1 prometheus简介

prometheus是由 SoundCloud 开源监控告警解决方案，是云原生时代最火热的监控系统之一。

详细功能可以参看官网文档。

2 prometheus部署

本文主要讲述prometheus的部署，云原生当然是用k8s来部署，所以本文不会讲源码安装、二进制安装等，先从docker 安装说起，k8s后面再讲。

2.1 部署组件

prometheus不只有prometheus，而是有一些列的组件，本次使用docker-compose部署prometheus、node_export、grafana、alertmanager、promtail和loki一套完整prometheus监控体系。

2.2 docker-compose.yml

version: '2'
 
networks:
    monitor:
        driver: bridge
 
services:
    prometheus:
        image: prom/prometheus
        container_name: prometheus
        hostname: prometheus
        restart: always
        volumes:
            - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
            - ./prometheus/node_down.yml:/etc/prometheus/node_down.yml
        ports:
            - "9090:9090"
        networks:
            - monitor
 
    alertmanager:
        image: prom/alertmanager
        container_name: alertmanager
        hostname: alertmanager
        restart: always
        volumes:
            - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml
        ports:
            - "9093:9093"
        networks:
            - monitor
 
    node-exporter:
        image: quay.io/prometheus/node-exporter
        container_name: node-exporter
        hostname: node-exporter
        restart: always
        ports:
            - "9100:9100"
        networks:
            - monitor
 
    pushgateway:
        image: prom/pushgateway
        container_name: pushgateway
        restart: always
        ports:
            - 9091:9091
        networks:
            - monitor

    loki:
       image: grafana/loki
       container_name: loki
       restart: always
       ports:
           - "3100:3100"
       volumes:
           - ./loki:/etc/loki
       command: -config.file=/etc/loki/local-config.yaml
       networks:
            - monitor

    promtail:
       image: grafana/promtail
       container_name: promtail
       restart: always
       volumes:
           - ./promtail:/etc/promtail
           - /home/blockchain-test/e-invoice/logs:/opt
       command: -config.file=/etc/promtail/docker-config.yaml
       networks:
            - monitor
            
    grafana:
        image: grafana/grafana
        container_name: grafana
        hostname: grafana
        restart: always
        ports:
            - "3000:3000"
        depends_on:
            - loki
            - promtail
        volumes:
            - ./grafana:/var/lib/grafana
        networks:
            - monitor

	# mysql的exporter，可选
    mysql-exporter:
        image: prom/mysqld-exporter
        container_name: mysql-exporter
        hostname: mysql-exporter
        restart: always
        ports:
            - "9104:9104"
        networks:
            - monitor
        environment:
            DATA_SOURCE_NAME: "root:123456@(192.168.0.1:3306)/"

	# redis的exporter，可选
    redis-exporter:
        image: oliver006/redis_exporter
        container_name: redis_exporter
        hostname: redis_exporter
        restart: always
        ports:
            - "9121:9121"
        networks:
            - monitor
        command:
            - '--redis.addr=redis://192.168.0.1:6379'
            - '--redis.password=123456'

描述文件中，部署了各个组件，暴露了相应的端口，挂载对应的目录主要是配置文件。

2.3 配置文件

prometheus配置文件：

prometheus.yml

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets: ['192.168.0.1:9093'] #alertmanager地址
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  #- "node_down.yml"
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  # prometheus抓取的监控的信息
  - job_name: 'prometheus'
    scrape_interval: 8s
    static_configs:
      - targets: ['192.168.0.1:9090']
      - targets: ['192.168.0.1:9100','192.168.0.2:9100']
        labels:
          group: 'client-node-exporter'
      - targets: ['192.168.0.1:9091']
        labels:
          group: 'client-pushgateway'
      - targets: ['192.168.0.1:9104']
        labels:
          group: 'client-mysql'
        

  # prometheus抓取redis监控的信息
  - job_name: 'redis_exporter'
    static_configs:
      - targets:
        - 192.168.0.1:9121
        
  # prometheus抓取GPU监控的信息      
  - job_name: 'dcgm' 
    static_configs:
      - targets: ['192.168.0.3:9400']
  - job_name: 'nvidia_gpu_exporter'
    static_configs:
      - targets: ['192.168.0.4:9835']

prometheus.yml主要配置prometheus抓取的监控信息，只要配置对应的exporter暴露的指标，都能被prometheus获取到。

node_down.yml：自定义的告警规则文件，文件中配置主机宕机和某个服务down 了之后的告警。可以根据需要自行配置。

groups:
- name: node_down
  rules:
  - alert: InstanceDown
    expr: up == 0
    for: 1m
    labels:
      user: test
    annotations:
      summary: "Instance {{ $labels.instance }} down"
      description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 1 minutes."
- name: node-up
  rules:
  - alert: node-up
    expr: up{job="xxx"} == 0
    for: 15s
    labels:
      severity: 1
      team: node
    annotations:
      summary: "{{ $labels.instance }} 已停止运行超过 15s！"

alertmanager配置文件：

alertmanager.yml，主要配置告警的频率，告警的邮箱等。除了邮件告警之外，还阔以配置短信、微信、钉钉等。

global:
  resolve_timeout: 5m
  smtp_from: 'xxx@126.com'
  smtp_smarthost: 'smtp.126.com:25'
  smtp_auth_username: 'xxxx@126.com'
  smtp_auth_password: 'SFSFEVCDFDGD'
  smtp_require_tls: false
  smtp_hello: 'xxx监控告警'

route:
  group_by: ['alertname']
  group_wait: 5s
  group_interval: 5s
  repeat_interval: 5m
  receiver: 'email'
receivers:
  - name: 'email'
    email_configs:
    - to: 'xxxxx@qq.com'
      send_resolved: true
    - to: 'cccccc@qq.com'
      send_resolved: true
    - to: 'dddddd@qq.com'
      send_resolved: true

loki配置：

loki是个轻量级的日志采集工具，原理prometheus几乎是一模一样的，有点就是轻量级了，如果只需要是看日志的话很适合用这个，没必要上ELK。

local-config.yaml，基本是官方的配置，如有需要自行修改。

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 5m
  chunk_retain_period: 30s
  max_transfer_retries: 0

schema_config:
  configs:
    - from: 2018-04-15
      store: boltdb
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 168h

storage_config:
  boltdb:
    directory: /loki/index

  filesystem:
    directory: /loki/chunks

limits_config:
  enforce_metric_name: false
  reject_old_samples: true 
  reject_old_samples_max_age: 168h
                           
chunk_store_config:        
  max_look_back_period: 0s 
                            
table_manager:              
  retention_deletes_enabled: false
  retention_period: 0s   

promtail配置

promtail和exporter类似，部署到要采集日志主机上，通过tail的方式去抓取日志，然后发到loki。

配置文件主要是配置要抓取的日志的服务和路径.

docker-config.yaml，抓取日志的配置：

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log # 抓取系统日志
- job_name: xxx
  static_configs:
  - targets:
      - localhost
    labels:
      job: xxx
      __path__: /opt/console/*log # 抓取某应用控制台日志
- job_name: xxx1
  static_configs:
  - targets:
      - localhost
    labels:
      job: xxxInfo
      __path__: /opt/info/*log # 抓取某应用info日志
- job_name: xxx2
  static_configs:
  - targets:
      - localhost
    labels:
      job: xxxError
      __path__: /opt/error/*log # 抓取某应用error日志

local-config.yaml ，默认配置。

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs1
      __path__: /var/log/*log

2.4 启动

完成描述文件和配置文件之后，就可以启动了。

docker-compose up -d

3 使用prometheus

输入ip+对应的端口打开应用

prometheus：http://192.168.0.1:9090/，可以自定义一些数据的图标。

查看相应的Targets信息等。

alertmanager http://192.168.0.1:9093/#/alerts

可以查看、配置、屏蔽一些告警信息。

xxx-exporter 通过对应的端口，可以查看exporter信息

主要是一些数据Metrics信息，这些数据看起来并不方便，所以可以使用grafana

grafana http://192.168.0.1:3000/login

登录进入之后，配置prometheus数据源，可以自定义一些监控图标，也可以去grafana的商店下载一些dashboard。详细grafana的使用自行了解。

Loki，Loki也集成在grafana中，切换数据源即可

选择对应的job，查看日志：

到此prometheus部署完成。