Prometheus支持的多种服务发现机制

• static_configs: 静态服务发现
• file_sd_configs: 文件服务发现
• dns_sd_configs: DNS 服务发现
• kubernetes_sd_configs: Kubernetes 服务发现
• consul_sd_configs: Consul 服务发现

安装参考

安装方法 请参阅我的两外博文： Prometheus 监控Kubernetes_叱咤少帅（少帅）的博客-CSDN博客_prometheus监控kubernetes

常规配置

Kube-prometheus 项目中对其他资源对象的管理是通过 Prometheus-Operator 项目实例来实现的；所以在管理和更新 Prometheus 服务是需要通过 Operator 来实现。
我们在使用 Kube-prometheus 项目配置 Prometheus 监控后，难免会进行一些调整，增加一些自定义的配置，这时候我们就需要更深入了解 Kube-prometheus 项目是如何配置 Prometheus 服务，并给我们预设什么样的调整配置方式。

### 配置文件在 secret 中
kubectl get secret -n monitoring

## 查询endpoint
kubectl get ep -n monitoring

prometheus-operator 文档参考

###
https://www.qikqiak.com/post/prometheus-operator-monitor-etcd/

##  工作原理参考
https://www.cnblogs.com/precipitation/p/15980967.html

##
https://www.cnblogs.com/wang-hongwei/p/15694644.html

##
https://help.aliyun.com/document_detail/123394.html

prometheus-operator 结合consul

在monitoring名称空间部署consul服务

为了监控Kafka，Mysql等等explorer客户端，我们结合consul去实现。vim consul.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: consul
  namespace: monitoring
  labels:
    app: consul
spec:
  selector:
    matchLabels:
      app: consul
  replicas: 1
  template:
    metadata:
      name: consul
      labels:
        app: consul
    spec:
      containers:
        - name: consul
          image: consul:1.6.1
          ports:
            - containerPort: 8500
              protocol: TCP
          imagePullPolicy: IfNotPresent
---
apiVersion: v1
kind: Service
metadata:
  name: consul
  namespace: monitoring
  labels:
    app: consul
spec:
  type: NodePort
  selector:
    app: consul
  ports:
    - name: consul
      protocol: TCP
      port: 8500
      targetPort: 8500
      nodePort: 30300

编写prometheus-additional并在monitoring名称空间创建secret资源

vim prometheus-additional.yaml

- job_name: consul
  consul_sd_configs:
    - server: consul.monitoring.svc:8500
  relabel_configs:
    - source_labels: [__meta_consul_tags]
      regex: .*,prome,.*
      action: keep
    - source_labels: [__meta_consul_service]
      target_label: job

如果是虚拟机的consul:

- job_name: 'consul-node'
  consul_sd_configs:
  - server: '192.168.10.13:8500'
    datacenter: 'dc1'
    scheme: http
    services: ['node-exporter']
  relabel_configs:
    - regex: __meta_consul_service_metadata_(.+)
      action: labelmap
    - source_labels: ["__meta_consul_service_id"]
      target_label: "id"

写入：

## 
kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml --dry-run=client -oyaml > additional-scrape-configs.yaml

##
kubectl apply -f additional-scrape-configs.yaml -n monitoring

##

kubectl get  secret  -n monitoring |grep additional

手动修改prometheus-prometheus.yaml

手动修改kube-prometheus/manifests中的prometheus-prometheus.yaml文件增加additionalScrapeConfigs配置段

vim kube-prometheus/manifests/prometheus-prometheus.yaml

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    app.kubernetes.io/component: prometheus
    app.kubernetes.io/instance: k8s
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 2.36.1
  name: k8s
  namespace: monitoring
spec:
  alerting:
    alertmanagers:
    - apiVersion: v2
      name: alertmanager-main
      namespace: monitoring
      port: web
  enableFeatures: []
  externalLabels: {}
  image: quay.io/prometheus/prometheus:v2.36.1
  nodeSelector:
    kubernetes.io/os: linux
  podMetadata:
    labels:
      app.kubernetes.io/component: prometheus
      app.kubernetes.io/instance: k8s
      app.kubernetes.io/name: prometheus
      app.kubernetes.io/part-of: kube-prometheus
      app.kubernetes.io/version: 2.36.1
  podMonitorNamespaceSelector: {}
  podMonitorSelector: {}
  probeNamespaceSelector: {}
  probeSelector: {}
  replicas: 2
  resources:
    requests:
      memory: 400Mi
  ruleNamespaceSelector: {}
  ruleSelector: {}
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  version: 2.36.1
  additionalScrapeConfigs:
    name: additional-configs
    key: prometheus-additional.yaml