我们的k8s集群是二进制部署,版本是1.20.4 同时选择的kube-prometheus版本是kube-prometheus-0.8.0

一、prometheus添加自定义监控与告警（etcd）

1、步骤及注意事项（前提，部署参考部署篇）

1.1 一般etcd集群会开启HTTPS认证，因此访问etcd需要对应的证书

1.2 使用证书创建etcd的secret

1.3 将etcd的secret挂在到prometheus

1.4创建etcd的servicemonitor对象（匹配kube-system空间下具有k8s-app=etcd标签的service）

1.5 创建service关联被监控对象

2、操作部署

2.1 创建etcd的secret

ETC的自建证书路径：/opt/etcd/ssl，cd /opt/etcd/ssl

kubectl create secret generic etcd-certs --from-file=server.pem --from-file=server-key.pem --from-file=ca.pem -n monitoring

可以用下面的命令验证下是否有内容产出，由内存说明是没有问题的

 curl --cert /opt/etcd/ssl/server.pem --key /opt/etcd/ssl/server-key.pem  https://192.168.7.108:2379/metrics -k |more

可以进入容器查看，查看证书挂载进去了

[root@master manifests]# kubectl exec -it -n monitoring prometheus-k8s-0 /bin/sh

/prometheus $ ls /etc/prometheus/secrets/etcd-certs/

ca.pem          server-key.pem  server.pem

2.2 添加secret到名为k8s的prometheus对象上（kubectl edit prometheus k8s -n monitoring或者修改yaml文件并更新资源）

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    prometheus: k8s
  name: k8s
  namespace: monitoring
spec:
  alerting:
    alertmanagers:
    - name: alertmanager-main
      namespace: monitoring
      port: web
  baseImage: quay.io/prometheus/prometheus
  nodeSelector:
    kubernetes.io/os: linux
  podMonitorNamespaceSelector: {}
  podMonitorSelector: {}
  replicas: 2
  secrets:
  - etcd-certs
  resources:
    requests:
      memory: 400Mi
  ruleSelector:
    matchLabels:
      prometheus: k8s
      role: alert-rules
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  version: v2.11.0

或者可以直接找到配置文件更新vim prometheus-prometheus.yaml

kubectl replace -f prometheus-prometheus.yaml

3、创建servicemonitoring对象

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: etcd1 #这个serviceMonitor的标签
  name: etcd
  namespace: monitoring
spec:
  endpoints:
  - interval: 30s
    port: etcd     #port名字就是service里面的spec.ports.name
    scheme: https  #访问的方式
    tlsConfig:
      caFile: /etc/prometheus/secrets/etcd-certs/ca.pem #证书位置/etc/prometheus/secrets,这个路径是默认的挂载路径
      certFile: /etc/prometheus/secrets/etcd-certs/server.pem
      keyFile: /etc/prometheus/secrets/etcd-certs/server-key.pem
  selector:
    matchLabels:
      k8s-app: etcd1
  namespaceSelector:
    matchNames:
    - monitoring  #匹配的命名空间

4、创建service并自定义endpoint

---
apiVersion: v1
kind: Endpoints
metadata:
  labels:
    k8s-app: etcd1
  name: etcd
  namespace: monitoring
subsets:
- addresses:
  - ip: 192.168.7.108
  - ip: 192.168.7.109
  - ip: 192.168.7.106
  ports:
  - name: etcd  #name
    port: 2379  #port
    protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: etcd1
  name: etcd
  namespace: monitoring
spec:
  ports:
  - name: etcd
    port: 2379
    protocol: TCP
    targetPort: 2379
  sessionAffinity: None
  type: ClusterIP

kubectl replace -f prometheus-prometheus.yaml

kubectl apply -f servicemonitor.yaml

kubectl apply -f service.yam

到这里就可以在prometheus中查看到etcd的监控信息了

添加告警

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    prometheus: k8s
    role: alert-rules
  name: etcd-rules
  namespace: monitoring
spec:
  groups:
  - name: etcd-exporter.rules
    rules:
    - alert: EtcdClusterUnavailable
      annotations:
        summary: etcd cluster small
        description: If one more etcd peer goes down the cluster will be unavailable
      expr: |
        count(up{job="etcd"} == 0) > (count(up{job="etcd"}) / 2-1)
      for: 3m
      labels:
        severity: critical

二、prometheus添加自定义监控与告警（kube-controller-manager）

Kube-prometheus默认是配置了kube-controller-manager的servicemonitor的，但是因为我们是二进制部署的，所以无法找到对应的kube-contorller-manager的service和endpoints,所以这里我们需要自己去手动创建service和endpoints

kubectl get servicemonitor -n monitoring

通过查看servicemonitor去查看需要匹配的service的labels

 vim kubernetes-serviceMonitorKubeControllerManager.yaml

以看到他是通过标签app.kubernetes.io/name=kube-controller-manager来匹配controller-manager的当我们查看的时候，并没有符合这个标签的svc所以prometheus找不到controller-manager地址。

好了，下面我们就开始创建service和endpoints了

创建service

首先创建一个endpoint，指向宿主机ip+10252，然后在创建一个同名的service，和上面查出来的标签

---
apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    app.kubernetes.io/name: kube-controller-manager
  name: kube-controller-manager-monitoring
  namespace: kube-system
subsets:
- addresses:
  - ip: 192.168.7.100
  ports:
  - name: https-metrics
    port: 10252
    protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: kube-controller-manager
  name: kube-controller-manager-monitoring
  namespace: kube-system
spec:
  ports:
  - name: https-metrics
    port: 10252
    protocol: TCP
    targetPort: 10252
  sessionAffinity: None
  type: ClusterIP

#注：kube-prometheus使用的是https、而暴露使用的是http，将https改成http

kubectl edit servicemonitor -n monitoring kube-controller-manager

     60     scheme: http

配置完成后就可以再prometheus界面查看到监控信息了，这样就是成功了

三、prometheus添加自定义监控与告警（kube-scheduler）

Kube-scheduler的配置和kube-controller-manager的配置类似

kubectl edit servicemonitor -n monitoring kube-scheduler

#scheme: https 改为scheme: http

好了，下面我们就开始创建service和endpoints了

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: kube-scheduler
  name: scheduler
  namespace: kube-system
spec:
  ports:
  - name: https-metrics
    port: 10251
    protocol: TCP
    targetPort: 10251
  sessionAffinity: None
  type: ClusterIP
---
apiVersion: v1
kind: Endpoints
metadata:
  labels:
    app.kubernetes.io/name: kube-scheduler
  name: scheduler
  namespace: kube-system
subsets:
- addresses:
  - ip: 192.168.7.100
  ports:
  - name: https-metrics
    port: 10251
    protocol: TCP

Kubectl apply -f svc-kube-scheduler.yaml

配置完成后就可以再prometheus界面查看到监控信息了，这样就是成功了

至此 controller-manager、scheduler 已经起来

备注：看到不少的资料都是说需要修改kube-controller-manager和kube-scheduler的监听地址，从127.0.0.1修改成0.0.0.0

但是因为我的配置中kube-controller-manager的配置文件一开是就是0.0.0.0所以没有修改，kube-scheduler的配置文件监听地址是127.0.0.1也没有进行修改但是依然成功了，所以这里还有待验证。