promehteus 监控超时_Promethues（八）监控kubernetes

prometheus相关的服务都部署在monitoring这个namespace下部署prometheus服务创建namespace$ cd /opt/k8s/prometheus$ cat>1-namespace.yml<apiVersion: v1kind: Namespacemetadata:name: monitoringEOF创建prometheus对应的配置文件，利用kub

尤文文

393人浏览 · 2020-12-23 03:27:24

尤文文 · 2020-12-23 03:27:24 发布

prometheus相关的服务都部署在monitoring这个namespace下

部署prometheus服务

创建namespace

$ cd /opt/k8s/prometheus

$ cat>1-namespace.yml<

apiVersion: v1

kind: Namespace

metadata:

EOF

创建prometheus对应的配置文件，利用kubernetes的ConfigMap

$ cd /opt/k8s/prometheus

$ cat>2-prom-cnfig.yml<

apiVersion: v1

kind: ConfigMap

metadata:

namespace: monitoring

data:

prometheus.yml: |

global:

scrape_interval: 15s

scrape_timeout: 15s

scrape_configs:

- job_name: 'prometheus'

static_configs:

- targets: ['localhost:9090']

EOF

创建用来存储prometheus数据的pv、pvc(采用本地存储、可以通过搭建NFS、GlusterFs等分布式文件系统代替)

$ cd /opt/k8s/prometheus

$ cat>3-prom-pv.yml<

kind: PersistentVolume

apiVersion: v1

metadata:

namespace: monitoring

labels:

type: local

app: prometheus

spec:

capacity:

storage: 10Gi

accessModes:

- ReadWriteOnce

hostPath:

path: /opt/k8s/prometheus/data

---

kind: PersistentVolumeClaim

apiVersion: v1

metadata:

namespace: monitoring

spec:

accessModes:

- ReadWriteOnce

resources:

requests:

storage: 10Gi

EOF

创建启动prometheus的文件，以deployment形式部署，外部访问通过NodePort类型的service

$ cd /opt/k8s/prometheus

$ cat>4-prometheus.yml<

apiVersion: apps/v1

kind: Deployment

metadata:

namespace: monitoring

labels:

app: prometheus

spec:

selector:

matchLabels:

app: prometheus

replicas: 1

template:

metadata:

labels:

app: prometheus

spec:

containers:

- name: prometheus

image: prom/prometheus:v2.16.0

args:

- '--config.file=/etc/prometheus/prometheus.yml'

- '--storage.tsdb.path=/prometheus'

- "--storage.tsdb.retention=7d"

- "--web.enable-lifecycle"

ports:

- containerPort: 9090

volumeMounts:

- mountPath: "/prometheus"

subPath: prometheus

- mountPath: "/etc/prometheus"

resources:

requests:

cpu: 500m

memory: 2Gi

limits:

cpu: 500m

memory: 2Gi

volumes:

- name: config

configMap:

- name: data

persistentVolumeClaim:

claimName: prometheus-claim

---

apiVersion: v1

kind: Service

metadata:

namespace: monitoring

spec:

type: NodePort

ports:

- port: 9090

targetPort: 9090

nodePort: 9090

selector:

app: prometheus

EOF

在Prometheus的启动命令中，传入了参数storage.tsdb.path、storage.tsdb.retention分别指定了Prometheus数据存储路径、存储时间

web.enable-lifecycle 当配置信息变更后通过/-/reload重新加载新的配置内容，不用重启服务

启动Prometheus服务

$ cd /opt/k8s/prometheus

$ mkdir data & chmod -R 777 data

$ kubectl create -f 1-namespace.yml -f 2-prom-cnfig.yml -f 3-prom-pv.yml -f 4-prometheus.yml

查看组件状态，确保所有服务正常启动

$ kubectl get all -n monitoring

NAME READY STATUS RESTARTS AGE

pod/prometheus-57cf64764d-xqnvl 1/1 Running 0 51s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

service/prometheus NodePort 10.254.209.164 9090:9090/TCP 51s

NAME READY UP-TO-DATE AVAILABLE AGE

deployment.apps/prometheus 1/1 1 1 51s

NAME DESIRED CURRENT READY AGE

replicaset.apps/prometheus-57cf64764d 1 1 1 51s

界面访问

和kubernetes监控相关的技术

cAdvisor google 开源的一款容器监控方案，收集容器自身的各种资源使用、性能相关信息，通过resutful API的形式提供给外部。kubernetes把cAdvisor对应的功能集成到了kubelet中，所以不需要再单独部署，直接从kubelet获取，并且kubelet还对容器信息进行summary，以pod为单位供外部调用。

metrics-server 是kubernetes核心监控流程中的一个组件，是Heapster的替代方案，从kubelet指标接口中获取信息，主要是CPU、Memory，再通过API server暴露出去。主要供kubectl top、HPA等kubernetes组件使用。只在内存中存储最后一次获取到的指标信息，不负责数据存储

kube-state-metrics 通过监听 API Server 生成有关资源对象的状态指标，比如 Deployment、replica sets等。只在内存中存储最后一次获取到的指标信息，不负责数据存储

node-exporter Prometheus官方提供的，专门用来收集*NIX系统自身、以及对应硬件的指标信息

kube-prometheus 一站式的kubernetes监控方案，将node-exporter、prometheus、kube-state-metrics、Grafana、metrics-server等组件收集起来，提供了更加便捷的脚本供使用者快速搭建一个完整的监控平台。

kubernetes监控内容

对集群自身状态的监控，如节点自身的CPU、Memory、IO、Network等信息

kubernetes系统自身组件的监控如 kube-schedule-manager、kube-proxy、kubelet等

集群中运行容器的监控，如容器、Pod等为单元的CPU、Memory信息

集群中编排组件对应的指标监控，如Deployment、Daemonset等本文采用自己部署组件的形式，看如何一步一步搭建一个监控平台。对于有快速搭建需求的，可以参考 kube-prometheus

部署node-exporter

因为要监控每一个节点，所以采用Daemonset控制器来部署node-exporter，在每个节点上都运行一个Pod

启动文件

$ cd /opt/k8s/prometheus

$ cat>5-node-exporter.yml<

apiVersion: apps/v1

kind: DaemonSet

metadata:

labels:

app: node-exporter

namespace: monitoring

spec:

selector:

matchLabels:

app: node-exporter

template:

metadata:

labels:

app: node-exporter

spec:

containers:

- name: node-exporter

image: 192.168.0.107/prometheus/node-exporter:v0.18.1

args:

- --web.listen-address=:9100

- --path.procfs=/host/proc

- --path.sysfs=/host/sys

- --path.rootfs=/host/root

- --no-collector.hwmon

- --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/)

resources:

limits:

cpu: 250m

memory: 180Mi

requests:

cpu: 102m

memory: 180Mi

ports:

- containerPort: 9100

volumeMounts:

- mountPath: /host/proc

readOnly: false

- mountPath: /host/sys

readOnly: false

- mountPath: /host/root

mountPropagation: HostToContainer

readOnly: true

hostNetwork: true

hostPID: true

nodeSelector:

kubernetes.io/os: linux

securityContext:

runAsNonRoot: true

runAsUser: 65534

tolerations:

- operator: Exists

volumes:

- hostPath:

path: /proc

- hostPath:

path: /sys

- hostPath:

path: /

EOF

启动node-exporter

$ cd /opt/k8s/prometheus

$ kubectl create -f 5-node-exporter.yml

$ kubectl -n monitoring get pod | grep node

node-exporter-854vr 1/1 Running 6 50m

node-exporter-lv9pv 1/1 Running 0 50m

通过prometheus收集node-exporter的指标信息

因为集群的节点之后可能会动态扩容和缩减，所以不便采用静态配置的形式，Prometheus给我们提供了Kubernetes对应的服务发现功能，可以实现对Kubernetes的动态监控。其中对节点的监控利用node的服务发现方式，在prometheus的配置文件中追加如下配置(对应的2-prom-cnfig.yml也需要追加，否则Configmap重建后这些信息就丢掉了)

$ kubectl -n monitoring edit configmaps prom-config

- job_name: "kubernetes-nodes"

kubernetes_sd_configs:

- role: node

relabel_configs:

- source_labels: [__address__]

regex: '(.*):10250'

replacement: '${1}:9100'

target_label: __address__

action: replace

- action: labelmap

regex: __meta_kubernetes_node_label_(.+)

追加完成后执行如下命令，重新加载配置项，至于为何这样配置，后面章节配置原理具体讲解

$ curl -XPOST http://192.168.0.107:9090/-/reload

此时，prometheus就会尝试获取集群node信息，查看promethes的日志信息，会发现如下错误提示

level=error ts=2020-03-22T10:37:13.856Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:333: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:monitoring:default" cannot list resource "nodes" in API group "" at the cluster scope"

意思是用默认的serviceaccount不能 list *v1.Node，因此需要我们为Prometheus重新创建serviceaccount，并赋予相应的权限

创建prometheus对应的serviceaccount,并赋予相应的权限

$ cd /opt/k8s/prometheus

$ cat>6-prometheus-serivceaccount-role.yaml<

apiVersion: v1

kind: ServiceAccount

metadata:

namespace: monitoring

---

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole

metadata:

rules:

- apiGroups: [""]

resources:

- nodes/proxy

- nodes

- namespaces

- endpoints

- pods

- services

verbs: ["get","list","watch"]

- apiGroups: [""]

resources:

- nodes/metrics

verbs: ["get"]

- nonResourceURLs:

- /metrics

verbs: ["get"]

- apiGroups:

- extensions

resources:

- ingresses

verbs: ["get", "list", "watch"]

---

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRoleBinding

metadata:

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: ClusterRole

subjects:

- kind: ServiceAccount

namespace: monitoring

EOF

$ cd /opt/k8s/prometheus

$ kubectl create -f 6-prometheus-serivceaccount-role.yaml

修改Prometheus的启动yaml，追加serivceaccount配置,重启Prometheus

...

spec:

serviceAccountName: prometheus-k8s

containers:

- name: prometheus

...

查看prometheus监控对象列表

目标地址查找

当kubernetes_sd_config配置的role是node时，prometheus启动后会调用kubernetes的 LIST Node API获取node相关信息，并从node对象中获取IP和PORT来构成监控地址。

其中IP获取按照如下顺序InternalIP, ExternalIP, LegacyHostIP, HostName查找

Port值默认采用Kubelet的HTTP port。

通过如下命令可查看 list Node接口返回的IP 和 Port信息

$ kubectl get node -o=jsonpath='{range .items[*]}{.status.addresses}{"t"}{.status.daemonEndpoints}{"n"}{end}'

[map[address:192.168.0.107 type:InternalIP] map[address:master type:Hostname]] map[kubeletEndpoint:map[Port:10250]]

[map[address:192.168.0.114 type:InternalIP] map[address:slave type:Hostname]] map[kubeletEndpoint:map[Port:10250]]

上述返回结果表示集群中有两个节点，构成的taget地址分别是

192.168.0.107:10250

192.168.0.114:10250

relabe_configs

Relabeling可以让Prometheus在抓取数据之前动态的修改标签的值。Prometheus有许多默认标签，其中下面几个和我们处理有关

__address__:初始化时会设置成目标地址对应的:

instance:__address__标签的值经过Relabel阶段后，会设置给标签instance, 即instance是__address__标签经过Relabel后的值

__scheme__:默认值 http

__metrics_path__ 默认值 /metrics

Prometheus拉取指标信息的目的地址是把这几个标签连接起来__scheme__://instance/__metrics_path__

我们启动node-expeorter后，在各个节点的:9100/metrics上暴露了node指标信息，然后在prometheus中追加了如下配置段

- job_name: "kubernetes-nodes"

kubernetes_sd_configs:

- role: node

relabel_configs:

- source_labels: [__address__]

regex: '(.*):10250'

replacement: '${1}:9100'

target_label: __address__

action: replace

- action: labelmap

regex: __meta_kubernetes_node_label_(.+)

其中relabel_configs的第一个配置片段

- source_labels: [__address__]

regex: '(.*):10250'

replacement: '${1}:9100'

target_label: __address__

action: replace

通过regex从__address__中匹配出IP地址

replacement：对应的值设置成 ${IP}:9100

target_label: 将__address__的值替换成replacement，即 ${IP}:9100

经过这些步骤后，拼接成的获取指标地址为[http://192.168.0.107:9100/metrics, http://192.168.0.114:9100/metrics],和我们的node-exporter暴露的指标地址匹配，就可以拉取Node的指标信息了

另外因为prometheus会将node对应的标签变成__meta_kubernetes_node_label_，所以追加了一个labelmap的动作，再把这些标签名字还原出来

追加收集kubelete提供的指标

kubelet会收集一些api server 、etcd等服务的指标信息,可以通过如下命令查看

$ kubectl get --raw https://192.168.0.107:10250/metrics

其中10250是kubelet的默认监听端口

接下来通过在promehteus中追加配置，让promehteus拉取这些信息

- job_name: "kubernetes-kubelet"

scheme: https

tls_config:

ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

kubernetes_sd_configs:

- role: node

relabel_configs:

- action: labelmap

regex: __meta_kubernetes_node_label_(.+)

追加收集cAdvisor指标,实现对集群容器的监控

kubelet默认集成了cAdvisor，用来对集群中容器信息进行收集，Kubernetes 1.7.3以及之后的版本中，把cAdvisor收集的指标信息(以container_开头)从Kubelet对应的/metrics中移除了，所以需要额外配置一个收集cAdvisor的job。cAdvisor指标调用命令

$ kubectl get --raw https://192.168.0.107:6443/api/v1/nodes/master/proxy/metrics/cadvisor

其中10250是kubelet的默认监听端口

接下来通过在promehteus中追加配置，让promehteus拉取这些信息

- job_name: "kubernetes-cadvisor"

scheme: https

tls_config:

ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

kubernetes_sd_configs:

- role: node

relabel_configs:

- action: labelmap

regex: __meta_kubernetes_node_label_(.+)

- target_label: __address__

replacement: kubernetes.default.svc:443

- source_labels: [__meta_kubernetes_node_name]

regex: (.+)

target_label: __metrics_path__

replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor

添加完成后重新load 配置文件，查看prometheus监控对象列表

完整的配置文件如下

$ cd /opt/k8s/prometheus

$ cat 2-prom-cnfig.yml

apiVersion: v1

kind: ConfigMap

metadata:

namespace: monitoring

data:

level=info ts=2020-03-22T12:15:02.551Z caller=head.go:625 component=tsdb msg="WAL segment loaded" segment=10 maxSegment=15

prometheus.yml: |

global:

scrape_interval: 15s

scrape_timeout: 15s

scrape_configs:

- job_name: 'prometheus'

static_configs:

- targets: ['localhost:9090']

- job_name: "kubernetes-nodes"

kubernetes_sd_configs:

- role: node

relabel_configs:

- source_labels: [__address__]

regex: '(.*):10250'

replacement: '${1}:9100'

target_label: __address__

action: replace

- action: labelmap

regex: __meta_kubernetes_node_label_(.+)

- job_name: "kubernetes-kubelet"

scheme: https

tls_config:

ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

kubernetes_sd_configs:

- role: node

relabel_configs:

- action: labelmap

regex: __meta_kubernetes_node_label_(.+)

- job_name: "kubernetes-cadvisor"

scheme: https

tls_config:

ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

kubernetes_sd_configs:

- role: node

relabel_configs:

- action: labelmap

regex: __meta_kubernetes_node_label_(.+)

- target_label: __address__

replacement: kubernetes.default.svc:443

- source_labels: [__meta_kubernetes_node_name]

regex: (.+)

target_label: __metrics_path__

replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor

部署grafana

创建用来存储grafana数据的pv、pvc(采用本地存储、可以通过搭建NFS、GlusterFs等分布式文件系统代替)

$ cd /opt/k8s/prometheus

$ cat>7-grafana-pv.yml<

kind: PersistentVolume

apiVersion: v1

metadata:

namespace: monitoring

labels:

type: local

app: grafana

spec:

capacity:

storage: 10Gi

accessModes:

- ReadWriteOnce

hostPath:

path: /opt/k8s/prometheus/grafana-pvc

---

kind: PersistentVolumeClaim

apiVersion: v1

metadata:

namespace: monitoring

spec:

accessModes:

- ReadWriteOnce

resources:

requests:

storage: 10Gi

EOF

grafana部署文件

$ cd /opt/k8s/prometheus

$ cat>8-grafana.yml<

apiVersion: apps/v1

kind: Deployment

metadata:

labels:

app: grafana

namespace: monitoring

spec:

replicas: 1

selector:

matchLabels:

app: grafana

template:

metadata:

labels:

app: grafana

spec:

containers:

- image: grafana/grafana:6.6.2

ports:

- containerPort: 3000

readinessProbe:

httpGet:

path: /api/health

port: http

resources:

limits:

cpu: 200m

memory: 400Mi

requests:

cpu: 100m

memory: 200Mi

volumeMounts:

- mountPath: /var/lib/grafana

readOnly: false

subPath: data

- mountPath: /etc/grafana/provisioning/datasources

readOnly: false

subPath: datasources

- mountPath: /etc/grafana/provisioning/dashboards

readOnly: false

subPath: dashboards-pro

- mountPath: /grafana-dashboard-definitions/0

readOnly: false

subPath: dashboards

nodeSelector:

beta.kubernetes.io/os: linux

securityContext:

runAsNonRoot: true

runAsUser: 65534

volumes:

- name: grafana-pvc

persistentVolumeClaim:

claimName: grafana-claim

---

apiVersion: v1

kind: Service

metadata:

namespace: monitoring

spec:

type: NodePort

ports:

- port: 3000

targetPort: 3000

nodePort: 3000

selector:

app: grafana

EOF

启动grafana

创建grafana挂载目录

$ cd /opt/k8s/prometheus

$ mkdir -p grafana-pvc/data

$ mkdir -p grafana-pvc/datasources

$ mkdir -p grafana-pvc/dashboards-pro

$ mkdir -p grafana-pvc/dashboards

$ chmod -R 777 grafana-pvc

data目录存放grafana的数据

datasources存放预定义的数据源

dashboards-pro存放dashboards管理文件，其中配置的dashboard的文件地址指向grafana-pvc/dashboards挂载到容器中的地址/grafana-dashboard-definitions/0

dashboards存放真正的dashboards定义文件(json)

创建默认的数据源文件

$ cd /opt/k8s/prometheus/grafana-pvc/datasources

$ cat > datasource.yaml<

apiVersion: 1

datasources:

- name: Prometheus

type: prometheus

access: proxy

url: http://prometheus.monitoring.svc:9090

EOF

创建默认的dashboards管理文件

$ cd /opt/k8s/prometheus/grafana-pvc/dashboards-pro

$ cat >dashboards.yaml<

apiVersion: 1

providers:

- name: '0'

orgId: 1

folder: ''

type: file

editable: true

updateIntervalSeconds: 10

allowUiUpdates: false

options:

path: /grafana-dashboard-definitions/0

EOF

创建默认的dashboard定义文件

可以到 a collection of shared dashboards,找到自己需要的dashboard模版，下载对应的json文件将对应文件存放到/opt/k8s/prometheus/grafana-pvc/dashboards),这里作为示例，下载1 Node Exporter for Prometheus Dashboard CN v20191102,对应的ID是8919。

$ cd /opt/k8s/prometheus/grafana-pvc/dashboards

$ wget https://grafana.com/api/dashboards/8919/revisions/11/download -o node-exporter-k8s.json

因为模版中的数据源默认用的是${DS_PROMETHEUS_111},从界面导入时有配置项供替换，我们直接下载json文件，所以通过直接修改文件，把数据源改成我们在/opt/k8s/prometheus/grafana-pvc/datasources下配置的数据源

$ cd /opt/k8s/prometheus/grafana-pvc/dashboards

$ sed -i "s/${DS_PROMETHEUS_111}/Prometheus/g" node-exporter-k8s.json

修改title

...

"timezone": "browser",

"title": "k8s-node-monitoring",

...

启动

$ cd /opt/k8s/prometheus/

$ kubectl create -f 7-grafana-pv.yml 8-grafana.yml

通过界面查看，因为我们已经默认设置过数据源、dashboard等信息，可以直接查看对应的dashboard

在部署grafana时，我们配置了默认的数据源、dashboard等信息，主要是为了实现，系统部署后这些默认监控指标可以直接观察，不需要实施人员现场配置。

其他的监控例如使用 Kube-state-metrics以及cAdvisor metrics实现对集群中Deployment、StatefulSet、容器、pod的监控也可以采用这种形式来实现。如可以利用1. Kubernetes Deployment Statefulset Daemonset metrics作为模版，稍微修改满足我们的监控需要，这里就不再展示具体步骤，读者可以自行尝试。

内容来源于网络如有侵权请私信删除

K8S/Kubernetes

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐

【深度】阿里巴巴万级规模 K8s 集群全局高可用体系之美

作者 | 韩堂、柘远、沉醉来源 | 阿里巴巴云原生公众号前言台湾作家林清玄在接受记者采访的时候，如此评价自己 30 多年写作生涯：“第一个十年我才华横溢，‘贼光闪现’，令周边黯然失色；第二个十年，我终于‘宝光现形’，不再去抢风头，反而与身边的美丽相得益彰；进入第三个十年，繁华落尽见真醇，我进入了‘醇光初现’的阶段，真正体味到了境界之美”。长夜有穷，真水无香。领略过了 K8s“身在江

K8S/Kubernetes

如何基于 K8s 构建下一代 DevOps 平台？

作者 | 孙健波（天元）导读：当前云原生 DevOps 体系现状如何？面临哪些挑战？如何通过 OAM 解决云原生 DevOps 场景下的诸多问题？云原生开发应用模型 OAM(Open Application Model) 社区核心成员孙健波将为大家一一解答，并分享如何基于 OAM 和 Kubernetes 打造无限能力的下一代 DevOps 平台。什么是 DevOps？为什么基于 Kub