【K8S系列】部署Prometheus+Grafana

yunqi1215

116人浏览 · 2026-01-23 12:57:26

yunqi1215 · 2026-01-23 12:57:26 发布

1、创建命名空间，使用helm添加Prometheus仓库

[root@k8s-master ~]# kubectl create namespace monitoring 

namespace/monitoring created 

[root@k8s-master ~]# helm repo add prometheus-community https://prometheus-community.github.io/helm-charts 

"prometheus-community" has been added to your repositories 

[root@k8s-master ~]# helm repo update 

Hang tight while we grab the latest from your chart repositories... 

...Successfully got an update from the "prometheus-community" chart repository 

...Successfully got an update from the "stable" chart repository 

...Successfully got an update from the "bitnami" chart repository 

Update Complete. ⎈Happy Helming!⎈

2、helm安装Prometheus完整监控（包括grafana，alertmanager）

[root@k8s-master ~]# helm install prometheus prometheus-community/kube-prometheus-stack \ 

>   --namespace monitoring \ 

>   --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \ 

>   --set grafana.adminPassword=admin \ 

>   --set prometheus.service.type=NodePort \ 

>   --set prometheus.service.nodePort=30090 \ 

>   --set grafana.service.type=NodePort \ 

>   --set grafana.service.nodePort=30091 \ 

>   --set alertmanager.service.type=NodePort \ 

>   --set alertmanager.service.nodePort=30092 

NAME: prometheus 

LAST DEPLOYED: Fri Jan 16 02:22:22 2026 

NAMESPACE: monitoring 

STATUS: deployed 

REVISION: 1 

NOTES: 

kube-prometheus-stack has been installed. Check its status by running: 

  kubectl --namespace monitoring get pods -l "release=prometheus" 

  

Get Grafana 'admin' user password by running: 

  

  kubectl --namespace monitoring get secrets prometheus-grafana -o jsonpath="{.data.admin-password}" | base64 -d ; echo 

  

Access Grafana local instance: 

  

  export POD_NAME=$(kubectl --namespace monitoring get pod -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=prometheus" -oname) 

  kubectl --namespace monitoring port-forward $POD_NAME 3000 

  

Get your grafana admin user password by running: 

  

  kubectl get secret --namespace monitoring -l app.kubernetes.io/component=admin-secret -o jsonpath="{.items[0].data.admin-password}" | base64 --decode ; echo 

  

  

Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.

登录信息：

admin/admin123

3、查看资源

[root@k8s-master ~]# kubectl get pods,svc,ep -n monitoring  

NAME                                                         READY   STATUS    RESTARTS   AGE 

pod/alertmanager-prometheus-kube-prometheus-alertmanager-0   2/2     Running   0          2m48s 

pod/prometheus-grafana-8bdd974f8-m2hmr                       3/3     Running   0          2m53s 

pod/prometheus-kube-prometheus-operator-784bff49d9-6jkrf     1/1     Running   0          2m53s 

pod/prometheus-kube-state-metrics-df8f49986-7xnrc            1/1     Running   0          2m53s 

pod/prometheus-prometheus-kube-prometheus-prometheus-0       2/2     Running   0          2m47s 

pod/prometheus-prometheus-node-exporter-5rcbz                1/1     Running   0          2m53s 

pod/prometheus-prometheus-node-exporter-j6dlz                1/1     Running   0          2m53s 

  

NAME                                              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE 

service/alertmanager-operated                     ClusterIP   None             <none>        9093/TCP,9094/TCP,9094/UDP      2m48s 

service/prometheus-grafana                        NodePort    10.107.228.134   <none>        80:30091/TCP                    2m54s 

service/prometheus-kube-prometheus-alertmanager   NodePort    10.99.78.217     <none>        9093:30092/TCP,8080:31446/TCP   2m54s 

service/prometheus-kube-prometheus-operator       ClusterIP   10.96.168.49     <none>        443/TCP                         2m54s 

service/prometheus-kube-prometheus-prometheus     NodePort    10.111.6.231     <none>        9090:30090/TCP,8080:31135/TCP   2m54s 

service/prometheus-kube-state-metrics             ClusterIP   10.107.156.237   <none>        8080/TCP                        2m54s 

service/prometheus-operated                       ClusterIP   None             <none>        9090/TCP                        2m47s 

service/prometheus-prometheus-node-exporter       ClusterIP   10.97.4.199      <none>        9100/TCP                        2m54s 

  

NAME                                                ENDPOINTS                                                  AGE 

endpoints/alertmanager-operated                     172.20.64.244:9094,172.20.64.244:9094,172.20.64.244:9093   2m48s 

endpoints/prometheus-grafana                        172.28.82.194:3000                                         2m54s 

endpoints/prometheus-kube-prometheus-alertmanager   172.20.64.244:8080,172.20.64.244:9093                      2m54s 

endpoints/prometheus-kube-prometheus-operator       172.28.82.255:10250                                        2m54s 

endpoints/prometheus-kube-prometheus-prometheus     172.28.82.254:9090,172.28.82.254:8080                      2m54s 

endpoints/prometheus-kube-state-metrics             172.28.82.195:8080                                         2m54s 

endpoints/prometheus-operated                       172.28.82.254:9090                                         2m47s 

endpoints/prometheus-prometheus-node-exporter       192.168.85.133:9100,192.168.85.134:9100                    2m54s

4、由于是使用二进制安装K8S集群，没有打开metrics，导致Prometheus识别不了，所以测试方式是使用匿名访问并打开metrics

sudo vi /etc/systemd/system/kube-scheduler.service 

# 在 ExecStart 中添加： 
--bind-address=0.0.0.0 
--secure-port=10259 
--authorization-always-allow-paths=/metrics  

  

sudo vi /etc/systemd/system/kube-controller-manager.service 

# 在 ExecStart 中添加： 
--bind-address=0.0.0.0 
--secure-port=10257 
--authorization-always-allow-paths=/metrics  

  

[root@k8s-master ~]# systemctl daemon-reload  
[root@k8s-master ~]# systemctl restart kube-scheduler.service kube-controller-manager.service

5、给kube-scheduler和kube-controller-manager添加监控的服务

[root@k8s-master ~]# cat > binary-control-plane-service.yaml <<'EOF' 

> apiVersion: v1 

> kind: Service 

> metadata: 

>   name: kube-controller-manager-metrics 

>   namespace: kube-system 

>   labels: 

>     k8s-app: kube-controller-manager 

> spec: 

>   type: ExternalName 

>   externalName: 192.168.85.133  # 你的 Master IP 

>   ports: 

>   - name: https-metrics 

>     port: 10257 

>     targetPort: 10257 

>     protocol: TCP 

> --- 

> apiVersion: v1 

> kind: Endpoints 

> metadata: 

>   name: kube-controller-manager-metrics 

>   namespace: kube-system 

>   labels: 

>     k8s-app: kube-controller-manager 

> subsets: 

> - addresses: 

>   - ip: 192.168.85.133  # 你的 Master IP 

>   ports: 

>   - name: https-metrics 

>     port: 10257 

>     protocol: TCP 

> --- 

> apiVersion: v1 

> kind: Service 

> metadata: 

>   name: kube-scheduler-metrics 

>   namespace: kube-system 

>   labels: 

>     k8s-app: kube-scheduler 

> spec: 

>   type: ExternalName 

>   externalName: 192.168.85.133  # 你的 Master IP 

>   ports: 

>   - name: https-metrics 

>     port: 10259 

>     targetPort: 10259 

>     protocol: TCP 

> --- 

> apiVersion: v1 

> kind: Endpoints 

> metadata: 

>   name: kube-scheduler-metrics 

>   namespace: kube-system 

>   labels: 

>     k8s-app: kube-scheduler 

> subsets: 

> - addresses: 

>   - ip: 192.168.85.133  # 你的 Master IP 

>   ports: 

>   - name: https-metrics 

>     port: 10259 

>     protocol: TCP 

> EOF 

[root@k8s-master ~]# kubectl apply -f binary-control-plane-service.yaml 

service/kube-controller-manager-metrics created 

endpoints/kube-controller-manager-metrics created 

service/kube-scheduler-metrics created 

endpoints/kube-scheduler-metrics created 



[root@k8s-master ~]# cat > binary-control-plane-servicemonitor.yaml <<'EOF' 

> apiVersion: monitoring.coreos.com/v1 

> kind: ServiceMonitor 

> metadata: 

>   name: binary-kube-controller-manager 

>   namespace: monitoring 

>   labels: 

>     release: prometheus 

> spec: 

>   jobLabel: k8s-app 

>   endpoints: 

>   - port: https-metrics 

>     scheme: https 

>     tlsConfig: 

>       insecureSkipVerify: true  # 跳过 TLS 验证 

>     interval: 30s 

>     honorLabels: true 

>   selector: 

>     matchLabels: 

>       k8s-app: kube-controller-manager 

>   namespaceSelector: 

>     matchNames: 

>     - kube-system 

> --- 

> apiVersion: monitoring.coreos.com/v1 

> kind: ServiceMonitor 

> metadata: 

>   name: binary-kube-scheduler 

>   namespace: monitoring 

>   labels: 

>     release: prometheus 

> spec: 

>   jobLabel: k8s-app 

>   endpoints: 

>   - port: https-metrics 

>     scheme: https 

>     tlsConfig: 

>       insecureSkipVerify: true  # 跳过 TLS 验证 

>     interval: 30s 

>     honorLabels: true 

>   selector: 

>     matchLabels: 

>       k8s-app: kube-scheduler 

>   namespaceSelector: 

>     matchNames: 

>     - kube-system 

> EOF 


[root@k8s-master ~]# kubectl apply -f binary-control-plane-servicemonitor.yaml 

servicemonitor.monitoring.coreos.com/binary-kube-controller-manager created 

servicemonitor.monitoring.coreos.com/binary-kube-scheduler created

6、查看监控，重启Prometheus

[root@k8s-master ~]# kubectl get prometheus -n monitoring 

NAME                                    VERSION   DESIRED   READY   RECONCILED   AVAILABLE   AGE 

prometheus-kube-prometheus-prometheus   v3.9.1    1         1       True         True        21m 


[root@k8s-master ~]# kubectl rollout restart statefulset prometheus-prometheus-kube-prometheus-prometheus -n monitoring 

statefulset.apps/prometheus-prometheus-kube-prometheus-prometheus restarted 


[root@k8s-master ~]# kubectl get pods -n monitoring            

NAME                                                     READY   STATUS    RESTARTS   AGE 

alertmanager-prometheus-kube-prometheus-alertmanager-0   2/2     Running   0          21m 

prometheus-grafana-8bdd974f8-m2hmr                       3/3     Running   0          21m 

prometheus-kube-prometheus-operator-784bff49d9-6jkrf     1/1     Running   0          21m 

prometheus-kube-state-metrics-df8f49986-7xnrc            1/1     Running   0          21m 

prometheus-prometheus-kube-prometheus-prometheus-0       1/2     Running   0          11s 

prometheus-prometheus-node-exporter-5rcbz                1/1     Running   0          21m 

prometheus-prometheus-node-exporter-j6dlz                1/1     Running   0          21m

7、同理kube-proxy也需要打开metrics

[root@k8s-master prome]# cat /etc/kubernetes/kube-proxy.yaml                     

apiVersion: kubeproxy.config.k8s.io/v1alpha1 

bindAddress: 0.0.0.0 

clientConnection: 

  acceptContentTypes: "" 

  burst: 10 

  contentType: application/vnd.kubernetes.protobuf 

  kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig 

  qps: 5 

clusterCIDR: 172.16.0.0/12,fc00::/48  

configSyncPeriod: 15m0s 

conntrack: 

  max: null 

  maxPerCore: 32768 

  min: 131072 

  tcpCloseWaitTimeout: 1h0m0s 

  tcpEstablishedTimeout: 24h0m0s 

enableProfiling: false 

healthzBindAddress: 0.0.0.0:10256 

hostnameOverride: "" 

iptables: 

  masqueradeAll: false 

  masqueradeBit: 14 

  minSyncPeriod: 0s 

  syncPeriod: 30s 

ipvs: 

  masqueradeAll: true 

  minSyncPeriod: 5s 

  scheduler: "rr" 

  syncPeriod: 30s 

kind: KubeProxyConfiguration 

metricsBindAddress: 0.0.0.0:10249 

mode: "ipvs" 

nodePortAddresses: null 

oomScoreAdj: -999 

portRange: "" 

udpIdleTimeout: 250ms

8、给kube-proxy添加监控服务

[root@k8s-master prome]# cat kube-proxy-monitor.yaml  

apiVersion: v1 

kind: Service 

metadata: 

  name: kube-proxy-metrics 

  namespace: kube-system 

  labels: 

    k8s-app: kube-proxy 

spec: 

  type: ClusterIP 

  clusterIP: None 

  ports: 

  - name: http-metrics 

    port: 10249 

    targetPort: 10249 

    protocol: TCP 

--- 

apiVersion: v1 

kind: Endpoints 

metadata: 

  name: kube-proxy-metrics 

  namespace: kube-system 

  labels: 

    k8s-app: kube-proxy 

subsets: 

- addresses: 

  - ip: 192.168.85.133  # 你的 Master IP，也是 Node IP 

  ports: 

  - name: http-metrics 

    port: 10249 

    protocol: TCP 

[root@k8s-master prome]# cat kube-proxy-servicemonitor.yaml  

apiVersion: monitoring.coreos.com/v1 

kind: ServiceMonitor 

metadata: 

  name: binary-kube-proxy 

  namespace: monitoring 

  labels: 

    release: prometheus 

spec: 

  jobLabel: k8s-app 

  endpoints: 

  - port: http-metrics 

    scheme: http 

    interval: 30s 

    honorLabels: true 

  selector: 

    matchLabels: 

      k8s-app: kube-proxy 

  namespaceSelector: 

    matchNames: 

    - kube-system

9、同理etcd也打开metrics，不使用证书

[root@k8s-master prome]# cat /etc/etcd/etcd.config.yml                                                                      

name: 'master' 

data-dir: /var/lib/etcd 

wal-dir: /var/lib/etcd/wal 

snapshot-count: 5000 

heartbeat-interval: 100 

election-timeout: 1000 

quota-backend-bytes: 0 

listen-peer-urls: 'https://192.168.85.133:2380' 

listen-client-urls: 'https://0.0.0.0:2379,http://0.0.0.0:2379' 

max-snapshots: 3 

max-wals: 5 

cors: 

initial-advertise-peer-urls: 'https://192.168.85.133:2380' 

advertise-client-urls: 'https://192.168.85.133:2379' 

discovery: 

discovery-fallback: 'proxy' 

discovery-proxy: 

discovery-srv: 

initial-cluster: 'master=https://192.168.85.133:2380' 

initial-cluster-token: 'etcd-k8s-cluster' 

initial-cluster-state: 'new' 

strict-reconfig-check: false 

enable-v2: true 

enable-pprof: true 

proxy: 'off' 

proxy-failure-wait: 5000 

proxy-refresh-interval: 30000 

proxy-dial-timeout: 1000 

proxy-write-timeout: 5000 

proxy-read-timeout: 0 

client-transport-security: 

  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem' 

  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem' 

  client-cert-auth: true 

  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem' 

  auto-tls: true 

peer-transport-security: 

  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem' 

  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem' 

  peer-client-cert-auth: true 

  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem' 

  auto-tls: true 

debug: false 

log-package-levels: 

log-outputs: [default] 

force-new-cluster: false

10、给etcd添加监控服务

[root@k8s-master prome]# cat etcd-monitor.yaml  

apiVersion: v1 

kind: Service 

metadata: 

  name: etcd-metrics 

  namespace: kube-system 

  labels: 

    k8s-app: etcd 

spec: 

  type: ExternalName 

  externalName: 192.168.85.133  # 你的 etcd 节点 IP 

  ports: 

  - name: https-metrics 

    port: 2379 

    targetPort: 2379 

    protocol: TCP 

--- 

apiVersion: v1 

kind: Endpoints 

metadata: 

  name: etcd-metrics 

  namespace: kube-system 

  labels: 

    k8s-app: etcd 

subsets: 

- addresses: 

  - ip: 192.168.85.133  # 你的 etcd 节点 IP 

  ports: 

  - name: https-metrics 

    port: 2379 

    protocol: TCP 



[root@k8s-master prome]# cat etcd-servicemonitor.yaml  

apiVersion: monitoring.coreos.com/v1 

kind: ServiceMonitor 

metadata: 

  name: binary-etcd 

  namespace: monitoring 

  labels: 

    release: prometheus 

spec: 

  jobLabel: k8s-app 

  endpoints: 

  - port: https-metrics 

    scheme: http 

    interval: 30s 

    honorLabels: true 

  selector: 

    matchLabels: 

      k8s-app: etcd 

  namespaceSelector: 

    matchNames: 

    - kube-system