导语：想再grafana上监控k8s

提前部署node_exporter

这个之前部署过了

提前部署kube-state-metrics

创建kube-state-metrics.json

{
  "apiVersion": "apps/v1",
  "kind": "Deployment",
  "metadata": {
    "name": "kube-state-metrics",
    "namespace": "kube-system"
  },
  "spec": {
    "selector": {
      "matchLabels": {
        "k8s-app": "kube-state-metrics",
        "grafanak8sapp": "true"
      }
    },
    "replicas": 1,
    "template": {
      "metadata": {
        "labels": {
          "k8s-app": "kube-state-metrics",
          "grafanak8sapp": "true"
        }
      },
      "spec": {
        "containers": [
          {
            "name": "kube-state-metrics",
            "image": "quay.io/coreos/kube-state-metrics:v1.1.0",
            "ports": [
              {
                "name": "http-metrics",
                "containerPort": 8080
              }
            ],
            "readinessProbe": {
              "httpGet": {
                "path": "/healthz",
                "port": 8080
              },
              "initialDelaySeconds": 5,
              "timeoutSeconds": 5
            }
          }
        ]
      }
    }
  }
}

修改prometheus的config

    - job_name: 'kubernetes-pods'
    - job_name: 'kubernetes-kubelet'
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics
    - job_name: 'kubernetes-cadvisor'
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
    - job_name: 'kubernetes-kube-state'
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: kubernetes_pod_name
      - source_labels: [__meta_kubernetes_pod_label_grafanak8sapp]
        regex: .*true.*
        action: keep
      - source_labels: ['__meta_kubernetes_pod_label_daemon', '__meta_kubernetes_pod_node_name']
        regex: 'node-exporter;(.*)'
        action: replace
        target_label: nodename

应用配置，重启prometheus的pod 使prometheus配置生效

安装kubernetes-app

已经安装并enable 这个插件

安装包下载地址: https://grafana.com/api/plugins/grafana-kubernetes-app/versions/1.0.1/download

cd /data/k8s/grafana/plugins
unzip grafana-kubernetes-app-31da28.zip

重启grafana的pod使插件生效 并enable 该插件

在Data Sources添加一个prometheus的源

我这里因为已经设置过了白名单 所以填一下URL就可以了

保存后 点击Kubernetes-app的图标 添加cluster

按图填写如下信息

apiserver 使用6443端口，以https形式提供服务。客户端访问apiserver需要认证客户端证书。该集群使用kubeadm 安装，会有一个/etc/kubernetes/admin.conf 文件，里面包含了客户端的证书和密码base64编码。复制 client-certificate-data和client-key-data的base64编码，分别执行 echo "<base64 code>" | base64 -d 就能还原成证书源文件。

通过以下命令获取相关信息

kubectl cluster-info

#CA Cert 
cat ~/.kube/config | grep certificate-authority-data | cut -d ' ' -f 6 | base64 -d
#Client Cert 
cat ~/.kube/config | grep client-certificate-data | cut -d ' ' -f 6 | base64 -d
#Client Key 
cat ~/.kube/config | grep client-key-data | cut -d ' ' -f 6 | base64 -d

点击save 第一次成功添加会提示绿色的成功 我这里点快了重复添加了

添加完成之后 在datasouces里可以看到它

我这里不知道是不是版本原因 也没报错 cluster界面一直是loading 就没继续下去了。没有自带的dashboard 不想自己去导入模版了。贴一下往网上的图

安装kubegraf

安装grafana插件并重启

# 进入pod安装
grafana-cli plugins install devopsprodigy-kubegraf-app
# 下载到插件文件夹解压也可
# cd /data/k8s/grafana/plugins/
# git clone https://github.com/devopsprodigy/kubegraf /var/lib/grafana/plugins/devopsprodigy-kubegraf-app # 这个后面是你plugins的路径

创建namespace及对应权限

kubectl create ns kubegraf
kubectl apply -f https://raw.githubusercontent.com/devopsprodigy/kubegraf/master/kubernetes/serviceaccount.yaml
kubectl apply -f https://raw.githubusercontent.com/devopsprodigy/kubegraf/master/kubernetes/clusterrole.yaml
kubectl apply -f https://raw.githubusercontent.com/devopsprodigy/kubegraf/master/kubernetes/clusterrolebinding.yaml
kubectl apply -f https://raw.githubusercontent.com/devopsprodigy/kubegraf/master/kubernetes/secret.yaml

创建private key and certificate

openssl genrsa -out ~/grafana-kubegraf.key 2048
openssl req -new -key ~/grafana-kubegraf.key -out ~/grafana-kubegraf.csr -subj "/CN=grafana-kubegraf/O=monitoring"
openssl x509 -req -in ~/grafana-kubegraf.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -out /etc/kubernetes/pki/grafana-kubegraf.crt -CAcreateserial

将证书拷贝到其余master

scp -r /etc/kubernetes/pki/grafana-kubegraf.crt prod-k8s-m002:/etc/kubernetes/pki/
scp -r /etc/kubernetes/pki/grafana-kubegraf.crt prod-k8s-m003:/etc/kubernetes/pki/

在grafana中enable 插件

添加cluster

填写对应信息及证书信息

添加成功

进入cluster界面

查看对应各个dashboard

参考地址

https://grafana.com/grafana/plugins/grafana-kubernetes-app/

https://grafana.com/grafana/plugins/devopsprodigy-kubegraf-app/