监控指标

Kubernetes本身监控

• Node资源利用率

• Node数量

• Pods数量（Node）

• 资源对象状态

Pod监控

• Pod数量（项目）

• 容器资源利用率

• 应用程序

准备一个kubernetes集群，目前只有一个集群。

Kubernetes是在Prometheus之外部署的。

准备就绪

监控K8s集群Pod步骤：

1、K8s RBAC授权
kubectl apply -f rbac.yaml

监控过程

监控kubernetes需要接口授权。直接访问kubernetes接口是不可以的。

授权第一步：
上传kubernetes授权文件到k8s节点：rabc.yaml文件

文件内容

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - services
  - endpoints
  - pods
  - nodes/proxy
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - "extensions"
  resources:
    - ingresses
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - configmaps
  - nodes/metrics
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: kube-system 

2、获取Token并保存到文件
kubectl get sa prometheus -n kube-system -o yaml
kubectl describe secret prometheus-token-xxx -n kube-system

获取token name

获取token

把token重定向到一个文件（token.k8s）：

把token.k8s上传到Prometheus节点上：

Prometheus节点上只保留token值:

3、创建Job和kubeconfig_sd_configs

书写prometheus.yml配置文件，内容如下：

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']

  - job_name: kubernetes-nodes-cadvisor
    metrics_path: /metrics
    scheme: https
    kubernetes_sd_configs:
    - role: node
    # 访问的api地址
      api_server: https://192.168.31.61:6443
      bearer_token_file: /opt/monitor/prometheus/token.k8s 
       # 忽略证书的校验
      tls_config:
        insecure_skip_verify: true
    bearer_token_file: /opt/monitor/prometheus/token.k8s 
    tls_config:
      insecure_skip_verify: true
    relabel_configs:
    # 将标签(.*)作为新标签名，原有值不变
    - action: labelmap
      regex: __meta_kubernetes_node_label_(.*)
    # 修改NodeIP:10250为APIServerIP:6443
    - action: replace
      regex: (.*)
      source_labels: ["__address__"]
      target_label: __address__
      replacement: 192.168.31.61:6443
    # 实际访问指标接口 https://NodeIP:10250/metrics/cadvisor 这个接口只能APISERVER访问，故此重新标记标签使用APISERVER代理访问
    - action: replace
      source_labels: [__meta_kubernetes_node_name]
      target_label: __metrics_path__
      regex: (.*)
      replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor

检查配置是否正确：

4、Grafana导入仪表盘

监控K8s资源对象状态步骤：

1、部署kube-state-metrics

在prometheus.yml文件中添加如下pod配置：

  - job_name: kubernetes-service-endpoints
    kubernetes_sd_configs:
    - role: endpoints
      api_server: https://192.168.31.61:6443
      bearer_token_file: /opt/monitor/prometheus/token.k8s
      tls_config:
        insecure_skip_verify: true
    bearer_token_file: /opt/monitor/prometheus/token.k8s
    tls_config:
      insecure_skip_verify: true
    # Service没配置注解prometheus.io/scrape的不采集
    relabel_configs:
    - action: keep
      regex: true
      source_labels:
      - __meta_kubernetes_service_annotation_prometheus_io_scrape
    # 重命名采集目标协议
    - action: replace
      regex: (https?)
      source_labels:
      - __meta_kubernetes_service_annotation_prometheus_io_scheme
      target_label: __scheme__
    # 重命名采集目标指标URL路径
    - action: replace
      regex: (.+)
      source_labels:
      - __meta_kubernetes_service_annotation_prometheus_io_path
      target_label: __metrics_path__
    # 重命名采集目标地址
    - action: replace
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2
      source_labels:
      - __address__
      - __meta_kubernetes_service_annotation_prometheus_io_port
      target_label: __address__
    # 将K8s标签(.*)作为新标签名，原有值不变
    - action: labelmap
      regex: __meta_kubernetes_service_label_(.+)
    # 生成命名空间标签
    - action: replace
      source_labels:
      - __meta_kubernetes_namespace
      target_label: kubernetes_namespace
    # 生成Service名称标签
    - action: replace
      source_labels:
      - __meta_kubernetes_service_name
      target_label: kubernetes_service_name

  - job_name: kubernetes-pods
    kubernetes_sd_configs:
    - role: pod
      api_server: https://192.168.31.61:6443
      bearer_token_file: /opt/monitor/prometheus/token.k8s
      tls_config:
        insecure_skip_verify: true
    bearer_token_file: /opt/monitor/prometheus/token.k8s
    tls_config:
      insecure_skip_verify: true
    # 重命名采集目标协议
    relabel_configs:
    - action: keep
      regex: true
      source_labels:
      - __meta_kubernetes_pod_annotation_prometheus_io_scrape
    # 重命名采集目标指标URL路径
    - action: replace
      regex: (.+)
      source_labels:
      - __meta_kubernetes_pod_annotation_prometheus_io_path
      target_label: __metrics_path__
    # 重命名采集目标地址
    - action: replace
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2
      source_labels:
      - __address__
      - __meta_kubernetes_pod_annotation_prometheus_io_port
      target_label: __address__
    # 将K8s标签(.*)作为新标签名，原有值不变
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)
    # 生成命名空间标签
    - action: replace
      source_labels:
      - __meta_kubernetes_namespace
      target_label: kubernetes_namespace
    # 生成Service名称标签
    - action: replace
      source_labels:
      - __meta_kubernetes_pod_name
      target_label: kubernetes_pod_name

配置路由规则让Prometheus可以访问到Kubernetes节点。

2、Grafana导入仪表盘