1. K8S使用系列篇(1) K8S集群监控(上)

1.1 使用Metrics Server监控K8S集群的CPU和内存使用数据

Metrics Server在部署完成后,将通过K8S核心API Server的/apis/metrics.k8s.io/v1beta1路径提供Node和Pod的监控数据。

Metrics Server：

• 用于基于CPU和内存的自动水平扩缩容(HPA)功能；
• 用于自动垂直扩缩容(VPA)功能。
• 用于提供核心指标(Core Metrics), 包括Node、Pod的CPU和内存使用指标。
• 是Kubernetes集群核心监控数据的聚合器。
• 可通过Metrics API的形式获取Metrics数据, 不进行存储和将指标转发给第三方服务。
• 可以与Kubectl工具结合使用,提供kubectl top命令来展示集群中的指标数据。

1.2 部署Metrics Server

Metrics Server的YAML配置如下:

1.2.1 Metrics的rbac策略

Kubernetes部署Metrics Server前需要先部署RBAC相关配置,Metrics Server有足够权限获取系统组件的信息。

$ cat metrics-rbac.yaml 
## ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
---
## ClusterRole aggregated-metrics-reader
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:aggregated-metrics-reader
  labels:
    k8s-app: metrics-server
    rbac.authorization.k8s.io/aggregate-to-view: "true"
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
rules:
- apiGroups: ["metrics.k8s.io"]
  resources: ["pods","nodes"]
  verbs: ["get","list","watch"]
---
## ClusterRole metrics-server
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:metrics-server
  labels:
    k8s-app: metrics-server
rules:
- apiGroups: [""]
  resources: ["pods","nodes","nodes/stats","namespaces","configmaps"]
  verbs: ["get","list","watch"]
---
## ClusterRoleBinding auth-delegator
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: metrics-server:system:auth-delegator
  labels:
    k8s-app: metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
## RoleBinding metrics-server-auth-reader
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: metrics-server-auth-reader
  namespace: kube-system
  labels:
    k8s-app: metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
## ClusterRoleBinding system:metrics-server
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:metrics-server
  labels:
    k8s-app: metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system

执行部署操作

$ kubectl apply -f metrics-rbac.yaml  -n kube-system

1.2.2 Metrics的APIService资源

设置扩展API Service工作于聚合层,允许使用其API扩展Kubernetes apiserver, 而这些API并不是核心 Kubernetes API的一部分。这里部署APIservice资源,来提供 Kubernetes Metrics指标API数据。

$ cat metrics-api-service.yaml 
## APIService
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  labels:
    k8s-app: metrics-server
  name: v1beta1.metrics.k8s.io
spec:
  group: metrics.k8s.io
  service:
    name: metrics-server
    namespace: kube-system
  version: v1beta1
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  versionPriority: 100

执行部署操作

$ kubectl apply -f metrics-api-service.yaml -n kube-system

1.2.3 metrics-server的pod部署

pod模板yaml文件如下:

$ cat metrics-server-deploy.yaml 
## Service
apiVersion: v1
kind: Service
metadata:
  labels:
    kubernetes.io/name: "Metrics-server"
    kubernetes.io/cluster-service: "true"
  name: metrics-server
  namespace: kube-system
spec:
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: main-port
  selector:
    k8s-app: metrics-server
---
## Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    k8s-app: metrics-server
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  strategy:
    rollingUpdate:
      maxUnavailable: 0
  template:
    metadata:
      name: metrics-server
      labels:
        k8s-app: metrics-server
    spec:
     # hostNetwork: true
      serviceAccountName: metrics-server
      containers:
      - name: metrics-server
        image: registry.aliyuncs.com/k8sxio/metrics-server:v0.3.7
        imagePullPolicy: IfNotPresent
        args:
          - --cert-dir=/tmp
          - --secure-port=4443
          - --kubelet-insecure-tls
          #- --kubelet-use-node-status-port
          - --kubelet-preferred-address-types=InternalDNS,InternalIP,ExternalDNS,ExternalIP,Hostname
        #livenessProbe:
        #  failureThreshold: 3
        #  httpGet:
        #    path: /livez
        #    port: https
        #    scheme: HTTPS
        #  periodSeconds: 10
        #readinessProbe:
        #  failureThreshold: 3
        #  httpGet:
        #    path: /readyz
        #    port: https
        #    scheme: HTTPS
        #  periodSeconds: 10
        ports:
        - name: main-port 
          containerPort: 4443
          protocol: TCP
        securityContext:
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
        resources:
          limits:
            memory: 1Gi
            cpu: 1000m
          requests:
            memory: 1Gi
            cpu: 1000m
        volumeMounts:
        - name: tmp-dir
          mountPath: /tmp
        - name: localtime
          readOnly: true
          mountPath: /etc/localtime
      volumes:
      - name: tmp-dir
        emptyDir: {}
      - name: localtime
        hostPath:
          type: File
          path: /etc/localtime
      nodeSelector:
        kubernetes.io/os: linux
        kubernetes.io/arch: "amd64"

执行部署操作

$ kubectl apply -f metrics-server-deploy.yaml -n kube-system

1.3 部署Metrics Server后的验证

• 通过标签匹配过滤出metrics-server的Pod
  执行命令:

$  kubectl get pod -l k8s-app=metrics-server -n kube-system

• 查看详细信息
  用describe命令查看明细:

$ kubectl describe pod metrics-server-9cd555b8c-rpsgq -n kube-system

• 查看pod日志

$ kubectl logs -f  metrics-server-9cd555b8c-rpsgq -n kube-system

• 部署完Metrics Server后,通过kubectl工具进行测试:

  • 获取Pod的CPU、Memory使用信息: kubectl top pod

  • 获取Node的CPU、Memory使用信息: kubectl top node
    

  • 获取全部节点指标信息: kubectl top node

  • 获取某个Namespace Pod的指标信息: kubectl top pods -n kube-system
    

  • 获取某个Namespace下某个Pod的指标信息: kubectl top pods coredns-7d75679df-rs8xc -n kube-system
    

  • 获取全部Namespace下的Pod的指标信息: kubectl top pods --all-namespaces