K8S测试/生产环境下部署metrics-server整理、记录

一、测试环境部署

1.1 获取yaml

wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

1.2 修改yaml

vim components.yaml
#部分内容如下
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  strategy:
    rollingUpdate:
      maxUnavailable: 0
  template:
    metadata:
      labels:
        k8s-app: metrics-server
    spec:
      containers:
      - args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        - --metric-resolution=15s
        #添加参数，禁用 TLS 验证
        - --kubelet-insecure-tls
        #修改镜像地址为阿里云
        image: registry.aliyuncs.com/google_containers/metrics-server:v0.6.4
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /livez
            port: https
            scheme: HTTPS
          periodSeconds: 10
        name: metrics-server
        ports:
        - containerPort: 4443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /readyz
            port: https
            scheme: HTTPS
          initialDelaySeconds: 20
          periodSeconds: 10
        resources:
          requests:
            cpu: 100m
            memory: 200Mi
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
        volumeMounts:
        - mountPath: /tmp
          name: tmp-dir
      nodeSelector:
        kubernetes.io/os: linux
      priorityClassName: system-cluster-critical
      serviceAccountName: metrics-server
      volumes:
      - emptyDir: {}
        name: tmp-dir

1.3 部署metrics-server服务

#部署服务
kubectl apply -f components.yaml
#验证服务
kubectl kubectl get apiservices |grep metrics
#内容如下， AVAILABLE显示True则正常
NAME                       SERVICE                      AVAILABLE   AGE
v1beta1.metrics.k8s.io     kube-system/metrics-server   True        4h28m
#查看节点指标
kubectl top nodes
#内容如下
NAME       CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
master01   1410m        70%    1216Mi          31%       
master02   1443m        72%    1188Mi          31%       
node01     531m         26%    591Mi           15%       
node02     489m         24%    637Mi           16%   

1.4 检查metrics-server的API Service是否正常

kubectl get apiservices |grep metrics

#排错参考

#官网
https://github.com/kubernetes-sigs/metrics-server
#CSDN
https://blog.csdn.net/qq_45097352/article/details/121509213
#阿里云
https://help.aliyun.com/document_detail/291404.html

二、生产环境部署

1.部署方式：kubeadm

2.k8s版本：v1.28.2

3.使用--kubelet-insecure-tls选项来配置Metrics Server会禁用 TLS 验证，这会降低安全性，因此不建议在生产环境中使用。

2.1 证书生成（参考二进制部署）

2.1.1 生成CA证书请求文件

vim ca-csr.json
#文件内容如下
{
  "CN": "kubernetes",
  "key": {
      "algo": "rsa",
      "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Hubei",
      "L": "Wuhan",
      "O": "k8s",
      "OU": "system"
    }
  ],
  "ca": {
          "expiry": "87600h"
  }
}
#执行生成
cfssl gencert -initca ca-csr.json  | cfssljson -bare ca

2.1.2 生成CA证书文件

vim ca-config.json
#文件内容如下
{
  "signing": {
      "default": {
          "expiry": "87600h"
        },
      "profiles": {
          "kubernetes": {
              "usages": [
                  "signing",
                  "key encipherment",
                  "server auth",
                  "client auth"
              ],
              "expiry": "87600h"
          }
      }
  }
}

2.1.3 生成metrics-server证书

vim metrics-server.json
#文件内容如下
{
  "CN": "metrics-server",  
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Hubei",
      "L": "Wuhan",
      "O": "k8s",
      "OU": "system"
    }
  ]
}
#生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes metrics-server.json | cfssljson -bare ./metrics-server

2.2 配置K8S服务

2.2.1 在master主机开启apiserver聚合服务

vim /etc/kubernetes/manifests/kube-apiserver.yaml
#spec.containers.command 在这一行下面手动添加如下一行
--enable-aggregator-routing=true

2.2.2 开启TLSBootstrap

2.2.2.1 修改config.yaml

vim /var/lib/kubelet/config.yaml
#所有节点（master和node）节点
serverTLSBootstrap: true

2.2.2.2 修改configmap

kubectl get configmap -n kube-system |grep kubelet-config
#内容如下
kubelet-config      1      22d
kubectl edit configmap kubelet-config -n kube-system
#添加内容同上

2.2.3 重启kubelet服务

systemctl restart kubelet

2.2.4 证书签发

#查看需要签发的证书
kubectl get certificatesigningrequests   或  kubectl get csr

#内容如下
NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION
csr-9j9r8 28s kubernetes.io/kubelet-serving system:node:master01 Pending
csr-sp7hc 28s kubernetes.io/kubelet-serving system:node:node01 Pending
csr-sqflv 28s kubernetes.io/kubelet-serving system:node:master02 Pending
csr-wcg7c 28s kubernetes.io/kubelet-serving system:node:node02 Pending

# 签发证书
kubectl certificate approve <CSR-名称>

2.2.5 RBAC配置

确保 Metrics Server 具有适当的 RBAC（Role-Based Access Control）权限，以便能够访问 kubelet API。在生产环境中，最好使用最小特权原则，只授予 Metrics Server 所需的权限。

vim metrics-server-rbac.yaml
#内容如下
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:metrics-server
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - nodes
  - nodes/stats
  - namespaces
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system

2.2.6 部署 Metrics Server

2.2.6.1 获取yaml

wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

2.2.6.2 修改yaml

#仅修改镜像地址，其他无需修改，内容如下
registry.aliyuncs.com/google_containers/metrics-server:v0.6.4

2.2.6.3 部署服务

kubectl apply -f metrics-server-deployment.yaml

2.2.7 服务验证

kubectl kubectl get apiservices |grep metrics
#内容如下， AVAILABLE显示True则正常
NAME                       SERVICE                      AVAILABLE   AGE
v1beta1.metrics.k8s.io     kube-system/metrics-server   True        4h28m
#查看节点指标
kubectl top nodes
#内容如下
NAME       CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
master01   1410m        70%    1216Mi          31%       
master02   1443m        72%    1188Mi          31%       
node01     531m         26%    591Mi           15%       
node02     489m         24%    637Mi           16%   

三、参考文献

http://www.javashuo.com/article/p-tfsnltqj-ke.html
https://blog.csdn.net/xhredeem/article/details/127314979
https://blog.csdn.net/vic_qxz/article/details/127569544