知识结构梳理

• HPA
  • metrics api
• API aggregation
  • metrics.k8s.io
    • metrics-server
  • custom.metrics.k8s.io
    • prometheus adapter
  • external.metrics.k8s.io
• 安装扩展API服务器

链接资料
https://www.cnblogs.com/wjoyxt/p/10003159.html

31. API聚合层 + metrics-server

一、metrics-server

1. 获取metrics-server

github：https://github.com/kubernetes-sigs/metrics-server
github获取k8s部署资源配置的参考清单，这里用的是V0.3.6版本，我们配置清单版本也选择这个版本（资源配置清单因具体情况还需一些小调整）
https://github.com/kubernetes-sigs/metrics-server/tree/release-0.3/deploy/1.8%2B

拉取metrics-server镜像
docker pull mirrorgooglecontainers/metrics-server-amd64:v0.3.6

2. 配置清单（原始清单）

• aggregated-metrics-reader.yaml

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: system:aggregated-metrics-reader
  labels:
    rbac.authorization.k8s.io/aggregate-to-view: "true"
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
rules:
- apiGroups: ["metrics.k8s.io"]
  resources: ["pods", "nodes"]
  verbs: ["get", "list", "watch"]

• auth-delegator.yaml

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: metrics-server:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system

• auth-reader.yaml

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: metrics-server-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system

• resource-reader.yaml

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:metrics-server
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - nodes
  - nodes/stats
  - namespaces
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system

• metrics-apiservice.yaml

---
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
  name: v1beta1.metrics.k8s.io
spec:
  service:
    name: metrics-server
    namespace: kube-system
  group: metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100

配置APIService资源，该资源是将apiserver的metrics.k8s.io转给metrics-server服务。HPA或其他组件访问apiserver的metrics.k8s.io接口时，apiserver会转访问metrics

• metrics-server-service.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    kubernetes.io/name: "Metrics-server"
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    k8s-app: metrics-server
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443

• metrics-server-deployment.yaml

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metrics-server
  namespace: kube-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    k8s-app: metrics-server
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  template:
    metadata:
      name: metrics-server
      labels:
        k8s-app: metrics-server
    spec:
      serviceAccountName: metrics-server
      volumes:
      # mount in tmp so we can safely use from-scratch images and/or read-only containers
      - name: tmp-dir
        emptyDir: {}
      containers:
      - name: metrics-server
        image: harbor.hzwod.com/k8s/metrics-server-amd64:v0.3.6
        imagePullPolicy: Always
        volumeMounts:
        - name: tmp-dir
          mountPath: /tmp

具体deployment部署metrics-server容器时，需添加一下自定的配置，我们下文详细介绍

3. metrics-server的关键配置说明

我们可以通过下条命名查看metrics-server的帮助文档：
docker run --rm harbor.hzwod.com/k8s/metrics-server-amd64:v0.3.6 /metrics-server -h
我们主要关注这几个参数

• --kubelet-preferred-address-types metrics默认使用hostname来通信的(默认值是[Hostname,InternalDNS,InternalIP,ExternalDNS,ExternalIP]),若我们某些节点无法通过hostname来访问，那我们可以将该参数直接设置成InternalIP，让metrics使用ip访问各个节点
• --kubeconfig=xxxx.kubeconfig 若我们集群中kubelet使用了自定义的证书控制访问，那么metrics访问kubelet可能会被拒绝，通过这个参数可以给metrics提供一个kubeconfig文件，保存了用k8s同CA证书签发的client证书，用于和kubelet通信
• --kubelet-insecure-tls 不校验kubelets出示的服务证书CA，只建议测试用
• --cert-dir 指定本地的TLS证书目录，若设置了下面两个参数则这个参数会被忽略，否则metrics会在该目录下生成自签证书
• --tls-cert-file、--tls-provate-key-file 配置metrics提供HTTPS服务的证书

4. 配置准备

• 准备kubeconfig文件
  参考部署kubelet时kubeconfig的制备过程，关键是配置文件中的证书和k8s集群证书源于同一个CA签发。这里这里直接使用部署kubelet时制备的k8s-node.kubeconfig文件
• 将kubeconfig文件存入scerts资源
  kubectl -n kube-system create secret generic k8snode-kubeconfig-secrets --from-file=/xxx/k8s-node.kubeconfig
k8snode-kubeconfig-secrets是secrets资源名称，下文就是通过这个名称挂载引用的
• deployment配置修改

...
      volumes:
      - name: k8snode-kubeconfig
        secret:
          secretName: k8snode-kubeconfig-secrets
      - name: tmp-dir
        emptyDir: {}
      containers:
      - name: metrics-server
        image: harbor.hzwod.com/k8s/metrics-server-amd64:v0.3.6
        imagePullPolicy: Always
        command:
          - /metrics-server
          - --kubelet-preferred-address-types=InternalIP
          - --kubeconfig=k8s-node.kubeconfig
          - --cert-dir=/tmp
        volumeMounts:
        - name: k8snode-kubeconfig
          mountPath: /kubeconfig
        - name: tmp-dir
          mountPath: /tmp

至此，我们的metrics就基本配置完成，我们将资源应用到k8s中；
但此时，上面配置的APIServer可能并不能正常访问metrics，如下所示

~]# kubectl get apiservice
v1beta1.metrics.k8s.io          kube-system/metrics-server    False (FailedDiscoveryCheck)
...

接下来我们要看看怎么修改kubenetes的启动参数来使用API聚合功能

二、kube-apiserver API聚合层（还在编写中）

Aggregation Layer（聚合层）可以允许kube apiserver使用其他API扩展，

1. kube-apiserver添加启动参数

    --requestheader-client-ca-file=/xxx/ca.pem
    --requestheader-allowed-names=aggregator
    --requestheader-extra-headers-prefix=X-Remote-Extra-
    --requestheader-group-headers=X-Remote-Group
    --requestheader-username-headers=X-Remote-User
    --proxy-client-cert-file=/xxx/proxy-client.pem
    --proxy-client-key-file=/xxx/proxy-client-key.pem

三、异常

启动后异常

异常二

通过metrics-server参数--kubeconfig配置上kubeconfig文件（内有证书信息）
kubeconfig通过scerts类资源保存到etcd中，再挂载到metrics-server容器中使用