个人博客原文： kubernetes(k8s)中部署metrics-server遇到的问题

metrics-server版本：k8s.gcr.io/metrics-server-amd64:v0.3.3
k8s版本：v1.14.1

The '--source' flag is unavailable right now (v0.3.0-alpha.1)
containers:
 - name: metrics-server
 image: k8s.gcr.io/metrics-server-amd64:v0.2.1
 command:
 - /metrics-server
 - --source=kubernetes.summary_api:''?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250&insecure=true
 - --requestheader-allowed-names=
所以这种方式不能使用

应用metrics-server的yaml文件后，查看metrics-server-v0.3.3-6cbfd7955f-v29n7状态为CrashLoopBackOff

[root@master metrics-server]# kubectl get pods -n kube-system 
NAME READY STATUS RESTARTS AGE
coredns-fb8b8dccf-9bnvc 1/1 Running 3 20d
coredns-fb8b8dccf-n5bzb 1/1 Running 3 20d
etcd-master 1/1 Running 4 20d
kube-apiserver-master 1/1 Running 4 20d
kube-controller-manager-master 1/1 Running 5 20d
kube-proxy-cmmbw 1/1 Running 4 20d
kube-proxy-dnw9v 1/1 Running 4 20d
kube-proxy-s8zwl 1/1 Running 3 20d
kube-proxy-tt2vb 1/1 Running 4 20d
kube-scheduler-master 1/1 Running 4 20d
kubernetes-dashboard-5f7b999d65-n56td 1/1 Running 0 2d21h
metrics-server-v0.3.3-6cbfd7955f-v29n7 0/2 CrashLoopBackOff 1 16s
weave-net-6pv9w 2/2 Running 11 20d
weave-net-9dsxr 2/2 Running 10 20d
weave-net-lt8vv 2/2 Running 9 20d
weave-net-m2m8t 2/2 Running 11 20d

于是查看日志，容器metrics-server出现如下报错

[root@master metrics-server]# kubectl logs metrics-server-v0.3.3-549cc669c7-gxvgw -c metrics-server -n kube-system 
Flag --deprecated-kubelet-completely-insecure has been deprecated, This is rarely the right option, since it leaves kubelet communication completely insecure. If you encounter auth errors, make sure you've enabled token webhook auth on the Kubelet, and if you're in a test cluster with self-signed Kubelet certificates, consider using kubelet-insecure-tls instead.
I0531 04:36:16.382275 1 serving.go:312] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0531 04:36:16.963800 1 secure_serving.go:116] Serving securely on [::]:443
E0531 04:36:46.981720 1 manager.go:111] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:worker1: unable to fetch metrics from Kubelet worker1 (10.5.24.223): Get http://10.5.24.223:10255/stats/summary/: dial tcp 10.5.24.223:10255: connect: connection refused, unable to fully scrape metrics from source kubelet_summary:worker3: unable to fetch metrics from Kubelet worker3 (10.5.24.225): Get http://10.5.24.225:10255/stats/summary/: dial tcp 10.5.24.225:10255: connect: connection refused, unable to fully scrape metrics from source kubelet_summary:master: unable to fetch metrics from Kubelet master (10.5.24.222): Get http://10.5.24.222:10255/stats/summary/: dial tcp 10.5.24.222:10255: connect: connection refused, unable to fully scrape metrics from source kubelet_summary:worker2: unable to fetch metrics from Kubelet worker2 (10.5.24.224): Get http://10.5.24.224:10255/stats/summary/: dial tcp 10.5.24.224:10255: connect: connection refused]
E0531 04:37:16.978839 1 manager.go:111] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:worker1: unable to fetch metrics from Kubelet worker1 (10.5.24.223): Get http://10.5.24.223:10255/stats/summary/: dial tcp 10.5.24.223:10255: connect: connection refused, unable to fully scrape metrics from source kubelet_summary:master: unable to fetch metrics from Kubelet master (10.5.24.222): Get http://10.5.24.222:10255/stats/summary/: dial tcp 10.5.24.222:10255: connect: connection refused, unable to fully scrape metrics from source kubelet_summary:worker3: unable to fetch metrics from Kubelet worker3 (10.5.24.225): Get http://10.5.24.225:10255/stats/summary/: dial tcp 10.5.24.225:10255: connect: connection refused, unable to fully scrape metrics from source kubelet_summary:worker2: unable to fetch metrics from Kubelet worker2 (10.5.24.224): Get http://10.5.24.224:10255/stats/summary/: dial tcp 10.5.24.224:10255: connect: connection refused]

容器metrics-server-nanny出现如下报错

[root@master metrics-server]# kubectl logs metrics-server-v0.3.3-55f94f8d76-ghlfb -n kube-system -c metrics-server-nanny
ERROR: logging before flag.Parse: I0531 06:20:40.721189 1 pod_nanny.go:65] Invoked by [/pod_nanny --config-dir=/etc/config --cpu={{ base_metrics_server_cpu }} --extra-cpu=0.5m --memory={{ base_metrics_server_memory }} --extra-memory={{ metrics_server_memory_per_node }}Mi --threshold=5 --deployment=metrics-server-v0.3.3 --container=metrics-server --poll-period=300000 --estimator=exponential --minClusterSize={{ metrics_server_min_cluster_size }}]
invalid argument "{{ metrics_server_min_cluster_size }}" for "--minClusterSize" flag: strconv.ParseUint: parsing "{{ metrics_server_min_cluster_size }}": invalid syntax
Usage of /pod_nanny:
 --config-dir string Path of configuration containing base resource requirements. (default "MISSING")
 --container string The name of the container to watch. This defaults to the nanny itself. (default "pod-nanny")
 --cpu string The base CPU resource requirement.
 --deployment string The name of the deployment being monitored. This is required.
 --estimator string The estimator to use. Currently supported: linear, exponential (default "linear")
 --extra-cpu string The amount of CPU to add per node.
 --extra-memory string The amount of memory to add per node.
 --extra-storage string The amount of storage to add per node. (default "0Gi")
 --memory string The base memory resource requirement.
 --minClusterSize uint The smallest number of nodes resources will be scaled to. Must be > 1. This flag is used only when an exponential estimator is used. (default 16)
 --namespace string The namespace of the ward. This defaults to the nanny pod's own namespace. (default "kube-system")
 --pod string The name of the pod to watch. This defaults to the nanny's own pod. (default "metrics-server-v0.3.3-55f94f8d76-ghlfb")
 --poll-period int The time, in milliseconds, to poll the dependent container. (default 10000)
 --storage string The base storage resource requirement. (default "MISSING")
 --threshold int A number between 0-100. The dependent's resources are rewritten when they deviate from expected by more than threshold.

解决方案：修改metrics-server-deployment.yaml为如下内容

[root@master metrics-server]# cat metrics-server-deployment.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
 name: metrics-server
 namespace: kube-system
 labels:
 kubernetes.io/cluster-service: "true"
 addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: v1
kind: ConfigMap
metadata:
 name: metrics-server-config
 namespace: kube-system
 labels:
 kubernetes.io/cluster-service: "true"
 addonmanager.kubernetes.io/mode: EnsureExists
data:
 NannyConfiguration: |-
 apiVersion: nannyconfig/v1alpha1
 kind: NannyConfiguration
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: metrics-server-v0.3.3
 namespace: kube-system
 labels:
 k8s-app: metrics-server
 kubernetes.io/cluster-service: "true"
 addonmanager.kubernetes.io/mode: Reconcile
 version: v0.3.3
spec:
 selector:
 matchLabels:
 k8s-app: metrics-server
 version: v0.3.3
 template:
 metadata:
 name: metrics-server
 labels:
 k8s-app: metrics-server
 version: v0.3.3
 annotations:
 scheduler.alpha.kubernetes.io/critical-pod: ''
 seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
 spec:
 priorityClassName: system-cluster-critical
 serviceAccountName: metrics-server
 containers:
 - name: metrics-server
 image: k8s.gcr.io/metrics-server-amd64:v0.3.3
 command:
 - /metrics-server
 - --metric-resolution=30s
 # These are needed for GKE, which doesn't support secure communication yet.
 # Remove these lines for non-GKE clusters, and when GKE supports token-based auth.
 #- --kubelet-port=10255
 #- --deprecated-kubelet-completely-insecure=true
 - --kubelet-insecure-tls
 - --kubelet-preferred-address-types=InternalIP
 #- --kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP
 ports:
 - containerPort: 443
 name: https
 protocol: TCP
 - name: metrics-server-nanny
 image: k8s.gcr.io/addon-resizer:1.8.5
 resources:
 limits:
 cpu: 100m
 memory: 300Mi
 requests:
 cpu: 5m
 memory: 50Mi
 env:
 - name: MY_POD_NAME
 valueFrom:
 fieldRef:
 fieldPath: metadata.name
 - name: MY_POD_NAMESPACE
 valueFrom:
 fieldRef:
 fieldPath: metadata.namespace
 volumeMounts:
 - name: metrics-server-config-volume
 mountPath: /etc/config
 command:
 - /pod_nanny
 - --config-dir=/etc/config
 #- --cpu={{ base_metrics_server_cpu }}
 - --extra-cpu=0.5m
 #- --memory={{ base_metrics_server_memory }}
 #- --extra-memory={{ metrics_server_memory_per_node }}Mi
 - --threshold=5
 - --deployment=metrics-server-v0.3.3
 - --container=metrics-server
 - --poll-period=300000
 - --estimator=exponential
 # Specifies the smallest cluster (defined in number of nodes)
 # resources will be scaled to.
 - --minClusterSize=2
 volumes:
 - name: metrics-server-config-volume
 configMap:
 name: metrics-server-config
 tolerations:
 - key: "CriticalAddonsOnly"
 operator: "Exists"

修改resource-reader.yaml为：

[root@master metrics-server]# cat resource-reader.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
 name: system:metrics-server
 labels:
 kubernetes.io/cluster-service: "true"
 addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups:
 - ""
 resources:
 - pods
 - nodes
 - nodes/stats
 - namespaces
 verbs:
 - get
 - list
 - watch
- apiGroups:
 - "extensions"
 resources:
 - deployments
 verbs:
 - get
 - list
 - update
 - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 name: system:metrics-server
 labels:
 kubernetes.io/cluster-service: "true"
 addonmanager.kubernetes.io/mode: Reconcile
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: system:metrics-server
subjects:
- kind: ServiceAccount
 name: metrics-server
 namespace: kube-system

修改后重新apply成功

[root@master metrics-server]# kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% 
master 336m 8% 2791Mi 36% 
worker1 258m 6% 1583Mi 59% 
worker2 172m 4% 1451Mi 54% 
worker3 312m 7% 1672Mi 62% 
[root@master metrics-server]# kubectl top pods -n kube-system 
NAME CPU(cores) MEMORY(bytes) 
coredns-fb8b8dccf-9bnvc 11m 16Mi 
coredns-fb8b8dccf-n5bzb 9m 17Mi 
etcd-master 39m 361Mi 
kube-apiserver-master 116m 410Mi 
kube-controller-manager-master 44m 66Mi 
kube-proxy-cmmbw 2m 17Mi 
kube-proxy-dnw9v 3m 23Mi 
kube-proxy-s8zwl 5m 16Mi 
kube-proxy-tt2vb 8m 22Mi 
kube-scheduler-master 3m 18Mi 
kubernetes-dashboard-5f7b999d65-n56td 1m 25Mi 
metrics-server-v0.3.3-7856b88bf4-hstvq 3m 22Mi 
weave-net-6pv9w 3m 114Mi 
weave-net-9dsxr 3m 123Mi 
weave-net-lt8vv 4m 105Mi 
weave-net-m2m8t 5m 115Mi

参考：https://github.com/kubernetes-incubator/metrics-server/issues/131

转载请注明：lampNick » kubernetes(k8s)中部署metrics-server遇到的问题