通过kubeadm安装的k8s集群获取kube-scheduler和kube-controller-manager组件状态异常

原因1:这两个pod的非安全端口没有开启，健康检查时报错，但是由于本身服务是正常的，只是健康检查的端口没启，所以不影响正常使用。

$ kubectl get cs
NAME                 STATUS      MESSAGE                                                                                     ERROR
controller-manager   Unhealthy   Get http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252: connect: connection refused
scheduler            Unhealthy   Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refused
etcd-0               Healthy     {"health":"true"}

 kubernetes版本：v1.18.6  镜像信息

$ sudo docker images
REPOSITORY                           TAG                 IMAGE ID            CREATED             SIZE
nginx                                latest              8cf1bfb43ff5        4 hours ago         132MB
k8s.gcr.io/kube-proxy                v1.18.6             c3d62d6fe412        6 days ago          117MB
k8s.gcr.io/kube-apiserver            v1.18.6             56acd67ea15a        6 days ago          173MB
k8s.gcr.io/kube-controller-manager   v1.18.6             ffce5e64d915        6 days ago          162MB
k8s.gcr.io/kube-scheduler            v1.18.6             0e0972b2b5d1        6 days ago          95.3MB
quay.io/coreos/flannel               v0.12.0-amd64       4e9f801d2217        4 months ago        52.8MB
k8s.gcr.io/pause                     3.2                 80d28bedfe5d        5 months ago        683kB
k8s.gcr.io/coredns                   1.6.7               67da37a9a360        5 months ago        43.8MB
k8s.gcr.io/etcd                      3.4.3-0             303ce5db0e90        9 months ago        288MB

 k8s组件pod状态

$ kubectl get pods -A
NAMESPACE     NAME                              READY   STATUS    RESTARTS   AGE
kube-system   coredns-66bff467f8-22hgp          1/1     Running   0          45m
kube-system   coredns-66bff467f8-ck6qq          1/1     Running   0          45m
kube-system   etcd-node-14                      1/1     Running   0          46m
kube-system   kube-apiserver-node-14            1/1     Running   0          46m
kube-system   kube-controller-manager-node-14   1/1     Running   0          17m
kube-system   kube-flannel-ds-amd64-lm7lt       1/1     Running   0          44m
kube-system   kube-proxy-5hghv                  1/1     Running   0          45m
kube-system   kube-scheduler-node-14            1/1     Running   0          17m

排查思路：

1、先查看本地的端口，可以确认没有启动10251、10252端口

2、确认kube-scheduler和kube-controller-manager组件配置是否禁用了非安全端口

配置文件路径：/etc/kubernetes/manifests/scheduler.conf 、/etc/kubernetes/manifests/controller-manager.conf

如controller-manager组件的配置如下：可以去掉--port=0这个设置，然后重启sudo systemctl restart kubelet

重启服务之后确认组件状态，显示就正常了

$ kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok
controller-manager   Healthy   ok
etcd-0               Healthy   {"health":"true"}

原因2: kubelet配置文件里没有指定静态pod的路径

解决办法：

（5.1）查找kubelet的配置文件位置 systemctl cat kubelet

[root@cka-node01 ~]# systemctl cat kubelet
    # /usr/lib/systemd/system/kubelet.service
[Unit]
Description=kubelet: The Kubernetes Node Agent
Documentation=https://kubernetes.io/docs/
 
[Service]
ExecStart=/usr/bin/kubelet
Restart=always
StartLimitInterval=0
RestartSec=10
 
[Install]
WantedBy=multi-user.target
 
    # /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf     这是kubelet的配置文件
    # Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
    # This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
    # This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
    # the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
 
 

（5.2）vim /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf 打开配置文件，添加以下内容

    # Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests"    要添加这个配置
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
    # This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
    # This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
    # the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

（5.3）重新加载daemon, 重启kubelet
[root@cka-node01 ~]# systemctl daemon-reload
[root@cka-node01 ~]# systemctl restart kubelet

添加前
在node上 ps -ef |grep kubelet 查看加载的配置文件

[root@cka-node01 ~]# ps -ef |grep kubelet
root        689      1  2 Aug16 ?        00:51:35 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2
root     126462 126161  0 11:57 pts/0    00:00:00 grep --color=auto kubelet

添加后

[root@cka-node01 ~]# ps -ef |grep kubelet
root       4457      1  6 12:17 ?        00:00:02 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests --config=/var/lib/kubelet/config.yaml --cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2
root       5089 126161  0 12:18 pts/0    00:00:00 grep --color=auto kubelet

复制

查找kubectl的帮助，grep manifest 可以看到

[root@cka-node01 ~]# kubelet --help |grep mani
manifest can be provided to the Kubelet.
(underspec'd currently) to submit a new manifest.
      --cgroup-driver string                                                                                      Driver that the kubelet uses to manipulate cgroups on the host.  Possible values: 'cgroupfs', 'systemd' (default "cgroupfs") (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
      --manifest-url string                                                                                       URL for accessing additional Pod specifications to run (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
      --manifest-url-header --manifest-url-header 'a:hello,b:again,c:world' --manifest-url-header 'b:beautiful'   Comma-separated list of HTTP headers to use when accessing the url provided to --manifest-url. Multiple headers with the same name will be added in the same order provided. This flag can be repeatedly invoked. For example: --manifest-url-header 'a:hello,b:again,c:world' --manifest-url-header 'b:beautiful' (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
      --pod-manifest-path string                                                                                  Path to the directory containing static pod files to run, or the path to a single static pod file. Files starting with dots will be ignored. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)