Kubernetes证书续期

Hadoop_Liang

212人浏览 · 2026-05-29 11:17:18

Hadoop_Liang · 2026-05-29 11:17:18 发布

问题背景

Kubernetes证书默认有效期是1年，证书过期后，查看kubectl get nodes命令会发现连接不上。问题如下：

[root@k8s-master01 ~]# kubectl get nodes
E0528 23:37:39.439307    3035 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.204.101:6443/api?timeout=32s\": dial tcp 192.168.204.101:6443: connect: connection refused"
E0528 23:37:39.456139    3035 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.204.101:6443/api?timeout=32s\": dial tcp 192.168.204.101:6443: connect: connection refused"
E0528 23:37:39.472313    3035 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.204.101:6443/api?timeout=32s\": dial tcp 192.168.204.101:6443: connect: connection refused"
E0528 23:37:39.489870    3035 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.204.101:6443/api?timeout=32s\": dial tcp 192.168.204.101:6443: connect: connection refused"
E0528 23:37:39.504756    3035 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.204.101:6443/api?timeout=32s\": dial tcp 192.168.204.101:6443: connect: connection refused"
The connection to the server 192.168.204.101:6443 was refused - did you specify the right host or port?
[root@k8s-master01 ~]#

在这里插入图片描述

问题排查

1.查看kubelet状态

[root@k8s-master01 ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: activating (auto-restart) (Result: exit-code) since Thu 2026-05-28 23:40:45 CST; 632m>
       Docs: https://kubernetes.io/docs/
    Process: 3852 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELE>
   Main PID: 3852 (code=exited, status=1/FAILURE)
        CPU: 76ms
lines 1-9/9 (END)

检查 kubelet 状态显示 active: activating (auto-restart)，进程退出码 1/FAILURE，说明服务处于不断重启失败的状态。这意味着控制平面（Master）的组件（如 kube-apiserver）未能正常启动。

按q返回命令行

2.查看是否监听6443端口

[root@k8s-master01 ~]# netstat -tlnp | grep 6443
[root@k8s-master01 ~]#

看到无输出，说明没有进程监听 6443 端口， API server 没有在运行。

3.查看docker状态

[root@k8s-master01 ~]# systemctl status docker
● docker.service - Docker Application Container Engine
     Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; preset: disabled)
     Active: active (running) since Thu 2026-05-28 23:32:24 CST; 11min ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
   Main PID: 1038 (dockerd)
      Tasks: 19
     Memory: 116.2M
        CPU: 1.328s
     CGroup: /system.slice/docker.service
             └─1038 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

May 28 23:32:21 k8s-master01 systemd[1]: Starting Docker Application Container Engine...
May 28 23:32:22 k8s-master01 dockerd[1038]: time="2026-05-28T23:32:22.596583795+08:00" level=info >
May 28 23:32:22 k8s-master01 dockerd[1038]: time="2026-05-28T23:32:22.815541741+08:00" level=info >
May 28 23:32:22 k8s-master01 dockerd[1038]: time="2026-05-28T23:32:22.933586285+08:00" level=info >
May 28 23:32:23 k8s-master01 dockerd[1038]: time="2026-05-28T23:32:23.841965900+08:00" level=info >
May 28 23:32:23 k8s-master01 dockerd[1038]: time="2026-05-28T23:32:23.913025913+08:00" level=info >
May 28 23:32:23 k8s-master01 dockerd[1038]: time="2026-05-28T23:32:23.941648967+08:00" level=info >
May 28 23:32:23 k8s-master01 dockerd[1038]: time="2026-05-28T23:32:23.942258143+08:00" level=info >
May 28 23:32:24 k8s-master01 systemd[1]: Started Docker Application Container Engine.
May 28 23:32:24 k8s-master01 dockerd[1038]: time="2026-05-28T23:32:24.021245253+08:00" level=info >
lines 1-22/22 (END)

看到Active: active (running)，说明docker正常运行

按q返回命令行

4.查看证书过期时间

[root@k8s-master01 ~]# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Sep 14, 2025 08:56 UTC   <invalid>       ca                      no     
apiserver                  Sep 14, 2025 08:56 UTC   <invalid>       ca                      no     
apiserver-etcd-client      Sep 14, 2025 08:56 UTC   <invalid>       etcd-ca                 no     
apiserver-kubelet-client   Sep 14, 2025 08:56 UTC   <invalid>       ca                      no     
controller-manager.conf    Sep 14, 2025 08:56 UTC   <invalid>       ca                      no     
etcd-healthcheck-client    Sep 14, 2025 08:56 UTC   <invalid>       etcd-ca                 no     
etcd-peer                  Sep 14, 2025 08:56 UTC   <invalid>       etcd-ca                 no     
etcd-server                Sep 14, 2025 08:56 UTC   <invalid>       etcd-ca                 no     
front-proxy-client         Sep 14, 2025 08:56 UTC   <invalid>       front-proxy-ca          no     
scheduler.conf             Sep 14, 2025 08:56 UTC   <invalid>       ca                      no     
super-admin.conf           Sep 14, 2025 08:56 UTC   <invalid>       ca                      no     

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Sep 12, 2034 08:56 UTC   8y              no
etcd-ca                 Sep 12, 2034 08:56 UTC   8y              no
front-proxy-ca          Sep 12, 2034 08:56 UTC   8y              no
[root@k8s-master01 ~]#

所有证书的过期时间都是 Sep 14, 2025 08:56 UTC，而当前系统时间是 Thu May 28 23:32:35 CST 2026（2026年5月28日）。这意味着当前时间已经超过了证书有效期（2025年9月14日）。证书确实已经过期（RESIDUAL TIME 显示）。这是导致 API Server 无法启动的根本原因之一：kube-apiserver 使用的证书已经过期，无法建立 TLS 连接。

问题的根本原因：kube-apiserver 使用的证书过期。

因此，需要更新所有过期的Kubernetes证书。

解决办法

备份现有证书

在控制平面（Master）节点执行如下命令备份证书

cp -r /etc/kubernetes/pki /etc/kubernetes/pki.bak
cp /etc/kubernetes/*.conf /etc/kubernetes/conf.bak/   # 若目录不存在则手动创建

操作过程

[root@k8s-master01 ~]# ls /etc/kubernetes/
admin.conf               kubelet.conf  pki             super-admin.conf
controller-manager.conf  manifests     scheduler.conf
[root@k8s-master01 ~]# cp -r /etc/kubernetes/pki /etc/kubernetes/pki.bak
[root@k8s-master01 ~]# mkdir /etc/kubernetes/conf.bak/
[root@k8s-master01 ~]# cp /etc/kubernetes/*.conf /etc/kubernetes/conf.bak/

使用 kubeadm 续期Master节点证书

在Master节点执行如下命令续期证书

kubeadm certs renew all

操作过程

[root@k8s-master01 ~]# kubeadm certs renew all
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
certificate embedded in the kubeconfig file for the super-admin renewed

Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.
[root@k8s-master01 ~]#

重启 kubelet

systemctl restart kubelet

操作过程

[root@k8s-master01 ~]# systemctl restart kubelet
[root@k8s-master01 ~]#

等待一会，确保kubelet重启完成

查看端口监听

netstat -tlnp | grep 6443

操作过程

[root@k8s-master01 ~]# netstat -tlnp | grep 6443
[root@k8s-master01 ~]# netstat -tlnp | grep 6443
[root@k8s-master01 ~]# netstat -tlnp | grep 6443
[root@k8s-master01 ~]#

大约3分钟后，执行多次监听，依然没有输出

使用kubectl命令查看节点

[root@k8s-master01 ~]# kubectl get nodes
E0529 00:07:15.249652   10799 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.204.101:6443/api?timeout=32s\": dial tcp 192.168.204.101:6443: connect: connection refused"
E0529 00:07:15.261214   10799 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.204.101:6443/api?timeout=32s\": dial tcp 192.168.204.101:6443: connect: connection refused"
E0529 00:07:15.277475   10799 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.204.101:6443/api?timeout=32s\": dial tcp 192.168.204.101:6443: connect: connection refused"
E0529 00:07:15.293347   10799 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.204.101:6443/api?timeout=32s\": dial tcp 192.168.204.101:6443: connect: connection refused"
E0529 00:07:15.308878   10799 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.204.101:6443/api?timeout=32s\": dial tcp 192.168.204.101:6443: connect: connection refused"
The connection to the server 192.168.204.101:6443 was refused - did you specify the right host or port?
[root@k8s-master01 ~]#

检查 kubelet 日志

[root@k8s-master01 ~]# journalctl -u kubelet -n 50 --no-pager
May 29 00:08:35 k8s-master01 systemd[1]: Started kubelet: The Kubernetes Node Agent.
May 29 00:08:35 k8s-master01 kubelet[11149]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 29 00:08:35 k8s-master01 kubelet[11149]: Flag --pod-infra-container-image has been deprecated, will be removed in a future release. Image garbage collector will get sandbox image information from CRI.
May 29 00:08:35 k8s-master01 kubelet[11149]: I0529 00:08:35.340684   11149 server.go:206] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 29 00:08:35 k8s-master01 kubelet[11149]: I0529 00:08:35.346137   11149 server.go:486] "Kubelet version" kubeletVersion="v1.31.1"
May 29 00:08:35 k8s-master01 kubelet[11149]: I0529 00:08:35.346175   11149 server.go:488] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 29 00:08:35 k8s-master01 kubelet[11149]: I0529 00:08:35.346503   11149 server.go:929] "Client rotation is on, will bootstrap in background"
May 29 00:08:35 k8s-master01 kubelet[11149]: E0529 00:08:35.347264   11149 bootstrap.go:266] "Unhandled Error" err="part of the existing bootstrap client certificate in /etc/kubernetes/kubelet.conf is expired: 2025-09-14 08:56:26 +0000 UTC" logger="UnhandledError"
May 29 00:08:35 k8s-master01 kubelet[11149]: E0529 00:08:35.364203   11149 run.go:72] "command failed" err="failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory"
May 29 00:08:35 k8s-master01 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
May 29 00:08:35 k8s-master01 systemd[1]: kubelet.service: Failed with result 'exit-code'.
May 29 00:08:45 k8s-master01 systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 212.
May 29 00:08:45 k8s-master01 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
May 29 00:08:45 k8s-master01 systemd[1]: Started kubelet: The Kubernetes Node Agent.
May 29 00:08:45 k8s-master01 kubelet[11193]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 29 00:08:45 k8s-master01 kubelet[11193]: Flag --pod-infra-container-image has been deprecated, will be removed in a future release. Image garbage collector will get sandbox image information from CRI.
May 29 00:08:45 k8s-master01 kubelet[11193]: I0529 00:08:45.626238   11193 server.go:206] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 29 00:08:45 k8s-master01 kubelet[11193]: I0529 00:08:45.634389   11193 server.go:486] "Kubelet version" kubeletVersion="v1.31.1"
May 29 00:08:45 k8s-master01 kubelet[11193]: I0529 00:08:45.634437   11193 server.go:488] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 29 00:08:45 k8s-master01 kubelet[11193]: I0529 00:08:45.634610   11193 server.go:929] "Client rotation is on, will bootstrap in background"
May 29 00:08:45 k8s-master01 kubelet[11193]: E0529 00:08:45.635145   11193 bootstrap.go:266] "Unhandled Error" err="part of the existing bootstrap client certificate in /etc/kubernetes/kubelet.conf is expired: 2025-09-14 08:56:26 +0000 UTC" logger="UnhandledError"
May 29 00:08:45 k8s-master01 kubelet[11193]: E0529 00:08:45.648971   11193 run.go:72] "command failed" err="failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory"
May 29 00:08:45 k8s-master01 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
May 29 00:08:45 k8s-master01 systemd[1]: kubelet.service: Failed with result 'exit-code'.
May 29 00:08:55 k8s-master01 systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 213.
May 29 00:08:55 k8s-master01 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
May 29 00:08:55 k8s-master01 systemd[1]: Started kubelet: The Kubernetes Node Agent.
May 29 00:08:55 k8s-master01 kubelet[11236]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 29 00:08:55 k8s-master01 kubelet[11236]: Flag --pod-infra-container-image has been deprecated, will be removed in a future release. Image garbage collector will get sandbox image information from CRI.
May 29 00:08:55 k8s-master01 kubelet[11236]: I0529 00:08:55.885679   11236 server.go:206] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 29 00:08:55 k8s-master01 kubelet[11236]: I0529 00:08:55.916286   11236 server.go:486] "Kubelet version" kubeletVersion="v1.31.1"
May 29 00:08:55 k8s-master01 kubelet[11236]: I0529 00:08:55.916385   11236 server.go:488] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 29 00:08:55 k8s-master01 kubelet[11236]: I0529 00:08:55.916745   11236 server.go:929] "Client rotation is on, will bootstrap in background"
May 29 00:08:55 k8s-master01 kubelet[11236]: E0529 00:08:55.918178   11236 bootstrap.go:266] "Unhandled Error" err="part of the existing bootstrap client certificate in /etc/kubernetes/kubelet.conf is expired: 2025-09-14 08:56:26 +0000 UTC" logger="UnhandledError"
May 29 00:08:55 k8s-master01 kubelet[11236]: E0529 00:08:55.926189   11236 run.go:72] "command failed" err="failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory"
May 29 00:08:55 k8s-master01 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
May 29 00:08:55 k8s-master01 systemd[1]: kubelet.service: Failed with result 'exit-code'.
May 29 00:09:05 k8s-master01 systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 214.
May 29 00:09:05 k8s-master01 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
May 29 00:09:06 k8s-master01 systemd[1]: Started kubelet: The Kubernetes Node Agent.
May 29 00:09:06 k8s-master01 kubelet[11279]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 29 00:09:06 k8s-master01 kubelet[11279]: Flag --pod-infra-container-image has been deprecated, will be removed in a future release. Image garbage collector will get sandbox image information from CRI.
May 29 00:09:06 k8s-master01 kubelet[11279]: I0529 00:09:06.079266   11279 server.go:206] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 29 00:09:06 k8s-master01 kubelet[11279]: I0529 00:09:06.082476   11279 server.go:486] "Kubelet version" kubeletVersion="v1.31.1"
May 29 00:09:06 k8s-master01 kubelet[11279]: I0529 00:09:06.082512   11279 server.go:488] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 29 00:09:06 k8s-master01 kubelet[11279]: I0529 00:09:06.082636   11279 server.go:929] "Client rotation is on, will bootstrap in background"
May 29 00:09:06 k8s-master01 kubelet[11279]: E0529 00:09:06.083137   11279 bootstrap.go:266] "Unhandled Error" err="part of the existing bootstrap client certificate in /etc/kubernetes/kubelet.conf is expired: 2025-09-14 08:56:26 +0000 UTC" logger="UnhandledError"
May 29 00:09:06 k8s-master01 kubelet[11279]: E0529 00:09:06.085477   11279 run.go:72] "command failed" err="failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory"
May 29 00:09:06 k8s-master01 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
May 29 00:09:06 k8s-master01 systemd[1]: kubelet.service: Failed with result 'exit-code'.
[root@k8s-master01 ~]#

看到kubelet启动失败。因为 /etc/kubernetes/kubelet.conf 中的客户端证书已过期（2025-09-14）。此前运行的 kubeadm certs renew all 只是续期了控制平面组件的证书（如 apiserver、etcd 等），但没有自动更新 kubelet.conf 中嵌入的客户端证书。kubelet.conf 是 kubelet 用来与 API Server 通信的 kubeconfig 文件，其中包含客户端证书。这个证书也需要单独续期或重新生成。

查看相关配置文件日期，有些还是2024年的，例如：kubelet.conf，说明kubelet.conf文件未更新

[root@k8s-master01 ~]# ll /etc/kubernetes/
total 44
-rw------- 1 root root 5655 May 29 00:02 admin.conf
drwxr-xr-x 2 root root  121 May 28 23:58 conf.bak
-rw------- 1 root root 5679 May 29 00:02 controller-manager.conf
-rw------- 1 root root 1995 Sep 14  2024 kubelet.conf
drwxr-xr-x 2 root root  113 Sep 14  2024 manifests
drwxr-xr-x 3 root root 4096 Sep 14  2024 pki
drwxr-xr-x 3 root root 4096 May 28 23:57 pki.bak
-rw------- 1 root root 5631 May 29 00:02 scheduler.conf
-rw------- 1 root root 5679 May 29 00:02 super-admin.conf
[root@k8s-master01 ~]# ll /etc/kubernetes/pki
total 56
-rw-r--r-- 1 root root 1123 May 29 00:02 apiserver-etcd-client.crt
-rw------- 1 root root 1675 May 29 00:02 apiserver-etcd-client.key
-rw-r--r-- 1 root root 1176 May 29 00:02 apiserver-kubelet-client.crt
-rw------- 1 root root 1675 May 29 00:02 apiserver-kubelet-client.key
-rw-r--r-- 1 root root 1289 May 29 00:02 apiserver.crt
-rw------- 1 root root 1675 May 29 00:02 apiserver.key
-rw-r--r-- 1 root root 1107 Sep 14  2024 ca.crt
-rw------- 1 root root 1679 Sep 14  2024 ca.key
drwxr-xr-x 2 root root  162 Sep 14  2024 etcd
-rw-r--r-- 1 root root 1123 Sep 14  2024 front-proxy-ca.crt
-rw------- 1 root root 1675 Sep 14  2024 front-proxy-ca.key
-rw-r--r-- 1 root root 1119 May 29 00:02 front-proxy-client.crt
-rw------- 1 root root 1675 May 29 00:02 front-proxy-client.key
-rw------- 1 root root 1675 Sep 14  2024 sa.key
-rw------- 1 root root  451 Sep 14  2024 sa.pub
[root@k8s-master01 ~]#

使用 kubeadm 重新生成 kubelet.conf

在Master节点重新生成kubelet配置

kubeadm init phase kubeconfig kubelet --node-name=k8s-master01

该命令会使用本地 CA 证书（/etc/kubernetes/pki/ca.crt 和 ca.key）为当前节点生成一个新的客户端证书，并写入 /etc/kubernetes/kubelet.conf。

操作过程

[root@k8s-master01 ~]# kubeadm init phase kubeconfig kubelet --node-name=k8s-master01
I0529 00:20:25.027150   14254 version.go:261] remote version is much newer: v1.36.1; falling back to: stable-1.31
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/kubelet.conf"
[root@k8s-master01 ~]#

重启kubelet

[root@k8s-master01 ~]# systemctl restart kubelet
[root@k8s-master01 ~]#

查看kubelet状态，还是自动重启状态

[root@k8s-master01 ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: activating (auto-restart) (Result: exit-code) since Fri 2026-05-29 00:22:08 CST; 9s a>
       Docs: https://kubernetes.io/docs/
    Process: 14720 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBEL>
   Main PID: 14720 (code=exited, status=1/FAILURE)
        CPU: 46ms
[root@k8s-master01 ~]#

查看6443端口监听，还是无输出

[root@k8s-master01 ~]# netstat -tlnp | grep 6443
[root@k8s-master01 ~]#

再次查看文件时间

[root@k8s-master01 ~]# ll /etc/kubernetes/
total 44
-rw------- 1 root root 5655 May 29 00:02 admin.conf
drwxr-xr-x 2 root root  121 May 28 23:58 conf.bak
-rw------- 1 root root 5679 May 29 00:02 controller-manager.conf
-rw------- 1 root root 1995 Sep 14  2024 kubelet.conf
drwxr-xr-x 2 root root  113 Sep 14  2024 manifests
drwxr-xr-x 3 root root 4096 Sep 14  2024 pki
drwxr-xr-x 3 root root 4096 May 28 23:57 pki.bak
-rw------- 1 root root 5631 May 29 00:02 scheduler.conf
-rw------- 1 root root 5679 May 29 00:02 super-admin.conf
[root@k8s-master01 ~]# ll /etc/kubernetes/pki
total 56
-rw-r--r-- 1 root root 1123 May 29 00:02 apiserver-etcd-client.crt
-rw------- 1 root root 1675 May 29 00:02 apiserver-etcd-client.key
-rw-r--r-- 1 root root 1176 May 29 00:02 apiserver-kubelet-client.crt
-rw------- 1 root root 1675 May 29 00:02 apiserver-kubelet-client.key
-rw-r--r-- 1 root root 1289 May 29 00:02 apiserver.crt
-rw------- 1 root root 1675 May 29 00:02 apiserver.key
-rw-r--r-- 1 root root 1107 Sep 14  2024 ca.crt
-rw------- 1 root root 1679 Sep 14  2024 ca.key
drwxr-xr-x 2 root root  162 Sep 14  2024 etcd
-rw-r--r-- 1 root root 1123 Sep 14  2024 front-proxy-ca.crt
-rw------- 1 root root 1675 Sep 14  2024 front-proxy-ca.key
-rw-r--r-- 1 root root 1119 May 29 00:02 front-proxy-client.crt
-rw------- 1 root root 1675 May 29 00:02 front-proxy-client.key
-rw------- 1 root root 1675 Sep 14  2024 sa.key
-rw------- 1 root root  451 Sep 14  2024 sa.pub

看到kubelet.conf文件还是2024年的日期，说明kubelet.conf文件存在时不会更新

需要先将原有文件删除或重命名为其他名字（重要！）

mv /etc/kubernetes/kubelet.conf /etc/kubernetes/kubelet.conf.old

再次执行kubelet配置命令

[root@k8s-master01 ~]# kubeadm init phase kubeconfig kubelet --node-name=k8s-master01
I0529 00:25:16.342198   15548 version.go:261] remote version is much newer: v1.36.1; falling back to: stable-1.31
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[root@k8s-master01 ~]#

重启kubelet

[root@k8s-master01 ~]# systemctl restart kubelet

查看kubelet状态

[root@k8s-master01 ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: active (running) since Fri 2026-05-29 00:25:28 CST; 11s ago
       Docs: https://kubernetes.io/docs/
   Main PID: 15652 (kubelet)
      Tasks: 12 (limit: 22964)
     Memory: 29.4M
        CPU: 494ms
     CGroup: /system.slice/kubelet.service
             └─15652 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.con>

看到是Active: active (running) ，说明控制平面的kubelet正常了

查看kubelet.conf日期，看到日期更新了

[root@k8s-master01 ~]# ll /etc/kubernetes/
total 52
-rw------- 1 root root 5655 May 29 00:02 admin.conf
drwxr-xr-x 2 root root  121 May 28 23:58 conf.bak
-rw------- 1 root root 5679 May 29 00:02 controller-manager.conf
-rw------- 1 root root 5691 May 29 00:25 kubelet.conf
-rw------- 1 root root 1995 Sep 14  2024 kubelet.conf.old
drwxr-xr-x 2 root root  113 Sep 14  2024 manifests
drwxr-xr-x 3 root root 4096 Sep 14  2024 pki
drwxr-xr-x 3 root root 4096 May 28 23:57 pki.bak
-rw------- 1 root root 5631 May 29 00:02 scheduler.conf
-rw------- 1 root root 5679 May 29 00:02 super-admin.conf
[root@k8s-master01 ~]# ll /etc/kubernetes/pki
total 56
-rw-r--r-- 1 root root 1123 May 29 00:02 apiserver-etcd-client.crt
-rw------- 1 root root 1675 May 29 00:02 apiserver-etcd-client.key
-rw-r--r-- 1 root root 1176 May 29 00:02 apiserver-kubelet-client.crt
-rw------- 1 root root 1675 May 29 00:02 apiserver-kubelet-client.key
-rw-r--r-- 1 root root 1289 May 29 00:02 apiserver.crt
-rw------- 1 root root 1675 May 29 00:02 apiserver.key
-rw-r--r-- 1 root root 1107 Sep 14  2024 ca.crt
-rw------- 1 root root 1679 Sep 14  2024 ca.key
drwxr-xr-x 2 root root  162 Sep 14  2024 etcd
-rw-r--r-- 1 root root 1123 Sep 14  2024 front-proxy-ca.crt
-rw------- 1 root root 1675 Sep 14  2024 front-proxy-ca.key
-rw-r--r-- 1 root root 1119 May 29 00:02 front-proxy-client.crt
-rw------- 1 root root 1675 May 29 00:02 front-proxy-client.key
-rw------- 1 root root 1675 Sep 14  2024 sa.key
-rw------- 1 root root  451 Sep 14  2024 sa.pub

查看6443端口监听情况，看到有输出，说明已经正常监听6443端口

[root@k8s-master01 ~]# netstat -tlnp | grep 6443
tcp6       0      0 :::6443                 :::*                    LISTEN      15977/kube-apiserve
[root@k8s-master01 ~]#

查看节点信息

[root@k8s-master01 ~]# kubectl get nodes
E0529 00:34:05.415700   28754 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E0529 00:34:05.438791   28754 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E0529 00:34:05.454602   28754 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E0529 00:34:05.469730   28754 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E0529 00:34:05.486839   28754 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
error: You must be logged in to the server (the server has asked for the client to provide credentials)
[root@k8s-master01 ~]#

报错原因是当前用户环境中的 $HOME/.kube/config 没有使用最新的 admin.conf。

更新用户环境下的admin.conf

复制新的admin.conf到用户环境

cp -f /etc/kubernetes/admin.conf $HOME/.kube/config

操作过程

[root@k8s-master01 ~]# cp -f /etc/kubernetes/admin.conf $HOME/.kube/config
cp: overwrite '/root/.kube/config'? y
[root@k8s-master01 ~]#

根据提示，输入y，回车

查看节点

[root@k8s-master01 ~]# kubectl get nodes
NAME           STATUS     ROLES           AGE    VERSION
k8s-master01   Ready      control-plane   621d   v1.31.1
k8s-node01     NotReady   <none>          621d   v1.31.1
k8s-node02     NotReady   <none>          621d   v1.31.1
[root@k8s-master01 ~]#

看到控制平面节点（k8s-master01）正常为Ready状态，但是其他的Worker节点（k8s-node01、k8s-node02）还是NotReady状态。

查看k8s-node01的信息

在Master节点执行如下命令查看Worker相应节点的情况

kubectl describe node <Work节点名称> | grep -A 5 Conditions

操作过程

[root@k8s-master01 ~]# kubectl describe node k8s-node01 | grep -A 5 Conditions
Conditions:
  Type                 Status    LastHeartbeatTime                 LastTransitionTime                Reason              Message
  ----                 ------    -----------------                 ------------------                ------              -------
  NetworkUnavailable   False     Sat, 14 Sep 2024 23:20:01 +0800   Sat, 14 Sep 2024 23:20:01 +0800   CalicoIsUp          Calico is running on this node
  MemoryPressure       Unknown   Sun, 15 Sep 2024 01:19:37 +0800   Fri, 29 May 2026 00:26:32 +0800   NodeStatusUnknown   Kubelet stopped posting node status.
  DiskPressure         Unknown   Sun, 15 Sep 2024 01:19:37 +0800   Fri, 29 May 2026 00:26:32 +0800   NodeStatusUnknown   Kubelet stopped posting node status.

从结果来看，Worker 节点的 kubelet 因证书过期已停止向 API Server 报告状态（LastHeartbeatTime 停留在 2024 年 9 月）

分别进入k8s-node01、k8s-node02查看kubelet.conf文件的日期

k8s-node01机器

[root@k8s-node01 ~]# ll /etc/kubernetes/
total 4
-rw------- 1 root root 1964 Sep 14  2024 kubelet.conf
drwxr-xr-x 2 root root    6 Sep 12  2024 manifests
drwxr-xr-x 2 root root   20 Sep 14  2024 pki
[root@k8s-node01 ~]#

k8s-node02机器

[root@k8s-node02 ~]# ll /etc/kubernetes/
total 4
-rw------- 1 root root 1964 Sep 14  2024 kubelet.conf
drwxr-xr-x 2 root root    6 Sep 12  2024 manifests
drwxr-xr-x 2 root root   20 Sep 14  2024 pki
[root@k8s-node02 ~]#

看到k8s-node01、k8s-node02节点的kubelet.conf文件的日期都还是2024年的（此时是2026年），有效期默认是1年，所以过期了，需要续期

修复 Worker 节点

Worker 节点（如 k8s-node01、k8s-node02）的证书无法通过 kubeadm certs renew 续期，需要重新生成Worker的kubelet.conf并重新让Worker节点加入Kubernetes集群。

在 Master 节点上生成新的 join 命令

kubeadm token create --print-join-command

操作过程

[root@k8s-master01 ~]# kubeadm token create --print-join-command
kubeadm join 192.168.204.101:6443 --token 7z02uj.qqg75txt56mxrf1y --discovery-token-ca-cert-hash sha256:e779fa48edb461d766be9e43c95eed9c172d5a460a2e5c3121662919b1e488ac
[root@k8s-master01 ~]#

注意：后面让Worker节点重新加入集群时，需要在生成的join命令后面加上–cri-socket unix:///var/run/cri-dockerd.sock，即：

kubeadm join 192.168.204.101:6443 --token 7z02uj.qqg75txt56mxrf1y --discovery-token-ca-cert-hash sha256:e779fa48edb461d766be9e43c95eed9c172d5a460a2e5c3121662919b1e488ac --cri-socket unix:///var/run/cri-dockerd.sock

停止 kubelet

分别停止Worker节点的kubelet

systemctl stop kubelet

操作过程

在k8s-node01机器操作

[root@k8s-node01 ~]# systemctl stop kubelet
[root@k8s-node01 ~]#

在k8s-node02机器操作

[root@k8s-node02 ~]# systemctl stop kubelet
[root@k8s-node02 ~]#

清除旧证书和配置

清除Work节点的证书及配置

kubeadm reset -f --cri-socket unix:///var/run/cri-dockerd.sock

操作过程

在k8s-node01机器操作

[root@k8s-node01 ~]# kubeadm reset -f --cri-socket unix:///var/run/cri-dockerd.sock
[preflight] Running pre-flight checks
W0529 01:04:21.437351   25436 removeetcdmember.go:106] [reset] No kubeadm config, using etcd pod spec to get data directory
[reset] Deleted contents of the etcd data directory: /var/lib/etcd
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Deleting contents of directories: [/etc/kubernetes/manifests /var/lib/kubelet /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/super-admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]

The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually by using the "iptables" command.

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.

The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.
[root@k8s-node01 ~]#

在k8s-node02机器操作

[root@k8s-node02 ~]# kubeadm reset -f --cri-socket unix:///var/run/cri-dockerd.sock
[preflight] Running pre-flight checks
W0529 01:04:42.915074   25435 removeetcdmember.go:106] [reset] No kubeadm config, using etcd pod spec to get data directory
[reset] Deleted contents of the etcd data directory: /var/lib/etcd
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
W0529 01:05:13.154392   25435 cleanupnode.go:105] [reset] Failed to remove containers: [failed to stop running pod 27971f42e49cc3c8ad587b04915f03e739fcb96ecb6913f8d7b100cb3e9804b9: rpc error: code = DeadlineExceeded desc = context deadline exceeded, failed to stop running pod c368d9fc827f62a2984859ae8bee766a953b675172d06ee4fe0b826dc5c3a3df: rpc error: code = DeadlineExceeded desc = context deadline exceeded, failed to stop running pod 343780070bb1044258134420a89645be6547a01734c35273781b6df9a70e65b6: rpc error: code = DeadlineExceeded desc = context deadline exceeded]
[reset] Deleting contents of directories: [/etc/kubernetes/manifests /var/lib/kubelet /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/super-admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]

The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually by using the "iptables" command.

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.

The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.
[root@k8s-node02 ~]#

虽然有报错，但不用理会。

Worker节点重新加入集群

让Worker节点重新加入集群

kubeadm join 192.168.204.101:6443 --token 7z02uj.qqg75txt56mxrf1y --discovery-token-ca-cert-hash sha256:e779fa48edb461d766be9e43c95eed9c172d5a460a2e5c3121662919b1e488ac --cri-socket unix:///var/run/cri-dockerd.sock

在k8s-node01机器操作

[root@k8s-node01 ~]# kubeadm join 192.168.204.101:6443 --token 7z02uj.qqg75txt56mxrf1y --discovery-token-ca-cert-hash sha256:e779fa48edb461d766be9e43c95eed9c172d5a460a2e5c3121662919b1e488ac --cri-socket unix:///var/run/cri-dockerd.sock
[preflight] Running pre-flight checks
        [WARNING FileExisting-socat]: socat not found in system path
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 510.33181ms
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

[root@k8s-node01 ~]#

在k8s-node02机器操作

[root@k8s-node02 ~]# kubeadm join 192.168.204.101:6443 --token 7z02uj.qqg75txt56mxrf1y --discovery-token-ca-cert-hash sha256:e779fa48edb461d766be9e43c95eed9c172d5a460a2e5c3121662919b1e488ac --cri-socket unix:///var/run/cri-dockerd.sock
[preflight] Running pre-flight checks
        [WARNING FileExisting-socat]: socat not found in system path
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 520.048853ms
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

[root@k8s-node02 ~]#

查看节点

在控制平面节点执行如下命令

[root@k8s-master01 ~]# kubectl get nodes
NAME           STATUS   ROLES           AGE    VERSION
k8s-master01   Ready    control-plane   621d   v1.31.1
k8s-node01     Ready    <none>          621d   v1.31.1
k8s-node02     Ready    <none>          621d   v1.31.1
[root@k8s-master01 ~]#

看到所有节点都变为了Ready状态。说明集群已经可以正常使用。

设置Worker节点自动更新kubelet.conf

为了防止一年后再次出现同样问题，设置自动更新Worker节点证书，在每个 Worker 节点上执行以下命令：

echo "rotateCertificates: true" >> /var/lib/kubelet/config.yaml
systemctl restart kubelet

操作过程

在k8s-node01机器操作

[root@k8s-node01 ~]# echo "rotateCertificates: true" >> /var/lib/kubelet/config.yaml
[root@k8s-node01 ~]# systemctl restart kubelet
[root@k8s-node01 ~]#

在k8s-node02机器操作

[root@k8s-node02 ~]# echo "rotateCertificates: true" >> /var/lib/kubelet/config.yaml
[root@k8s-node02 ~]# systemctl restart kubelet
[root@k8s-node02 ~]#

再次查看节点，正常如下：

[root@k8s-master01 ~]# kubectl get nodes
NAME           STATUS   ROLES           AGE    VERSION
k8s-master01   Ready    control-plane   621d   v1.31.1
k8s-node01     Ready    <none>          621d   v1.31.1
k8s-node02     Ready    <none>          621d   v1.31.1

总结

Master 节点：备份证书 → 续期控制平面证书 → 重新生成 kubelet.conf → 重启 kubelet → 更新 admin.conf → 生成新的 Worker 加入命令

Worker 节点：停止 kubelet → 重置节点配置 → 使用提供的 join 命令重新加入集群 → 启用证书自动轮换

一键证书续期脚本

手动一步一步执行前面的命令，可完成证书续期修复。如果想实现一键修复，可编写一键修复脚本，提交修复效率。

1.创建一键修复脚本

[root@k8s-master01 ~]# vim fix-k8s-cert.sh

脚本内容如下：

#!/bin/bash
set -e

RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[0;33m'
NC='\033[0m'

log_info()  { echo -e "${GREEN}[INFO]${NC} $1"; }
log_warn()  { echo -e "${YELLOW}[WARN]${NC} $1"; }
log_error() { echo -e "${RED}[ERROR]${NC} $1"; exit 1; }

if [[ $EUID -ne 0 ]]; then
    log_error "请使用 root 用户执行此脚本"
fi

# 安装 sshpass
if ! command -v sshpass &>/dev/null; then
    log_warn "未安装 sshpass，正在尝试安装..."
    if command -v yum &>/dev/null; then
        yum install -y sshpass
    elif command -v apt &>/dev/null; then
        apt update && apt install -y sshpass
    else
        log_error "请手动安装 sshpass"
    fi
fi

# 修复 Master
log_info "========== 修复 Master 节点 =========="
BACKUP_DIR="/etc/kubernetes/backup-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$BACKUP_DIR"
cp -r /etc/kubernetes/pki "$BACKUP_DIR/pki" 2>/dev/null || true
cp /etc/kubernetes/*.conf "$BACKUP_DIR/" 2>/dev/null || true
log_info "备份已保存至 $BACKUP_DIR"

kubeadm certs renew all

[[ -f /etc/kubernetes/kubelet.conf ]] && mv /etc/kubernetes/kubelet.conf /etc/kubernetes/kubelet.conf.old
kubeadm init phase kubeconfig kubelet --node-name="$(hostname)"

systemctl restart kubelet

log_info "等待 API Server 启动..."
for i in {1..30}; do
    if netstat -tlnp 2>/dev/null | grep -q ":6443.*LISTEN"; then
        log_info "API Server 已启动"
        break
    fi
    sleep 2
done

mkdir -p "$HOME/.kube"
cp -f /etc/kubernetes/admin.conf "$HOME/.kube/config"

# 生成 Worker 修复脚本
log_info "========== 生成 Worker 修复脚本 =========="
JOIN_CMD=$(kubeadm token create --print-join-command)
if systemctl status cri-docker &>/dev/null || command -v cri-dockerd &>/dev/null; then
    JOIN_CMD="$JOIN_CMD --cri-socket unix:///var/run/cri-dockerd.sock"
fi

cat > /root/worker_fix.sh << 'EOF'
#!/bin/bash
set -e
if [[ $EUID -ne 0 ]]; then
    echo "请使用 root 用户执行此脚本"
    exit 1
fi
JOIN_CMD="__JOIN_CMD_PLACEHOLDER__"
echo "[INFO] 停止 kubelet..."
systemctl stop kubelet || true
CRI_SOCKET=""
if systemctl status cri-docker &>/dev/null || command -v cri-dockerd &>/dev/null; then
    CRI_SOCKET="--cri-socket unix:///var/run/cri-dockerd.sock"
fi
echo "[INFO] 重置节点..."
kubeadm reset -f $CRI_SOCKET || echo "[WARN] kubeadm reset 出现非致命错误"
echo "[INFO] 清理 CNI 配置..."
rm -rf /etc/cni/net.d 2>/dev/null || true
echo "[INFO] 重新加入集群..."
eval "$JOIN_CMD"
echo "[INFO] 启用证书自动轮换..."
CONFIG_FILE="/var/lib/kubelet/config.yaml"
if [[ -f "$CONFIG_FILE" ]]; then
    if ! grep -q "rotateCertificates:" "$CONFIG_FILE"; then
        echo "rotateCertificates: true" >> "$CONFIG_FILE"
    else
        sed -i 's/rotateCertificates:.*/rotateCertificates: true/' "$CONFIG_FILE"
    fi
    systemctl restart kubelet
fi
echo "[INFO] Worker 修复完成"
EOF

sed -i "s|__JOIN_CMD_PLACEHOLDER__|$JOIN_CMD|g" /root/worker_fix.sh
chmod +x /root/worker_fix.sh

# 获取 Worker 节点
MASTER_NAME=$(hostname)
WORKER_NODES=$(kubectl get nodes -o name | grep -v "$MASTER_NAME" | cut -d'/' -f2)
if [[ -z "$WORKER_NODES" ]]; then
    log_info "没有发现 Worker 节点，集群修复完成。"
    exit 0
fi
log_info "发现 Worker 节点：$WORKER_NODES"

read -s -p "请输入 Worker 节点的 root 密码（所有节点密码需相同）: " WORKER_PASS
echo ""

FIRST_WORKER=$(echo "$WORKER_NODES" | head -1)
if ! sshpass -p "$WORKER_PASS" ssh -o ConnectTimeout=5 -o StrictHostKeyChecking=no "$FIRST_WORKER" "echo ok" &>/dev/null; then
    log_error "密码错误或无法连接到 $FIRST_WORKER"
fi
log_info "密码验证通过"

# 并发修复 Worker
for NODE in $WORKER_NODES; do
    log_info "正在修复节点: $NODE"
    sshpass -p "$WORKER_PASS" scp -o StrictHostKeyChecking=no /root/worker_fix.sh "$NODE:/root/worker_fix.sh" >/dev/null
    sshpass -p "$WORKER_PASS" ssh -o StrictHostKeyChecking=no "$NODE" "chmod +x /root/worker_fix.sh && /root/worker_fix.sh" &
done
wait

log_info "所有 Worker 节点已执行修复脚本，等待节点状态变为 Ready..."

# 等待所有节点 Ready（最多 10 分钟，每 5 秒检查一次）
MAX_WAIT=120  # 120次 * 5秒 = 600秒 = 10分钟
for i in $(seq 1 $MAX_WAIT); do
    sleep 5
    NOT_READY=$(kubectl get nodes --no-headers 2>/dev/null | grep -v Ready | wc -l)
    if [[ $NOT_READY -eq 0 ]]; then
        log_info "所有节点均已 Ready！"
        break
    fi
    # 每 10 次（50秒）输出一次状态
    if [[ $((i % 10)) -eq 0 ]]; then
        log_warn "等待节点就绪中... ($i/$MAX_WAIT) 当前未就绪节点数: $NOT_READY"
        kubectl get nodes
    fi
    # 如果等待超过 3 分钟且仍有节点 NotReady，尝试重启网络插件
    if [[ $i -eq 40 ]] && [[ $NOT_READY -gt 0 ]]; then
        log_warn "部分节点长时间未 Ready，尝试重启网络插件..."
        kubectl delete pod -n kube-system -l k8s-app=calico-node --force --grace-period=0 2>/dev/null || true
        kubectl delete pod -n kube-system -l k8s-app=calico-kube-controllers --force --grace-period=0 2>/dev/null || true
    fi
done

FINAL_NOT_READY=$(kubectl get nodes --no-headers 2>/dev/null | grep -v Ready | wc -l)
if [[ $FINAL_NOT_READY -eq 0 ]]; then
    log_info "========== 全部修复完成 =========="
    kubectl get nodes
else
    log_error "仍有节点未 Ready，请手动检查：kubectl get nodes"
fi

添加执行权限

[root@k8s-master01 ~]# chmod +x fix-k8s-cert.sh

执行脚本之前，先使用kubectl查看节点

[root@k8s-master01 ~]# kubectl get nodes
E0529 15:28:28.548835    1806 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.204.101:6443/api?timeout=32s\": dial tcp 192.168.204.101:6443: connect: connection refused"
E0529 15:28:28.550509    1806 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.204.101:6443/api?timeout=32s\": dial tcp 192.168.204.101:6443: connect: connection refused"
E0529 15:28:28.552138    1806 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.204.101:6443/api?timeout=32s\": dial tcp 192.168.204.101:6443: connect: connection refused"
E0529 15:28:28.553622    1806 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.204.101:6443/api?timeout=32s\": dial tcp 192.168.204.101:6443: connect: connection refused"
E0529 15:28:28.555281    1806 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.204.101:6443/api?timeout=32s\": dial tcp 192.168.204.101:6443: connect: connection refused"
The connection to the server 192.168.204.101:6443 was refused - did you specify the right host or port?
[root@k8s-master01 ~]#

执行一键修复脚本

./fix-k8s-cert.sh

过程如下

[root@k8s-master01 ~]# ./fix-k8s-cert.sh
[WARN] 未安装 sshpass，正在尝试安装...
Docker CE Stable - x86_64                         1.2 kB/s | 2.0 kB     00:01
Docker CE Stable - x86_64                         101 kB/s |  78 kB     00:00
Extra Packages for Enterprise Linux 9 - x86_64    9.4 kB/s | 5.8 kB     00:00
Extra Packages for Enterprise Linux 9 - x86_64    408 kB/s |  21 MB     00:51
Extra Packages for Enterprise Linux 9 openh264 (F 506  B/s | 993  B     00:01
Kubernetes                                        404  B/s | 1.7 kB     00:04
Kubernetes                                        9.9 kB/s |  36 kB     00:03
Rocky Linux 9 - BaseOS                            2.7 kB/s | 4.3 kB     00:01
Rocky Linux 9 - BaseOS                            354 kB/s | 2.6 MB     00:07
Rocky Linux 9 - AppStream                         2.0 kB/s | 4.8 kB     00:02
Rocky Linux 9 - AppStream                         533 kB/s | 9.9 MB     00:18
Rocky Linux 9 - Extras                            835  B/s | 3.1 kB     00:03
Rocky Linux 9 - Extras                            8.8 kB/s |  17 kB     00:01
Dependencies resolved.
==================================================================================
 Package          Architecture    Version                Repository          Size
==================================================================================
Installing:
 sshpass          x86_64          1.09-4.el9             appstream           27 k

Transaction Summary
==================================================================================
Install  1 Package

Total download size: 27 k
Installed size: 47 k
Downloading Packages:
sshpass-1.09-4.el9.x86_64.rpm                     6.1 kB/s |  27 kB     00:04
----------------------------------------------------------------------------------
Total                                             6.1 kB/s |  27 kB     00:04
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                          1/1
  Installing       : sshpass-1.09-4.el9.x86_64                                1/1
  Running scriptlet: sshpass-1.09-4.el9.x86_64                                1/1
  Verifying        : sshpass-1.09-4.el9.x86_64                                1/1

Installed:
  sshpass-1.09-4.el9.x86_64

Complete!
[INFO] ========== 修复 Master 节点 ==========
[INFO] 备份已保存至 /etc/kubernetes/backup-20260529-153305
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
certificate embedded in the kubeconfig file for the super-admin renewed

Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.
I0529 15:33:11.302766    3191 version.go:261] remote version is much newer: v1.36.1; falling back to: stable-1.31
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[INFO] 等待 API Server 启动...
[INFO] ========== 生成 Worker 修复脚本 ==========
[INFO] 发现 Worker 节点：k8s-node01
k8s-node02
请输入 Worker 节点的 root 密码（所有节点密码需相同）:
[INFO] 密码验证通过
[INFO] 正在修复节点: k8s-node01
[INFO] 正在修复节点: k8s-node02
[INFO] 停止 kubelet...
[INFO] 重置节点...
[preflight] Running pre-flight checks
[reset] Deleted contents of the etcd data directory: /var/lib/etcd
[reset] Stopping the kubelet service
W0529 15:34:52.100383    3442 removeetcdmember.go:106] [reset] No kubeadm config, using etcd pod spec to get data directory
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[INFO] 停止 kubelet...
[reset] Deleting contents of directories: [/etc/kubernetes/manifests /var/lib/kubelet /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/super-admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]

The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually by using the "iptables" command.

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.

The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.
[INFO] 清理 CNI 配置...
[INFO] 重新加入集群...
[INFO] 重置节点...
[preflight] Running pre-flight checks
        [WARNING FileExisting-socat]: socat not found in system path
[preflight] Running pre-flight checks
[reset] Deleted contents of the etcd data directory: /var/lib/etcd
[reset] Stopping the kubelet service
W0529 15:34:52.507858    3381 removeetcdmember.go:106] [reset] No kubeadm config, using etcd pod spec to get data directory
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 1.010947359s
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

[INFO] 启用证书自动轮换...
[INFO] Worker 修复完成
[reset] Deleting contents of directories: [/etc/kubernetes/manifests /var/lib/kubelet /etc/kubernetes/pki]
W0529 15:35:22.789824    3381 cleanupnode.go:105] [reset] Failed to remove containers: [failed to stop running pod 27971f42e49cc3c8ad587b04915f03e739fcb96ecb6913f8d7b100cb3e9804b9: rpc error: code = DeadlineExceeded desc = context deadline exceeded, failed to stop running pod c368d9fc827f62a2984859ae8bee766a953b675172d06ee4fe0b826dc5c3a3df: rpc error: code = DeadlineExceeded desc = context deadline exceeded, failed to stop running pod 343780070bb1044258134420a89645be6547a01734c35273781b6df9a70e65b6: rpc error: code = DeadlineExceeded desc = context deadline exceeded]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/super-admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]

The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually by using the "iptables" command.

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.

The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.
[INFO] 清理 CNI 配置...
[INFO] 重新加入集群...
[preflight] Running pre-flight checks
        [WARNING FileExisting-socat]: socat not found in system path
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 1.011804188s
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

[INFO] 启用证书自动轮换...
[INFO] Worker 修复完成
[INFO] 所有 Worker 节点已执行修复脚本，等待节点状态变为 Ready...
[INFO] 所有节点均已 Ready！
[INFO] ========== 全部修复完成 ==========
NAME           STATUS   ROLES           AGE    VERSION
k8s-master01   Ready    control-plane   621d   v1.31.1
k8s-node01     Ready    <none>          621d   v1.31.1
k8s-node02     Ready    <none>          621d   v1.31.1
[root@k8s-master01 ~]#

提示：执行过程中，根据提示输入Linux的登录密码

执行脚本后，再次查看节点

[root@k8s-master01 ~]# kubectl get nodes
NAME           STATUS   ROLES           AGE    VERSION
k8s-master01   Ready    control-plane   621d   v1.31.1
k8s-node01     Ready    <none>          621d   v1.31.1
k8s-node02     Ready    <none>          621d   v1.31.1

看到节点均为Ready状态，说明一键修复脚本正常可用。

为验证是否稳定，重启集群所有机器，等待一会后，再次查看

[root@k8s-master01 ~]# kubectl get nodes
NAME           STATUS   ROLES           AGE    VERSION
k8s-master01   Ready    control-plane   621d   v1.31.1
k8s-node01     Ready    <none>          621d   v1.31.1
k8s-node02     Ready    <none>          621d   v1.31.1

看到节点均为Ready状态，说明一键修复脚本正常可用且稳定。

完成！enjoy it！

亚马逊云科技技术品牌专区

更多推荐

open-code-review Codex 插件教程：对话式调用评审技能指南

本文是一份面向新手的 **open-code-review Codex 插件教程**，教你在 Codex 中通过对话式方式调用 open-code-review 的 AI 代码评审技能。open-code-review 是阿里巴巴开源的 AI 代码评审工具，采用"确定性流水线 + LLM Agent"混合架构，内置多语言评审规则（NPE、线程安全、XSS、SQL 注入等），可在 Codex 里用一

亚马逊云科技技术品牌专区

OBS Studio 完整指南：免费开源录屏与直播软件从零上手

想做一场像模像样的直播，或者把屏幕内容录成教程视频，但一想到商用软件的订阅费就有些犹豫？其实你不需要花一分钱。OBS Studio 是一款免费开源的直播与录屏软件，支持 Windows、macOS 和 Linux，采集、混音、编码、推流、录制，一条流水线全包了。## 先搞清楚 OBS Studio 是什么OBS Studio 是一个"场景化"的采集合成工具：你把摄像头、窗口、浏览器、图片、

亚马逊云科技技术品牌专区

如何用MediaCrawler爬取小红书抖音数据？7大平台爬虫从安装到实战完整指南

MediaCrawler 是一个支持小红书、抖音、快手、B站、微博、百度贴吧、知乎的开源自媒体数据采集工具。它能按关键词、按帖子、按创作者主页三种方式抓取笔记/视频内容、评论甚至二级评论，并把结果存成 JSON、CSV、Excel 或数据库。适合做市场调研、竞品监控、内容选题的运营人员，以及想要现成爬虫框架而不想从零写 JS 逆向的开发者。## 从一次"手动抄数据"说起想象这个场景：你负责