1 遇到的问题

这个 PI 有一个性能测试(Performance test, 简称 PT)的任务，尘封了 1 年的用于 PT 的 k8s 集群将要大派用场。

但正当负责 PT 的同事磨刀霍霍，准备使用 kubectl 大展身手的时候，出现了以下错误：

root@xxxxx[03:47][/home/root]$ kubectl get po
Unable to connect to the server: x509: certificate has expired or is not yet valid: current time 2022-02-09T03:52:01+01:00 is after 2022-02-03T02:58:17Z

上面的错误提示，证书过期了。于是，编写了本文简单记录一下这次更新证书操作的过程。

2 master 节点上的证书

2.1 证书和key的路径

k8s 集群使用的 证书 和 私钥(key) 存放在 master 节点的 /etc/kubernetes/pki/ 下：

root@xxxxxx[04:01][/home/root]$ ls -alt /etc/kubernetes/pki
total 60
drwxr-xr-x 4 root root  125 Feb  3  2021 ..
drwxr-xr-x 3 root root 4096 Feb  3  2021 .
-rw------- 1 root root 1679 Feb  3  2021 sa.key
-rw------- 1 root root  451 Feb  3  2021 sa.pub
-rw-r--r-- 1 root root 1135 Feb  3  2021 apiserver-etcd-client.crt
-rw------- 1 root root 1679 Feb  3  2021 apiserver-etcd-client.key
drwxr-xr-x 2 root root  162 Feb  3  2021 etcd
-rw-r--r-- 1 root root 1103 Feb  3  2021 front-proxy-client.crt
-rw------- 1 root root 1679 Feb  3  2021 front-proxy-client.key
-rw-r--r-- 1 root root 1078 Feb  3  2021 front-proxy-ca.crt
-rw------- 1 root root 1675 Feb  3  2021 front-proxy-ca.key
-rw-r--r-- 1 root root 1143 Feb  3  2021 apiserver-kubelet-client.crt
-rw------- 1 root root 1679 Feb  3  2021 apiserver-kubelet-client.key
-rw-r--r-- 1 root root 1273 Feb  3  2021 apiserver.crt
-rw------- 1 root root 1675 Feb  3  2021 apiserver.key
-rw-r--r-- 1 root root 1066 Feb  3  2021 ca.crt
-rw------- 1 root root 1679 Feb  3  2021 ca.key

上面所列文件中，包含 ca 的属根证书及其对应的 私钥，是在签发证书时候用，并不在本次更新的证书范围内。例如，apiserver-kubelet-client.crt 的 ca证书 就是 ca.crt。用 openssl 查看 apiserver-kubelet-client.crt 的 issuer:

root@xxxxx[08:00][/home/root]$ openssl x509 -noout -issuer -subject -in /etc/kubernetes/pki/apiserver-kubelet-client.crt
issuer= /CN=kubernetes
subject= /O=system:masters/CN=kube-apiserver-kubelet-client

从上面，可知，颁发 apiserver-kubelet-client.crt 的 issuer 是 /CN=kubernetes。再用 openssl 查看 ca.crt 的 subject 和 有效时间：

root@xxxxx[08:01][/home/root]$ openssl x509 -noout -issuer -subject -enddate -in /etc/kubernetes/pki/ca.crt
issuer= /CN=kubernetes
subject= /CN=kubernetes
notAfter=Feb  1 02:58:17 2031 GMT

ca.crt 的 subject 和 issuer 为 /CN=kubernetes，与 apiserver-kubelet-client.crt 的 issuer 一致，因此，ca.crt，为根证书，用于签发 apiserver-kubelet-client.crt。

2.2 检查证书有效性

使用 openssl 检查 apiserver.crt 证书的有效期：

root@xxxxx[04:10][/home/root]$ openssl x509 -enddate -noout -in /etc/kubernetes/pki/apiserver.crt
notAfter=Feb  3 02:58:17 2022 GMT

证书的有效期至 2022 年 2 月 3 日，而当天时间已是 2022 年 2 月 9 日了，已经过期了。

3 使用 kubeadm 更新集群相关证书及conf文件

如果 k8s 集群的 kubeadm 版本为 1.8 或更高的版本，则可以使用 kubeadm 进行证书的更新。按下面的步骤进行操作。

3.1 使用 kubeadm 检查证书情况

在上一节，我们手动检查 /etc/kubernetes/pki 下的证书情况，现在，我们可以通过 kubeadm 方便地检查集群中相关证书的信息：

root@xxxxx[09:25][/home/root]$ sudo kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Feb 03, 2022 02:58 UTC   <invalid>                               no
apiserver                  Feb 03, 2022 02:58 UTC   <invalid>       ca                      no
apiserver-etcd-client      Feb 03, 2022 02:58 UTC   <invalid>       etcd-ca                 no
apiserver-kubelet-client   Feb 03, 2022 02:58 UTC   <invalid>       ca                      no
controller-manager.conf    Feb 03, 2022 02:58 UTC   <invalid>                               no
etcd-healthcheck-client    Feb 03, 2022 02:58 UTC   <invalid>       etcd-ca                 no
etcd-peer                  Feb 03, 2022 02:58 UTC   <invalid>       etcd-ca                 no
etcd-server                Feb 03, 2022 02:58 UTC   <invalid>       etcd-ca                 no
front-proxy-client         Feb 03, 2022 02:58 UTC   <invalid>       front-proxy-ca          no
scheduler.conf             Feb 03, 2022 02:58 UTC   <invalid>                               no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Feb 01, 2031 02:58 UTC   8y              no
etcd-ca                 Feb 01, 2031 02:58 UTC   8y              no
front-proxy-ca          Feb 01, 2031 02:58 UTC   8y              no

我们可以通过 kubeadm certs check-expiration 命令查看集群相关证书情况，其中 CERTIFICATE 部分为证书部分，CERTIFICATE AUTHORITY 为 ca 部分。

3.2 更新相关证书

可以使用下面命令一键更新所有证书和configuration文件:

kubeadm certs renew all

为了演示，下面将逐个更新证书。

• 更新 apiserver 证书

root@xxxxx[09:29][/home/root]$ sudo kubeadm certs renew apiserver
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration

certificate for serving the Kubernetes API renewed

• 更新 apiserver-etcd-client 证书

root@xxxxx[09:31][/home/root]$ sudo kubeadm certs renew apiserver-etcd-client
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration

certificate the apiserver uses to access etcd renewed

• 更新 apiserver-kubelet-client 证书

root@xxxxx[09:36][/home/root]$ sudo kubeadm certs renew apiserver-kubelet-client
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration

certificate for the API server to connect to kubelet renewed

• 更新 etcd-healthcheck-client 证书

root@xxxxx[09:38][/home/root]$ sudo kubeadm certs renew etcd-healthcheck-client
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration

certificate for liveness probes to healthcheck etcd renewed

• 更新 etcd-peer 证书

root@xxxx[09:41][/home/root]$ sudo kubeadm certs renew etcd-peer
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration

certificate for etcd nodes to communicate with each other renewed

• 更新 etcd-server 证书

root@xxxxx[09:42][/home/root]$ sudo kubeadm certs renew etcd-server
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration

certificate for serving etcd renewed

• 更新 front-proxy-client 证书

root@xxxxx[09:43][/home/root]$ sudo kubeadm certs renew front-proxy-client
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration

certificate for the front proxy client renewed

3.3 更新相关的 conf 文件

在 /etc/kubernetes 下，有几个 conf 文件：

root@xxx[09:54][/home/root]$ sudo ls -alt /etc/kubernetes | grep -E .*conf$
-rw-------    1 root root  1942 Feb  3  2021 kubelet.conf
-rw-------    1 root root  5550 Feb  3  2021 scheduler.conf
-rw-------    1 root root  5602 Feb  3  2021 controller-manager.conf
-rw-------    1 root root  5570 Feb  3  2021 admin.conf

其实这些 conf 文件就是 k8s 集群各核心组件所用的 kubeconfig 文件：

• admin.conf

admin 用户使用的 kubeconfig。

更新 admin.conf

root@xxxxx[09:57][/home/root]$ sudo kubeadm certs renew admin.conf
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed

• controller-manager.conf

controller manager 使用的 kubeconfig。

更新 controller-manager.conf

root@xxxxx[10:21][/home/root]$ sudo kubeadm certs renew controller-manager.conf
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

certificate embedded in the kubeconfig file for the controller manager to use renewed

• scheduler.conf

scheduler manager 使用的 kubeconfig。

更新 scheduler.conf

root@xxxxx[10:22][/home/root]$ sudo kubeadm certs renew scheduler.conf
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

certificate embedded in the kubeconfig file for the scheduler manager to use renewed

至此，为 k8s 集群更新证书的操作已经完成了。

4 验证证书更新效果

再检查集群相关证书情况

root@xxxxx[10:37][/home/root]$ sudo kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Feb 09, 2023 09:21 UTC   364d                                    no
apiserver                  Feb 09, 2023 08:29 UTC   364d            ca                      no
apiserver-etcd-client      Feb 09, 2023 08:34 UTC   364d            etcd-ca                 no
apiserver-kubelet-client   Feb 09, 2023 08:38 UTC   364d            ca                      no
controller-manager.conf    Feb 09, 2023 09:22 UTC   364d                                    no
etcd-healthcheck-client    Feb 09, 2023 08:41 UTC   364d            etcd-ca                 no
etcd-peer                  Feb 09, 2023 08:42 UTC   364d            etcd-ca                 no
etcd-server                Feb 09, 2023 08:43 UTC   364d            etcd-ca                 no
front-proxy-client         Feb 09, 2023 08:43 UTC   364d            front-proxy-ca          no
scheduler.conf             Feb 09, 2023 09:24 UTC   364d                                    no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Feb 01, 2031 02:58 UTC   8y              no
etcd-ca                 Feb 01, 2031 02:58 UTC   8y              no
front-proxy-ca          Feb 01, 2031 02:58 UTC   8y              no

相关证书的有效期已经被设置成 2023 年 2 月 9 了。

再执行 kubectl get nodes：

root@xxxxx[10:46][/tmp]$ sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get nodes
NAME           STATUS   ROLES                  AGE    VERSION
node-1   Ready    <none>                 371d   v1.20.2
master   Ready    control-plane,master   371d   v1.20.2
node-2   Ready    <none>                 349d   v1.20.4
node-3   Ready    <none>                 371d   v1.20.4