版本信息

该方案在kubernetes 1.12.5下验证有效，其他版本可能会略有不同，但基本原理相似。

故障现象

使用kubeadm部署的集群，默认根证书有效期是10年，其余证书的有效期为1年。当证书到期之后，k8s各核心组件之间的交互和协作会因为证书验证失效到时失败，从而导致k8s集群不可用。

故障排查和验证

故障排查

查看k8s各组件的日志会发现ssl认证错误信息。

故障验证

通过openssl x509 -in [证书名称] -noout -text命令能够查看证书的相关信息，从而确定是否证书到期：

$ cd /etc/kubernetes/pki
$ openssl x509 -in apiserver-kubelet-client.crt -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 5363063852586150726 (0x4a6d6e28c122ab46)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: Dec 16 12:31:51 2019 GMT
            Not After : Dec 15 12:31:52 2020 GMT
        Subject: O=system:masters, CN=kube-apiserver-kubelet-client
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:df:72:40:f1:6f:0d:02:7d:d6:ce:d4:55:0a:e2:
                    aa:12:e9:dc:2d:7e:b7:a7:5f:42:e5:9b:e2:3d:dd:
                    b8:ec:7c:d8:b9:36:33:5b:07:76:ec:e0:f4:75:34:
                    8a:94:0d:21:f4:3d:66:2f:9e:95:58:62:d3:d7:ee:
                    25:f0:c3:ea:fb:4c:30:6e:18:ef:78:40:61:6b:3b:
                    55:12:f7:72:42:b6:87:c0:85:2f:03:d2:b9:a9:72:
                    3e:12:ae:db:a0:e4:37:13:18:35:ad:7f:41:ab:7a:
                    3f:55:a9:68:9e:24:2a:69:db:9f:1e:6f:10:9f:9f:
                    d0:6b:76:83:28:02:36:29:2b:41:53:87:63:36:c1:
                    79:aa:58:35:6a:3f:46:9f:b7:7d:58:cc:1a:c2:d8:
                    85:de:5b:81:07:2d:08:cd:ef:d0:08:e2:b2:05:2d:
                    9c:27:cb:4e:e4:55:e6:2b:26:86:4b:11:a3:d1:27:
                    53:f4:b9:75:5d:1d:fc:ee:c5:1d:07:6a:27:e3:a1:
                    0f:ae:b7:2b:14:01:b4:ff:46:0b:d0:4b:4e:fa:5b:
                    94:93:66:84:2f:49:ed:9c:23:65:bf:09:ee:69:c8:
                    6a:66:be:54:09:46:d0:5c:7f:a4:bd:9a:2c:5d:66:
                    b0:c6:ba:3c:f5:b7:8a:3c:b1:ba:3f:9c:b3:98:44:
                    41:7d
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Client Authentication
    Signature Algorithm: sha256WithRSAEncryption
         97:fa:aa:29:13:16:43:3c:66:f4:3e:c9:6c:18:9b:86:fb:ef:
         fb:12:2a:b8:64:cd:f6:44:b4:49:70:5b:b7:f1:a6:df:68:55:
         61:24:18:2a:ef:5e:19:6e:0e:c7:48:1f:ff:cc:bc:6e:d0:5e:
         b9:3d:59:46:a2:0b:0b:ce:6d:ca:ce:58:8e:b2:7c:fe:0d:70:
         14:11:20:1c:e8:9b:0c:e4:fe:f3:81:c3:d4:e5:65:16:f3:d8:
         a1:3e:c1:c5:e9:31:e9:6e:7c:7e:29:69:52:06:cc:f2:4c:ad:
         bd:52:09:55:3f:a9:ce:fb:84:df:d4:b4:04:de:dd:01:35:4f:
         6e:25:d2:d7:12:5b:34:05:8a:f0:f2:19:34:16:9f:f3:96:c3:
         24:ab:e0:be:01:70:fa:bc:4f:f2:31:43:bf:db:f8:de:6a:6d:
         0d:78:b1:8d:16:3e:53:6d:92:17:41:e1:c3:6a:92:e3:83:f9:
         3e:82:0e:56:7f:6d:cb:b8:aa:e2:bf:77:69:8f:13:a2:67:e4:
         df:e3:21:a6:12:f8:a4:bc:02:c7:70:0b:67:07:5b:61:97:f6:
         fa:74:0a:36:9c:46:5a:08:c3:70:2a:64:9b:66:ff:ec:fe:f2:
         80:84:06:fa:8f:c4:45:de:28:45:34:1b:f3:18:8f:db:5a:b8:
         59:a5:9f:76

数据备份

etcd数据备份

通常，etcd数据存储在/var/lib/etcd目录下，如果不是，请核实etcd数据存储路径，然后备份。

$ cp -r /var/lib/etcd /var/lib/etcd-bak

证书备份

$ mkdir -p /etc/kubernetes/pki-bak
$ mv /etc/kubernetes/pki/*.crt /etc/kubernetes/pki-bak
$ mv /etc/kubernetes/pki/*.key /etc/kubernetes/pki-bak

conf文件备份

$ mkdir -p /etc/kubernetes/conf
$ mv /etc/kubernetes/*.conf /etc/kubernetes/conf

生成新证书

$ kubeadm alpha phase certs apiserver --apiserver-advertise-address ${MASTER_API_SERVER_IP}
$ kubeadm alpha phase certs apiserver-kubelet-client
$ kubeadm alpha phase certs front-proxy-client

生成新配置文件

$ kubeadm alpha phase kubeconfig all --apiserver-advertise-address ${MASTER_API_SERVER_IP}

更新admin配置文件

$ cp /etc/kubernetes/admin.conf /root/.kube/conf

重建服务

$ cd /etc/kubernetes/manifests
$ mv *.yaml ..
$ mv ../*.yaml .

验证

通过如下命令查看node的健康状态，如果证书无误，所有节点将变为Ready状态。

$ kubectl get node