kubeadm安装的k8s集群证书过期处理

kubeadm安装的k8s证书过期处理

一、背景说明

1. kubeadm默认证书为一年，一年过期后，会导致api service不可用，使用过程中会出现：x509: certificate has expired or is not yet valid
2. kubelet证书分为server和client两种， k8s1.9开始默认启用了client证书的自动轮换，但server证书自动轮换需要用户配置开启

二、开启server证书自动轮换

此方案适用于证书还未过期

1. 增加kubelet参数

#在/etc/sysconfig/kubelet增加，若多master，都需要配置：
KUBELET_EXTRA_ARGS=--feature-gates=RotateKubeletServerCertificate=true --rotate-server-certificates=true

2. 配置kube-controller-manager

cat /etc/kubernetes/manifests/kube-controller-manager.yaml
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
component: kube-controller-manager
tier: control-plane
name: kube-controller-manager
namespace: kube-system
spec:
containers:
- command:
- kube-controller-manager
- --experimental-cluster-signing-duration=87600h0m0s #增加证书颁发时间参数
- --feature-gates=RotateKubeletServerCertificate=true #开启server证书签发
- --allocate-node-cidrs=true

3. 创建rbac对象，允许节点轮换kubelet server证书

cat > ca-update.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
rules:
- apiGroups:
- certificates.k8s.io
resources:
- certificatesigningrequests/selfnodeserver
verbs:
- create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kubeadm:node-autoapprove-certificate-server
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:nodes
EOF
kubectl apply -f ca-update.yaml

4. 重启kubelet

systemctl restart kubelet
#查看csr,状态会由Pending to Approved
kubectl get csr

5. 多master其他节点一直处于Pending

#出于安全原因,处于pending状态的master节点需要手动审批
kubectl certificate approve <name>

三、替换server证书

此方案适用于证书已过期，处理完成后，再执行开启server证书自动轮换

1. 报错信息

kubectl get po
Unable to connect to the server: x509: certificate has expired or is not yet valid

2. 证书备份

cp -Ra /etc/kubernetes /opt/kubernetes-backup-time

3. 删除过期证书

#apiserver证书
rm -f /etc/kubernetes/pki/apiserver*
#front-proxy-client证书
rm -f /etc/kubernetes/pki/front-proxy-client.*
#etcd证书，若etcd是部署在集群外自签证书，不执行以下命令
rm -rf /etc/kubernetes/pki/etcd/healthcheck-client.*
rm -rf /etc/kubernetes/pki/etcd/server.*
rm -rf /etc/kubernetes/pki/etcd/peer.*

4. 重新生成证书

#下载对应版本的kubeadm
wget https://dl.k8s.io/release/v1.10.1/bin/linux/amd64/kubeadm
chmod a+x kubeadm
#生成证书,若使用HA需要配置成vip地址
./kubeadm alpha phase certs all --apiserver-advertise-address <IP address of your master server>

5. 重新生成配置文件

#备份配置文件
mv /etc/kubernetes/*.conf /tmp
#生成配置文件
./kubeadm alpha phase kubeconfig all --apiserver-advertise-address <IP address of your master server>

6. 重启kubelet

systemctl restart kubelet

7. 验证集群

#查看证书过期时间
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep ' Not '
#集群节点状态
kubectl get no

四、参考

• kubelet-tls-bootstrapping
• certificate-rotation