kubernetes节点减容与扩容
k8s缩容扩容，含二进制与kubeadm环境
·
前言
k8s分二进制安装与kubeadm安装，本次我刚好有两套部署好的环境，就都试一下减容与缩容
二进制
[root@cm1 pv]# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
cm1 Ready <none> 3d20h v1.25.0 10.10.21.197 <none> CentOS Linux 7 (Core) 6.0.8-1.el7.elrepo.x86_64 docker://20.10.21
cm2 Ready <none> 3d20h v1.25.0 10.10.21.198 <none> CentOS Linux 7 (Core) 6.0.8-1.el7.elrepo.x86_64 docker://20.10.21
cm3 Ready <none> 3d20h v1.25.0 10.10.21.199 <none> CentOS Linux 7 (Core) 6.0.8-1.el7.elrepo.x86_64 docker://20.10.21
cn1 Ready <none> 3d20h v1.25.0 10.10.21.200 <none> CentOS Linux 7 (Core) 6.0.8-1.el7.elrepo.x86_64 docker://20.10.21
cn2 Ready <none> 3d20h v1.25.0 10.10.21.201 <none> CentOS Linux 7 (Core) 6.0.8-1.el7.elrepo.x86_64 docker://20.10.21
cn3 Ready <none> 3d20h v1.25.0 10.10.21.202 <none> CentOS Linux 7 (Core) 6.0.8-1.el7.elrepo.x86_64 docker://20.10.21
cn4 Ready <none> 7m v1.25.0 10.10.21.205 <none> CentOS Linux 7 (Core) 6.0.8-1.el7.elrepo.x86_64 docker://20.10.21
kubeadm
root@master1:~/rbd_yaml# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master1.hu.org Ready control-plane 37d v1.25.2 10.10.21.170 <none> Ubuntu 20.04.5 LTS 5.4.0-131-generic docker://20.10.12
master2.hu.org Ready control-plane 36d v1.25.2 10.10.21.172 <none> Ubuntu 20.04.5 LTS 5.4.0-90-generic docker://20.10.12
master3.hu.org Ready control-plane 36d v1.25.2 10.10.21.175 <none> Ubuntu 20.04.5 LTS 5.4.0-90-generic docker://20.10.12
node1.hu.org Ready <none> 37d v1.25.2 10.10.21.171 <none> Ubuntu 20.04.5 LTS 5.4.0-90-generic docker://20.10.12
node2.hu.org Ready <none> 37d v1.25.2 10.10.21.173 <none> Ubuntu 20.04.5 LTS 5.4.0-90-generic docker://20.10.12
node3.hu.org Ready <none> 37d v1.25.2 10.10.21.176 <none> Ubuntu 20.04.5 LTS 5.4.0-90-generic docker://20.10.12
缩容
root@master1:~/rbd_yaml# kubectl get node
NAME STATUS ROLES AGE VERSION
master1.hu.org Ready control-plane 37d v1.25.2
master2.hu.org Ready control-plane 36d v1.25.2
master3.hu.org Ready control-plane 36d v1.25.2
node1.hu.org Ready <none> 37d v1.25.2
node2.hu.org Ready <none> 37d v1.25.2
node3.hu.org Ready <none> 37d v1.25.2
root@master1:~/rbd_yaml# kubectl cordon node3.hu.org # 设置节点不可调度
node/node3.hu.org cordoned
root@master1:~/rbd_yaml# kubectl drain node3.hu.org --delete-emptydir-data --ignore-daemonsets
# 驱逐pod
node/node3.hu.org already cordoned
Warning: ignoring DaemonSet-managed Pods: kube-flannel/kube-flannel-ds-ff5k4, kube-system/kube-proxy-5pnh5, monitor/node-exporter-65pmz
evicting pod monitor/blackbox-exporter-59b7cc76b-br7p9
evicting pod kubegems/kubegems-argo-cd-app-controller-777d8fffb-4l6nr
evicting pod default/nginx-6768c68f7b-dnt7z
evicting pod default/nginx2-b648d744f-kp925
evicting pod kube-system/coredns-c676cc86f-7l2c8
evicting pod kubegems-local/kubegems-local-kubectl-bf9f98658-7t4jv
evicting pod kubegems/kubegems-mysql-0
evicting pod kubegems/kubegems-dashboard-78bd4449c4-ghn2g
evicting pod kubegems/kubegems-gitea-0
evicting pod kubegems/kubegems-redis-master-0
evicting pod kubernetes-dashboard/kubernetes-dashboard-566f484578-gmnnt
pod/kubegems-dashboard-78bd4449c4-ghn2g evicted
pod/kubernetes-dashboard-566f484578-gmnnt evicted
I1118 02:41:07.415515 137843 request.go:682] Waited for 1.088277469s due to client-side throttling, not priority and fairness, request: GET:https://10.10.21.178:6443/api/v1/namespaces/kubegems/pods/kubegems-argo-cd-app-controller-777d8fffb-4l6nr
pod/nginx-6768c68f7b-dnt7z evicted
pod/blackbox-exporter-59b7cc76b-br7p9 evicted
pod/kubegems-local-kubectl-bf9f98658-7t4jv evicted
pod/nginx2-b648d744f-kp925 evicted
pod/kubegems-redis-master-0 evicted
pod/kubegems-argo-cd-app-controller-777d8fffb-4l6nr evicted
pod/kubegems-gitea-0 evicted
pod/kubegems-mysql-0 evicted
pod/coredns-c676cc86f-7l2c8 evicted
node/node3.hu.org drained
root@master1:~/rbd_yaml# kubectl get node
NAME STATUS ROLES AGE VERSION
master1.hu.org Ready control-plane 37d v1.25.2
master2.hu.org Ready control-plane 36d v1.25.2
master3.hu.org Ready control-plane 36d v1.25.2
node1.hu.org Ready <none> 37d v1.25.2
node2.hu.org Ready <none> 37d v1.25.2
node3.hu.org Ready,SchedulingDisabled <none> 37d v1.25.2
root@master1:~/rbd_yaml# kubectl delete node node3.hu.org
node "node3.hu.org" deleted
root@master1:~/rbd_yaml# kubectl get node
NAME STATUS ROLES AGE VERSION
master1.hu.org Ready control-plane 37d v1.25.2
master2.hu.org Ready control-plane 36d v1.25.2
master3.hu.org Ready control-plane 36d v1.25.2
node1.hu.org Ready <none> 37d v1.25.2
node2.hu.org Ready <none> 37d v1.25.2
扩容
以下操作默认已经装好了容器引擎，完成了安装前必要条件
二进制扩容
拷贝证书到新节点上
[root@cn4 kubernetes]# pwd
/etc/kubernetes
[root@cn4 kubernetes]# scp -r cn3:/etc/kubernetes/* .
root@cn3's password:
bootstrap-kubelet.kubeconfig 100% 2232 3.5MB/s 00:00
kubelet-conf.yml 100% 1628 4.9MB/s 00:00
kubelet.kubeconfig 100% 2302 8.4MB/s 00:00
kube-proxy.kubeconfig 100% 6381 18.6MB/s 00:00
kube-proxy.yaml 100% 825 3.6MB/s 00:00
ca.pem 100% 1363 5.2MB/s 00:00
ca-key.pem 100% 1675 1.1MB/s 00:00
front-proxy-ca.pem
启动kubelet和kube-proxy
[root@cn4 kubernetes]# systemctl start kubelet kube-proxy.service
[root@cn4 kubernetes]# systemctl status kubelet kube-proxy.service
● kubelet.service - Kubernetes Kubelet
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2022-11-18 11:03:16 CST; 9s ago
Docs: https://github.com/kubernetes/kubernetes
Main PID: 70687 (kubelet)
Tasks: 14
Memory: 31.4M
CGroup: /system.slice/kubelet.service
└─70687 /usr/local/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --config=/etc/kubernetes/kub...
Nov 18 11:03:17 cn4 kubelet[70687]: I1118 11:03:17.956399 70687 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-local-net-dir\" (UniqueNa...
Nov 18 11:03:17 cn4 kubelet[70687]: I1118 11:03:17.956482 70687 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2glx\" (Uniqu...
Nov 18 11:03:17 cn4 kubelet[70687]: I1118 11:03:17.956497 70687 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"k...
Nov 18 11:03:17 cn4 kubelet[70687]: I1118 11:03:17.956513 70687 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubern...
Nov 18 11:03:17 cn4 kubelet[70687]: I1118 11:03:17.956552 70687 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bpffs\" (UniqueName: \"kuberne...
Nov 18 11:03:17 cn4 kubelet[70687]: I1118 11:03:17.956572 70687 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-bin-dir\" (UniqueName: \"k...
Nov 18 11:03:17 cn4 kubelet[70687]: I1118 11:03:17.956586 70687 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-net-dir\" (UniqueName: \"k...
Nov 18 11:03:17 cn4 kubelet[70687]: I1118 11:03:17.956626 70687 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-log-dir\" (UniqueName: \"k...
Nov 18 11:03:17 cn4 kubelet[70687]: I1118 11:03:17.956642 70687 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"policysync\" (UniqueName: \"ku...
Nov 18 11:03:17 cn4 kubelet[70687]: I1118 11:03:17.956650 70687 reconciler.go:169] "Reconciler: start to sync state"
● kube-proxy.service - Kubernetes Kube Proxy
Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2022-11-18 11:03:16 CST; 9s ago
Docs: https://github.com/kubernetes/kubernetes
Main PID: 70688 (kube-proxy)
Tasks: 5
Memory: 11.1M
CGroup: /system.slice/kube-proxy.service
└─70688 /usr/local/bin/kube-proxy --config=/etc/kubernetes/kube-proxy.yaml --v=2
Nov 18 11:03:17 cn4 kube-proxy[70688]: I1118 11:03:17.884482 70688 service.go:440] "Adding new service port" portName="kube-system/calico-typha:calico-typha" servicePort="1...67:5473/TCP"
Nov 18 11:03:17 cn4 kube-proxy[70688]: I1118 11:03:17.884491 70688 service.go:440] "Adding new service port" portName="kube-system/kube-dns:dns" servicePort="10.96.0.10:53/UDP"
Nov 18 11:03:17 cn4 kube-proxy[70688]: I1118 11:03:17.884498 70688 service.go:440] "Adding new service port" portName="kube-system/kube-dns:dns-tcp" servicePort="10.96.0.10:53/TCP"
Nov 18 11:03:17 cn4 kube-proxy[70688]: I1118 11:03:17.884504 70688 service.go:440] "Adding new service port" portName="kube-system/kube-dns:metrics" servicePort="10.96.0.10:9153/TCP"
Nov 18 11:03:17 cn4 kube-proxy[70688]: I1118 11:03:17.884568 70688 service.go:440] "Adding new service port" portName="kube-system/metrics-server:https" servicePort="10.97....193:443/TCP"
Nov 18 11:03:17 cn4 kube-proxy[70688]: I1118 11:03:17.884579 70688 service.go:440] "Adding new service port" portName="ingress-nginx/ingress-nginx-controller:http" serviceP....189:80/TCP"
Nov 18 11:03:17 cn4 kube-proxy[70688]: I1118 11:03:17.884587 70688 service.go:440] "Adding new service port" portName="ingress-nginx/ingress-nginx-controller:https" service...189:443/TCP"
Nov 18 11:03:17 cn4 kube-proxy[70688]: I1118 11:03:17.884595 70688 service.go:440] "Adding new service port" portName="kube-system/default-http-backend" servicePort="10.107....129:80/TCP"
Nov 18 11:03:17 cn4 kube-proxy[70688]: I1118 11:03:17.884605 70688 service.go:440] "Adding new service port" portName="default/kubernetes:https" servicePort="10.96.0.1:443/TCP"
Nov 18 11:03:17 cn4 kube-proxy[70688]: I1118 11:03:17.884855 70688 proxier.go:1030] "Stale service" protocol="udp" servicePortName="kube-system/kube-dns:dns" clusterIP="10.96.0.10"
Hint: Some lines were ellipsized, use -l to show in full.
# 没报错的话大概率成了，我这边因为证书之前有记录过这个节点，开启服务就直接扩进去了
# 如果没有记录的话kubelt会提示node名字找不到，这个时候需要到master节点操作
kubectl get csr
kubectl certificate approve csr-vx866 # csr是上面get出来的
扩容完成检查一下
[root@cm1 pv]# kubectl get node,cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS ROLES AGE VERSION
node/cm1 Ready <none> 3d21h v1.25.0
node/cm2 Ready <none> 3d21h v1.25.0
node/cm3 Ready <none> 3d21h v1.25.0
node/cn1 Ready <none> 3d21h v1.25.0
node/cn2 Ready <none> 3d21h v1.25.0
node/cn3 Ready <none> 3d21h v1.25.0
node/cn4 Ready <none> 7m50s v1.25.0
NAME STATUS MESSAGE ERROR
componentstatus/scheduler Healthy ok
componentstatus/controller-manager Healthy ok
componentstatus/etcd-2 Healthy {"health":"true","reason":""}
componentstatus/etcd-1 Healthy {"health":"true","reason":""}
componentstatus/etcd-0 Healthy {"health":"true","reason":""}
kubeadm扩容
master节点生成token
root
@master1:~/rbd_yaml# kubeadm token create --print-join-command
kubeadm join 10.10.21.178:6443 --token uo89v1.99wf83u43ntilt41 --discovery-token-ca-cert-hash sha256:44aba1ef82b6b34c40fe748a9c2cd321be91aa3c22dd23e706001b65affb9dc9
新节点扩进来
登录到需要扩容的节点上
root@node3:~# kubeadm join 10.10.21.178:6443 --token uo89v1.99wf83u43ntilt41 --discovery-token-ca-cert-hash sha256:44aba1ef82b6b34c40fe748a9c2cd321be91aa3c22dd23e706001b65affb9dc9
Found multiple CRI endpoints on the host. Please define which one do you wish to use by setting the 'criSocket' field in the kubeadm configuration file: unix:///var/run/containerd/containerd.sock, unix:///var/run/cri-dockerd.sock
To see the stack trace of this error execute with --v=5 or higher
#直接执行master上生成的指令报错
这是因为我容器引擎是docker，我们这边的cri接口不上containerd，所以真实命令应该为
root@node3:~# kubeadm join 10.10.21.178:6443 --token uo89v1.99wf83u43ntilt41 --discovery-token-ca-cert-hash sha256:44aba1ef82b6b34c40fe748a9c2cd321be91aa3c22dd23e706001b65affb9dc9 --cri-socket unix:///run/cri-dockerd.sock
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W1118 02:53:23.454899 2669242 cluster.go:94] error unmarshaling configuration schema.GroupVersionKind{Group:"kubeadm.k8s.io", Version:"v1beta3", Kind:"ClusterConfiguration"}: strict decoding error: unknown field "apiServer.certSANS"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
扩容成功检查一下
root@master1:~/rbd_yaml# kubectl get node,cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS ROLES AGE VERSION
node/master1.hu.org Ready control-plane 37d v1.25.2
node/master2.hu.org Ready control-plane 36d v1.25.2
node/master3.hu.org Ready control-plane 36d v1.25.2
node/node1.hu.org Ready <none> 37d v1.25.2
node/node2.hu.org Ready <none> 37d v1.25.2
node/node3.hu.org Ready <none> 18m v1.25.2
NAME STATUS MESSAGE ERROR
componentstatus/scheduler Healthy ok
componentstatus/controller-manager Healthy ok
componentstatus/etcd-2 Healthy {"health":"true"}
componentstatus/etcd-0 Healthy {"health":"true"}
componentstatus/etcd-1 Healthy {"health":"true"}
小提示
如果遇到如以下的报错，多半是node节点之前没有清除好
[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR FileAvailable--etc-kubernetes-kubelet.conf]: /etc/kubernetes/kubelet.conf already exists
[ERROR Port-10250]: Port 10250 is in use
[ERROR FileAvailable--etc-kubernetes-pki-ca.crt]: /etc/kubernetes/pki/ca.crt already exists
这时候需要将/etc/kubernetes/下清除掉，然后再关闭kubelet再执行扩容命令即可
K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容
更多推荐
0
0- 0
已为社区贡献8条内容
所有评论(0)