问题

testing-node1节点由于资源不足导入状态为NotReady。

解决方案

• 为testing-node1节点重新分配资源（具体方法略）
• 删除testing-node1，此时，原来运行在testing-node1节点上的服务被重新分配到其他节点运行。然后，在master节点执行kubectl delete node testing-node1
• 在master节点执行kubeadm token create --print-join-command生成节点加入集群命令

[root@testing-master ingress]# kubeadm token create --print-join-command
W0611 10:56:39.214887   94437 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
kubeadm join 192.168.8.109:6443 --token 370t4t.lh6bpdlcmiclhg7s     --discovery-token-ca-cert-hash sha256:aa445e2bb5b4e072f12b55395eb146cafbf83f84814218e3a5d0191e2c211e36

• 在testing-node1上执行命令加入集群

kubeadm join 192.168.8.109:6443 --token 370t4t.lh6bpdlcmiclhg7s     --discovery-token-ca-cert-hash sha256:aa445e2bb5b4e072f12b55395eb146cafbf83f84814218e3a5d0191e2c211e36

• 由于之前 kubeadm init 初始化过，所以一些配置文件及服务均已存在，重新执行 kubeadm join 时导致冲突并报错如下

[preflight] Running pre-flight checks.
    [WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 17.12.1-ce. Max validated version: 17.03
    [WARNING FileExisting-crictl]: crictl not found in system path
[preflight] Some fatal errors occurred:
    [ERROR Port-10250]: Port 10250 is in use
    [ERROR FileAvailable--etc-kubernetes-pki-ca.crt]: /etc/kubernetes/pki/ca.crt already exists
    [ERROR FileAvailable--etc-kubernetes-kubelet.conf]: /etc/kubernetes/kubelet.conf already exists12345678

• 在testing-node1节点执行命令kubeadm reset

[root@testing-node1 ~]# kubeadm reset
[reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
[reset] Are you sure you want to proceed? [y/N]: y
[preflight] Running pre-flight checks
W0611 10:59:08.049741   24340 removeetcdmember.go:79] [reset] No kubeadm config, using etcd pod spec to get data directory
[reset] No etcd config found. Assuming external etcd
[reset] Please, manually reset etcd to prevent further issues
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
[reset] Deleting contents of stateful directories: [/var/lib/kubelet /var/lib/dockershim /var/run/kubernetes /var/lib/cni]

The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually by using the "iptables" command.

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.

The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.

• 在testing-node1节点执行kubeadm join 命令

[root@testing-node1 ~]# kubeadm join 192.168.8.109:6443 --token 370t4t.lh6bpdlcmiclhg7s     --discovery-token-ca-cert-hash sha256:aa445e2bb5b4e072f12b55395eb146cafbf83f84814218e3a5d0191e2c211e36
W0611 10:59:29.961899   25370 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
	[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

• 大功告成