一. 环境

• Ubutun20
• 一个master结点，两个node结点，均为2CPU,4G
• 部署Ngnix 测试可用性
• 机器IP

Node	IP	HostName
master	172.17.57.47	master-node
node1	172.17.57.48	k8snode000001
node2	172.17.57.50	k8snode000002

二. 安装

1. 安装Docker (3台机器都安装)

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
sudo apt update
apt install docker.io

cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

2. 安装kubelet kubeadm kubectl (3台机器都安装)

curl -s https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -
cat << EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://mirrors.ustc.edu.cn/kubernetes/apt kubernetes-xenial main
EOF
apt update && apt install -y kubelet kubeadm kubectl

3. 其他系统配置 (3台机器都配置)

# 关闭防火墙
systemctl stop firewalld
systemctl disable firewalld

# 禁用Selinux
apt install selinux-utils
setenforce 0

# 关闭swap
swapoff -a

# 将桥接的IPv4流量传递到iptables的链
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
# 生效
sysctl --system 

4. 启动Docker (3台机器都启动)

systemctl daemon-reload
systemctl enable docker
systemctl start docker

检查启动状态(3台机器都检查一遍)

root@k8snode000001:~# systemctl status docker
● docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2021-09-26 10:26:56 CST; 2 days ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
   Main PID: 985782 (dockerd)
      Tasks: 17
     Memory: 624.8M
     CGroup: /system.slice/docker.service
             ├─ 985782 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
             └─2099528 /usr/bin/unpigz -d -c

或者执行 sudo docker run hello-world

5. 部署master结点

root@lean-master:~# kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=172.17.57.47 --kubernetes-version=v1.22.2 --ignore-preflight-errors=Swap --image-repository registry.aliyuncs.com/google_containers
...
...
To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities 
and service account keys on each node and then running the following as root:

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.17.57.47:6443 --token 1g54n2.zrzzpe32qaxwe90u --discovery-token-ca-cert-hash sha256:acf1b1c06045ffa9e1a8ffb8f5f9f96b138471f3a10e71d0e5b64d0015be4e8

若出现上面log则成功，根据提示还得执行以下命令

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

执行完成后，检查master节点状态

root@lean-master:~# kubectl get nodes
NAME            STATUS      ROLES                  AGE    VERSION
lean-master     NotReady    control-plane,master   2d2h   v1.22.2

能够看到，目前有一个master节点已经运行了，但是还处于未准备状态
下面我们还需要在Node节点执行其它的命令，将node1和node2加入到我们的master节点上

6. 部署Node节点

master节点上获取join命令

root@lean-master:~# kubeadm token create --print-join-command
kubeadm join 172.17.57.47:6443 --token xy3hnh.mxpgpgxncwebv32n --discovery-token-ca-cert-hash sha256:acf1b1c06045ffa9e1a8ffb8f5f9f96b138471f3a10e71d0e5b64d0015be4e84 

复制join命令到各个Node节点上执行

kubeadm join 172.17.57.47:6443 --token xy3hnh.mxpgpgxncwebv32n --discovery-token-ca-cert-hash sha256:acf1b1c06045ffa9e1a8ffb8f5f9f96b138471f3a10e71d0e5b64d0015be4e84

再次查看master节点状态

root@lean-master:~# kubectl get nodes
NAME            STATUS   	ROLES                  AGE    VERSION
k8snode000001   NotReady    <none>                 2d2h   v1.22.2
k8snode000002   NotReady    <none>                 2d1h   v1.22.2
lean-master     NotReady    control-plane,master   2d3h   v1.22.2

发现还是NotReady，接下来按照网络插件

7. 按照网络插件CNI （master节点）

# 下载网络插件配置
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

# 添加
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

查看pods状态

root@lean-master:~# kubectl get pods --all-namespaces
NAMESPACE     NAME                                  READY   STATUS    RESTARTS   AGE
kube-system   coredns-7f6cbbb7b8-97222              1/1     Running   0          2d3h
kube-system   coredns-7f6cbbb7b8-kpnmg              1/1     Running   0          2d3h
kube-system   etcd-lean-master                      1/1     Running   7          2d3h
kube-system   kube-apiserver-lean-master            1/1     Running   6          2d3h
kube-system   kube-controller-manager-lean-master   1/1     Running   0          44h
kube-system   kube-flannel-ds-97npx                 1/1     Running   0          2d3h
kube-system   kube-flannel-ds-l8zv6                 1/1     Running   0          2d3h
kube-system   kube-flannel-ds-n8gmf                 1/1     Running   3          2d1h
kube-system   kube-proxy-98tj7                      1/1     Running   0          2d3h
kube-system   kube-proxy-lsvqj                      1/1     Running   2          2d1h
kube-system   kube-proxy-sxzsh                      1/1     Running   0          2d3h
kube-system   kube-scheduler-lean-master            1/1     Running   0          44h

等待状态Running后，查看master状态

root@lean-master:~# kubectl get nodes
NAME            STATUS   ROLES                  AGE    VERSION
k8snode000001   Ready    <none>                 2d2h   v1.22.2
k8snode000002   Ready    <none>                 2d1h   v1.22.2
lean-master     Ready    control-plane,master   2d2h   v1.22.2

状态全为Ready了，如果还有NotReady的，可以在Master将该节点删除

kubectl delete node k8snode000001
 
# 然后到k8snode000001节点进行重置
 kubeadm reset
# 重置完后在加入
kubeadm join 172.17.57.47:6443 --token 1g54n2.zrzzpe32qaxwe90u --discovery-token-ca-cert-hash sha256:acf1b1c06045ffa9e1a8ffb8f5f9f96b138471f3a10e71d0e5b64d0015be4e84

8. 测试集群

部署一个Ngnix服务

# 下载nginx 【会联网拉取nginx镜像】
root@lean-master:~# kubectl create deployment nginx --image=nginx
deployment.apps/nginx created

# 查看状态
root@lean-master:~# kubectl get pod
NAME                     READY   STATUS    RESTARTS   AGE
nginx-6799fc88d8-czpj4   1/1     Running   0          12s

下面我们就需要将端口暴露出去，让其它外界能够访问

#暴露端口
root@lean-master:~# kubectl expose deployment nginx --port=80 --type=NodePort
service/nginx exposed

#查看服务端口
root@lean-master:~# kubectl get pod,svc
NAME                         READY   STATUS    RESTARTS   AGE
pod/nginx-6799fc88d8-czpj4   1/1     Running   0          68s

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
service/kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP          2d3h
service/nginx        NodePort    10.108.175.89   <none>        80:32218/TCP     5s

查看端口为32218
然后浏览器访问http://<masterip>:32218

三. 常见安装错误

错误一

我们在给node1节点使用 kubernetes join命令的时候，出现以下错误

error execution phase preflight: [preflight] Some fatal errors occurred:
	[ERROR Swap]: running with swap on is not supported. Please disable swap

错误原因是我们需要关闭swap

# 关闭swap
# 临时
swapoff -a 
# 临时
sed -ri 's/.*swap.*/#&/' /etc/fstab

错误二

在给node1节点使用 kubernetes join命令的时候，出现以下错误

The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused

解决方法，首先需要到 master 节点，创建一个文件

# 创建文件
vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

# 添加如下内容
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --fail-swap-on=false"

# 重置
kubeadm reset

# 然后删除刚刚创建的配置目录
rm -rf $HOME/.kube

#然后 在master重新初始化
kubeadm init ....

#然后重新node上
kubeadm join ....

如果依然无法解决检查下 /etc/docker/daemon.json, native.cgroupdriver=systemd

cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF