1、kubernetes的组织架构

2、kubernetes的集群部署模型

3、k8s安装的环境准备

我这里是自己本地装了3台虚拟机，需要注意k8s对虚拟机的配置有强制要求。
划重点：

• 虚拟机配置必须>=2C2GB(好在VMware可以随时修改虚拟机配置，中途修改成2核4G)
• k8s网络组件地址不能和虚拟机ip重复，后续如果使用虚拟机启用calico作为网络组件，稍改配置即可。
• 我的三台虚拟机ip分别是192.168.149.128/129/130(中间设涉及3台虚拟机通信问题，1是配置/etc/hosts，2是关闭防火墙)

#1、设置主机名(主机名不得重复)
#centos设置方式
hostnamectl set-hostname k8s-master
hostnamectl set-hostname k8s-node1
hostnamectl set-hostname k8s-node2

#2、将SELinux 设置为 permissive 模式（相当于将其禁用）
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

#3、关闭swap
swapoff -a  
sed -ri 's/.*swap.*/#&/' /etc/fstab

#4、允许 iptables 检查桥接流量 将ipv6转至ipv4
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF 

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system

4 、安装kubelet、kubeadm、kubectl

#设置安装镜像地址
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
   http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

#安装三组件
sudo yum install -y kubelet-1.20.9 kubeadm-1.20.9 kubectl-1.20.9 --disableexcludes=kubernetes

sudo systemctl enable --now kubelet

#检查kubelet(厂长)运行状态
systemctl status kubelet

5、使用kubeadm引导集群

5.1 下载机器所需要的镜像容器

#下载k8s内部组件(每台机器都执行)
sudo tee ./images.sh <<-'EOF'
#!/bin/bash
images=(
kube-apiserver:v1.20.9
kube-proxy:v1.20.9
kube-controller-manager:v1.20.9
kube-scheduler:v1.20.9
coredns:1.7.0
etcd:3.4.13-0
pause:3.2
)
for imageName in ${images[@]} ; do
docker pull registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/$imageName
done
EOF

#给images.sh文件赋权限   
chmod +x ./images.sh && ./images.sh

5.2 初始化主节点

#所有机器添加master域名映射，以下需要修改为自己的
#告诉k8s集群谁是master节点
echo "192.168.149.128  cluster-endpoint" >> /etc/hosts

#主节点初始化(kubeadm reset)
kubeadm init \
--apiserver-advertise-address=192.168.149.128 \
--control-plane-endpoint=cluster-endpoint \
--image-repository registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images \
--kubernetes-version v1.20.9 \
--service-cidr=10.96.0.0/16 \
--pod-network-cidr=172.16.0.0/16

#--pod-network-cidr=原始内容是192.168.0.0/16(16是掩码，配置可用子网网段数)
#下面是在网上看到的伪大神提供的虚拟机配置calocl的解决方案，无效大神最终用了Flannel作为网路组件(特此吐槽)
--pod-network-cidr=192.168.149.1/16(IP地址与宿主机地址重复)
#所有网络范围不重叠

下图是初始化成功的界面，初始化失败需要kubeadm reset(切记，后文踩坑记录还有详情)

[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

#1、开启集群前，需要配置再主节点设置常规配置
To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf
  
#2、添加网络插件
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/


#3、添加master节点
You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

  kubeadm join cluster-endpoint:6443 --token xj69sq.eqwhu8s2h9ifrcq9 \
    --discovery-token-ca-cert-hash sha256:2aae2083882147c6e56f877b7d6bf8419c7d2225b96b45509dce89b3fb80cf57 \
    --control-plane 

#4、添加worker节点  
Then you can join any number of worker nodes by running the following on each as root:

kubeadm join cluster-endpoint:6443 --token xj69sq.eqwhu8s2h9ifrcq9 \
    --discovery-token-ca-cert-hash sha256:2aae2083882147c6e56f877b7d6bf8419c7d2225b96b45509dce89b3fb80cf57

5.2.1 设置.kube/config

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

5.2.2 安装网络组件

https://kubernetes.io/docs/concepts/cluster-administration/addons/ [可选的网络组件]
https://projectcalico.docs.tigera.io/about/about-calico [calico]

#添加网络插件的命令
#这里是坑，详情看踩坑下文 
#curl https://docs.projectcalico.org/manifests/calico.yaml -O
curl https://docs.projectcalico.org/v3.20/manifests/calico.yaml -O

kubectl apply -f calico.yaml

5.2.3 修改calico.yaml地址

这里就是前文将的本地虚拟机ip和默认的calico配置地址网段重复，需要另设网段

vi calico.yaml
#由于在kubeadm做初始化时，
#将--pod-network-cidr地址从192.168.0.0/16修改为172.16.0.0/16
#现在需要在calico地址同样修改为改地址172.16.0.0/16
kubectl apply -f calico.yaml

5.2.4 检查k8s启动的pod

#查看集群所有节点
kubectl get nodes

#根据配置文件，给集群创建资源
kubectl apply -f xxxx.yaml

#查看集群部署了哪些应用？ 运行中的应用在docker里面叫容器，在k8s里面叫Pod
docker ps  (docker)
kubectl get pods -A  (k8s)

5.2.5 主节点启动成功

5.3 加入worker节点

#将前述工作节点命令在子节点执行(命令24h内有效) 
Then you can join any number of worker nodes by running the following on each as root:

kubeadm join cluster-endpoint:6443 --token xj69sq.eqwhu8s2h9ifrcq9 \
    --discovery-token-ca-cert-hash sha256:2aae2083882147c6e56f877b7d6bf8419c7d2225b96b45509dce89b3fb80cf57 
        
#当令牌过期后，在master执行以下命令重新获取令牌
kubeadm token create --print-join-command

6、安装可视化界面dashboard

6.1 部署dashboard

//kubernetes官方提供的可视化界面
//https://github.com/kubernetes/dashboard
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.3.1/aio/deploy/recommended.yaml

将yaml中的type：参数从ClusterIP 修改为NodePort

spec:
  clusterIP: 10.96.222.117
  clusterIPs:
  - 10.96.222.117
  externalTrafficPolicy: Cluster
  ports:
  - nodePort: 31019
    port: 443
    protocol: TCP
    targetPort: 8443
  selector:
    k8s-app: kubernetes-dashboard
  sessionAffinity: None
  type: NodePort

暴露可视化界面的端口

//[注意需要开放下图31019的端口]
kubectl get svc -A |grep kubernetes-dashboard

6.3 访问dashboard

使用k8s集群的任意节点+port即可访问

6.4 创建访问账号

由于6.3节中，需要创建Token令牌才能真正的登录，故需执行以下命令

vi dash.yaml
#创建访问账号，准备一个yaml文件； vi dash.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard

执行

kubectl apply -f dash.yaml

6.4 令牌访问

#获取访问令牌
kubectl -n kubernetes-dashboard get secret $(kubectl -n kubernetes-dashboard get sa/admin-user -o jsonpath="{.secrets[0].name}") -o go-template="{{.data.token | base64decode}}"

令牌内容

eyJhbGciOiJSUzI1NiIsImtpZCI6IkFmaENpRTJkVDB2VDVuWmVrc1N1VHYzNktwNDgyN1JGZnlwU2hkRUstc0EifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLWx2ZnZqIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJlYzNhY2M5Mi0xZDhkLTQ3NzktOTM4Yy1kM2I3ZmUyZDdhMzAiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZXJuZXRlcy1kYXNoYm9hcmQ6YWRtaW4tdXNlciJ9.Ke3zR8SN5_JUCjYlxY3i2evaf05cDeT0cSBUuORd09andEsQIhuBjPx21MCU8byIMxz5X2sVLWUCZRvzT0rstoFt7gdTuSwFALYAryciGyK4OCTBCokRsVaH1SHiOYTezHSEo-_glSSdYOFCTEveuogExmnnpnv8QqeOSFmAd5yGs62FcFdlMX14ecm7umKIW2-se9E2PLLP3tDYCZxzM4ouGMFxf-uf1b_ruEg6K01En2xXP-QYA8YOhV7ogL3aUv7EKceiNw-tcpAcL_bX0D3zCgbZMYkB2sCjQ9UMpjD-VYbxPcapwsFmR4AchSrmK_qP5e5KdUx8D0G3nhXm5A

提取令牌过程

6.5 界面输入令牌并访问

7、过程中报错

注意：

kubeadm init\ 执行失败或者强制终止后，需要先执行kubeadm reset重置

7.1 CPU核数小于最低要求2C(内存也必须在2G以上)

解决方法：因为是虚拟机，直接修改内存和核心数即可

7.2 Calico设置桥接数

#[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1

#解决方法：将桥接数设置为1 
echo "1" >/proc/sys/net/bridge/bridge-nf-call-iptables

7.3 Calico版本和kubernetes版本不兼容

报错信息：

[root@k8s-master ~]$ curl https://docs.projectcalico.org/manifests/calico.yaml -O
[root@k8s-master ~]$ kubectl apply -f calico.yaml
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
error: unable to recognize "calico.yaml": no matches for kind "PodDisruptionBudget" in version "policy/v1"

报错原因：Calico与kubernetes版本不兼容
https://projectcalico.docs.tigera.io/archive/v3.20/getting-started/kubernetes/requirements

解决方案：

[root@k8s-master ~]$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.9", GitCommit:"7a576bc3935a6b555e33346fd73ad77c925e9e4a", GitTreeState:"clean", BuildDate:"2021-07-15T21:01:38Z", GoVersion:"go1.15.14", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.9", GitCommit:"7a576bc3935a6b555e33346fd73ad77c925e9e4a", GitTreeState:"clean", BuildDate:"2021-07-15T20:56:38Z", GoVersion:"go1.15.14", Compiler:"gc", Platform:"linux/amd64"}

 curl https://docs.projectcalico.org/v3.20/manifests/calico.yaml -O

7.4 calico更改配置后，k8s配置未更新

报错信息：

Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")

报错原因：
该问题由配置未更新造成（之前的配置还在）
解决方案：

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

#再次执行即可  
kubectl apply -f calico.yaml