一、节点规划

主机名	主机IP
k8s-master	172.17.185.91
k8s-node01	172.17.185.92
k8s-node02	172.17.185.93
k8s-node03	172.17.185.94

二、所有节点操作

1、关闭防火墙及selinux

systemctl stop firewalld && systemctl disable firewalld

sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config  && setenforce 0

2、关闭 swap 分区

# 临时
swapoff -a 

#永久
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab 

3、设置主机名

# 在172.17.185.91主机执行
hostnamectl set-hostname k8s-master

# 在172.17.185.92主机执行
hostnamectl set-hostname k8s-node01

# 在172.17.185.93主机执行
hostnamectl set-hostname k8s-node02

# 在172.17.185.94主机执行
hostnamectl set-hostname k8s-node03

4、配置本地域名解析

cat > /etc/hosts <<EOF
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.17.185.91 k8s-master
172.17.185.92 k8s-node01
172.17.185.93 k8s-node02
172.17.185.94 k8s-node03
EOF

5、将桥接的IPv4流量传递到iptables的链

cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

sysctl --system

6、时间同步配置

yum install -y ntpdate

为保证节点的时间请保持一致,在crontab中配置：

(echo "*/5 * * * * /usr/sbin/ntpdate -u cn.pool.ntp.org") | crontab

systemctl restart crond

7、安装docker

# 安装yum源
yum -y install yum-utils
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

# 安装最新的docker软件包
yum -y install docker-ce

# 配置docker仓库源地址

mkdir -p /etc/docker/

cat > /etc/docker/daemon.json <<EOF
{
    "registry-mirrors":[
        "https://docker.mirrors.ustc.edu.cn",
        "http://hub-mirror.c.163.com"
    ]
}
EOF

# 启动并查看docker服务
systemctl enable docker && systemctl start docker

8、添加kubernetes的阿里YUM软件源

cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

9、所有主机安装kubeadm,kubelet和kubectl

yum install -y kubelet-1.18.0 kubeadm-1.18.0 kubectl-1.18.0

systemctl enable kubelet

三、部署Kubernetes Master

本章节只需要在Master节点执行。

1、使用kubeadm部署master节点

kubeadm init \
--apiserver-advertise-address=172.17.185.91 \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.18.0 \
--service-cidr=10.1.0.0/16 \
--service-cidr=10.10.0.0/16 \
--pod-network-cidr=10.122.0.0/16

说明：(1)由于默认拉取镜像地址k8s.gcr.io国内无法访问，这里指定阿里云镜像仓库地址。(2) apiserver-advertise-address需要修改成自己的master地址。（3）当命令执行错误机器时，可用命令kubeadm reset对节点进行恢复

W1019 11:38:39.482069    2578 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.18.0
[preflight] Running pre-flight checks
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
        [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.10.0.1 172.17.185.91]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [172.17.185.91 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [172.17.185.91 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W1019 11:39:33.273004    2578 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W1019 11:39:33.273620    2578 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 21.005309 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.18" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-master as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: nofewt.tgotrk7m72yjnxwz
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.17.185.91:6443 --token nofewt.tgotrk7m72yjnxwz \
    --discovery-token-ca-cert-hash sha256:57f15cdef16774b7f4089a4f2b09410c7c52631ead802f59a646e89edb76bd0d

根据输出提示操作：

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

2、安装calico网络

说明：尝试网上的安装flannel网络时会被墙。

kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

四、加入Kubernetes Node

本章在两个 Node 节点执行。

使用kubeadm join 注册Node节点到Matser,kubeadm join 的内容，在上面kubeadm init 已经生成好了

# 在172.17.185.92主机执行
kubeadm join 172.17.185.91:6443 --token nofewt.tgotrk7m72yjnxwz \
    --discovery-token-ca-cert-hash sha256:57f15cdef16774b7f4089a4f2b09410c7c52631ead802f59a646e89edb76bd0d

# 在172.17.185.93主机执行
kubeadm join 172.17.185.91:6443 --token nofewt.tgotrk7m72yjnxwz \
    --discovery-token-ca-cert-hash sha256:57f15cdef16774b7f4089a4f2b09410c7c52631ead802f59a646e89edb76bd0d

# 在172.17.185.94主机执行
kubeadm join 172.17.185.91:6443 --token nofewt.tgotrk7m72yjnxwz \
    --discovery-token-ca-cert-hash sha256:57f15cdef16774b7f4089a4f2b09410c7c52631ead802f59a646e89edb76bd0d

加入node节点后，回到master节点执行：kubectl get nodes ,知道STATUS均为Ready状态时代表集群安装完成。

[root@localhost ~]# kubectl get nodes
NAME         STATUS   ROLES    AGE     VERSION
k8s-master   Ready    master   2m54s   v1.18.0
k8s-node01   Ready    <none>   89s     v1.18.0
k8s-node02   Ready    <none>   87s     v1.18.0
k8s-node03   Ready    <none>   84s     v1.18.0

五、kubeadm添加新master

首先，在master上生成新的token

# kubeadm token create --print-join-command
W1111 17:50:25.985706  292853 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
kubeadm join apiserver.cluster.local:6443 --token sc2ty3.ej38ceisi5lmt9ad     --discovery-token-ca-cert-hash sha256:42bf6e526b795854b61b7c0ca875f9a8292b989d44f0f51a4d8dec450711b89e

然后，在master上生成用于新master加入的证书

# kubeadm init phase upload-certs --upload-certs
I1111 17:50:52.634857  293705 version.go:252] remote version is much newer: v1.19.3; falling back to: stable-1.18
W1111 17:50:53.498664  293705 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
c5c77a2b5989c75c0ec98fae91f771c569e5764523fd8daa102a1cb074c07e2f

最后，添加新master节点，把最会一行加到–control-plane --certificate-key后。

# kubeadm join apiserver.cluster.local:6443 --token sc2ty3.ej38ceisi5lmt9ad \
  --discovery-token-ca-cert-hash sha256:42bf6e526b795854b61b7c0ca875f9a8292b989d44f0f51a4d8dec450711b89e \
  --control-plane --certificate-key c5c77a2b5989c75c0ec98fae91f771c569e5764523fd8daa102a1cb074c07e2f

六、安装k8s管理工具

1、kuboard

请参考文档：https://www.kuboard.cn/install/install-dashboard.html

2、rancher

docker run -d --privileged --restart=always --name rancher -p 80:80 -p 443:443 rancher/rancher:stable

七、证书过期问题

kubeadm 默认证书为一年，一年过期后，会导致api service不可用，使用过程中会出现：x509: certificate has expired or is not yet valid.

处理方法：[1] https://www.cnblogs.com/skymyyang/p/11093686.html

[2] https://www.kancloud.cn/pshizhsysu/kubernetes/2204250