操作系统：CentOS Linux release 7.7.1908 (Core)
docker版本：19.03
kubernetes版本：v1.19.0
至少2G内存、2CPU
网络良好
特定端口是开放的(前期可以把防火墙给禁用掉)

一、配置网络

１．主机网络采用静态网址，例：

vim  /etc/sysconfig/network-scripts/ifcfg-ens33

TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static    #设置为静态地址
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens33
UUID=e4f81192-b2e7-407b-90ea-a36bd0cc230a
DEVICE=ens33
ONBOOT=YES
IPADDR="192.168.１.100" #本机ip地址
PREFIX="24"
GATEWAY="192.168.１.2"  #默认网关
DNS1="8.8.8.8"          #DNS

２．重启网络

 systemctr restart network

其他主机采用类似方法安装。

二、配置负载均衡代理

这里使用keepalived代理，实现统一入口，(三节点的集群API请求不会很高，可以不需要负载均衡，只需实现高可用即可。)可以参考官网文档，使用keepalived+nginx、haproxy、LVS(三选一)。

参考k8s官网高可用集群方案

１．所有主机安装keepalived

yum -y install keepalived

２．修改配置文件

vim  /etc/keepalived/keepalived.conf

global_defs {
   router_id k8s-node1  #主机唯一标识，主机间不可相同
   vrrp_skip_check_adv_addr
   #vrrp_strict  #要注释掉，否则可能ping不通VIP
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_instance k8s {
    state BACKUP  #服务启动时的角色
    interface ens192  #绑定的网卡名
    virtual_router_id 110  #虚拟路由，三台主机必须一致
    priority 80  #服务启动时，抢占VIP的优先级
    advert_int 1
    nopreempt  #故障恢复后，不抢占VIP，避免主节点恢复后网络再次动荡
    authentication {
        auth_type PASS
        auth_pass 4567  #广播时的密码，台主机必须一致
    }
    virtual_ipaddress {
        10.95.10.129  #VIP地址
    }
}

３．启动服务

systemctl start keepalived
systemctl enable keepalived

４．查看

ip a s | grep 10.95.10.129

补丁：在操作这一步时，出现过使用“systemctl stop keepalived”命令无法关闭keepalived进程的问题，且启动日志中有　Can’t open PID file /var/run/keepalived.pid (yet?) after start:　报错。

解决方案：
vim /etc/systemd/system/multi-user.target.wants/keepalived.service

注释掉　KillMode=process
#表示只杀掉程序的主进程，打开的子进程不管。我们keepalived肯定要全部杀掉。所以注释掉这一行。默认值是control-group，即当前控制组里的所有进程全部杀掉。
systemctl daemon-reload

systemctl restart keepalived

参考文档：https://blog.csdn.net/Matt_li/article/details/104469950

5．分别在主机上设置主机名

hostnamectl set-hostname airport-k8s-m1

6．配置hosts

  cat >>/etc/host <<EOF
    192.168.１.100 airport-k8s-m1
    192.168.１.101 airport-k8s-m2
    192.168.１.102 airport-k8s-m3
    EOF

三、生成免密登录密钥(可选)

ssh-keygen -f /root/.ssh/id_rsa -N ''

for i in airport-k8s-m1 airport-k8s-m2 airport-k8s-m3
do
ssh-copy-id $i
done

四、关闭selinux

sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config && setenforce 0

五、时间同步

安装 chrony 服务

yum install -y chrony

systemctl start chronyd

systemctl enable chronyd

六、关闭防火墙

在生产环境中，应该是配置防火墙，开放必要的端口

systemctl stop firewalld

systemctl disable firewalld

七、关闭swap分区

sed -i '11s/\/dev/# \/dev/g' /etc/fstab

swapoff -a

八、配置IPVS内核

１．默认情况下，Kube-proxy将在kubeadm部署的集群中以iptables模式运行

以下内容引用《Kubernetes权威指南 第4版》第5.5章

iptables模式虽然实现起来简单,但存在无法避免的缺陷:在集群中 的Service和Pod大量增加以后,iptables中的规则会急速膨胀,导致性能 显著下降,在某些极端情况下甚至会出现规则丢失的情况,并且这种故 障难以重现与排查,于是Kubernetes从1.8版本开始引入第3代的 IPVS(IP Virtual Server)模式,如图5.16所示。IPVS在Kubernetes 1.11 中升级为GA稳定版。
应用程序编程访问API Server iptables与IPVS虽然都是基于Netfilter实现的,但因为定位不同,二者有着本质的差别:iptables是为防火墙而设计的;IPVS则专门用于高性能负载均衡,并使用更高效的数据结构(Hash表),允许几乎无限的 规模扩张,因此被kube-proxy采纳为第三代模式。
与iptables相比,IPVS拥有以下明显优势:
◎ 为大型集群提供了更好的可扩展性和性能;
◎支持比iptables更复杂的复制均衡算法(最小负载、最少连接、 加权等);
◎ 支持服务器健康检查和连接重试等功能;
◎可以动态修改ipset的集合,即使iptables的规则正在使用这个集 合。
由于IPVS无法提供包过滤、airpin-masquerade tricks(地址伪装)、 SNAT等功能,因此在某些场景(如NodePort的实现)下还要与iptables 搭配使用。在IPVS模式下,kube-proxy又做了重要的升级,即使用 iptables的扩展ipset,而不是直接调用iptables来生成规则链。
iptables规则链是一个线性的数据结构,ipset则引入了带索引的数据 结构,因此当规则很多时,也可以很高效地查找和匹配。我们可以将 ipset简单理解为一个IP(段)的集合,这个集合的内容可以是IP地址、 IP网段、端口等,iptables可以直接添加规则对这个“可变的集合”进行操 作,这样做的好处在于可以大大减少iptables规则的数量,从而减少性能 损耗。
假设要禁止上万个IP访问我们的服务器,则用iptables的话,就需要 一条一条地添加规则,会在iptables中生成大量的规则;但是用ipset的话,只需将相关的IP地址(网段)加入ipset集合中即可,这样只需设置 少量的iptables规则即可实现目标。

yum install -y ipset ipvsadm

cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
EOF

chmod +x /etc/sysconfig/modules/ipvs.modules

bash /etc/sysconfig/modules/ipvs.modules

２．配置内核参数

cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.ip_forward = 1
vm.swappiness=0
EOF

modprobe br_netfilter

sysctl -p /etc/sysctl.d/k8s.conf

(后续还有操作！)

３．打开文件数

echo "* soft nofile 65536" >> /etc/security/limits.conf

echo "* hard nofile 65536" >> /etc/security/limits.conf

九、安装docker(参考官网)

sudo yum remove docker docker-client  docker-client-latest  docker-common docker-latest docker-latest-logrotate docker-logrotate docker-engine

rm -rf /var/lib/docker

sudo yum install -y yum-utils  device-mapper-persistent-data  lvm2

sudo yum-config-manager    --add-repo    https://download.docker.com/linux/centos/docker-ce.repo

sudo yum-config-manager --enable docker-ce-nightly

sudo yum-config-manager --enable docker-ce-test

sudo yum-config-manager --disable docker-ce-nightly

sudo yum install docker-ce docker-ce-cli containerd.io

如需指定版本：

sudo yum -y install docker-ce-18.09.1 docker-ce-cli-18.09.1 containerd.io

yum list docker-ce --showduplicates | sort -r

需要配合 kubernetes的地方有:

１．设置开机自动重启

systemctl enable docker && systemctl start docker

２．设置 cgroup驱动使用systemd以及存储

cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ]
}
EOF

３．然后重启一下 docker, 因为修改了配置文件.

systemctl restart docker

十、kubeadm安装

1.安装 kubeadm, kubelet and kubectl

还是一样的所有的节点都要安装这个.

添加源(谷歌源，国内可能访问不到)

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

替换成阿里的源如下:

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

２．安装

sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

如需指定版本

sudo yum install -y kubelet-1.19.0 kubeadm-1.19.0 kubectl-1.19.0 --disableexcludes=kubernetes

sudo systemctl enable --now kubelet

３．kubectl 自动补全

默认安装的有

sudo yum install bash-completion -y

echo "source <(kubectl completion bash)" >> ~/.bashrc

source ~/.bashrc

４．配置 kubelet
kubelet 默认的 cgroup 的驱动为 cgroupfs，而 Docker 默认的 cgroup 的驱动为 systemd，所有这里可以将 kubectl 的默认驱动改为同 Docker 的一样，否则下边执行 kubeadm init 的时候会报错。当然也可以修改 Docker 的驱动为 cgroupfs。

参考官网https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

vim /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf

Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml --cgroup-driver=systemd"

修改完成之后，需要重新 reload 一下 kubelet 服务。

systemctl daemon-reload

systemctl restart kubelet

十一、安装kebenetes集群

安装airport-k8s-m1
１．准备配置文件

cat >kubeadm-config.yaml<<EOF
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: v1.19.0
apiServer:
  certSANs:
  - "10.95.10.129"
controlPlaneEndpoint: "10.95.10.129:6443"
imageRepository: registry.aliyuncs.com/google_containers
networking:
  dnsDomain: cluster.local
  podSubnet: "10.244.0.0/16"
  serviceSubnet: 10.96.0.0/12
EOF

２．下载所需镜像

kubeadm config images pull --config kubeadm-config.yaml

３．初始化

kubeadm init --config kubeadm-config.yaml --upload-certs

这个初始化过程和之前的情况有所不同,加入了一个新的参数:--upload-certs。
这个参数专门用于高可用部署,可以将需要在不同的控制平面之间传递的证书文件上传到集群中,以Secret形式保存起来,并且使用Token进行加密。值得注意的是,这个Secret会在两个小时后过期,如果过期就需要使用kubeadm init phase upload-certs --upload-certs命令重新生成。

输出如下：

[init] Using Kubernetes version: v1.19.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [node1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local api.k8s.gy.com api.k8s.gy.com] and IPs [10.96.0.1 192.168.１.100]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [node1 localhost] and IPs [192.168.１.100 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [node1 localhost] and IPs [192.168.１.100 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 19.007860 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.16" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node node1 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node node1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: ks271v.t4pp7n7k0qd5danp
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:
  kubeadm join 10.95.10.129:6443 --token iet5r9.u5g2m7dknk3eexgr     \
  --discovery-token-ca-cert-hash sha256:dfa5645c58b1abe04842819dce17c56883c5a7c6ed03288585b86342f12da009     \
  --control-plane --certificate-key b89e3ff968c959789845055ca92dd71b42ab01de6112eac60e2d1ac8e4636e33
   
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 10.95.10.129:6443 --token ks271v.t4pp7n7k0qd5danp \
    --discovery-token-ca-cert-hash sha256:6ab48ebd0024ae79aff1a2fcfa63aa2b61e58083b4aa265ef8fc17d54e6dcca6

４．在需对集群管理的节点上执行下列命令：

mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config

十二、安装网络插件

可选的网络插件：https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#pod-network

１．安装calico网络插件

wget https://docs.projectcalico.org/v3.17/manifests/calico.yaml 

编辑calico.yaml文件，修改其中的网段改为前面指定的集群网段10.244.0.0/16，默认calico的网段是192.168.0.0/16，在vi编辑器中键入 \192.168.0.0 快速定位修改处

vim calico.yaml

> - name: CALICO_IPV4POOL_CIDR
>   value: "10.244.0.0/16"

２． 安装网络插件

kubectl apply -f calico.yaml

３．插件安装完成后，可以检查 CoreDNS pod 是否运行正常.

kubectl get pods --all-namespace

十三、以master模式加入集群

１．(kubeadm init时如果加入了--upload-certs参数时，可以忽略此步骤，直接进行第2步。)
当节点需要以master加入集群时，需要主master的证书和key，下面的脚本将本地的证书拷贝到其他master节点。（还有一种方法是使用kubeadm命令拷贝证书）

权限拷贝脚本sync.master.ca.sh。

#!/bin/sh

vhost="airport-k8s-m2 airport-k8s-m3"
usr=root
who=`whoami`

if [[ "$who" != "$usr" ]];then
  echo "请使用 root 用户执行或者 sudo ./sync.master.ca.sh"
  exit 1
fi
echo $who

# 需要从 node1 拷贝的 ca 文件

caFiles=(
/etc/kubernetes/pki/ca.crt
/etc/kubernetes/pki/ca.key
/etc/kubernetes/pki/sa.key
/etc/kubernetes/pki/sa.pub
/etc/kubernetes/pki/front-proxy-ca.crt
/etc/kubernetes/pki/front-proxy-ca.key
/etc/kubernetes/pki/etcd/ca.crt
/etc/kubernetes/pki/etcd/ca.key
/etc/kubernetes/admin.conf
)
pkiDir=/etc/kubernetes/pki/etcd

for h in $vhost
do
  ssh ${usr}@$h "mkdir -p $pkiDir" 
  echo "Dirs for ca scp created, start to scp..."
  # scp 文件到目标机
  	scp /etc/kubernetes/pki/ca.crt /etc/kubernetes/pki/ca.key /etc/kubernetes/pki/sa.key /etc/kubernetes/pki/sa.pub /etc/kubernetes/pki/front-proxy-ca.crt /etc/kubernetes/pki/front-proxy-ca.key    ${usr}@$h:/etc/kubernetes/pki/
    scp /etc/kubernetes/pki/etcd/ca.crt /etc/kubernetes/pki/etcd/ca.key  ${usr}@$h:/etc/kubernetes/pki/etcd/
    scp /etc/kubernetes/admin.conf  ${usr}@$h:/etc/kubernetes/
  echo "Ca files transfered for $h ... ok"
done

２．在airport-k8s-m2和airport-k8s-m3上以下列命令加入集群：（初始化的输出里，sha256值是唯一的，请注意！）

下列命令以master加入集群

kubeadm join 10.95.10.129:6443  --token iet5r9.u5g2m7dknk3eexgr    \
 --discovery-token-ca-cert-hash sha256:dfa5645c58b1abe04842819dce17c56883c5a7c6ed03288585b86342f12da009    \
  --control-plane --certificate-key b89e3ff968c959789845055ca92dd71b42ab01de6112eac60e2d1ac8e4636e33

下列命令可以将节点作为node加入集群

kubeadm join api.k8s.airport.com:6443 --token ks271v.t4pp7n7k0qd5danp \
    --discovery-token-ca-cert-hash sha256:6ab48ebd0024ae79aff1a2fcfa63aa2b61e58083b4aa265ef8fc17d54e6dcca6

３．检查结果

kubectl get nodes

NAME    　　　　STATUS  ROLES    AGE    VERSION
airport-k8s-m1  Ready    master  127m  v1.19.0
airport-k8s-m2  Ready    master  106m  v1.19.0
airport-k8s-m3  Ready    master  103m  v1.19.0

kubectl get pods --all-namespaces

拉取镜像时间较长，耐心等待

４．让master节点可以运行pod（去污）

kubectl taint nodes --all node-role.kubernetes.io/master-

５．如果加入失败，可以清除k8s节点信息，重新加入

以airport-k8s-m2为例

kubectl drain airport-k8s-m2
kubectl delete node airport-k8s-m2

登录已退出集群的airport-k8s-m2服务器重置 kubelet 配置并重新加入集群。

kubeadm reset

注意看kubeadm reset命令的输出信息。

重新加入

kubeadm join 10.95.10.129:6443 --token iet5r9.u5g2m7dknk3eexgr    \
 --discovery-token-ca-cert-hash sha256:dfa5645c58b1abe04842819dce17c56883c5a7c6ed03288585b86342f12da009    \
  --control-plane --certificate-key b89e3ff968c959789845055ca92dd71b42ab01de6112eac60e2d1ac8e4636e33

７．k8s集群token过期重新生成

默认情况下，token会在24小时后过期。如果要在令牌过期后重新向集群中添加新的节点，则需要重新生成token，并获取ca证书sha256编码hash值

# 重新生成token
kubeadm token create
wbekm9.d7m4p57rro7xuwzw

# 获取ca证书sha256编码hash值
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
9db128fe4c68b0e65c19bb49226fc717e64e790d23816c5615ad6e21fbe92020

#在不同的控制平面之间传递证书文件上传到集群的Secret
kubeadm init phase upload-certs --upload-certs
[upload-certs] Using certificate key:
4c75a78f1105910b8566a71aabaa952cf4149840a25bc4e0c1099901e440e402

# 将新的node节点加入到k8s集群中
kubeadm join 10.95.10.129:6443 --token wbekm9.d7m4p57rro7xuwzw \
    --discovery-token-ca-cert-hash sha256:9db128fe4c68b0e65c19bb49226fc717e64e790d23816c5615ad6e21fbe92020 \
    --control-plane --certificate-key 4c75a78f1105910b8566a71aabaa952cf4149840a25bc4e0c1099901e440e

十四、kubernetes启用ipvs模式

１．所有节点验证开启了ipvs：

 lsmod |grep ip_vs

２．安装ipvsadm工具：（所有节点安装ipvsadm）

yum install ipset ipvsadm -y　#上面已经安装

３．编辑kube-proxy配置文件：

 kubectl edit  configmap -n kube-system  kube-proxy

４．重启kube-proxy:

 kubectl get pod -n kube-system              ##查看
 kubectl delete  pod -n kube-system  kube-proxy-2nhf6     ##注意3个kube-proxy依次重启

或者批量重启

 kubectl get pod -n kube-system | grep kube-proxy | awk '{system("kubectl delete pod "$1" -n kube-system")}'

５．查看：

 ipvsadm -ln