K8s 二进制扩容node节点方案

基于以下链接文档，k8s环境扩容
链接: https://pan.baidu.com/s/1_DopTLhdWujC3_6Ed3gyxg 提取码: eu2g

网络上的文档没有能够让你直接copy走拿去就能用的文档，但是大多数作者，都是根据自己的需求来进行的，二进制虽然不如kubeadm方式简单，一条两条命令基本搞定，但是基本上就是按照当时集群的部署文档copy的一遍而已，在扩容node节点的时候首先考虑几个问题。

1.node节点上的必备服务

docker，cni插件（calio，fannel），kubelet，kube-proxy

2.他们直接如果和k8s master建立关系

也就是kubelet和kube-proxy如何于apiserver建立连接呢？

通过下面的例子我们就能看到所有的组件的配置文件都有记录在与apiserver建立连接，kube-proxy和kubelet自然也不会例外。

[root@master kubernetes]# find . | grep -nr 6443
scheduler.kubeconfig:5:    server: https://192.168.0.110:6443
kubelet.kubeconfig:5:    server: https://192.168.0.110:6443
kube-proxy.kubeconfig:5:    server: https://192.168.0.110:6443
bootstrap-kubelet.kubeconfig:5:    server: https://192.168.0.110:6443
admin.kubeconfig:5:    server: https://192.168.0.110:6443
controller-manager.kubeconfig:5:    server: https://192.168.0.110:6443

除此以为需要证书文件，才可以建立连接，因为我们可以看到https://192.168.0.110:6443，这里用的则是https协议，另外k8s二进制的通信原理大家也都知道，ssl证书啦，tls协议啦，不懂的自己先百度一下吧。

3.如果单master节点扩容为多master节点呢

也简单，一通百通，改天我在把方案补充出来，与之前单master节点的部署方式并无太大区别，只是在扩容之前考虑一个问题为什么要多master，肯定大家都知道高可用什么什么的，那高可用的核心问题点在哪里？我们要了解两个最最最核心的组件，apiserver和etcd，apiserver是所有组件的枢纽，etcd存储k8s集群的元数据

apiserver要考虑的是因为所有的组件，任何的交互信息都要通过apiserver来进行，那apiserver如果不堪重负挂掉啦，那集群不就瘫掉了吗，所以多master的好处就显而易见，那要怎么做呢，其实也特别简单，我这里比如用什么haproxy+keeplived，通过master节点的ip:6443端口，实现轮询，这里的问题不久解决了吗。

etcd主要承担的则是数据的冗余，及时我挂了一个master节点，至少还有一个节点的etcd保存了集群的相关信息etcd 目前默认使用 2379 端口提供 HTTP API 服务， 2380 端口和 peer 通信，直接在配置中建立etcd集群的关系即可，如下所示

通过etcd的配置文件即可建立

initial-cluster: 'k8s-master01=https://192.168.0.107:2380,k8s-master02=https://192.168.0.108:2380,k8s-master03=https://192.168.0.109:2380

1.初始化系统配置

1.1 配置hosts

192.168.0.110   master1
192.168.0.102   node1
192.168.0.63    node2
192.168.0.55    node3

1.2 配置yum源

curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
sed -i -e '/mirrors.cloud.aliyuncs.com/d' -e '/mirrors.aliyuncs.com/d' /etc/yum.repos.d/CentOS-Base.repo

1.3必备工具安装

yum install wget jq psmisc vim net-tools telnet yum-utils device-mapper-persistent-data lvm2 git  wget jq psmisc -y

1.4节点关闭firewalld 、dnsmasq、selinux

systemctl disable --now firewalld 
systemctl disable --now dnsmasq
systemctl disable --now NetworkManager

setenforce 0
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config

1.5节点关闭swap分区，fstab注释swap

swapoff -a && sysctl -w vm.swappiness=0
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab

1.6 节点同步时间

rpm -ivh http://mirrors.wlnmp.com/centos/wlnmp-release-centos.noarch.rpm
yum install ntpdate -y

ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
echo 'Asia/Shanghai' >/etc/timezone
ntpdate time2.aliyun.com
# 加入到crontab
*/5 * * * * /usr/sbin/ntpdate time2.aliyun.com

1.7 节点配置limit：

ulimit -SHn 65535

vim /etc/security/limits.conf
# 末尾添加如下内容
* soft nofile 65535
* hard nofile 65535
* soft nproc 65535
* hard nproc 65535
* soft memlock unlimited
* hard memlock unlimited　

1.8 免密登陆

#master节点执行

ssh-copy-id -i .ssh/id_rsa.pub 192.168.0.55 

1.9 拷贝节点安装文件

#master节点执行

scp -r /root/k8s-ha-install/ node3:/root/

1.10内核升级

yum update -y --exclude=kernel* 

cd /root
wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm
wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm
cd /root && yum localinstall -y kernel-ml*

grub2-set-default  0 && grub2-mkconfig -o /etc/grub2.cfg
grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"
grubby --default-kernel

reboot 

uname -a

1.11更改主机名

hostname node3
hostnamectl set-hostname node3
bash

1.12安装ipvsadm

#所有节点配置ipvs模块，在内核4.19+版本nf_conntrack_ipv4已经改为nf_conntrack， 4.18以下使用nf_conntrack_ipv4即可

yum install ipvsadm ipset sysstat conntrack libseccomp -y


modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
vim /etc/modules-load.d/ipvs.conf 
	# 加入以下内容
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip



#启动服务
systemctl enable --now systemd-modules-load.service

#检查是否加载
[root@node3 ~]#  lsmod | grep -e ip_vs -e nf_conntrack
nf_conntrack_netlink    40960  0 
nfnetlink              16384  3 nf_conntrack_netlink,ip_set
ip_vs_ftp              16384  0 
nf_nat                 32768  2 nf_nat_ipv4,ip_vs_ftp
ip_vs_sed              16384  0 
ip_vs_nq               16384  0 
ip_vs_fo               16384  0 
ip_vs_dh               16384  0 
ip_vs_lblcr            16384  0 
ip_vs_lblc             16384  0 
ip_vs_wlc              16384  0 
ip_vs_lc               16384  0 
ip_vs_sh               16384  0 
ip_vs_wrr              16384  0 
ip_vs_rr               16384  13 
ip_vs                 151552  37 ip_vs_wlc,ip_vs_rr,ip_vs_dh,ip_vs_lblcr,ip_vs_sh,ip_vs_fo,ip_vs_nq,ip_vs_lblc,ip_vs_wrr,ip_vs_lc,ip_vs_sed,ip_vs_ftp
nf_conntrack          143360  6 xt_conntrack,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink,ip_vs
nf_defrag_ipv6         20480  1 nf_conntrack
nf_defrag_ipv4         16384  1 nf_conntrack
libcrc32c              16384  3 nf_conntrack,nf_nat,ip_vs

1.13开启一些k8s集群中必须的内核参数，所有节点配置k8s内核：

cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720

net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
EOF


#执行命令
sysctl --system && sysctl -p

2.安装Docker

yum install docker-ce-19.03.* -y

[root@node3 ~]# vim /etc/docker/daemon.json 
#将文本改成如下，推荐从其他节点copy
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
      "max-size": "100m"
  },
  "insecure-registries": ["https://hub.test.com:18443"]
}


#启动docker
systemctl daemon-reload && systemctl enable --now docker

3.安装核心服务前准备

mkdir -p /var/lib/kubelet /var/log/kubernetes /etc/systemd/system/kubelet.service.d /etc/kubernetes/manifests/  /etc/kubernetes/pki /opt/cni/bin

#在master节点执行
scp /usr/local/bin/kube{let,-proxy} node3:/usr/local/bin/ 

#复制相关证书到node节点
[root@master1 ~]# for FILE in etcd-ca.pem etcd.pem etcd-key.pem;do  scp /etc/etcd/ssl/$FILE node3:/etc/etcd/ssl/; done
etcd-ca.pem                                                                                                                                                  100% 1367   672.2KB/s   00:00    
etcd.pem                                                                                                                                                     100% 1452     1.3MB/s   00:00    
etcd-key.pem                                                                                                                                                 100% 1675     1.3MB/s   00:00    
[root@master1 ~]# for FILE in pki/ca.pem pki/ca-key.pem pki/front-proxy-ca.pem bootstrap-kubelet.kubeconfig; do  scp /etc/kubernetes/$FILE node3:/etc/kubernetes/${FILE}; done
ca.pem                                                                                                                                                       100% 1411   966.2KB/s   00:00    
ca-key.pem                                                                                                                                                   100% 1675     1.3MB/s   00:00    
front-proxy-ca.pem                                                                                                                                           100% 1143   899.7KB/s   00:00    
bootstrap-kubelet.kubeconfig                                                                                                                                 100% 2300   139.2KB/s   00:00   

4.配置和安装kubelet

vim  /usr/lib/systemd/system/kubelet.service

[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=docker.service
Requires=docker.service

[Service]
ExecStart=/usr/local/bin/kubelet

Restart=always
StartLimitInterval=0
RestartSec=10

[Install]
WantedBy=multi-user.target



vim /etc/systemd/system/kubelet.service.d/10-kubelet.conf

[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig"
Environment="KUBELET_SYSTEM_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_CONFIG_ARGS=--config=/etc/kubernetes/kubelet-conf.yml --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.2"
Environment="KUBELET_EXTRA_ARGS=--node-labels=node.kubernetes.io/node='' "
ExecStart=
ExecStart=/usr/local/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_SYSTEM_ARGS $KUBELET_EXTRA_ARGS



vim /etc/kubernetes/kubelet-conf.yml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.pem
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
cgroupsPerQOS: true
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
enforceNodeAllocatable:
- pods
eventBurst: 10
eventRecordQPS: 5
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kubeAPIBurst: 10
kubeAPIQPS: 5
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
registryBurst: 10
registryPullQPS: 5
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s

#启动服务

systemctl daemon-reload
systemctl enable --now kubelet

#master执行
[root@master1 ~]# kubectl get nodes
NAME      STATUS   ROLES    AGE     VERSION
master1   Ready    <none>   29d     v1.20.0
node1     Ready    <none>   29d     v1.20.0
node2     Ready    <none>   29d     v1.20.0
node3     Ready    <none>   3m      v1.20.0

问题梳理：

[root@node3 ~]# journalctl -xefu kubelet | grep fai
Jun 02 19:33:39 node3 kubelet[24182]: F0602 19:33:39.786231   24182 server.go:269] failed to run Kubelet: misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs"
Jun 02 19:33:39 node3 systemd[1]: Unit kubelet.service entered failed state.
Jun 02 19:33:39 node3 systemd[1]: kubelet.service failed.
Jun 02 19:33:54 node3 kubelet[24431]: E0602 19:33:54.914011   24431 kubelet.go:1271] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data in memory cache
Jun 02 19:33:54 node3 kubelet[24431]: E0602 19:33:54.922192   24431 nodelease.go:49] failed to get node "node3" when trying to set owner ref to the node lease: nodes "node3" not found
Jun 02 19:33:54 node3 kubelet[24431]: E0602 19:33:54.993972   24431 eviction_manager.go:260] eviction manager: failed to get summary stats: failed to get node info: node "node3" not found
Jun 02 19:33:55 node3 kubelet[24431]: W0602 19:33:55.518361   24431 driver-call.go:149] FlexVolume: driver call failed: executable: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds/uds, args: [init], error: fork/exec /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds/uds: no such file or directory, output: ""
##该问题检查/etc/docker/daemon.json文件配置，最好从其他健康的node节点直接复制

Jun 02 19:36:34 node3 kubelet[26467]: E0602 19:36:34.012055   26467 pod_workers.go:191] Error syncing pod 5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5 ("calico-node-7zvxw_kube-system(5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5)"), skipping: failed to "StartContainer" for "calico-node" with CrashLoopBackOff: "back-off 10s restarting failed container=calico-node pod=calico-node-7zvxw_kube-system(5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5)"
Jun 02 19:36:42 node3 kubelet[26467]: E0602 19:36:42.628358   26467 pod_workers.go:191] Error syncing pod 5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5 ("calico-node-7zvxw_kube-system(5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5)"), skipping: failed to "StartContainer" for "calico-node" with CrashLoopBackOff: "back-off 10s restarting failed container=calico-node pod=calico-node-7zvxw_kube-system(5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5)"
Jun 02 19:36:59 node3 kubelet[26467]: E0602 19:36:59.097043   26467 pod_workers.go:191] Error syncing pod 5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5 ("calico-node-7zvxw_kube-system(5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5)"), skipping: failed to "StartContainer" for "calico-node" with CrashLoopBackOff: "back-off 20s restarting failed container=calico-node pod=calico-node-7zvxw_kube-system(5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5)"
Jun 02 19:37:02 node3 kubelet[26467]: E0602 19:37:02.628171   26467 pod_workers.go:191] Error syncing pod 5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5 ("calico-node-7zvxw_kube-system(5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5)"), skipping: failed to "StartContainer" for "calico-node" with CrashLoopBackOff: "back-off 20s restarting failed container=calico-node pod=calico-node-7zvxw_kube-system(5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5)"

#无关紧要，因为我没安装kube-proxy，因此cni插件CrashLoopBackOff

#还有一些奇怪的日志，但是kubelet如果能够正常启动的话，先不要管这些，因为类似于kube-proxy和cni的依赖问题等等，安装后重启即可解决

5.配置和安装kube-proxy

master节点操作

 scp /etc/kubernetes/kube-proxy.kubeconfig node3:/etc/kubernetes/kube-proxy.kubeconfig
 scp kube-proxy/kube-proxy.conf node3:/etc/kubernetes/kube-proxy.conf
 scp kube-proxy/kube-proxy.service node3:/usr/lib/systemd/system/kube-proxy.service

Node3节点启动kube-proxy

systemctl daemon-reload
systemctl enable --now kube-proxy

6.cni

我这里用的是calico，因为类似于系统插件都是用的daemonset方式部署，所以直接apply即可

kubectl apply -f calico-etcd.yaml

kubectl  get po -n kube-system

如果容器状态异常可以使用kubectl describe 或者logs查看容器的日志