K8S/Kubernetes

HA的2种部署方式
一种是将etcd与Master节点组件混布在一起

另外一种方式是，使用独立的Etcd集群，不与Master节点混布

本章是用第一种叠加式安装的

通过kubeadm搭建一个高可用的k8s集群，kubeadm可以帮助我们快速的搭建k8s集群，高可用主要体现在对master节点组件及etcd存储的高可用，文中使用到的服务器ip及角色对应如下：

192.168.200.3 master1
192.168.200.4 master2
192.168.200.5 master3
192.168.200.6 node1
192.168.200.7 node2
192.168.200.8 node3
192.168.200.16 VIP

1、更新仓库

yum -y update

2、导入ELRepo仓库的公共密钥

rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org

3、安装ELRepo仓库的yum源

rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm

4、查询可用内核版本

yum --disablerepo="*" --enablerepo="elrepo-kernel" list available

5、安装最新内核

yum --enablerepo=elrepo-kernel install kernel-ml

6、查看系统上的所有可用内核

awk -F\' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg

7、修改 内核启动顺序

方法1 用 grub2-set-default 0 设置

方法2 编辑 /etc/default/grub 文件，把 GRUB_DEFAULT 设置为 0，生成 grub 配置文件并重启 grub2-mkconfig -o /boot/grub2/grub.cfg

8、重启系统

reboot

9、查看内核

uname -r

10、各节点下载docker源

wget https://download.docker.com/linux/centos/docker-ce.repo -o /etc/yum.repos.d/docker-ce.repo

11、各节点安装docker服务并加入开机启动

yum -y install docker-ce
systemctl start docker && systemctl enable docker

12、各节点配置docker加速器并修改成k8s驱动
daemon.json文件如果没有自己创建

cat /etc/docker/daemon.json

{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ]
}

13、重启docker服务

systemctl restart docker

14、更改各节点主机名

hostnamectl set-hostname 主机名

15、配置各节点hosts文件

cat /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.200.3 master1
192.168.200.4 master2
192.168.200.5 master3
192.168.200.6 node1
192.168.200.7 node2
192.168.200.8 node3

16、关闭各个节点防火墙

systemctl stop firewalld && systemctl disable firewalld

17、关闭各节点SElinux (需要重启服务器)

cat /etc/selinux/config

sed -i 's/^ *SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

18、关闭各节点swap分区

cat /etc/fstab

sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

19、同步各节点的时间

yum -y install chrony && systemctl enable chronyd && systemctl start chronyd

timedatectl set-timezone Asia/Shanghai && chronyc -a makestep

20、各节点内核调整,将桥接的IPv4流量传递到iptables的链

cat /etc/sysctl.d/k8s.conf

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.ip_forward = 1

sysctl -p /etc/sysctl.d/k8s.conf

21、配置各节点k8s的yum源

cat /etc/yum.repos.d/kubernetes.repo

[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

22、各节点安装ipset服务

yum -y install ipvsadm ipset sysstat conntrack libseccomp

23、各节点开启ipvs模块

cat /etc/sysconfig/modules/ipvs.modules 

     #!/bin/sh
     modprobe -- ip_vs
     modprobe -- ip_vs_rr
     modprobe -- ip_vs_wrr
     modprobe -- ip_vs_sh
     modprobe -- nf_conntrack_ipv4

chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4

24、所有master节点安装haproxy和keepalived服务

yum -y install haproxy keepalived

25、修改master1节点keepalived配置文件

cat /etc/keepalived/keepalived.conf

 ! Configuration File for keepalived

global_defs {
   router_id LVS_DEVEL

# 添加如下内容
   script_user root
   enable_script_security
}



vrrp_script check_haproxy {
    script "/etc/keepalived/check_haproxy.sh"         # 检测脚本路径
    interval 3
    weight -2 
    fall 10
    rise 2
}



vrrp_instance VI_1 {
    state MASTER            # MASTER
    interface ens33         # 本机网卡名
    virtual_router_id 51
    priority 100             # 权重100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.200.16      # 虚拟IP
    }
    track_script {
        check_haproxy       # 模块
    }
}

26、修改master2节点keepalived配置文件

cat /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
   router_id LVS_DEVEL

# 添加如下内容
   script_user root
   enable_script_security
}



vrrp_script check_haproxy {
    script "/etc/keepalived/check_haproxy.sh"         # 检测脚本路径
    interval 3
    weight -2 
    fall 10
    rise 2
}



vrrp_instance VI_1 {
    state BACKUP            # BACKUP
    interface ens33         # 本机网卡名
    virtual_router_id 51
    priority 99             # 权重99
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.200.16      # 虚拟IP
    }
    track_script {
        check_haproxy       # 模块
    }
}

27、修改master3节点keepalived配置文件

cat /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
   router_id LVS_DEVEL

# 添加如下内容
   script_user root
   enable_script_security
}



vrrp_script check_haproxy {
    script "/etc/keepalived/check_haproxy.sh"         # 检测脚本路径
    interval 3
    weight -2 
    fall 10
    rise 2
}



vrrp_instance VI_1 {
    state BACKUP            # BACKUP
    interface ens33         # 本机网卡名
    virtual_router_id 51
    priority 98            # 权重98
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.200.16      # 虚拟IP
    }
    track_script {
        check_haproxy       # 模块
    }
}

28、三台master节点haproxy配置都一样

cat /etc/haproxy/haproxy.cfg 

#---------------------------------------------------------------------
# Example configuration for a possible web application.  See the
# full configuration options online.
#
#   http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
#
#---------------------------------------------------------------------

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend  kubernetes-apiserver
    mode                        tcp
    bind                        *:16443
    option                      tcplog
    default_backend             kubernetes-apiserver

#---------------------------------------------------------------------
# static backend for serving up images, stylesheets and such
#---------------------------------------------------------------------
listen stats
    bind            *:1080
    stats auth      admin:awesomePassword
    stats refresh   5s
    stats realm     HAProxy\ Statistics
    stats uri       /admin?stats

#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend kubernetes-apiserver
    mode        tcp
    balance     roundrobin
    server  master1 192.168.200.3:6443 check
    server  master2 192.168.200.4:6443 check
    server  master3 192.168.200.5:6443 check

29、每台master节点编写健康监测脚本

cat /etc/keepalived/check_haproxy.sh 

#!/bin/sh
# HAPROXY down
A=`ps -C haproxy --no-header | wc -l`
if [ $A -eq 0 ]
then
systmectl start haproxy
if [ ps -C haproxy --no-header | wc -l -eq 0 ]
then
killall -9 haproxy
echo "HAPROXY down" | mail -s "haproxy"
sleep 3600
fi 

fi

30、给脚本增加执行权限

chmod +x check_haproxy.sh

31、启动keepalived和haproxy

systemctl start keepalived && systemctl enable keepalived
systemctl start haproxy && systemctl enable haproxy

32、查看 VIP 地址

33、每个节点安装kubeadm,kubelet和kubectl # 安装的kubeadm、kubectl和kubelet要和kubernetes版本一致，kubelet加入开机启动之后不手动启动，要不然会报错，初始化集群之后集群会自动启动kubelet服务！！！

yum -y install kubeadm-1.18.2 kubelet-1.18.2 kubectl-1.18.2
systemctl enable kubelet && systemctl daemon-reload

34、获取默认配置文件

kubeadm config print init-defaults > kubeadm-config.yaml

35、修改初始化配置文件

cat kubeadm-config.yaml

apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.200.3     # 本机IP
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: master1        # 本主机名
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: "192.168.200.16:16443"    # 虚拟IP和haproxy端口
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io    # 镜像仓库源要根据自己实际情况修改
kind: ClusterConfiguration
kubernetesVersion: v1.18.2     # k8s版本
networking:
  dnsDomain: cluster.local
  podSubnet: "10.244.0.0/16"
  serviceSubnet: 10.96.0.0/12
scheduler: {}
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
featureGates:
  SupportIPVSProxyMode: true
mode: ipvs

36、下载相关镜像

kubeadm config images pull --config kubeadm-config.yaml

37、初始化集群

kubeadm init --config kubeadm-config.yaml

38、加入集群认证

39、在其它两个master节点创建以下目录

mkdir -p /etc/kubernetes/pki/etcd

40、把主master节点证书分别复制到从master节点

scp /etc/kubernetes/pki/ca.* root@192.168.200.4:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/sa.* root@192.168.200.4:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/front-proxy-ca.* root@192.168.200.4:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/etcd/ca.* root@192.168.200.4:/etc/kubernetes/pki/etcd/
scp /etc/kubernetes/admin.conf root@192.168.200.4:/etc/kubernetes/
scp /etc/kubernetes/pki/ca.* root@192.168.200.5:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/sa.* root@192.168.200.5:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/front-proxy-ca.* root@192.168.200.5:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/etcd/ca.* root@192.168.200.5:/etc/kubernetes/pki/etcd/
scp /etc/kubernetes/admin.conf root@192.168.200.5:/etc/kubernetes/

41、把主节点admin.conf证书复制到其他node节点

scp /etc/kubernetes/admin.conf root@192.168.200.6:/etc/kubernetes/

42、其他master节点加入集群执行以下命令

kubeadm join 192.168.200.16:16443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:f0489748e3b77a9a29443dae2c4c0dfe6ff4bde0daf3ca8740dd9ab6a9693a78 \
    --control-plane

43、node节点加入集群执行以下命令

kubeadm join 192.168.200.16:16443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:f0489748e3b77a9a29443dae2c4c0dfe6ff4bde0daf3ca8740dd9ab6a9693a78

44、所有master节点执行以下命令，node节点随意

root用户执行以下命令

echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
source .bash_profile

非root用户执行以下命令

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

45、查看所有节点状态

kubectl get nodes 

46、安装网络插件

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

47、查看节点状态

kubectl get pods --all-namespaces

48、下载etcdctl客户端命令行工具

wget https://github.com/etcd-io/etcd/releases/download/v3.4.14/etcd-v3.4.14-linux-amd64.tar.gz

49、解压并加入环境变量

tar -zxf etcd-v3.4.14-linux-amd64.tar.gz && mv etcd-v3.4.14-linux-amd64/etcdctl /usr/local/bin/ && chmod +x /usr/local/bin/

50、验证etcdctl是否能用，出现以下结果代表已经成功了

etcdctl

51、查看etcd高可用集群健康状态

ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key --write-out=table --endpoints=192.168.200.3:2379,192.168.200.4:2379,192.168.200.5:2379 endpoint health

52、查看etcd高可用集群列表

ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key --write-out=table --endpoints=192.168.200.3:2379,192.168.200.4:2379,192.168.200.5:2379 member list

53、查看etcd高可用集群leader

ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key --write-out=table --endpoints=192.168.200.3:2379,192.168.200.4:2379,192.168.200.5:2379 endpoint status