2. Kubeadm方式搭建k8s集群

Kubeadm方式搭建集群优缺点：优点：简单优雅，支持高可用，升级方便缺点：不易维护，文档不够细致将master1作为deploy节点，未指定节点时默认在master1上进行操作。建议deploy节点与其它节点配置ssh免密登录，配置过程参考：批量实现SSH免密登录。环境准备环境准备工作请在所有节点进行。主机说明：系统ip角色c...

最爱喝酸奶

789人浏览 · 2019-11-14 18:26:36

最爱喝酸奶 · 2019-11-14 18:26:36 发布

Kubeadm方式搭建集群优缺点：

优点：
    简单优雅，支持高可用，升级方便
    
缺点：
    不易维护，文档不够细致

将master1作为deploy节点，未指定节点时默认在master1上进行操作。

建议deploy节点与其它节点配置ssh免密登录，配置过程参考：批量实现SSH免密登录。

环境准备

环境准备工作请在所有节点进行。

主机说明：

系统	ip	角色	cpu	内存	hostname
CentOS 7.7	192.168.1.51	master、deploy	>=2	>=2G	master1
CentOS 7.7	192.168.1.52	master	>=2	>=2G	master2
CentOS 7.7	192.168.1.53	master	>=2	>=2G	master3
CentOS 7.7	192.168.1.54	node	>=2	>=2G	node1
CentOS 7.7	192.168.1.55	node	>=2	>=2G	node2
CentOS 7.7	192.168.1.56	node	>=2	>=2G	node3

设置主机名：

每个节点的主机名必须不一样，且保证节点之间可以通过hostname互相访问。

以master1为例，

# hostnamectl set-hostname master1

# vim /etc/hosts

192.168.1.51 master1
192.168.1.52 master2
192.168.1.53 master3
192.168.1.54 node1
192.168.1.55 node2
192.168.1.56 node3

安装依赖包：

# yum update -y

# yum install -y curl git iptables conntrack ipvsadm ipset jq sysstat libseccomp

关闭防火墙、selinux和swap，重置iptables：

# systemctl stop firewalld && systemctl disable firewalld

# sed -i 's/=enforcing/=disabled/g' /etc/selinux/config && setenforce 0

# iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat && iptables -P FORWARD ACCEPT

# swapoff -a

# sed -i '/swap/s/^\(.*\)$/#\1/g' /etc/fstab

# systemctl stop dnsmasq && systemctl disable dnsmasq               #否则可能导致docker容器无法解析域名

系统参数设置：

# cat > /etc/sysctl.d/kubernetes.conf <<EOF
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
vm.swappiness=0
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
EOF

# sysctl -p /etc/sysctl.d/kubernetes.conf

安装docker：

# curl http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -o /etc/yum.repos.d/docker.repo

# yum makecache fast

# yum install -y docker-ce

# systemctl start docker && systemctl enable docker

# cat <<EOF > /etc/docker/daemon.json
{
	"exec-opts":["native.cgroupdriver=systemd"]
}
EOF

# systemctl restart docker

安装必要工具：

kubeadm     用于部署集群

bukelet     集群中各节点需要运行的组件，负责管理pod、容器的生命周期

kubectl     集群管理工具（master节点安装即可）

# cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
       http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

# yum install -y kubeadm-1.14.0-0 kubelet-1.14.0-0 kubectl-1.14.0-0 --disableexcludes=kubernetes

# systemctl start kubelet && systemctl enable kubelet

配置文件（部署节点）：

# mkdir /software && cd /software

# git clone https://git.imooc.com/LZXLINUX/kubernetes-ha-kubeadm.git

# cd kubernetes-ha-kubeadm/ && ls
addons  configs  global-config.properties  init.sh  LICENSE  README.md  scripts

文件说明：

addons      kubernetes的插件，比如calico和dashboard

configs     包含了部署集群过程中用到的各种配置文件

scripts     包含部署集群过程中用到的脚本，如keepalived检查脚本

global-configs.properties   全局配置，包含各种易变的配置内容

init.sh     初始化脚本，配置好global-config之后，会自动生成所有配置文件

# vim global-config.properties

#kubernetes版本
VERSION=v1.14.0

#POD网段
POD_CIDR=172.10.0.0/16

#master虚拟ip（建议为同网段地址）
MASTER_VIP=192.168.1.188

#keepalived用到的网卡接口名
VIP_IF=ens192

# ./init.sh

====替换变量列表====
VERSION=v1.14.0
POD_CIDR=172.10.0.0/16
MASTER_VIP=192.168.1.188
VIP_IF=ens192

====替换脚本====
scripts/check-apiserver.sh

====替换配置文件====
configs/keepalived-backup.conf
configs/keepalived-master.conf
configs/kubeadm-config.yaml
addons/calico-rbac-kdd.yaml
addons/calico.yaml
addons/dashboard-all.yaml
配置生成成功，位置: /home/kubernetes-ha-kubeadm/target

# find target/ -type f

target/configs/keepalived-backup.conf
target/configs/keepalived-master.conf
target/configs/kubeadm-config.yaml
target/scripts/check-apiserver.sh
target/addons/calico-rbac-kdd.yaml
target/addons/calico.yaml
target/addons/dashboard-all.yaml

搭建高可用集群

3个master节点安装keepalived，保证master节点的api-server进程高可用。

注意：云服务器一般不支持自定义虚拟ip，请跳过安装keepalived。高可用可以使用云商的负载均衡服务（比如阿里云的SLB），把backends设置成你的3个master节点，然后虚拟ip就配置成负载均衡的内网ip即可。

安装keepalived：

# ssh root@master1 "yum install -y keepalived"

# ssh root@master2 "yum install -y keepalived"

# ssh root@master3 "yum install -y keepalived"

拷贝keepalived配置：

# ssh root@master1 "mkdir /etc/keepalived"

# ssh root@master2 "mkdir /etc/keepalived"

# ssh root@master3 "mkdir /etc/keepalived"

# cd /software/kubernetes-ha-kubeadm/

# scp target/configs/keepalived-master.conf master1:/etc/keepalived/keepalived.conf

# scp target/configs/keepalived-backup.conf master2:/etc/keepalived/keepalived.conf

# scp target/configs/keepalived-backup.conf master3:/etc/keepalived/keepalived.conf

# scp target/scripts/check-apiserver.sh master1:/etc/keepalived/

# scp target/scripts/check-apiserver.sh master2:/etc/keepalived/

# scp target/scripts/check-apiserver.sh master3:/etc/keepalived/

修改keepalived配置：

master1

# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived
global_defs {
 router_id keepalive-master
}

vrrp_script check_apiserver {
 script "/etc/keepalived/check-apiserver.sh"
 interval 3
 weight -3
}

vrrp_instance VI-kube-master {
   state MASTER
   interface ens192
   virtual_router_id 68
   priority 100
   dont_track_primary
   advert_int 3
   virtual_ipaddress {
     192.168.1.188
   }
   track_script {
       check_apiserver
   }
}

master2

# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived
global_defs {
 router_id keepalive-backup1
}

vrrp_script check_apiserver {
 script "/etc/keepalived/check-apiserver.sh"
 interval 3
 weight -3
}

vrrp_instance VI-kube-master {
   state BACKUP
   interface ens192
   virtual_router_id 68
   priority 99
   dont_track_primary
   advert_int 3
   virtual_ipaddress {
     192.168.1.188
   }
   track_script {
       check_apiserver
   }
}

master3

# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived
global_defs {
 router_id keepalive-backup2
}

vrrp_script check_apiserver {
 script "/etc/keepalived/check-apiserver.sh"
 interval 3
 weight -3
}

vrrp_instance VI-kube-master {
   state BACKUP
   interface ens192
   virtual_router_id 68
   priority 98
   dont_track_primary
   advert_int 3
   virtual_ipaddress {
     192.168.1.188
   }
   track_script {
       check_apiserver
   }
}

启动keepalived：

# ssh root@master1 "systemctl enable keepalived && systemctl start keepalived"

# ssh root@master1 "systemctl enable keepalived && systemctl start keepalived"

# ssh root@master1 "systemctl enable keepalived && systemctl start keepalived"

# systemctl status keepalived

# ip addr               #查看vip

部署第一个master节点：

# cd /software/kubernetes-ha-kubeadm/

# scp target/configs/kubeadm-config.yaml master1:/software/

# kubeadm init --config=/software/kubeadm-config.yaml --experimental-upload-certs

# mkdir -p ~/.kube

# cp -i /etc/kubernetes/admin.conf ~/.kube/config

# kubectl get pods -n kube-system

NAME                              READY   STATUS    RESTARTS   AGE
coredns-8567978547-g7m8q          0/1     Pending   0          2m28s
coredns-8567978547-j2gnp          0/1     Pending   0          2m28s
etcd-master1                      1/1     Running   0          88s
kube-apiserver-master1            1/1     Running   0          93s
kube-controller-manager-master1   1/1     Running   0          102s
kube-proxy-ldm4l                  1/1     Running   0          2m28s
kube-scheduler-master1            1/1     Running   0          102s

注意备份上面初始化之后打印的join命令，这里分别是以master节点加入集群、以node节点加入集群。

kubeadm join 192.168.1.188:6443 --token mxv077.npi9bcbh6qf0hy44 \
    --discovery-token-ca-cert-hash sha256:9b71c47b979b950169395b2806d532aff60f9e82567d06bab27b1ff3ffb39cff \
    --experimental-control-plane --certificate-key 5560a43ae5f9a59db25cf1be39a6b6084628a8be9329e223caef3695a2249049

kubeadm join 192.168.1.188:6443 --token mxv077.npi9bcbh6qf0hy44 \
    --discovery-token-ca-cert-hash sha256:9b71c47b979b950169395b2806d532aff60f9e82567d06bab27b1ff3ffb39cff

部署calico网络插件：

上传calico配置到配置好kubectl的节点（一个节点即可）。

# mkdir /etc/kubernetes/addons

# cd /software/kubernetes-ha-kubeadm/

# scp target/addons/calico* master1:/etc/kubernetes/addons/

# kubectl apply -f /etc/kubernetes/addons/calico-rbac-kdd.yaml

# kubectl apply -f /etc/kubernetes/addons/calico.yaml

# kubectl get pods -n kube-system

这一步需要pull镜像quay.io/calico/node:v3.1.3和quay.io/calico/cni:v3.1.3，可能等待几分钟，如果报错可以手动pull镜像。

注意：如果发现calico的pod只启动一个且一直重启，日志报503警告，可以先不用管，继续往下做。

加入其它master节点：

master2和master3

# kubeadm join 192.168.1.188:6443 --token mxv077.npi9bcbh6qf0hy44 \
    --discovery-token-ca-cert-hash sha256:9b71c47b979b950169395b2806d532aff60f9e82567d06bab27b1ff3ffb39cff \
    --experimental-control-plane --certificate-key 5560a43ae5f9a59db25cf1be39a6b6084628a8be9329e223caef3695a2249049

# mkdir -p ~/.kube

# cp -i /etc/kubernetes/admin.conf ~/.kube/config

加入其它node节点：

node1、node2和node3

# kubeadm join 192.168.1.188:6443 --token mxv077.npi9bcbh6qf0hy44 \
    --discovery-token-ca-cert-hash sha256:9b71c47b979b950169395b2806d532aff60f9e82567d06bab27b1ff3ffb39cff

查看集群节点：

任选一个master节点

# kubectl get nodes

NAME      STATUS   ROLES    AGE     VERSION
master1   Ready    master   5m18s   v1.14.0
master2   Ready    master   71s     v1.14.0
master3   Ready    master   19s     v1.14.0
node1     Ready    <none>   19s     v1.14.0
node2     Ready    <none>   20s     v1.14.0
node3     Ready    <none>   17s     v1.14.0

可以看到，整个集群有3个master节点和3个node节点，都处于ready状态。

可用性测试

集群已经初步搭建起来，下面进行集群的可用性测试。

一个DaemonSet对象能确保其创建的Pod在集群中的每一台（或指定）Node上都运行一个副本。如果集群中动态加入了新的Node，DaemonSet中的Pod也会被添加在新加入的Node上运行。删除一个DaemonSet也会级联删除所有其创建的Pod。

因此，创建一个DaemonSet对象来测试可用性比较合适。

创建nginx daemonset：

# cd /software && vim nginx-ds.yaml

apiVersion: v1
kind: Service
metadata:
  name: nginx-ds
  labels:
    app: nginx-ds
spec:
  type: NodePort
  selector:
    app: nginx-ds
  ports:
  - name: http
    port: 80
    targetPort: 80
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: nginx-ds
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  template:
    metadata:
      labels:
        app: nginx-ds
    spec:
      containers:
      - name: my-nginx
        image: nginx:1.14.0
        ports:
        - containerPort: 80

# kubectl create -f nginx-ds.yaml
service/nginx-ds created
daemonset.extensions/nginx-ds created

检查ip连通性：

# kubectl get pods -o wide

NAME             READY   STATUS    RESTARTS   AGE   IP           NODE    NOMINATED NODE   READINESS GATES
nginx-ds-qsfp5   1/1     Running   0          68s   172.10.5.2   node3   <none>           <none>
nginx-ds-xw722   1/1     Running   0          68s   172.10.4.2   node1   <none>           <none>
nginx-ds-zzn64   1/1     Running   0          68s   172.10.2.2   node2   <none>           <none>

# kubectl get svc

NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP        24m
nginx-ds     NodePort    10.109.224.125   <none>        80:31388/TCP   3m42s

在每个节点上ping pod ip，同时访问服务ip及其端口，在每个节点检查node-port可用性。

检查dns可用性：

# vim pod-nginx.yaml

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx:1.14.0
    ports:
    - containerPort: 80

# kubectl create -f pod-nginx.yaml
pod/nginx created

# kubectl exec -it nginx bash

root@nginx:/# apt-get update

root@nginx:/# apt install -y iputils-ping

root@nginx:/# ping nginx-ds
PING nginx-ds.default.svc.cluster.local (10.109.224.125) 56(84) bytes of data.

# kubectl get svc
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP        40m
nginx-ds     NodePort    10.109.224.125   <none>        80:31388/TCP   19m

可以看到，在nginx pod中ping nginx-ds时dns解析没问题，返回的是nginx-ds的cluster-ip。这说明之前搭建的集群正常可用。

部署dashboard

部署dashboard：

# cd /software/kubernetes-ha-kubeadm/

# scp target/addons/dashboard-all.yaml master1:/etc/kubernetes/addons/

# kubectl apply -f /etc/kubernetes/addons/dashboard-all.yaml

# kubectl get deploy kubernetes-dashboard -n kube-system

NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
kubernetes-dashboard   1/1     1            1           105s

# kubectl get svc kubernetes-dashboard -n kube-system

NAME                   TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)         AGE
kubernetes-dashboard   NodePort   10.98.87.253   <none>        443:30005/TCP   2m35s

访问dashboard：

dashboard只允许通过https访问，不过使用nodeport的方式暴露服务后，可以使用https://NodeIP:NodePort地址访问。

关于自定义证书，默认dashboard的证书是自动生成的非安全的证书。如果有域名和对应的安全证书可以自行替换，使用安全的域名方式访问dashboard。

在dashboard-all.yaml中增加dashboard启动参数，可以指定证书文件，其中证书文件是通过secret注进来的。

- –tls-cert-file 
- dashboard.cer 
- –tls-key-file 
- dashboard.key

登录dashboard：

dashboard默认只支持token认证，所以如果使用KubeConfig文件，需要在该文件中指定token，这里使用token的方式登录。

# kubectl create sa dashboard-admin -n kube-system

# kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin

# ADMIN_SECRET=$(kubectl get secrets -n kube-system | grep dashboard-admin | awk '{print $1}')

# kubectl describe secret -n kube-system ${ADMIN_SECRET} | grep -E '^token' | awk '{print $2}'              #打印token

使用打印的token登录dashboard，

在这里插入图片描述

使用谷歌浏览器无法访问，不过可以使用火狐浏览器访问，

在这里插入图片描述

如果想要使用谷歌浏览器访问，可以这样做，

# mkdir /software/key && cd /software/key

# openssl genrsa -out dashboard.key 2048

# openssl req -new -out dashboard.csr -key dashboard.key -subj '/CN=192.168.1.51'

# openssl x509 -req -in dashboard.csr -signkey dashboard.key -out dashboard.crt

# kubectl delete secret kubernetes-dashboard-certs -n kube-system

# kubectl create secret generic kubernetes-dashboard-certs --from-file=dashboard.key --from-file=dashboard.crt -n kube-system

# kubectl get pod -n kube-system |grep dashboard
kubernetes-dashboard-5bd4bfc87-t8mmj   1/1     Running   0          75m

# kubectl delete pod kubernetes-dashboard-5bd4bfc87-t8mmj -n kube-system

在这里插入图片描述

kubernetes dashboard部署完成，kubernetes的高可用集群到此也搭建完成。

K8S/Kubernetes

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐

【深度】阿里巴巴万级规模 K8s 集群全局高可用体系之美

作者 | 韩堂、柘远、沉醉来源 | 阿里巴巴云原生公众号前言台湾作家林清玄在接受记者采访的时候，如此评价自己 30 多年写作生涯：“第一个十年我才华横溢，‘贼光闪现’，令周边黯然失色；第二个十年，我终于‘宝光现形’，不再去抢风头，反而与身边的美丽相得益彰；进入第三个十年，繁华落尽见真醇，我进入了‘醇光初现’的阶段，真正体味到了境界之美”。长夜有穷，真水无香。领略过了 K8s“身在江

K8S/Kubernetes

如何基于 K8s 构建下一代 DevOps 平台？

作者 | 孙健波（天元）导读：当前云原生 DevOps 体系现状如何？面临哪些挑战？如何通过 OAM 解决云原生 DevOps 场景下的诸多问题？云原生开发应用模型 OAM(Open Application Model) 社区核心成员孙健波将为大家一一解答，并分享如何基于 OAM 和 Kubernetes 打造无限能力的下一代 DevOps 平台。什么是 DevOps？为什么基于 Kub