目录

写在之前

一. k8s简介

1. k8s 的整体架构

2. 部署说明

3. ansible host文件说明

二. 基础系统部署

1. 设置关闭防火墙及SELINUX

2. 关闭Swap

3.升级linux内核

4.优化内核参数

5. 开启 ipvs

6. 设置系统时区

7.创建相关目录

8.设置环境变量脚本

三. 创建证书

1. 安装cfssl工具集

2.创建证书 

3. 分发证书

4. 部署kubectl命令行工具

5. 生成admin 证书

6.生成kubeconfig文件

四.ETCD部署

1. 分发文件

2. 创建etcd证书和私钥

3. 创建启动文件

五. 部署Flannel网络

1. 分发文件

2. 创建秘钥

3. 将网段信息写入etcd

4. 创建启动文件，并启动服务

六. 部署服务高可用

1. kube-nginx 部署

2. keepalived 部署

七. kube-apiserver 部署

1. 解压文件并分发 

2. 创建Kubernetes 证书和私钥

3. 创建加密配置文件

4. 创建审计策略文件

5. 创建证书签名请求

6. 创建kube-apiserver启动文件

 八. 部署kube-controller-manager

1. 创建证书签名

2. 创建和分发kubeconfig文件

3. 创建kube-controller-manager启动文件

4.启动服务

5.kube-controller-manager 创建权限

九. 部署kube-scheduler

2. 创建和分发 kubeconfig 文件

3.创建 kube-scheduler 配置文件

4. 创建启动文件

5. 启动服务并检查

十. docker的安装

1. 安装docker服务

2. 创建配置文件

3. 启动服务并检查

十一. 部署kubelet组件

1. 创建kubelet bootstrap kubeconfig文件

2. 查看kubeadm为各个节点创建的token

4. 创建和分发kubelet启动文件

5. approve CSR请求

6. bear token认证和授权

十二. 部署kube-proxy组件

1. 创建kube-proxy证书签名请求

2. 创建和分发 kubeconfig 文件

3. 创建kube-proxy配置文件

4. 启动 kube-proxy 服务

十三. 测试集群功能

1. 创建yml文件

2. 查看pod

3. 测试连通性

4. 查看服务情况

十四. CoreDNS安装

1. 创建yaml文件

2. 创建coredns

3. 测试dns功能 

十五. 后记

写在之前

只2017年开始，断断续续的接触k8s也是有几个年头了，从1.11开始到现在马上都1.20了，真快呀！之前就有想整理下所有技术的想法，毕竟已经35岁的人，闯荡江湖已经12个年头了。是该总结下了，这行说实话干不了几年了。话不多说，等专门写一篇感慨的文章。开整！

一. k8s简介

这块我不会讲的太多，毕竟k8s一个很大的体系，不是一两句话或者两三句话就可以说清楚的。可能说的太深了我也未必能说的清楚，主要是说不清楚吧。但是简单的我还是要说下，主要是应对面试吧。但是说实话理论这事儿干的深了很重要的，在排错的时候，如果有理论加持会事半功倍，很容易地位错误。如果对于理论不大理解，那就是撞大运。想想是不是应该多写点理论？ 还是算了吧，等等看，时间充足了，再说吧。

1. k8s 的整体架构

这块不太多说了，先上图。不是我自己画的，copy哪个大仙的，有侵权的喊我。我换其他大仙的。

总体来说k8s可以分为两个部分，master和node两个部分。

  • master 由四个模块组成，APIServer，schedule,controller-manager,etcd.

APIServer: APIServer负责对外提供RESTful的kubernetes API的服务，它是系统管理指令的统一接口，任何对资源的增删该查都要交给APIServer处理后再交给etcd，kubectl(kubernetes提供的客户端工具，该工具内部是对kubernetes API的调用）是直接和APIServer交互的。

schedule: schedule负责调度Pod到合适的Node上，如果把scheduler看成一个黑匣子，那么它的输入是pod和由多个Node组成的列表，输出是Pod和一个Node的绑定。 kubernetes目前提供了调度算法，同样也保留了接口。用户根据自己的需求定义自己的调度算法。

controller manager: 如果APIServer做的是前台的工作的话，那么controller manager就是负责后台的。每一个资源都对应一个控制器。而controller manager就是负责管理这些控制器的，比如我们通过APIServer创建了一个Pod，当这个Pod创建成功后，APIServer的任务就算完成了。

etcd：etcd是一个高可用的键值存储系统，kubernetes使用它来存储各个资源的状态，从而实现了Restful的API。

  • Node节点主要由三个模板组成：kublet, kube-proxy,Docker

kube-proxy: 该模块实现了kubernetes中的服务发现和反向代理功能。kube-proxy支持TCP和UDP连接转发，默认基Round Robin算法将客户端流量转发到与service对应的一组后端pod。服务发现方面，kube-proxy使用etcd的watch机制监控集群中service和endpoint对象数据的动态变化，并且维护一个service到endpoint的映射关系，从而保证了后端pod的IP变化不会对访问者造成影响，另外，kube-proxy还支持session affinity。

kublet：kublet是Master在每个Node节点上面的agent，是Node节点上面最重要的模块，它负责维护和管理该Node上的所有容器，但是如果容器不是通过kubernetes创建的，它并不会管理。本质上，它负责使Pod的运行状态与期望的状态一致。

Docker:进行容器生成、配置和使用，作为pod节点的重要支撑。

这玩意儿就说到这里，再详细的不多学了，我不做大自然的搬运工了。面试的话基本上就问这些基本的，往深了就是pod，pv，pvc这类的。等专门写写这块内容。其他也就没嘛了，面试官也不太敢往深了搞，关键是他也的会哈，百分之八十的都是这样，当然牛人也是很多的。

2. 部署说明

首先需要说明的是，为了快捷很多东西直接用ansible做的，简单的ansible的命令，没有涉及到playbook的东西，以后可能会做一套这东西。但是现在觉得没拿必要。如果哪位同志想搞这块的内容，可以单独的找我，咱们一起探讨。还有，我内网做了本地dns，直接采用相关域名操作，没有进行主机的后饰条文件设置。最后，我自己做了本地的yum源，很多包可能在网络部署的时候会没有，这个别担心自己找找就可以，没什么大不了的事儿。不能指着一篇文档发家致富，工作很多时候还的靠自己。我这边做的是我们的测试环境，线上环境大同小异，也是这个样做的，我们的node节点大概会有十几台，我这里会写一台吧，如果说不清可能最多写两台node节点。另外我的ansible是单独的一台机器。各位战友可根据自己的环境进行调整，不多说了，具体规划如下：

角色

ip

主机名称

组件

master-01

10.120.200.2

k8s-master01.k8s.com

kube-apiserver，kube-controller-manager，kube-scheduler，etcd

master-02

10.120.200.3

k8s-master02.k8s.com

kube-apiserver，kube-controller-manager，kube-scheduler，etcd

node-01

10.120.200.10

k8s-node01.k8s.com

kubelet，kube-proxy，docker， etcd

node-02

10.120.200.4

k8s-node02.k8s.com

kubelet，kube-proxy，docker ，ansible

3. ansible host文件说明

[master]

k8s-master01.k8s.com
k8s-master02.k8s.com

[node]
k8s-node01.k8s.com
k8s-node02.k8s.com

[etcd]
k8s-master01.k8s.com
k8s-master02.k8s.com
k8s-node07.k8s.com

二. 基础系统部署

1. 设置关闭防火墙及SELINUX

ansible all -m shell -a "systemctl stop firewalld && systemctl disable firewalld"
ansible all -m shell -a "setenforce 0"
ansible all -m shell -a "yum install yum-utils -y" 
ansible all -m shell -a "sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config"

2. 关闭Swap

ansible all -m shell -a "swapoff -a && sysctl -w vm.swappiness=0" 
ansible all -m shell -a "sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab"

3.升级linux内核

ansible all -m shell -a "mkdir /data/"
ansible all -m copy -a "src=./kernel-ml-5.11.8-1.el7.elrepo.x86_64.rpm dest=/data/ "
ansible all -m shell -a "chdir=/data rpm -ivh kernel-ml-5.11.8-1.el7.elrepo.x86_64.rpm"
ansible all -m shell -a "grub2-set-default  0 && grub2-mkconfig -o /etc/grub2.cfg"
#查看启动默认内核指向上面安装的内核
ansible all -m shell -a "grubby --default-kernel"
#顺手更新下系统，由于使用yum update命令更新，时间或者显示的问题可能有用ansible会有错误。可以到每台机器上去执行命令
ansible all -m shell -a "yum update -y"
ansible all -m shell -a "reboot"

4.优化内核参数

cat > kubernetes.conf <<EOF
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
#net.ipv4.tcp_tw_recycle=0
vm.swappiness=0 # 禁止使用 swap 空间，只有当系统 OOM 时才允许使用它
vm.overcommit_memory=1 # 不检查物理内存是否够用
vm.panic_on_oom=0 # 开启 OOM
fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=1048576
fs.file-max=52706963
fs.nr_open=52706963
net.ipv6.conf.all.disable_ipv6=1
net.netfilter.nf_conntrack_max=2310720
EOF

ansible all -m copy -a "src=./kubernetes.conf dest=/etc/sysctl.d/"
ansible all -m shell -a "sysctl -p /etc/sysctl.d/kubernetes.conf"

5. 开启 ipvs

cat > ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
EOF

ansible all -m copy -a "src=./ipvs.modules dest=/etc/sysconfig/modules/"
ansible all -m shell -a "chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4"

6. 设置系统时区

ansible all -m shell -a "timedatectl set-timezone Asia/Shanghai"
 #将当前的 UTC 时间写入硬件时钟
ansible all -m shell -a "timedatectl set-local-rtc 0"
 #重启依赖于系统时间的服务
ansible all -m shell -a "systemctl restart rsyslog "
ansible all -m shell -a "systemctl restart crond"

7.创建相关目录

ansible all -m shell -a "mkdir -p  /opt/k8s/{bin,work} /etc/{kubernetes,etcd}/cert"

8.设置环境变量脚本

所有环境变量都定义在environment.sh中，需要根据环境修改。并且需要拷贝到所有节点的/opt/k8s/bin目录下

#!/usr/bin/bash
# 生成 EncryptionConfig 所需的加密 key
export ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64)
# 集群各机器 IP 数组
export NODE_IPS=( 10.120.200.2 10.120.200.3  10.120.200.10 )
# 集群各 IP 对应的主机名数组
export NODE_NAMES=(k8s-master02 k8s-master01 k8s-node07)
# 集群MASTER机器 IP 数组
export MASTER_IPS=(10.120.200.2 10.120.200.3  )
# 集群所有的master Ip对应的主机
export MASTER_NAMES=(k8s-master02 k8s-master01)
# etcd 对应主机列表
export ETCD_NAMES=(k8s-master02 k8s-master01 k8s-node07)
# etcd 集群服务地址列表
export ETCD_ENDPOINTS="https://10.120.200.2:2379,https://10.120.200.3:2379,https://10.120.200.10:2379"
# etcd 集群间通信的 IP 和端口
export ETCD_NODES="k8s-01=https://10.120.200.2:2380,k8s-02=https://10.120.200.3:2380,k8s-03=https://10.120.200.10:2380"
# etcd 集群所有node ip
export ETCD_IPS=(192.110.120.200.2 10.120.200.3  10.120.200.10 )
# kube-apiserver 的反向代理(kube-nginx)地址端口
export KUBE_APISERVER="https://10.120.200.100:8443"
# 节点间互联网络接口名称
export IFACE="eth0"
# etcd 数据目录
export ETCD_DATA_DIR="/data/k8s/etcd/data"
# etcd WAL 目录，建议是 SSD 磁盘分区，或者和 ETCD_DATA_DIR 不同的磁盘分区
export ETCD_WAL_DIR="/data/k8s/etcd/wal"
# k8s 各组件数据目录
export K8S_DIR="/data/k8s/k8s"
# docker 数据目录
#export DOCKER_DIR="/data/k8s/docker"
## 以下参数一般不需要修改
# TLS Bootstrapping 使用的 Token，可以使用命令 head -c 16 /dev/urandom | od -An -t x | tr -d ' ' 生成
#BOOTSTRAP_TOKEN="41f7e4ba8b7be874fcff18bf5cf41a7c"
# 最好使用 当前未用的网段 来定义服务网段和 Pod 网段
# 服务网段，部署前路由不可达，部署后集群内路由可达(kube-proxy 保证)
SERVICE_CIDR="10.254.0.0/16"
# Pod 网段，建议 /16 段地址，部署前路由不可达，部署后集群内路由可达(flanneld 保证)
CLUSTER_CIDR="172.30.0.0/16"
# 服务端口范围 (NodePort Range)
export NODE_PORT_RANGE="1024-32767"
# flanneld 网络配置前缀
export FLANNEL_ETCD_PREFIX="/kubernetes/network"
# kubernetes 服务 IP (一般是 SERVICE_CIDR 中第一个IP)
export CLUSTER_KUBERNETES_SVC_IP="10.254.0.1"
# 集群 DNS 服务 IP (从 SERVICE_CIDR 中预分配)
export CLUSTER_DNS_SVC_IP="10.254.0.2"
# 集群 DNS 域名（末尾不带点号）
export CLUSTER_DNS_DOMAIN="cluster.local"
# 将二进制目录 /opt/k8s/bin 加到 PATH 中
export PATH=/opt/k8s/bin:$PATH
ansible all -m copy -a "src=./environment.sh dest=/opt/k8s/bin"
ansible all -m shell -a "chdir=/opt/k8s/bin chmod 755 ./environment.sh"

基本上系统前期基础部署到这里就完成了。这些内容每天master和node均需要操作。在ansible中采用了all 这个参数选项。接下来开始做master组件部署。

三. 创建证书

1. 安装cfssl工具集

ansible k8s-master01.k8s.com -m shell -a "mkdir -p /opt/k8s/cert"
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
mv cfssl_linux-amd64 cfssl
ansible k8s-master01.k8s.com -m copy  -a "src=./cfssl dest=/opt/k8s/bin/" 
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
mv cfssljson_linux-amd64 cfssljson
ansible k8s-master01.k8s.com -m copy  -a "src=./cfssljson dest=/opt/k8s/bin/" 
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
mv cfssl-certinfo_linux-amd64 cfssl-certinfo
ansible k8s-master01.k8s.com -m copy  -a "src=./cfssl-certinfo dest=/opt/k8s/bin/" 
ansible k8s-master01.k8s.com -m shell  -a "chmod +x /opt/k8s/bin/*"
ansible k8s-master01.k8s.com -m shell  -a "export PATH=/opt/k8s/bin:$PATH"

这部分可以在master01机器上操作，但是我这环境master 没有外网，只能在ansible机器下载然后推过去了。

2.创建证书 

cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "87600h"
      }
    }
  }
}
EOF
cat > ca-csr.json <<EOF
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "TJ",
      "L": "TJ",
      "O": "k8s",
      "OU": "4Paradigm"
    }
  ],
  "ca": {
    "expiry": "876000h"
 }
}
EOF
ansible k8s-master01.k8s.com -m copy  -a "src=./ca-config.json dest=/opt/k8s/work/" 
ansible k8s-master01.k8s.com -m copy  -a "src=./ca-csr.json dest=/opt/k8s/work/"

将生成的json文件推送到master01中。


ansible k8s-master01.k8s.com -m shell  -a "chdir=/opt/k8s/work  /opt/k8s/bin/cfssl gencert -initca ca-csr.json | /opt/k8s/bin/cfssljson -bare ca"
#查看生成的文件
ansible k8s-master01.k8s.com -m shell  -a "chdir=/opt/k8s/work  ls ca*"

3. 分发证书

ansible k8s-master01.k8s.com -m shell  -a "source /opt/k8s/bin/environment.sh"
source /opt/k8s/bin/environment.sh
ansible all -m shell  -a "mkdir -p /etc/kubernetes/cert"
for node_ip in ${NODE_IPS[@]};   do     echo ">>> ${node_ip}"; ansible k8s-master01.k8s.com -m shell  -a "chdir=/opt/k8s/work scp ca*.pem ca-config.json root@${node_ip}:/etc/kubernetes/cert"; done

4. 部署kubectl命令行工具

ansible all -m copy -a "src=./kubectl dest=/opt/k8s/bin"
ansible all -m shell  -a "chdir=/opt/k8s/bin chmod +x ./* "

5. 生成admin 证书

生成admin-csr.json的文件

{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "TJ",
      "L": "TJ",
      "O": "system:masters",
      "OU": "4Paradigm"
    }
  ]
}

ansible k8s-master01.k8s.com -m copy  -a "src=./admin-csr.json dest=/opt/k8s/work/"
ansible k8s-master01.k8s.com -m shell  -a "chdir=/opt/k8s/work  /opt/k8s/bin/cfssl gencert -ca=/opt/k8s/work/ca.pem -ca-key=/opt/k8s/work/ca-key.pem -config=/opt/k8s/work/ca-config.json -profile=kubernetes admin-csr.json | /opt/k8s/bin/cfssljson -bare admin"
#查看文件
ansible k8s-master01.k8s.com -m shell  -a "chdir=/opt/k8s/work  ls admin*" 

admin.csr
admin-csr.json
admin-key.pem
admin.pem

6.生成kubeconfig文件

ansible k8s-master01.k8s.com -m shell  -a "chdir=/opt/k8s/work  source /opt/k8s/bin/environment.sh"   
## 设置集群参数
ansible k8s-master01.k8s.com -m shell  -a "chdir=/opt/k8s/work  /opt/k8s/bin/kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/k8s/work/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=kubectl.kubeconfig"
#设置客户端认证参数
ansible k8s-master01.k8s.com -m shell  -a "chdir=/opt/k8s/work  /opt/k8s/bin/kubectl config set-credentials admin \                                 
  --client-certificate=/opt/k8s/work/admin.pem \
  --client-key=/opt/k8s/work/admin-key.pem \                      
  --embed-certs=true \                    
  --kubeconfig=kubectl.kubeconfig"
# 设置上下文参数

ansible k8s-master01.k8s.com -m shell  -a "chdir=/opt/k8s/work  /opt/k8s/bin/kubectl config set-context kubernetes \                                 
  --cluster=kubernetes \                        
  --user=admin \                                                  
  --kubeconfig=kubectl.kubeconfig" 
# 设置默认上下文
ansible k8s-master01.k8s.com -m shell  -a "chdir=/opt/k8s/work  /opt/k8s/bin/kubectl config use-context kubernetes --kubeconfig=kubectl.kubeconfig" 

#文件分发

ansible all  -m shell  -a "mkdir -p ~/.kube  "

for node_ip in ${NODE_IPS[@]};   do     echo ">>> ${node_ip}"; ansible k8s-master01.k8s.com -m shell  -a "chdir=/opt/k8s/work scp kubectl.kubeconfig root@${node_ip}:~/.kube/config"; done

四.ETCD部署

1. 分发文件

tar -zxf etcd-v3.3.25-linux-amd64.tar.gz 
cd etcd-v3.3.25-linux-amd64
ansible etcd  -m copy -a "src=./etcd  dest=/opt/k8s/bin"
ansible etcd  -m copy -a "src=./etcdctl  dest=/opt/k8s/bin"
ansible etcd  -m shell -a "chdir=/opt/k8s/bin chmod +x *"

2. 创建etcd证书和私钥

生成etcd-csr.json 文件

{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "10.120.200.2",
    "10.120.200.3",
    "10.120.200.10"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "TJ",
      "L": "TJ",
      "O": "k8s",
      "OU": "4Paradigm"
    }
  ]
}
ansible k8s-master01.k8s.com -m copy -a "src=./etcd-csr.json dest=/opt/k8s/work"
ansible k8s-master01.k8s.com -m shell -a "chdir=/opt/k8s/work /opt/k8s/bin/cfssl gencert -ca=/opt/k8s/work/ca.pem \
    -ca-key=/opt/k8s/work/ca-key.pem \
    -config=/opt/k8s/work/ca-config.json \
    -profile=kubernetes etcd-csr.json | /opt/k8s/bin/cfssljson -bare etcd"
ansible all -m shell -a "source /opt/k8s/bin/environment.sh"
for node_ip in ${ETCD_IPS[@]};   do     echo ">>> ${node_ip}"; ansible k8s-master01.k8s.com -m shell  -a "chdir=/opt/k8s/work scp ./etcd.pem root@${node_ip}:/etc/etcd/cert"; done
for node_ip in ${ETCD_IPS[@]};   do     echo ">>> ${node_ip}"; ansible k8s-master01.k8s.com -m shell  -a "chdir=/opt/k8s/work scp ./etcd-key.pem root@${node_ip}:/etc/etcd/cert"; done

3. 创建启动文件

生成etcd启动模板文件etcd.service.template

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
[Service]
Type=notify
WorkingDirectory=/data/k8s/etcd/data
ExecStart=/opt/k8s/bin/etcd \
  --data-dir=/data/k8s/etcd/data \
  --wal-dir=/data/k8s/etcd/wal \
  --name=##NODE_NAME## \
  --cert-file=/etc/etcd/cert/etcd.pem \
  --key-file=/etc/etcd/cert/etcd-key.pem \
  --trusted-ca-file=/etc/kubernetes/cert/ca.pem \
  --peer-cert-file=/etc/etcd/cert/etcd.pem \
  --peer-key-file=/etc/etcd/cert/etcd-key.pem \
  --peer-trusted-ca-file=/etc/kubernetes/cert/ca.pem \
  --peer-client-cert-auth \
  --client-cert-auth \
  --listen-peer-urls=https://##NODE_IP##:2380 \
  --initial-advertise-peer-urls=https://##NODE_IP##:2380 \
  --listen-client-urls=https://##NODE_IP##:2379 \
  --advertise-client-urls=https://##NODE_IP##:2379 \
  --initial-cluster-token=etcd-cluster-0 \
  --initial-cluster=k8s-master01=https://10.120.200.2:2380,k8s-master02=https://10.120.200.3:2380,k8s-node07=https://10.120.200.10:2380 \
  --initial-cluster-state=new \
  --auto-compaction-mode=periodic \
  --auto-compaction-retention=1 \
  --max-request-bytes=33554432 \
  --quota-backend-bytes=6442450944 \
  --heartbeat-interval=250 \
  --election-timeout=2000
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target

生成节点启动文件 

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for (( i=0; i < 3; i++ ))
  do
    sed -e "s/##NODE_NAME##/${MASTER_NAMES[i]}/" -e "s/##NODE_IP##/${ETCD_IPS[i]}/" etcd.service.template > etcd-${ETCD_IPS[i]}.service 
  done
ls *.service

ansible k8s-master01.k8s.com  -m copy -a "src=./etcd-10.120.200.2.service dest=/etc/systemd/system/etcd.service"
ansible k8s-master02.k8s.com  -m copy -a "src=./etcd-10.120.200.3.service dest=/etc/systemd/system/etcd.service"
ansible k8s-node01.k8s.com  -m copy -a "src=./etcd-10.120.200.10.service dest=/etc/systemd/system/etcd.service"
ansible etcd -m shell -a "mkdir /data/k8s/etcd/data/ -p"
ansible etcd -m shell -a "chmod 700 /data/k8s/etcd/data/ -R"[root@k8s-node-01 data]# 
ansible etcd -m shell -a "mkdir -p ${ETCD_DATA_DIR} ${ETCD_WAL_DIR}"
ansible etcd -m shell -a "systemctl daemon-reload && systemctl enable etcd && systemctl restart etcd"

检查结果

ansible etcd -m shell -a "systemctl status etcd|grep Active"
结果
k8s-node07.k8s.com | CHANGED | rc=0 >>
   Active: active (running) since 三 2020-12-23 19:55:56 CST; 2h 5min ago
k8s-master01.k8s.com | CHANGED | rc=0 >>
   Active: active (running) since 三 2020-12-23 19:55:06 CST; 2h 5min ago
k8s-master02.k8s.com | CHANGED | rc=0 >>
   Active: active (running) since 三 2020-12-23 19:55:06 CST; 2h 5min ago

任意一个etcd节点执行

for node_ip in ${ETCD_IPS[@]};   do     echo ">>> ${node_ip}";     ETCDCTL_API=3 /opt/k8s/bin/etcdctl     --endpoints=https://${node_ip}:2379     --cacert=/etc/kubernetes/cert/ca.pem     --cert=/etc/etcd/cert/etcd.pem     --key=/etc/etcd/cert/etcd-key.pem endpoint health;   done
结果：      
>>> 10.120.200.2
https://10.120.200.2:2379 is healthy: successfully committed proposal: took = 25.451793ms
>>> 10.120.200.3
https://10.120.200.3:2379 is healthy: successfully committed proposal: took = 24.119252ms
>>> 10.120.200.10
https://10.120.200.10:2379 is healthy: successfully committed proposal: took = 28.700118ms

查看etcd的leader

ETCDCTL_API=3 /opt/k8s/bin/etcdctl \
   -w table --cacert=/etc/kubernetes/cert/ca.pem \
   --cert=/etc/etcd/cert/etcd.pem \
   --key=/etc/etcd/cert/etcd-key.pem \
   --endpoints=${ETCD_ENDPOINTS} endpoint status
结果:
+----------------------------+------------------+---------+---------+-----------+-----------+------------+
|          ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+----------------------------+------------------+---------+---------+-----------+-----------+------------+
|  https://10.120.200.2:2379 | ae550aa9096a80a7 |  3.3.25 |   20 kB |      true |        49 |         11 |
|  https://10.120.200.3:2379 | 3cab77a8b9fc02e5 |  3.3.25 |   20 kB |     false |        49 |         11 |
| https://10.120.200.10:2379 | ab964a290e2d8de9 |  3.3.25 |   20 kB |     false |        49 |         11 |
+----------------------------+------------------+---------+---------+-----------+-----------+------------+

五. 部署Flannel网络

k8s集群需要master和node各个节点pod网络互通。我们可以通过flannel的xvlan技术，在各个创建虚拟网卡，组成pod网络。pod网络使用的端口为8472，第一次启动时，从etcd获取配置的Pod网络，为本节点分配一个未使用的地址段，然后创建flannel.1网络接口。

1. 分发文件

tar -zxf flannel-v0.12.0-linux-amd64.tar.gz
ansible all -m copy -a "src=./flanneld dest=/opt/k8s/bin/ "
ansible all -m copy -a "src=./mk-docker-opts.sh dest=/opt/k8s/bin/ "
ansible all -m shell -a " chmod +x /opt/k8s/bin/*  "

2. 创建秘钥


cat > flanneld-csr.json <<EOF
{
  "CN": "flanneld",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "TJ",
      "L": "TJ",
      "O": "k8s",
      "OU": "4Paradigm"
    }
  ]
}
EOF
#分发到etcd节点
ansible k8s-master01.k8s.com  -m copy -a 'src=./flanneld-csr.json  dest=/opt/k8s/work'
ansible k8s-master01.k8s.com -m shell  -a "chdir=/opt/k8s/work /opt/k8s/bin/cfssl gencert -ca=/opt/k8s/work/ca.pem  -ca-key=/opt/k8s/work/ca-key.pem -config=/opt/k8s/work/ca-config.json -profile=kubernetes flanneld-csr.json | /opt/k8s/bin/cfssljson -bare flanneld"
#将秘钥分发到各个服务器，包括master和nod
ansible all -m shell -a "mkdir -p /etc/flannel/cert"
for node_ip in ${NODE_IPS[@]};   do     echo ">>> ${node_ip}"; ansible k8s-master01.k8s.com -m shell  -a "chdir=/opt/k8s/work scp ./flanneld.pem root@${node_ip}:/etc/flannel/cert"; done
for node_ip in ${MASTER_IPS[@]};   do     echo ">>> ${node_ip}"; ansible k8s-master01.k8s.com -m shell  -a "chdir=/opt/k8s/work scp ./flanneld.pem root@${node_ip}:/etc/flannel/cert"; done
for node_ip in ${NODE_IPS[@]};   do     echo ">>> ${node_ip}"; ansible k8s-master01.k8s.com -m shell  -a "chdir=/opt/k8s/work scp ./flanneld-key.pem root@${node_ip}:/etc/flannel/cert"; done
for node_ip in ${MASTER_IPS[@]};   do     echo ">>> ${node_ip}"; ansible k8s-master01.k8s.com -m shell  -a "chdir=/opt/k8s/work scp ./flanneld-key.pem root@${node_ip}:/etc/flannel/cert"; done

3. 将网段信息写入etcd

#由于涉及到符号问题，在etcd节点任意一台执行
source /opt/k8s/bin/environment.sh
etcdctl   --endpoints=${ETCD_ENDPOINTS}   --ca-file=/etc/kubernetes/cert/ca.pem   --cert-file=/etc/flannel/cert/flanneld.pem   --key-file=/etc/flannel/cert/flanneld-key.pem   mk ${FLANNEL_ETCD_PREFIX}/config '{"Network":"'${CLUSTER_CIDR}'", "SubnetLen": 21, "Backend": {"Type": "vxlan"}}'

4. 创建启动文件，并启动服务

创建启动文件

cat > flanneld.service << EOF
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service
[Service]
Type=notify
ExecStart=/opt/k8s/bin/flanneld \\
  -etcd-cafile=/etc/kubernetes/cert/ca.pem \\
  -etcd-certfile=/etc/flannel/cert/flanneld.pem \\
  -etcd-keyfile=/etc/flannel/cert/flanneld-key.pem \\
  -etcd-endpoints=${ETCD_ENDPOINTS} \\
  -etcd-prefix=${FLANNEL_ETCD_PREFIX} \\
  -iface=${IFACE} \\
  -ip-masq
ExecStartPost=/opt/k8s/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=always
RestartSec=5
StartLimitInterval=0
[Install]
WantedBy=multi-user.target
RequiredBy=docker.service
EOF

#分发到节点

ansible all -m copy -a 'src=./flanneld.service dest=/etc/systemd/system/'

启动服务

ansible all -m shell -a 'systemctl daemon-reload && systemctl enable flanneld && systemctl restart flanneld'

检查服务进程

ansible all  -m shell  -a "systemctl status flanneld|grep Active"

检查分配给flanneld的Pod网段信息

etcdctl \
   --endpoints=${ETCD_ENDPOINTS} \
   --ca-file=/etc/kubernetes/cert/ca.pem \
   --cert-file=/etc/flannel/cert/flanneld.pem \
   --key-file=/etc/flannel/cert/flanneld-key.pem \
   get ${FLANNEL_ETCD_PREFIX}/config

结果：
{"Network":"172.30.0.0/16", "SubnetLen": 21, "Backend": {"Type": "vxlan"}}

查看已分配的Pod子网网段列表

etcdctl \
   --endpoints=${ETCD_ENDPOINTS} \
   --ca-file=/etc/kubernetes/cert/ca.pem \
   --cert-file=/etc/flannel/cert/flanneld.pem \
   --key-file=/etc/flannel/cert/flanneld-key.pem \
   ls ${FLANNEL_ETCD_PREFIX}/subnets

结果：
/kubernetes/network/subnets/172.30.80.0-21
/kubernetes/network/subnets/172.30.96.0-21
/kubernetes/network/subnets/172.30.184.0-21
/kubernetes/network/subnets/172.30.32.0-21

查看某Pod网段对应节点IP和flannel接口地址

etcdctl   --endpoints=${ETCD_ENDPOINTS}   --ca-file=/etc/kubernetes/cert/ca.pem   --cert-file=/etc/flannel/cert/flanneld.pem   --key-file=/etc/flannel/cert/flanneld-key.pem   get ${FLANNEL_ETCD_PREFIX}/subnets/172.30.184.0-21

结果：
{"PublicIP":"10.120.200.4","BackendType":"vxlan","BackendData":{"VtepMAC":"36:da:bd:63:39:f9"}}

六. 部署服务高可用

1. kube-nginx 部署

##由于安装nginx，方便起见直接在master01上操作。

#编译
cd /opt/k8s/work/nginx-1.15.3
mkdir nginx-prefix
./configure --with-stream --without-http --prefix=$(pwd)/nginx-prefix --without-http_uwsgi_module 
make && make install

加载动态库

ldd ./nginx-prefix/sbin/nginx

分发文件

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}
  done
#可能拷贝过程中报错，需要重复执行
for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp /opt/k8s/work/nginx-1.15.3/nginx-prefix/sbin/nginx  root@${node_ip}:/opt/k8s/kube-nginx/sbin/kube-nginx
    ssh root@${node_ip} "chmod a+x /opt/k8s/kube-nginx/sbin/*"
    ssh root@${node_ip} "mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}"
    sleep 3
  done

创建配置文件并分发

cat > kube-nginx.conf <<EOF
worker_processes 1;
events {
    worker_connections  1024;
}
stream {
    upstream backend {
        hash $remote_addr consistent;
        server 192.168.0.50:6443        max_fails=3 fail_timeout=30s;
        server 192.168.0.51:6443        max_fails=3 fail_timeout=30s;
        server 192.168.0.52:6443        max_fails=3 fail_timeout=30s;
    }
    server {
        listen *:8443;
        proxy_connect_timeout 1s;
        proxy_pass backend;
    }
}
EOF
 
for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp kube-nginx.conf  root@${node_ip}:/opt/k8s/kube-nginx/conf/kube-nginx.conf
  done

创建启动文件并启动

cat > kube-nginx.service <<EOF
[Unit]
Description=kube-apiserver nginx proxy
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=forking
ExecStartPre=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -t
ExecStart=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx
ExecReload=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -s reload
PrivateTmp=true
Restart=always
RestartSec=5
StartLimitInterval=0
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
#分发
for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp kube-nginx.service  root@${node_ip}:/etc/systemd/system/
  done

#启动
for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kube-nginx && systemctl start kube-nginx"
  done
检查状态
for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl status kube-nginx |grep 'Active:'"
  done

2. keepalived 部署

keekalived只再master端部署即可。首先安装服务。

ansible master -m shell -a 'yum  install -y keepalived'

创建配置文件，并分发

cat ./keepalived.conf 
global_defs {
   router_id 10.120.200.2
}
vrrp_script chk_nginx {
    script "/etc/keepalived/check_port.sh 8443"
    interval 2
    weight -20
}
vrrp_instance VI_1 {
    state MASTER
    interface ens32
    virtual_router_id 251
    priority 100
    advert_int 1
    mcast_src_ip 10.120.200.2
    nopreempt
    authentication {
        auth_type PASS
        auth_pass 11111111
    }
    track_script {
         chk_nginx
    }
    virtual_ipaddress {
        10.120.200.100
    }

ansible master -m copy -a "src=./keepalived.conf dest=/etc/keepalived/"  
#因为按照master01 做的需要在master02 上修改ip
ansible k8s-master02.k8s.com -m shell -a "sed -i 's#10.120.200.2#10.120.200.3#g'  /etc/keepalived/keepalived.conf"

创建健康检查脚本

cat check_port.sh   
CHK_PORT=$1
 if [ -n "$CHK_PORT" ];then
        PORT_PROCESS=`ss -lt|grep $CHK_PORT|wc -l`
        if [ $PORT_PROCESS -eq 0 ];then
                echo "Port $CHK_PORT Is Not Used,End."
                exit 1
        fi
 else
        echo "Check Port Cant Be Empty!

#分发文件

ansible master -m copy -a "src=./check_port.sh dest=/etc/keepalived/"

启动服务

ansible master -m shell -a  'systemctl enable --now keepalived'
#可能一次起不来 

ansible master -m shell -a  'systemctl start keepalived'

检查服务情况

# ping 虚拟ip
ping 10.120.200.250 

结果：

PING 10.120.200.250 (10.120.200.250) 56(84) bytes of data.
64 bytes from 10.120.200.250: icmp_seq=1 ttl=64 time=0.593 ms
64 bytes from 10.120.200.250: icmp_seq=2 ttl=64 time=0.932 ms
^C
--- 10.120.200.250 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1051ms
rtt min/avg/max/mdev = 0.593/0.762/0.932/0.171 ms

七. kube-apiserver 部署

1. 解压文件并分发 

tar -xzvf kubernetes-server-linux-amd64.tar.gz
cd kubernetes/server


#将k8s相关文件拷贝到所有master节点上
ansible master -m copy -a "src=./bin  dest=/opt/k8s/"
ansible master -m shell -a  'chmod +x /opt/k8s/bin/*'
#将相关文件拷贝到node节点
ansible node -m copy -a "src=./bin/kubelet  dest=/opt/k8s/bin"
ansible node -m copy -a "src=./bin/kube-proxy  dest=/opt/k8s/bin"
ansible node -m shell -a  'chmod +x /opt/k8s/bin/*'

2. 创建Kubernetes 证书和私钥

cat kubernetes-csr.json 
{
  "CN": "kubernetes",
  "hosts": [
    "127.0.0.1",
    "10.120.200.2",
    "10.120.200.3",
    "10.120.200.4",
    "10.120.200.10",
    "10.254.0.1",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local."
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "TJ",
      "L": "TJ",
      "O": "k8s",
      "OU": "4Paradigm"
    }
  ]
}
ansible k8s-master01.k8s.com  -m copy -a "src=./kubernetes-csr.json   dest=/opt/k8s/work"

#生成私钥和证书
ansible k8s-master01.k8s.com -m shell -a "chdir=/opt/k8s/work /opt/k8s/bin/cfssl gencert -ca=/opt/k8s/work/ca.pem \
      -ca-key=/opt/k8s/work/ca-key.pem \
      -config=/opt/k8s/work/ca-config.json \
      -profile=kubernetes kubernetes-csr.json | /opt/k8s/bin/cfssljson -bare kubernetes"
#分发到master节点
for node_ip in ${MASTER_IPS[@]};   do     echo ">>> ${node_ip}"; ansible k8s-master01.k8s.com -m shell  -a "chdir=/opt/k8s/work scp ./kubernetes-key.pem root@${node_ip}:/etc/kubernetes/cert"; done
for node_ip in ${MASTER_IPS[@]};   do     echo ">>> ${node_ip}"; ansible k8s-master01.k8s.com -m shell  -a "chdir=/opt/k8s/work scp ./kubernetes.pem root@${node_ip}:/etc/kubernetes/cert"; done

3. 创建加密配置文件

cat > encryption-config.yaml <<EOF
kind: EncryptionConfig
apiVersion: v1
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: ${ENCRYPTION_KEY}
      - identity: {}
EOF

#分发文件
ansible master -m copy -a "src=./encryption-config.yaml  dest=/etc/kubernetes/"

4. 创建审计策略文件

source /opt/k8s/bin/environment.sh
cat > audit-policy.yaml <<EOF
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
  # The following requests were manually identified as high-volume and low-risk, so drop them.
  - level: None
    resources:
      - group: ""
        resources:
          - endpoints
          - services
          - services/status
    users:
      - 'system:kube-proxy'
    verbs:
      - watch
  - level: None
    resources:
      - group: ""
        resources:
          - nodes
          - nodes/status
    userGroups:
      - 'system:nodes'
    verbs:
      - get
  - level: None
    namespaces:
      - kube-system
    resources:
      - group: ""
        resources:
          - endpoints
    users:
      - 'system:kube-controller-manager'
      - 'system:kube-scheduler'
      - 'system:serviceaccount:kube-system:endpoint-controller'
    verbs:
      - get
      - update
  - level: None
    resources:
      - group: ""
        resources:
          - namespaces
          - namespaces/status
          - namespaces/finalize
    users:
      - 'system:apiserver'
    verbs:
      - get
  # Don't log HPA fetching metrics.
  - level: None
    resources:
      - group: metrics.k8s.io
    users:
      - 'system:kube-controller-manager'
    verbs:
      - get
      - list
  # Don't log these read-only URLs.
  - level: None
    nonResourceURLs:
      - '/healthz*'
      - /version
      - '/swagger*'
  # Don't log events requests.
  - level: None
    resources:
      - group: ""
        resources:
          - events
  # node and pod status calls from nodes are high-volume and can be large, don't log responses for expected updates from nodes
  - level: Request
    omitStages:
      - RequestReceived
    resources:
      - group: ""
        resources:
          - nodes/status
          - pods/status
    users:
      - kubelet
      - 'system:node-problem-detector'
      - 'system:serviceaccount:kube-system:node-problem-detector'
    verbs:
      - update
      - patch
  - level: Request
    omitStages:
      - RequestReceived
    resources:
      - group: ""
        resources:
          - nodes/status
          - pods/status
    userGroups:
      - 'system:nodes'
    verbs:
      - update
      - patch
  # deletecollection calls can be large, don't log responses for expected namespace deletions
  - level: Request
    omitStages:
      - RequestReceived
    users:
      - 'system:serviceaccount:kube-system:namespace-controller'
    verbs:
      - deletecollection
  # Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data,
  # so only log at the Metadata level.
  - level: Metadata
    omitStages:
      - RequestReceived
    resources:
      - group: ""
        resources:
          - secrets
          - configmaps
      - group: authentication.k8s.io
        resources:
          - tokenreviews
  # Get repsonses can be large; skip them.
  - level: Request
    omitStages:
      - RequestReceived
    resources:
      - group: ""
      - group: admissionregistration.k8s.io
      - group: apiextensions.k8s.io
      - group: apiregistration.k8s.io
      - group: apps
      - group: authentication.k8s.io
      - group: authorization.k8s.io
      - group: autoscaling
      - group: batch
      - group: certificates.k8s.io
      - group: extensions
      - group: metrics.k8s.io
      - group: networking.k8s.io
      - group: policy
      - group: rbac.authorization.k8s.io
      - group: scheduling.k8s.io
      - group: settings.k8s.io
      - group: storage.k8s.io
    verbs:
      - get
      - list
      - watch
  # Default level for known APIs
  - level: RequestResponse
    omitStages:
      - RequestReceived
    resources:
      - group: ""
      - group: admissionregistration.k8s.io
      - group: apiextensions.k8s.io
      - group: apiregistration.k8s.io
      - group: apps
      - group: authentication.k8s.io
      - group: authorization.k8s.io
      - group: autoscaling
      - group: batch
      - group: certificates.k8s.io
      - group: extensions
      - group: metrics.k8s.io
      - group: networking.k8s.io
      - group: policy
      - group: rbac.authorization.k8s.io
      - group: scheduling.k8s.io
      - group: settings.k8s.io
      - group: storage.k8s.io
  # Default level for all other requests.
  - level: Metadata
    omitStages:
      - RequestReceived
EOF

分发文件

ansible master -m copy -a "src=./audit-policy.yaml  dest=/etc/kubernetes/"

5. 创建证书签名请求

 cat > proxy-client-csr.json <<EOF                                         
{
  "CN": "aggregator",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "TJ",
      "L": "TJ",
      "O": "k8s",
      "OU": "4Paradigm"
    }
  ]
}
EOF

ansible k8s-master01.k8s.com  -m copy -a "src=./proxy-client-csr.json  dest=/opt/k8s/work"

#生成证书及私钥

ansible k8s-master01.k8s.com -m shell -a "chdir=/opt/k8s/work  /opt/k8s/bin/cfssl gencert -ca=/etc/kubernetes/cert/ca.pem \
  -ca-key=/etc/kubernetes/cert/ca-key.pem  \
  -config=/etc/kubernetes/cert/ca-config.json  \
  -profile=kubernetes proxy-client-csr.json | /opt/k8s/bin/cfssljson -bare proxy-client"

#分发证书
for node_ip in ${MASTER_IPS[@]};   do     echo ">>> ${node_ip}"; ansible k8s-master01.k8s.com -m shell  -a "chdir=/opt/k8s/work scp ./proxy-client-key.pem root@${node_ip}:/etc/kubernetes/cert"; done
for node_ip in ${MASTER_IPS[@]};   do     echo ">>> ${node_ip}"; ansible k8s-master01.k8s.com -m shell  -a "chdir=/opt/k8s/work scp ./proxy-client.pem root@${node_ip}:/etc/kubernetes/cert"; done

6. 创建kube-apiserver启动文件

创建模板文件

 cat > kube-apiserver.service.template <<EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
[Service]
WorkingDirectory=${K8S_DIR}/kube-apiserver
ExecStart=/opt/k8s/bin/kube-apiserver \\
  --advertise-address=##NODE_IP## \\
  --default-not-ready-toleration-seconds=360 \\
  --default-unreachable-toleration-seconds=360 \\
  --max-mutating-requests-inflight=2000 \\
  --max-requests-inflight=4000 \\
  --default-watch-cache-size=200 \\
  --delete-collection-workers=2 \\
  --encryption-provider-config=/etc/kubernetes/encryption-config.yaml \\
  --etcd-cafile=/etc/kubernetes/cert/ca.pem \\
  --etcd-certfile=/etc/kubernetes/cert/kubernetes.pem \\
  --etcd-keyfile=/etc/kubernetes/cert/kubernetes-key.pem \\
  --etcd-servers=${ETCD_ENDPOINTS} \\
  --bind-address=##NODE_IP## \\
  --secure-port=6443 \\
  --tls-cert-file=/etc/kubernetes/cert/kubernetes.pem \\
  --tls-private-key-file=/etc/kubernetes/cert/kubernetes-key.pem \\
  --insecure-port=0 \\
  --audit-log-maxage=15 \\
  --audit-log-maxbackup=3 \\
  --audit-log-maxsize=100 \\
  --audit-log-truncate-enabled \\
  --audit-log-path=${K8S_DIR}/kube-apiserver/audit.log \\
  --audit-policy-file=/etc/kubernetes/audit-policy.yaml \\
  --profiling \\
  --anonymous-auth=false \\
  --client-ca-file=/etc/kubernetes/cert/ca.pem \\
  --enable-bootstrap-token-auth \\
  --requestheader-allowed-names="aggregator" \\
  --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \\
  --requestheader-extra-headers-prefix="X-Remote-Extra-" \\
  --requestheader-group-headers=X-Remote-Group \\
  --requestheader-username-headers=X-Remote-User \\
  --service-account-key-file=/etc/kubernetes/cert/ca.pem \\
  --authorization-mode=Node,RBAC \\
  --runtime-config=api/all=true \\
  --enable-admission-plugins=NodeRestriction \\
  --allow-privileged=true \\
  --apiserver-count=3 \\
  --event-ttl=168h \\
  --kubelet-certificate-authority=/etc/kubernetes/cert/ca.pem \\
  --kubelet-client-certificate=/etc/kubernetes/cert/kubernetes.pem \\
  --kubelet-client-key=/etc/kubernetes/cert/kubernetes-key.pem \\
  --kubelet-https=true \\
  --kubelet-timeout=10s \\
  --proxy-client-cert-file=/etc/kubernetes/cert/proxy-client.pem \\
  --proxy-client-key-file=/etc/kubernetes/cert/proxy-client-key.pem \\
  --service-cluster-ip-range=${SERVICE_CIDR} \\
  --service-node-port-range=${NODE_PORT_RANGE} \\
  --logtostderr=true \\
  --v=2
Restart=on-failure
RestartSec=10
Type=notify
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

生成节点配置文件并分发

for (( i=0; i < 3; i++ ))    do     sed -e "s/##NODE_NAME##/${MASTER_NAMES[i]}/" -e "s/##NODE_IP##/${MASTER_IPS[i]}/" kube-apiserver.service.template > kube-apiserver-${MASTER_IPS[i]}.service ;   done

ansible k8s-master01.k8s.com  -m copy -a "src=./kube-apiserver-10.120.200.2.service  dest=/etc/systemd/system/kube-apiserver.service"
ansible k8s-master02.k8s.com  -m copy -a "src=./kube-apiserver-10.120.200.3.service  dest=/etc/systemd/system/kube-apiserver.service"

启动并检查服务

ansible master -m shell -a  'mkdir -p /data/k8s/k8s/kube-apiserver'
ansible master -m shell -a  'systemctl daemon-reload && systemctl enable kube-apiserver && systemctl restart kube-apiserver' 

#检查启动进程
ansible master -m shell -a "systemctl status kube-apiserver |grep 'Active:'"

结果：
k8s-master02.k8s.com | CHANGED | rc=0 >>
   Active: active (running) since 二 2020-12-29 16:45:57 CST; 2m ago
k8s-master01.k8s.com | CHANGED | rc=0 >>
   Active: active (running) since 二 2020-12-29 16:44:19 CST; 2m ago

打印kube-apiserver写入etcd数据

ansible k8s-master01.k8s.com -m shell -a "ETCDCTL_API=3 /opt/k8s/bin/etcdctl \
    --endpoints=${ETCD_ENDPOINTS} \
    --cacert=/opt/k8s/work/ca.pem \
    --cert=/opt/k8s/work/etcd.pem \
    --key=/opt/k8s/work/etcd-key.pem \
    get /registry/ --prefix --keys-only"

#结果很多就不在这里复制了

检查kube-apiserver监听的端口

ansible master -m shell -a "netstat -lntup|grep kube"

#结果为

k8s-master02.k8s.com | CHANGED | rc=0 >>
tcp        0      0 10.120.200.3:6443       0.0.0.0:*               LISTEN      19745/kube-apiserve 
k8s-master01.k8s.com | CHANGED | rc=0 >>
tcp        0      0 10.120.200.2:6443       0.0.0.0:*               LISTEN      48893/kube-apiserve

集群检查

ansible k8s-master01.k8s.com -m shell -a "kubectl cluster-info"

#结果为

k8s-master01.k8s.com | CHANGED | rc=0 >>
Kubernetes master is running at https://10.120.200.100:8443
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

ansible k8s-master01.k8s.com -m shell -a "kubectl get all --all-namespaces"

#结果为
k8s-master01.k8s.com | CHANGED | rc=0 >>
NAMESPACE   NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
default     service/kubernetes   ClusterIP   10.254.0.1   <none>        443/TCP   19h

ansible k8s-master01.k8s.com -m shell -a "kubectl get componentstatuses"
#结果为
k8s-master01.k8s.com | CHANGED | rc=0 >>
NAME                 STATUS      MESSAGE                                                                                       ERROR
scheduler            Unhealthy   Get "http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect: connection refused   
controller-manager   Unhealthy   Get "http://127.0.0.1:10252/healthz": dial tcp 127.0.0.1:10252: connect: connection refused   
etcd-2               Healthy     {"health":"true"}                                                                             
etcd-0               Healthy     {"health":"true"}                                                                             
etcd-1               Healthy     {"health":"true"}                                                                             Warning: v1 ComponentStatus is deprecated in v1.19+

授权kube-apiserver访问kubelet API的权限

ansible k8s-master01.k8s.com -m shell -a  'kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes'

 八. 部署kube-controller-manager

1. 创建证书签名

cat kube-controller-manager-csr.json 
{
    "CN": "system:kube-controller-manager",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "hosts": [
      "127.0.0.1",
      "10.120.200.2",
      "10.120.200.3"
    ],
    "names": [
      {
        "C": "CN",
        "ST": "TJ",
        "L": "TJ",
        "O": "system:kube-controller-manager",
        "OU": "4Paradigm"
      }
    ]
}

ansible k8s-master01.k8s.com  -m copy -a "src=./kube-controller-manager-csr.json  dest=/opt/k8s/work"  

#生成证书和私钥

ansible k8s-master01.k8s.com -m shell -a "chdir=/opt/k8s/work  cfssl gencert -ca=/opt/k8s/work/ca.pem \
  -ca-key=/opt/k8s/work/ca-key.pem \
  -config=/opt/k8s/work/ca-config.json \
  -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager"

#分发到节点服务器
for node_ip in ${MASTER_IPS[@]};   do     echo ">>> ${node_ip}"; ansible k8s-master01.k8s.com -m shell  -a "chdir=/opt/k8s/work scp ./kube-controller-manager-key.pem root@${node_ip}:/etc/kubernetes/cert"; done

for node_ip in ${MASTER_IPS[@]};   do     echo ">>> ${node_ip}"; ansible k8s-master01.k8s.com -m shell  -a "chdir=/opt/k8s/work scp ./kube-controller-manager.pem root@${node_ip}:/etc/kubernetes/cert"; done

2. 创建和分发kubeconfig文件

source /opt/k8s/bin/environment.sh
ansible k8s-master01.k8s.com -m shell -a "chdir=/opt/k8s/work  kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/k8s/work/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=kube-controller-manager.kubeconfig"

ansible k8s-master01.k8s.com -m shell -a "chdir=/opt/k8s/work  kubectl config set-credentials system:kube-controller-manager \
  --client-certificate=kube-controller-manager.pem \
  --client-key=kube-controller-manager-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-controller-manager.kubeconfig"

ansible k8s-master01.k8s.com -m shell -a "chdir=/opt/k8s/work kubectl config set-context system:kube-controller-manager \     
  --cluster=kubernetes \                            
  --user=system:kube-controller-manager \       
  --kubeconfig=kube-controller-manager.kubeconfig"

ansible k8s-master01.k8s.com -m shell -a "chdir=/opt/k8s/work  kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig"

#分发到节点服务器

for node_ip in ${MASTER_IPS[@]};   do     echo ">>> ${node_ip}"; ansible k8s-master01.k8s.com -m shell  -a "chdir=/opt/k8s/work scp ./kube-controller-manager.kubeconfig root@${node_ip}:/etc/kubernetes/"; done

3. 创建kube-controller-manager启动文件

cat > kube-controller-manager.service.template <<EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
WorkingDirectory=${K8S_DIR}/kube-controller-manager
ExecStart=/opt/k8s/bin/kube-controller-manager \\
  --profiling \\
  --cluster-name=kubernetes \\
  --controllers=*,bootstrapsigner,tokencleaner \\
  --kube-api-qps=1000 \\
  --kube-api-burst=2000 \\
  --leader-elect \\
  --use-service-account-credentials\\
  --concurrent-service-syncs=2 \\
  --bind-address=0.0.0.0 \\
  --tls-cert-file=/etc/kubernetes/cert/kube-controller-manager.pem \\
  --tls-private-key-file=/etc/kubernetes/cert/kube-controller-manager-key.pem \\
  --authentication-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\
  --client-ca-file=/etc/kubernetes/cert/ca.pem \\
  --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \\
  --requestheader-extra-headers-prefix="X-Remote-Extra-" \\
  --requestheader-group-headers=X-Remote-Group \\
  --requestheader-username-headers=X-Remote-User \\
  --authorization-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\
  --cluster-signing-cert-file=/etc/kubernetes/cert/ca.pem \\
  --cluster-signing-key-file=/etc/kubernetes/cert/ca-key.pem \\
  --experimental-cluster-signing-duration=876000h \\
  --horizontal-pod-autoscaler-sync-period=10s \\
  --concurrent-deployment-syncs=10 \\
  --concurrent-gc-syncs=30 \\
  --node-cidr-mask-size=24 \\
  --service-cluster-ip-range=${SERVICE_CIDR} \\
  --pod-eviction-timeout=6m \\
  --terminated-pod-gc-threshold=10000 \\
  --root-ca-file=/etc/kubernetes/cert/ca.pem \\
  --service-account-private-key-file=/etc/kubernetes/cert/ca-key.pem \\
  --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\
  --logtostderr=true \\
  --v=2
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF

#替换启动文件，并分发脚本
source /opt/k8s/bin/environment.sh
for (( i=0; i < 3; i++ ))
  do
    sed -e "s/##NODE_NAME##/${MASTER_NAMES[i]}/" -e "s/##NODE_IP##/${MASTER_IPS[i]}/" kube-controller-manager.service.template > kube-controller-manager-${MASTER_IPS[i]}.service 
  done

#分发到节点
ansible k8s-master01.k8s.com  -m copy -a "src=./kube-controller-manager-10.120.200.2.service   dest=/etc/systemd/system/kube-controller-manager.service "
ansible k8s-master02.k8s.com  -m copy -a "src=./kube-controller-manager-10.120.200.3.service   dest=/etc/systemd/system/kube-controller-manager.service "

4.启动服务

ansible master -m shell -a  'mkdir -p /data/k8s/k8s/kube-controller-manager'
ansible master -m shell -a  'systemctl daemon-reload && systemctl enable kube-controller-manager && systemctl restart kube-controller-manager'  
#检查进程
ansible master -m shell -a  'systemctl status kube-controller-manager|grep Active'                                                                          
#检查端口
ansible master -m shell -a  'netstat -lnpt | grep kube-cont'
##结果为：

k8s-master02.k8s.com | CHANGED | rc=0 >>
tcp6       0      0 :::10252                :::*                    LISTEN      119404/kube-control 
tcp6       0      0 :::10257                :::*                    LISTEN      119404/kube-control 
k8s-master01.k8s.com | CHANGED | rc=0 >>
tcp6       0      0 :::10252                :::*                    LISTEN      18696/kube-controll 
tcp6       0      0 :::10257                :::*                    LISTEN      18696/kube-controll

5.kube-controller-manager 创建权限

当前ClusteRole system:kube-controller-manager权限过小，只能创建secret、serviceaccount等资源,需要将controller权限分配到ClusterRole system:controller:xxx中。在启动文件增加 --use-service-account-credentials=true ，这样主控会建ServiceAccount XXX-controlle， ClusterRoleBinding system:controller:XXX 会给各 XXX-controller ServiceAccount 对应的 ClusterRole system:controller:XXX 权限。

ansible k8s-master01.k8s.com -m shell -a " kubectl describe clusterrole system:kube-controller-manager"
##结果：
k8s-master01.k8s.com | CHANGED | rc=0 >>
Name:         system:kube-controller-manager
Labels:       kubernetes.io/bootstrapping=rbac-defaults
Annotations:  rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
  Resources                                  Non-Resource URLs  Resource Names             Verbs
  ---------                                  -----------------  --------------             -----
  secrets                                    []                 []                         [create delete get update]
  serviceaccounts                            []                 []                         [create get update]
  events                                     []                 []                         [create patch update]
  events.events.k8s.io                       []                 []                         [create patch update]
  endpoints                                  []                 []                         [create]
  serviceaccounts/token                      []                 []                         [create]
  tokenreviews.authentication.k8s.io         []                 []                         [create]
  subjectaccessreviews.authorization.k8s.io  []                 []                         [create]
  leases.coordination.k8s.io                 []                 []                         [create]
  endpoints                                  []                 [kube-controller-manager]  [get update]
  leases.coordination.k8s.io                 []                 [kube-controller-manager]  [get update]
  configmaps                                 []                 []                         [get]
  namespaces                                 []                 []                         [get]
  *.*                                        []                 []                         [list watch]
 ansible k8s-master01.k8s.com -m shell -a " kubectl get clusterrole|grep controller"                    
k8s-master01.k8s.com | CHANGED | rc=0 >>
system:controller:attachdetach-controller                              2020-12-29T08:44:24Z
system:controller:certificate-controller                               2020-12-29T08:44:26Z
system:controller:clusterrole-aggregation-controller                   2020-12-29T08:44:24Z
system:controller:cronjob-controller                                   2020-12-29T08:44:24Z
system:controller:daemon-set-controller                                2020-12-29T08:44:24Z
system:controller:deployment-controller                                2020-12-29T08:44:24Z
system:controller:disruption-controller                                2020-12-29T08:44:24Z
system:controller:endpoint-controller                                  2020-12-29T08:44:24Z
system:controller:endpointslice-controller                             2020-12-29T08:44:25Z
system:controller:endpointslicemirroring-controller                    2020-12-29T08:44:25Z
system:controller:expand-controller                                    2020-12-29T08:44:25Z
system:controller:generic-garbage-collector                            2020-12-29T08:44:25Z
system:controller:horizontal-pod-autoscaler                            2020-12-29T08:44:25Z
system:controller:job-controller                                       2020-12-29T08:44:25Z
system:controller:namespace-controller                                 2020-12-29T08:44:25Z
system:controller:node-controller                                      2020-12-29T08:44:25Z
system:controller:persistent-volume-binder                             2020-12-29T08:44:25Z
system:controller:pod-garbage-collector                                2020-12-29T08:44:25Z
system:controller:pv-protection-controller                             2020-12-29T08:44:26Z
system:controller:pvc-protection-controller                            2020-12-29T08:44:26Z
system:controller:replicaset-controller                                2020-12-29T08:44:25Z
system:controller:replication-controller                               2020-12-29T08:44:25Z
system:controller:resourcequota-controller                             2020-12-29T08:44:26Z
system:controller:route-controller                                     2020-12-29T08:44:26Z
system:controller:service-account-controller                           2020-12-29T08:44:26Z
system:controller:service-controller                                   2020-12-29T08:44:26Z
system:controller:statefulset-controller                               2020-12-29T08:44:26Z
system:controller:ttl-controller                                       2020-12-29T08:44:26Z
system:kube-controller-manager                                         2020-12-29T08:44:23Z

以system:controller:attachdetach-controller为例

ansible k8s-master01.k8s.com -m shell -a " kubectl describe clusterrole system:controller:attachdetach-controller"
k8s-master01.k8s.com | CHANGED | rc=0 >>
Name:         system:controller:attachdetach-controller
Labels:       kubernetes.io/bootstrapping=rbac-defaults
Annotations:  rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
  Resources                         Non-Resource URLs  Resource Names  Verbs
  ---------                         -----------------  --------------  -----
  volumeattachments.storage.k8s.io  []                 []              [create delete get list watch]
  events                            []                 []              [create patch update]
  events.events.k8s.io              []                 []              [create patch update]
  nodes                             []                 []              [get list watch]
  csidrivers.storage.k8s.io         []                 []              [get list watch]
  csinodes.storage.k8s.io           []                 []              [get list watch]
  persistentvolumeclaims            []                 []              [list watch]
  persistentvolumes                 []                 []              [list watch]
  pods                              []                 []              [list watch]
  nodes/status                      []                 []              [patch update]

查看当前的 leader

ansible k8s-master01.k8s.com -m shell -a " kubectl get endpoints kube-controller-manager --namespace=kube-system  -o yaml"
k8s-master01.k8s.com | CHANGED | rc=0 >>
apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"k8s-master01_06a89fb9-a6c8-404a-a3fc-0988e7b8960c","leaseDurationSeconds":15,"acquireTime":"2020-12-30T06:31:02Z","renewTime":"2020-12-30T06:40:17Z","leaderTransitions":0}'
  creationTimestamp: "2020-12-30T06:31:03Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:control-plane.alpha.kubernetes.io/leader: {}
    manager: kube-controller-manager
    operation: Update
    time: "2020-12-30T06:40:17Z"
  name: kube-controller-manager
  namespace: kube-system
  resourceVersion: "16869"
  selfLink: /api/v1/namespaces/kube-system/endpoints/kube-controller-manager
  uid: 84b77801-1b03-4d95-b5fa-cc604106171f

九. 部署kube-scheduler

1. 创建证书签名

cat kube-scheduler-csr.json 
{
    "CN": "system:kube-scheduler",
    "hosts": [
      "127.0.0.1",
      "10.120.200.2",
      "10.120.200.3"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
      {
        "C": "CN",
        "ST": "TJ",
        "L": "TJ",
        "O": "system:kube-scheduler",
        "OU": "4Paradigm"
      }
    ]
}

#发送到master01

ansible k8s-master01.k8s.com  -m copy -a "src=./kube-scheduler-csr.json  dest=/opt/k8s/work"

#生成证书和私钥
ansible k8s-master01.k8s.com -m shell -a " chdir=/opt/k8s/work cfssl gencert -ca=/opt/k8s/work/ca.pem \           
  -ca-key=/opt/k8s/work/ca-key.pem \
  -config=/opt/k8s/work/ca-config.json \
  -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler"

#分发到master节点
 for node_ip in ${MASTER_IPS[@]};   do     echo ">>> ${node_ip}"; ansible k8s-master01.k8s.com -m shell  -a "chdir=/opt/k8s/work scp ./kube-scheduler-key.pem  root@${node_ip}:/etc/kubernetes/cert"; done

for node_ip in ${MASTER_IPS[@]};   do     echo ">>> ${node_ip}"; ansible k8s-master01.k8s.com -m shell  -a "chdir=/opt/k8s/work scp ./kube-scheduler.pem  root@${node_ip}:/etc/kubernetes/cert"; done

2. 创建和分发 kubeconfig 文件

ansible k8s-master01.k8s.com -m shell -a "chdir=/opt/k8s/work kubectl config set-cluster kubernetes \                                                                                                     
  --certificate-authority=/opt/k8s/work/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=kube-scheduler.kubeconfig"

ansible k8s-master01.k8s.com -m shell -a "chdir=/opt/k8s/work kubectl config set-credentials system:kube-scheduler \                                                                                       
  --client-certificate=kube-scheduler.pem \          
  --client-key=kube-scheduler-key.pem \                 
  --embed-certs=true \                
  --kubeconfig=kube-scheduler.kubeconfig"

ansible k8s-master01.k8s.com -m shell -a "chdir=/opt/k8s/work kubectl config set-context system:kube-scheduler \                                                                                          
  --cluster=kubernetes \                             
  --user=system:kube-scheduler \                        
  --kubeconfig=kube-scheduler.kubeconfig"

ansible k8s-master01.k8s.com -m shell -a "chdir=/opt/k8s/work kubectl config use-context system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig"

#分发文件

for node_ip in ${MASTER_IPS[@]};   do     echo ">>> ${node_ip}"; ansible k8s-master01.k8s.com -m shell  -a "chdir=/opt/k8s/work scp ./kube-scheduler.kubeconfig root@${node_ip}:/etc/kubernetes/"; done

3.创建 kube-scheduler 配置文件

cat /etc/kubernetes/kube-scheduler.yaml   
apiVersion: kubescheduler.config.k8s.io/v1beta1
kind: KubeSchedulerConfiguration
clientConnection:
  kubeconfig: "/etc/kubernetes/kube-scheduler.kubeconfig"
  qps: 100
enableContentionProfiling: false
enableProfiling: true
healthzBindAddress: 127.0.0.1:10251
leaderElection:
  leaderElect: true
metricsBindAddress: ##NODE_IP##:10251

for (( i=0; i < 3; i++ ));   do     sed -e "s/##NODE_NAME##/${MASTER_NAMES[i]}/" -e "s/##NODE_IP##/${NODE_IPS[i]}/" kube-scheduler.yaml.template > kube-scheduler-${MASTER_IPS[i]}.yaml; done


#分发
ansible k8s-master01.k8s.com  -m copy -a "src=./kube-scheduler-10.120.200.2.yaml    dest=/etc/kubernetes/kube-scheduler.yaml "

ansible k8s-master02.k8s.com  -m copy -a "src=./kube-scheduler-10.120.200.3.yaml    dest=/etc/kubernetes/kube-scheduler.yaml "

4. 创建启动文件

 cat > kube-scheduler.service.template <<EOF                                                                                      
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
WorkingDirectory=${K8S_DIR}/kube-scheduler
ExecStart=/opt/k8s/bin/kube-scheduler \\
  --config=/etc/kubernetes/kube-scheduler.yaml \\
  --bind-address=##NODE_IP## \\
  --secure-port=10259 \\
  --port=0 \\
  --tls-cert-file=/etc/kubernetes/cert/kube-scheduler.pem \\
  --tls-private-key-file=/etc/kubernetes/cert/kube-scheduler-key.pem \\
  --authentication-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \\
  --client-ca-file=/etc/kubernetes/cert/ca.pem \\
  --requestheader-allowed-names="" \\
  --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \\
  --requestheader-extra-headers-prefix="X-Remote-Extra-" \\
  --requestheader-group-headers=X-Remote-Group \\
  --requestheader-username-headers=X-Remote-User \\
  --authorization-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \\
  --logtostderr=true \\
  --v=2
Restart=always
RestartSec=5
StartLimitInterval=0
[Install]
WantedBy=multi-user.target
EOF

for (( i=0; i < 3; i++ ));   do     sed -e "s/##NODE_NAME##/${MASTER_NAMES[i]}/" -e "s/##NODE_IP##/${MASTER_IPS[i]}/" kube-scheduler.service.template > kube-scheduler-${NODE_IPS[i]}.service ;   done

#分发
ansible k8s-master01.k8s.com  -m copy -a "src=./kube-scheduler-10.120.200.2.service    dest=/etc/systemd/system/kube-scheduler.service"

ansible k8s-master02.k8s.com  -m copy -a "src=./kube-scheduler-10.120.200.3.service    dest=/etc/systemd/system/kube-scheduler.service"

5. 启动服务并检查

ansible master -m shell -a  'mkdir -p /opt/k8s/work/kube-scheduler'

ansible master -m shell -a  'systemctl daemon-reload && systemctl enable kube-scheduler && systemctl restart kube-scheduler'

ansible master -m shell -a  'systemctl status kube-scheduler|grep Active'

#查看leader
kubectl get endpoints kube-scheduler --namespace=kube-system  -o yaml
#结果:
apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"k8s-master01_427509fc-39d9-4fd2-8fb4-3637735e0849","leaseDurationSeconds":15,"acquireTime":"2020-12-30T15:09:21Z","renewTime":"2020-12-31T02:46:43Z","leaderTransitions":1}'
  creationTimestamp: "2020-12-30T15:09:05Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:control-plane.alpha.kubernetes.io/leader: {}
    manager: kube-scheduler
    operation: Update
    time: "2020-12-31T02:46:43Z"
  name: kube-scheduler
  namespace: kube-system
  resourceVersion: "140866"
  selfLink: /api/v1/namespaces/kube-system/endpoints/kube-scheduler
  uid: 853fb510-a05c-418f-890f-6fa34f2bdf4d
'

这周事儿太多，文档中断了一周。今天开始接着往下继续。开始安装各个node的节点。看到胜利的曙光了

十. docker的安装

1. 安装docker服务

由于我这边做的内部yum源，直接可以yum安装。在实际情况中根据自己的实际状况进行安装。这个比较简单，这里不做过度的介绍。

ansible all -m shell -a  'yum install -y docker-ce-19.03.12-3.el7.x86_64.rpm docker-ce-cli-19.03.12-3.el7.x86_64.rpm  containerd.io-1.2.13-3.2.el7.x86_64.rpm'

2. 创建配置文件

cat daemon.json 
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "registry-mirrors": ["https://aad0405c.m.daocloud.io"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "insecure-registries": ["harbor.k8s.com"]
}

ansible all -m copy -a "src=./daemon.json dest=/etc/docker/"
cat docker.service   
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
#EnvironmentFile=/run/flannel/subnet.env
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS
#ExecStart=/usr/bin/dockerd  $DOCKER_NETWORK_OPTIONS -H fd:// --containerd=/run/containerd/containerd.sock
EnvironmentFile=-/run/flannel/docker
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target


ansible all -m copy -a "src=./docker.service dest=/usr/lib/systemd/system/"

3. 启动服务并检查

ansible all -m shell -a "systemctl daemon-reload && systemctl enable docker && systemctl restart docker"

#检查检查存活

ansible all -m shell -a "systemctl status docker|grep Active"      

# 查看docker0 网卡信息

ansible all -m shell -a "ip addr show docker0" 

#查看存储驱动
ansible all -m shell -a "docker info|grep overlay2"

十一. 部署kubelet组件

1. 创建kubelet bootstrap kubeconfig文件

source /opt/k8s/bin/environment.sh

for node_name in ${NODE_NAMES[@]}
  do
    echo ">>> ${node_name}"
    # 创建 token
    export BOOTSTRAP_TOKEN=$(kubeadm token create \
      --description kubelet-bootstrap-token \
      --groups system:bootstrappers:${node_name} \
      --kubeconfig ~/.kube/config)
    # 设置集群参数
    kubectl config set-cluster kubernetes \
      --certificate-authority=/etc/kubernetes/cert/ca.pem \
      --embed-certs=true \
      --server=${KUBE_APISERVER} \
      --kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig
    # 设置客户端认证参数
    kubectl config set-credentials kubelet-bootstrap \
      --token=${BOOTSTRAP_TOKEN} \
      --kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig
    # 设置上下文参数
    kubectl config set-context default \
      --cluster=kubernetes \
      --user=kubelet-bootstrap \
      --kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig
    # 设置默认上下文
    kubectl config use-context default --kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig
  done

#分发文件
 for node_name in ${NODE_NAMES[@]};   do     echo ">>> ${node_name}";     scp kubelet-bootstrap-${node_name}.kubeconfig root@${node_name}.k8s.com:/etc/kubernetes/bootstrap-kubelet.conf;   done



#分发文件

 for node_name in ${NODE_NAMES[@]};   do     echo ">>> ${node_name}";     scp kubelet-bootstrap-${node_name}.kubeconfig root@${node_name}.k8s.com:/etc/kubernetes/kubelet-bootstrap.kubeconfig;   done

2. 查看kubeadm为各个节点创建的token

kubeadm token list --kubeconfig ~/.kube/config


kubectl get secrets  -n kube-system|grep bootstrap-token

3. 创建配置文件

cat > kubelet-config.yaml.template <<EOF
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: "##NODE_IP##"
staticPodPath: ""
syncFrequency: 1m
fileCheckFrequency: 20s
httpCheckFrequency: 20s
staticPodURL: ""
port: 10250
readOnlyPort: 0
rotateCertificates: true
serverTLSBootstrap: true
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: true
  x509:
    clientCAFile: "/etc/kubernetes/cert/ca.pem"
authorization:
  mode: Webhook
registryPullQPS: 0
registryBurst: 20
eventRecordQPS: 0
eventBurst: 20
enableDebuggingHandlers: true
enableContentionProfiling: true
healthzPort: 10248
healthzBindAddress: "##NODE_IP##"
clusterDomain: "${CLUSTER_DNS_DOMAIN}"
clusterDNS:
  - "${CLUSTER_DNS_SVC_IP}"
nodeStatusUpdateFrequency: 10s
nodeStatusReportFrequency: 1m
imageMinimumGCAge: 2m
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
volumeStatsAggPeriod: 1m
kubeletCgroups: ""
systemCgroups: ""
cgroupRoot: ""
cgroupsPerQOS: true
cgroupDriver: systemd
runtimeRequestTimeout: 10m
hairpinMode: promiscuous-bridge
maxPods: 220
podCIDR: "${CLUSTER_CIDR}"
podPidsLimit: -1
resolvConf: /etc/resolv.conf
maxOpenFiles: 1000000
kubeAPIQPS: 1000
kubeAPIBurst: 2000
serializeImagePulls: false
evictionHard:
  memory.available:  "100Mi"
nodefs.available:  "10%"
nodefs.inodesFree: "5%"
imagefs.available: "15%"
evictionSoft: {}
enableControllerAttachDetach: true
failSwapOn: true
containerLogMaxSize: 20Mi
containerLogMaxFiles: 10
systemReserved: {}
kubeReserved: {}
systemReservedCgroup: ""
kubeReservedCgroup: ""
enforceNodeAllocatable: ["pods"]
EOF

for node_ip in ${NODE_IPS[@]}
  do 
    echo ">>> ${node_ip}"
    sed -e "s/##NODE_IP##/${node_ip}/" kubelet-config.yaml.template > kubelet-config-${node_ip}.yaml.template
    scp kubelet-config-${node_ip}.yaml.template root@${node_ip}:/etc/kubernetes/kubelet.kubeconfig
  done

4. 创建和分发kubelet启动文件

cat > kubelet.service.template <<EOF
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service
[Service]
WorkingDirectory=${K8S_DIR}/kubelet
ExecStart=/opt/k8s/bin/kubelet \\
  --allow-privileged=true \\
  --bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig \\
  --cert-dir=/etc/kubernetes/cert \\
  --cni-conf-dir=/etc/cni/net.d \\
  --container-runtime=docker \\
  --container-runtime-endpoint=unix:///var/run/dockershim.sock \\
  --root-dir=${K8S_DIR}/kubelet \\
  --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \\
  --config=/etc/kubernetes/kubelet-config.yaml \\
  --hostname-override=##NODE_NAME## \\
  --pod-infra-container-image=harbor.k8s.com/k8s-base/pause-amd64:3.1 \\
  --image-pull-progress-deadline=15m \\
  --volume-plugin-dir=${K8S_DIR}/kubelet/kubelet-plugins/volume/exec/ \\
  --logtostderr=true \\
  --v=2
Restart=always
RestartSec=5
StartLimitInterval=0
[Install]
WantedBy=multi-user.target
EOF

for node_name in ${NODE_NAMES[@]}
  do 
    echo ">>> ${node_name}"
    sed -e "s/##NODE_NAME##/${node_name}/" kubelet.service.template > kubelet-${node_name}.service
    scp kubelet-${node_name}.service root@${node_name}:/etc/systemd/system/kubelet.service
  done

#创建user和group的CSR权限

kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --group=system:bootstrappers

# 启动服务 

ansible all -m shell -a "systemctl daemon-reload && systemctl enable kubelet && systemctl restart kubelet"

5. approve CSR请求

## 自动approve CSR请求。需要等待几分钟
cat > csr-crb.yaml <<EOF
 # Approve all CSRs for the group "system:bootstrappers"
 kind: ClusterRoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
   name: auto-approve-csrs-for-group
 subjects:
 - kind: Group
   name: system:bootstrappers
   apiGroup: rbac.authorization.k8s.io
 roleRef:
   kind: ClusterRole
   name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
   apiGroup: rbac.authorization.k8s.io
---
 # To let a node of the group "system:nodes" renew its own credentials
 kind: ClusterRoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
   name: node-client-cert-renewal
 subjects:
 - kind: Group
   name: system:nodes
   apiGroup: rbac.authorization.k8s.io
 roleRef:
   kind: ClusterRole
   name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
   apiGroup: rbac.authorization.k8s.io
---
# A ClusterRole which instructs the CSR approver to approve a node requesting a
# serving cert matching its client cert.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: approve-node-server-renewal-csr
rules:
- apiGroups: ["certificates.k8s.io"]
  resources: ["certificatesigningrequests/selfnodeserver"]
  verbs: ["create"]
---
 # To let a node of the group "system:nodes" renew its own server credentials
 kind: ClusterRoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
   name: node-server-cert-renewal
 subjects:
 - kind: Group
   name: system:nodes
   apiGroup: rbac.authorization.k8s.io
 roleRef:
   kind: ClusterRole
   name: approve-node-server-renewal-csr
   apiGroup: rbac.authorization.k8s.io
EOF
kubectl apply -f csr-crb.yaml

##手动approve CSR请求
kubectl get csr | grep Pending | awk '{print $1}' | xargs kubectl certificate approve

#查看csr 是不是已经Approved,Issued 状态

kubectl get csr
NAME        AGE     SIGNERNAME                      REQUESTOR                CONDITION
csr-4gthz   3h42m   kubernetes.io/kubelet-serving   system:node:k8s-node02   Approved,Issued
csr-4lpcn   5h30m   kubernetes.io/kubelet-serving   system:node:k8s-node02   Approved,Issued
csr-4n4d9   82m     kubernetes.io/kubelet-serving   system:node:k8s-node04   Approved,Issued
csr-4nzxs   4h44m   kubernetes.io/kubelet-serving   system:node:k8s-node02   Approved,Issued

#查看nodes

kubectl get nodes
NAME           STATUS   ROLES    AGE     VERSION
k8s-master01   Ready    <none>   3m    v1.19.2
k8s-master02   Ready    <none>   3m    v1.19.2
k8s-node01     Ready    <none>   3m    v1.19.2
k8s-node07     Ready    <none>   3m    v1.19.2

#查看API 接口

netstat -lntup|grep kubelet
tcp        0      0 10.120.200.4:10248      0.0.0.0:*               LISTEN      72037/kubelet       
tcp        0      0 10.120.200.4:10250      0.0.0.0:*               LISTEN      72037/kubelet       
tcp        0      0 127.0.0.1:34957         0.0.0.0:*               LISTEN      72037/kubelet

6. bear token认证和授权

kubectl create sa kubelet-api-test
kubectl create clusterrolebinding kubelet-api-test --clusterrole=system:kubelet-api-admin --serviceaccount=default:kubelet-api-test
SECRET=$(kubectl get secrets | grep kubelet-api-test | awk '{print $1}')
TOKEN=$(kubectl describe secret ${SECRET} | grep -E '^token' | awk '{print $2}')
echo ${TOKEN}

十二. 部署kube-proxy组件

1. 创建kube-proxy证书签名请求

cat > kube-proxy-csr.json <<EOF                                                                                                                                                         
{
  "CN": "system:kube-proxy",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "TJ",
      "L": "TJ",
      "O": "k8s",
      "OU": "4Paradigm"
    }
  ]
}
EOF

ansible k8s-master01.k8s.com -m copy -a 'src=./kube-proxy-csr.json dest=/opt/k8s/work/kube-proxy-csr.json'

ansible k8s-master01.k8s.com -m shell -a 'chdir=/opt/k8s/work /opt/k8s/bin/cfssl gencert -ca=/opt/k8s/work/ca.pem \
  -ca-key=/opt/k8s/work/ca-key.pem \
  -config=/opt/k8s/work/ca-config.json \
  -profile=kubernetes  kube-proxy-csr.json | /opt/k8s/bin/cfssljson -bare kube-proxy'

for node_name in ${NODE_NAMES[@]}; do ansible k8s-master01.k8s.com -m shell -a "chidr=/opt/k8s/work scp /opt/k8s/work/kube-proxy-key.pem root@${node_name}.k8s.com :/etc/kubernetes/cert/"; done

for node_name in ${NODE_NAMES[@]}; do ansible k8s-master01.k8s.com -m shell -a "chidr=/opt/k8s/work scp /opt/k8s/work/kube-proxy.pem root@${node_name}:/etc/kubernetes/cert/"; done

2. 创建和分发 kubeconfig 文件

source /opt/k8s/bin/environment.sh

ansible k8s-master01.k8s.com -m shell -a 'chdir=/opt/k8s/work kubectl config set-cluster kubernetes \                             
  --certificate-authority=/opt/k8s/work/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=kube-proxy.kubeconfig'

ansible k8s-master01.k8s.com -m shell -a 'chdir=/opt/k8s/work kubectl config set-credentials kube-proxy \                             
  --client-certificate=kube-proxy.pem \         
  --client-key=kube-proxy-key.pem \
  --embed-certs=true \        
  --kubeconfig=kube-proxy.kubeconfig'

ansible k8s-master01.k8s.com -m shell -a 'chdir=/opt/k8s/work kubectl config set-context default \                                    
  --cluster=kubernetes \                        
  --user=kube-proxy \              
  --kubeconfig=kube-proxy.kubeconfig'
ansible k8s-master01.k8s.com -m shell -a 'chdir=/opt/k8s/work kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig'  

for node_name in ${NODE_NAMES[@]}; do ansible k8s-master01.k8s.com -m shell -a "scp /opt/k8s/work/kube-proxy.kubeconfig root@${node_name}:/etc/kubernetes/"; done

3. 创建kube-proxy配置文件

cat > kube-proxy-config.yaml.template <<EOF                                                                                                                                    
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
clientConnection:
  burst: 200
  kubeconfig: "/etc/kubernetes/kube-proxy.kubeconfig"
  qps: 100
bindAddress: ##NODE_IP##
healthzBindAddress: ##NODE_IP##:10256
metricsBindAddress: ##NODE_IP##:10249
enableProfiling: true
clusterCIDR: ${CLUSTER_CIDR}
hostnameOverride: ##NODE_NAME##
mode: "ipvs"
portRange: ""
kubeProxyIPTablesConfiguration:
  masqueradeAll: false
kubeProxyIPVSConfiguration:
  scheduler: rr
  excludeCIDRs: []
EOF

source /opt/k8s/bin/environment.sh  

for (( i=0; i < 10; i++ ));   do      echo ">>> ${NODE_NAMES[i]}";     sed -e "s/##NODE_NAME##/${NODE_NAMES[i]}/" -e "s/##NODE_IP##/${NODE_IPS[i]}/" kube-proxy-config.yaml.template > kube-proxy-config-${NODE_NAMES[i]}.yaml.template;     scp kube-proxy-config-${NODE_NAMES[i]}.yaml.template root@${node_name}:/etc/kubernetes/kube-proxy-config.yaml; done

# 创建启动文件
cat > kube-proxy.service <<EOF                                                                                                                                                                                   [Unit]                                                                                                                                                        
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
[Service]
WorkingDirectory=${K8S_DIR}/kube-proxy
ExecStart=/opt/k8s/bin/kube-proxy \\
  --config=/etc/kubernetes/kube-proxy-config.yaml \\
  --logtostderr=true \\
  --v=2
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

ansible all -m copy -a 'src=./kube-proxy.service  dest=/etc/systemd/system/'

4. 启动 kube-proxy 服务

ansible all -m shell -a  'mkdir -p /data/k8s/k8s/kube-proxy'

ansible all -m shell -a  'modprobe ip_vs_rr' 

ansible all -m shell -a  'systemctl daemon-reload && systemctl enable kube-proxy && systemctl restart kube-proxy'

# 检测服务状态


 ansible all -m shell -a  'systemctl status kube-proxy|grep Active'                                               
k8s-node01.k8s.com | CHANGED | rc=0 >>
   Active: active (running) since 二 2021-01-19 10:26:15 CST; 2m ago
k8s-master01.k8s.com | CHANGED | rc=0 >>
   Active: active (running) since 二 2021-01-19 10:10:45 CST; 2m ago
k8s-node07.k8s.com | CHANGED | rc=0 >>
   Active: active (running) since 二 2021-01-19 10:10:45 CST; 2m ago
k8s-master02.k8s.com | CHANGED | rc=0 >>
   Active: active (running) since 二 2021-01-19 10:10:46 CST; 2m ago
#检测端口
ansible all -m shell -a  'netstat -lnpt|grep kube-prox'                               
k8s-node01.k8s.com | CHANGED | rc=0 >>
tcp        0      0 10.120.200.4:10249      0.0.0.0:*               LISTEN      37747/kube-proxy    
tcp        0      0 10.120.200.4:10256      0.0.0.0:*               LISTEN      37747/kube-proxy    

k8s-master01.k8s.com | CHANGED | rc=0 >>
tcp        0      0 10.120.200.2:10249      0.0.0.0:*               LISTEN      95352/kube-proxy    
tcp        0      0 10.120.200.2:10256      0.0.0.0:*               LISTEN      95352/kube-proxy    

k8s-node07.k8s.com | CHANGED | rc=0 >>
tcp        0      0 10.120.200.10:10249     0.0.0.0:*               LISTEN      82438/kube-proxy    
tcp        0      0 10.120.200.10:10256     0.0.0.0:*               LISTEN      82438/kube-proxy    
k8s-master02.k8s.com | CHANGED | rc=0 >>
tcp        0      0 10.120.200.3:10249      0.0.0.0:*               LISTEN      20054/kube-proxy    
tcp        0      0 10.120.200.3:10256      0.0.0.0:*               LISTEN      20054/kube-proxy 


#查看ipvs路由规则
ansible all -m shell -a  '/usr/sbin/ipvsadm -ln'                                      
k8s-node01.k8s.com | CHANGED | rc=0 >>
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.254.0.1:443 rr
  -> 10.120.200.2:6443            Masq    1      0          0         
  -> 10.120.200.3:6443            Masq    1      0          0                    
k8s-master01.k8s.com | CHANGED | rc=0 >>
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.254.0.1:443 rr
  -> 10.120.200.2:6443            Masq    1      0          0         
  -> 10.120.200.3:6443            Masq    1      0          0         
k8s-node07.k8s.com | CHANGED | rc=0 >>
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.254.0.1:443 rr
  -> 10.120.200.2:6443            Masq    1      0          0         
  -> 10.120.200.3:6443            Masq    1      0          0         
k8s-master02.k8s.com | CHANGED | rc=0 >>
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.254.0.1:443 rr
  -> 10.120.200.2:6443            Masq    1      0          0         
  -> 10.120.200.3:6443            Masq    1      0          0
#验证集群

kubectl get node
NAME           STATUS     ROLES    AGE   VERSION
k8s-master01   Ready      <none>   5d    v1.19.2
k8s-master02   Ready      <none>   5d    v1.19.2
k8s-node01     Ready      <none>   5d    v1.19.2
k8s-node07     Ready      <none>   5d    v1.19.2

十三. 测试集群功能

1. 创建yml文件

cat nginx-ds.yml
apiVersion: v1
kind: Service
metadata:
  name: nginx-ds
  labels:
    app: nginx-ds
spec:
  type: NodePort
  selector:
    app: nginx-ds
  ports:
  - name: http
    port: 80
    targetPort: 80
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nginx-ds
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  selector:
    matchLabels:
      app: nginx-ds
  template:
    metadata:
      labels:
        app: nginx-ds
    spec:
      containers:
      - name: my-nginx
        image: harbor.k8s.com/k8s-base/nginx:1.13.0-alpine
        ports:
        - containerPort: 80
# 执行yml文件
kubectl create -f nginx-ds.yml

2. 查看pod

kubectl get pod  -o wide
NAME             READY   STATUS    RESTARTS   AGE     IP             NODE           NOMINATED NODE   READINESS GATES
nginx-ds-7j5px   1/1     Running   0          3h26m   172.30.184.2   k8s-node01     <none>           <none>
nginx-ds-9jb9s   1/1     Running   0          3h26m   172.30.192.2   k8s-node07     <none>           <none>
nginx-ds-scxvg   1/1     Running   0          24m     172.30.208.3   k8s-master02   <none>           <none>
nginx-ds-snsjq   1/1     Running   0          3h26m   172.30.224.2   k8s-master01   <none>           <none>

3. 测试连通性

ping 172.30.184.2
PING 172.30.184.2 (172.30.184.2) 56(84) bytes of data.
64 bytes from 172.30.184.2: icmp_seq=1 ttl=63 time=0.745 ms
^C64 bytes from 172.30.184.2: icmp_seq=2 ttl=63 time=1.07 ms
^C
--- 172.30.184.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1029ms
rtt min/avg/max/mdev = 0.745/0.907/1.070/0.165 ms


 ping 172.30.192.2
PING 172.30.192.2 (172.30.192.2) 56(84) bytes of data.
64 bytes from 172.30.192.2: icmp_seq=1 ttl=63 time=0.904 ms
64 bytes from 172.30.192.2: icmp_seq=2 ttl=63 time=0.966 ms
^C
--- 172.30.192.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.904/0.935/0.966/0.031 ms


ping 172.30.208.3
PING 172.30.208.3 (172.30.208.3) 56(84) bytes of data.
64 bytes from 172.30.208.3: icmp_seq=1 ttl=63 time=0.631 ms
64 bytes from 172.30.208.3: icmp_seq=2 ttl=63 time=0.530 ms
^C
--- 172.30.208.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1063ms
rtt min/avg/max/mdev = 0.530/0.580/0.631/0.055 ms


ping 172.30.224.2 
PING 172.30.224.2 (172.30.224.2) 56(84) bytes of data.
64 bytes from 172.30.224.2: icmp_seq=1 ttl=64 time=0.150 ms
64 bytes from 172.30.224.2: icmp_seq=2 ttl=64 time=0.074 ms
^C
--- 172.30.224.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1030ms
rtt min/avg/max/mdev = 0.074/0.112/0.150/0.038 ms

4. 查看服务情况

kubectl get svc |grep nginx-ds
nginx-ds     NodePort    10.254.49.54   <none>        80:9249/TCP   4h46m

 curl -s 10.254.49.54:80  
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

使用浏览器可以查看

十四. CoreDNS安装

1. 创建yaml文件

cat coredns.yaml 
apiVersion: v1
kind: ServiceAccount
metadata:
  name: coredns
  namespace: kube-system
  labels:
      kubernetes.io/cluster-service: "true"
      addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
    addonmanager.kubernetes.io/mode: Reconcile
  name: system:coredns
rules:
- apiGroups:
  - ""
  resources:
  - endpoints
  - services
  - pods
  - namespaces
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
    addonmanager.kubernetes.io/mode: EnsureExists
  name: system:coredns
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:coredns
subjects:
- kind: ServiceAccount
  name: coredns
  namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
  labels:
      addonmanager.kubernetes.io/mode: EnsureExists
data:
  Corefile: |
    .:53 {
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            upstream
            fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: coredns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/name: "CoreDNS"
spec:
  replicas: 2
  # replicas: not specified here:
  # 1. In order to make Addon Manager do not reconcile this replicas parameter.
  # 2. Default is 1.
  # 3. Will be tuned in real time if DNS horizontal auto-scaling is turned on.
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  selector:
    matchLabels:
      k8s-app: kube-dns
  template:
    metadata:
      labels:
        k8s-app: kube-dns
      annotations:
        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
    spec:
      priorityClassName: system-cluster-critical
      serviceAccountName: coredns
      tolerations:
        - key: "CriticalAddonsOnly"
          operator: "Exists"
      nodeSelector:
        beta.kubernetes.io/os: linux
      containers:
      - name: coredns
        image: harbor.k8s.com/k8s-base/coredns:1.4.0 
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            memory: 170Mi
          requests:
            cpu: 100m
            memory: 70Mi
        args: [ "-conf", "/etc/coredns/Corefile" ]
        volumeMounts:
        - name: host-time
          mountPath: /etc/localtime
          readOnly: true
        - name: config-volume
          mountPath: /etc/coredns
          readOnly: true
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        - containerPort: 9153
          name: metrics
          protocol: TCP
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
            scheme: HTTP
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - all
          readOnlyRootFilesystem: true
      dnsPolicy: Default
      volumes:
        - name: host-time
          hostPath:
            path: /etc/localtime
        - name: config-volume
          configMap:
            name: coredns
            items:
            - key: Corefile
              path: Corefile
---
apiVersion: v1
kind: Service
metadata:
  name: kube-dns
  namespace: kube-system
  annotations:
    prometheus.io/port: "9153"
    prometheus.io/scrape: "true"
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/name: "CoreDNS"
spec:
  selector:
    k8s-app: kube-dns
  clusterIP: 10.254.0.2
  ports:
  - name: dns
    port: 53
    protocol: UDP
  - name: dns-tcp
    port: 53
    protocol: TCP
  - name: metrics
    port: 9153
    protocol: TCP

2. 创建coredns

kubectl create -f ./coredns.yaml

#需要注意镜像，我已经把镜像放入带harbor中，在实际操作中根据自己实际情况而定

#查看pod

kubectl get pod -n kube-system  -l k8s-app=kube-dns -o wide  
NAME                       READY   STATUS    RESTARTS   AGE   IP             NODE           NOMINATED NODE   READINESS GATES
coredns-5477df9596-9pvw2   1/1     Running   0          35m   172.30.192.2   k8s-node07     <none>           <none>
coredns-5477df9596-f98z4   1/1     Running   0          35m   172.30.224.2   k8s-master01   <none>           <none>

3. 测试dns功能 

我使用busybox来测试下dns是否正常，不建议使用高版本的。可能nslookup有问题。活生生的例子，折腾了半天。

#创建yaml

cat busybox.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - name: busybox
    image: harbor.k8s.com/k8s-base/busybox:1.28.3
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Alway

kubectl create -f busybox.yaml   

#查看下pod
kubectl get pod
NAME             READY   STATUS    RESTARTS   AGE
busybox          1/1     Running   0          27m




##使用nslookup查看

kubectl exec -ti busybox -- nslookup kubernetes
Server:    10.254.0.2
Address 1: 10.254.0.2 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes
Address 1: 10.254.0.1 kubernetes.default.svc.cluster.local

十五. 后记

总算是写完了，前后写了有一个月的时间，本打算一到两个星期完事儿。一来工作安排的很满没有过多的时间来写。二来为保证文档质量，做一点写一点。虽然耽误些时间，但是总体来说按照这个文档能把k8s搭建起来，这就没有枉费一个多月的努力。

本文现阶段只写了k8s基本框架的搭建，在实际环境中不可能只是这些，还需有一些其他辅助的模块。比如说web界面、ingress、prometheus监控等等，在后续的文章中都会涉及。这是是一个开始，精彩的还在后面，比如说结合jenkins和gitlab实现自动化发布等等。青山不改，绿水长流。以后再见！

# kubernetes # docker # 运维
Logo
K8S/Kubernetes

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐

【深度】阿里巴巴万级规模 K8s 集群全局高可用体系之美

作者 |  韩堂、柘远、沉醉来源 | 阿里巴巴云原生公众号​前言台湾作家林清玄在接受记者采访的时候，如此评价自己 30 多年写作生涯：“第一个十年我才华横溢，‘贼光闪现’，令周边黯然失色；第二个十年，我终于‘宝光现形’，不再去抢风头，反而与身边的美丽相得益彰；进入第三个十年，繁华落尽见真醇，我进入了‘醇光初现’的阶段，真正体味到了境界之美”。​长夜有穷，真水无香。领略过了 K8s“身在江

avatar K8S/Kubernetes

如何基于 K8s 构建下一代 DevOps 平台？

作者 | 孙健波（天元）导读：当前云原生 DevOps 体系现状如何？面临哪些挑战？如何通过 OAM 解决云原生 DevOps 场景下的诸多问题？云原生开发应用模型 OAM(Open Application Model) 社区核心成员孙健波将为大家一一解答，并分享如何基于 OAM 和 Kubernetes 打造无限能力的下一代 DevOps 平台。什么是 DevOps？为什么基于 Kub

avatar K8S/Kubernetes

k8s 火了！

2020，上云之年，产品云端化成为一种趋势。在一线城市，很多公司都已经构建了自己的私有云环境，比如阿里云、网易云、华为云等。而Kubernetes 作为基于容器编排领域的王者，具备扩展...

avatar K8S/Kubernetes