一、生成Flannel网络TLS证书

在所有集群节点都安装Flannel，下面的操作在k8s-master1上进行，其他节点重复执行即可。（证书生成一次就行）

1、创建证书签名请求

cat > /tmp/certs/flanneld-csr.json <<EOF
{
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "O": "k8s",
      "OU": "k8s Security",
      "L": "ChengDU",
      "ST": "SiChuan",
      "C": "CN"
    }
  ],
  "CN": "flanneld",
  "hosts": []
}
EOF

该证书只会被 kubectl 当做 client 证书使用，所以 hosts 字段为空；
生成证书和私钥：

cfssl gencert -ca=/tmp/certs/ca.pem \
  -ca-key=/tmp/certs/ca-key.pem \
  -config=/tmp/certs/gencert.json \
  -profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld
  
# verify
openssl x509 -in /tmp/certs/flanneld.pem -text -noout

2、将证书分发到所有集群节点/etc/kubernetes/cert/目录下


ssh-keygen
ssh-copy-id -i /root/.ssh/id_rsa.pub root@k8s-master1
ssh-copy-id -i /root/.ssh/id_rsa.pub root@k8s-master2
ssh-copy-id -i /root/.ssh/id_rsa.pub root@k8s-master3
ssh-copy-id -i /root/.ssh/id_rsa.pub root@node1
ssh-copy-id -i /root/.ssh/id_rsa.pub root@node2



ssh k8s-master1 'mkdir -pv /etc/kubernetes/cert/'
ssh k8s-master2 'mkdir -pv /etc/kubernetes/cert/'
ssh k8s-master3 'mkdir -pv /etc/kubernetes/cert/'
ssh node1 'mkdir -pv /etc/kubernetes/cert/'
ssh node2 'mkdir -pv /etc/kubernetes/cert/'

mkdir -pv /etc/kubernetes/cert/

scp flanneld*.pem k8s-master1:/etc/kubernetes/cert/
scp flanneld*.pem k8s-master2:/etc/kubernetes/cert/
scp flanneld*.pem k8s-master3:/etc/kubernetes/cert/
scp flanneld*.pem node1:/etc/kubernetes/cert/
scp flanneld*.pem node2:/etc/kubernetes/cert/

scp ca.pem k8s-master1:/etc/kubernetes/cert/
scp ca.pem k8s-master2:/etc/kubernetes/cert/
scp ca.pem k8s-master3:/etc/kubernetes/cert/
scp ca.pem node1:/etc/kubernetes/cert/
scp ca.pem node2:/etc/kubernetes/cert/

二、部署 Flannel

1、下载安装Flannel

wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz

scp flannel-v0.11.0-linux-amd64.tar.gz k8s-master1:/root/
scp flannel-v0.11.0-linux-amd64.tar.gz k8s-master2:/root/
scp flannel-v0.11.0-linux-amd64.tar.gz k8s-master3:/root/
scp flannel-v0.11.0-linux-amd64.tar.gz node1:/root/
scp flannel-v0.11.0-linux-amd64.tar.gz node2:/root/

mkdir -pv /usr/local/flannel/
tar -xzvf flannel-v0.11.0-linux-amd64.tar.gz -C /usr/local/flannel/
cd /usr/local/flannel/
cp -av {flanneld,mk-docker-opts.sh} /usr/local/bin/
2、向 etcd 写入网段信息

下面2条命令在etcd集群中任意一台执行一次即可，也是是创建一个flannel网段供docker分配使用

etcdctl --endpoints https://10.0.0.11:2379,https://10.0.0.11:22379,https://10.0.0.11:32379  \
--ca-file=/etc/k8s/ssl/ca.pem \
--cert-file=/etc/k8s/ssl/etcd.pem \
--key-file=/etc/k8s/ssl/etcd-key.pem \
mkdir /kubernetes/network


etcdctl --endpoints https://10.0.0.11:2379,https://10.0.0.11:22379,https://10.0.0.11:32379  \
--ca-file=/etc/k8s/ssl/ca.pem \
--cert-file=/etc/k8s/ssl/etcd.pem \
--key-file=/etc/k8s/ssl/etcd-key.pem \
mk /kubernetes/network/config '{"Network":"172.30.0.0/16","SubnetLen":24,"Backend":{"Type":"vxlan"}}'


3、创建system unit文件
cat > /etc/systemd/system/flanneld.service << EOF
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service

[Service]
Type=notify
ExecStart=/usr/local/bin/flanneld \
  -etcd-cafile=/etc/kubernetes/cert/ca.pem \
  -etcd-certfile=/etc/kubernetes/cert/flanneld.pem \
  -etcd-keyfile=/etc/kubernetes/cert/flanneld-key.pem \
  -etcd-endpoints=https://10.0.0.11:2379,https://10.0.0.11:2379,https://10.0.0.11:2379 \
  -etcd-prefix=/kubernetes/network
ExecStartPost=/usr/local/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=on-failure

[Install]
WantedBy=multi-user.target
RequiredBy=docker.service
EOF

mk-docker-opts.sh 脚本将分配给 flanneld 的 Pod 子网网段信息写入到 /run/flannel/docker 文件中，后续 docker 启动时使用这个文件中参数值设置 docker0 网桥。
flanneld 使用系统缺省路由所在的接口和其它节点通信，对于有多个网络接口的机器（如，内网和公网），可以用 -iface=enpxx 选项值指定通信接口。

4、启动flannel并且设置开机自启动
systemctl daemon-reload
systemctl enable flanneld
systemctl start flanneld

5、查看flannel分配的子网信息
# cat /run/flannel/docker 
DOCKER_OPT_BIP="--bip=172.30.18.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=true"
DOCKER_OPT_MTU="--mtu=1450"
DOCKER_NETWORK_OPTIONS=" --bip=172.30.18.1/24 --ip-masq=true --mtu=1450"

# cat /run/flannel/subnet.env
FLANNEL_NETWORK=172.30.0.0/16
FLANNEL_SUBNET=172.30.18.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=false

/run/flannel/docker是flannel分配给docker的子网信息，/run/flannel/subnet.env包含了flannel整个大网段以及在此节点上的子网段。

6、查看flannel网络是否生效
# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.0.21  netmask 255.255.255.0  broadcast 10.0.0.255
        inet6 fe80::250:56ff:feae:fe2e  prefixlen 64  scopeid 0x20<link>
        ether 00:50:56:ae:fe:2e  txqueuelen 1000  (Ethernet)
        RX packets 3797  bytes 20455365 (19.5 MiB)
        RX errors 0  dropped 51  overruns 0  frame 0
        TX packets 3535  bytes 329140 (321.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 172.30.18.0  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::909d:e1ff:fed2:b6ab  prefixlen 64  scopeid 0x20<link>
        ether 92:9d:e1:d2:b6:ab  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 8 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 501  bytes 25496 (24.8 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 501  bytes 25496 (24.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

三、配置docker支持flannel网络

1、所有node安装docker
关于安装docker，请参考：安装指定版本的docker
安装docker-ce：
安装必要的一些系统工具
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
添加软件源信息
sudo yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
更新并安装 Docker-CE
sudo yum makecache fast
sudo yum -y install docker-ce

配置docker
# mkdir /etc/docker/

创建docker配置文件：
# cat > /etc/docker/daemon.json <<EOF 
{
"registry-mirrors": ["https://fgl80ig9.mirror.aliyuncs.com","http://04be47cf.m.daocloud.io"],
"exec-opts": ["native.cgroupdriver=systemd"]
}
EOF
启动docker服务：
systemctl enable docker && systemctl start docker

2、配置docker支持flannel网络，所有docker节点都操作
vim /usr/lib/systemd/system/docker.service 
EnvironmentFile=/run/flannel/docker
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS

3、重启docker，使配置生效

systemctl daemon-reload
systemctl restart docker
systemctl enable docker && systemctl status docker

4、查看docker网络是否生效

启动一个容器（如有现有容器可以不run一个新的）
# docker run -itd centos
Unable to find image 'centos:latest' locally
latest: Pulling from library/centos
8ba884070f61: Pull complete 
Digest: sha256:b5e66c4651870a1ad435cd75922fe2cb943c9e973a9673822d1414824a1d0475
Status: Downloaded newer image for centos:latest
dec1a01acba00ba92d871e9b79f0e3d2ad8d486d884bdc554c3e761f92229c41


查看ip地址是否是flannel网络分配的网段
# docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' dec
172.30.18.2

5、查看所有集群主机的网络情况
# etcdctl --endpoints https://10.0.0.11:2379,https://10.0.0.11:22379,https://10.0.0.11:32379  \
--ca-file=/etc/k8s/ssl/ca.pem \--cert-file=/etc/k8s/ssl/etcd.pem \
--key-file=/etc/k8s/ssl/etcd-key.pem \
ls /kubernetes/network/subnets

/kubernetes/network/subnets/172.30.57.0-24
/kubernetes/network/subnets/172.30.18.0-24
/kubernetes/network/subnets/172.30.28.0-24
/kubernetes/network/subnets/172.30.74.0-24
/kubernetes/network/subnets/172.30.71.0-24