kubernetes-flannel 网络组件CNI配置过程

测试环境：k8s v1.10.5CNI Plugin : 0.6.0flannel: 0.11.0flannel cni 配置过程：Container Network Interface (CNI) 最早是由CoreOS发起的容器网络规范，是Kubernetes网络插件的基础。其基本思想为：Container Runtime在创建容器时，先创建好network namespace...

大飞哥2

10233人浏览 · 2019-03-14 08:54:04

大飞哥2 · 2019-03-14 08:54:04 发布

测试环境：

k8s v1.10.5
CNI Plugin : 0.6.0
flannel: 0.11.0

flannel cni 配置过程：

Container Network Interface (CNI) 最早是由CoreOS发起的容器网络规范，是Kubernetes网络插件的基础。其基本思想为：Container Runtime在创建容器时，先创建好network namespace，然后调用CNI插件为这个netns配置网络，其后再启动容器内的进程。现已加入CNCF，成为CNCF主推的网络模型。

CNI插件包括两部分：

1. CNI Plugin负责给容器配置网络，它包括两个基本的接口
1.1 配置网络: AddNetwork(net *NetworkConfig, rt *RuntimeConf) (types.Result, error)
1.2 清理网络: DelNetwork(net *NetworkConfig, rt *RuntimeConf) error
2. IPAM Plugin负责给容器分配IP地址，主要实现包括host-local和dhcp。

Kubernetes Pod 中的其他容器都是Pod所属pause容器的网络，创建过程为：
1. kubelet 先创建pause容器生成network namespace
2. 调用网络CNI driver
3. CNI driver 根据配置调用具体的cni 插件
4. cni 插件给pause 容器配置网络
5. pod 中其他的容器都使用 pause 容器的网络

如下：

k8s安装：
参考： https://github.com/opsnull/follow-me-install-kubernetes-cluster

cni配置过程：
1. 删除原有的网络组件，如果你按上面的k8s安装文档配置了flannel,cni方式使用二进制文件安装比较困难，这里采用容器方式重新安装。

2. kube-controller-manager ：
启动参数增加如下两项：

--allocate-node-cidrs=true   #  是否应在云提供商上分配和设置Pod的CIDR。
--cluster-cidr=10.244.0.0/16 #    集群中Pod的CIDR范围。要求--allocate-node-cidrs为true

kube-controller-manager参数详细说明参考：https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/以下是完整配置：

# cat /etc/systemd/system/kube-controller-manager.service 
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
ExecStart=/opt/k8s/bin/kube-controller-manager \
  --port=0 \
  --secure-port=10252 \
  --bind-address=127.0.0.1 \
  --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \
  --service-cluster-ip-range=10.254.0.0/16 \
  --allocate-node-cidrs=true \
  --cluster-cidr=10.244.0.0/16 \
  --cluster-name=kubernetes \
  --cluster-signing-cert-file=/etc/kubernetes/cert/ca.pem \
  --cluster-signing-key-file=/etc/kubernetes/cert/ca-key.pem \
  --experimental-cluster-signing-duration=8760h \
  --root-ca-file=/etc/kubernetes/cert/ca.pem \
  --service-account-private-key-file=/etc/kubernetes/cert/ca-key.pem \
  --leader-elect=true \
  --feature-gates=RotateKubeletServerCertificate=true \
  --controllers=*,bootstrapsigner,tokencleaner \
  --horizontal-pod-autoscaler-use-rest-clients=true \
  --horizontal-pod-autoscaler-sync-period=10s \
  --tls-cert-file=/etc/kubernetes/cert/kube-controller-manager.pem \
  --tls-private-key-file=/etc/kubernetes/cert/kube-controller-manager-key.pem \
  --use-service-account-credentials=true \
  --v=2 \
  --horizontal-pod-autoscaler-use-rest-clients=true
Restart=on
Restart=on-failure
RestartSec=5
User=k8s

[Install]
WantedBy=multi-user.target

3. kubelet
启动参数增加：

--network-plugin=cni # 要为kubelet / pod生命周期中的各种事件调用的网络插件的名称

kubelet参数详细说明参考：https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/以下是完整配置：

# cat /etc/systemd/system/kubelet.service 
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/opt/k8s/bin/kubelet \
  --bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig \
  --cert-dir=/etc/kubernetes/cert \
  --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
  --config=/etc/kubernetes/kubelet.config.json \
  --hostname-override=consul-02 \
  --network-plugin=cni \
  --pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest \
  --logtostderr=true \
  --allow-privileged=true \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

4. docker

启动改为： ExecStart=/usr/bin/dockerd --log-level=error

5. 安装flannel

5.1 下载cni插件：
关于 CNI 的版本可以查询 https://github.com/kubernetes/kubernetes/blob/master/build/debian-hyperkube-base/Makefile
下载：
https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-${CNI_VER}.tgz
或
https://github.com/containernetworking/plugins/releases

解压后，放在各个节点的 /opt/cni/bin 下：

# ls /opt/cni/bin/ -l
总用量 44828
-rwxr-xr-x 1 root root 3890407 3月   5 22:06 bridge
-rwxr-xr-x 1 root root 9921982 3月   5 22:06 dhcp
-rwxr-xr-x 1 root root 2814104 3月   5 22:06 flannel
-rwxr-xr-x 1 root root 2991965 3月   5 22:06 host-local
-rwxr-xr-x 1 root root 3475802 3月   5 22:06 ipvlan
-rwxr-xr-x 1 root root 3026388 3月   5 22:06 loopback
-rwxr-xr-x 1 root root 3520724 3月   5 22:06 macvlan
-rwxr-xr-x 1 root root 3470464 3月   5 22:06 portmap
-rwxr-xr-x 1 root root 3877986 3月   5 22:06 ptp
-rwxr-xr-x 1 root root 2605279 3月   5 22:06 sample
-rwxr-xr-x 1 root root 2808402 3月   5 22:06 tuning
-rwxr-xr-x 1 root root 3475750 3月   5 22:06 vlan

5.2 安装flannel

$ wget https://raw.githubusercontent.com/coreos/flannel/v0.11.0/Documentation/kube-flannel.yml

$ kubectl apply -f ./kube-flannel.yml

如果flannel images下载不回来，可以在这里下载（jmgao1983/flannel:v0.11.0-amd64）

5.3 安装完flannel后，我们启动一个pod,查看IP分配情况：

$ kubectl get po,svc -o wide --all-namespaces=true
NAMESPACE     NAME                                        READY     STATUS    RESTARTS   AGE       IP            NODE
default       po/nginx-deployment-549b8578f9-rvsh6        1/1       Running   0          45m       10.244.0.2    consul-02
kube-system   po/coredns-77c989547b-mck5p                 1/1       Running   7          137d      10.244.0.3    consul-02
kube-system   po/coredns-77c989547b-tdq8f                 1/1       Running   2          21h       10.244.1.2    consul-03
kube-system   po/kube-flannel-ds-amd64-npdgk              1/1       Running   0          46m       192.168.0.6   consul-02
kube-system   po/kube-flannel-ds-amd64-td7zt              1/1       Running   0          46m       192.168.0.7   consul-03
kube-system   po/metrics-server-v0.2.1-7958856dcd-92s5g   2/2       Running   14         137d      10.244.0.4    consul-02

NAMESPACE     NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE       SELECTOR
default       svc/kubernetes       ClusterIP   10.254.0.1       <none>        443/TCP         251d      <none>
default       svc/nginx            NodePort    10.254.125.130   <none>        80:30088/TCP    22h       app=nginx
kube-system   svc/coredns          ClusterIP   10.254.0.2       <none>        53/UDP,53/TCP   245d      k8s-app=coredns
kube-system   svc/metrics-server   ClusterIP   10.254.217.197   <none>        443/TCP         215d      k8s-app=metrics-server

5.4 查看节点的网络：

k8s节点1：

# ifconfig 
cni0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.244.0.1  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::d02c:deff:fe4a:2b09  prefixlen 64  scopeid 0x20<link>
        ether 0a:58:0a:f4:00:01  txqueuelen 1000  (Ethernet)
        RX packets 45431  bytes 3426825 (3.2 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 47263  bytes 90306689 (86.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.30.59.1  netmask 255.255.255.0  broadcast 172.30.59.255
        ether 02:42:31:81:21:f7  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp0s3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.6  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::a00:27ff:fe65:eff7  prefixlen 64  scopeid 0x20<link>
        ether 08:00:27:65:ef:f7  txqueuelen 1000  (Ethernet)
        RX packets 1299683  bytes 374382878 (357.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1039608  bytes 120600259 (115.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.244.0.0  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::800c:13ff:fe42:3e1b  prefixlen 64  scopeid 0x20<link>
        ether 82:0c:13:42:3e:1b  txqueuelen 0  (Ethernet)
        RX packets 14  bytes 906 (906.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 10  bytes 2234 (2.1 KiB)
        TX errors 0  dropped 8 overruns 0  carrier 0  collisions 0
......

k8s节点2：

# ifconfig 
cni0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.244.1.1  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::84ee:1eff:fe99:f991  prefixlen 64  scopeid 0x20<link>
        ether 0a:58:0a:f4:01:01  txqueuelen 1000  (Ethernet)
        RX packets 17363  bytes 1115044 (1.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 17591  bytes 6517620 (6.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.30.72.1  netmask 255.255.255.0  broadcast 172.30.72.255
        ether 02:42:ac:43:1d:8c  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp0s3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.7  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::a00:27ff:fec9:5dc1  prefixlen 64  scopeid 0x20<link>
        ether 08:00:27:c9:5d:c1  txqueuelen 1000  (Ethernet)
        RX packets 2016010  bytes 340498015 (324.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1862182  bytes 372521420 (355.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.244.1.0  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::4832:76ff:fe06:fa2f  prefixlen 64  scopeid 0x20<link>
        ether 4a:32:76:06:fa:2f  txqueuelen 0  (Ethernet)
        RX packets 10  bytes 2234 (2.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 14  bytes 906 (906.0 B)
        TX errors 0  dropped 8 overruns 0  carrier 0  collisions 0
......

参考：
https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/#installation
https://github.com/coreos/flannel
https://github.com/containernetworking/plugins
https://github.com/containernetworking/plugins/releases （cni-plugins插件下载地址）
https://github.com/feiskyer/kubernetes-handbook/blob/master/network/cni/index.md (cni介绍，推荐看)
https://mritd.me/2017/09/20/set-up-ha-kubernetes-cluster-on-aliyun-ecs/

https://jiayi.space/post/kubernetescong-ru-men-dao-fang-qi-3-wang-luo-yuan-li ()

K8S/Kubernetes

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐

【深度】阿里巴巴万级规模 K8s 集群全局高可用体系之美

作者 | 韩堂、柘远、沉醉来源 | 阿里巴巴云原生公众号前言台湾作家林清玄在接受记者采访的时候，如此评价自己 30 多年写作生涯：“第一个十年我才华横溢，‘贼光闪现’，令周边黯然失色；第二个十年，我终于‘宝光现形’，不再去抢风头，反而与身边的美丽相得益彰；进入第三个十年，繁华落尽见真醇，我进入了‘醇光初现’的阶段，真正体味到了境界之美”。长夜有穷，真水无香。领略过了 K8s“身在江

K8S/Kubernetes

如何基于 K8s 构建下一代 DevOps 平台？

作者 | 孙健波（天元）导读：当前云原生 DevOps 体系现状如何？面临哪些挑战？如何通过 OAM 解决云原生 DevOps 场景下的诸多问题？云原生开发应用模型 OAM(Open Application Model) 社区核心成员孙健波将为大家一一解答，并分享如何基于 OAM 和 Kubernetes 打造无限能力的下一代 DevOps 平台。什么是 DevOps？为什么基于 Kub