用multus模拟k8s pod的多块网卡

用multus给k8s的pod模拟多块网卡。

ioriogami

1100人浏览 · 2019-04-29 19:56:52

ioriogami · 2019-04-29 19:56:52 发布

Multiple Network Interfaces for a k8s pod

Introduction

Install multus binary on a VM

Introduction

In some cases we may need more than one network interface for a single k8s pod. One example is that if we'd like to make an Oracle RAC database into k8s cluster, one pod per instance, then we need at least two network interfaces for its public and private IP. This page describes how we can get this done by leveraging a 3rd party CNI (Container Network Interface) plug-in called "multus".

In short, multus is a CNI plug-in that takes over the k8s network and delegates the work to other existing CNI plug-ins such as flannel, wave-net, macvlan .etc, and thus makes applying multiple network CNIs possible. It is developed by Intel.

Install multus binary on a VM

First let's clone multus-cni repository (change "io" to your UID ):

bash-4.2$ cd /home/io/

bash-4.2$ mkdir intel

bash-4.2$ cd intel

bash-4.2$ mkdir multus-cni

bash-4.2$ cd multus-cni

bash-4.2$ git clone https://github.com/intel/multus-cni.git .

To install multus binary we need to install go, we can download it from google official site, e.g.

bash-4.2$ mkdir go

bash-4.2$ cd go

bash-4.2$ pwd

/home/io/go

bash-4.2$ su -

[root@myhost ~]# export http_proxy=...

[root@myhost ~]# export https_proxy=...

[root@myhost ~]# cd /home/io/go

[root@myhost go]# wget https://dl.google.com/go/go1.11.1.linux-amd64.tar.gz

[root@myhost go]# tar -C /usr/local -xzf go1.11.1.linux-amd64.tar.gz

[root@myhost go]# export PATH=$PATH:/usr/local/go/bin

[root@myhost go]# which go

/usr/local/go/bin/go

Now build our multus-cni plug-in:

[root@myhost go]# cd /home/io/intel/multus-cni

[root@myhost multus-cni]# ./build

Then copy the output binary to /opt/cni/bin/

[root@myhost multus-cni]# cd bin

[root@myhost bin]# ls

multus

[root@myhost bin]# cp multus /opt/cni/bin/

Configure multus

First create multus cluster role and cluster role binding (change "myhost" to your host name)

bash-4.2$ cd /home/io/intel/multus-cni/examples

bash-4.2$ kubectl apply -f clusterrole.yml

bash-4.2$ kubectl create clusterrolebinding myhost --clusterrole=multus-crd-overpowered --user=system:node:myhost

Delete any existing CNI config under directory /etc/cni/net.d/, e.g.

[root@myhost intel]# cd /etc/cni/net.d

[root@myhost net.d]# mv 10-flannel.conf /home/io/intel/

Then apply the multus DaemonSet:

bash-4.2$ cd /home/io/intel/multus-cni/examples

bash-4.2$ kubectl apply -f multus-with-flannel.yml

After that there should be a new file generated in /etc/cni/net.d:

bash-4.2$ ls /etc/cni/net.d/

10-multus-with-flannel.conf

Now create the CRD (Custom Resource Definition) and the flannel/macvlan instances

bash-4.2$ kubectl apply -f crd.yml

bash-4.2$ kubectl create -f flannel-conf.yml

bash-4.2$ kubectl create -f macvlan-conf.yml

At this point we probably have two flannel daemon processes running on our k8s node:

bash-4.2$ ps -ef | grep flanneld

root 69153 69135 0 Oct28 ? 00:00:06 /opt/bin/flanneld --ip-masq --kube-subnet-mgr

io 103613 10149 0 02:05 pts/0 00:00:00 grep flanneld

This is because our k8s installation script had already started a flannel DaemonSet as our default network CNI plugin, and our multus DaemonSet launched a second one. We can delete the original flannel DaemonSet (though I don't see any problem with both of them running):

bash-4.2$ kubectl delete daemonset kube-flannel-ds -n kube-system

Try it out

Now let's create a simple pod (still under the dir /home/io/intel/multus-cni/examples):

bash-4.2$ kubectl create -f sample-pod.yml

Wait a while for the image to be pulled and the pod to be created.

As in sample-pod.yml we specifies the macvlan as its extra network interface (flannel is the default one), the pod should have two IP addresses:

bash-4.2$ kubectl exec -it samplepod -- ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

valid_lft forever preferred_lft forever

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

3: eth0@if629: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP

link/ether 0a:58:0a:f4:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0

inet 10.244.0.2/24 scope global eth0

valid_lft forever preferred_lft forever

inet6 fe80::8cbd:5aff:fee1:442/64 scope link

valid_lft forever preferred_lft forever

4: net1@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN

link/ether 0a:58:c0:a8:01:c8 brd ff:ff:ff:ff:ff:ff link-netnsid 0

inet 192.168.1.200/24 scope global net1

valid_lft forever preferred_lft forever

inet6 fe80::858:c0ff:fea8:1c8/64 scope link

valid_lft forever preferred_lft forever

Create a file sample-pod-flannel.yml with following content (the only difference between it and sample-pod.yml is that we commented out the networks annotation) and apply it:

apiVersion: v1

kind: Pod

metadata:

name: samplepod2

annotations:

# k8s.v1.cni.cncf.io/networks: macvlan-conf

spec:

containers:

- name: samplepod2

command: ["/bin/bash", "-c", "sleep 2000000000000"]

image: dougbtv/centos-network

ports:

- containerPort: 80

Wait a while for the pod to create, and we will see that it has only the default flannel network address:

bash-4.2$ kubectl exec -it samplepod2 -- ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

valid_lft forever preferred_lft forever

3: eth0@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP

link/ether 0a:58:0a:f4:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0

inet 10.244.0.3/24 scope global eth0

valid_lft forever preferred_lft forever

Form samplepod2 we can ping samplepod by 10.244.0.2, but not 192.168.1.200, because samplepod2 is not in the macvlan network.

If we update samplepod2 adding back the annotation "k8s.v1.cni.cncf.io/networks: macvlan-conf", it will be able to ping samplepod by both of the IP addresses.