容器和calico的veth pair通讯过程

本文章分析的前提是在calico部署在k8s上，并开启了ipip的功能

root@test-master-113 ~]# calicoctl get ippool -oyaml
- apiVersion: v1
  kind: ipPool
  metadata:
    cidr: 192.168.0.0/16
  spec:
    ipip:
      enabled: true
    nat-outgoing: true

每一个容器通过calico cni-plugin 插件设置默认的路由，都会在指定的network-ns创建veth-pair,位于容器内的veth将会被设置成169.254.1.1

# ip route
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0  scope link

容器内默认路由的mac地址被设置成主机节点上的某个一cali开头的网卡

# arp
Address                  HWtype  HWaddress           Flags Mask            Iface
169.254.1.1              ether   4e:e1:53:f4:e0:06   C                     eth0

# ip neigh
169.254.1.1 dev eth0 lladdr 4e:e1:53:f4:e0:06 REACHABLE

mac地址为4e:e1:53:f4:e0:06

cni-plugin创建了endpoint之后，会将其保存到etcd中，felix从而感知到endpoint的变化。felix会在host端设置一条静态arp:

root@test-slave-114 ~]# ip link show cali34d45bc515f
18704: cali34d45bc515f@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT
    link/ether 4e:e1:53:f4:e0:06 brd ff:ff:ff:ff:ff:ff link-netnsid 14

查看容器的网卡信息

[root@test-master-113 ~]# calicoctl get workloadendpoint --workload=tanyanliao.beegotest-3324938723-l2tkm -oyaml
- apiVersion: v1
  kind: workloadEndpoint
  metadata:
    labels:
      ClusterID: CID-ca4135da3326
      UserID: "102"
      calico/k8s_ns: tanyanliao
      diskType: ""
      name: beegotest
      pod-template-hash: "3324938723"
      tenxcloud.com/appName: beegotest
      tenxcloud.com/svcName: beegotest
    name: eth0
    node: test-slave-114
    orchestrator: k8s
    workload: tanyanliao.beegotest-3324938723-l2tkm
  spec:
    interfaceName: cali34d45bc515f
    ipNetworks:
    - 192.168.134.64/32
    mac: d6:e5:13:88:83:8c
    profiles:
    - k8s_ns.tanyanliao

可以看到这个容器tanyanliao.beegotest-3324938723-l2tkm的网卡名称为cali34d45bc515f,在主机上查看对应的网卡cali34d45bc515f的信息

[root@test-slave-114 ~]# ip link show cali34d45bc515f
18704: cali34d45bc515f@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT
    link/ether 4e:e1:53:f4:e0:06 brd ff:ff:ff:ff:ff:ff link-netnsid 14

可以看到网卡的mac地址是4e:e1:53:f4:e0:06跟在容器内执行arp命令查询出来的mac地址信息是一致的，也就是说容器内的流量不管是流进还是流出都是通过默认的路由流进流出的

容器内的网卡mac地址

# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1
    link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if18704: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether d6:e5:13:88:83:8c brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.134.64/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::d4e5:13ff:fe88:838c/64 scope link
       valid_lft forever preferred_lft forever

以上说明：
cni-plugin插件在容器生成了默认的路由，由于默认的IP169.254.1.1是无效的路由，因此，该mac地址设置成了该容器在node节点上的mac地址4e:e1:53:f4:e0:06(这一步是通过calico felix完成的)，也就是说,容器内的veth mac地址为d6:e5:13:88:83:8c的流量转发是通过默认路由转发到主机生对应的网卡，然后通过ipip隧道走出去的(前提是开了ipip功能的calico)，IPIP隧道是通过tunl0，通讯的

[root@test-slave-114 ~]# ip addr  |grep tun
4: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1440 qdisc noqueue state UNKNOWN qlen 1
    inet 192.168.134.113/32 brd 192.168.134.113 scope global tunl0

参考:
calico的架构设计与组件交互过程