calico分配网络使k8s节点指定固定网段

需求：测试环境的k8s的pod网络需要和阿里云zk通信（10.188.0.0）（pod的ip注册到zk），这里要固定node节点docker网段，避免当node节点重启后docker的网段会改变，避免对应的路由也需要重新指定网络。

ip地址	指定网段	备注
10.188.28.7	10.188.248.0/22	固定地址段
10.188.28.8	10.188.252.0/22	固定地址段

1.配置calicoctl

1.1 下载calicoctl

# wget https://github.com/projectcalico/calicoctl/releases/download/v3.5.4/calicoctl -O /usr/bin/calicoctl
# chmod  +x /usr/local/bin/calicoctl

1.2 配置calicoctl

# cat /etc/calico/calicoctl.cfg
apiVersion: projectcalico.org/v3
kind: CalicoAPIConfig
metadata:
spec:
  datastoreType: "kubernetes"
  kubeconfig: "/root/.kube/config"

1.3 测试calicoctl

# calicoctl get ippool
NAME                       CIDR              SELECTOR         
default-ipv4-ippool        192.168.0.0/16    all() 

# calicoctl get ippool -o json default-ipv4-ippool
{
  "kind": "IPPool",
  "apiVersion": "projectcalico.org/v3",
  "metadata": {
    "name": "default-ipv4-ippool",
    "uid": "e0b3c2f2-6d2c-4552-95df-cb75506b4e65",
    "resourceVersion": "1613",
    "creationTimestamp": "2021-10-26T11:11:00Z"
  },
  "spec": {
    "cidr": "192.168.0.0/16",
    "ipipMode": "Always",
    "natOutgoing": true,
    "blockSize": 26,
    "nodeSelector": "all()"
  }
}

2.配置ippool

使分配到k8s02上的pod，分配的ip池是10.188.252.0/22，分配到k8s01上的pod，分配的ip池是10.188.248.0/22

# cat 248-ippool.yaml
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  name: 248-ipv4pool
spec:
  blockSize: 26
  cidr: 10.188.248.0/22
  ipipMode: Always
  nodeSelector: calico-network == 'k8s01'
  natOutgoing: true
# cat 252-ippool.yaml
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  name: 252-ipv4pool
spec:
  blockSize: 26
  cidr: 10.188.252.0/22
  ipipMode: Always
  nodeSelector: calico-network == 'k8s02'
  natOutgoing: true

下一步给node节点打上标签

# kubectl  label nodes bj-smzy-ce-k8s01-28-7 calico-network=k8s01
# kubectl  label nodes bj-smzy-ce-k8s02-28-8 calico-network=k8s02
# kubectl get nodes --show-labels #查看label是否添加成功

3.添加ippool

# calicoctl create -f 248-ippool.yaml
# calicoctl create -f 252-ippool.yaml
# calicoctl get ippool
NAME           CIDR              SELECTOR
248-ipv4pool   10.188.248.0/22   calico-network == 'k8s01'
252-ipv4pool   10.188.252.0/22   calico-network == 'k8s02'
default-ipv4-ippool   10.5.0.0/16   all()

删除默认的ippool

# kubectl delete  ippool default-ipv4-ippool

4.创建pod测试ippool网段划分

# cat  nginx-deplay.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 10
  template:
    metadata:
      labels:
        app: nginx
      #annotations: #可以不添加注释（已经把默认ipool删除）
      #  "cni.projectcalico.org/ipv4pools": "[\"248-ipv4pool\",\"252-ipv4pool\"]"
    spec:
      containers:
      - name: nginx
        image: nginx:alpine
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
          
# kubectl   create  -f nginx-deplay
# kubectl   get  pod  -o wide
NAME                                READY   STATUS    RESTARTS   AGE    IP               NODE                    NOMINATED NODE   READINESS GATES
nginx-deployment-7fb7fd49b4-5w982   1/1     Running   0          117m   10.188.252.10    bj-smzy-ce-k8s02-28-8   <none>           <none>
nginx-deployment-7fb7fd49b4-8qqrx   1/1     Running   0          117m   10.188.248.205   bj-smzy-ce-k8s01-28-7   <none>           <none>
nginx-deployment-7fb7fd49b4-hnw76   1/1     Running   0          117m   10.188.248.208   bj-smzy-ce-k8s01-28-7   <none>           <none>
nginx-deployment-7fb7fd49b4-jsq2r   1/1     Running   0          117m   10.188.252.15    bj-smzy-ce-k8s02-28-8   <none>           <none>
nginx-deployment-7fb7fd49b4-kbwqz   1/1     Running   0          117m   10.188.248.206   bj-smzy-ce-k8s01-28-7   <none>           <none>
nginx-deployment-7fb7fd49b4-klbhj   1/1     Running   0          117m   10.188.252.14    bj-smzy-ce-k8s02-28-8   <none>           <none>
nginx-deployment-7fb7fd49b4-mwd7z   1/1     Running   0          117m   10.188.252.12    bj-smzy-ce-k8s02-28-8   <none>           <none>
nginx-deployment-7fb7fd49b4-s7rwx   1/1     Running   0          117m   10.188.252.11    bj-smzy-ce-k8s02-28-8   <none>           <none>
nginx-deployment-7fb7fd49b4-sgr6l   1/1     Running   0          117m   10.188.248.207   bj-smzy-ce-k8s01-28-7   <none>           <none>
nginx-deployment-7fb7fd49b4-zhrfz   1/1     Running   0          117m   10.188.252.13    bj-smzy-ce-k8s02-28-8   <none>           <none>

可以看到创建的pod的ip地址

分配到bj-smzy-ce-k8s01-28-7节点的pod是10.188.248.0/22段

分配到bj-smzy-ce-k8s02-28-8节点的pod是10.188.252.0/22段

注意：当新节点接入集群时，需要再次添加新节点的网段的ippool，不然分配的pod落在新节点上是启动不成功的（无法分配pod的ip）

5.核心交换机上添加静态路由

# ip route 10.188.248.0/22（node的docker网段） 10.188.28.7（node节点ip）
# ip route 10.188.252.0/22（node的docker网段） 10.188.28.8（node节点ip）

查看node节点的路由，如若不通信，还需要手动指定路由

# route add -net 10.180.0.0/16 gw 10.188.28.1

6.测试

在10.180.0.0网段的机器上ping podIP

在pod里ping 10.180.0.0网段的ip