K8s 固定IP
k8s固定IP
K8s 固定IP
介绍
Kube-ipam基于etcd分布式存储实现kubernetes动态IP网络分配管理,确保集群中IP地址的唯一性。
本文是基于calico网络环境来操作的,经过两个不同集群测试,可以正常在Deployment和StatefulSet 中通过注释来固定IP,即使重启IP还是和注释配置中的一样。
安装kube-ipam
检查:
请确保你的kubelet正确的配置了network-plugin、cni-conf-dir 和 cni-bin-dir参数。下面给出一个kubelet的配置示例供你参考:
# cat /etc/systemd/system/kubelet.service
...
ExecStart=/usr/local/bin/kubelet \
...
--network-plugin=cni \
--cni-conf-dir=/etc/cni/net.d \
--cni-bin-dir=/opt/cni/bin/ \
...
或者
# cat /etc/systemd/system/kubelet.service.d/10-kubelet.conf
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig"
Environment="KUBELET_SYSTEM_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_CONFIG_ARGS=--config=/etc/kubernetes/kubelet-conf.yml --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.5"
Environment="KUBELET_EXTRA_ARGS=--node-labels=node.kubernetes.io/node='' "
ExecStart=
ExecStart=/usr/local/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_SYSTEM_ARGS $KUBELET_EXTRA_ARGS
总之在kubelet启动的时候要带有 --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin
下载安装kube-ipam
wget https://github.com/cloudnativer/kube-ipam/releases/download/v0.2.0/kube-ipam-v0.2.0-x86.tgz
tar -zxvf kube-ipam-v0.2.0-x86.tgz
mv kube-ipam/kube-ipam /opt/cni/bin/kube-ipam
然后复制kube-ipam到所有节点的 /opt/cni/bin/ 这个目录下并且该kube-ipam要有执行权限,如果没有权限可执行:
chmod 755 /opt/cni/bin/kube-ipam
ls -l /opt/cni/bin/kube-ipam
#也可以用ansible一步到位
ansible masters:nodes -m copy -a "src=/opt/cni/bin/kube-ipam dest=/opt/cni/bin/kube-ipam mode=755"
分发etcd证书密钥和admin.kubeconfig
我用ansible 批量分发到所有节点
ansible masters:nodes -m copy -a "src=/etc/kubernetes/admin.kubeconfig dest=/etc/kubernetes/admin.kubeconfig"
ansible masters:nodes -m copy -a "src=/etc/kubernetes/pki/etcd dest=/etc/kubernetes/pki/"
注意admin.kubeconfig 即是所在k8s集群的config文件,用来管理k8s的证书,通常是 /root/.kube/config 文件
安装或者更新calico cni
更新修改calico.yaml文件,然后使用该calico.yaml在k8s中安装更新
截图如下
配置参数说明
type
(string, required): 填写CNI插件的类型, 例如 macvlan、ipvlan、kube-router、bridge、calico等(还可以与Multus
结合支持更多CNI插件)。routes
(string, optional): 要添加到容器命名空间的路由列表。 每个路由都是一个带有“dst”和可选“gw”字段。 如果省略“gw”,将使用“网关”的值。resolvConf
(string, optional): 主机上要解析并作为 DNS 配置返回的resolv.conf
文件路径。etcdConfig
etcd 地址信息的对象etcdURL
(string, required): etcd的endpoint URL地址。etcdCertFile
(string, required): etcd的cert文件。etcdKeyFile
(string, required): etcd的key文件。etcdTrustedCAFileFile
(string, required): etcd的ca文件。kubeConfig
(string, required): kubernetes集群的kubeconfig文件。
- ranges
subnet
(string, required): 要分配出去的 CIDR 块。rangeStart
(string, optional): 从subnet
子网内开始分配的IP地址,默认为subnet
子网段内的“.2”这个IP地址。rangeEnd
(string, optional): 从subnet
子网内结束分配的IP地址,默认为subnet
子网段内的“.254”这个IP地址。gateway
(string, optional): 从subnet
子网内分配的网关IP地址,默认为subnet
子网段内的“.1”这个IP地址。
配置如下:
"ipam": {
"name": "kube-subnet",
"type": "kube-ipam",
"kubeConfig": "/etc/kubernetes/admin.kubeconfig",
"etcdConfig": {
"etcdURL": "https://10.2.0.21:2379,https://10.2.0.22:2379,https://10.2.0.23:2379",
"etcdCertFile": "/etc/kubernetes/pki/etcd/etcd.pem",
"etcdKeyFile": "/etc/kubernetes/pki/etcd/etcd-key.pem",
"etcdTrustedCAFileFile": "/etc/kubernetes/pki/etcd/etcd-ca.pem"
},
"subnet": "172.20.0.0/16",
"fixedStart": "172.20.0.10",
"fixedEnd": "172.20.0.255",
"rangeStart": "172.20.1.0",
"rangeEnd": "172.20.255.254",
"gateway": "172.20.0.1",
"routes": [{
"dst": "0.0.0.0/0"
}],
"resolvConf": "/etc/resolv.conf"
},
kubeConfig 用的是管理集群的 admin.kubeconfig ,可以放置在/root/.kube/ 下面并改名config,然后随便执行一条命令,测试是否可用。
etcdURL 是连接etcd集群的地址,在这里我的是: https://10.2.0.21:2379,https://10.2.0.22:2379,https://10.2.0.23:2379,这3个集群。
etcdCertFile 对应的是etcd的cert文件,etcdKeyFile对应的是etcd的key文件,etcdTrustedCAFileFile对应的是etcd的cacert文件, 如果我们不确认文件是否正确,那我们可以用命令来尝试连接一些etcd集群看看是否成功,命令如下
etcdctl --endpoints="10.2.0.21:2379,10.2.0.22:2379,10.2.0.23:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem endpoint status --write-out=table
subnet 对应的是K8s Pod网段:172.20.0.0/16,要和calico.yaml中CALICO_IPV4POOL_CIDR的配置一摸一样
然后执行calico.yaml
kubectl apply -f calico.yaml
kubectl get po -nkube-system
等待所有calico服务都正常后,我们进行最后的测试,分别进行Deployment和StatefulSet 重启测试
# cat fixed-ip-test-Deployment.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: fixed-ip-test
namespace: default
labels:
k8s-app: cloudnativer-test
spec:
replicas: 1
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
selector:
matchLabels:
k8s-app: cloudnativer-test
template:
metadata:
labels:
k8s-app: cloudnativer-test
annotations:
kube-ipam.ip: "172.20.166.178"
kube-ipam.netmask: "255.255.0.0"
kube-ipam.gateway: "172.20.0.1"
spec:
nodeName: node1
containers:
- name: fixed-ip-test
image: nginx:1.7.9
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 80
---
# cat fixed-ip-test-Statefulset.yaml
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: fixed-ip-test-statefulset
namespace: default
labels:
k8s-app: cloudnativer-test
spec:
replicas: 1
selector:
matchLabels:
k8s-app: cloudnativer-test
serviceName: ''
template:
metadata:
labels:
k8s-app: cloudnativer-test
annotations:
kube-ipam.ip: "172.20.166.179"
kube-ipam.netmask: "255.255.0.0"
kube-ipam.gateway: "172.20.0.1"
spec:
nodeName: node1
containers:
- name: fixed-ip-test-statefulset
image: nginx:1.7.9
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 80
---
完成✅
参考:
https://github.com/cloudnativer/kube-ipam/blob/main/README0.2-zh.md
更多推荐
所有评论(0)