K8s 中 iptables 和ipvs 的理解
Iptables定义service 类型为 ClusterIPkubectl get svc -owidenginx-basicClusterIP10.101.157.173<none>80/TCP42happ=nginxkubectl get pods -owidenginx-deployment-6799fc88d8-dmhk91/1Running027m10.224.0.11
·
Iptables
定义service 类型为 ClusterIP
kubectl get svc -owide
nginx-basic ClusterIP 10.101.157.173 <none> 80/TCP 42h app=nginx
kubectl get pods -owide
nginx-deployment-6799fc88d8-dmhk9 1/1 Running 0 27m 10.224.0.11 node1 <none> <none>
nginx-deployment-6799fc88d8-m6vsc 1/1 Running 0 27m 10.224.0.9 node1 <none> <none>
nginx-deployment-6799fc88d8-mmgxl 1/1 Running 0 27m 10.224.0.10 node1 <none> <none>
# 使用iptables-save 查看路由规则
iptables-save -t nat
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A add -m message -j jump 所有进来的包在出prerouting 这个包都要去kube-service 中去看看
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A add -m message -j jump 所有出来的包在出prerouting 这个包都要去kube-service 中去看看
-A KUBE-SERVICES -d 10.101.157.173/32 -p tcp -m comment --comment "default/nginx-basic:http cluster IP" -m tcp --dport 80 -j KUBE-SVC-WWRFY3PZ7W3FGMQW
-d destination(目标)
iptable 规则说明一条一套执行 有百分之33的几率被KUBE-SEP-ZMFA63VDZWTWAEBQ 这个chain 处理
-A KUBE-SVC-WWRFY3PZ7W3FGMQW -m comment --comment "default/nginx-basic:http" -m statistic --mode random --probability 0.33333333349 -j KUBE-SEP-ZMFA63VDZWTWAEBQ
-A KUBE-SEP-ZMFA63VDZWTWAEBQ -s 10.224.0.10/32 -m comment --comment "default/nginx-basic:http" -j KUBE-MARK-MASQ
-A KUBE-SEP-ZMFA63VDZWTWAEBQ -p tcp -m comment --comment "default/nginx-basic:http" -m tcp -j DNAT --to-destination :0 --persistent --to-destination :0 --persistent --to-destination
-A KUBE-SVC-WWRFY3PZ7W3FGMQW -m comment --comment "default/nginx-basic:http" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-B7KA7M3EBT33ZNQX
-A KUBE-SEP-B7KA7M3EBT33ZNQX -s 10.224.0.11/32 -m comment --comment "default/nginx-basic:http" -j KUBE-MARK-MASQ
-A KUBE-SEP-B7KA7M3EBT33ZNQX -p tcp -m comment --comment "default/nginx-basic:http" -m tcp -j DNAT --to-destination :0 --persistent --to-destination :0 --persistent --to-destination
-A KUBE-SVC-WWRFY3PZ7W3FGMQW -m comment --comment "default/nginx-basic:http" -j KUBE-SEP-YUJMBOAQBKLEZT3R
-A KUBE-SEP-YUJMBOAQBKLEZT3R -s 10.224.0.9/32 -m comment --comment "default/nginx-basic:http" -j KUBE-MARK-MASQ
-A KUBE-SEP-YUJMBOAQBKLEZT3R -p tcp -m comment --comment "default/nginx-basic:http" -m tcp -j DNAT --to-destination :0 --persistent --to-destination :0 --persistent --to-destination
最终结论为:从上可以看出k8s 通过service 的流量会路由到pod 中去
定义service 类型为 NodePort
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nginx-basic:http" -m tcp --dport 31219 -j KUBE-SVC-WWRFY3PZ7W3FGMQW
netstat -na|grep 31219
tcp 0 0 0.0.0.0:31219 0.0.0.0:* LISTEN
netstat -nap|grep 31219
tcp 0 0 0.0.0.0:31219 0.0.0.0:* LISTEN 52480/kube-proxy
查看-j KUBE-SVC-WWRFY3PZ7W3FGMQW 与上图service类型为ClusterIP一致 也是往三个Pod 中转发流量
ipvs
IPVS
修改kube-proxy 的configmap中修改对应的mode 然后改为ipvs
ipvsadm -l
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP node1:31219 rr
-> 10.224.0.9:http Masq 1 0 0
-> 10.224.0.10:http Masq 1 0 0
-> 10.224.0.11:http Masq 1 0 0
TCP node1:31219 rr
-> 10.224.0.9:http Masq 1 0 0
-> 10.224.0.10:http Masq 1 0 0
-> 10.224.0.11:http Masq 1 0 0
TCP node1:https rr
-> node1:sun-sr-https Masq 1 4 0
TCP node1:domain rr
-> 10.224.0.7:domain Masq 1 0 0
-> 10.224.0.8:domain Masq 1 0 0
TCP node1:9153 rr
-> 10.224.0.7:9153 Masq 1 0 0
-> 10.224.0.8:9153 Masq 1 0 0
TCP node1:http rr
-> 10.224.0.9:http Masq 1 0 0
-> 10.224.0.10:http Masq 1 0 0
-> 10.224.0.11:http Masq 1 0 0
TCP node1:31219 rr
-> 10.224.0.9:http Masq 1 0 0
-> 10.224.0.10:http Masq 1 0 0
-> 10.224.0.11:http Masq 1 0 0
TCP node1:31219 rr
-> 10.224.0.9:http Masq 1 0 0
-> 10.224.0.10:http Masq 1 0 0
-> 10.224.0.11:http Masq 1 0 0
UDP node1:domain rr
-> 10.224.0.7:domain Masq 1 0 0
-> 10.224.0.8:domain Masq 1 0 0
使用iptables -F -t nat
然后再次查看下面的规则 会发现多了一条IPSET 现在使用ipvs 的话会做一个 IP 伪装 KUBE-MARK-MASQ
iptables-save -t nat
# Generated by iptables-save v1.4.21 on Wed Feb 16 13:31:44 2022
*nat
:PREROUTING ACCEPT [32:1440]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [47:2820]
:POSTROUTING ACCEPT [47:2820]
:DOCKER - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-LOAD-BALANCER - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODE-PORT - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-PROXY-CANARY - [0:0]
:KUBE-SEP-B7KA7M3EBT33ZNQX - [0:0]
:KUBE-SEP-DWSBRBKMJKQCC6K6 - [0:0]
:KUBE-SEP-H7GRETGDBNJH3TNF - [0:0]
:KUBE-SEP-M7D4TRPAWXXY3RIC - [0:0]
:KUBE-SEP-OZ2EEVAFCTBCWHHW - [0:0]
:KUBE-SEP-P6X6TGMQCZ3YXVTJ - [0:0]
:KUBE-SEP-TGTH74NZUASQJRDW - [0:0]
:KUBE-SEP-WQDGZCN3UF6UPQZM - [0:0]
:KUBE-SEP-YUJMBOAQBKLEZT3R - [0:0]
:KUBE-SEP-ZMFA63VDZWTWAEBQ - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-ERIFXISQEP7F7OF4 - [0:0]
:KUBE-SVC-JD5MR3NA4I4DYORP - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
:KUBE-SVC-TCOU7JCQXEZGVUNU - [0:0]
:KUBE-SVC-WWRFY3PZ7W3FGMQW - [0:0]
:cali-OUTPUT - [0:0]
:cali-POSTROUTING - [0:0]
:cali-PREROUTING - [0:0]
:cali-fip-dnat - [0:0]
:cali-fip-snat - [0:0]
:cali-nat-outgoing - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A KUBE-FIREWALL -j KUBE-MARK-DROP
-A KUBE-LOAD-BALANCER -j KUBE-MARK-MASQ
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-NODE-PORT -p tcp -m comment --comment "Kubernetes nodeport TCP port for masquerade purpose" -m set --match-set KUBE-NODE-PORT-TCP dst -j KUBE-MARK-MASQ
-A KUBE-POSTROUTING -m comment --comment "Kubernetes endpoints dst ip:port, source ip for solving hairpin purpose" -m set --match-set KUBE-LOOP-BACK dst,dst,src -j MASQUERADE
-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE
-A KUBE-SERVICES ! -s 10.244.0.0/16 -m comment --comment "Kubernetes service cluster ip + port for masquerade purpose" -m set --match-set KUBE-CLUSTER-IP dst,dst -j KUBE-MARK-MASQ
-A KUBE-SERVICES -m addrtype --dst-type LOCAL -j KUBE-NODE-PORT
-A KUBE-SERVICES -m set --match-set KUBE-CLUSTER-IP dst,dst -j ACCEPT
COMMIT
更多推荐
已为社区贡献1条内容
所有评论(0)