修改K8S Master节点IP后使用kubeadm join无法添加节点

背景： 目前接了一个需求：修改使用kubeadm部署好的K8S环境所在服务器IP。很明显，修改服务器IP后，kube-apiserver等服务是无法启动的。在使用脚本重新生成了apiserver证书等一系列操作后，将所有的服务恢复后，测试使用kubeadm join添加新的节点一直失败，看日志是kubeadm期间会使用到服务器的旧IP，调试了很久，终于发现使用旧IP的地方，以下是排查过程，记录以下，共后续参考。

一、kubeadm join报错

旧IP： 192.168.12.110
新IP： 192.168.12.224

root@intellif-3:~# kubeadm join 192.168.12.224:6443 --token dperzp.19444xvuwd5lhr2n     --discovery-token-ca-cert-hash sha256:1db95da7507bd5b509eddc7aec24d4921334a175ab5b6b2e5bde66ccc128388a --cri-socket=/run/containerd/containerd.sock
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

error execution phase preflight: unable to fetch the kubeadm-config ConfigMap: failed to get config map: Get https://192.168.12.110:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config: dial tcp 192.168.12.110:6443: i/o timeout

二、打开kubeadm调试信息

• 在需要加入k8s集群的新服务器上执行如下命令：

kubeadm join 192.168.12.224:6443
 --token dperzp.19444xvuwd5lhr2n \
 --discovery-token-ca-cert-hash sha256:1db95da7507bd5b509eddc7aec24d4921334a175ab5b6b2e5bde66ccc128388a  \ 
 --cri-socket=/run/containerd/containerd.sock \
 -v=10

• 输出日志如下：

I0711 15:03:41.339810   32744 join.go:367] [preflight] found NodeName empty; using OS hostname as NodeName
[preflight] Running pre-flight checks
I0711 15:03:41.339943   32744 preflight.go:90] [preflight] Running general checks
I0711 15:03:41.339987   32744 checks.go:254] validating the existence and emptiness of directory /etc/kubernetes/manifests
I0711 15:03:41.340005   32744 checks.go:292] validating the existence of file /etc/kubernetes/kubelet.conf
I0711 15:03:41.340013   32744 checks.go:292] validating the existence of file /etc/kubernetes/bootstrap-kubelet.conf
I0711 15:03:41.340021   32744 checks.go:105] validating the container runtime
I0711 15:03:41.357074   32744 checks.go:382] validating the presence of executable crictl
I0711 15:03:41.357218   32744 checks.go:341] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables
I0711 15:03:41.357295   32744 checks.go:341] validating the contents of file /proc/sys/net/ipv4/ip_forward
I0711 15:03:41.357340   32744 checks.go:653] validating whether swap is enabled or not
I0711 15:03:41.357385   32744 checks.go:382] validating the presence of executable ip
I0711 15:03:41.357423   32744 checks.go:382] validating the presence of executable iptables
I0711 15:03:41.357454   32744 checks.go:382] validating the presence of executable mount
I0711 15:03:41.357485   32744 checks.go:382] validating the presence of executable nsenter
I0711 15:03:41.357512   32744 checks.go:382] validating the presence of executable ebtables
I0711 15:03:41.357542   32744 checks.go:382] validating the presence of executable ethtool
I0711 15:03:41.357571   32744 checks.go:382] validating the presence of executable socat
I0711 15:03:41.357595   32744 checks.go:382] validating the presence of executable tc
I0711 15:03:41.357633   32744 checks.go:382] validating the presence of executable touch
I0711 15:03:41.357669   32744 checks.go:524] running all checks
I0711 15:03:41.375863   32744 checks.go:412] checking whether the given node name is reachable using net.LookupHost
I0711 15:03:41.376138   32744 checks.go:622] validating kubelet version
I0711 15:03:41.469718   32744 checks.go:131] validating if the service is enabled and active
I0711 15:03:41.487644   32744 checks.go:209] validating availability of port 10250
I0711 15:03:41.487877   32744 checks.go:292] validating the existence of file /etc/kubernetes/pki/ca.crt
I0711 15:03:41.487902   32744 checks.go:439] validating if the connectivity type is via proxy or direct
I0711 15:03:41.487956   32744 join.go:427] [preflight] Discovering cluster-info
I0711 15:03:41.488127   32744 token.go:200] [discovery] Trying to connect to API Server "192.168.12.224:6443"
I0711 15:03:41.488996   32744 token.go:75] [discovery] Created cluster-info discovery client, requesting info from "https://192.168.12.224:6443"
I0711 15:03:41.489139   32744 round_trippers.go:419] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubeadm/v1.14.0 (linux/amd64) kubernetes/641856d" 'https://192.168.12.224:6443/api/v1/namespaces/kube-public/configmaps/cluster-info'
I0711 15:03:41.502120   32744 round_trippers.go:438] GET https://192.168.12.224:6443/api/v1/namespaces/kube-public/configmaps/cluster-info 200 OK in 12 milliseconds
I0711 15:03:41.502153   32744 round_trippers.go:444] Response Headers:
I0711 15:03:41.502164   32744 round_trippers.go:447]     Content-Type: application/json
I0711 15:03:41.502174   32744 round_trippers.go:447]     Content-Length: 2331
I0711 15:03:41.502183   32744 round_trippers.go:447]     Date: Thu, 11 Jul 2019 07:03:41 GMT
I0711 15:03:41.502242   32744 request.go:942] Response Body: {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"cluster-info","namespace":"kube-public","selfLink":"/api/v1/namespaces/kube-public/configmaps/cluster-info","uid":"ab35c11a-a133-11e9-9ab6-005056bffc6e","resourceVersion":"355456","creationTimestamp":"2019-07-08T03:51:29Z"},"data":{"jws-kubeconfig-83brsz":"eyJhbGciOiJIUzI1NiIsImtpZCI6IjgzYnJzeiJ9..duFmZ7WgLbFEj8B_dDGrxinP1lu9JCfvOjV5NhtyIg0","jws-kubeconfig-bnavms":"eyJhbGciOiJIUzI1NiIsImtpZCI6ImJuYXZtcyJ9..XXbqwRq7fWlCrnI-zK2_yaKbi9YSXrUOHIeoUrO0iAs","jws-kubeconfig-dperzp":"eyJhbGciOiJIUzI1NiIsImtpZCI6ImRwZXJ6cCJ9..mEZckISMk5j_oOLg_oiROQ7vdcR88rFL_RgUkImdFUs","jws-kubeconfig-xcj9ft":"eyJhbGciOiJIUzI1NiIsImtpZCI6InhjajlmdCJ9..KoNl4BXu9oUXqvl2b4gelCCpKFsRq0vexLCdGjoOVCQ","kubeconfig":"apiVersion: v1\nclusters:\n- cluster:\n    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRFNU1EY3dPREF6TlRFd09Wb1hEVEk1TURjd05UQXpOVEV3T1Zvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBS1FjCmFZVEVMMXZLS2RGcytUZFQ2ZWkzZjNoUzYyTFRZc3MrTFdLMmVKTXJOTllCSXlQOGZvYWE1cXd0anh4KzdxcG4KeFNUQW5OSGpMZnR0b00zQUV6WjE4TTRkVE5xWHU4VDE0bllsamx4MzMxUFZHSDIxS0tidDdyS3ZYU3JSeG8zcwo0dG1EMVpUNUFMaGltK3Z3QnZ5MHZlTDlnWXpJeklIbVpZNmFOamF4YUdUa0VMWWlQOU9IbFpnKzVTVmloMWJ3CjZydTlMWDQrWDJQL2J6QUxGcW45bGtub2RlUFp4TTVKRHFIT0h3aDUyWHc2TlREcU1VTUFrSG4xcTRmU3RZNW4KZ3VsZEw3RVVXVEFxaU5OVnQ3dVpYZk9OeFdEdnRWTmg4ekVJUHRyaWhiWUdHRTFjVUxndFB6UDVqbTN2c1FhRAo2bklZYU9NcmVyNmhPSVFXTlZjQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFHbHc0ZTVLdE1tM1VTZUxNekgyYVcxdVZZZ1gKTWFrQlFnMThJMTloenRIZ0tYUFJQVVNOUE4rMWtRTHdxQ0xwUUwwUm1tWXVHaVhoUTY3eUVlTU1pVXpRcTVqVwpSUUd4Q0FUQlJzV3FRK2lXY0FPSHcxVy9OdWdBdFQxMWxndjUzMVZUREs3cnA0aDgrS1Uway8xY0REWWZycXYrCkRjaGZoYTdUT1NkNW1FQzdQbkpEOUFDaWtQS2w3cHYwZ0ZaelFWUXhCZ0RJN3g0a0JXZjVnN05OclhIa3dMcXcKMmRUaHBNZU9TR2NoM3FjbkU5eW1TNUZoSG1vSHZDN1VFOCtlNUpHMjVkTUlWSXc1LytkL0lscmE4akRZVktoWQpvQXM0dGpWdmtsUnJWRDRwSEVXNDdJZ0FUV2pOMDRuMmE2Y01peFRvRHhtMlM1cXlPU2R6SlkvQ3BJST0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=\n    server: https://192.168.12.110:6443\n  name: \"\"\ncontexts: []\ncurrent-context: \"\"\nkind: Config\npreferences: {}\nusers: []\n"}}
I0711 15:03:41.506734   32744 token.go:141] [discovery] Requesting info from "https://192.168.12.224:6443" again to validate TLS against the pinned public key
I0711 15:03:41.506852   32744 round_trippers.go:419] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubeadm/v1.14.0 (linux/amd64) kubernetes/641856d" 'https://192.168.12.224:6443/api/v1/namespaces/kube-public/configmaps/cluster-info'
I0711 15:03:41.519193   32744 round_trippers.go:438] GET https://192.168.12.224:6443/api/v1/namespaces/kube-public/configmaps/cluster-info 200 OK in 12 milliseconds
I0711 15:03:41.519242   32744 round_trippers.go:444] Response Headers:
I0711 15:03:41.519254   32744 round_trippers.go:447]     Content-Type: application/json
I0711 15:03:41.519264   32744 round_trippers.go:447]     Content-Length: 2331
I0711 15:03:41.519273   32744 round_trippers.go:447]     Date: Thu, 11 Jul 2019 07:03:41 GMT
I0711 15:03:41.519338   32744 request.go:942] Response Body: {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"cluster-info","namespace":"kube-public","selfLink":"/api/v1/namespaces/kube-public/configmaps/cluster-info","uid":"ab35c11a-a133-11e9-9ab6-005056bffc6e","resourceVersion":"355456","creationTimestamp":"2019-07-08T03:51:29Z"},"data":{"jws-kubeconfig-83brsz":"eyJhbGciOiJIUzI1NiIsImtpZCI6IjgzYnJzeiJ9..duFmZ7WgLbFEj8B_dDGrxinP1lu9JCfvOjV5NhtyIg0","jws-kubeconfig-bnavms":"eyJhbGciOiJIUzI1NiIsImtpZCI6ImJuYXZtcyJ9..XXbqwRq7fWlCrnI-zK2_yaKbi9YSXrUOHIeoUrO0iAs","jws-kubeconfig-dperzp":"eyJhbGciOiJIUzI1NiIsImtpZCI6ImRwZXJ6cCJ9..mEZckISMk5j_oOLg_oiROQ7vdcR88rFL_RgUkImdFUs","jws-kubeconfig-xcj9ft":"eyJhbGciOiJIUzI1NiIsImtpZCI6InhjajlmdCJ9..KoNl4BXu9oUXqvl2b4gelCCpKFsRq0vexLCdGjoOVCQ","kubeconfig":"apiVersion: v1\nclusters:\n- cluster:\n    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRFNU1EY3dPREF6TlRFd09Wb1hEVEk1TURjd05UQXpOVEV3T1Zvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBS1FjCmFZVEVMMXZLS2RGcytUZFQ2ZWkzZjNoUzYyTFRZc3MrTFdLMmVKTXJOTllCSXlQOGZvYWE1cXd0anh4KzdxcG4KeFNUQW5OSGpMZnR0b00zQUV6WjE4TTRkVE5xWHU4VDE0bllsamx4MzMxUFZHSDIxS0tidDdyS3ZYU3JSeG8zcwo0dG1EMVpUNUFMaGltK3Z3QnZ5MHZlTDlnWXpJeklIbVpZNmFOamF4YUdUa0VMWWlQOU9IbFpnKzVTVmloMWJ3CjZydTlMWDQrWDJQL2J6QUxGcW45bGtub2RlUFp4TTVKRHFIT0h3aDUyWHc2TlREcU1VTUFrSG4xcTRmU3RZNW4KZ3VsZEw3RVVXVEFxaU5OVnQ3dVpYZk9OeFdEdnRWTmg4ekVJUHRyaWhiWUdHRTFjVUxndFB6UDVqbTN2c1FhRAo2bklZYU9NcmVyNmhPSVFXTlZjQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFHbHc0ZTVLdE1tM1VTZUxNekgyYVcxdVZZZ1gKTWFrQlFnMThJMTloenRIZ0tYUFJQVVNOUE4rMWtRTHdxQ0xwUUwwUm1tWXVHaVhoUTY3eUVlTU1pVXpRcTVqVwpSUUd4Q0FUQlJzV3FRK2lXY0FPSHcxVy9OdWdBdFQxMWxndjUzMVZUREs3cnA0aDgrS1Uway8xY0REWWZycXYrCkRjaGZoYTdUT1NkNW1FQzdQbkpEOUFDaWtQS2w3cHYwZ0ZaelFWUXhCZ0RJN3g0a0JXZjVnN05OclhIa3dMcXcKMmRUaHBNZU9TR2NoM3FjbkU5eW1TNUZoSG1vSHZDN1VFOCtlNUpHMjVkTUlWSXc1LytkL0lscmE4akRZVktoWQpvQXM0dGpWdmtsUnJWRDRwSEVXNDdJZ0FUV2pOMDRuMmE2Y01peFRvRHhtMlM1cXlPU2R6SlkvQ3BJST0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=\n    server: https://192.168.12.110:6443\n  name: \"\"\ncontexts: []\ncurrent-context: \"\"\nkind: Config\npreferences: {}\nusers: []\n"}}
I0711 15:03:41.520091   32744 token.go:164] [discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.12.224:6443"
I0711 15:03:41.520118   32744 token.go:206] [discovery] Successfully established connection with API Server "192.168.12.224:6443"
I0711 15:03:41.520160   32744 join.go:441] [preflight] Fetching init configuration
I0711 15:03:41.520172   32744 join.go:474] [preflight] Retrieving KubeConfig objects
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
I0711 15:03:41.522726   32744 round_trippers.go:419] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubeadm/v1.14.0 (linux/amd64) kubernetes/641856d" -H "Authorization: Bearer dperzp.19444xvuwd5lhr2n" 'https://192.168.12.110:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config'
I0711 15:04:11.523073   32744 round_trippers.go:438] GET https://192.168.12.110:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config  in 30000 milliseconds
I0711 15:04:11.523144   32744 round_trippers.go:444] Response Headers:
error execution phase preflight: unable to fetch the kubeadm-config ConfigMap: failed to get config map: Get https://192.168.12.110:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config: dial tcp 192.168.12.110:6443: i/o timeout

• 分析以上日志发现kubeadm join命令会去kube-public命名空间获取名为cluster-info的ConfigMap
• 检查该configmap: kubectl -n kube-public get configmaps cluster-info -o yaml，发现是这里面的配置还使用的是旧IP，修改后就正常了

apiVersion: v1
data:
  jws-kubeconfig-83brsz: eyJhbGciOiJIUzI1NiIsImtpZCI6IjgzYnJzeiJ9..mGxgno2SegPpoSEQ0k1jp6lwR5FtsVQPupvhJYo-2Bw
  jws-kubeconfig-bnavms: eyJhbGciOiJIUzI1NiIsImtpZCI6ImJuYXZtcyJ9..e3cMCBmDlCEjdmkIrCOBGFzLpIl-GXUBmbLJyOGOP7Q
  jws-kubeconfig-dperzp: eyJhbGciOiJIUzI1NiIsImtpZCI6ImRwZXJ6cCJ9..eJq_-srBIT9LhNlRjmZc8e_sN2xHEVaNA3VVEeiGHV8
  jws-kubeconfig-xcj9ft: eyJhbGciOiJIUzI1NiIsImtpZCI6InhjajlmdCJ9..LHpv5VLLOlTQcXKi9_eFtNrAsViqdvHWEaHGQMb98yU
  jws-kubeconfig-ya6mz7: eyJhbGciOiJIUzI1NiIsImtpZCI6InlhNm16NyJ9..juycWvz0QrqCsbBqXN0-CS4-MutO2Dxkd5eEaaJYg8I
  kubeconfig: |
    apiVersion: v1
    clusters:
    - cluster:
        certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRFNU1EY3dPREF6TlRFd09Wb1hEVEk1TURjd05UQXpOVEV3T1Zvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBS1FjCmFZVEVMMXZLS2RGcytUZFQ2ZWkzZjNoUzYyTFRZc3MrTFdLMmVKTXJOTllCSXlQOGZvYWE1cXd0anh4KzdxcG4KeFNUQW5OSGpMZnR0b00zQUV6WjE4TTRkVE5xWHU4VDE0bllsamx4MzMxUFZHSDIxS0tidDdyS3ZYU3JSeG8zcwo0dG1EMVpUNUFMaGltK3Z3QnZ5MHZlTDlnWXpJeklIbVpZNmFOamF4YUdUa0VMWWlQOU9IbFpnKzVTVmloMWJ3CjZydTlMWDQrWDJQL2J6QUxGcW45bGtub2RlUFp4TTVKRHFIT0h3aDUyWHc2TlREcU1VTUFrSG4xcTRmU3RZNW4KZ3VsZEw3RVVXVEFxaU5OVnQ3dVpYZk9OeFdEdnRWTmg4ekVJUHRyaWhiWUdHRTFjVUxndFB6UDVqbTN2c1FhRAo2bklZYU9NcmVyNmhPSVFXTlZjQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFHbHc0ZTVLdE1tM1VTZUxNekgyYVcxdVZZZ1gKTWFrQlFnMThJMTloenRIZ0tYUFJQVVNOUE4rMWtRTHdxQ0xwUUwwUm1tWXVHaVhoUTY3eUVlTU1pVXpRcTVqVwpSUUd4Q0FUQlJzV3FRK2lXY0FPSHcxVy9OdWdBdFQxMWxndjUzMVZUREs3cnA0aDgrS1Uway8xY0REWWZycXYrCkRjaGZoYTdUT1NkNW1FQzdQbkpEOUFDaWtQS2w3cHYwZ0ZaelFWUXhCZ0RJN3g0a0JXZjVnN05OclhIa3dMcXcKMmRUaHBNZU9TR2NoM3FjbkU5eW1TNUZoSG1vSHZDN1VFOCtlNUpHMjVkTUlWSXc1LytkL0lscmE4akRZVktoWQpvQXM0dGpWdmtsUnJWRDRwSEVXNDdJZ0FUV2pOMDRuMmE2Y01peFRvRHhtMlM1cXlPU2R6SlkvQ3BJST0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
        server: https://192.168.12.110:6443
      name: ""
    contexts: []
    current-context: ""
    kind: Config
    preferences: {}
    users: []
kind: ConfigMap
metadata:
  creationTimestamp: "2019-07-08T03:51:29Z"
  name: cluster-info
  namespace: kube-public
  resourceVersion: "386966"
  selfLink: /api/v1/namespaces/kube-public/configmaps/cluster-info
  uid: ab35c11a-a133-11e9-9ab6-005056bffc6e
root@intellif-0:/var/local/ifaascloud-bootstrap/deploy/cmd/kubernetes# 

三、总结

1、使用kubedam和 kubectl恰当的日志级别对调试会有意想不到的帮助
2、多去看源码，学习命令内部原理