VM虚拟机安装k8s，重启虚拟机之后连接k8s拒绝

环境：centos7
前提：已经设置静态ip
原因：查看master主机的k8s容器是否启动
发现：

[root@master k8s-deployment]# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
[root@master k8s-deployment]# docker ps -a
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS                      PORTS               NAMES
af535c7d7b1d        cfaa4ad74c37           "kube-apiserver --ad…"   9 hours ago         Exited (137) 9 hours ago                        k8s_kube-apiserver_kube-apiserver-master_kube-system_d94cbf4c733bf4cc2a7a1a646adf5507_5
7bc05f750da3        eb516548c180           "/coredns -conf /etc…"   9 hours ago         Exited (2) 9 hours ago                          k8s_coredns_coredns-fb8b8dccf-h47tc_kube-system_5e98a723-5387-11ec-8209-000c29326e93_4
aa2d8bda2911        eb516548c180           "/coredns -conf /etc…"   9 hours ago         Exited (2) 9 hours ago                          k8s_coredns_coredns-fb8b8dccf-mhwmn_kube-system_5e919123-5387-11ec-8209-000c29326e93_4
81f2df55e910        cfaa4ad74c37           "kube-apiserver --ad…"   9 hours ago         Exited (0) 9 hours ago                          k8s_kube-apiserver_kube-apiserver-master_kube-system_d94cbf4c733bf4cc2a7a1a646adf5507_4
b7f45e42ae2a        efb3887b411d           "kube-controller-man…"   16 hours ago        Exited (2) 9 hours ago                          k8s_kube-controller-manager_kube-controller-manager-master_kube-system_ecf9c37413eace225bc60becabeddb3b_1
ca086629f638        8931473d5bdb           "kube-scheduler --bi…"   16 hours ago        Exited (2) 9 hours ago                          k8s_kube-scheduler_kube-scheduler-master_kube-system_f44110a0ca540009109bfc32a7eb0baa_1
7539ea641479        k8s.gcr.io/pause:3.1   "/pause"                 20 hours ago        Exited (0) 9 hours ago                          k8s_POD_coredns-fb8b8dccf-mhwmn_kube-system_5e919123-5387-11ec-8209-000c29326e93_0
e26ac6287091        k8s.gcr.io/pause:3.1   "/pause"                 20 hours ago        Exited (0) 9 hours ago                          k8s_POD_coredns-fb8b8dccf-h47tc_kube-system_5e98a723-5387-11ec-8209-000c29326e93_0
45a3429588b9        ff281650a721           "/opt/bin/flanneld -…"   20 hours ago        Exited (137) 9 hours ago                        k8s_kube-flannel_kube-flannel-ds-amd64-q7pb7_kube-system_7043194a-5387-11ec-8209-000c29326e93_0
04bdb6ca3214        ff281650a721           "cp -f /etc/kube-fla…"   20 hours ago        Exited (0) 20 hours ago                         k8s_install-cni_kube-flannel-ds-amd64-q7pb7_kube-system_7043194a-5387-11ec-8209-000c29326e93_0
0f4f6df97494        k8s.gcr.io/pause:3.1   "/pause"                 20 hours ago        Exited (0) 9 hours ago                          k8s_POD_kube-flannel-ds-amd64-q7pb7_kube-system_7043194a-5387-11ec-8209-000c29326e93_0
9b97bad3f8fc        20a2d7035165           "/usr/local/bin/kube…"   20 hours ago        Exited (2) 9 hours ago                          k8s_kube-proxy_kube-proxy-tws8v_kube-system_5e75e3d0-5387-11ec-8209-000c29326e93_0
035daae1272c        k8s.gcr.io/pause:3.1   "/pause"                 20 hours ago        Exited (0) 9 hours ago                          k8s_POD_kube-proxy-tws8v_kube-system_5e75e3d0-5387-11ec-8209-000c29326e93_0
9bbce988e279        2c4adeb21b4f           "etcd --advertise-cl…"   20 hours ago        Exited (0) 9 hours ago                          k8s_etcd_etcd-master_kube-system_8bb9e48436e393a6bb8cd7a0cb41d33b_0
5b166fa382b1        8931473d5bdb           "kube-scheduler --bi…"   20 hours ago        Exited (1) 16 hours ago                         k8s_kube-scheduler_kube-scheduler-master_kube-system_f44110a0ca540009109bfc32a7eb0baa_0
11ebcb6bdef3        efb3887b411d           "kube-controller-man…"   20 hours ago        Exited (255) 16 hours ago                       k8s_kube-controller-manager_kube-controller-manager-master_kube-system_ecf9c37413eace225bc60becabeddb3b_0
fb076e784e45        k8s.gcr.io/pause:3.1   "/pause"                 20 hours ago        Exited (0) 9 hours ago                          k8s_POD_kube-apiserver-master_kube-system_d94cbf4c733bf4cc2a7a1a646adf5507_0
2265f7ad617f        k8s.gcr.io/pause:3.1   "/pause"                 20 hours ago        Exited (0) 9 hours ago                          k8s_POD_etcd-master_kube-system_8bb9e48436e393a6bb8cd7a0cb41d33b_0
78f717e32a33        k8s.gcr.io/pause:3.1   "/pause"                 20 hours ago        Exited (0) 9 hours ago                          k8s_POD_kube-scheduler-master_kube-system_f44110a0ca540009109bfc32a7eb0baa_0
31cb5d977ae4        k8s.gcr.io/pause:3.1   "/pause"                 20 hours ago        Exited (0) 9 hours ago                          k8s_POD_kube-controller-manager-master_kube-system_ecf9c37413eace225bc60becabeddb3b_0

所有的容器都没有启动。
解决：将所有的容器全部启动，并将容器设置为启动自动运行

[root@master k8s-deployment]# docker start af535c7d7b1d 7bc05f750da3 aa2d8bda2911 81f2df55e910 b7f45e42ae2a ca086629f638 7539ea641479 e26ac6287091 45a3429588b9 04bdb6ca3214 0f4f6df97494 9b97bad3f8fc 035daae1272c 9bbce988e279 5b166fa382b1 11ebcb6bdef3 fb076e784e45 2265f7ad617f 78f717e32a33 31cb5d977ae4
Error response from daemon: cannot join network of a non running container: fb076e784e451f2c26b63c2cc39a4a0e1316c39d76bde5f2ff7871b923f49a8e
Error response from daemon: cannot join network of a non running container: e26ac62870918f787a15cc4b760c03c15fccebbb1b8712ae17a2e07b6ec04849
Error response from daemon: cannot join network of a non running container: 7539ea64147910b68c4c83c13c02931dc4a993529465abca5e6863fda050d93a
Error response from daemon: cannot join network of a non running container: fb076e784e451f2c26b63c2cc39a4a0e1316c39d76bde5f2ff7871b923f49a8e
Error response from daemon: cannot join network of a non running container: 31cb5d977ae4b6ba76767e6eace90409b5b9a977765b89e1baa7002d18d8f9ee
Error response from daemon: cannot join network of a non running container: 78f717e32a33bf24a1f67df6429d16acce45fd79244b11b564210b17e40de470
7539ea641479
e26ac6287091
Error response from daemon: cannot join network of a non running container: 0f4f6df97494d3229840fd97c93bf8e22c4a538c6195625e9fb2bd421e725886
Error response from daemon: cannot join network of a non running container: 0f4f6df97494d3229840fd97c93bf8e22c4a538c6195625e9fb2bd421e725886
0f4f6df97494
Error response from daemon: cannot join network of a non running container: 035daae1272cebe32da924567b4765600935e0cace11bd0aad8679baa9a19a1b
035daae1272c
Error response from daemon: cannot join network of a non running container: 2265f7ad617fa8f2510b646edbb15893a85f20761e4f2d7d08ea5b0d77fe307f
Error response from daemon: cannot join network of a non running container: 78f717e32a33bf24a1f67df6429d16acce45fd79244b11b564210b17e40de470
Error response from daemon: cannot join network of a non running container: 31cb5d977ae4b6ba76767e6eace90409b5b9a977765b89e1baa7002d18d8f9ee
fb076e784e45
2265f7ad617f
78f717e32a33
31cb5d977ae4
Error: failed to start containers: af535c7d7b1d, 7bc05f750da3, aa2d8bda2911, 81f2df55e910, b7f45e42ae2a, ca086629f638, 45a3429588b9, 04bdb6ca3214, 9b97bad3f8fc, 9bbce988e279, 5b166fa382b1, 11ebcb6bdef3
[root@master k8s-deployment]# docker ps 
CONTAINER ID        IMAGE                  COMMAND             CREATED             STATUS              PORTS               NAMES
7539ea641479        k8s.gcr.io/pause:3.1   "/pause"            20 hours ago        Up 23 seconds                           k8s_POD_coredns-fb8b8dccf-mhwmn_kube-system_5e919123-5387-11ec-8209-000c29326e93_0
e26ac6287091        k8s.gcr.io/pause:3.1   "/pause"            20 hours ago        Up 22 seconds                           k8s_POD_coredns-fb8b8dccf-h47tc_kube-system_5e98a723-5387-11ec-8209-000c29326e93_0
0f4f6df97494        k8s.gcr.io/pause:3.1   "/pause"            20 hours ago        Up 22 seconds                           k8s_POD_kube-flannel-ds-amd64-q7pb7_kube-system_7043194a-5387-11ec-8209-000c29326e93_0
035daae1272c        k8s.gcr.io/pause:3.1   "/pause"            20 hours ago        Up 21 seconds                           k8s_POD_kube-proxy-tws8v_kube-system_5e75e3d0-5387-11ec-8209-000c29326e93_0
fb076e784e45        k8s.gcr.io/pause:3.1   "/pause"            20 hours ago        Up 21 seconds                           k8s_POD_kube-apiserver-master_kube-system_d94cbf4c733bf4cc2a7a1a646adf5507_0
2265f7ad617f        k8s.gcr.io/pause:3.1   "/pause"            20 hours ago        Up 20 seconds                           k8s_POD_etcd-master_kube-system_8bb9e48436e393a6bb8cd7a0cb41d33b_0
78f717e32a33        k8s.gcr.io/pause:3.1   "/pause"            20 hours ago        Up 20 seconds                           k8s_POD_kube-scheduler-master_kube-system_f44110a0ca540009109bfc32a7eb0baa_0
31cb5d977ae4        k8s.gcr.io/pause:3.1   "/pause"            20 hours ago        Up 19 seconds                           k8s_POD_kube-controller-manager-master_kube-system_ecf9c37413eace225bc60becabeddb3b_0

一次没有全部启动成功，再试一次
[root@master k8s-deployment]# docker start af535c7d7b1d 7bc05f750da3 aa2d8bda2911 81f2df55e910 b7f45e42ae2a ca086629f638 7539ea641479 e26ac6287091 45a3429588b9 04bdb6ca3214 0f4f6df97494 9b97bad3f8fc 035daae1272c 9bbce988e279 5b166fa382b1 11ebcb6bdef3 fb076e784e45 2265f7ad617f 78f717e32a33 31cb5d977ae4
af535c7d7b1d
7bc05f750da3
aa2d8bda2911
81f2df55e910
b7f45e42ae2a
ca086629f638
7539ea641479
e26ac6287091
45a3429588b9
04bdb6ca3214
0f4f6df97494
9b97bad3f8fc
035daae1272c
9bbce988e279
5b166fa382b1
11ebcb6bdef3
fb076e784e45
2265f7ad617f
78f717e32a33
31cb5d977ae4

验证
[root@master k8s-deployment]# kubectl get svc
NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
kubernetes       ClusterIP   10.96.0.1       <none>        443/TCP          19h
tomcat-service   NodePort    10.97.136.163   <none>        8000:32500/TCP   19h

设置容器启动自动开启
[root@master k8s-deployment]# docker update af535c7d7b1d 7bc05f750da3 aa2d8bda2911 81f2df55e910 b7f45e42ae2a ca086629f638 7539ea641479 e26ac6287091 45a3429588b9 04bdb6ca3214 0f4f6df97494 9b97bad3f8fc 035daae1272c 9bbce988e279 5b166fa382b1 11ebcb6bdef3 fb076e784e45 2265f7ad617f 78f717e32a33 31cb5d977ae4 --restart=always
af535c7d7b1d
7bc05f750da3
aa2d8bda2911
81f2df55e910
b7f45e42ae2a
ca086629f638
7539ea641479
e26ac6287091
45a3429588b9
04bdb6ca3214
0f4f6df97494
9b97bad3f8fc
035daae1272c
9bbce988e279
5b166fa382b1
11ebcb6bdef3
fb076e784e45
2265f7ad617f
78f717e32a33
31cb5d977ae4

ok