检查etcd集群的状态：

[root@master ~]# etcdctl cluster-health
member 8e9e05c52164694d is healthy: got healthy result from http://192.168.2.178:2379
cluster is healthy

 

[root@master ~]# vi /etc/sysconfig/flanneld 
# Flanneld configuration options

# etcd url location.  Point this to the server where etcd runs
FLANNEL_ETCD_ENDPOINTS="http://192.168.2.178:2379"

# etcd config key.  This is the configuration key that flannel queries
# For address range assignment
#FLANNEL_ETCD_PREFIX="/atomic.io/network"
FLANNEL_ETCD_PREFIX="/k8s/network"

# Any additional options that you want to pass
FLANNEL_OPTIONS="--iface=ens33"

此时启动flanneld服务，半天起不来，其实报错了：

systemctl restart flanneld

查看系统日志/var/log/messages

Dec 18 22:06:03 k8s01 flanneld-start: E1218 22:06:03.423708    1735 network.go:102] failed to retrieve network config: 100: Key not found (/k8s) [3]
Dec 18 22:06:04 k8s01 flanneld-start: E1218 22:06:04.428807    1735 network.go:102] failed to retrieve network config: 100: Key not found (/k8s) [3]
Dec 18 22:06:05 k8s01 flanneld-start: E1218 22:06:05.431055    1735 network.go:102] failed to retrieve network config: 100: Key not found (/k8s) [3]
Dec 18 22:06:06 k8s01 flanneld-start: E1218 22:06:06.434662    1735 network.go:102] failed to retrieve network config: 100: Key not found (/k8s) [3]
Dec 18 22:06:07 k8s01 flanneld-start: E1218 22:06:07.440863    1735 network.go:102] failed to retrieve network config: 100: Key not found (/k8s) [3]
Dec 18 22:06:08 k8s01 flanneld-start: E1218 22:06:08.444012    1735 network.go:102] failed to retrieve network config: 100: Key not found (/k8s) [3]
Dec 18 22:06:09 k8s01 flanneld-start: E1218 22:06:09.447245    1735 network.go:102] failed to retrieve network config: 100: Key not found (/k8s) [3]
Dec 18 22:06:10 k8s01 flanneld-start: E1218 22:06:10.452601    1735 network.go:102] failed to retrieve network config: 100: Key not found (/k8s) [3]
Dec 18 22:06:11 k8s01 flanneld-start: E1218 22:06:11.453889    1735 network.go:102] failed to retrieve network config: 100: Key not found (/k8s) [3]

会发现大量这样的报错。

[root@master ~]# etcdctl set /k8s/network/config '{"Network":"10.255.0.0/16"}'
{"Network":"10.255.0.0/16"}
[root@master ~]# etcdctl get /k8s/network/config
{"Network":"10.255.0.0/16"}

执行一下上面的命令，设置一下/k8s/network/config即可。

[root@master ~]# systemctl restart flanneld
[root@master ~]# systemctl status flanneld

● flanneld.service - Flanneld overlay address etcd agent
   Loaded: loaded (/usr/lib/systemd/system/flanneld.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2019-12-18 22:17:52 CST; 21s ago
  Process: 1854 ExecStartPost=/usr/libexec/flannel/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker (code=exited, status=0/SUCCESS)
 Main PID: 1843 (flanneld)
   Memory: 7.0M
   CGroup: /system.slice/flanneld.service
           └─1843 /usr/bin/flanneld -etcd-endpoints=http://192.168.2.178:2379...

Dec 18 22:17:52 master systemd[1]: Starting Flanneld overlay address etcd a.....
Dec 18 22:17:52 master flanneld-start[1843]: I1218 22:17:52.454158    1843 m...s
Dec 18 22:17:52 master flanneld-start[1843]: I1218 22:17:52.454556    1843 m...8
Dec 18 22:17:52 master flanneld-start[1843]: I1218 22:17:52.454568    1843 m...)
Dec 18 22:17:52 master flanneld-start[1843]: I1218 22:17:52.457115    1843 l...0
Dec 18 22:17:52 master flanneld-start[1843]: I1218 22:17:52.467605    1843 m...4
Dec 18 22:17:52 master flanneld-start[1843]: I1218 22:17:52.467830    1843 n...s
Dec 18 22:17:52 master systemd[1]: Started Flanneld overlay address etcd agent.
Hint: Some lines were ellipsized, use -l to show in full.

[root@master ~]# systemctl restart kube-apiserver kube-controller-manager kube-scheduler
[root@master ~]# systemctl status kube-apiserver kube-controller-manager kube-scheduler
● kube-apiserver.service - Kubernetes API Server
   Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2019-12-18 22:24:13 CST; 21s ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
 Main PID: 1981 (kube-apiserver)
   Memory: 36.4M
   CGroup: /system.slice/kube-apiserver.service
           └─1981 /usr/bin/kube-apiserver --logtostderr=true --v=0 --etcd-servers=http://192.168.2.178:2379 --insecure-bind-address=0.0.0.0 --allow-privileged=false --service-cluster-ip-range=10.254.0....

Dec 18 22:24:12 master systemd[1]: Starting Kubernetes API Server...
Dec 18 22:24:12 master kube-apiserver[1981]: I1218 22:24:12.900368    1981 config.go:562] Will report 192.168.2.178 as public IP address.
Dec 18 22:24:12 master kube-apiserver[1981]: W1218 22:24:12.900898    1981 handlers.go:50] Authentication is disabled
Dec 18 22:24:12 master kube-apiserver[1981]: [restful] 2019/12/18 22:24:12 log.go:30: [restful/swagger] listing is available at https://192.168.2.178:6443/swaggerapi/
Dec 18 22:24:12 master kube-apiserver[1981]: [restful] 2019/12/18 22:24:12 log.go:30: [restful/swagger] https://192.168.2.178:6443/swaggerui/ is mapped to folder /swagger-ui/
Dec 18 22:24:13 master kube-apiserver[1981]: I1218 22:24:13.028655    1981 serve.go:95] Serving securely on 0.0.0.0:6443
Dec 18 22:24:13 master kube-apiserver[1981]: I1218 22:24:13.028712    1981 serve.go:109] Serving insecurely on 0.0.0.0:8080
Dec 18 22:24:13 master systemd[1]: Started Kubernetes API Server.

● kube-controller-manager.service - Kubernetes Controller Manager
   Loaded: loaded (/usr/lib/systemd/system/kube-controller-manager.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2019-12-18 22:24:12 CST; 21s ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
 Main PID: 1980 (kube-controller)
   Memory: 20.2M
   CGroup: /system.slice/kube-controller-manager.service
           └─1980 /usr/bin/kube-controller-manager --logtostderr=true --v=0 --master=http://192.168.2.178:8080

Dec 18 22:24:16 master kube-controller-manager[1980]: I1218 22:24:16.539626    1980 pet_set.go:146] Starting statefulset controller
Dec 18 22:24:16 master kube-controller-manager[1980]: I1218 22:24:16.551133    1980 controllermanager.go:544] Attempting to start certificates, full resource map map[autoscaling/v1:&APIReso...orizontalPod
Dec 18 22:24:16 master kube-controller-manager[1980]: ationcontrollers true ReplicationControllerDummy} {replicationcontrollers/scale true Scale} {thirdpartyresources false ThirdPartyResour...gets/status 
Dec 18 22:24:16 master kube-controller-manager[1980]: I1218 22:24:16.551305    1980 controllermanager.go:546] Starting certificates.k8s.io/v1alpha1 apis
Dec 18 22:24:16 master kube-controller-manager[1980]: I1218 22:24:16.551322    1980 controllermanager.go:548] Starting certificate request controller
Dec 18 22:24:16 master kube-controller-manager[1980]: E1218 22:24:16.551544    1980 controllermanager.go:558] Failed to start certificate controller: open /etc/kubernetes/ca/ca.pem: no such...or directory
Dec 18 22:24:16 master kube-controller-manager[1980]: I1218 22:24:16.553035    1980 attach_detach_controller.go:235] Starting Attach Detach Controller
Dec 18 22:24:16 master kube-controller-manager[1980]: I1218 22:24:16.553089    1980 serviceaccounts_controller.go:120] Starting ServiceAccount controller
Dec 18 22:24:16 master kube-controller-manager[1980]: I1218 22:24:16.562581    1980 garbagecollector.go:766] Garbage Collector: Initializing
Dec 18 22:24:26 master kube-controller-manager[1980]: I1218 22:24:26.563231    1980 garbagecollector.go:780] Garbage Collector: All monitored resources synced. Proceeding to collect garbage

● kube-scheduler.service - Kubernetes Scheduler Plugin
   Loaded: loaded (/usr/lib/systemd/system/kube-scheduler.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2019-12-18 22:24:12 CST; 21s ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
 Main PID: 1979 (kube-scheduler)
   Memory: 15.4M
   CGroup: /system.slice/kube-scheduler.service
           └─1979 /usr/bin/kube-scheduler --logtostderr=true --v=0 --master=http://192.168.2.178:8080 --address=0.0.0.0

Dec 18 22:24:13 master kube-scheduler[1979]: E1218 22:24:13.000603    1979 reflector.go:199] pkg/controller/informers/factory.go:89: Failed to list *api.PersistentVolumeClaim: Get http://1...ction refused
Dec 18 22:24:13 master kube-scheduler[1979]: E1218 22:24:13.000633    1979 reflector.go:199] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:473: Failed to list *api.ReplicationC...ction refused
Dec 18 22:24:13 master kube-scheduler[1979]: E1218 22:24:13.000658    1979 reflector.go:199] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:470: Failed to list *api.Service: Get...ction refused
Dec 18 22:24:13 master kube-scheduler[1979]: E1218 22:24:13.000686    1979 reflector.go:199] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:466: Failed to list *api.PersistentVo...ction refused
Dec 18 22:24:13 master kube-scheduler[1979]: E1218 22:24:13.000718    1979 reflector.go:199] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:463: Failed to list *api.Node: Get ht...ction refused
Dec 18 22:24:13 master kube-scheduler[1979]: E1218 22:24:13.000757    1979 reflector.go:199] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:460: Failed to list *api.Pod: Get http://192.168.2...
Dec 18 22:24:13 master kube-scheduler[1979]: E1218 22:24:13.000807    1979 leaderelection.go:228] error retrieving resource lock kube-system/kube-scheduler: Get http://192.168.2.178:8080/a...ction refused
Dec 18 22:24:13 master kube-scheduler[1979]: E1218 22:24:13.000894    1979 reflector.go:199] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:457: Failed to list *api.Pod: Get http://192.168.2...
Dec 18 22:24:16 master kube-scheduler[1979]: I1218 22:24:16.457251    1979 leaderelection.go:188] sucessfully acquired lease kube-system/kube-scheduler
Dec 18 22:24:16 master kube-scheduler[1979]: I1218 22:24:16.457483    1979 event.go:217] Event(api.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"kube-scheduler", UID:"12...became leader
Hint: Some lines were ellipsized, use -l to show in full.
[root@master ~]# 

 

停止服务master节点：

systemctl stop kube-apiserver kube-controller-manager kube-scheduler flanneld

systemctl status kube-apiserver kube-controller-manager kube-scheduler flanneld

systemctl start kube-apiserver kube-controller-manager kube-scheduler flanneld