网上讲述如何部署k8s集群的文章很多，特别是k8s中文社区里面，每个平台的部署方式都有详细说明。但就是看了中文社区的指导，发现一路是坑，第一个源访问的时候就404 NotFound， 更别说那生涩的翻译和版本匹配问题。

     如此一来还不如自己写一个。

  1. 开虚机

      系统环境CentOS 7.2 . 这方面不再赘述

  2. 设置/etc/hosts 

      简单来说就是把master和minion主机都用域名在hosts文件中记录一下。

  3. 集群主机都安装kubernetes和etcd

       直接yum install . 安装kubernetes 时会顺带把docker等都安装上。 此文章编写时，阿里云的kubernetes版本为1.5.2.

      这里还需要注意：应该在etcdctl中配置flannel，否则flannel无法正常启动。

[root@k8s-master home]# etcdctl set /flannel/network/config '{ "Network": "172.16.0.0/16", "SubnetLen": 24, "Backend": { "Type": "vxlan" } }' 

  4. 配置apiserver

      按照中文社区的指导配置（但最终并没有使用该配置，看下去就知道）

      修改/etc/kubernetes/apiserver

      修改/etc/kubernetes/config

     由于这条路不通，具体修改内容就不贴了。

  5. master配置启动脚本

#/bin/bashfor SERVICES in etcd kube-apiserver kube-controller-manager kube-scheduler; dosystemctl restart $SERVICESsystemctl enable $SERVICESsystemctl status $SERVICESdone

   走到这里，按照中文社区的指导，应该就能直接起来了。但其实并不能！！！

   收到以下错误：

Sep 29 17:06:15 debug010000002015 kube-apiserver: W0929 17:06:15.881473   21259 handlers.go:50] Authentication is disabledSep 29 17:06:15 debug010000002015 kube-apiserver: [restful] 2018/09/29 17:06:15 log.go:30: [restful/swagger] listing is available at https://172.16.7.93:6443/swaggerapi/Sep 29 17:06:15 debug010000002015 kube-apiserver: [restful] 2018/09/29 17:06:15 log.go:30: [restful/swagger] https://172.16.7.93:6443/swaggerui/ is mapped to folder /swagger-ui/Sep 29 17:06:15 debug010000002015 kube-apiserver: E0929 17:06:15.984071   21259 reflector.go:199] k8s.io/kubernetes/plugin/pkg/admission/resourcequota/resource_access.go:83: Failed to list *api.ResourceQuota: Get http://127.0.0.1:18080/api/v1/resourcequotas?resourceVersion=0: dial tcp 127.0.0.1:18080: getsockopt: connection refusedSep 29 17:06:15 debug010000002015 kube-apiserver: E0929 17:06:15.984217   21259 reflector.go:199] pkg/controller/informers/factory.go:89: Failed to list *api.Namespace: Get http://127.0.0.1:18080/api/v1/namespaces?resourceVersion=0: dial tcp 127.0.0.1:18080: getsockopt: connection refusedSep 29 17:06:15 debug010000002015 kube-apiserver: E0929 17:06:15.987986   21259 reflector.go:199] pkg/controller/informers/factory.go:89: Failed to list *api.LimitRange: Get http://127.0.0.1:18080/api/v1/limitranges?resourceVersion=0: dial tcp 127.0.0.1:18080: getsockopt: connection refusedSep 29 17:06:16 debug010000002015 kube-apiserver: F0929 17:06:16.058072   21259 genericapiserver.go:189] unable to load server certificate: open /var/run/kubernetes/apiserver.key: permission deniedSep 29 17:06:16 debug010000002015 systemd: kube-apiserver.service: main process exited, code=exited, status=255/n/aSep 29 17:06:16 debug010000002015 systemd: Failed to start Kubernetes API Server.Sep 29 17:06:16 debug010000002015 systemd: Unit kube-apiserver.service entered failed state.Sep 29 17:06:16 debug010000002015 systemd: kube-apiserver.service failed.Sep 29 17:06:16 debug010000002015 systemd: kube-apiserver.service holdoff time over, scheduling restart.

 查询了Google和Baidu都无果。

 但是测试发现直接用命令行启动kube-api 是成功的。因此只好采取直接修改systemctl service文件的做法。

 修改kube-apiserver.service的启动脚本， 路径是 /lib/systemd/system/kube-apiserver.service

 内容如下：

[root@k8s-master home]# vi /lib/systemd/system/kube-apiserver.service  [Unit]Description=Kubernetes API ServiceDocumentation=https://github.com/GoogleCloudPlatform/kubernetesAfter=network.targetAfter=etcd.service [Service]EnvironmentFile=-/etc/kubernetes/configEnvironmentFile=-/etc/kubernetes/apiserver#ExecStart=/usr/bin/kube-apiserver \#           $KUBE_LOGTOSTDERR \#           $KUBE_LOG_LEVEL \#           $KUBE_ETCD_SERVERS \#           $KUBE_API_ADDRESS \#           $KUBE_API_PORT \#           $KUBELET_PORT \#           $KUBE_ALLOW_PRIV \#           $KUBE_SERVICE_ADDRESSES \#           $KUBE_ADMISSION_CONTROL \#           $KUBE_API_ARGS ExecStart=/usr/bin/kube-apiserver --allow_privileged=true --logtostderr=false --v=6 --log-dir=/var/log/k8s/kube-apiserver --insecure-bind-address=0.0.0.0 --insecure-port=8080 --admission_control=NamespaceLifecycle,NamespaceExists,LimitRanger,ResourceQuota,ServiceAccount,AlwaysPullImages,SecurityContextDeny --etcd_servers=http://x.x.x.x:2379 --master-service-namespace=master --secure-port=6443 --bind-address=0.0.0.0 --service-cluster-ip-range=10.0.0.0/16 --max-requests-inflight=1000 --storage-backend=etcd3 --tls-cert-file=/etc/kubernetes/pki/apiserver.pem \            --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \            --client-ca-file=/etc/kubernetes/pki/ca.pem \            --service-account-key-file=/etc/kubernetes/pki/ca-key.pemKillMode=control-groupRestart=on-failureRestartSec=10 [Install]WantedBy=multi-user.target

其中 ： --etcd_servers=http://xx.xx.xx.xx:2379   为本机eth0网卡IP,需要替换。

             相关ssl文件需要用openssl自己生成。或者使用insecure模式。

将 etcd的配置文件，路径如下：/etc/etcd/etcd.conf ， 其中下述内容由监听本地回环改为监听0.0.0.0

ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379"

修改flannel.service文件

[root@k8s-master home]# vi /lib/systemd/system/flanneld.service  [Unit]Description=Flanneld overlay address etcd agentAfter=network.targetAfter=network-online.targetWants=network-online.targetAfter=etcd.serviceBefore=docker.service [Service]Type=notifyEnvironmentFile=/etc/sysconfig/flanneldEnvironmentFile=-/etc/sysconfig/docker-networkExecStart=/usr/bin/flanneld -etcd-endpoints=http://x.x.x.x:2379 -etcd-prefix=/flannel/network -iface=eth0#ExecStart=/usr/bin/flanneld-start $FLANNEL_OPTIONS#ExecStartPost=/usr/libexec/flannel/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/dockerRestart=on-failure [Install]WantedBy=multi-user.targetWantedBy=docker.service

其中 ： --etcd_servers=http://xx.xx.xx.xx:2379   为本机eth0网卡IP,需要替换。

修改kube-controller-manager.service文件

[root@k8s-master home]# vi /lib/systemd/system/kube-controller-manager.service  Description=Kubernetes Controller ManagerDocumentation=https://github.com/GoogleCloudPlatform/kubernetes [Service]EnvironmentFile=-/etc/kubernetes/configEnvironmentFile=-/etc/kubernetes/controller-manager#ExecStart=/usr/bin/kube-controller-manager \#           $KUBE_LOGTOSTDERR \#           $KUBE_LOG_LEVEL \#           $KUBE_MASTER \#           $KUBE_CONTROLLER_MANAGER_ARGS ExecStart=/usr/bin/kube-controller-manager --logtostderr=false --v=6 --log-dir=/var/log/k8s/kube-controller-manager --namespace-sync-period=5m0s --node-monitor-grace-period=40s --node-monitor-period=5s --node-startup-grace-period=1m0s --node-sync-period=10s --pod-eviction-timeout=5m0s --pvclaimbinder-sync-period=10s --register-retry-count=20    --kubeconfig=/etc/kubernetes/controller-manager.conf \            --cluster-name=kubernetes \            --service-cluster-ip-range=10.0.0.0/16 \            --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \            --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \            --service-account-private-key-file=/etc/kubernetes/pki/ca-key.pem \            --root-ca-file=/etc/kubernetes/pki/ca.pem Restart=on-failureLimitNOFILE=65536 [Install]WantedBy=multi-user.target

修改kube-scheduler.service文件

[root@k8s-master home]# vi /lib/systemd/system/kube-scheduler.service  [Unit]Description=Kubernetes Scheduler PluginDocumentation=https://github.com/GoogleCloudPlatform/kubernetes [Service]EnvironmentFile=-/etc/kubernetes/configEnvironmentFile=-/etc/kubernetes/scheduler#ExecStart=/usr/bin/kube-scheduler \#           $KUBE_LOGTOSTDERR \#           $KUBE_LOG_LEVEL \#           $KUBE_MASTER \#           $KUBE_SCHEDULER_ARGS ExecStart=/usr/bin/kube-scheduler --logtostderr=false --v=6 --log-dir=/var/log/k8s/kube-scheduler --algorithm-provider=DefaultProvider --kubeconfig=/etc/kubernetes/scheduler.conf Restart=on-failureLimitNOFILE=65536 [Install]WantedBy=multi-user.target

修改kube-proxy.service文件

[root@k8s-master home]# vi /lib/systemd/system/kube-proxy.service  [Unit]Description=Kubernetes Kube-Proxy ServerDocumentation=https://github.com/GoogleCloudPlatform/kubernetesAfter=network.target [Service]EnvironmentFile=-/etc/kubernetes/configEnvironmentFile=-/etc/kubernetes/proxy#ExecStart=/usr/bin/kube-proxy \#           $KUBE_LOGTOSTDERR \#           $KUBE_LOG_LEVEL \#           $KUBE_MASTER \#           $KUBE_PROXY_ARGS ExecStart=/usr/bin/kube-proxy --master=http://x.x.x.x:8080 --hostname-override=k8s-master --proxy-mode=iptables -v=6 --logtostderr=false --log-dir=/var/log/k8s/kube-proxyRestart=on-failureLimitNOFILE=65536 [Install]WantedBy=multi-user.target

其中 xx.xx.xx.xx:8080   为本机eth0网卡IP,需要替换。

修改kubelet.service文件

[root@k8s-master home]# vi /lib/systemd/system/kubelet.service  [Unit]Description=Kubernetes Kubelet ServerDocumentation=https://github.com/GoogleCloudPlatform/kubernetesAfter=docker.serviceRequires=docker.service [Service]WorkingDirectory=/var/lib/kubeletEnvironmentFile=-/etc/kubernetes/configEnvironmentFile=-/etc/kubernetes/kubelet#ExecStart=/usr/bin/kubelet \#           $KUBE_LOGTOSTDERR \#           $KUBE_LOG_LEVEL \#           $KUBELET_API_SERVER \#           $KUBELET_ADDRESS \#           $KUBELET_PORT \#           $KUBELET_HOSTNAME \#           $KUBE_ALLOW_PRIV \#           $KUBELET_POD_INFRA_CONTAINER \#           $KUBELET_ARGS ExecStart=/usr/bin/kubelet --allow-privileged=true \        --logtostderr=false \        --v=6 \        --log-dir=/var/log/k8s/kubelet \        --address=x.x.x.x \        --cluster-dns=10.0.1.10 \        --hostname-override=k8s-master \        --cluster-domain=cluster.local \        --kubeconfig=/etc/kubernetes/kubelet.conf \        --pod-manifest-path=/etc/kubernetes/manifest \        --allow-privileged=true \        --authorization-mode=AlwaysAllow \        --fail-swap-on=false \        --cgroup-driver=systemd \        --pod-infra-container-image=registry.aliyuncs.com/archon/pause-amd64:3.0 Restart=on-failure [Install]WantedBy=multi-user.target

其中 xx.xx.xx.xx   为本机eth0网卡IP,需要替换。

registry.aliyuncs.com/archon/pause-amd64:3.0  来源自https://segmentfault.com/q/1010000008763165/a-1020000008824481

  完毕后，再运行启动脚本，此时所有组件都能够正常启动。

 6.  minion配置文件

     路径为： /etc/kubernetes/kubelet 以及 /etc/kubernetes/config

   config文件内容如下：

#### kubernetes system config## The following values are used to configure various aspects of all# kubernetes services, including##   kube-apiserver.service#   kube-controller-manager.service#   kube-scheduler.service#   kubelet.service#   kube-proxy.service# logging to stderr means we get it in the systemd journalKUBE_LOGTOSTDERR="--logtostderr=true" # journal message level, 0 is debugKUBE_LOG_LEVEL="--v=0" # Should this cluster be allowed to run privileged docker containersKUBE_ALLOW_PRIV="--allow-privileged=false" # How the controller-manager, scheduler, and proxy find the apiserverKUBE_MASTER="--master=http://k8s-master:8080" # Comma separated list of nodes in the etcd clusterKUBE_ETCD_SERVERS=”–etcd_servers=http://k8s-master:4001″ # logging to stderr means we get it in the systemd journal

 kubelet文件如下：

 #### kubernetes kubelet (minion) config # The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)KUBELET_ADDRESS="--address=0.0.0.0" # The port for the info server to serve on# KUBELET_PORT="--port=10250" # You may leave this blank to use the actual hostnameKUBELET_HOSTNAME="--hostname-override=k8s-slave" # location of the api-serverKUBELET_API_SERVER="--api-servers=http://k8s-master:8080" # pod infrastructure containerKUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest" # Add your own!KUBELET_ARGS=""

验证服务状态：

[root@k8s-slave home]# kubectl get csNAME                 STATUS    MESSAGE              ERRORscheduler            Healthy   ok                   controller-manager   Healthy   ok                   etcd-0               Healthy   {"health": "true"}  

  6. minion配置启动脚本

修改kube-proxy.service文件

[root@k8s-slave home]# vi /lib/systemd/system/kube-proxy.service  [Unit]Description=Kubernetes Kube-Proxy ServerDocumentation=https://github.com/GoogleCloudPlatform/kubernetesAfter=network.target [Service]EnvironmentFile=-/etc/kubernetes/configEnvironmentFile=-/etc/kubernetes/proxy#ExecStart=/usr/bin/kube-proxy \#           $KUBE_LOGTOSTDERR \#           $KUBE_LOG_LEVEL \#           $KUBE_MASTER \#           $KUBE_PROXY_ARGS ExecStart=/usr/bin/kube-proxy --master=http://x.x.x.x:8080 --hostname-override=k8s-slave --proxy-mode=iptables -v=6 --logtostderr=false --log-dir=/var/log/k8s/kube-proxy Restart=on-failureLimitNOFILE=65536 [Install]WantedBy=multi-user.target

其中 xx.xx.xx.xx   为master eth0网卡IP,需要替换。

修改kubelet配置文件

[root@k8s-slave home]# vi /lib/systemd/system/kubelet.service  [Unit]Description=Kubernetes Kubelet ServerDocumentation=https://github.com/GoogleCloudPlatform/kubernetesAfter=docker.serviceRequires=docker.service [Service]WorkingDirectory=/var/lib/kubeletEnvironmentFile=-/etc/kubernetes/configEnvironmentFile=-/etc/kubernetes/kubelet#ExecStart=/usr/bin/kubelet \#           $KUBE_LOGTOSTDERR \#           $KUBE_LOG_LEVEL \#           $KUBELET_API_SERVER \#           $KUBELET_ADDRESS \#           $KUBELET_PORT \#           $KUBELET_HOSTNAME \#           $KUBE_ALLOW_PRIV \#           $KUBELET_POD_INFRA_CONTAINER \#           $KUBELET_ARGS ExecStart=/usr/bin/kubelet --allow-privileged=true \        --logtostderr=false \        --v=6 \        --log-dir=/var/log/k8s/kubelet \        --address=0.0.0.0 \        --cluster-dns=10.0.1.10 \        --hostname-override=k8s-slave \        --cluster-domain=cluster.local \        --kubeconfig=/etc/kubernetes/kubelet.conf \        --pod-manifest-path=/etc/kubernetes/manifest \        --allow-privileged=true \        --authorization-mode=AlwaysAllow \        --fail-swap-on=false \        --cgroup-driver=systemd \        --pod-infra-container-image=registry.aliyuncs.com/archon/pause-amd64:3.0 Restart=on-failure [Install]WantedBy=multi-user.target

在etc/profile文件末尾添加以下内容：

export KUBERNETES_MASTER=http://x.x.x.x:8080

其中 xx.xx.xx.xx   为master eth0网卡IP,需要替换。

for SERVICES in kube-proxy kubelet docker; dosystemctl restart $SERVICESsystemctl enable $SERVICESsystemctl status $SERVICESdone

   运行启动脚本，此时所有服务可以正常启动。

   验证服务状态：

[root@k8s-slave home]# kubectl get nodesNAME         STATUS    ROLES     AGE       VERSIONk8s-master   Ready     <none>    3h        v1.9.0k8s-slave    Ready     <none>    2h        v1.9.0

   至此集群部署完毕。