K8s Master 三个组件:

• kube-apiserver
• kube-controller-manager
• kube-scheduler

kubernetes master 节点运行如下组件： kube-apiserver kube-scheduler kube-controller-manager. 其中kube-scheduler 和 kube-controller-manager 可以以集群模式运行，通过 leader 选举产生一个工作进程，其它进程处于阻塞模式，master高可用模式下可用

github 官网

https://github.com/kubernetes/kubernetes

基本流程图

基本功能图

安装步骤

• 下载文件
• 制作证书
• 创建TLS Bootstrapping Token
• 部署apiserver组件
• 部署kube-scheduler组件
• 部署kube-controller-manager组件
• 验证服务

下载文件,解压缩

wget https://dl.k8s.io/v1.13.6/kubernetes-server-linux-amd64.tar.gz
[root@k8s ~]# mkdir /opt/kubernetes/{bin,cfg,ssl} -p
[root@k8s ~]# tar zxf kubernetes-server-linux-amd64.tar.gz 
[root@k8s ~]# cd kubernetes/server/bin/
[root@k8s bin]# cp kube-scheduler kube-apiserver kube-controller-manager /opt/kubernetes/bin/
[root@k8s bin]# cp kubectl /usr/bin/
[root@k8s bin]# 

制作kubernetes ca证书

• 制作ca配置文件

cd  /opt/kubernetes/ssl/
cat << EOF | tee /opt/kubernetes/ssl/ca-config.json
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF


cat << EOF | tee /opt/kubernetes/ssl/ca-csr.json
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

• 生成ca证书

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
[root@k8s ssl]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
2019/05/28 14:47:18 [INFO] generating a new CA key and certificate from CSR
2019/05/28 14:47:18 [INFO] generate received request
2019/05/28 14:47:18 [INFO] received CSR
2019/05/28 14:47:18 [INFO] generating key: rsa-2048
2019/05/28 14:47:18 [INFO] encoded CSR
2019/05/28 14:47:19 [INFO] signed certificate with serial number 34219464473634319112180195944445301722929678647
[root@k8s ssl]# 

• 制作apiserver证书

cat << EOF | tee server-csr.json
{
    "CN": "kubernetes",
    "hosts": [
      "10.0.0.1",
      "127.0.0.1",
      "10.0.52.13",
      "10.0.52.7",
      "10.0.52.8",
      "10.0.52.9",
      "10.0.52.10",
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

[root@k8s ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server
2019/05/28 15:03:31 [INFO] generate received request
2019/05/28 15:03:31 [INFO] received CSR
2019/05/28 15:03:31 [INFO] generating key: rsa-2048
2019/05/28 15:03:31 [INFO] encoded CSR
2019/05/28 15:03:31 [INFO] signed certificate with serial number 114040551556369232239873744650692828468613738631
2019/05/28 15:03:31 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@k8s ssl]# ls
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem  server.csr  server-csr.json  server-key.pem  server.pem
[root@k8s ssl]# 

server-csr.json文件中的hosts配置,需要"10.0.52.13", “10.0.52.7”, “10.0.52.8”, “10.0.52.9”, “10.0.52.10”,这几个IP是自己设置的,其他的都是内置的,无需改动,其中我们k8smaster的ip,如果高可用的话,几个master的ip,lb的ip都需要设置,否则无法连接apiserver.

创建TLS Bootstrapping Token

[root@k8s ssl]# BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ' ')
[root@k8s ssl]# cat << EOF | tee /opt/kubernetes/cfg/token.csv
> ${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap"
> EOF
7d558bb3a5206cf78f881de7d7b82ca6,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
[root@k8s ssl]# cat /opt/kubernetes/cfg/token.csv
7d558bb3a5206cf78f881de7d7b82ca6,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
[root@k8s ssl]# 

部署kube-apiserver组件

1. 创建kube-apiserver配置文件

Apiserver配置文件里面需要配置的有etcd-servers地址,bind-address,advertise-address都是当前master节点的IP地址.token-auth-file和各种pem认证文件,需要把对应的文件位置输入即可.

cat << EOF | tee /opt/kubernetes/cfg/kube-apiserver
KUBE_APISERVER_OPTS="--logtostderr=true \\
--v=4 \\
--etcd-servers=https://10.0.52.13:2379,https://10.0.52.14:2379,https://10.0.52.6:2379 \\
--bind-address=10.0.52.13 \\
--secure-port=6443 \\
--advertise-address=10.0.52.13 \\
--allow-privileged=true \\
--service-cluster-ip-range=10.0.0.0/24 \\
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\
--authorization-mode=RBAC,Node \\
--kubelet-https=true \\
--enable-bootstrap-token-auth \\
--token-auth-file=/opt/kubernetes/cfg/token.csv \\
--service-node-port-range=30000-50000 \\
--tls-cert-file=/opt/kubernetes/ssl/server.pem  \\
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \\
--client-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--etcd-cafile=/opt/etcd/ssl/ca.pem \\
--etcd-certfile=/opt/etcd/ssl/server.pem \\
--etcd-keyfile=/opt/etcd/ssl/server-key.pem"
EOF

2.创建apiserver systemd文件

cat << EOF | tee /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-apiserver
ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

3.启动服务

systemctl daemon-reload
systemctl enable kube-apiserver
systemctl start kube-apiserver

[root@k8s ssl]# ps -ef |grep kube-apiserver
root     19404     1 89 15:50 ?        00:00:09 /opt/kubernetes/bin/kube-apiserver --logtostderr=true --v=4 --etcd-servers=https://10.0.52.13:2379,https://10.0.52.14:2379,https://10.0.52.6:2379 --bind-address=10.0.52.13 --secure-port=6443 --advertise-address=10.0.52.13 --allow-privileged=true --service-cluster-ip-range=10.0.0.0/24 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction --authorization-mode=RBAC,Node --kubelet-https=true --enable-bootstrap-token-auth --token-auth-file=/opt/kubernetes/cfg/token.csv --service-node-port-range=30000-50000 --tls-cert-file=/opt/kubernetes/ssl/server.pem --tls-private-key-file=/opt/kubernetes/ssl/server-key.pem --client-ca-file=/opt/kubernetes/ssl/ca.pem --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem --etcd-cafile=/opt/etcd/ssl/ca.pem --etcd-certfile=/opt/etcd/ssl/server.pem --etcd-keyfile=/opt/etcd/ssl/server-key.pem
root     19418 19122  0 15:50 pts/1    00:00:00 grep --color=auto kube-apiserver
[root@k8s ssl]# systemctl status kube-apiserver
● kube-apiserver.service - Kubernetes API Server
   Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2019-05-28 15:50:21 CST; 26s ago
     Docs: https://github.com/kubernetes/kubernetes
 Main PID: 19404 (kube-apiserver)
   Memory: 221.2M
   CGroup: /system.slice/kube-apiserver.service
           └─19404 /opt/kubernetes/bin/kube-apiserver --logtostderr=true --v=4 --etcd-servers=https://10.0.52.13:2379,https://10.0.52.14:2379,https://10.0.52.6:2379 --bind-address=10.0.52.13 --secure-port=6443 --advertise-address=10.0.52.13 --allow-privileged=true --...

May 28 15:50:32 k8s.master kube-apiserver[19404]: I0528 15:50:32.057378   19404 wrap.go:47] GET /api/v1/namespaces/kube-system: (1.709711ms) 200 [kube-apiserver/v1.13.6 (linux/amd64) kubernetes/abdda3f 10.0.52.13:41238]
May 28 15:50:32 k8s.master kube-apiserver[19404]: I0528 15:50:32.076300   19404 wrap.go:47] POST /apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/roles: (2.984796ms) 201 [kube-apiserver/v1.13.6 (linux/amd64) kubernetes/abdda3f 10.0.52.13:41238]
May 28 15:50:32 k8s.master kube-apiserver[19404]: I0528 15:50:32.076874   19404 storage_rbac.go:246] created role.rbac.authorization.k8s.io/system:controller:bootstrap-signer in kube-system
May 28 15:50:32 k8s.master kube-apiserver[19404]: I0528 15:50:32.095073   19404 wrap.go:47] GET /apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/roles/system:controller:cloud-provider: (1.887241ms) 404 [kube-apiserver/v1.13.6 (linux/amd64)...f 10.0.52.13:41238]
May 28 15:50:32 k8s.master kube-apiserver[19404]: I0528 15:50:32.097100   19404 wrap.go:47] GET /api/v1/namespaces/kube-system: (1.654384ms) 200 [kube-apiserver/v1.13.6 (linux/amd64) kubernetes/abdda3f 10.0.52.13:41238]
May 28 15:50:32 k8s.master kube-apiserver[19404]: I0528 15:50:32.115586   19404 wrap.go:47] POST /apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/roles: (2.390436ms) 201 [kube-apiserver/v1.13.6 (linux/amd64) kubernetes/abdda3f 10.0.52.13:41238]
May 28 15:50:32 k8s.master kube-apiserver[19404]: I0528 15:50:32.115766   19404 storage_rbac.go:246] created role.rbac.authorization.k8s.io/system:controller:cloud-provider in kube-system
May 28 15:50:32 k8s.master kube-apiserver[19404]: I0528 15:50:32.134609   19404 wrap.go:47] GET /apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/roles/system:controller:token-cleaner: (1.458696ms) 404 [kube-apiserver/v1.13.6 (linux/amd64) ...f 10.0.52.13:41238]
May 28 15:50:32 k8s.master kube-apiserver[19404]: I0528 15:50:32.136356   19404 wrap.go:47] GET /api/v1/namespaces/kube-system: (1.420447ms) 200 [kube-apiserver/v1.13.6 (linux/amd64) kubernetes/abdda3f 10.0.52.13:41238]
May 28 15:50:32 k8s.master kube-apiserver[19404]: I0528 15:50:32.155628   19404 wrap.go:47] POST /apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/roles: (2.433057ms) 201 [kube-apiserver/v1.13.6 (linux/amd64) kubernetes/abdda3f 10.0.52.13:41238]
Hint: Some lines were ellipsized, use -l to show in full.

[root@k8s ssl]# ps -ef |grep -v grep |grep kube-apiserver 
root     19404     1  1 15:50 ?        00:00:25 /opt/kubernetes/bin/kube-apiserver --logtostderr=true --v=4 --etcd-servers=https://10.0.52.13:2379,https://10.0.52.14:2379,https://10.0.52.6:2379 --bind-address=10.0.52.13 --secure-port=6443 --advertise-address=10.0.52.13 --allow-privileged=true --service-cluster-ip-range=10.0.0.0/24 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction --authorization-mode=RBAC,Node --kubelet-https=true --enable-bootstrap-token-auth --token-auth-file=/opt/kubernetes/cfg/token.csv --service-node-port-range=30000-50000 --tls-cert-file=/opt/kubernetes/ssl/server.pem --tls-private-key-file=/opt/kubernetes/ssl/server-key.pem --client-ca-file=/opt/kubernetes/ssl/ca.pem --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem --etcd-cafile=/opt/etcd/ssl/ca.pem --etcd-certfile=/opt/etcd/ssl/server.pem --etcd-keyfile=/opt/etcd/ssl/server-key.pem
[root@k8s ssl]# 

[root@k8s ssl]# netstat -tulpn |grep kube-apiserve
tcp        0      0 10.0.52.13:6443         0.0.0.0:*               LISTEN      19404/kube-apiserve 
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      19404/kube-apiserve 
[root@k8s ssl]# 

部署kube-scheduler组件

1.创建kube-scheduler配置文件

–master: kube-scheduler 使用它连接 kube-apiserver； –leader-elect=true：集群运行模式，启用选举功能；被选为 leader 的节点负责处理工作，其它节点为阻塞状态；

cat << EOF | tee /opt/kubernetes/cfg/kube-scheduler
KUBE_SCHEDULER_OPTS="--logtostderr=true \\
--v=4 \\
--master=127.0.0.1:8080 \\
--leader-elect"
EOF

2.创建kube-scheduler systemd文件

cat << EOF | tee /usr/lib/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-scheduler
ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

3.启动&验证

systemctl daemon-reload
systemctl enable kube-scheduler
systemctl start kube-scheduler
[root@k8s ssl]# systemctl status kube-scheduler.service
● kube-scheduler.service - Kubernetes Scheduler
   Loaded: loaded (/usr/lib/systemd/system/kube-scheduler.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2019-05-28 16:06:21 CST; 9s ago
     Docs: https://github.com/kubernetes/kubernetes
 Main PID: 19524 (kube-scheduler)
   Memory: 10.8M
   CGroup: /system.slice/kube-scheduler.service
           └─19524 /opt/kubernetes/bin/kube-scheduler --logtostderr=true --v=4 --master=127.0.0.1:8080 --leader-elect

May 28 16:06:22 k8s.master kube-scheduler[19524]: I0528 16:06:22.942604   19524 shared_informer.go:123] caches populated
May 28 16:06:23 k8s.master kube-scheduler[19524]: I0528 16:06:23.042738   19524 shared_informer.go:123] caches populated
May 28 16:06:23 k8s.master kube-scheduler[19524]: I0528 16:06:23.142882   19524 shared_informer.go:123] caches populated
May 28 16:06:23 k8s.master kube-scheduler[19524]: I0528 16:06:23.243024   19524 shared_informer.go:123] caches populated
May 28 16:06:23 k8s.master kube-scheduler[19524]: I0528 16:06:23.243057   19524 controller_utils.go:1027] Waiting for caches to sync for scheduler controller
May 28 16:06:23 k8s.master kube-scheduler[19524]: I0528 16:06:23.343173   19524 shared_informer.go:123] caches populated
May 28 16:06:23 k8s.master kube-scheduler[19524]: I0528 16:06:23.343195   19524 controller_utils.go:1034] Caches are synced for scheduler controller
May 28 16:06:23 k8s.master kube-scheduler[19524]: I0528 16:06:23.343249   19524 leaderelection.go:205] attempting to acquire leader lease  kube-system/kube-scheduler...
May 28 16:06:23 k8s.master kube-scheduler[19524]: I0528 16:06:23.351601   19524 leaderelection.go:214] successfully acquired lease kube-system/kube-scheduler
May 28 16:06:23 k8s.master kube-scheduler[19524]: I0528 16:06:23.451916   19524 shared_informer.go:123] caches populated

部署kube-controller-manager组件

1.创建kube-controller-manager配置文件

cat << EOF | tee /opt/kubernetes/cfg/kube-controller-manager
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=true \\
--v=4 \\
--master=127.0.0.1:8080 \\
--leader-elect=true \\
--address=127.0.0.1 \\
--service-cluster-ip-range=10.0.0.0/24 \\
--cluster-name=kubernetes \\
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \\
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem  \\
--root-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--experimental-cluster-signing-duration=87600h0m0s"
EOF

2.创建kube-controller-manager systemd文件

cat << EOF | tee /usr/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-controller-manager
ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

3.启动&验证

systemctl daemon-reload
systemctl enable kube-controller-manager
systemctl start kube-controller-manager

[root@k8s ssl]# systemctl status kube-controller-manager
● kube-controller-manager.service - Kubernetes Controller Manager
   Loaded: loaded (/usr/lib/systemd/system/kube-controller-manager.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2019-05-28 16:18:52 CST; 10s ago
     Docs: https://github.com/kubernetes/kubernetes
 Main PID: 19606 (kube-controller)
   Memory: 31.9M
   CGroup: /system.slice/kube-controller-manager.service
           └─19606 /opt/kubernetes/bin/kube-controller-manager --logtostderr=true --v=4 --master=127.0.0.1:8080 --leader-elect=true --address=127.0.0.1 --service-cluster-ip-range=10.0.0.0/24 --cluster-name=kubernetes --cluster-signing-cert-file=/opt/kubernetes/ssl/ca...

May 28 16:18:55 k8s.master kube-controller-manager[19606]: I0528 16:18:55.140091   19606 controller_utils.go:1034] Caches are synced for garbage collector controller
May 28 16:18:55 k8s.master kube-controller-manager[19606]: I0528 16:18:55.140098   19606 garbagecollector.go:142] Garbage collector: all resource monitors have synced. Proceeding to collect garbage
May 28 16:18:55 k8s.master kube-controller-manager[19606]: I0528 16:18:55.168470   19606 request.go:530] Throttling request took 1.399047743s, request: GET:http://127.0.0.1:8080/apis/apiextensions.k8s.io/v1beta1?timeout=32s
May 28 16:18:55 k8s.master kube-controller-manager[19606]: I0528 16:18:55.169456   19606 resource_quota_controller.go:427] syncing resource quota controller with updated resources from discovery: map[/v1, Resource=replicationcontrollers:{} extensio...beta1, Resource=eve
May 28 16:18:55 k8s.master kube-controller-manager[19606]: I0528 16:18:55.169593   19606 resource_quota_monitor.go:180] QuotaMonitor unable to use a shared informer for resource "extensions/v1beta1, Resource=networkpolicies": no informer found for ...rce=networkpolicies
May 28 16:18:55 k8s.master kube-controller-manager[19606]: I0528 16:18:55.169632   19606 resource_quota_monitor.go:243] quota synced monitors; added 0, kept 29, removed 0
May 28 16:18:55 k8s.master kube-controller-manager[19606]: E0528 16:18:55.169647   19606 resource_quota_controller.go:437] failed to sync resource monitors: couldn't start monitor for resource "extensions/v1beta1, Resource=networkpolicies": unable ...ce=networkpolicies"
May 28 16:18:55 k8s.master kube-controller-manager[19606]: I0528 16:18:55.225106   19606 shared_informer.go:123] caches populated
May 28 16:18:55 k8s.master kube-controller-manager[19606]: I0528 16:18:55.225138   19606 controller_utils.go:1034] Caches are synced for garbage collector controller
May 28 16:18:55 k8s.master kube-controller-manager[19606]: I0528 16:18:55.225146   19606 garbagecollector.go:245] synced garbage collector
Hint: Some lines were ellipsized, use -l to show in full.
[root@k8s ssl]# 

验证master服务状态

[root@k8s ssl]# kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
controller-manager   Healthy   ok                  
scheduler            Healthy   ok                  
etcd-1               Healthy   {"health":"true"}   
etcd-0               Healthy   {"health":"true"}   
etcd-2               Healthy   {"health":"true"}   
[root@k8s ssl]# 

如果出现如上界面,表示Master安装成功!