kubernetes 完全二进制安装部署

1. 网络结构，准备工作master: 192.168.17.4node1:192.168.17.5node2:192.168.17.6在上面的三台机器上安装好docker，并配置好私有库，上传pod-infrastructure镜像(这个后面搭建k8s需要）## 1. 安装docker,使用yum方式安装或是二进制安装，这里使用yum安装yum search...

zhongyoubing

756人浏览 · 2018-12-05 16:46:15

zhongyoubing · 2018-12-05 16:46:15 发布

1. 网络结构，准备工作

master: 192.168.17.4
node1:  192.168.17.5
node2:  192.168.17.6

在上面的三台机器上安装好docker，并配置好私有库，上传pod-infrastructure镜像(这个后面搭建k8s需要）

## 1. 安装docker,使用yum方式安装或是二进制安装，这里使用yum安装
yum search docker|grep docker 
yum install docker.x86_64 

## 2. 配置私有库，使用docker container的方式配置

docker pull registry
docker run -d -p 5000:5000 --restart=always --name registry registry

配资docker私有库,私有库使用https方式，不配置无法push和pull 
vim /etc/docker/daemon.json
{
    "insecure-registries":[IP_address:5000"]
}
## 3.上传pod-infrastructure镜像
docker search pod-infrastructure 
选择需要的镜像，拉取下来，这里以docker.io/openshift/origin-pod 为例
docker pull docker.io/openshift/origin-pod 
docker tag docker.io/openshift/origin-pod  local_ip:5000/origin-pod
doker push local_ip:5000/origin-pod

2. 创建加密通信相关的配置文件，在master上操作

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
chmod +x cfssl_linux-amd64
sudo mv cfssl_linux-amd64 /usr/local/bin/cfssl

wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
chmod +x cfssljson_linux-amd64
sudo mv cfssljson_linux-amd64 /usr/local/bin/cfssljson

wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl-certinfo_linux-amd64
sudo mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo

创建文件夹

mkdir /etc/kubernetes/ssl -p
cd /etc/kubernetes/ssl

创建ca.pem， ca-key,pem

cat >> ca-config.json <<EOF 
{
    "signing": {
        "default": {
            "expiry": "8760h"
        },
        "profiles": {
            "kubernetes": {
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ],
                "expiry": "8760h"
            }
        }
    }
}

ca-config.json：可以定义多个profiles，分别指定不同的过期时间、使用场景等参数；后续在签名证书时使用某个profile
signing：表示该证书可用于签名其它证书；生成的ca.pem证书中CA=TRUE
server auth：表示client可以用该CA对server提供的证书进行验证
client auth：表示server可以用该CA对client提供的证书进行验证

ca证书申请表：

cat <<EOF > ca-csr.json
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "BeiJing",
            "L": "BeiJing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}

“CN”：Common Name，kube-apiserver从证书中提取该字段作为申请的用户名(User Name)；浏览器使用该字段验证网站是否合法
“O”：Organization，kube-apiserver从证书中提取该字段作为申请用户所属的组 (Group)

生成ca.pem/ca-key.pem

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

<<'COMMENT'
2017/04/21 14:41:50 [INFO] generating a new CA key and certificate from CSR
2017/04/21 14:41:50 [INFO] generate received request
2017/04/21 14:41:50 [INFO] received CSR
2017/04/21 14:41:50 [INFO] generating key: rsa-2048
2017/04/21 14:41:50 [INFO] encoded CSR
2017/04/21 14:41:50 [INFO] signed certificate with serial number 251797296407837937517157206505247063834323020724
COMMENT


ls ca*
<<'COMMENT'
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem
COMMENT

创建etcd.pem和etcd-key.pem

etcd证书申请表

cat > etcd-csr.json <<EOF
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "192.168.17.4",
    "192.168.17.5",
    "192.168.17.6"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

hosts 字段分别指定了etcd集群(192.168.17.4/192.168.17.5/192.168.17.6)、k8s-master的IP(192.168.1.171)

生成etcd.pem/etcd-key.pem

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd

<<'COMMENT'
2017/05/12 14:13:51 [INFO] generate received request
2017/05/12 14:13:51 [INFO] received CSR
2017/05/12 14:13:51 [INFO] generating key: rsa-2048
2017/05/12 14:13:52 [INFO] encoded CSR
2017/05/12 14:13:52 [INFO] signed certificate with serial number 513529097933554910472311565391610507353262197901
2017/05/12 14:13:52 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
COMMENT

创建kubernetes.pem和kubernetes-key.pem

kubernetes证书申请表

cat <<EOF > kubernetes-csr.json
{
  "CN": "kubernetes",
  "hosts": [
    "127.0.0.1",
    "192.168.17.4",
    "10.254.0.1",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

hosts 字段分别指定了k8s-master的IP(192.168.1.171)
添加 kube-apiserver注册的名为kubernetes的服务IP(Service Cluster IP)，一般是kube-apiserver --service-cluster-ip-range选项值指定的网段的第一个IP，如 “10.254.0.1”

生成kubernetes.pem/kubernetes-key.pem

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes

<<'COMMENT'
2017/04/21 14:52:32 [INFO] generate received request
2017/04/21 14:52:32 [INFO] received CSR
2017/04/21 14:52:32 [INFO] generating key: rsa-2048
2017/04/21 14:52:32 [INFO] encoded CSR
2017/04/21 14:52:32 [INFO] signed certificate with serial number 675534892777997310707325450009893653396769335719
2017/04/21 14:52:32 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

创建admin.pem和admin-key.pem

admin证书申请表

cat <<EOF > admin-csr.json
{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}

后续kube-apiserver使用RBAC(Role-Based Access Control)对客户端(如kubelet、kube-proxy、Pod)请求进行授权
kube-apiserver预定义了一些RBAC使用的RoleBindings，如cluster-admin将Group system:masters与Role cluster-admin绑定，该Role授予了调用kube-apiserver所有 API的权限
OU指定该证书的Group为system:masters，kubelet使用该证书访问kube-apiserver 时，由于证书被CA签名，所以认证通过，同时由于证书用户组为经过预授权的 system:masters，所以被授予访问所有API的权限

生成admin.pem/admin-key.pem

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
<<'COMMENT'
2017/04/21 14:58:44 [INFO] generate received request
2017/04/21 14:58:44 [INFO] received CSR
2017/04/21 14:58:44 [INFO] generating key: rsa-2048
2017/04/21 14:58:45 [INFO] encoded CSR
2017/04/21 14:58:45 [INFO] signed certificate with serial number 592438256014219038650472041230298814450491905528
2017/04/21 14:58:45 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
COMMENT

创建kube-proxy.pem和kube-proxy-key.pem

kube-proxy证书申请表

cat <<EOF > kube-proxy-csr.json
{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

CN 指定该证书的User为system:kube-proxy
kube-apiserver预定义的RoleBinding system:node-proxier将User system:kube-proxy 与Role system:node-proxier绑定，该Role授予了调用kube-apiserver Proxy相关API的权限

生成kube-proxy.pem/kube-proxy-key.pem

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes  kube-proxy-csr.json | cfssljson -bare kube-proxy
<<'COMMENT'
2017/04/21 15:08:44 [INFO] generate received request
2017/04/21 15:08:44 [INFO] received CSR
2017/04/21 15:08:44 [INFO] generating key: rsa-2048
2017/04/21 15:08:45 [INFO] encoded CSR
2017/04/21 15:08:45 [INFO] signed certificate with serial number 290129049837776761536725457428661161889494017049
2017/04/21 15:08:45 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
COMMENT

将生成的整数分发给其余的结点：

scp *.pem 192.168.17.5:/etc/kubernetes/ssl/
scp *.pem 192.168.17.6:/etc/kubernetes/ssl/

验证证书可用性

以kubernentes.pem为例

利用openssl验证

openssl x509  -noout -text -in  kubernetes.pem
<<'COMMENT'
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            76:54:08:41:9c:14:91:ce:23:59:d8:db:d5:39:66:37:67:85:e9:a7
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=CN, ST=BeiJing, L=BeiJing, O=k8s, OU=System, CN=kubernetes
        Validity
            Not Before: Apr 21 06:48:00 2017 GMT
            Not After : Apr 21 06:48:00 2018 GMT
        Subject: C=CN, ST=BeiJing, L=BeiJing, O=k8s, OU=System, CN=kubernetes
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
...
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier: 
                24:FA:8A:54:54:39:D3:65:21:3F:80:E7:5C:B8:4C:F8:B9:21:B4:B0
            X509v3 Authority Key Identifier: 
                keyid:0F:64:BF:83:F1:43:0F:32:0A:E1:D8:90:7D:C6:49:7B:59:00:95:84

            X509v3 Subject Alternative Name: 
                DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster, DNS:kubernetes.default.svc.cluster.local, IP Address:127.0.0.1, IP Address:192.168.1.171, IP Address:192.168.1.175, IP Address:192.168.1.176, IP Address:192.168.1.177, IP Address:10.254.0.1
    Signature Algorithm: sha256WithRSAEncryption

3. 安装 etcd

去官网下载：etcd-v3.3.10-linux-amd64.tar.gz，flannel-v0.10.0-linux-amd64.tar.gz ，kubernetes.tar.gz

解压后，将etcd, etcdctl 复制到所有的结点/usr/local/bin下，将flanneld ，mk-docker-opts.sh 复制到所有结点/usr/local/bin下

将kubelet, kubectl，kube-apiserver，kube-controller-manager, kube-scheduler复制到master结点的/usr/local/bin下，将kubelet, kubectl，kube-proxy复制node结点

安装之前需要在master和nodes中如下配置，否者网络会出错：

# 关闭防火墙
sudo systemctl stop firewalld
sudo systemctl disable firewalld

#关闭 swapoff 
swapoff -a

master中etcd配置如下：

cat /usr/lib/systemd/system/etcd.service

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/local/bin/etcd \
  --name=etcd1 \
  --cert-file=/etc/kubernetes/ssl/etcd.pem \
  --key-file=/etc/kubernetes/ssl/etcd-key.pem \
  --peer-cert-file=/etc/kubernetes/ssl/etcd.pem \
  --peer-key-file=/etc/kubernetes/ssl/etcd-key.pem \
  --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
  --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
  --initial-advertise-peer-urls=https://192.168.17.4:2380 \
  --listen-peer-urls=https://192.168.17.4:2380 \
  --listen-client-urls=https://192.168.17.4:2379,http://127.0.0.1:2379 \
  --advertise-client-urls=https://192.168.17.4:2379 \
  --initial-cluster-token=etcd-cluster-0 \
  --initial-cluster=etcd1=https://192.168.17.4:2380,etcd2=https://192.168.17.5:2380,etcd3=https://192.168.17.6:2380 \
  --initial-cluster-state=new \
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

其余两个node的结点配置类似

启动etcd

在master和node中启动etcd

systemctl daemon-reload
systemctl enable etcd
systemctl start etcd

检查etcd集群情况：

etcdctl --ca-file=/etc/kubernetes/ssl/ca.pem --cert-file=/etc/kubernetes/ssl/etcd.pem --key-file=/etc/kubernetes/ssl/etcd-key.pem cluster-health

4. 安装flanneld

master和nodes的flannel配置如下：

[root@i kubernetes]# pwd
/etc/kubernetes
[root@i kubernetes]# cat flannel 
# Flanneld configuration options  
# etcd url location.  Point this to the server where etcd runs
ETCD_ENDPOINTS="https://192.168.17.4:2379,https://192.168.17.5:2379,https://192.168.17.6:2379"
# etcd config key.  This is the configuration key that flannel queries
# For address range assignment
ETCD_PREFIX="/kubernetes/network"
# Any additional options that you want to pass
FLANNEL_OPTIONS="-etcd-cafile=/etc/kubernetes/ssl/ca.pem -etcd-certfile=/etc/kubernetes/ssl/etcd.pem -etcd-keyfile=/etc/kubernetes/ssl/etcd-key.pem -iface=eth2"   

#iface需要根据机器的实际情况配置

flanneld.service配置：

 vim /usr/lib/systemd/system/flanneld.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service
[Service]
Type=notify
EnvironmentFile=/etc/kubernetes/flannel
ExecStart=/usr/local/bin/flanneld \
  -etcd-endpoints=${ETCD_ENDPOINTS} \
  -etcd-prefix=${ETCD_PREFIX} \
  $FLANNEL_OPTIONS
ExecStartPost=/usr/local/bin/mk-docker-opts.sh -d /run/flannel/docker
Restart=on-failure
LimitNOFILE=1000000
LimitNPROC=1000000
LimitCORE=1000000
[Install]
WantedBy=multi-user.target
RequiredBy=docker.service

master上操作或其中任意一台，只操作一次，在etcd中创建网络配置：

etcdctl --endpoints=https://192.168.17.4:2379,https://192.168.17.5:2379,https://192.168.17.6:2379 \
  --ca-file=/etc/kubernetes/ssl/ca.pem \
  --cert-file=/etc/kubernetes/ssl/kubernetes.pem \
  --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
  set /kubernetes/network/config '{"Network":"172.30.0.0/16","SubnetLen":24,"Backend":{ "Type": "vxlan", "VNI": 1 }}'

在master及node上启动flanneld

systemctl daemon-reload
systemctl enable flanneld
systemctl start flanneld

任何一台上面验证节点：

[root@localhost kubernetes]# etcdctl --endpoints=https://192.168.17.4:2379,https://192.168.17.5:2379,https://192.168.17.6:2379   --ca-file=/etc/kubernetes/ssl/ca.pem   --cert-file=/etc/kubernetes/ssl/kubernetes.pem   --key-file=/etc/kubernetes/ssl/kubernetes-key.pem get /kubernetes/network/config
{"Network":"172.30.0.0/16","SubnetLen":24,"Backend":{ "Type": "vxlan", "VNI": 1 }}

[root@localhost kubernetes]# etcdctl --endpoints=https://192.168.17.4:2379,https://192.168.17.5:2379,https://192.168.17.6:2379   --ca-file=/etc/kubernetes/ssl/ca.pem   --cert-file=/etc/kubernetes/ssl/kubernetes.pem   --key-file=/etc/kubernetes/ssl/kubernetes-key.pem ls /kubernetes/network/subnets
/kubernetes/network/subnets/172.30.49.0-24
/kubernetes/network/subnets/172.30.29.0-24
/kubernetes/network/subnets/172.30.1.0-24

5. kubeconfig 生成

kubeconfig文件记录k8s集群的各种信息，对集群构建非常重要。

kubectl命令行工具从~/.kube/config，即kubectl的kubeconfig文件中获取访问kube-apiserver的地址，证书和用户名等信息
kubelet/kube-proxy等在Node上的程序进程同样通过bootstrap.kubeconfig和kube-proxy.kubeconfig上提供的认证与授权信息与Master进行通讯

kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER}

在Master上执行，即192.168.17.4

生成的文件在~/.kube/config下

Kubectl kubeconfig

声明kube apiserver

export KUBE_APISERVER="https://192.168.17.4:6443"

设置集群参数：

kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER}

设置客户端认证参数：

kubectl config set-credentials admin \
  --client-certificate=/etc/kubernetes/ssl/admin.pem \
  --embed-certs=true \
  --client-key=/etc/kubernetes/ssl/admin-key.pem

设置上下文参数：

kubectl config set-context kubernetes \
  --cluster=kubernetes \
  --user=admin

设置默认上下文：

kubectl config use-context kubernetes

Kubelet kubeconfig

声明kube apiserver

export KUBE_APISERVER="https://192.168.17.4:6443"

设置集群参数：

cd /etc/kubernetes
kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=bootstrap.kubeconfig

设置客户端认证参数

cd /etc/kubernetes
kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=bootstrap.kubeconfig

设置上下文：

cd /etc/kubernetes
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kubelet-bootstrap \
  --kubeconfig=bootstrap.kubeconfig

设置默认上下文：

cd /etc/kubernetes
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

Kube-proxy kubeconfig

生成Token认证文件：

cd /etc/kubernetes
cat > token.csv <<EOF
${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF

声明kube apiserver

export KUBE_APISERVER="https://192.168.17.4:6443"

设置集群参数：

cd /etc/kubernetes
kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=kube-proxy.kubeconfig

设置客户端认证参数：

cd /etc/kubernetes
kubectl config set-credentials kube-proxy \
  --client-certificate=/etc/kubernetes/ssl/kube-proxy.pem \
  --client-key=/etc/kubernetes/ssl/kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig

设置上下文参数：

cd /etc/kubernetes
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig

设置默认上下文：

cd /etc/kubernetes
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

将生产的kubeconfig分发到其余的结点上：

scp ~/.kube/config 192.168.17.5:/etc/kubernetes/kubelet.kubeconfig
scp /etc/kubernetes/bootstrap.kubeconfig 192.168.17.5:/etc/kubernetes/bootstrap.kubeconfig
scp /etc/kubernetes/kube-proxy.kubeconfig 192.168.17.5:/etc/kubernetes/kube-proxy.kubeconfig
scp ~/.kube/config 192.168.17.6:/etc/kubernetes/kubeconfig
scp /etc/kubernetes/bootstrap.kubeconfig 192.168.17.6:/etc/kubernetes/bootstrap.kubeconfig
scp /etc/kubernetes/kube-proxy.kubeconfig 192.168.17.6:/etc/kubernetes/kube-proxy.kubeconfig

6. master 搭建

通用配置文件如下：

[root@i kubernetes]# pwd 
/etc/kubernetes
[root@i kubernetes]# cat config
###
# kubernetes system config
#
# The following values are used to configure various aspects of all
# kubernetes services, including
#
#   kube-apiserver.service
#   kube-controller-manager.service
#   kube-scheduler.service
#   kubelet.service
#   kube-proxy.service
# logging to stderr means we get it in the systemd journal
KUBE_LOGTOSTDERR="--logtostderr=true"
# journal message level, 0 is debug
KUBE_LOG_LEVEL="--v=0"
# Should this cluster be allowed to run privileged docker containers
KUBE_ALLOW_PRIV="--allow-privileged=true"
# How the controller-manager, scheduler, and proxy find the apiserver
KUBE_MASTER="--master=http://192.168.17.4:8080"

kube-apiserver配置：

[root@i kubernetes]# pwd 
/etc/kubernetes
[root@i kubernetes]# cat kube-apiserver 
## The address on the local server to listen to.
KUBE_API_ADDRESS="--advertise-address=192.168.17.4 --bind-address=192.168.17.4 --insecure-bind-address=192.168.17.4"
#
## The port on the local server to listen on.
#KUBE_API_PORT="--port=8080"
#
## Port minions listen on
#KUBELET_PORT="--kubelet-port=10250"
#
## Comma separated list of nodes in the etcd cluster
KUBE_ETCD_SERVERS="--etcd-servers=https://192.168.17.4:2379,https://192.168.17.5:2379,https://192.168.17.6:2379"
#
## Address range to use for services
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=172.31.0.0/16"
#
## default admission control policies
#KUBE_ADMISSION_CONTROL="--admission-control=ServiceAccount,NamespaceLifecycle,NamespaceExists,LimitRanger,ResourceQuota"
KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota,NodeRestriction"
#
## Add your own!
KUBE_API_ARGS=" --enable-bootstrap-token-auth \
                --authorization-mode=RBAC,Node \
                --runtime-config=rbac.authorization.k8s.io/v1 \
                --kubelet-https=true \
                --token-auth-file=/etc/kubernetes/token.csv \
                --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem \
                --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
                --client-ca-file=/etc/kubernetes/ssl/ca.pem \
                --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \
                --etcd-cafile=/etc/kubernetes/ssl/ca.pem \
                --etcd-certfile=/etc/kubernetes/ssl/etcd.pem \
                --etcd-keyfile=/etc/kubernetes/ssl/etcd-key.pem \
                --enable-swagger-ui=true \
                --apiserver-count=3 \
                --audit-log-maxage=30 \
                --audit-log-maxbackup=3 \
                --audit-log-maxsize=100 \
                --audit-log-path=/var/lib/audit.log \
                --event-ttl=1h"

kube-apiserver.service：

[root@i kubernetes]# cat kube-apiserver 
## The address on the local server to listen to.
KUBE_API_ADDRESS="--advertise-address=192.168.17.4 --bind-address=192.168.17.4 --insecure-bind-address=192.168.17.4"
#
## The port on the local server to listen on.
#KUBE_API_PORT="--port=8080"
#
## Port minions listen on
#KUBELET_PORT="--kubelet-port=10250"
#
## Comma separated list of nodes in the etcd cluster
KUBE_ETCD_SERVERS="--etcd-servers=https://192.168.17.4:2379,https://192.168.17.5:2379,https://192.168.17.6:2379"
#
## Address range to use for services
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=172.31.0.0/16"
#
## default admission control policies
#KUBE_ADMISSION_CONTROL="--admission-control=ServiceAccount,NamespaceLifecycle,NamespaceExists,LimitRanger,ResourceQuota"
KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota,NodeRestriction"
#
## Add your own!
KUBE_API_ARGS=" --enable-bootstrap-token-auth \
                --authorization-mode=RBAC,Node \
                --runtime-config=rbac.authorization.k8s.io/v1 \
                --kubelet-https=true \
                --token-auth-file=/etc/kubernetes/token.csv \
                --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem \
                --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
                --client-ca-file=/etc/kubernetes/ssl/ca.pem \
                --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \
                --etcd-cafile=/etc/kubernetes/ssl/ca.pem \
                --etcd-certfile=/etc/kubernetes/ssl/etcd.pem \
                --etcd-keyfile=/etc/kubernetes/ssl/etcd-key.pem \
                --enable-swagger-ui=true \
                --apiserver-count=3 \
                --audit-log-maxage=30 \
                --audit-log-maxbackup=3 \
                --audit-log-maxsize=100 \
                --audit-log-path=/var/lib/audit.log \
                --event-ttl=1h"
[root@i kubernetes]# cat /usr/lib/systemd/system/kube-apiserver.service 
[Unit]
Description=Kubernetes API Service
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
After=etcd.service
[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/kube-apiserver
ExecStart=/usr/local/bin/kube-apiserver \
        $KUBE_LOGTOSTDERR \
        $KUBE_LOG_LEVEL \
        $KUBE_ETCD_SERVERS \
        $KUBE_API_ADDRESS \
        $KUBE_API_PORT \
        $KUBELET_PORT \
        $KUBE_ALLOW_PRIV \
        $KUBE_SERVICE_ADDRESSES \
        $KUBE_ADMISSION_CONTROL \
        $KUBE_API_ARGS
Restart=on-failure
RestartSec=15
Type=notify
LimitNOFILE=1000000
LimitNPROC=1000000
LimitCORE=1000000
[Install]
WantedBy=multi-user.target

kube-controller-manager配置：

[root@i kubernetes]# pwd
/etc/kubernetes
[root@i kubernetes]# cat kube-controller-manager 
# The following values are used to configure the kubernetes controller-manager
# defaults from config and apiserver should be adequate
# Add your own!
KUBE_CONTROLLER_MANAGER_ARGS="--address=127.0.0.1 \
                              --service-cluster-ip-range=172.31.0.0/16 \
                              --cluster-cidr=172.30.0.0/16 \
                              --allocate-node-cidrs=true \
                              --cluster-name=kubernetes \
                              --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \
                              --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem  \
                              --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \
                              --root-ca-file=/etc/kubernetes/ssl/ca.pem \
                              --leader-elect=true \
                              --v=2 \
--horizontal-pod-autoscaler-use-rest-clients=false"

kube-controller-manager.service:

[root@i kubernetes]# cat /usr/lib/systemd/system/kube-controller-manager.service 
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/kube-controller-manager
ExecStart=/usr/local/bin/kube-controller-manager \
        $KUBE_LOGTOSTDERR \
        $KUBE_LOG_LEVEL \
        $KUBE_MASTER \
        $KUBE_CONTROLLER_MANAGER_ARGS
Restart=on-failure
LimitNOFILE=1000000
LimitNPROC=1000000
LimitCORE=1000000
[Install]
WantedBy=multi-user.target

kube-scheduler配置：

[root@i kubernetes]# cat kube-scheduler 
# kubernetes scheduler config
# default config should be adequate
# Add your own!
KUBE_SCHEDULER_ARGS="--leader-elect=true \
                     --address=127.0.0.1"   
#此处是127.0.0.1, 不是IP地址，否则检测health会有问题

kube-scheduler.service:

[root@i kubernetes]# cat /usr/lib/systemd/system/kube-scheduler.service 
[Unit]
Description=Kubernetes Scheduler Plugin
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/kube-scheduler
ExecStart=/usr/local/bin/kube-scheduler \
            $KUBE_LOGTOSTDERR \
            $KUBE_LOG_LEVEL \
            $KUBE_MASTER \
            $KUBE_SCHEDULER_ARGS
Restart=on-failure
LimitNOFILE=1000000
LimitNPROC=1000000
LimitCORE=1000000
[Install]
WantedBy=multi-user.target

在master上启动上述服务：

for i in kube-apiserver kube-controller-manager kube-scheduler
do
    systemctl daemon-reload
	systemctl start $i
	systemctl enable $i
done

7 node的搭建

通用配置文件config:

[root@localhost kubernetes]# pwd 
/etc/kubernetes
[root@localhost kubernetes]# cat config 
# kubernetes system config
#
# The following values are used to configure various aspects of all
# kubernetes services, including
#
#   kube-apiserver.service
#   kube-controller-manager.service
#   kube-scheduler.service
#   kubelet.service
#   kube-proxy.service
# logging to stderr means we get it in the systemd journal
KUBE_LOGTOSTDERR="--logtostderr=true"
# journal message level, 0 is debug
KUBE_LOG_LEVEL="--v=0"
# Should this cluster be allowed to run privileged docker containers
KUBE_ALLOW_PRIV="--allow-privileged=true"
# How the controller-manager, scheduler, and proxy find the apiserver
KUBE_MASTER="--master=http://192.168.17.4:8080"

kubelet配置：

[root@localhost kubernetes]# cat kubelet
## kubernetes kubelet (minion) config
#
## The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--address=192.168.17.5"
#
## The port for the info server to serve on
#
## You may leave this blank to use the actual hostname
KUBELET_HOSTNAME="--hostname-override=192.168.17.5"
#
## pod infrastructure container
KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=192.168.17.5:5000/origin-pod:latest"
# 此处的pod-infra-container-image就是私有库中的origin-pod
#
## Add your own!
# 需要指定--cgroup-driver为systemd
KUBELET_ARGS="--cgroup-driver=systemd \
	      --cluster-dns=172.31.0.2 \
              --bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig \
              --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
              --cert-dir=/etc/kubernetes/ssl \
              --cluster-domain=cluster.local \
              --hairpin-mode promiscuous-bridge \
              --serialize-image-pulls=false \
              --container-runtime=docker \
              --register-node \
              --tls-cert-file=/etc/kubernetes/ssl/ca.pem \
              --tls-private-key-file=/etc/kubernetes/ssl/ca-key.pem \
              --root-dir=/data/kubelet"

[root@localhost kubernetes]# cat /usr/lib/systemd/system/kubelet.service 
[Unit]
Description=Kubernetes Kubelet Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service
[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/kubelet
ExecStart=/usr/local/bin/kubelet \
            $KUBE_LOGTOSTDERR \
            $KUBE_LOG_LEVEL \
            $KUBELET_API_SERVER \
            $KUBELET_ADDRESS \
            $KUBELET_PORT \
            $KUBELET_HOSTNAME \
            $KUBE_ALLOW_PRIV \
            $KUBELET_POD_INFRA_CONTAINER \
            $KUBELET_ARGS
Restart=on-failure
LimitNOFILE=1000000
LimitNPROC=1000000
LimitCORE=1000000
[Install]
WantedBy=multi-user.target

kube-proxy配置：

root@localhost kubernetes]# cat kube-proxy
# kubernetes proxy config
# default config should be adequate
# Add your own!
KUBE_PROXY_ARGS="--bind-address=192.168.17.5 \
                 --hostname-override=192.168.17.5 \
                 --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig \
                 --cluster-cidr=172.31.0.0/16"
[root@localhost kubernetes]# cat /usr/lib/systemd/system/kubelet-proxy
cat: /usr/lib/systemd/system/kubelet-proxy: No such file or directory
[root@localhost kubernetes]# cat kube-proxy
# kubernetes proxy config
# default config should be adequate
# Add your own!
KUBE_PROXY_ARGS="--bind-address=192.168.17.5 \
                 --hostname-override=192.168.17.5 \
                 --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig \
                 --cluster-cidr=172.31.0.0/16"
[root@localhost kubernetes]# cat /usr/lib/systemd/system/kube-proxy.service 
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/kube-proxy
ExecStart=/usr/local/bin/kube-proxy \
        $KUBE_LOGTOSTDERR \
        $KUBE_LOG_LEVEL \
        $KUBE_MASTER \
        $KUBE_PROXY_ARGS
Restart=on-failure
LimitNOFILE=1000000
LimitNPROC=1000000
LimitCORE=1000000
[Install]
WantedBy=multi-user.target

所有的node配置相似，在node上启动：

systemctl daemon-reload
systemctl enable kubelet
systemctl enable kube-proxy
systemctl start kubelet
systemctl start kube-proxy

检查集群：

#master上操作：
kubectl get nodes
[root@i k8s]# kubectl get nodes
NAME           STATUS    ROLES     AGE       VERSION
192.168.17.5   Ready     <none>    7h        v1.11.4
192.168.17.6   Ready     <none>    20h       v1.11.4

8. 检查集群可用性：

在master上操作

[root@i kubernetes_app]# cat nginx-ds.yaml 
apiVersion: v1
kind: Service
metadata:
  name: nginx-ds
  labels:
    app: nginx-ds
spec:
  type: NodePort
  selector:
    app: nginx-ds
  ports:
  - name: http
    port: 80
    targetPort: 80

---

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: nginx-ds
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  template:
    metadata:
      labels:
        app: nginx-ds
    spec:
      containers:
      - name: my-nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80

创建pod和svc:

kubectl create -f nginx-ds.yaml

[root@i kubernetes_app]# kubectl get pods --all-namespaces -o wide
NAMESPACE   NAME             READY     STATUS    RESTARTS   AGE       IP           NODE           NOMINATED NODE
default     nginx-ds-86dj5   1/1       Running   0          2h        172.17.0.3   192.168.17.5   <none>
default     nginx-ds-j8dn9   1/1       Running   0          2h        172.17.0.3   192.168.17.6   <none>
[root@i kubernetes_app]# kubectl get svc --all-namespaces
NAMESPACE   NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
default     kubernetes   ClusterIP   172.31.0.1      <none>        443/TCP        23h
default     nginx-ds     NodePort    172.31.128.30   <none>        80:30730/TCP   2h

在任何一台node上可以访问网站：

[root@localhost kubernetes]# curl 172.31.128.30
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

参考：https://blog.linuxeye.cn/458.html

https://o-my-chenjian.com/2017/04/26/Deploy-Node-Of-K8s/

https://blog.csdn.net/A632189007/article/details/78730903

K8S/Kubernetes

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐

【深度】阿里巴巴万级规模 K8s 集群全局高可用体系之美

作者 | 韩堂、柘远、沉醉来源 | 阿里巴巴云原生公众号前言台湾作家林清玄在接受记者采访的时候，如此评价自己 30 多年写作生涯：“第一个十年我才华横溢，‘贼光闪现’，令周边黯然失色；第二个十年，我终于‘宝光现形’，不再去抢风头，反而与身边的美丽相得益彰；进入第三个十年，繁华落尽见真醇，我进入了‘醇光初现’的阶段，真正体味到了境界之美”。长夜有穷，真水无香。领略过了 K8s“身在江

K8S/Kubernetes

如何基于 K8s 构建下一代 DevOps 平台？

作者 | 孙健波（天元）导读：当前云原生 DevOps 体系现状如何？面临哪些挑战？如何通过 OAM 解决云原生 DevOps 场景下的诸多问题？云原生开发应用模型 OAM(Open Application Model) 社区核心成员孙健波将为大家一一解答，并分享如何基于 OAM 和 Kubernetes 打造无限能力的下一代 DevOps 平台。什么是 DevOps？为什么基于 Kub