K8S部署

1 安装说明

各个服务安装节点
作为集群的Master、服务环境控制相关模块、api网管控制相关模块、平台管理控制台模块
docker、etcd、api-server、scheduler、controller-manager、kubelet、flannel、docker-compose、harbor、kube-dns、kube- dashboard
服务节点、用于容器化服务部署和运行
docker、etcd、kubelet、proxy、flannel
该文档为单机部署，安装环境为64位CentOS7.3。
安装过程中可能会遇到提示没有安装依赖包，可以自己手动设置yum源，之后直接yum install 安装包 进行安装

1.1配置yum源
挂载 mount –t /dev/cdrom /mnt
创建文件夹 mkdir –p /usr/local/yum
将挂载镜像中的rpm文件夹复制到yum文件夹中

cp –r /mnt/Packages /usr/local/yum 
yum clean all
createrepo /usr/local/yum

配置yum.repo文件


```powershell
[local-resource]				--该库的名字
name=myrepo					--该库的说明
baseurl=file:///usr/local/yum	--该库的传输方式，具体路径 传输方式有(http://、ftp://、file:///)
enable=1					--启用该库，0表示不启动
gpgcheck=0					--使用gpg文件检查软件包的签名
gpgkey=					--表示gpg文件存放的位置(不用设置)
yum list  --拉取安装包列表
yum install 安装包  --rpm包的安装


1.2关闭防火墙和swap分区

```powershell
关闭防火墙 
systemctl stop firewalld.service		停止防火墙
systemctl disable firewalld.service	禁止防火墙开机启动
swapoff –a						关闭swap分区

2安装docker(每台)

上传以下rpm至服务器

强制安装docker及所需安装包
rpm -ivh --nodeps *rpm
启动docker并设置开机启动
systemctl daemon-reload
systemctl enable docker
systemctl start docker
systemctl status docker

3 创建kubernetes各个组件所需要的TLS通信证书和密钥

(密钥分发至每台)

3.1 生成文件列表

ca-key.pem、ca.pem、kubernetes-key.pem、kubernetes.pem、kube-proxy.pem、kube-proxy-key.pem、admin.pem、admin-key.pem

使用证书分类

etcd：ca.pem / kubernetes-key.pem / kubernetes.pem;
kube-apiserver：ca.pem / kubernetes-key.pem / kubernetes.pem;
kubelet：ca.pem;
kube-proxy：ca.pem / kube-proxy-key.pem / kube-proxy.pem; 
kubectl：ca.pem / admin-key.pem / admin.pem

3.2安装CFSSL

将cfssl_linux-amd64.bin、cfssl-certinfo_linux-amd64.bin、cfssljson_linux-amd64.bin移动文件至usr/local/bin下，并配置环境变量
执行以下命令
[root@localhost cfssl]# mv cfssl_linux-amd64.bin /usr/local/bin/cfssl
[root@localhost cfssl]# mv cfssljson_linux-amd64.bin /usr/local/bin/cfssljson
[root@localhost cfssl]# mv cfssl-certinfo_linux-amd64.bin /usr/local/bin/cfssl-certinfo
[root@localhost cfssl]# vi /etc/profile
添加export PATH=/root/local/bin:$PATH
[root@localhost ssl]# source /etc/profile	###立即生效
[root@localhost ssl]# chmod +x /usr/local/bin/cfssl*

3.3创建CA配置文件

[root@localhost /]# mkdir -p /root/ssl
[root@localhost /]# cd /root/ssl/
[root@localhost ssl]# cfssl print-defaults config > config.json
[root@localhost ssl]# cfssl print-defaults csr > csr.json
[root@localhost ssl]# vi ca-config.json
{
"signing":	{
"default":	{
"expiry": "8760h"
},
"profiles":	{
"kubernetes":	{
"usages":	[
"signing",
"key encipherment",
"server	auth",
"client	auth"
],
"expiry":"8760h"
}
}
}
}

3.3.1创建CA证书签名请求

[root@localhost ssl]# vi  ca-csr.json
{
"CN":"kubernetes",
"key":{
"algo":	"rsa",
"size":	2048
},
"names":[
{
"C":	"CN",
"ST":	"BeiJing",
"L":	"BeiJing",
"O":	"k8s",
"OU":	"System"
}
]
}

3.3.2生成CA证书和密钥

[root@localhost ssl]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca
[root@localhost ssl]# ls
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem  config.json  csr.json
创建kubernetes证书

3.4创建kubernetes证书签名请求

[root@localhost ssl]# vi kubernetes-csr.json
{
"CN":"kubernetes",
"hosts":[
"127.0.0.1",
"169.254.6.20",
"168.254.0.1",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo":"rsa",
"size":2048
},
"names":[
{
"C":"CN",
"ST":"BeiJing",
"L":"BeiJing",
"O":"k8s",
"OU":"System"
}
]
}

3.4.1生成kubernetes证书和密钥

[root@localhost ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem  -config=ca-config.json  -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
2019/04/12 20:28:07 [INFO] generate received request
2019/04/12 20:28:07 [INFO] received CSR
2019/04/12 20:28:07 [INFO] generating key: rsa-2048
2019/04/12 20:28:07 [INFO] encoded CSR
2019/04/12 20:28:08 [INFO] signed certificate with serial number 341419041177557148191513412510722414927398891570
2019/04/12 20:28:08 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

3.5创建admin证书

[root@localhost ssl]# cat admin-csr.json 
{
        "CN": "admin",
        "hosts": [],
        "key": {
                "algo": "rsa",
                "size": 2048
        },
        "names": [
         {
                "C": "CN",
                "ST": "BeiJing",
                "L": "BeiJing",
                "O": "system:masters",
                "OU": "System"
         }
        ]
}

3.5.1生成admin证书和密钥

[root@localhost ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
2019/04/12 20:33:19 [INFO] generate received request
2019/04/12 20:33:19 [INFO] received CSR
2019/04/12 20:33:19 [INFO] generating key: rsa-2048
2019/04/12 20:33:19 [INFO] encoded CSR
2019/04/12 20:33:19 [INFO] signed certificate with serial number 523319795135016522528190714496735510305838417368
2019/04/12 20:33:19 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

3.6创建kube-proxy证书

[root@localhost ssl]# cat kube-proxy-csr.json 
{
        "CN": "system:kube-proxy",
        "hosts": [],
        "key": {
                "algo": "rsa",
                "size": 2048
        },
        "names": [
                {
                        "C": "CN",
                        "ST": "BeiJing",
                        "L": "BeiJing",
                        "O": "k8s",
                        "OU": "System"
                }
        ]
}

3.6.1生成kube-proxy证书和密钥

[root@localhost ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
2019/04/12 20:36:40 [INFO] generate received request
2019/04/12 20:36:40 [INFO] received CSR
2019/04/12 20:36:40 [INFO] generating key: rsa-2048
2019/04/12 20:36:40 [INFO] encoded CSR
2019/04/12 20:36:40 [INFO] signed certificate with serial number 322993585988121148066141818038794839711351512333
2019/04/12 20:36:40 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

3.7使用openssl验证证书

[root@localhost ssl]# openssl x509 -noout -text -in kubernetes.pem
验证方式如下：
Issuer 与ca-csr.json一致
Subject 与kubernetes-csr.json一致
X509v3	Subject	Alternative	Name与kubernetes-csr.json一致
X509v3	Key	Usage/Extended	Key	Usage与 ca-config.jso中的kubernetes profile一致
使用cfssl-certinfo命令查看证书
[root@localhost ssl]# cfssl-certinfo -cert kubernetes.pem

3.8分发证书

[root@localhost ssl]# mkdir -p /etc/kubernetes/ssl
[root@localhost ssl]# cp *.pem /etc/kubernetes/ssl/

4 创建kubeconfig文件(密钥分发至每台)

4.1创建TLS Bootstrapping Token

添加环境变量
export BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ' ')
[root@localhost ssl]# vi /etc/profile
[root@localhost ssl]# touch  token.csv
[root@localhost ssl]# vi token.csv 
添加
{BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap"
分发至所有节点
[root@localhost ssl]# cp token.csv /etc/kubernetes/

4.2创建kubeconfig文件

4.2.1生成bootstrap.kubeconfig

设置集群参数
[root@localhost kubernetes]# kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=bootstrap.kubeconfig

设置客户端认证参数
[root@localhost kubernetes]# kubectl config set-credentials kubelet-bootstrap \
--token=${BOOTSTRAP_TOKEN} \
--kubeconfig=bootstrap.kubeconfig

设置上下文参数
[root@localhost kubernetes]# kubectl config set-context default \
--cluster=kubernetes \
--user=kubelet-bootstrap \
--kubeconfig=bootstrap.kubeconfig

设置默认上下文
[root@localhost kubernetes]# kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

4.2.2生成kube-proxy.kubeconfig

–embed-certs为true表示将certificate-authority证书写到bootstrap.kubeconfig中
设置客户端认证参数是没有指定密钥和证书，后续有 kube-apiserver自动生成

 [root@localhost kubernetes]#export KUBE_APISERVER="https://169.254.6.20:6443"
设置集群参数
[root@localhost kubernetes]#kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=kube-proxy.kubeconfig
设置客户端认证参数
[root@localhost kubernetes]#kubectl config set-credentials kube-proxy \
--client-certificate=/etc/kubernetes/ssl/kube-proxy.pem \
--client-key=/etc/kubernetes/ssl/kube-proxy-key.pem \
--embed-certs=true \
--kubeconfig=kube-proxy.kubeconfig
设置上下文参数
 [root@localhost kubernetes]#kubectl config set-context default \
--cluster=kubernetes \
--user=kube-proxy \
--kubeconfig=kube-proxy.kubeconfig
	设置默认上下文
[root@localhost kubernetes]#kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

4.2.3创建 kubectl.kubeconfig文件

[root@localhost /]# chmod a+x /usr/bin/kube*
[root@localhost /]# export KUBE_APISERVER="https://169.254.6.20:6443"
设置集群参数
[root@localhost /]# kubectl config set-cluster kubernetes \
> --certificate-authority=/etc/kubernetes/ssl/ca.pem \
> --embed-certs=true \
> --server=${KUBE_APISERVER}
执行结果:Cluster "kubernetes" set.
设置客户端认证参数
[root@localhost /]# kubectl config set-credentials admin \
> --client-certificate=/etc/kubernetes/ssl/admin.pem \
> --embed-certs=true \
> --client-key=/etc/kubernetes/ssl/admin-key.pem
执行结果:User "admin" set.
设置上下文参数
[root@localhost /]# kubectl config set-context kubernetes \
> --cluster=kubernetes \
> --user=admin
执行结果:Context "kubernetes" set.
设置默认上下文
[root@localhost /]# kubectl config use-context kubernetes
执行结果:Switched to context "kubernetes".
以上配置会直接可以通过一下命令查看
[root@localhost /]# cat ~/.kube/config

5 部署etcd集群

5.1上传并解压

[root@localhost etcd]# tar -zxvf etcd-v3.1.7-linux-amd64.tar.gz 
[root@localhost etcd-v3.1.7-linux-amd64]# mv etcd* /usr/local/bin/
export ETCD_NAME=sz-pg-oam-docker-test-001.tendcloud.com
export INTERNAL_IP=169.254.6.20

5.2 编辑etct.service启动文件及conf配置文件
5.2.1编辑启动文件

[root@localhost etcd]#vi /erc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/local/bin/etcd \
--name=sz-pg-oam-docker-test-001.tendcloud.com \
--cert-file=/etc/kubernetes/ssl/kubernetes.pem  \
--key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
--peer-cert-file=/etc/kubernetes/ssl/kubernetes.pem \
--peer-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
--trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
--peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
--initial-advertise-peer-urls   https://169.254.6.20:2380 \
--listen-peer-urls      https://169.254.6.20:2380 \
--listen-client-urls    https://169.254.6.20:2379,https://127.0.0.1:2739 \
--advertise-client-urls https://169.254.6.20:2379 \
--initial-cluster-token etcd-cluster-0 \
--initial-cluster       sz-pg-oam-docker-test-001.tendcloud.com=https://169.254.6.20:2380 \
--initial-cluster-state new \
--data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target

5.2.2编辑配置文件

[root@localhost etcd]#mkdir -p /etc/etcd/
[root@localhost etcd]#vi /etd/etcd/etcd.conf
[member]
ETCD_NAME="sz-pg-oam-docker-test-001.tendcloud.com"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://0.0.0.0:2380"
ETCD_LISTEN_CLIENT_URLS="https://0.0.0.0:2379"

[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://169.254.6.20:2380"
ETCD_INITIAL_CLUSTER="sz-pg-oam-docker-test-001.tendcloud.com=https://169.254.6.20:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-0"
ETCD_ADVERTISE_CLIENT_URLS="https://169.254.6.20:2379"

5.3启动并添加开机启动

systemctl daemon-reload
systemctl enable etcd
systemctl start etcd
systemctl stop etcd

5.4查看集群配置

etcdctl --ca-file=/etc/kubernetes/ssl/ca.pem --cert-file=/etc/kubernetes/ssl/kubernetes.pem --key-file=/etc/kubernetes/ssl/kubernetes-key.pem cluster-health

etcdctl --ca-file=/etc/kubernetes/ssl/ca.pem --cert-file=/etc/kubernetes/ssl/kubernetes.pem --key-file=/etc/kubernetes/ssl/kubernetes-key.pem  member list

6Harbor安装(master节点)

上传harbor-v1.5.2.tgz和docker-compose-Linux-x86_64.bin

6.1安装docker-compose

[root@localhost harbor]# cp docker-compose-Linux-x86_64 /usr/local/kubernetes/bin/docker-compose
[root@localhost harbor]# chmod +x /usr/local/kubernetes/bin/docker-compose

6.2安装配置harbor

[root@localhost harbor]#tar xf harbor-offline-installer-v1.5.0.tgz

6.2.1配置harbor.cfg

修改此处为本地IP
hostname = 10.10.10.1
修改登陆密码
harbor_admin_password = Harbor12345

6.2.2安装

###一定要进入harbor目录，日志放在/var/log/harbor/
[root@localhost harbor]# cd /usr/local/kubernetes/harbor/         
[root@localhost harbor]# ./prepare 
[root@localhost harbor]# ./install.sh
查看生成的镜像是否启动
[root@localhost harbor]# docker ps        ###状态为up

6.2.3加入认证

在docker启动文件中添加 --insecure-registry=IP /
在daemon.json中添加 {"insecure-registries": ["IP"]}

6.2.4测试
6.2.4.1docker命令登陆
[root@localhost harbor]# docker login 10.10.10.1 ###账号：admin 密码：Harbor12345
Username: admin
Password:
Login Succeeded

7 部署kubernetes master 节点

节点包括
kube-apiserver
kube-scheduler
kube-controller-manager
上传并解压 kubernetes-server-linux-amd64.tar.gz

[root@localhost /]#tar -zxvf  kubernetes-server-linux-amd64.tar.gz
复制二进制文件至/usr/local/bin/中
[root@localhost bin]# cp -r {kube-apiserver,kube-controller-manager,kube-scheduler,kubectl,kube-proxy,kubelet} /usr/local/bin/

7.1配置启动kube-apiserver
7.1.1配置启动文件

[root@localhost /]# vi /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Service
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
After=etcd.service
[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/apiserver
ExecStart=/usr/local/bin/kube-apiserver \
				$KUBE_LOGTOSTDERR \
				$KUBE_LOG_LEVEL \
				$KUBE_ETCD_SERVERS \
				$KUBE_API_ADDRESS \
				$KUBE_API_PORT \
				$KUBELET_PORT \
				$KUBE_ALLOW_PRIV \
				$KUBE_SERVICE_ADDRESSES \
				$KUBE_ADMISSION_CONTROL \
				$KUBE_API_ARGS
Restart=on-failure
Type=notify
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target

7.1.2配置config

该配置文件被kube-apiserver/kube-controller-manager/kube-scheduler/kubelet/kubeproxy使用
[root@localhost /]# vi /etc/kubernetes/config
KUBE_LOGTOSTDERR="--logtostderr=true"
KUBE_LOG_LEVEL="--v=0"
KUBE_ALLOW_PRIV="--allow-privileged=true"
KUBE_MASTER="--master=http://169.254.6.20:8080"

7.1.3配置 apiserver

[root@localhost /]# vi /etc/kubernetes/apiserver
KUBE_API_ADDRESS="--advertise-address=169.254.6.20 --bind-address=0.0.0.0 --insecure-bind-address=0.0.0.0"
KUBE_ETCD_SERVERS="--etcd-servers=http://169.254.6.20:2379"
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"
KUBE_ADMISSION_CONTROL="--admission-control=ServiceAccount,NamespaceLifecycle,NamespaceExists,LimitRanger,ResourceQuota"
KUBE_API_ARGS="--authorization-mode=RBAC --runtime-config=rbac.authorization.k8s.io/v1beta1 --kubelet-https=true --experimental-bootstrap-token-auth --token-auth-file=/etc/kubernetes/token.csv --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --client-ca-file=/etc/kubernetes/ssl/ca.pem --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem --etcd-cafile=/etc/kubernetes/ssl/ca.pem --etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem --etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem --enable-swagger-ui=true --apiserver-count=3 --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/var/lib/audit.log --event-ttl=1h"

7.1.4启动并设置开机启动

systemctl daemon-reload
systemctl enable kube-apiserver
systemctl start kube-apiserver
systemctl status kube-apiserver

7.2 配置启动kube-controller-manager
7.2.1配置启动文件

[root@localhost /]# vi /usr/lib/systemd/system/kube-controller-manager.service
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/controller-manager
ExecStart=/usr/bin/local/kube-controller-manager \
    $KUBE_LOGTOSTDERR \
    $KUBE_LOG_LEVEL \
    $KUBE_MASTER \
    $KUBE_CONTROLLER_MANAGER_ARGS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target

7.2.2编辑配置文件controller-manager

[root@localhost /]# vi /etc/kubernetes/controller-manager
KUBE_CONTROLLER_MANAGER_ARGS="--address=127.0.0.1 --service-cluster-ip-range=10.254.0.0/16 --cluster-name=kubernetes --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem  --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem --root-ca-file=/etc/kubernetes/ssl/ca.pem --leader-elect=true"

7.2.3启动并设置开机启动

systemctl daemon-reload
systemctl enable kube-controller-manager
systemctl start kube-controller-manager
systemctl status kube-controller-manager

7.3配置启动kube-scheduler
7.3.1配置启动文件kube-scheduler

[root@localhost /]# vi /usr/lib/systemd/system/kube-kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler Plugin
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/scheduler
ExecStart=/usr/local/bin/kube-scheduler \
            $KUBE_LOGTOSTDERR \
            $KUBE_LOG_LEVEL \
            $KUBE_MASTER \
            $KUBE_SCHEDULER_ARGS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target

7.3.2编辑配置文件scheduler

[root@localhost /]# vi /etc/kubernetes/scheduler
KUBE_SCHEDULER_ARGS="--leader-elect=true --address=127.0.0.1"

7.3.3启动并设置开机启动

systemctl daemon-reload
systemctl enable kube-kube-scheduler
systemctl start kube-kube-scheduler
systemctl status kube-kube-scheduler

7.4查看master节点功能

[root@localhost kubernetes]# kubectl get componentstatuses
NAME                 STATUS    MESSAGE              ERROR
etcd-0               Healthy   {"health": "true"}   
scheduler            Healthy   ok                   
controller-manager   Healthy   ok

8部署kubernetes node节点

8.1配置启动Flanneld
8.1.1配置前操作

[root@localhost flannel]# ls
flannel-v0.7.1-linux-amd64.tar.gz
[root@localhost flannel]# tar -zxvf flannel-v0.7.1-linux-amd64.tar.gz 
[root@localhost k8s]# mv flannel-v0.7.1-linux-amd64.tar.gz flannel/
[root@localhost k8s]# cd flannel/
移动文件至
[root@localhost bin]# cp -r flanneld /usr/bin/
[root@localhost bin]# cp -r mk-docker-opts.sh /usr/libexec/flanneld/

8.1.2配置启动文件

[root@localhost bin]# vi /usr/lib/systemd/system/flanneld.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service

[Service]
Type=notify
EnvironmentFile=/etc/sysconfig/flanneld
EnvironmentFile=-/etc/sysconfig/docker-network
ExecStart=/usr/local/bin/flanneld \
  -etcd-endpoints=${FLANNEL_ETCD_ENDPOINTS} \
  -etcd-prefix=${FLANNEL_ETCD_PREFIX} \
  $FLANNEL_OPTIONS
ExecStartPost=/usr/libexec/flannel/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=on-failure

[Install]
WantedBy=multi-user.target
RequiredBy=docker.service

8.1.3编辑配置文件flanneld

[root@localhost sysconfig]# vi /etc/sysconfig/flanneld
FLANNEL_ETCD_ENDPOINTS="http://169.254.6.20:2379"
FLANNEL_ETCD_PREFIX="/kube-centos/network"
FLANNEL_OPTIONS="-etcd-cafile=/etc/kubernetes/ssl/ca.pem -etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem -etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem"

8.1.4设置etcd-key

[root@localhost sysconfig]# etcdctl mkdir /atomic.io/network
[root@localhost sysconfig]# etcdctl mk /atomic.io/network/config  "{ \"Network\": \"172.17.0.0/16\", \"SubnetLen\": 24, \"Backend\": { \"Type\": \"vxlan\" } }"
上面IP与docker本身的IP地址一个网段

8.1.5添加docker配置

因为docker要使用flanneld，在docker启动文件中添加中加入
EnvironmentFile=-/etc/sysconfig/flanneld
EnvironmentFile=-/run/flannel/subnet.env
--bip=${FLANNEL_SUBNET}

8.1.6启动flanneld并重启docker

systemctl daemon-reload
systemctl enable flanneld
systemctl start flanneld
systemctl status flanneld
systemctl daemon-reload 
systemctl restart docker.service

8.1.7验证

使用ifconfig查看网路配置
flannel地址与docker在同一个网段即可

8.2安装配置kubelet
8.2.1安装配置前操作

kubelet启动需要向kube-apiserver发送TLS bootstrapping请求，需要将bootstrap token文件中的kubelet-bootstrap用户授予system:node-bootstrapper权限之后kubelet才有创建认证的请求的权限(certificate signing requests)

[root@localhost sysconfig]# kubectl create clusterrolebinding kubelet-bootstrap \
###赋予system:node-bootstrapper角色
[root@localhost sysconfig]# --clusterrole=system:node-bootstrapper \				
###指定/etc/kubernetes/token.csv中的用户名并写入etc/kubernetes/bootstrap.kubeconfig
[root@localhost sysconfig]# --user=kubelet-bootstrap	

[root@localhost sysconfig]# kubectl create clusterrolebinding kubelet-bootstrap \
[root@localhost sysconfig]#> --clusterrole=system:node-bootstrapper \
[root@localhost sysconfig]#> --user=kubelet-bootstrap
clusterrolebinding "kubelet-bootstrap" created

移动文件至/usr/local/bin/
[root@localhost bin]# cp -r {kube-proxy,kubelet} /usr/local/bin/
创建kubelet文件
mkdir -p /var/lib/kubelet
8.2.2创建kubelet.service启动文件
[root@localhost sysconfig]# vi /usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service
[Service]
WorkingDirectory=/var/lib/kubelet
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/kubelet
ExecStart=/usr/local/bin/kubelet \
            $KUBE_LOGTOSTDERR \
            $KUBE_LOG_LEVEL \
            $KUBELET_API_SERVER \
            $KUBELET_ADDRESS \
            $KUBELET_PORT \
            $KUBELET_HOSTNAME \
            $KUBE_ALLOW_PRIV \
            $KUBELET_POD_INFRA_CONTAINER \
            $KUBELET_ARGS
Restart=on-failure
[Install]
WantedBy=multi-user.target

8.2.3编辑配置文件kubectl

[root@localhost sysconfig]# vi /etc/sysconfig/kubelet
KUBELET_ADDRESS="--address=169.254.6.20"
KUBELET_HOSTNAME="--hostname-override=169.254.6.20"
KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=k8s.gcr.io/pause-amd64:3.1
KUBELET_ARGS="--cgroup-driver=systemd --cluster-dns=10.254.0.2 --experimental-bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --require-kubeconfig --cert-dir=/etc/kubernetes/ssl --cluster-domain=cluster.local --hairpin-mode promiscuous-bridge --serialize-image-pulls=false --allow-privileged=true --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice"

8.2.4启动并设置开机启动

systemctl daemon-reload
systemctl enable kubelet
systemctl start kubelet
systemctl status kubelet

8.3安装配置kube-proxy
安装conntrack-tools.x86_64

[root@localhost cfssl]# yum install conntrack-tools.x86_64

8.3.1创建kube-proxy.service启动文件

[root@localhost sysconfig]# vi /usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/proxy
ExecStart=/usr/local/bin/kube-proxy \
        $KUBE_LOGTOSTDERR \
        $KUBE_LOG_LEVEL \
        $KUBE_MASTER \
        $KUBE_PROXY_ARGS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

8.3.2创建proxy配置文件

[root@localhost sysconfig]# vi /etc/sysconfig/kube-proxy
KUBE_PROXY_ARGS="--bind-address=169.254.6.20 --hostname-override=169.254.6.20 --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig --cluster-cidr=10.254.0.0/16"

8.3.3启动并设置开机启动

systemctl daemon-reload
systemctl enable kube-proxy
systemctl start kube-proxy
systemctl status kube-proxy

8.4创建nginx服务进行测试
8.4.1推送docker镜像之harbor中
标记镜像
docker tag <169.254.6.20/library/images>
推送镜像
dockers push 169.254.6.20/library/images
8.4.2创建nginx

执行下列命令创建并运行
[root@localhost cfssl]# kubectl run nginx --replicas=2 --labels="run=load-balancer-example" --image=169.254.6.20/library/vmware/nginx-photon  --port=80
[root@localhost cfssl]# kubectl expose deployment nginx --type=NodePort --name=example-service
使用命令查看服务信息
[root@localhost cfssl]#  kubectl describe svc example-service
Name:                     example-service
Namespace:                default
Labels:                   run=load-balancer-example
Annotations:              <none>
Selector:                 run=load-balancer-example
Type:                     NodePort
IP:                       10.254.21.167
Port:                     <unset>  80/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  32262/TCP
Endpoints:                172.17.86.2:80,172.17.86.3:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>
使用kubectl命令查看该服务的pod、service、deployment为running

8.4.3登陆验证
curl 172.17.86.2:80或者通过浏览器登陆本机ip: 32262验证
8.4.4kubectl常用命令

产看服务
kubectl get svc
查看部署的容器
kubectl get deployment 
删除
kubectl delete 类型  名称
产看日志
kubectl describe 类型 名称
修改参数
kubectl edit 类型/名称 -n default
类型为(pod svc ec deployment)

9安装配置kubedns(master节点)

9.1推送镜像至harbor

标记镜像
docker tag k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.14.8  169.254.6.20/library/k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.14.8
docker tag k8s.gcr.io/k8s-dns-sidecar-amd64:1.14.8  169.254.6.20/library/k8s.gcr.io/k8s-dns-sidecar-amd64:1.14.8
docker tag k8s.gcr.io/k8s-dns-kube-dns-amd64:1.14.8  169.254.6.20/library/k8s.gcr.io/k8s-dns-kube-dns-amd64:1.14.8
推送至harbor
docker push 169.254.6.20/library/k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.14.8
docker push 169.254.6.20/library/k8s.gcr.io/k8s-dns-sidecar-amd64:1.14.8
docker push 169.254.6.20/library/k8s.gcr.io/k8s-dns-kube-dns-amd64:1.14.8

9.2配置yaml文件

配置KUBEDNS需要kubedns-cm.yaml  kubedns-sa.yaml  kubedns-controller.yaml  kubedns-svc.yaml文件，修改yaml配置文件中使用的镜像为本地镜像
vi kubedns-cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
vi kubedns-sa.yaml 
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile

vi kubedns-controller.yaml 
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  strategy:
    rollingUpdate:
      maxSurge: 10%
      maxUnavailable: 0
  selector:
    matchLabels:
      k8s-app: kube-dns
  template:
    metadata:
      labels:
        k8s-app: kube-dns
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      tolerations:
      - key: "CriticalAddonsOnly"
        operator: "Exists"
      volumes:
      - name: kube-dns-config
        configMap:
          name: kube-dns
          optional: true
      containers:
      - name: kubedns
        image: 169.254.6.20/library/k8s.gcr.io/k8s-dns-kube-dns-amd64:1.14.8
        resources:
          limits:
            memory: 170Mi
          requests:
            cpu: 100m
            memory: 70Mi
        livenessProbe:
          httpGet:
            path: /healthcheck/kubedns
            port: 10054
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        readinessProbe:
          httpGet:
            path: /readiness
            port: 8081
            scheme: HTTP
          # we poll on pod startup for the Kubernetes master service and
          # only setup the /readiness HTTP server once that's available.
          initialDelaySeconds: 3
          timeoutSeconds: 5
        args:
        - --domain=cluster.local.
        - --dns-port=10053
        - --config-dir=/kube-dns-config
        - --v=2
        - --kube-master-url=http://169.254.6.20:8080  ###新增配置不添加会导致网络拒绝链接
        #__PILLAR__FEDERATIONS__DOMAIN__MAP__
        env:
        - name: PROMETHEUS_PORT
          value: "10055"
        ports:
        - containerPort: 10053
          name: dns-local
          protocol: UDP
        - containerPort: 10053
          name: dns-tcp-local
          protocol: TCP
        - containerPort: 10055
          name: metrics
          protocol: TCP
        volumeMounts:
        - name: kube-dns-config
          mountPath: /kube-dns-config
      - name: dnsmasq
        image: 169.254.6.20/library/k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.14.8
        livenessProbe:
          httpGet:
            path: /healthcheck/dnsmasq
            port: 10054
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        args:
        - -v=2
        - -logtostderr
        - -configDir=/etc/k8s/dns/dnsmasq-nanny
        - -restartDnsmasq=true
        - --
        - -k
        - --cache-size=1000
        - --log-facility=-
        - --server=/cluster.local./127.0.0.1#10053
        - --server=/in-addr.arpa/127.0.0.1#10053
        - --server=/ip6.arpa/127.0.0.1#10053
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        # see: https://github.com/kubernetes/kubernetes/issues/29055 for details
        resources:
          requests:
            cpu: 150m
            memory: 20Mi
        volumeMounts:
        - name: kube-dns-config
          mountPath: /etc/k8s/dns/dnsmasq-nanny
      - name: sidecar
        image: 169.254.6.20/library/k8s.gcr.io/k8s-dns-sidecar-amd64:1.14.8
        livenessProbe:
          httpGet:
            path: /metrics
            port: 10054
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        args:
        - --v=2
        - --logtostderr
        - --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local.,5,A
        - --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local.,5,A
        ports:
        - containerPort: 10054
          name: metrics
          protocol: TCP
        resources:
          requests:
            memory: 20Mi
            cpu: 10m
      dnsPolicy: Default  # Don't use cluster DNS.
      serviceAccountName: kube-dns

vi kubedns-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/name: "KubeDNS"
spec:
  selector:
    k8s-app: kube-dns
  clusterIP: 10.254.0.2
  ports:
  - name: dns
    port: 53
    protocol: UDP
  - name: dns-tcp
    port: 53
    protocol: TCP

9.3执行所有定义文件

命令及执行结果
[root@localhost kubedns]# kubectl create -f .
configmap "kube-dns" created
deployment "kube-dns" created
serviceaccount "kube-dns" created
service "kube-dns" created
删除
kubectl delete -f .

9.4使用kubectl产看kubedns状态
kubectl get svc -n kube-system
kubectl get deployment -n kube-system
kubectl get pods -n kube-system
以上命令执行结果状态均为running则表示正常
查看日志是否有报错
kubectl describe pod -n kube-system pod名称

10 安装dashboard插件(master节点)

10.1推送镜像至harbor

将镜像上传至docker
docker load -i kubernetes-dashboard-amd64-v1.8.3.tar.gz
标记并上传至Harbor中
docker tag k8s.gcr.io/kubernetes-dashboard-amd64:v1.8.3 169.254.6.20/library/k8s.gcr.io/kubernetes-dashboard-amd64:v1.8.3
docker push 169.254.6.20/library/k8s.gcr.io/kubernetes-dashboard-amd64:v1.8.3

10.2编辑yaml文件

文件中的dashboard镜像为本地镜像地址需要修改为对应的镜像地址和版本：

vi dashboard-controller.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: kubernetes-dashboard
  namespace: kube-system
  labels:
    k8s-app: kubernetes-dashboard
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  selector:
    matchLabels:
      k8s-app: kubernetes-dashboard
  template:
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      serviceAccountName: dashboard
      containers:
      - name: kubernetes-dashboard
        image: 169.254.6.20/library/k8s.gcr.io/kubernetes-dashboard-amd64:v1.8.3
        resources:
          limits:
            cpu: 100m
            memory: 50Mi
          requests:
            cpu: 100m
            memory: 50Mi
        ports:
        - containerPort: 9090
        args:
          - --apiserver-host=http://169.254.6.20:8080    ###新增配置不添加会导致网络拒绝链接
        livenessProbe:
          httpGet:
            path: /
            port: 9090
          initialDelaySeconds: 30
          timeoutSeconds: 30
      tolerations:
      - key: "CriticalAddonsOnly"
        operator: "Exists"
vi dashboard-service.yaml
指定端口类型为 NodePort，这样外界可以通过地址 nodeIP:nodePort 访问 dashboard
apiVersion: v1
kind: Service
metadata:
  name: kubernetes-dashboard
  namespace: kube-system
  labels:
    k8s-app: kubernetes-dashboard
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  type: NodePort 
  selector:
    k8s-app: kubernetes-dashboard
  ports:
  - port: 80
    targetPort: 9090

vidashboard-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: dashboard
  namespace: kube-system

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: dashboard
subjects:
  - kind: ServiceAccount
    name: dashboard
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

10.3执行所有定义文件

命令及执行结果
[root@localhost kubedns]# kubectl create -f .

10.4使用kubectl查看dashboard状态

kubectl get svc -n kube-system
kubectl get deployment -n kube-system
kubectl get pods -n kube-system
以上命令执行结果状态均为running则表示正常
查看日志是否有报错
kubectl describe pod -n kube-system pod名称

10.5验证
http://169.254.6.20:端口 登陆成功则证明部署成功

11 全部服务及配置文件

11.1服务

systemctl start docker
systemctl start etcd
systemctl start kube-apiserver
systemctl start kube-scheduler
systemctl start kube-controller-manager
systemctl start flanneld
systemctl start docker
systemctl start kubelet
systemctl start kube-proxy

12 k8S 常用命令：

查看指定pod 的输出
kubectl describe  pod dlsupd-0 

查看所有pod 的信息
kubectl get pod -o wide  

查看指定pod的日志
kubectl logs kernel-0 kernel 

cd 到pod项目目录
删除pod
kubectl delete -f .
重新加载pod
kubectl apply -f .

查看所有pod命名空间
kubectl get po --all-namespaces 
查看指定pod的sts
kubectl get sts front-reverse 
查看指定pod的输出
kubectl describe  sts front-reverse 

登陆进入k8s 进入 pod
kubectl exec -it paas-oss-0 -c paas-oss sh
查看生成的配置文件
kubectl edit pod web-0（pod名称） 

当没有pod 的 yaml 配置文件的时候，用这个命令重启
kubectl get pod -n apm filebeat-xpdqr -o yaml | kubectl replace --force -f -

查看当前所有node
kubectl get nodes
kubectl config use-context context-WL0005-admin
kubectl config get-clusters
kubectl config delete-cluster AC0013
kubectl config get-contexts
# linux # centos # docker # kubernetes
Logo
K8S/Kubernetes

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐

【深度】阿里巴巴万级规模 K8s 集群全局高可用体系之美

作者 |  韩堂、柘远、沉醉来源 | 阿里巴巴云原生公众号​前言台湾作家林清玄在接受记者采访的时候，如此评价自己 30 多年写作生涯：“第一个十年我才华横溢，‘贼光闪现’，令周边黯然失色；第二个十年，我终于‘宝光现形’，不再去抢风头，反而与身边的美丽相得益彰；进入第三个十年，繁华落尽见真醇，我进入了‘醇光初现’的阶段，真正体味到了境界之美”。​长夜有穷，真水无香。领略过了 K8s“身在江

avatar K8S/Kubernetes

如何基于 K8s 构建下一代 DevOps 平台？

作者 | 孙健波（天元）导读：当前云原生 DevOps 体系现状如何？面临哪些挑战？如何通过 OAM 解决云原生 DevOps 场景下的诸多问题？云原生开发应用模型 OAM(Open Application Model) 社区核心成员孙健波将为大家一一解答，并分享如何基于 OAM 和 Kubernetes 打造无限能力的下一代 DevOps 平台。什么是 DevOps？为什么基于 Kub

avatar K8S/Kubernetes

k8s 火了！

2020，上云之年，产品云端化成为一种趋势。在一线城市，很多公司都已经构建了自己的私有云环境，比如阿里云、网易云、华为云等。而Kubernetes 作为基于容器编排领域的王者，具备扩展...

avatar K8S/Kubernetes