kubernetes之二进制搭建k8s(1.18.4版本)
yum update && wget -O /etc/yum.repos.d/epel.repo http://mirrors.cloud.tencent.com/repo/epel-7.repo && yum clean all&& yum makecache# 关闭selinux,firewalld,swapsed -i 's/enforcing
部署规划
主机 | ip | 作用 | 部署的软件 |
---|---|---|---|
centos-7 | 172.31.15.17 | master | kube-apiserver,kube-controller-manager,kube-scheduler,docker,etcd |
centos-7 | 172.31.15.2 | node | kubelet,kube-proxy,docker,etcd |
centos-7 | 172.31.15.11 | node | kubelet,kube-proxy,docker,etcd |
系统初始化(所有节点执行)
软件安装路径默认路径为/data
yum update && wget -O /etc/yum.repos.d/epel.repo http://mirrors.cloud.tencent.com/repo/epel-7.repo && yum clean all
&& yum makecache
# 关闭selinux,firewalld,swap
sed -i 's/enforcing/disabled/' /etc/selinux/config
systemctl disable firewalld && systemctl stop firewalld
sed -ri 's/.*swap.*/#&/' /etc/fstab && swapoff -a
# 设置好主机名与主机名解析
cat >> /etc/hosts << EOF
172.31.15.17 centos7-node4
172.31.15.2 centos7-node5
172.31.15.11 centos7-node6
172.31.15.17 k8s-master
172.31.15.2 k8s-node1
172.31.15.11 k8s-node2
EOF
# 将桥接的IPv4流量传递到iptables的链
modprobe br_netfilter
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sysctl -p /etc/sysctl.d/k8s.conf #配置生效
#时间同步
yum install chrony -y && systemctl enable chronyd && systemctl start chronyd
timedatectl set-timezone Asia/Shanghai && timedatectl set-ntp yes
部署ETCD集群
Etcd 是一个分布式键值存储系统,Kubernetes使用Etcd进行数据存储,所以先准备一个Etcd数据库, 为解决Etcd单点故障,应采用集群方式部署,这里使用3台组建集群,可容忍1台机器故障,当然,你也 可以使用5台组建集群,可容忍2台机器故障。
节点hostname 节点名称 ip
centos7-node4 etcd-1 172.31.15.17
centos7-node5 etcd-2 172.31.15.2
centos7-node6 etcd-3 172.31.15.11
注:为了节省机器,这里与K8s节点机器复用。也可以独立于k8s集群之外部署,只要apiserver能 连接到就行。
生成etcd证书配置
- 准备cfssl证书管理工具,使用json文件生成证书,相比openssl更方便使用
软件安装
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -O /usr/local/bin/cfssl
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -O /usr/local/bin/cfssljson
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -O /usr/local/bin/cfssl-certinfo
chmod +x /usr/local/bin/cfssl*
- 准备ca与证书配置
mkdir -p ~/TLS/{etcd,k8s} && cd ~/TLS/etcd
# 自谦CA配置文件
cat > ca-config.json << EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"www": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
EOF
# 自签csr配置文件
cat > ca-csr.json << EOF
{
"CN": "etcd CA",
"key": {
"algo": "rsa",
"size": 2048
},
"ca": {
"expiry": "87600h"
},
"names": [
{
"C": "CN",
"L": "BJ",
"ST": "BeiJing"
}
]
}
EOF
# 生成CA证书
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
ls *pem
# 签发ETCD https证书
cat > server.json << EOF
{
"CN": "etcd",
"hosts": [
"172.31.15.17",
"172.31.15.2",
"172.31.15.11",
"172.31.15.15"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing"
}
]
}
EOF
- 注:上述文件hosts字段中IP为所有etcd节点的集群内部通信IP,一个都不能少!为了方便后期扩 容可以多写几个预留的IP。
签发证书
gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server.json | cfssljson -bare server
ls server*pem #签发证书文件
部署ETCD集群
单个节点配置
安装路径准备
mkdir /data/etcd/{bin,cfg,ssl,data} -p
# 二进制文件准备
wget https://github.wuyanzheshui.workers.dev/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz && tar xf etcd-v3.4.9-linux-amd64.tar.gz
mv etcd-v3.4.9-linux-amd64/etcd* /data/etcd/bin/
# 当前节点172.31.15.17配置文件
cat > /data/etcd/cfg/etcd.conf <<EOF
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/data/etcd/data/default.etcd"
ETCD_LISTEN_PEER_URLS="https://172.31.15.17:2380"
ETCD_LISTEN_CLIENT_URLS="https://172.31.15.17:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.31.15.17:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://172.31.15.17:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://172.31.15.17:2380,etcd-2=https://172.31.15.2:2380,etcd-3=https://172.31.15.11:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF
# systemctl 启动管理文件配置(所有节点配置是一致的)
cat /usr/lib/systemd/system/etcd.service <<EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/data/etcd/cfg/etcd.conf
ExecStart=/data/etcd/bin/etcd \
--cert-file=/data/etcd/ssl/server.pem \
--key-file=/data/etcd/ssl/server-key.pem \
--peer-cert-file=/data/etcd/ssl/server.pem \
--peer-key-file=/data/etcd/ssl/server-key.pem \
--trusted-ca-file=/data/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/data/etcd/ssl/ca.pem \
--logger=zap
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
# 拷贝证书文件
mv ~/TLS/etcd/*pem /data/etcd/ssl
# 启动当前节点
systemctl daemon-reload && systemctl enable etcd && systemctl start etcd
配置文件字段介绍
ETCD_NAME:节点名称,集群中唯一
ETCD_DATA_DIR:数据目录
ETCD_LISTEN_PEER_URLS:集群通信监听地址 ETCD_LISTEN_CLIENT_URLS:客户端访问监听地址 ETCD_INITIAL_ADVERTISE_PEER_URLS:集群通告地址 ETCD_ADVERTISE_CLIENT_URLS:客户端通告地址
ETCD_INITIAL_CLUSTER:集群节点地址
ETCD_INITIAL_CLUSTER_TOKEN:集群Token ETCD_INITIAL_CLUSTER_STATE:加入集群的当前状态,new是新集群,existing表示加入已有集 群
其余两个节点配置
vi /data/etcd/cfg/etcd.conf
#[Member]
ETCD_NAME="etcd-1" # 修改此处,节点2改为etcd-2,节点3改为etcd-3 ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://172.31.15.17:2380" # 修改此处为当前服务器IP ETCD_LISTEN_CLIENT_URLS="https://172.31.15.17:2379" # 修改此处为当前服务器IP
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.31.15.17:2380" # 修改此处为当前服务 器IP
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.56.14:2379" # 修改此处为当前服务器IP ETCD_INITIAL_CLUSTER="etcd-1=https://172.31.15.17:2380,etcd-2=https://172.31.15.2:2380,etcd-3=https://172.31.15.11:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
最后启动etcd并设置开机启动。
systemctl daemon-reload && systemctl enable etcd && systemctl start etcd
验证Etcd部署状态是否成功
任意节点执行如下命令
ETCDCTL_API=3 /data/etcd/bin/etcdctl --cacert=/data/etcd/ssl/ca.pem --cert=/data/etcd/ssl/server.pem --key=/data/etcd/ssl/server-key.pem --endpoints="https://172.31.15.17:2379,https://172.31.15.2:2379,https://172.31.15.11:2379" endpoint health
成功输出
https://172.31.15.11:2379 is healthy: successfully committed proposal: took = 9.777432ms
https://172.31.15.17:2379 is healthy: successfully committed proposal: took = 10.150672ms
https://172.31.15.2:2379 is healthy: successfully committed proposal: took = 11.418273ms
出现问题排查手段
- 查看/var/log/message日志或者journalctl -xe -f -uetcd
- 一般配置文件没问题的话就ok,最大的问题还有一点就是网络通信和防火墙,注意响应的策略放开即可
实际出现的问题
master-16 etcd[25461]: request cluster ID mismatch (got bdd7c7c32bde1208 want 698851d67fea3c17)
master-16 etcd[25461]: request cluster ID mismatch (got bdd7c7c32bde1208 want 698851d67fea3c17)
master-16 etcd[25461]: request cluster ID mismatch (got bdd7c7c32bde1208 want 698851d67fea3c17)
master-16 etcd[25461]: request cluster ID mismatch (got bdd7c7c32bde1208 want 698851d67fea3c17)
master-16 etcd[25461]: request cluster ID mismatch (got bdd7c7c32bde1208 want 698851d67fea3c17)
master-16 etcd[25461]: request cluster ID mismatch (got bdd7c7c32bde1208 want 698851d67fea3c17)
master-16 etcd[25461]: request cluster ID mismatch (got bdd7c7c32bde1208 want 698851d67fea3c17)
master-16 etcd[25461]: request cluster ID mismatch (got bdd7c7c32bde1208 want 698851d67fea3c17)
master-16 etcd[25461]: request cluster ID mismatch (got bdd7c7c32bde1208 want 698851d67fea3c17)
master-16 etcd[25461]: request cluster ID mismatch (got bdd7c7c32bde1208 want 698851d67fea3c17)
解决办法
删除了etcd集群所有节点中的–data_dir的内容
分析: 因为集群搭建过程,单独启动过单一etcd,做为测试验证,集群内第一次启动其他etcd服务时候,是通过发现服务引导的,所以需要删除旧的成员信息
所有节点安装docker
3台机器下载
# 下载和解压docker二进制文件
wget https://download.docker.com/linux/static/stable/x86_64/docker-19.03.9.tgz && tar xf docker-19.03.9.tgz
mv docker/* /usr/bin/
# 配置systemd管理docker (其余的两个节点也需要安装)
cat > /usr/lib/systemd/system/docker.service << EOF
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
EOF
docker配置与启动
# 配置docker阿里云镜像加速和存储路径(graph)
mkdir /etc/docker
cat > /etc/docker/daemon.json << EOF
{
"graph": "/data/docker",
"registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"]
}
EOF
# 服务启动
systemctl daemon-reload && systemctl restart docker && systemctl enable docker
出现的问题
容器启动报 container init caused “write /proc/self/attr/keycreate: permission denied”": unknown
容器启动时报container init caused “write /proc/self/attr/keycreate: permission denied”": unknown, 从错误信息看是权限问题,
从目录权限看启动用户是有权限操作。再查看服务器防火墙,发现防火墙是开启的状态,关闭防火墙。
getenforce
Enforcing
cat /etc/selinux/config
SELINUX=enforcing
#关闭selinux
setenforce 0
vim /etc/selinux/config
SELINUX=disabled
k8s master节点的安装与部署
当前部署master节点 172.31.15.17
生成k8s证书配置
cd ~/TLS/k8s
cat > ca-config.json <<EOF
{
"signing":{
"default":{
"expiry":"87600h"
},
"profiles":{
"kubernetes":{
"expiry":"87600h",
"usages":[
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
EOF
at > ca-csr.json <<EOF
{
"CN":"kubernetes",
"key":{
"algo":"rsa",
"size":2048
},
"names":[
{
"C":"CN",
"L":"Beijing",
"ST":"Beijing",
"O":"k8s",
"OU":"System"
}
]
}
EOF
# 生成CA证书
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
自签apiserver证书
cd ~/TLS/k8s
cat > server-csr.json << EOF
{
"CN":"kubernetes",
"hosts":[
"10.0.0.1",
"172.0.0.1",
"127.0.0.1",
"172.31.15.17",
"172.31.15.2",
"172.31.15.11",
"172.31.15.15",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key":{
"algo":"rsa",
"size":2048
},
"names":[
{
"C":"CN",
"L":"BeiJing",
"ST":"BeiJing",
"O":"k8s",
"OU":"System"
}
]
}
EOF
注:为了方便后期扩容可以多 写几个预留的IP。
生成apiserver证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server
ls *pem
安装ApiServer
# 创建软件目录
mkdir -p /data/kubernetes/{cfg,bin,ssl,logs}
# 文件下载与拷贝
wget https://dl.k8s.io/v1.18.4/kubernetes-server-linux-amd64.tar.gz && tar xf kubernetes-server-linux-amd64.tar.gz
cp kubernetes/server/bin/kube-apiserver /data/kubernetes/bin/
cp kubernetes/server/bin/kube-controller-manager /data/kubernetes/bin/
cp kubernetes/server/bin/kube-scheduler /data/kubernetes/bin/
cp kubernetes/server/bin/kubectl /usr/bin/
创建apiserver配置文件
# 创建配置文件
cat > /data/kubernetes/cfg/kube-apiserver.conf << EOF
KUBE_APISERVER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/data/kubernetes/logs \\
--etcd-servers=https://172.31.15.17:2379,https://172.31.15.2:2379,https://172.31.15.11:2379 \\
--bind-address=172.31.15.17 \\
--secure-port=6443 \\
--advertise-address=172.31.15.17 \\
--allow-privileged=true \\
--service-cluster-ip-range=10.0.0.0/24 \\
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestricti on \\
--authorization-mode=RBAC,Node \\
--enable-bootstrap-token-auth=true \\
--token-auth-file=/data/kubernetes/cfg/token.csv \\
--service-node-port-range=30000-32767 \\
--kubelet-client-certificate=/data/kubernetes/ssl/server.pem \\
--kubelet-client-key=/data/kubernetes/ssl/server-key.pem \\
--tls-cert-file=/data/kubernetes/ssl/server.pem \\
--tls-private-key-file=/data/kubernetes/ssl/server-key.pem \\
--client-ca-file=/data/kubernetes/ssl/ca.pem \\
--service-account-key-file=/data/kubernetes/ssl/ca-key.pem \\
--etcd-cafile=/data/etcd/ssl/ca.pem \\
--etcd-certfile=/data/etcd/ssl/server.pem \\
--etcd-keyfile=/data/etcd/ssl/server-key.pem \\
--audit-log-maxage=30 \\
--audit-log-maxbackup=3 \\
--audit-log-maxsize=100 \\
--audit-log-path=/data/kubernetes/logs/k8s-audit.log"
EOF
# 拷贝证书
mv ~/TLS/k8s/*pem /data/kubernetes/ssl/
注意事项:
--logtostderr:启用日志
---v:日志等级
--log-dir:日志目录
--etcd-servers:etcd集群地址
--bind-address:监听地址
--secure-port:https安全端口
--advertise-address:集群通告地址
--allow-privileged:启用授权
--service-cluster-ip-range:Service虚拟IP地址段
--enable-admission-plugins:准入控制模块
--authorization-mode:认证授权,启用RBAC授权和节点自管理
--enable-bootstrap-token-auth:启用TLS bootstrap机制
--token-auth-file:bootstrap token文件
--service-node-port-range:Service nodeport类型默认分配端口范围
--kubelet-client-xxx:apiserver访问kubelet客户端证书
--tls-xxx-file:apiserver https证书
--etcd-xxxfile:连接Etcd集群证书 --audit-log-xxx:审计日志
启用TLS Bootstrap机制
TLS Bootstraping:Master apiserver启用TLS认证后,Node节点kubelet和kube-proxy要与kube- apiserver进行通信,必须使用CA签发的有效证书才可以,当Node节点很多时,这种客户端证书颁发需 要大量工作,同样也会增加集群扩展复杂度。为了简化流程,Kubernetes引入了TLS bootstraping机制 来自动颁发客户端证书,kubelet会以一个低权限用户自动向apiserver申请证书,kubelet的证书由 apiserver动态签署。所以强烈建议在Node上使用这种方式,目前主要用于kubelet,kube-proxy还是 由我们统一颁发一个证书。
TLS bootstraping 工作流程:
- 根据上述配置token文件
cat > /data/kubernetes/cfg/token.csv << EOF
7d41ac76c8d4768a9adcc4b26f86a193,kubelet-bootstrap,10001,"system:node-bootstrapper"
EOF
格式:token,用户名,UID,用户组 token也可自行生成替换:
head -c 16 /dev/urandom | od -An -t x | tr -d ’ ’
使用systemctl管理apiserver
- 生成配置文件
cat > /usr/lib/systemd/system/kube-apiserver.service << EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/data/kubernetes/cfg/kube-apiserver.conf
ExecStart=/data/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
- 启动并设置开机自启动
systemctl daemon-reload && systemctl start kube-apiserver && systemctl enable kube-apiserver
- 授权kubelet-bootstrap用户允许请求证书
kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
部署kube-controller-manager
创建配置文件
cat > /data/kubernetes/cfg/kube-controller-manager.conf << EOF
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/data/kubernetes/logs \\
--leader-elect=true \\
--master=127.0.0.1:8080 \\
--bind-address=127.0.0.1 \\
--allocate-node-cidrs=true \\
--cluster-cidr=10.244.0.0/16 \\
--service-cluster-ip-range=10.0.0.0/24 \\
--cluster-signing-cert-file=/data/kubernetes/ssl/ca.pem \\
--cluster-signing-key-file=/data/kubernetes/ssl/ca-key.pem \\
--root-ca-file=/data/kubernetes/ssl/ca.pem \\
--service-account-private-key-file=/data/kubernetes/ssl/ca-key.pem \\
--experimental-cluster-signing-duration=87600h0m0s"
EOF
–master:通过本地非安全本地端口8080连接apiserver。
–leader-elect:当该组件启动多个时,自动选举(HA)
–cluster-signing-cert-file/–cluster-signing-key-file:自动为kubelet颁发证书的CA,与apiserver 保持一致
systemctl管理controller-manager
cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/data/kubernetes/cfg/kube-controller-manager.conf
ExecStart=/data/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
- 启动&开机自启
systemctl daemon-reload && systemctl start kube-controller-manager && systemctl enable kube-controller-manager
部署kube-scheduler
创建配置文件
cat > /data/kubernetes/cfg/kube-scheduler.conf << EOF
KUBE_SCHEDULER_dataS="--logtostderr=false \\
--v=2 \\
--log-dir=/data/kubernetes/logs \\
--leader-elect \\
--master=127.0.0.1:8080 \\
--bind-address=127.0.0.1"
EOF
–master:通过本地非安全本地端口8080连接apiserver。
–leader-elect:当该组件启动多个时,自动选举(HA)
systemctl管理kube-scheduler
cat > /usr/lib/systemd/system/kube-scheduler.service << EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/data/kubernetes/cfg/kube-scheduler.conf
ExecStart=/data/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_dataS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
- 启动&开机自启
systemctl daemon-reload && systemctl start kube-scheduler && systemctl enable kube-scheduler
至此master部署完成,集群状态查看
kubectl get cs
返回如下结果,证明mater部署ok
scheduler Healthy ok
etcd-1 Healthy {"health":"true"}
etcd-2 Healthy {"health":"true"}
controller-manager Healthy ok
etcd-0 Healthy {"health":"true"}
部署WorkNode
当前操作节点是: 172.31.15.17(将这个master也作为node),生成环境建议分开部署
- 需要的软件:
- kubelet
- kube-proxy
基础软件包准备
# 创建软件目录
mkdir -p /data/kubernetes/{cfg,bin,ssl,logs}
# 文件下载与拷贝
wget https://dl.k8s.io/v1.18.4/kubernetes-server-linux-amd64.tar.gz && tar xf kubernetes-server-linux-amd64.tar.gz
cp kubernetes/server/bin/kube-proxy /data/kubernetes/bin/
cp kubernetes/server/bin/kubelet /data/kubernetes/bin/
部署kubelet
创建kubelet配置文件
cat > /data/kubernetes/cfg/kubelet.conf << EOF
KUBELET_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/data/kubernetes/logs \\
--hostname-override=k8s-master \\
--network-plugin=cni \\
--kubeconfig=/data/kubernetes/cfg/kubelet.kubeconfig \\
--bootstrap-kubeconfig=/data/kubernetes/cfg/bootstrap.kubeconfig \\
--config=/data/kubernetes/cfg/kubelet-config.yml \\
--cert-dir=/data/kubernetes/ssl \\
--pod-infra-container-image=lizhenliang/pause-amd64:3.0"
EOF
–hostname-override:显示名称,集群中唯一
–network-plugin:启用CNI
–kubeconfig:空路径,会自动生成,后面用于连接apiserver
–bootstrap-kubeconfig:首次启动向apiserver申请证书
–config:配置参数文件
–cert-dir:kubelet证书生成目录
–pod-infra-container-image:管理Pod网络容器的镜像
创建参数配置文件
cat > /data/kubernetes/cfg/kubelet-config.yml << EOF
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS:
- 10.0.0.2
clusterDomain: cluster.local
failSwapOn: false
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 2m0s
enabled: true
x509:
clientCAFile: /data/kubernetes/ssl/ca.pem
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 5m0s
cacheUnauthorizedTTL: 30s
evictionHard:
imagefs.available: 15%
memory.available: 100Mi
nodefs.available: 10%
nodefs.inodesFree: 5%
maxOpenFiles: 1000000
maxPods: 110
EOF
生成bootstrap.kubeconfig文件
KUBE_APISERVER="https://172.31.15.17:6443"
# 生成 kubelet bootstrap kubeconfig 配置文件
kubectl config set-cluster kubernetes --certificate-authority=/data/kubernetes/ssl/ca.pem --embed-certs=true --server=https://172.31.15.17:6443 --kubeconfig=bootstrap.kubeconfig
# 与token.csv里保持一致
kubectl config set-credentials "kubelet-bootstrap" --token=2b4b65d2e33e24dc0beafddda6dd4b23 --kubeconfig=bootstrap.kubeconfig
kubectl config set-context default --cluster=kubernetes --user="kubelet-bootstrap" --kubeconfig=bootstrap.kubeconfig
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
#拷贝生成的配置到cfg
cp bootstrap.kubeconfig /data/kubernetes/cfg
systemctl管理kubelet
- 创建启动文件
cat > /usr/lib/systemd/system/kubelet.service << EOF
[Unit]
Description=Kubernetes Kubelet
After=docker.service
[Service]
EnvironmentFile=/data/kubernetes/cfg/kubelet.conf
ExecStart=/data/kubernetes/bin/kubelet $KUBELET_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
- 启动kubelet&开机自启动
systemctl daemon-reload && systemctl start kubelet && systemctl enable kubelet
如有异常,及时查看日志解决,大多数问题会出现在/data/kubernetes/cfg/kubelet-config.yml格式上
批准kubelet证书申请并加入集群
- 查看kubelet证书请求
kubectl get csr
返回结果
NAME AGE SIGNERNAME REQUESTOR CONDITION
node-csr-zJmrG00TW4zKRNPKoNo3ag0ojgPwEM2M3ARCsvVVyiI 60s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending
- 批准kubelet证书申请,并加入集群
kubectl certificate approve node-csr-zJmrG00TW4zKRNPKoNo3ag0ojgPwEM2M3ARCsvVVyiI
- 查看节点
kubectl get node
- 返回结果
NAME STATUS ROLES AGE VERSION
k8s-master NotReady <none> 2s v1.18.4
注:由于网络插件还没有部署,节点会没有准备就绪 NotReady
部署kube-proxy
创建配置文件
cat > /data/kubernetes/cfg/kube-proxy.conf << EOF
KUBE_PROXY_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/data/kubernetes/logs \\
--config=/data/kubernetes/cfg/kube-proxy-config.yml"
EOF
配置参数文件
cat > /data/kubernetes/cfg/kube-proxy-config.yml << EOF
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
metricsBindAddress: 0.0.0.0:10249
clientConnection:
kubeconfig: /data/kubernetes/cfg/kube-proxy.kubeconfig
hostnameOverride: k8s-master
clusterCIDR: 10.0.0.0/24
EOF
生成kube-proxy.kubeconfig文件
- 证书签发
cd ~/TLS/k8s/
cat > kube-proxy-csr.json << EOF
{
"CN":"system:kube-proxy",
"hosts":[
],
"key":{
"algo":"rsa",
"size":2048
},
"names":[
{
"C":"CN",
"L":"BeiJing",
"ST":"BeiJing",
"O":"k8s",
"OU":"System"
}
]
}
EOF
# 生成CA证书
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
# 生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
生成kubeconfig配置文件
kubectl config set-cluster kubernetes --certificate-authority=/data/kubernetes/ssl/ca.pem --embed-certs=true --server=https://172.31.15.17:6443 --kubeconfig=kube-proxy.kubeconfig
kubectl config set-credentials kube-proxy --client-certificate=./kube-proxy.pem --client-key=./kube-proxy-key.pem --embed-certs=true --kubeconfig=kube-proxy.kubeconfig
kubectl config set-context default --cluster=kubernetes --user=kube-proxy --kubeconfig=kube-proxy.kubeconfig
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
cp kube-proxy.kubeconfig /data/kubernetes/cfg/
systemctl管理kube-proxy
- 创建启动文件
cat > /usr/lib/systemd/system/kube-proxy.service << EOF
[Unit]
Description=Kubernetes Proxy
After=network.target
[Service]
EnvironmentFile=/data/kubernetes/cfg/kube-proxy.conf
ExecStart=/data/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
- 设置开机自启动
systemctl daemon-reload && systemctl start kube-proxy && systemctl enable kube-proxy
部署CNI网络
wget https://github.com/containernetworking/plugins/releases/download/v0.8.6/cni-plugins-linux-amd64-v0.8.6.tgz && mkdir /opt/cni/bin -p
tar zxvf cni-plugins-linux-amd64-v0.8.6.tgz -C /opt/cni/bin/
- 部署flannel
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
sed -i -r "s#quay.io/coreos/flannel:.*-amd64#lizhenliang/flannel:v0.12.0-amd64#g" kube-flannel.yml
kubectl apply -f kube-flannel.yml
- 查看部署状态
kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master Ready <none> 79m v1.18.4
授权apiserver访问kubelet
- 创建配置
cat > apiserver-to-kubelet-rbac.yaml <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:kube-apiserver-to-kubelet
rules:
- apiGroups:
- ""
resources:
- nodes/proxy
- nodes/stats
- nodes/log
- nodes/spec
- nodes/metrics
- pods/log
verbs:
- "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: system:kube-apiserver
namespace: ""
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:kube-apiserver-to-kubelet
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: kubernetes
EOF
## 创建授权
kubectl apply -f apiserver-to-kubelet-rbac.yaml
新增workNode
同步文件不配置
将172.31.15.17 相关的配置拷贝到172.31.15.11 172.31.15.5上
# kubelet,kube-proxy拷贝
scp /data/kubernetes/bin/kubelet root@172.31.15.5:/data/kubernetes/bin/
scp /data/kubernetes/bin/kube-proxy root@172.31.15.5:/data/kubernetes/bin/
scp /data/kubernetes/bin/kubelet root@172.31.15.11:/data/kubernetes/bin/
scp /data/kubernetes/bin/kube-proxy root@172.31.15.11:/data/kubernetes/bin/
# cni插件拷贝
scp -rp /opt/cni/ root@172.31.15.5:/opt
scp -rp /opt/cni/ root@172.31.15.11:/opt
# 证书拷贝
scp /data/kubernetes/ssl/ca.pem 172.31.15.5:/data/kubernetes/ssl/
scp /data/kubernetes/ssl/ca.pem 172.31.15.11:/data/kubernetes/ssl/
# 配置文件拷贝
scp /data/kubernetes/cfg/kube-proxy* 172.31.15.5:/data/kubernetes/cfg/
scp /data/kubernetes/cfg/kube-proxy* 172.31.15.11:/data/kubernetes/cfg/
scp /data/kubernetes/cfg/kubelet* 172.31.15.5:/data/kubernetes/cfg/
scp /data/kubernetes/cfg/kubelet* 172.31.15.11:/data/kubernetes/cfg/
scp /data/kubernetes/cfg/bootstrap.kubeconfig 172.31.15.5:/data/kubernetes/cfg/
scp /data/kubernetes/cfg/bootstrap.kubeconfig 172.31.15.11:/data/kubernetes/cfg/
# 启动文件拷贝
scp /usr/lib/systemd/system/kubelet.service 172.31.15.5:/usr/lib/systemd/system/
scp /usr/lib/systemd/system/kubelet.service 172.31.15.11:/usr/lib/systemd/system/
scp /usr/lib/systemd/system/kube-proxy.service 172.31.15.5:/usr/lib/systemd/system/
scp /usr/lib/systemd/system/kube-proxy.service 172.31.15.11:/usr/lib/systemd/system/
####删除证书和配置文件
在172.31.15.17 master机器上
rm /data/kubernetes/cfg/kubelet.kubeconfig
rm -f /data/kubernetes/ssl/kubelet*
- 注:这几个文件是证书申请审批后自动生成的,每个Node不同,必须删除重新生成。
配置新的Node节点
- 修改kubelet和kube-proxy配置文件
vi /opt/kubernetes/cfg/kubelet.conf
--hostname-override=k8s-node1
vi /opt/kubernetes/cfg/kube-proxy-config.yml
hostnameOverride: k8s-node1
配置kubectl和kube-proxy开机启动
systemctl daemon-reload && systemctl start kubelet && systemctl start kube-proxy
systemctl enable kubelet && systemctl enable kube-proxy
在master节点上准许node加入
- 获取准入的node信息
kubectl get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
node-csr-63HqXs5ifBWopOS6dZAO8bRJ8PImXljxbOt-2wV5hHg 7m57s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending
node-csr-t6XNO793xatm4gCwQiYH4QDOeIY4yMx8C0SUXSNye7c 38s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending
- 准入node
kubectl certificate approve node-csr-63HqXs5ifBWopOS6dZAO8bRJ8PImXljxbOt-2wV5hHg
kubectl certificate approve node-csr-t6XNO793xatm4gCwQiYH4QDOeIY4yMx8C0SUXSNye7c
- 查看状态
kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master Ready <none> 3h32m v1.18.4
k8s-node1 Ready <none> 106s v1.18.4
k8s-node2 Ready <none> 105s v1.18.4
- 如果新加的node不是Ready,那就重新apply 一下kube-flannel.yml,或者重新启动下systemctl restart kubelet && systemctl restart kube-proxy
更多推荐
所有评论(0)