【体系】Kubernetes容器管理
k8s容器管理平台入门教程
1. 基本概念
概述与特性:k8s是谷歌在2014年开源的容器化集群管理系统,主要用于容器应用部署,具有易于应用扩展的特点,让容器化应用部署更加简洁高效。传统的应用部署方式是通过插件或脚本来安装应用。这样做的缺点是应用的运行、配 置、管理、所有生存周期将与当前操作系统绑定,这样做并不利于应用的升级更新/回滚等 操作,当然也可以通过创建虚拟机的方式来实现某些功能,但是虚拟机非常重,并不利于可移植性;新的方式是通过部署容器方式实现,每个容器之间互相隔离,每个容器有自己的文件 系统 ,容器之间进程不会相互影响,能区分计算资源。相对于虚拟机,容器能快速部署, 由于容器与底层设施、机器文件系统解耦的,所以它能在不同云、不同版本操作系统间进行迁移。
架构组件:k8s具有自动装箱、自我修复(自愈能力)、水平扩展、服务发现、滚动更新、版本回退、密钥和配置管理、存储编排和批处理等功能特点
- Master Node:k8s 集群控制节点,对集群进行调度管理,接受集群外用户去集群操作请求; Master Node 由 API Server、Scheduler、ClusterState Store(ETCD 数据库)和 Controller MangerServer 所组成
- apiserver:集群统一入口,以restfull方式,交给etcd存储
- scheduler:节点调度,选择node节点应用部署
- controller-manager:处理集群中常规后台任务,一个资源对应一个控制器
- etcd:存储系统,保存集群相关的数据
- Worker Node:集群工作节点,运行用户业务应用容器,包含 kubelet、kube proxy 和 ContainerRuntime
- kubelet:master排到node节点代表,管理本机容器
- kube-proxy:提供网络代理,负载均衡等操作
核心概念:pod、controller、service
- pod:最小部署单元;一组容器的集合;共享网络;生命周期是短暂的
- controller:确保预期的pod副本数量;无状态应用部署;有状态应用部署;确保所有的node运行同一个pod;一次性任务和定时任务
- service:定义一组pod访问规则
2. 搭建集群
k8s硬件要求:测试环境中master(2核 4G内存 20G硬盘)和node(4核 8G内存 40G硬盘);生产环境中master和node均有更高要求
搭建集群方式:kubeadm方式(Kubeadm 是一个 K8s 部署工具,提供 kubeadm init 和 kubeadm join,用于快速部 署 Kubernetes 集群。官方地址,见链接)和二进制包方式(从 github 下载发行版的二进制包,手动部署每个组件,组成 Kubernetes 集群)
k8s平台规划:环境平台规划分为单master集群和多master集群(常用,避免单master故障导致系统崩溃)
平台搭建1:单master集群的kubeadm方式安装
- 准备虚拟主机(记得做好快照,以便环境快速恢复)。网络要求:集群机器之间能够互通,且能够上外网(操作系统此例为CentOS7)
- 三个虚拟机的初始化(记得做好快照,以便环境快速恢复)
# 1.三台虚拟机均关闭防火墙: systemctl stop firewalld #临时 systemctl disable firewalld #永久 # 2.三台虚拟机均关闭selinux setenforce 0 #临时 sed -i 's/enforcing/disabled/' /etc/selinux/config #永久 # 3.三台虚拟机均关闭swap分区 swapoff -a #临时 sed -ri 's/.*swap.*/#&/' /etc/fstab #永久 # 4.三台虚拟机分别设置主机名 hostnamectl set-hostname k8s-master # 5.在master添加 hosts [root@k8s-master ~]# cat >> /etc/hosts << EOF > 172.16.90.146 k8s-master > 172.16.90.145 k8s-node1 > 172.16.90.144 k8s-node2 > EOF # 6. 将三台虚拟机桥接的IPv4流量传递到iptables的链,并使之生效 [root@k8s-master ~]# cat > /etc/sysctl.d/k8s.conf << EOF > net.bridge.bridge-nf-call-ip6tables = 1 > net.bridge.bridge-nf-call-iptables = 1 > EOF [root@k8s-master ~]# sysctl --system # 7. 三台虚拟机均设置时间同步 yum install ntpdate -y ntpdate time.windows.com
- 所有节点安装 Docker/kubeadm/kubelet
# 安装wget [root@k8s-master ~]# yum install wget # 下载docker源文件 [root@k8s-master ~]# wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo # 安装docker [root@k8s-master ~]# yum -y install docker-ce-18.06.1.ce-3.el7 # 启动docker [root@k8s-master ~]# systemctl enable docker && systemctl start docker # 检查是否安装成功 [root@k8s-master ~]# docker --version # 添加阿里云YUM软件源 # 设置仓库地址 [root@k8s-master ~]# cat > /etc/docker/daemon.json << EOF > { > "registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"] > } > EOF # 添加yum源 [root@k8s-master ~]# cat > /etc/yum.repos.d/kubernetes.repo << EOF > [kubernetes] > name=Kubernetes > baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 > enabled=1 > gpgcheck=0 > repo_gpgcheck=0 > gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg > https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg > EOF # 安装 kubeadm,kubelet 和 kubectl [root@k8s-master ~]# yum install -y kubelet kubeadm kubectl # 设置开机启动 [root@k8s-master ~]# systemctl enable kubelet
- 在master节点执行kubeadm init命令进行初始化
# 在master节点机部署Kubernetes Master,并执行一下目录 # 由于默认拉取镜像地址 k8s.gcr.io 国内无法访问,这里指定阿里云镜像仓库地址。 [root@k8s-master ~]# kubeadm init --apiserver-advertise-address=172.16.90.146 --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.21.0 --service-cidr=10.96.0.0/12 --pod-network-cidr=10.244.0.0/16 ... error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR ImagePull]: failed to pull image registry.aliyuncs.com/google_containers/coredns:v1.8.0: output: Error response from daemon: manifest for registry.aliyuncs.com/google_containers/coredns:v1.8.0 not found , error: exit status 1 ... # 检查发现缺少镜像coredns [root@k8s-master ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE registry.aliyuncs.com/google_containers/kube-apiserver v1.21.0 4d217480042e 3 months ago 126MB registry.aliyuncs.com/google_containers/kube-proxy v1.21.0 38ddd85fe90e 3 months ago 122MB registry.aliyuncs.com/google_containers/kube-controller-manager v1.21.0 09708983cc37 3 months ago 120MB registry.aliyuncs.com/google_containers/kube-scheduler v1.21.0 62ad3129eca8 3 months ago 50.6MB registry.aliyuncs.com/google_containers/pause 3.4.1 0f8457a4c2ec 6 months ago 683kB registry.aliyuncs.com/google_containers/etcd 3.4.13-0 0369cf4303ff 11 months ago 253MB # 解决该问题:Kubernetes 需要的是 registry.aliyuncs.com/google_containers/coredns:v1.8.0 这个镜像,使用 docker tag 命令重命名 # 拉取镜像 [root@k8s-master ~]# docker pull registry.aliyuncs.com/google_containers/coredns:1.8.0 # 重命名 [root@k8s-master ~]# docker tag registry.aliyuncs.com/google_containers/coredns:1.8.0 registry.aliyuncs.com/google_containers/coredns:v1.8.0 # 删除原有镜像 [root@k8s-master ~]# docker rmi registry.aliyuncs.com/google_containers/coredns:1.8.0 # 检查 [root@k8s-master ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE registry.aliyuncs.com/google_containers/kube-apiserver v1.21.0 4d217480042e 3 months ago 126MB registry.aliyuncs.com/google_containers/kube-proxy v1.21.0 38ddd85fe90e 3 months ago 122MB registry.aliyuncs.com/google_containers/kube-controller-manager v1.21.0 09708983cc37 3 months ago 120MB registry.aliyuncs.com/google_containers/kube-scheduler v1.21.0 62ad3129eca8 3 months ago 50.6MB registry.aliyuncs.com/google_containers/pause 3.4.1 0f8457a4c2ec 6 months ago 683kB registry.aliyuncs.com/google_containers/coredns v1.8.0 296a6d5035e2 9 months ago 42.5MB registry.aliyuncs.com/google_containers/etcd 3.4.13-0 0369cf4303ff 11 months ago 253MB # 再次执行kubeadm init ... Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config ... kubeadm join 172.16.90.146:6443 --token y761gh.vxrkrulwu0tt74sw \ --discovery-token-ca-cert-hash sha256:b611e2e88052ec60ac4716b0a9a48a9fa45d99a4b457563593dc29805214bbc5 # 使用 kubectl 工具: [root@k8s-master ~]# mkdir -p $HOME/.kube [root@k8s-master ~]# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config [root@k8s-master ~]# chown $(id -u):$(id -g) $HOME/.kube/config [root@k8s-master ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master NotReady control-plane,master 3m25s v1.21.3
- 在node节点上执行kubeadm join命令把node节点添加到当前集群里面
[root@k8s-node1 ~]# kubeadm join 172.16.90.146:6443 --token y761gh.vxrkrulwu0tt74sw --discovery-token-ca-cert-hash sha256:b611e2e88052ec60ac4716b0a9a48a9fa45d99a4b457563593dc29805214bbc5 [root@k8s-node2 ~]# kubeadm join 172.16.90.146:6443 --token y761gh.vxrkrulwu0tt74sw --discovery-token-ca-cert-hash sha256:b611e2e88052ec60ac4716b0a9a48a9fa45d99a4b457563593dc29805214bbc5 # 检查是否添加成功 [root@k8s-master ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master NotReady control-plane,master 11m v1.21.3 k8s-node1 NotReady <none> 2m6s v1.21.3 k8s-node2 NotReady <none> 60s v1.21.3
- 配置网络插件
[root@k8s-master ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml # 查看是否有运行 [root@k8s-master ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-59d64cd4d4-dqx8m 1/1 Running 0 31m coredns-59d64cd4d4-z8pdq 1/1 Running 0 31m etcd-k8s-master 1/1 Running 0 31m kube-apiserver-k8s-master 1/1 Running 0 31m kube-controller-manager-k8s-master 1/1 Running 0 31m kube-flannel-ds-h7v2g 1/1 Running 0 2m46s kube-flannel-ds-xmzfh 1/1 Running 0 2m46s kube-flannel-ds-z9nbj 1/1 Running 0 2m46s kube-proxy-6c9cd 1/1 Running 0 20m kube-proxy-cnvfg 1/1 Running 0 31m kube-proxy-p4nx4 1/1 Running 0 22m kube-scheduler-k8s-master 1/1 Running 0 31m # 验证是否启动 [root@k8s-master ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master Ready control-plane,master 32m v1.21.3 k8s-node1 Ready <none> 23m v1.21.3 k8s-node2 Ready <none> 22m v1.21.3
- 测试 kubernetes 集群
# 在 Kubernetes 集群中创建一个 pod,验证是否正常运行 # 联网下载nginx镜像 [root@k8s-master ~]# kubectl create deployment nginx --image=nginx deployment.apps/nginx created # 查看pod状态 [root@k8s-master ~]# kubectl get pod NAME READY STATUS RESTARTS AGE nginx-6799fc88d8-lj24f 1/1 Running 0 62s # 对外暴露80端口 [root@k8s-master ~]# kubectl expose deployment nginx --port=80 --type=NodePort service/nginx exposed # 查看对外端口 [root@k8s-master ~]# kubectl get pod,svc NAME READY STATUS RESTARTS AGE pod/nginx-6799fc88d8-lj24f 1/1 Running 0 3m4s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 39m service/nginx NodePort 10.105.227.129 <none> 80:30640/TCP 20s # 访问地址:http://NodeIP:Port(NodeIP为任意节点ip,port为暴露出来的端口号)
平台搭建2:单master集群的二进制方式安装
- 准备虚拟主机(记得做好快照,以便环境快速恢复)。网络要求:集群机器之间能够互通,且能够上外网(操作系统此例为CentOS7)
角色 IP 组件 k8s-master 172.16.90.147 kube-apiserver,kube-controller-manager,kube -scheduler,etcd k8s-node1 172.16.90.148 kubelet,kube-proxy,docker etcd - 两个虚拟机的初始化(记得做好快照,以便环境快速恢复)
# 关闭防火墙 systemctl stop firewalld systemctl disable firewalld # 关闭 selinux sed -i 's/enforcing/disabled/' /etc/selinux/config # 永久 setenforce 0 # 临时 # 关闭 swap swapoff -a # 临时 sed -ri 's/.*swap.*/#&/' /etc/fstab # 永久 # 根据规划设置主机名 hostnamectl set-hostname <hostname> # 在 master 添加 hosts cat >> /etc/hosts << EOF 172.16.90.147 m1 172.16.90.148 n1 EOF # 将桥接的 IPv4 流量传递到 iptables 的链 cat > /etc/sysctl.d/k8s.conf << EOF net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 EOF # 生效 sysctl --system # 时间同步 yum install ntpdate -y ntpdate time.windows.com
- 为etcd自签证书
# 准备 cfssl 证书生成工具 # cfssl 是一个开源的证书管理工具,使用 json 文件生成证书,相比 openssl 更方便使用。 找任意一台服务器操作,这里用 Master 节点 # 下载 wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64 mv cfssl_linux-amd64 /usr/local/bin/cfssl mv cfssljson_linux-amd64 /usr/local/bin/cfssljson mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo # 生成 Etcd 证书 # 自签证书颁发机构(CA) # 创建工作目录 [root@m1 TSL]# mkdir -p ~/TLS/{etcd,k8s} [root@m1 TSL]# cd TLS/etcd # 自签CA [root@m1 TSL]# cat > ca-config.json<< EOF { "signing": { "default": { "expiry": "87600h" }, "profiles": { "www": { "expiry": "87600h", "usages": ["signing", "key encipherment", "server auth", "client auth"] } } } } EOF [root@m1 TSL]# cat > ca-csr.json<< EOF { "CN": "etcd CA", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Beijing", "ST": "Beijing" } ] } EOF # 生成证书 [root@m1 etcd]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca - [root@m1 etcd]# ls *pem ca-key.pem ca.pem # 使用自签 CA 签发 Etcd HTTPS 证书 # 创建证书申请文件 [root@m1 etcd]# cat > server-csr.json<< EOF { "CN": "etcd", "hosts": [ "172.16.90.147", "172.16.90.148" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing" } ] } EOF # 注意:上述文件 hosts 字段中 IP 为所有 etcd 节点的集群内部通信 IP,一个都不能少!为了 方便后期扩容可以多写几个预留的 IP。 # 生成证书 [root@m1 etcd]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server #查看结果 [root@m1 etcd]# ls server*pem server-key.pem server.pem
- 部署etcd集群
# 以下在节点 1 上操作,为简化操作,待会将节点 1 生成的所有文件拷贝到节点 2 和节点 3. # 创建工作目录并解压二进制包 [root@m1 ~]# mkdir -p /opt/etcd/{bin,cfg,ssl} [root@m1 ~]# tar zxvf etcd-v3.4.9-linux-amd64.tar.gz [root@m1 ~]# mv etcd-v3.4.9-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/ [root@m1 opt]# tree etcd etcd ├── bin │ ├── etcd │ └── etcdctl ├── cfg └── ssl # 创建etcd配置文件 [root@m1 ~]# cat > /opt/etcd/cfg/etcd.conf << EOF #[Member] #ETCD_NAME:节点名称,集群中唯一 ETCD_NAME="etcd-1" #ETCD_DATA_DIR:数据目录 ETCD_DATA_DIR="/var/lib/etcd/default.etcd" #ETCD_LISTEN_PEER_URLS:集群通信监听地址 ETCD_LISTEN_PEER_URLS="https://172.16.90.147:2380" #ETCD_LISTEN_CLIENT_URLS:客户端访问监听地址 ETCD_LISTEN_CLIENT_URLS="https://172.16.90.147:2379" #[Clustering] #ETCD_INITIAL_ADVERTISE_PEER_URLS:集群通告地址 ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.16.90.147:2380" #ETCD_ADVERTISE_CLIENT_URLS:客户端通告地址 ETCD_ADVERTISE_CLIENT_URLS="https://172.16.90.147:2379" #ETCD_INITIAL_CLUSTER:集群节点地址 ETCD_INITIAL_CLUSTER="etcd-1=https://172.16.90.147:2380,etcd-2=https://172.16.90.148:2380" #ETCD_INITIAL_CLUSTER_TOKEN:集群 Token ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" #ETCD_INITIAL_CLUSTER_STATE:加入集群的当前状态,new 是新集群,existing 表示加入 已有集群 ETCD_INITIAL_CLUSTER_STATE="new" #systemd 管理 etcd [root@m1 /]# cat > /usr/lib/systemd/system/etcd.service << EOF [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] Type=notify EnvironmentFile=/opt/etcd/cfg/etcd.conf ExecStart=/opt/etcd/bin/etcd \ --cert-file=/opt/etcd/ssl/server.pem \ --key-file=/opt/etcd/ssl/server-key.pem \ --peer-cert-file=/opt/etcd/ssl/server.pem \ --peer-key-file=/opt/etcd/ssl/server-key.pem \ --trusted-ca-file=/opt/etcd/ssl/ca.pem \ --peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \ --logger=zap Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF # 笔者再启动etcd服务时报错,经排查因为 \ 引发,改后可行 [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] Type=notify EnvironmentFile=/opt/etcd/cfg/etcd.conf ExecStart=/opt/etcd/bin/etcd --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --peer-cert-file=/opt/etcd/ssl/server.pem --peer-key-file=/opt/etcd/ssl/server-key.pem --trusted-ca-file=/opt/etcd/ssl/ca.pem --peer-trusted-ca-file=/opt/etcd/ssl/ca.pem --logger=zap Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target # 拷贝刚才生成的证书 [root@m1 ssl]# cp ~/TLS/etcd/ca*pem ~/TLS/etcd/server*pem /opt/etcd/ssl/ [root@m1 ssl]# ls ca-key.pem ca.pem server-key.pem server.pem # 将上面mastert所有生成的文件拷贝到从节点 [root@m1 system]# scp -r /opt/etcd/ root@172.16.90.148:/opt/ [root@m1 system]# scp /usr/lib/systemd/system/etcd.service root@172.16.90.148:/usr/lib/systemd/system/ # 到从节点点修改配置文件 [root@n1 ~]# vim /opt/etcd/cfg/etcd.conf #[Member] ETCD_NAME="etcd-2" ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://172.16.90.148:2380" ETCD_LISTEN_CLIENT_URLS="https://172.16.90.148:2379" #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.16.90.148:2380" ETCD_ADVERTISE_CLIENT_URLS="https://172.16.90.148:2379" ETCD_INITIAL_CLUSTER="etcd-1=https://172.16.90.147:2380,etcd-2=https://172.16.90.148:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" # 启动并设置开机启动 # 重新服务的配置文件 [root@m1 system]# systemctl daemon-reload #启动etcd服务 [root@m1 system]# systemctl start etcd #有错误可用此命令查看日志;也可检查是否启动 [root@m1 system]# systemctl status etcd.service #将etcd服务设置为开机启动 [root@m1 system]# systemctl enable etcd # 在master节点执行以下命令,检查是否启动成功 [root@m1 ~]# ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://172.16.90.147:2379,https://172.16.90.148:2379" endpoint health https://172.16.90.147:2379 is healthy: successfully committed proposal: took = 23.120604ms https://172.16.90.148:2379 is healthy: successfully committed proposal: took = 24.304144ms #如果输出上面信息,就说明集群部署成功。如果有问题第一步先看日志: /var/log/message 或 journalctl -u etcd
- 安装docker
# 以下在所有节点操作。这里采用二进制安装,用 yum 安装也一样。 # 解压二进制包 [root@m1 ~]# tar zxvf docker-19.03.9.tgz [root@m1 ~]# mv docker/* /usr/bin # systemd 管理 docker [root@m1 ~]# vim /usr/lib/systemd/system/docker.service [Unit] Description=Docker Application Container Engine Documentation=https://docs.docker.com After=network-online.target firewalld.service Wants=network-online.target [Service] Type=notify ExecStart=/usr/bin/dockerd ExecReload=/bin/kill -s HUP $MAINPID LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity TimeoutStartSec=0 Delegate=yes KillMode=process Restart=on-failure StartLimitBurst=3 StartLimitInterval=60s [Install] WantedBy=multi-user.target # 创建配置文件 # registry-mirrors 阿里云镜像加速器 [root@m1 ~]# cat > /etc/docker/daemon.json << EOF > { > "registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"] > } > EOF
- 为apiserver自签证书
# 主从节点间的访问:添加可信任的ip列表 或者 写到CA证书发送 # 下列以 添加可信任的ip列表 为例进行演示 # 创建CA配置json文件 # 自签证书颁发机构(CA) [root@m1 k8s]# vim ca-config.json { "signing": { "default": { "expiry": "87600h" }, "profiles": { "kubernetes": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } } } [root@m1 k8s]# vim ca-csr.json { "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "HuBei", "ST": "WuHan", "O": "k8s", "OU": "System" } ] } # 使用自签 CA 签发 kube-apiserver HTTPS 证书 # 创建apiserver证书的所需配置文件 [root@m1 k8s]# vim kube-proxy-csr.json { "CN": "system:kube-proxy", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "HuBei", "ST": "WuHan", "O": "k8s", "OU": "System" } ] } # 创建证书申请文件: [root@m1 k8s]# vim server-csr.json { "CN": "kubernetes", "hosts": [ "10.0.0.1", "127.0.0.1", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local", "172.16.90.147", "172.16.90.148", "172.16.90.149", "172.16.90.150", "172.16.90.151" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "HuBei", "ST": "WuHan", "O": "k8s", "OU": "System" } ] } # 注:host中的最后几个IP为需要连接apiserver的IP,一般为master集群的所有IP,和负载均衡LB的所有IP和VIP,本文中的IP # "10.16.8.150", master01 # "10.16.8.151", master02 # "10.16.8.156", LB # "10.16.8.155", 备用IP # "10.16.8.164" 备用IP # 其中10.16.8.168即可信任的IP列表 # 原配置: # ... # "10.0.0.1", # "127.0.0.1", # "kubernetes", # "kubernetes.default", # "kubernetes.default.svc", # "kubernetes.default.svc.cluster", # "kubernetes.default.svc.cluster.local", # "10.16.8.150", # "10.16.8.151", # "10.16.8.156", # "10.16.8.155", # "10.16.8.164" # ... # 自建CA,生成证书 [root@m1 k8s]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca - # 生成证书 [root@k8s-master01 k8s]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server [root@k8s-master01 k8s]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy # 检查证书生成情况 [root@m1 k8s]# ll *.pem [root@m1 k8s]# ll *.pem -rw-------. 1 root root 1679 7月 28 19:14 ca-key.pem -rw-r--r--. 1 root root 1346 7月 28 19:14 ca.pem -rw-------. 1 root root 1675 7月 28 19:24 kube-proxy-key.pem -rw-r--r--. 1 root root 1391 7月 28 19:24 kube-proxy.pem -rw-------. 1 root root 1675 7月 28 19:22 server-key.pem -rw-r--r--. 1 root root 1635 7月 28 19:22 server.pem # 启动并设置开机启动 systemctl daemon-reload systemctl start docker systemctl enable docker
- 部署master组件(由于未知错误,笔者CentOS7无法识别 “”,此处有*.conf 和 *.server文件集合,验证码:nht1)
# 注:打开链接你会发现里面有很多包,下载一个 server 包就够了,包含了 Master 和 Worker Node 二进制文件。 # 解压二进制包 [root@m1 ~]# mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs} [root@m1 kubernetes]# tar -xzvf kubernetes-server-linux-amd64.tar.gz [root@m1 ~]# cd kubernetes/server/bin [root@m1 bin]# cp kube-apiserver kube-scheduler kube-controller-manager /opt/kubernetes/bin [root@m1 bin]# cp kubectl /usr/bin/ # 部署 kube-apiserver # 创建配置文件 [root@m1 bin]# vim /opt/kubernetes/cfg/kube-apiserver.conf KUBE_APISERVER_OPTS="--logtostderr=false \ --v=2 \ --log-dir=/opt/kubernetes/logs \ --etcd-servers=https://172.16.90.147:2379,https://172.16.90.148:2379 \ --bind-address=172.16.90.147 \ --secure-port=6443 \ --advertise-address=172.16.90.147 \ --allow-privileged=true \ --service-cluster-ip-range=10.0.0.0/24 \ --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \ --authorization-mode=RBAC,Node \ --enable-bootstrap-token-auth=true \ --token-auth-file=/opt/kubernetes/cfg/token.csv \ --service-node-port-range=30000-32767 \ --kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \ --kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \ --tls-cert-file=/opt/kubernetes/ssl/server.pem \ --tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \ --client-ca-file=/opt/kubernetes/ssl/ca.pem \ --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \ --etcd-cafile=/opt/etcd/ssl/ca.pem \ --etcd-certfile=/opt/etcd/ssl/server.pem \ --etcd-keyfile=/opt/etcd/ssl/server-key.pem \ --audit-log-maxage=30 \ --audit-log-maxbackup=3 \ --audit-log-maxsize=100 \ --audit-log-path=/opt/kubernetes/logs/k8s-audit.log" # 配置详解 # –logtostderr:启用日志 # —v:日志等级 # –log-dir:日志目录 # –etcd-servers:etcd 集群地址 # –bind-address:监听地址 # –secure-port:https 安全端口 # –advertise-address:集群通告地址 # –allow-privileged:启用授权 # –service-cluster-ip-range:Service 虚拟 IP 地址段 # –enable-admission-plugins:准入控制模块 # –authorization-mode:认证授权,启用 RBAC 授权和节点自管理 # –enable-bootstrap-token-auth:启用 TLS bootstrap 机制 # –token-auth-file:bootstrap token 文件 # –service-node-port-range:Service nodeport 类型默认分配端口范围 # –kubelet-client-xxx:apiserver 访问 kubelet 客户端证书 # –tls-xxx-file:apiserver https 证书 # –etcd-xxxfile:连接 Etcd 集群证书 # –audit-log-xxx:审计日志 # 拷贝刚才生成的证书,把刚才生成的证书拷贝到配置文件中的路径 [root@m1 bin]# cp ~/TLS/k8s/ca*pem ~/TLS/k8s/server*pem /opt/kubernetes/ssl/ # 启用 TLS Bootstrapping 机制 # TLS Bootstraping:Master apiserver 启用 TLS 认证后,Node 节点 kubelet 和 kube- proxy 要与 kube-apiserver 进行通信,必须使用 CA 签发的有效证书才可以,当 Node 节点很多时,这种客户端证书颁发需要大量工作,同样也会增加集群扩展复杂度。为了 简化流程,Kubernetes 引入了 TLS bootstraping 机制来自动颁发客户端证书,kubelet 会以一个低权限用户自动向 apiserver 申请证书,kubelet 的证书由 apiserver 动态签署。 # 所以强烈建议在 Node 上使用这种方式,目前主要用于 kubelet,kube-proxy 还是由我 们统一颁发一个证书。 # 创建上述配置文件中 token 文件: [root@m1 bin]# vim /opt/kubernetes/cfg/token.csv c47ffb939f5ca36231d9e3121a252940,kubelet-bootstrap,10001,"system:node-bootstrapper" # 格式:token,用户名,UID,用户组。token 也可自行生成替换: # head -c 16 /dev/urandom | od -An -t x | tr -d ' ' # systemd 管理 apiserver [root@m1 bin]# vim /usr/lib/systemd/system/kube-apiserver.service [Unit] Description=Kubernetes API Server Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS Restart=on-failure [Install] WantedBy=multi-user.target # 启动并设置开机启动 [root@m1 ~]# systemctl daemon-reload [root@m1 ~]# systemctl start kube-apiserver [root@m1 ~]# systemctl enable kube-apiserver # 授权 kubelet-bootstrap 用户允许请求证书 [root@m1 k8s]# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap # 部署 kube-controller-manager [root@m1 k8s]# vim /opt/kubernetes/cfg/kube-controller-manager.conf KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \ --v=2 \ --log-dir=/opt/kubernetes/logs \ --leader-elect=true \ --master=127.0.0.1:8080 \ --bind-address=127.0.0.1 \ --allocate-node-cidrs=true \ --cluster-cidr=10.244.0.0/16 \ --service-cluster-ip-range=10.0.0.0/24 \ --cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \ --cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \ --root-ca-file=/opt/kubernetes/ssl/ca.pem \ --service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \ --experimental-cluster-signing-duration=87600h0m0s" # -master:通过本地非安全本地端口 8080 连接 apiserver # -leader-elect:当该组件启动多个时,自动选举(HA) # -cluster-signing-cert-file/–cluster-signing-key-file:自动为 kubelet 颁发证书 的 CA,与 apiserver 保持一致 # systemd 管理 controller-manager [root@m1 k8s]# vim /usr/lib/systemd/system/kube-controller-manager.service [Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS Restart=on-failure [Install] WantedBy=multi-user.target # 启动并设置开机启动 [root@m1 k8s]# systemctl daemon-reload [root@m1 k8s]# systemctl start kube-controller-manager [root@m1 k8s]# systemctl enable kube-controller-manager # 部署 kube-scheduler [root@m1 k8s]# vim /opt/kubernetes/cfg/kube-scheduler.conf KUBE_SCHEDULER_OPTS="--logtostderr=false \ --v=2 \ --log-dir=/opt/kubernetes/logs \ --leader-elect \ --master=127.0.0.1:8080 \ --bind-address=127.0.0.1" # –master:通过本地非安全本地端口 8080 连接 apiserver。 # –leader-elect:当该组件启动多个时,自动选举(HA) # systemd 管理 scheduler [root@m1 k8s]# vim /usr/lib/systemd/system/kube-scheduler.service [Unit] Description=Kubernetes Scheduler Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS Restart=on-failure [Install] WantedBy=multi-user.target # 启动并设置开机启动 [root@m1 k8s]# systemctl daemon-reload [root@m1 k8s]# systemctl start kube-scheduler [root@m1 k8s]# systemctl enable kube-scheduler # 查看集群状态 # 所有组件都已经启动成功,通过 kubectl 工具查看当前集群组件状态: Warning: v1 ComponentStatus is deprecated in v1.19+ NAME STATUS MESSAGE ERROR controller-manager Healthy ok scheduler Healthy ok etcd-0 Healthy {"health":"true"} etcd-1 Healthy {"health":"true"} # 如上输出说明 Master 节点组件运行正常。
- 部署node组件(由于未知错误,笔者CentOS7无法识别 “”,此处有*.conf 和 *.server文件集合,验证码:nht1)
# 创建工作目录并拷贝二进制文件 # 在所有 worker node 创建工作目录 [root@n1 ~]# mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs} # 将m1上证书文件拷贝到n1 [root@m1 k8s]# scp ~/TLS/k8s/ca*pem ~/TLS/k8s/kube-proxy*pem root@172.16.90.148:/opt/kubernetes/ssl/ # 解压文件并拷贝配置 [root@n1 ~]# tar -xzvf kubernetes-node-linux-amd64.tar.gz [root@n1 ~]# cd kubernetes/node/bin/ [root@n1 bin]# ls kubeadm kubectl kubelet kube-proxy [root@n1 bin]# cp kubelet kube-proxy /opt/kubernetes/bin [root@n1 bin]# cp kubectl /usr/bin/ # 部署 kubelet # 创建配置文件 [root@n1 bin]# vim /opt/kubernetes/cfg/kubelet.conf KUBELET_OPTS="--logtostderr=false \ --v=2 \ --log-dir=/opt/kubernetes/logs \ --hostname-override=m1 \ --network-plugin=cni \ --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \ --bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \ --config=/opt/kubernetes/cfg/kubelet-config.yml \ --cert-dir=/opt/kubernetes/ssl \ --pod-infra-container-image=lizhenliang/pause-amd64:3.0" # –hostname-override:显示名称,集群中唯一 # –network-plugin:启用 CNI # –kubeconfig:空路径,会自动生成,后面用于连接 apiserver # –bootstrap-kubeconfig:首次启动向 apiserver 申请证书 # –config:配置参数文件 # –cert-dir:kubelet 证书生成目录 # –pod-infra-container-image:管理 Pod 网络容器的镜像 # 配置参数文件 [root@n1 bin]# vim /opt/kubernetes/cfg/kubelet-config.yml kind: KubeletConfiguration apiVersion: kubelet.config.k8s.io/v1beta1 address: 0.0.0.0 port: 10250 readOnlyPort: 10255 cgroupDriver: cgroupfs clusterDNS: - 10.0.0.2 clusterDomain: cluster.local failSwapOn: false authentication: anonymous: enabled: false webhook: cacheTTL: 2m0s enabled: true x509: clientCAFile: /opt/kubernetes/ssl/ca.pem authorization: mode: Webhook webhook: cacheAuthorizedTTL: 5m0s cacheUnauthorizedTTL: 30s evictionHard: imagefs.available: 15% memory.available: 100Mi nodefs.available: 10% nodefs.inodesFree: 5% maxOpenFiles: 1000000 maxPods: 110 # 生成 bootstrap.kubeconfig 文件 # 指定apiserver地址 [root@n1 kubernetes]# export KUBE_APISERVER="https://172.16.90.147:6443" # apiserver IP:PORT # 指定TOKEN值 [root@n1 kubernetes]# export TOKEN="c47ffb939f5ca36231d9e3121a252940" # 与 token.csv 里保持一致 # 设置集群参数 [root@n1 kubernetes]# kubectl config set-cluster kubernetes \ > --certificate-authority=/opt/kubernetes/ssl/ca.pem \ > --embed-certs=true \ > --server=${KUBE_APISERVER} \ > --kubeconfig=bootstrap.kubeconfig Cluster "kubernetes" set. # 设置客户端认证参数 [root@n1 kubernetes]# kubectl config set-credentials "kubelet-bootstrap" \ > --token=${TOKEN} \ > --kubeconfig=bootstrap.kubeconfig User "kubelet-bootstrap" set. # 设置上下文参数 [root@n1 kubernetes]# kubectl config set-context default \ > --cluster=kubernetes \ > --user="kubelet-bootstrap" \ > --kubeconfig=bootstrap.kubeconfig Context "default" created. # 设置默认上下文 [root@n1 kubernetes]# kubectl config use-context default --kubeconfig=bootstrap.kubeconfig Switched to context "default". # –embed-certs 为 true 时表示将 certificate-authority 证书写入到生成的 bootstrap.kubeconfig 文件中 # 拷贝到配置文件路径 [root@n1 kubernetes]# cp bootstrap.kubeconfig /opt/kubernetes/cfg # systemd 管理 kubelet [root@n1 kubernetes]# vim /usr/lib/systemd/system/kubelet.service [Unit] Description=Kubernetes Kubelet After=docker.service [Service] EnvironmentFile=/opt/kubernetes/cfg/kubelet.conf ExecStart=/opt/kubernetes/bin/kubelet \$KUBELET_OPTS Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target # 启动并设置开机启动 [root@n1 kubernetes]# systemctl daemon-reload [root@n1 kubernetes]# systemctl start kubelet [root@n1 kubernetes]# systemctl enable kubelet # 批准 kubelet 证书申请并加入集群 # 查看 kubelet 证书请求 [root@m1 ~]# kubectl get csr NAME AGE SIGNERNAME REQUESTOR CONDITION node-csr-j8VQGozwmrCDsJfBXTtA7MYwlmSMULb8WacDPTSniDY 61s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending # 批准申请 [root@m1 ~]# kubectl certificate approve node-csr-j8VQGozwmrCDsJfBXTtA7MYwlmSMULb8WacDPTSniDY certificatesigningrequest.certificates.k8s.io/node-csr-j8VQGozwmrCDsJfBXTtA7MYwlmSMULb8WacDPTSniDY approved # 查看节点 [root@m1 ~]# kubectl get node NAME STATUS ROLES AGE VERSION m1 NotReady <none> 28s v1.19.13 # 注:由于网络插件还没有部署,节点会没有准备就绪 NotReady # 部署 kube-proxy # 创建配置文件 [root@n1 kubernetes]# vim /opt/kubernetes/cfg/kube-proxy.conf KUBE_PROXY_OPTS="--logtostderr=false \ --v=2 \ --log-dir=/opt/kubernetes/logs \ --config=/opt/kubernetes/cfg/kube-proxy-config.yml" # 配置参数文件 [root@n1 kubernetes]# vim /opt/kubernetes/cfg/kube-proxy-config.yml kind: KubeProxyConfiguration apiVersion: kubeproxy.config.k8s.io/v1alpha1 bindAddress: 0.0.0.0 metricsBindAddress: 0.0.0.0:10249 clientConnection: kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig hostnameOverride: n1 clusterCIDR: 10.0.0.0/24 # 生成 kubeconfig 文件 # m1已经设置集群参数KUBE_APISERVER,这里不再设置 # 设置集群参数 [root@n1 kubernetes]# kubectl config set-cluster kubernetes \ > --certificate-authority=/opt/kubernetes/ssl/ca.pem \ > --embed-certs=true \ > --server=${KUBE_APISERVER} \ > --kubeconfig=kube-proxy.kubeconfig Cluster "kubernetes" set. # 设置客户端认证参数 [root@n1 kubernetes]# kubectl config set-credentials kube-proxy \ > --client-certificate=/opt/kubernetes/ssl/kube-proxy.pem \ > --client-key=/opt/kubernetes/ssl/kube-proxy-key.pem \ > --embed-certs=true \ > --kubeconfig=kube-proxy.kubeconfig User "kube-proxy" set. # 设置上下文参数 [root@n1 kubernetes]# kubectl config set-context default \ > --cluster=kubernetes \ > --user=kube-proxy \ > --kubeconfig=kube-proxy.kubeconfig Context "default" created. # 设置默认上下文 [root@n1 kubernetes]# kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig Switched to context "default". # 设置集群参数和客户端认证参数时 –embed-certs 都为 true,这会将 certificate-authority、client-certificate 和 client-key 指向的证书文件内容写入到生成的 kube-proxy.kubeconfig 文件中 # 新增节点时只需,将bootstrap.kubeconfig和kube-proxy.kubeconfig文件分发到各node节点上 # 拷贝到配置文件指定路径 [root@n1 kubernetes]# cp kube-proxy.kubeconfig /opt/kubernetes/cfg/ # systemd 管理 kube-proxy [root@n1 kubernetes]# vim /usr/lib/systemd/system/kube-proxy.service [Unit] Description=Kubernetes Proxy After=network.target [Service] EnvironmentFile=/opt/kubernetes/cfg/kube-proxy.conf ExecStart=/opt/kubernetes/bin/kube-proxy \$KUBE_PROXY_OPTS Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target # 启动并设置开机启动 [root@n1 kubernetes]# systemctl daemon-reload [root@n1 kubernetes]# systemctl start kube-proxy [root@n1 kubernetes]# systemctl status kube-proxy ● kube-proxy.service - Kubernetes Proxy Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; disabled; vendor preset: disabled) Active: active (running) since 六 2021-07-31 19:03:13 CST; 2s ago ... [root@n1 kubernetes]# systemctl enable kube-proxy
- 部署集群(CNI)网络
# 解压二进制包并移动到默认工作目录 [root@n1 ~]# mkdir -p /opt/cni/bin [root@n1 ~]# tar zxvf cni-plugins-linux-amd64-v0.8.6.tgz -C /opt/cni/bin # 部署CNI网络 [root@m1 ~]# wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml [root@m1 ~]# sed -i -r "s#quay.io/coreos/flannel:.*-amd64#lizhenliang/flannel:v0.12.0- amd64#g" kube-flannel.yml # 默认镜像地址无法访问,修改为 docker hub 镜像仓库 [root@m1 ~]# kubectl apply -f kube-flannel.yml [root@m1 ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE kube-flannel-ds-2kqmz 1/1 Running 0 6m26s [root@m1 ~]# kubectl get node NAME STATUS ROLES AGE VERSION m1 Ready <none> 81m v1.19.13 # 部署好网络插件,Node准备就绪 # 授权 apiserver 访问 kubelet [root@m1 ~]# vim apiserver-to-kubelet-rbac.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" labels: kubernetes.io/bootstrapping: rbac-defaults name: system:kube-apiserver-to-kubelet rules: - apiGroups: - "" resources: - nodes/proxy - nodes/stats - nodes/log - nodes/spec - nodes/metrics - pods/log verbs: - "*" --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: system:kube-apiserver namespace: "" roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:kube-apiserver-to-kubelet subjects: - apiGroup: rbac.authorization.k8s.io kind: User name: kubernetes [root@m1 ~]# kubectl apply -f apiserver-to-kubelet-rbac.yaml
- 新增加 Worker Node
# 拷贝已部署好的 Node 相关文件到新节点 # 在n1(172.16.90.148)节点将Worker Node涉及文件拷贝到新节点172.16.90.149 [root@n1 ~]# scp -r /opt/kubernetes root@172.16.90.149:/opt/ [root@n1 ~]# scp -r /usr/lib/systemd/system/{kubelet,kube-proxy}.service root@172.16.90.149:/usr/lib/systemd/system [root@n1 ~]# scp -r /opt/cni/ root@172.16.90.149:/opt/ [root@n1 ~]# scp /opt/kubernetes/ssl/ca.pem root@172.16.90.149:/opt/kubernetes/ssl # 删除kubelet证书和kubeconfig文件 [root@n2 ~]# rm /opt/kubernetes/cfg/kubelet.kubeconfig [root@n2 ~]# rm -f /opt/kubernetes/ssl/kubelet* # 注:这几个文件是证书申请审批后自动生成的,每个Node不同,必须删除重新生成 # 修改主机名 [root@n2 ~]# vim /opt/kubernetes/cfg/kubelet.conf ... --hostname-override=n2 ... [root@n2 ~]# vim /opt/kubernetes/cfg/kube-proxy-config.yml ... hostnameOverride: n2 ... # 启动并设置开机启动 [root@n2 ~]# systemctl daemon-reload [root@n2 ~]# systemctl start kubelet [root@n2 ~]# systemctl enable kubelet [root@n2 ~]# systemctl start kube-proxy [root@n2 ~]# systemctl enable kube-proxy # 在Master上批准新Node kubelet证书申请 # 与n1相同,此处略过 # 查看 Node 状态 [root@m1 ~]# Kubectl get node
- 测试集群
[root@m1 ~]# kubectl create deployment nginx --image=nginx deployment.apps/nginx created [root@m1 ~]# kubectl expose deployment nginx --port=80 --type=NodePort service/nginx exposed [root@m1 ~]# kubectl get pod,svc NAME READY STATUS RESTARTS AGE pod/nginx-6799fc88d8-j5fn2 0/1 ContainerCreating 0 20s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 3d23h service/nginx NodePort 10.0.0.70 <none> 80:30465/TCP 7s # 浏览器访问地址:http://NodeIP:Port(例如:http://172.16.90.148:30465)
两种搭建方式对比:对两种搭建方式的总结
- kubeadm搭建k8s集群
- 安装linux系统虚拟机,并对系统进行初始化操作
- 在所有节点(master、node)安装docker(包括下载、修改仓库地址和yum源成阿里地址)、kubeadm(
kubeadm join <Master 节点的 IP 和端口 >
)、kubelet和kubectl - 在master节点执行初始化命令操作(
kubeadm init \
,指定镜像源,使用阿里云镜像) - 部署网络CNI插件(
kubectl apply -f kube-flannel.yml
) - 在所有node节点上使用join命令把node节点添加到master
- 二进制方式搭建k8s集群
- 安装linux系统虚拟机,并对系统进行初始化操作
- 生成cfssl自签证书(
ca-key.pem、ca.pem、server-key.pem、server.pem
) - 部署ectd集群(本质是把etcd服务,交给systemd管理。把证书复制过来,启动并设置开机启动)
- 为apiserver自签证书(生成过程和etcd类似)
- 部署master组件(apiserver、controller-manager、scheduler下载二进制文件进行安装,交给systemd管理组件启动并设置开机启动)
- 部署node组件(docker、kubelet、kube-proxy下载二进制文件进行安装,交给systemd管理组件启动并设置开机启动,最后批准kubelet证书申请并加入集群)
- 部署CNI网络插件
3. 核心技术(上)
kubernetes 集群命令行工具 kubectl:kubectl 是 Kubernetes 集群的命令行工具,通过 kubectl 能够对集群本身进行管理,并能 够在集群上进行容器化应用的安装部署。kubectl命令语法:kubectl [command] [TYPE] [NAME] [flags]
(command指定要对资源执行的操作,例如create、get、describe和delete;TYPE指定资源类型,资源类型是大小写敏感的,开发者能够以单数、复数和缩略的形式,例如kubectl get pod pod1
;NAME指定资源名称,名称大小写敏感,如果省略名称,则会显示所有的资源例如:kubectl get pods
);flags指定可选的参数,例如可用-s或者-server参数指定kubernetes API server的地址和端口。获取kubectl帮助的方法:kubectl --help
,具体看某个操作kubectl get --help
。kubectl 子命令使用分类:
- 基础命令
- 部署和集群管理命令
- 故障和调试命令
- 其他命令
kubernetes集群YAML文件详解:k8s集群中对资源管理和资源对象编排部署都可以通过声明样式(YAML)文件来解决,我们把这种文件叫做资源清单文件,通过 kubectl 命令直接使用资源清单文件就可以实现对大量的资源对象进行编排部署了。语法格式:通过缩进表示层级关系;不能用tab进行缩进,只能用空格;一般开通缩进两个空格;字符后缩进一个空格,比如冒号,逗号等后面;使用—表示新的yaml文件开始;使用#表示注释。yaml文件有控制器定义和被控制对象组成:
快速编写yaml文件:使用kubectl create命令生成yaml文件,命令:kubectl create deployment web --image=nginx -o yaml --dry-run > web.yaml
(用于还未真正部署);使用kubectl get命令导出yaml文件kubectl get deploy nginx -o=yaml --export > web.yaml
(用于已经部署好的情况)
Pod简介:Pod 是 k8s 系统中可以创建和管理的最小单元,是资源对象模型中由用户创建或部署的最 小资源对象模型,也是在 k8s 上运行容器化应用的资源对象,其他的资源对象都是用来支 撑或者扩展 Pod 对象功能的。k8s 不会直接处理容器,而是 Pod,Pod 是由一个或多个container组成
Pod 是 Kubernetes 的最重要概念,每一个 Pod 都有一个特殊的被称为”根容器“的 Pause 容器。Pause 容器对应的镜 像属于 Kubernetes 平台的一部分,除了 Pause 容器,每个 Pod 还包含一个或多个紧密相关的用户业务容器(加入到Pause容器中)
pod基本概念:最小部署单元、包含多个容器(一组容器的集合)、各个pod中容器共享网络命名空间、pod是短暂的
pod存在意义:创建容器使用docker,一个docker对应一个容器(便于管理),一个容器对应一个进程,一个容器运行一个应用程序;pod是多进程设计,运行多个应用程序(一个pod有多个容器,一个容器里面运行一个应用程序);pod存在为了亲密性应用(两个应用之间进行交互、网络之间调用、两个应用需要频繁调用)
pod与应用、容器、节点、pos差别:每个pod都是应用的一个实例,有专用的IP;一个pod可以有多个容器,彼此之间共享网络和存储资源,每个pod中有一个pause容器保存所有的容器状态,通过管理pause容器,达到管理pod中所有容器的效果;同一个pod中的容器总会被调度到相同node节点,不同节点间pod的通信基于虚拟二层网络技术实现;普通的pod和静态pod
pod特性:共享网络,pod中容器之间通过linux的namespace和group机制进行隔离,所以要想实现网络共享其前提是pod中所有容器都在一个namespace里面,其实现原理是先创建pause容器(也称作info容器),然后将其他业务容器加入到pause容器中,使得所有业务容器处于同一namespace中,对外则暴露info容器的ip、mac、port等信息;共享存储,采用docker的Volumn数据卷进行持久化存储到某一特定区间,所有node都可以访问该区间;生命周期短暂,当pod所在节点发生故障,那么该节点的pod会被调度到其他节点,而且被重新调度的pod是一个全新的pod;网络平坦,K8S集群中的所有Pod都在同一个共享网络地址空间中,也就是说每个Pod都可以通过其他Pod的IP地址来实现访问
pod常见配置:拉取策略、资源限制、重启机制和健康检查
- 镜像拉取策略
- 资源限制
- 重启策略
- 健康检查
pod调度策略:主要包括创建pod流程和pod调度影响因素两部分
- 创建pod流程:首先在master通过apiserver创建pod节点,随后相关信息持久化到etcd中,此时scheduler是实时监控apiserver当检查到有pod创建时,它会通过apiserver读取到该pod存放在etcd的信息,并通过自身调度算法将该pod调度到某个node节点上;被调度的node节点通过kubelet访问apiserver,并读取到etcd中存放的信息,随后通知docker创建该容器。
- 影响pod调度的因素:主要有资源限制(前面所讲的request、limit)、节点选择器、节点亲和性、污点和污点容忍这几个方面
controller简介:在集群上管理和运行容器的对象,Pod是通过Controller实现应用的运维(比如伸缩、滚动升级等等),Pod和Controller之间通过label标签建立关系,其图示如下:
常用控制器deployment:用于部署无状态应用(例如web应用、之前部署的nginx,都是无状态应用)、管理Pod和ReplicaSet、部署和滚动升级等功能。常用于web服务和微服务的场景,以下是使用deploy部署应用的过程:
- 导出yaml文件
kubectl create deployment web --image=nginx --dry-run -o yaml > web.yaml
- 使用yaml部署应用
kubectl apply -f web.yaml
- 查看应用的启动情况
kubectl get pods
- 对外发布(暴露端口)
kubectl expose deployment web --port=80 --type=NodePort --target-port=80 --name=web1 -o yaml > web1.yaml
,并执行该yaml文件kubectl apply -f web1.yaml
- 查看运行情况
kubectl get pods,svc
- 应用升级
kubectl set image deployment web nginx=nignx:1.15
,其升级过程,首先k8s下载1.15版本的nginx(期间1.1.4并不停止服务),下载完成后1.15版本的nginx会替换1.14版本的nginx,其升级过程中服务不中断,最后查看升级状态kubectl rollout status deployment web
- 弹性回滚,首先可通过
kubectl rollout history deployment web
查看升级版本,通过kubectl rollout undo deployment web
回滚到上一个版本,也可以通过kubectl rollout undo deployment web --to-revision=2
回到指定版本,可通过kubectl rollout status deployment web
查看当前状态。利用命令kubectl scale deployment web --relicas=10
可弹性伸缩新增10个相同的pod
常用控制器statefueset:pod分为无状态pod(认为pod都是一样的;没有顺序要求;不用考虑在哪个node运行;随意进行伸缩和拓展)和有状态pod(以上因素都需要考虑到;让每个pod独立的保持pod启动顺序和唯一性;唯一的网络表示符,持久存储;有序,比如mysql主从),而statefueset用于有状态pod控制。其创建过程分为无头service和有状态应用:
常用控制器DaemonSet:用于部署守护进程,在每个node上运行一个pod,新加入的node也同样运行在一个pod里面(例如:在每个node节点安装数据采集工具)
常用控制器Job:一次性任务
常用控制器cronJob:定时任务
Service简介:service用于定义一组pod访问规则。作用:防止pod失联、定义一组pod访问策略(负载均衡)。起源如下图:
常用的Service类型:ClusterIP、NodePort、LoadBalancer
- ClusterIP:集群内部使用(例如node1节点访问启动的pod,可利用
kubectl get svc
查看分配的ip) - NodePort:对外访问应用使用(例如:通过ip访问系统,在浏览器访问pod里面的nginx)
- loadBalancer:对外访问应用使用,公有云亦可
4. 核心技术(下)
配置管理:以是否加密来区分使用,分为Secret(常用作凭证,作用是将加密数据存在etcd里面,让pod容器以挂载volume方式进行访问)和ConfigMap(常用作配置文件,作用是存储不加密数据到etcd,让Pod以变量或者Volume挂载到容器中)
- secret的使用
- ConfigMap的使用
集群安全机制:访问k8s集群时候,需要经过以下三个步骤完成具体操作。而在进行访问的时候,过程中都需要经过apiserver,apiserver作为统一协调(类比于门卫)。访问过程中需要证书、token或者用户名+密码;如果访问pod则需要serviceAccount
- 认证:传输安全(对外不暴露8080端口,只能内部访问,对外使用端口6443)、认证(客户端认证常用方式有:https证书认证,基于ca证书;http tokens认证,通过token识别用户;http基本认证,用户名+密码方式认证)
- 鉴权(授权):基于RBAC进行鉴权操作;基于角色访问控制
- 准入控制:就是准入控制器的列表,如果列表有请求内容则通过,否则就拒绝
RBAC简介:基于角色的访问控制
- 角色:role(特定命名空间访问权限)和ClusterRole(所有命名空间访问权限)
- 角色绑定:roleBinding(角色绑定到主体)和ClusterRoleBinding(集群角色绑定到主体)
- 主体:user(用户)、group(用户组)、serviceAccount(服务账号)
RBAC实现鉴权:以下用实例的方式表述其过程(以下TLS文件是以二进制安装方式环境为例)
引入Ingress控制器:在使用Service里面的NodePort(把端口对外暴露,通过ip:port方式进行访问)时,其存在如下缺陷:在每个节点上都会起到端口,在访问时候通过任何检点,通过节点ip:port实现访问;意味着每个端口只能使用一次,一个端口对应一个应用。但是实际访问中都是用域名,根据不同域名跳转到不同服务,Ingress正是为了解决此问题而提出的解决方案,它使得pod和ingress通过service关联,ingress作为统一入口由service关联一组pod,其原理如下图所示:
使用ingress控制器:这里选择官方维护的nginx控制器进行部署,其步骤大致分为部署ingress controller和创建ingress规则,以下过程是使用Ingress对外暴露应用
-
创建nginx应用,对外暴露端口使用NodePort
-
部署ingress controller
apiVersion: v1 kind: Namespace metadata: name: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- kind: ConfigMap apiVersion: v1 metadata: name: nginx-configuration namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- kind: ConfigMap apiVersion: v1 metadata: name: tcp-services namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- kind: ConfigMap apiVersion: v1 metadata: name: udp-services namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- apiVersion: v1 kind: ServiceAccount metadata: name: nginx-ingress-serviceaccount namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: nginx-ingress-clusterrole labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx rules: - apiGroups: - "" resources: - configmaps - endpoints - nodes - pods - secrets verbs: - list - watch - apiGroups: - "" resources: - nodes verbs: - get - apiGroups: - "" resources: - services verbs: - get - list - watch - apiGroups: - "" resources: - events verbs: - create - patch - apiGroups: - "extensions" - "networking.k8s.io" resources: - ingresses verbs: - get - list - watch - apiGroups: - "extensions" - "networking.k8s.io" resources: - ingresses/status verbs: - update --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: Role metadata: name: nginx-ingress-role namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx rules: - apiGroups: - "" resources: - configmaps - pods - secrets - namespaces verbs: - get - apiGroups: - "" resources: - configmaps resourceNames: # Defaults to "<election-id>-<ingress-class>" # Here: "<ingress-controller-leader>-<nginx>" # This has to be adapted if you change either parameter # when launching the nginx-ingress-controller. - "ingress-controller-leader-nginx" verbs: - get - update - apiGroups: - "" resources: - configmaps verbs: - create - apiGroups: - "" resources: - endpoints verbs: - get --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: RoleBinding metadata: name: nginx-ingress-role-nisa-binding namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: nginx-ingress-role subjects: - kind: ServiceAccount name: nginx-ingress-serviceaccount namespace: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: nginx-ingress-clusterrole-nisa-binding labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: nginx-ingress-clusterrole subjects: - kind: ServiceAccount name: nginx-ingress-serviceaccount namespace: ingress-nginx --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-ingress-controller namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx spec: replicas: 1 selector: matchLabels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx template: metadata: labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx annotations: prometheus.io/port: "10254" prometheus.io/scrape: "true" spec: hostNetwork: true # wait up to five minutes for the drain of connections terminationGracePeriodSeconds: 300 serviceAccountName: nginx-ingress-serviceaccount nodeSelector: kubernetes.io/os: linux containers: - name: nginx-ingress-controller image: lizhenliang/nginx-ingress-controller:0.30.0 args: - /nginx-ingress-controller - --configmap=$(POD_NAMESPACE)/nginx-configuration - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services - --udp-services-configmap=$(POD_NAMESPACE)/udp-services - --publish-service=$(POD_NAMESPACE)/ingress-nginx - --annotations-prefix=nginx.ingress.kubernetes.io securityContext: allowPrivilegeEscalation: true capabilities: drop: - ALL add: - NET_BIND_SERVICE # www-data -> 101 runAsUser: 101 env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace ports: - name: http containerPort: 80 protocol: TCP - name: https containerPort: 443 protocol: TCP livenessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 10 readinessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP periodSeconds: 10 successThreshold: 1 timeoutSeconds: 10 lifecycle: preStop: exec: command: - /wait-shutdown --- apiVersion: v1 kind: LimitRange metadata: name: ingress-nginx namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx spec: limits: - min: memory: 90Mi cpu: 100m type: Container
-
创建ingress规则
apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: example-ingress spec: rules: - host: example.ingredemo.com http: paths: - path: / backend: serviceName: web servicePort: 80
-
在windows系统hosts文件中添加域名访问规则
-
验证
helm引入:之前部署应用的基本过程都是编写yaml文件、deployment、service最后ingress,该方式是和部署单一的、少数服务的应用,但是如果部署微服务项目,则可能有几十个服务,每个服务都有一套yaml文件,此时需要维护大量yaml文件,版本管理特别不方便。Helm就是为解决此问题而产生的解决方案。helm是一个Kubernetes的包管理工具,就像Linux下的包管理器,如yum/apt等,可以很方便的将之前打包好的yaml文件部署到kubernetes上。使用helm可以把所有yaml作为一个整体管理,以实现yaml高效复用。helm有三个重要概念分别是helm(一个命令行客户端工具)、chart(把yaml打包,是yaml集合)和release(基础chart部署实体,应用级别的版本管理)。helm在2019年发布V3版本,和之前版本相比有明显变化:V3版本中删除Tiller、release可以在不同命名空间重用、将chart推送到dorcker仓库中。
helm安装使用:主要包括helm安装、配置helm仓库、使用helm快速部署应用
# 安装helm
[root@k8s-master ~]# tar -xzvf helm-v3.0.0-linux-amd64.tar.gz
[root@k8s-master ~]# cd linux-amd64/
[root@k8s-master linux-amd64]# ls
helm LICENSE README.md
[root@k8s-master linux-amd64]# mv helm /usr/bin
# 配置helm仓库
# 添加存储库
[root@k8s-master linux-amd64]# helm repo add stable http://mirror.azure.cn/kubernetes/charts #微软仓库,这个仓库推荐,基本 上官网有的 chart 这里都有
[root@k8s-master linux-amd64]# helm repo add aliyun https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts #阿里云仓库
# 更新仓库源
[root@k8s-master linux-amd64]# helm repo update
# 查看配置的存储库
[root@k8s-master linux-amd64]# helm repo list
# 删除仓库
[root@k8s-master linux-amd64]# helm repo remove stable
# 使用helm快速部署应用
# 使用命令搜索应用,格式:helm search repo 名称
[root@k8s-master ~]# helm search repo weave
NAME CHART VERSION APP VERSION DESCRIPTION
aliyun/weave-cloud 0.1.2 Weave Cloud is a add-on to Kubernetes which pro...
aliyun/weave-scope 0.9.2 1.6.5 A Helm chart for the Weave Scope cluster visual...
# 根据搜索内容选择安装,格式:helm install 安装之后名称 搜索之后应用名称
[root@k8s-master ~]# helm install ui aliyun/weave-scope
# 查看安装之后状态
[root@k8s-master ~]# helm list
[root@k8s-master ~]# helm status ui
# 查应用状态发现,ui-weave-scope其TYPE为ClusterIP
[root@k8s-master ~]# kubectl get svc
# 修改其TYPE
[root@k8s-master ~]# kubectl edit svc ui-weave-scope
...
type: NodePort
...
# 查看修改后状态发现,ui-weave-scope其TYPE为NodePort
[root@k8s-master ~]# kubectl get svc
自定义chart部署:过程见以下命令
# 使用命令创建chart,格式为:helm create chart 名称
[root@k8s-master ~]# helm create mychart
Creating mychart
[root@k8s-master ~]# cd mychart/
#Chart.yaml:当前chart属性配置信息
#templates:编写yaml文件放在这个目录
#values.yaml:yaml文件可以使用全局变量
[root@k8s-master mychart]# ls
charts Chart.yaml templates values.yaml
# 在templates文件夹中创建两个yaml文件deployment.yaml和service.yaml
[root@k8s-master mychart]# cd templates/
[root@k8s-master templates]# ls
deployment.yaml _helpers.tpl ingress.yaml NOTES.txt serviceaccount.yaml service.yaml tests
[root@k8s-master templates]# rm -rf ./*
[root@k8s-master templates]# kubectl create deployment web1 --image=nginx --dry-run -o yaml > deployment.yaml
W0817 20:20:53.148698 15888 helpers.go:557] --dry-run is deprecated and can be replaced with --dry-run=client.
[root@k8s-master templates]# ls
deployment.yaml
[root@k8s-master templates]# kubectl expose deployment web1 --port=80 --target-port=80 --type=NodePort --dry-run -o yaml > service.yaml
W0817 20:24:22.445043 21372 helpers.go:557] --dry-run is deprecated and can be replaced with --dry-run=client.
Error from server (NotFound): deployments.apps "web1" not found
[root@k8s-master templates]# ls
deployment.yaml service.yaml
# 此时打开service.yaml发现为空
[root@k8s-master templates]# kubectl create deployment web1 --image=nginx
deployment.apps/web1 created
[root@k8s-master templates]# kubectl expose deployment web1 --port=80 --target-port=80 --type=NodePort --dry-run -o yaml > service.yaml
# 此时service.yaml里面就有内容
# 删掉web1,以helm方式部署
[root@k8s-master templates]# kubectl delete deployment web1
deployment.apps "web1" deleted
# 安装mychart
[root@k8s-master ~]# helm install web1 mychart/
# 检查
[root@k8s-master ~]# kubectl get pods
[root@k8s-master ~]# kubectl get svc
# 应用升级
[root@k8s-master ~]# helm upgrade web1 mychart/
chart模板使用:chart模板可实现yaml高效复用,其方式是通过chart的values.yaml文件定义全局变量传递参数(在values.yaml定义变量和值,在具体yaml文件中获取定义变量值),动态渲染模板,yaml内容动态传入参数生成(在yaml文件中大体只有image、tag、label、port和replicas这几个地方不同而已),其实现过程如下
# 在values.yaml定义变量和值
[root@k8s-master ~]# cd mychart/
[root@k8s-master mychart]# ls
charts Chart.yaml templates values.yaml
[root@k8s-master mychart]# vim values.yaml
...
replicas: 1
image: nginx
tag: 1.16
label: nginx
port: 80
...
# 在templates的yaml文件使用values.yaml定义变量
# 通过表达式使用全局变量,其格式为:{{.Values.变量名称}}
# 也常用{{.Release.Name}}避免生成随机名称
[root@k8s-master mychart]# cd templates/
[root@k8s-master templates]# vim deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Release.Name}}-deploy
spec:
replicas: 1
selector:
matchLabels:
app: {{ .Values.label}}
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: {{ .Values.label}}
spec:
containers:
- image: {{ .Values.image}}
name: nginx
resources: {}
status: {}
[root@k8s-master templates]# vim deployment.yaml
apiVersion: v1
kind: Service
metadata:
name: {{ .Release.Name}}-svc
spec:
ports:
- port: {{ .Values.port}}
protocol: TCP
targetPort: 80
selector:
app: {{ .Values.label}}
type: NodePort
status:
loadBalancer: {}
[root@k8s-master ~]# helm install --dry-run web2 mychart/
#检查
[root@k8s-master ~]# kubectl get pods
[root@k8s-master ~]# kubectl get svc
持久化存储:数据卷emptydir是本地存储,pod重启后数据会丢失,为了解决这一问题就需要对数据进行持久化存储。实现这一方案,k8s使用nfs网络存储实现pod重启数据还在,以下是实现过程
# 1. 准备环境:创建一台服务器安装nfs服务端(笔者这里ip设置为172.16.90.134)并关闭防火墙
[root@90143-k8s-nfs ~]# systemctl disable --now firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@90143-k8s-nfs ~]# firewall-cmd --list-all
FirewallD is not running
# 2. 安装nfs
[root@90143-k8s-nfs ~]# yum install -y nfs-utils
# 3. 创建挂载路径
[root@90143-k8s-nfs ~]# mkdir -p /data/nfs
# 4. 设置挂载路径
[root@90143-k8s-nfs ~]# vim /etc/exports
/data/nfs *(rw,no_root_squash)
# 5. 在k8s的node节点安装nfs
[root@k8s-node1 ~]# yum install -y nfs-utils
[root@k8s-node2 ~]# yum install -y nfs-utils
# 6. 在nfs服务器中启动服务
[root@90143-k8s-nfs ~]# systemctl start nfs
[root@90143-k8s-nfs ~]# ps -ef | grep nfs
avahi 5936 1 0 14:52 ? 00:00:00 avahi-daemon: running [90143-k8s-nfs.local]
root 27593 2 0 15:10 ? 00:00:00 [nfsd4_callbacks]
root 27599 2 0 15:10 ? 00:00:00 [nfsd]
root 27600 2 0 15:10 ? 00:00:00 [nfsd]
root 27601 2 0 15:10 ? 00:00:00 [nfsd]
root 27602 2 0 15:10 ? 00:00:00 [nfsd]
root 27603 2 0 15:10 ? 00:00:00 [nfsd]
root 27604 2 0 15:10 ? 00:00:00 [nfsd]
root 27605 2 0 15:10 ? 00:00:00 [nfsd]
root 27606 2 0 15:10 ? 00:00:00 [nfsd]
root 28030 7550 0 15:10 pts/0 00:00:00 grep --color=auto nfs
# 7. 在k8s集群部署应该用使用nfs持久网络存储
[root@k8s-master ~]# mkdir pv
[root@k8s-master ~]# vim pv/nfs-nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-dep1
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
volumeMounts:
- name: wwwroot
mountPath: /usr/share/nginx/html
ports:
- containerPort: 80
volumes:
- name: wwwroot
nfs:
server: 172.16.90.143
path: /data/nfs
[root@k8s-master ~]# cd pv
[root@k8s-master pv]# kubectl apply -f nfs-nginx.yaml
deployment.apps/nginx-dep1 created
[root@k8s-master pv]# kubectl describe pod nginx-dep1-776574d4d-hg647
[root@k8s-master pv]# kubectl exec -it nginx-dep1-776574d4d-mdtcd bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@nginx-dep1-776574d4d-mdtcd:/# ls /usr/share/nginx/html
[root@90143-k8s-nfs ~]# cd /data/nfs/
hello nfs
root@nginx-dep1-776574d4d-mdtcd:/# ls /usr/share/nginx/html
index.html
root@nginx-dep1-776574d4d-mdtcd:/# exit
exit
# 8. 验证,通过NodeIP:Port可浏览器查看
[root@k8s-master pv]# kubectl expose deployment nginx-dep1 --port=80 --target-port=80 --type=NodePort
service/nginx-dep1 exposed
[root@k8s-master pv]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 22d
nginx NodePort 10.105.227.129 <none> 80:30640/TCP 22d
nginx-dep1 NodePort 10.96.151.113 <none> 80:31987/TCP 12s
web1 NodePort 10.101.103.2 <none> 80:31093/TCP 2d5h
PV与PVC:PV(持久化存储,对存储资源进行抽象,对外提供可以调用的地方,本质是生产者)与PVC(用于调用,不需要关心内部实现细节,本质是消费者)其实现流程如下图所示:
# 避免影响删除nfs-nginx.yaml
[root@k8s-master pv]# kubectl delete -f nfs-nginx.yaml
deployment.apps "nginx-dep1" deleted
[root@k8s-master pv]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-6799fc88d8-lj24f 1/1 Running 0 22d
web-7866dfdb9f-7zg68 0/1 ImagePullBackOff 0 2d5h
web-96d5df5c8-br8md 1/1 Running 0 2d5h
# 定义pvc
[root@k8s-master pv]# vim pvc.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-dep1
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
volumeMounts:
- name: wwwroot
mountPath: /usr/share/nginx/html
ports:
- containerPort: 80
volumes:
- name: wwwroot
persistentVolumeClaim:
claimName: my-pvc
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: my-pvc
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi
[root@k8s-master pv]# kubectl apply -f pvc.yaml
deployment.apps/nginx-dep1 created
persistentvolumeclaim/my-pvc created
[root@k8s-master pv]# vim pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: my-pv
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteMany
nfs:
path: /data/nfs
server: 172.16.90.143
[root@k8s-master pv]# kubectl apply -f pv.yaml
persistentvolume/my-pv created
#检查
[root@k8s-master pv]# kubectl get pv,pvc
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/my-pv 5Gi RWX Retain Bound default/my-pvc 43s
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/my-pvc Bound my-pv 5Gi RWX 3m13s
[root@k8s-master pv]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-6799fc88d8-lj24f 1/1 Running 0 22d
nginx-dep1-69f5bb95b-bwn2g 1/1 Running 0 4m11s
nginx-dep1-69f5bb95b-gpn9f 1/1 Running 0 4m11s
nginx-dep1-69f5bb95b-k9xhw 1/1 Running 0 4m11s
web-7866dfdb9f-7zg68 0/1 ImagePullBackOff 0 2d5h
web-96d5df5c8-br8md 1/1 Running 0 2d5h
[root@k8s-master pv]# kubectl exec -it nginx-dep1-69f5bb95b-bwn2g bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@nginx-dep1-69f5bb95b-bwn2g:/# ls /usr/share/nginx/html/
index.html
5. 日志管理
6. 监控平台
集群资源监控:包括监控指标和监控平台,其中
- 监控指标
- 集群监控:节点资源利用率、节点数、运行pods
- Pod监控:容器指标、应用程序
- 监控平台搭建方案(prometheus+Grafana)
- prometheus:开源的;监控、报警、数据库;以HTTP协议周期性抓取被监控组件状态;不需要复杂的集成过程,使用http接口接入就可以
- Grafana:开源的数据分析和可视化工具;支持多种数据源
搭建过程:相关yaml文件地址(验证码:mzzv)
- 启动Prometheus和Grafana
[root@k8s-master ~]# mkdir pgmonitor [root@k8s-master ~]# cd pgmonitor/ # 将yaml文件上传到 [root@k8s-master pgmonitor]# ls grafana node-exporter.yaml prometheus [root@k8s-master pgmonitor]# vim node-exporter.yaml --- apiVersion: apps/v1 kind: DaemonSet metadata: name: node-exporter namespace: kube-system labels: k8s-app: node-exporter spec: selector: matchLabels: k8s-app: node-exporter ... # 部署守护进程 [root@k8s-master pgmonitor]# kubectl create -f node-exporter.yaml # 部署prometheus [root@k8s-master prometheus]# kubectl create -f rbac-setup.yaml [root@k8s-master prometheus]# kubectl create -f configmap.yaml [root@k8s-master prometheus]# vim prometheus.deploy.yml --- apiVersion: apps/v1 ... [root@k8s-master prometheus]# kubectl create -f prometheus.deploy.yml [root@k8s-master prometheus]# kubectl create -f prometheus.svc.yml # 检查 [root@k8s-master prometheus]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-59d64cd4d4-dqx8m 1/1 Running 0 23d coredns-59d64cd4d4-z8pdq 1/1 Running 0 23d etcd-k8s-master 1/1 Running 0 23d kube-apiserver-k8s-master 1/1 Running 0 23d kube-controller-manager-k8s-master 1/1 Running 0 23d kube-flannel-ds-h7v2g 1/1 Running 0 23d kube-flannel-ds-xmzfh 1/1 Running 0 23d kube-flannel-ds-z9nbj 1/1 Running 0 23d kube-proxy-6c9cd 1/1 Running 0 23d kube-proxy-cnvfg 1/1 Running 0 23d kube-proxy-p4nx4 1/1 Running 0 23d kube-scheduler-k8s-master 1/1 Running 0 23d node-exporter-g68xs 1/1 Running 0 7m57s node-exporter-rk2rg 1/1 Running 0 7m57s prometheus-68546b8d9-xk7tx 1/1 Running 0 116s # 部署Grafana [root@k8s-master pgmonitor]# cd grafana/ [root@k8s-master grafana]# vim grafana-deploy.yaml apiVersion: apps/v1 kind: Deployment metadata: name: grafana-core namespace: kube-system labels: app: grafana component: core spec: replicas: 1 selector: matchLabels: app: grafana component: core ... [root@k8s-master grafana]# kubectl create -f grafana-deploy.yaml [root@k8s-master grafana]# kubectl create -f grafana-svc.yaml [root@k8s-master grafana]# kubectl create -f grafana-ing.yaml # 检查 [root@k8s-master grafana]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-59d64cd4d4-dqx8m 1/1 Running 0 23d coredns-59d64cd4d4-z8pdq 1/1 Running 0 23d etcd-k8s-master 1/1 Running 0 23d grafana-core-85587c9c49-khvnk 1/1 Running 0 83s kube-apiserver-k8s-master 1/1 Running 0 23d kube-controller-manager-k8s-master 1/1 Running 0 23d kube-flannel-ds-h7v2g 1/1 Running 0 23d kube-flannel-ds-xmzfh 1/1 Running 0 23d kube-flannel-ds-z9nbj 1/1 Running 0 23d kube-proxy-6c9cd 1/1 Running 0 23d kube-proxy-cnvfg 1/1 Running 0 23d kube-proxy-p4nx4 1/1 Running 0 23d kube-scheduler-k8s-master 1/1 Running 0 23d node-exporter-g68xs 1/1 Running 0 17m node-exporter-rk2rg 1/1 Running 0 17m prometheus-68546b8d9-xk7tx 1/1 Running 0 11m # 查看打开的端口号 [root@k8s-master grafana]# kubectl get svc -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE grafana NodePort 10.111.191.90 <none> 3000:31708/TCP 4m32s kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 23d node-exporter NodePort 10.97.61.70 <none> 9100:31672/TCP 20m prometheus NodePort 10.111.182.252 <none> 9090:30003/TCP 14m [root@k8s-master grafana]# kubectl get svc -n kube-system -o wide
- 打开Grafana,配置数据源,导入显示模板,默认用户名密码都是admin。最后配置prometheus数据源
7. 搭建高可用集群
步骤文档地址:验证码0x23
8. 集群项目部署实操
k8s集群部署java项目:以下将以java项目(验证码c78o)为例实现这一流程
-
准备Java项目
-
通过maven进行打包
[root@15-package demo]# ls demojenkins [root@15-package demo]# cd demojenkins [root@15-package demojenkins]# mvn clean package [root@15-package demojenkins]# ls demojenkins.iml Dockerfile HELP.md mvnw mvnw.cmd pom.xml src target [root@15-package demojenkins]# cd target/ [root@15-package target]# ls classes demojenkins.jar demojenkins.jar.original generated-sources generated-test-sources maven-archiver maven-status surefire-reports test-classes
-
制作镜像
[root@15-package demojenkins]# ls demojenkins.iml Dockerfile HELP.md mvnw mvnw.cmd pom.xml src target [root@15-package demojenkins]# docker build -t java-demo-01:latest . [root@15-package demojenkins]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE java-demo-01 latest 3563dd6e175a 3 minutes ago 122MB openjdk 8-jdk-alpine a3562aa0b991 2 years ago 105MB # 简单测试镜像 [root@62-cent demojenkins]# docker run -d -p 8111:8111 java-demo-01:latest -t
-
上传镜像到镜像服务器中(以阿里云为例)
# 登录阿里云仓库 [root@62-cent demojenkins]# docker login --username=xxxx registry.cn-hangzhou.aliyuncs.com # 为镜像添加版本号 [root@62-cent demojenkins]# docker tag 3563dd6e175a registry.cn-hangzhou.aliyuncs.com/my_demo_space/java-project-01:1.0.0 # 实现推送 [root@62-cent demojenkins]# docker push registry.cn-hangzhou.aliyuncs.com/my_demo_space/java-project-01:1.0.0
-
部署镜像暴露应用
# 导出yaml [root@k8s-master ~]# kubectl create deployment javademo1 --image=registry.cn-hangzhou.aliyuncs.com/my_demo_space/java-project-01:1.0.0 --dry-run -o yaml > javademo1.yaml # 创建yaml [root@k8s-master ~]# kubectl apply -f javademo1.yaml deployment.apps/javademo1 created # 查看创建情况 [root@k8s-master ~]# kubectl get pods # 扩容 [root@k8s-master ~]# kubectl scale deployment javademo1 --replicas=3 # 暴露端口 [root@k8s-master ~]# kubectl expose deployment javademo1 --port=8111 --target-port=8111 --type=NodePort #通过NodeIp:port访问即可
更多推荐
所有评论(0)