一.k8s简介

kubernetes的本质是一组服务器集群，它可以在集群的每个节点上运行特定的程序，来对节点中的容器进行管理。目的是实现资源管理的自动化，主要提供了如下的主要功能：

• 自我修复：一旦某一个容器崩溃，能够在1秒中左右迅速启动新的容器

• 弹性伸缩：可以根据需要，自动对集群中正在运行的容器数量进行调整

• 服务发现：服务可以通过自动发现的形式找到它所依赖的服务

• 负载均衡：如果一个服务起动了多个容器，能够自动实现请求的负载均衡

• 版本回退：如果发现新发布的程序版本有问题，可以立即回退到原来的版本

• 存储编排：可以根据容器自身的需求自动创建存储卷

二.k8s设计框架

1 master：集群的控制平面，负责集群的决策

• ApiServer : 资源操作的唯一入口，接收用户输入的命令，提供认证、授权、API注册和发现等机制

• Scheduler : 负责集群资源调度，按照预定的调度策略将Pod调度到相应的node节点上

• ControllerManager : 负责维护集群的状态，比如程序部署安排、故障检测、自动扩展、滚动更新等

• Etcd ：负责存储集群中各种资源对象的信息

2 node：集群的数据平面，负责为容器提供运行环境

• kubelet：负责维护容器的生命周期，同时也负责Volume（CVI）和网络（CNI）的管理

• Container runtime：负责镜像管理以及Pod和容器的真正运行（CRI）

• kube-proxy：负责为Service提供cluster内部的服务发现和负载均衡

三.K8S集群环境搭建

主机名	ip	角色
reg.ddp.org	172.25.250.250	harbor仓库
k8s-master	172.25.250.100	master，k8s集群控制节点
k8s-node1	172.25.250.10	worker，k8s集群工作节点
k8s-node2	172.25.250.20	worker，k8s集群工作节点

构建harbor镜像仓库

1.部署docker本地仓库

[root@harbor ~]# cat > /etc/yum.repos.d/docker.repo <<EOF
[docker]
name = docker
baseurl = https://mirrors.aliyun.com/docker-ce/linux/rhel/9.6/x86_64/stable/
gpgcheck = 0
EOF

[root@harbor ~]# dnf install httpd createrepo-y
[root@harbor ~]# mkdir  /var/www/html/docker/ -p
[root@harbor ~]# vim /etc/httpd/conf/httpd.conf
Listen 4444
[root@harbor ~]# systemctl enable --now httpd

[root@harbor ~]# dnf  install docker-ce -downloadonly  --destdir /mnt/ -y
[root@harbor ~]# mv /mnt/*.rpm /var/www/html/docker/
[root@harbor ~]# createrepo -v /var/www/html/docker/

[root@harbor ~]# cat > /etc/yum.repos.d/docker.repo <<EOF
[docker]
name = docker
baseurl = http://172.25.250.250:4444/docker
gpgcheck = 0
EOF

[root@harbor ~]# dnf install docker-ce -y

2.安装docker在harbor仓库节点

[root@harbor ~]# echo br_netfilter > /etc/modules-load.d/docker_mod.conf
[root@harbor ~]# modprobe -a br_netfilter
[root@harbor ~]# vim /etc/sysctl.d/docker.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1

[root@harbor ~]# systemctl restart  systemd-modules-load.service
[root@harbor ~]# sysctl  --system

[root@harbor ~]# vim /lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --iptables=true
[root@harbor ~]# systemctl daemon-reload
[root@harbor ~]# systemctl enable --now docker

3.生成key

[root@harbor ~]# mkdir /data/certs -p
[root@harbor ~]# mkdir /data/certs -p
[root@harbor ~]# openssl req -newkey  rsa:4096 \
-nodes -sha256 -keyout /data/certs/ddp.org.key \
-addext "subjectAltName = DNS:reg.ddp.org" \
-x509 -days 365 -out /data/certs/ddp.org.crt

You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:Shannxi
Locality Name (eg, city) [Default City]:Xi'an
Organization Name (eg, company) [Default Company Ltd]:kubernetes
Organizational Unit Name (eg, section) []:harbor
Common Name (eg, your name or your server's hostname) []:reg.ddp.org
Email Address []:admin@ddp.org

4.编辑harbor配置文件

[root@harbor ~]# tar zxf  harbor-offline-installer-v2.5.4.tgz -C /opt/
[root@harbor ~]# cd /opt/harbor/
[root@harbor harbor]# ls
common.sh  harbor.v2.5.4.tar.gz  harbor.yml.tmpl  install.sh  LICENSE  prepare
[root@harbor harbor]# cp harbor.yml.tmpl harbor.yml
[root@harbor harbor]# vim harbor.yml

[root@harbor harbor]# ./install.sh --with-chartmuseum


#编写启动脚本
[root@harbor ~]# vim /lib/systemd/system/harbor.service


[root@harbor harbor]# docker compose  down
[root@harbor harbor]# docker compose  down
 
[root@harbor ~]# systemctl enable --now harbor
Created symlink /etc/systemd/system/multi-user.target.wants/harbor.service → /usr/lib/systemd/system/harbor.service.

[root@harbor harbor]# docker compose  ps

构建部署kubernetes所需主机

1.所有主机配置

#关闭主机上所有swap分区
systemctl mask swap.target
sed '/swap/s/^/#/g' -i /etc/fstab

mkdir  /etc/docker/certs.d/reg.ddp.org/ -p
#在harbor主机中分发证书到所有主机
[root@harbor ~]# for i in 100 10 20
> do
> scp /data/certs/ddp.org.crt root@172.25.250.$i:/etc/docker/certs.d/reg.ddp.org/ca.crt
> done

systemctl enable  docker
systemctl restart docker

#所有主机配置docker加速器
cat >/etc/docker/daemon.json <<EOF
{
  "registry-mirrors":["https://reg.timinglee.org"]
}
EOF

docker info

#所有主机去下载k8s插件
vim /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name = kubernetes
baseurl = https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.35/rpm/
gpgcheck = 0


#检测
dnf list kubelet

#检测环境swap分区是否存在
swapon  -s

2.kubernetes的部署

二进制包方式部署cri-dockerd

#下载二进制包
[root@docker-node1 ~]# wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.4.4/cri-dockerd-0.4.4.amd64.tgz

#解压二进制压缩包
[root@docker-node1 ~]# tar zxf cri-dockerd-0.4.4.amd64.tgz

[root@docker-node1 ~]# cd cri-dockerd/
[root@docker-node1 cri-dockerd]# ls
cri-dockerd  cri-docker.service  cri-docker.socket



#安装cri-docker命令到系统
[root@docker-node1 cri-dockerd]# install -o root -g root -m 0755 cri-dockerd  /usr/local/bin/cri-dockerd

[root@docker-node1 cri-dockerd]# ls -l /usr/bin/cri-dockerd
-rwxr-xr-x 1 root root 51638434 Aug 16 19:11 /usr/bin/cri-dockerd

#生成启动文件
[root@docker-node1 cri-dockerd]# cp cri-docker.s* /lib/systemd/system
cp: overwrite '/lib/systemd/system/cri-docker.service'? y
[root@docker-node1 cri-dockerd]# chmod  +x /lib/systemd/system/cri-docker.s*


[root@docker-node1 cri-dockerd]# vim /lib/systemd/system/cri-docker.service

[Service]
Type=notify
ExecStart=/usr/local/bin/cri-dockerd --network-plugin=cni --pod-infra-container-image=reg.ddp.org/k8s/pause:3.10.1 --container-runtime-endpoint fd://
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

[root@docker-node1 cri-dockerd]# systemctl daemon-reload

#启动服务
[root@docker-node1 cri-dockerd]# systemctl enable --now cri-docker.service
Created symlink /etc/systemd/system/multi-user.target.wants/cri-docker.service → /usr/lib/systemd/system/cri-docker.service.
[root@docker-node1 cri-dockerd]# systemctl status cri-docker.service

安装构建kubernetes 集群所需软件

#Master节点
dnf install kubelet kubeadm kubectl -y
systemctl enable --now kubelet.service

#master节点中 kubectl 和kubeadm 补齐
[root@master ~]# echo "source <(kubectl completion bash)" >> ~/.bashrc
[root@master ~]# echo "source <(kubeadm completion bash)" >> ~/.bashrc
[root@master ~]# source  ~/.bashrc


#node节点
dnf install kubelet kubeadm  -y
systemctl enable --now kubelet.service

下载kubernetes集群所需镜像

[root@master ~]# kubeadm config images pull \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.35.7 \
--cri-socket=unix:///var/run/cri-dockerd.sock

#上传镜像到本地harbor
[root@master ~]# docker  login  reg.ddp.org -u admin
Password:

WARNING! Your credentials are stored unencrypted in '/root/.docker/config.json'.
Configure a credential helper to remove this warning. See
https://docs.docker.com/go/credential-store/

Login Succeeded


[root@master ~]# docker images  --format  "{{.Repository}}:{{.Tag}}" | awk -F "/" '/google/{system("docker tag "$0" reg.ddp.org/k8s/"$3)}'

[root@master ~]# docker images  --format  "{{.Repository}}:{{.Tag}}" | awk -F "/" '/ddp/{system("docker push "$0)}'

在master中完成集群初始化

[root@master ~]# kubeadm init --pod-network-cidr=10.244.0.0/16 \
--image-repository reg.ddp.org/k8s \
--kubernetes-version v1.35.7 \
--cri-socket=unix:///var/run/cri-dockerd.sock

....

kubeadm join 172.25.250.100:6443 --token es2x1q.778q7gumtk3x854q --discovery-token-ca-cert-hash sha256:02c776da984d082ffb3e220d570c668a333fa9240a01ea39c2e77e2dce81c53e

添加kubernets环境变量到本机

[root@master ~]# echo "export KUBECONFIG=/etc/kubernetes/admin.conf" > ~/.bash_profile
[root@master ~]# source  ~/.bash_profile
[root@master ~]# kubectl get nodes
NAME     STATUS     ROLES           AGE    VERSION
master   NotReady   control-plane   102s   v1.35.7

[root@k8s-master ~]# tar zxf kube-flannel-v0.28.9.tar.gz
[root@k8s-master ~]# ls
flannel  kube-flannel-v0.28.9.tar.gz
[root@k8s-master ~]# cd flannel/

[root@k8s-master flannel]# docker load  -i kube-flannel-v0.28.9.tar
Loaded image: ghcr.io/flannel-io/flannel-cni-plugin:v1.9.1-flannel3
Loaded image: ghcr.io/flannel-io/flannel:v0.28.9


[root@master ~]# docker tag ghcr.io/flannel-io/flannel-cni-plugin:v1.9.0-flannel1 reg.ddp.org/flannel-io/flannel-cni-plugin:v1.9.0-flannel1
[root@master ~]# docker push  reg.ddp.org/flannel-io/flannel-cni-plugin:v1.9.0-flannel1
The push refers to repository [reg.ddp.org/flannel-io/flannel-cni-plugin]
2c8aa52d4746: Pushed
5aa68bbbc67e: Pushed
v1.9.0-flannel1: digest: sha256:b3d30c221113b30fea3e8a7fccb145e929b097d0319b9eeb6b5a591b10b5c671 size: 739

[root@master ~]# docker tag ghcr.io/flannel-io/flannel:v0.28.1 reg.ddp.org/flannel-io/flannel:v0.28.1
[root@master ~]# docker push reg.ddp.org/flannel-io/flannel:v0.28.1
The push refers to repository [reg.ddp.org/flannel-io/flannel]
5668da16a30b: Pushed
5f70bf18a086: Pushed
00012e17b6cc: Pushed
9738bb9596cf: Pushed
b6233bc105d7: Pushed
03767449b95b: Pushed
9a7d8cf9ff51: Pushed
13a60c10faeb: Pushed
70dc5e033175: Pushed
256f393e029f: Pushed
v0.28.1: digest: sha256:e671adbc267460164555159210066d3304a43e3b5dd85cc0b5b6ad62e83aab52 size: 2414


[root@master ~]# vim kube-flannel.yml
        image: flannel-io/flannel:v0.28.1
        image: flannel-io/flannel-cni-plugin:v1.9.0-flannel1
        image: flannel-io/flannel:v0.28.1
        
[root@k8s-master flannel]# grep -n  image: kube-flannel.yml  #要改内容所在行的位置
148:        image: flannel-io/flannel:v0.28.9
175:        image: flannel-io/flannel-cni-plugin:v1.9.1-flannel3
186:        image: flannel-io/flannel:v0.28.9


[root@master ~]# kubectl apply -f kube-flannel.yml
namespace/kube-flannel created
serviceaccount/flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

#测试
[root@master ~]# kubectl get  nodes

NAME         STATUS   ROLES           AGE     VERSION
k8s-master   Ready    control-plane   2d23h   v1.35.7
k8s-node1    Ready    <none>          2d23h   v1.35.7
k8s-node2    Ready    <none>          2d23h   v1.35.7