基于Kubernetes 1.22搭建基准测试系统(train-ticket)手册

本手册需要在搭建好kubernetes的基础上，再去运行基准测试系统。由于基准测试系统会占用大量的资源，所以要再真实的机器上搭建，或者是云主机上。

目录

基于Kubernetes 1.22搭建基准测试系统(train-ticket)手册

一、实验环境

二、环境检查

三、安装CRI-这里使用Docker

四、安装kubectl kubeadm kubelet

五、安装集群

六、部署基准测试系统

七、引用参考连接

---

一、实验环境

服务器系统	内核版本	节点IP	节点名称
Ubuntu 22.04	Linux 5.15.0-43-generic	192.168.105.132	master
Ubuntu 22.04	Linux 5.15.0-43-generic	192.168.105.133	worker-01
Ubuntu 22.04	Linux 5.15.0-43-generic	192.168.105.134	worker-02

二、环境检查

1. 检查3台机器是否互相联通，并且均正常上网。

2. 设置主机名

   vim /etc/hostname # 命令

    # 192.168.105.132 主机修改为 master
    # 192.168.105.133 主机修改为 worker-01
    # 192.168.105.134 主机修改为 worker-01

3. 主机名与IP进行映射

   Note-1: 以下步骤需要在三台机器上执行，执行完成后重启服务器。

   Note-2：修改完成后重启服务器。使用命令ping master 或 worker-01检查网络是否联通。

   vim /etc/hosts # 命令

    127.0.0.1 localhost
    # 此处也需要修改为主机名
    127.0.1.1 master 
    ​
    # 添加映射
    192.168.105.132 master
    192.168.105.133 worker-01
    192.168.105.134 worker-01
    ​
    # The following lines are desirable for IPv6 capable hosts
    ::1     ip6-localhost ip6-loopback
    fe00::0 ip6-localnet
    ff00::0 ip6-mcastprefix
    ff02::1 ip6-allnodes
    ff02::2 ip6-allrouters

4. 禁用swap

   swap分区：交换分区，从磁盘里分一块空间来充当内存使用，性能比真正的物理内存要差。 docker容器在内存里运行，k8s不允许容器到swap分区运行，要关闭swap分区，所以关闭swap分区是k8s为了追求高性能。

   (1)检查swap

    root@woker-01:~# free -h
                   total        used        free      shared  buff/cache   available
    Mem:           125Gi       1.3Gi       123Gi       5.0Mi       312Mi       123Gi
    Swap:          8.0Gi          0B       8.0Gi

   swap不为0,说明未禁用swap交换区

   (2)临时关闭swap

   sudo swapoff -a

   再一次检查swap

    root@woker-01:~# free -h
                   total        used        free      shared  buff/cache   available
    Mem:           125Gi       1.3Gi       123Gi       5.0Mi       313Mi       123Gi
    Swap:             0B          0B          0B

   (3)永久关闭swap

    vim /etc/fstab
    /dev/disk/by-id/dm-uuid-LVM-r61BVsAoX3nEuGdY0vbFEnil06zm7gxIdciulVrVkn5zLvW7iRSXr5b14zeM3nnj / ext4 defaults 0 1
    # /boot was on /dev/sda2 during curtin installation
    /dev/disk/by-uuid/eeeb35a5-0d56-4a0a-960b-468fdc6820d9 /boot ext4 defaults 0 1
    # /boot/efi was on /dev/sda1 during curtin installation
    /dev/disk/by-uuid/2D61-3C9B /boot/efi vfat defaults 0 1
    ​
    # 注释下面一行的内容
    #/swap.img      none    swap    sw      0       0

   1. 关闭SElinux

   需要安装相关工具包才能关闭

    sudo apt-get install -y policycoreutils

    root@worker-02:~# setenforce 0  #关闭selinux
    setenforce: SELinux is disabled

   1. 将文件系统设置为可读写

    sudo mount -n -o remount,rw /

三、安装CRI-这里使用Docker

Note-1: 以下步骤需要在三台机器上执行。

1. 卸载旧版本，旧版本的 Docker 被称为 docker、docker.io 或 docker-engine。如果安装了这些，请卸载它们

   sudo apt-get remove docker docker-engine docker.io containerd runc

2. 更新 apt 包索引并安装包以允许 apt 通过 HTTPS 使用存储库

    sudo apt-get update

    sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg \
    lsb-release

3. 添加Docker官方的GPG密钥

    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

4. 设置稳定存储库

    echo \
    "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
    $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

5. 安装 Docker 引擎更新apt包索引

   (1).更新apt包索

    sudo apt-get update

   (2)安装最新版本的Docker Engine和containerd

    sudo apt-get install docker-ce docker-ce-cli containerd.io

   Note-2:根据基准测试系统(train-ticket)的特殊，未能兼容最新版本。所以参考下图，安装以下版本

查询存储库中可用版本

 apt-cache madison docker-ce

使用第二列中的版本字符串安装特定版本，例如 5:20.10.18~3-0~ubuntu-jammy

 docker-ce | 5:20.10.18~3-0~ubuntu-jammy | https://download.docker.com/linux/ubuntu jammy/stable amd64 Packages

(3) 安装指定版本得docker （5:20.10.18~3-0~ubuntu-jammy）

 sudo apt-get install docker-ce=5:20.10.18~3-0~ubuntu-jammy docker-ce-cli=5:20.10.18~3-0~ubuntu-jammy containerd.io=1.5.11-1

(4) 测试部署docker 是否成功

 docker run hello-world

Unable to find image 'hello-world:latest' locally
 latest: Pulling from library/hello-world
 719385e32844: Pull complete 
 Digest: sha256:fc6cf906cbfa013e80938cdf0bb199fbdbb86d6e3e013783e5a766f50f5dbce0
 Status: Downloaded newer image for hello-world:latest
 ​
 Hello from Docker!
 This message shows that your installation appears to be working correctly.
 ​
 To generate this message, Docker took the following steps:
  1. The Docker client contacted the Docker daemon.
  2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
     (amd64)
  3. The Docker daemon created a new container from that image which runs the
     executable that produces the output you are currently reading.
  4. The Docker daemon streamed that output to the Docker client, which sent it
     to your terminal.
 ​
 To try something more ambitious, you can run an Ubuntu container with:
  $ docker run -it ubuntu bash
 ​
 Share images, automate workflows, and more with a free Docker ID:
  https://hub.docker.com/
 ​
 For more examples and ideas, visit:
  https://docs.docker.com/get-started/

6.使普通用户也可以执行docker命令

 sudo gpasswd -a $USER docker #将登陆用户加入到docker用户组中
 newgrp docker #更新用户组

7.检查套接字

 ls /var/run/docker.sock

四、安装kubectl kubeadm kubelet

1.安装curl和apt-transport-https

sudo apt-get update && sudo apt-get install -y apt-transport-https curl

2.下载GPG

 sudo wget https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg

3.添加GPG

 sudo apt-key add apt-key.gpg

4.写入镜像源文件

注：没有此目录，直接在下面创建这个文件，并增加以下内容。

 sudo cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
 deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
 EOF

5.更新索引

 sudo apt-get update

6.安装kubectl kubeadm kubelet（安装指定版本1.22.4）

 sudo apt-get install kubeadm=1.22.4-00 kubelet=1.22.4-00 kubectl=1.22.4-00 -y

五、安装集群

1.检查需要哪些镜像

 root@master:~# kubeadm config images list
 I0511 10:00:28.341332    9281 version.go:255] remote version is much newer: v1.27.1; falling back to: stable-1.22
 k8s.gcr.io/kube-apiserver:v1.22.17
 k8s.gcr.io/kube-controller-manager:v1.22.17
 k8s.gcr.io/kube-scheduler:v1.22.17
 k8s.gcr.io/kube-proxy:v1.22.17
 k8s.gcr.io/pause:3.5
 k8s.gcr.io/etcd:3.5.0-0
 k8s.gcr.io/coredns/coredns:v1.8.4

2.拉取镜像文件到本地

 vim push.sh  # 执行命令
 # 将以下内容写入文件
 #!/bin/sh
 sudo docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.22.4
 sudo docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.22.4
 sudo docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.22.4
 sudo docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.22.4
 sudo docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.5
 sudo docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.0-0
 sudo docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.8.4

 sudo chmod +x push.sh # 赋予执行权限
 ./push.sh   # 拉取镜像

3.查看镜像文件

4.修改镜像名

vim tag.sh  # 执行命令
 # 将以下内容写入文件
 #!/bin/sh
 sudo docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.22.17 k8s.gcr.io/kube-apiserver:v1.22.17
 sudo docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.22.17 k8s.gcr.io/kube-controller-manager:v1.22.17
 sudo docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.22.17 k8s.gcr.io/kube-scheduler:v1.22.17
 sudo docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.22.17 k8s.gcr.io/kube-proxy:v1.22.17
 sudo docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.5 k8s.gcr.io/pause:3.5
 sudo docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.0-0 k8s.gcr.io/etcd:3.5.0-0
 sudo docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.8.4 k8s.gcr.io/coredns/coredns:v1.8.4

 sudo chmod +x tag.sh # 赋予执行权限
 ./tag.sh

5.修改docker默认的Cgroup Driver

(1).Kubernetes 推荐使用 systemd 来代替 cgroupfs。systemd是Kubernetes自带的cgroup管理器, 负责为每个进程分配cgroups。docker的cgroup driver默认是cgroupfs,这样就同时运行有两个cgroup控制管理器。当资源有压力的情况时,有可能出现不稳定的情况

 vim /etc/docker/daemon.json

 # 将以下内容写入文件
 {"exec-opts": ["native.cgroupdriver=systemd"]}

(2).重启docker

 sudo systemctl restart docker

(3).重启kubelet

 sudo systemctl restart kubelet

6.配置集群

(1).master执行--初始化集群

 kubeadm init --apiserver-advertise-address=192.168.105.132 --image-repository=registry.aliyuncs.com/google_containers --kubernetes-version=v1.22.17 --service-cidr=10.96.0.0/12 --pod-network-cidr=10.244.0.0/16 --ignore-preflight-errors=all --v=6

初始化完成显示

(2).迁移配置

 mkdir -p $HOME/.kube
 sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
 sudo chown $(id -u):$(id -g) $HOME/.kube/config

(3)工作节点(worker)加入集群

 kubeadm join 172.16.2.146:6443 --token j4gpc3.l19pwajnb3xb60nu \
     --discovery-token-ca-cert-hash sha256:fc7649fda9a81f88e0089e8bf4df51cc119ee9d30038d4afc4bf058cba7cf362

查看节点状态

 root@master:~# kubectl get node
 NAME             STATUS     ROLES                  AGE    VERSION
 gzdx-master      NotReady   control-plane,master   100s   v1.22.4
 gzdx-worker-01   NotReady   <none>                 21s    v1.22.4
 gzdx-worker-02   NotReady   <none>                 11s    v1.22.4

Note: 由于是低版本的kubernetes，所以节点状态为NotReady是正常的。原因是需要配置网络插件Calico.

7.配置集群网络插件

(1).下载官方Calico

 wget https://docs.projectcalico.org/v3.20/manifests/calico.yaml

(2).修改calico.yaml

 vim calico.yaml +3888
 # 大约在3888行，取消注释，并改为前面kubeadm init中pod-network-cidr字段定义的网段
 ...
 - name: CALICO_IPV4POOL_CIDR
   value: "10.244.0.0/16"
 ...

(3)修改完后应用清单

 kubectl apply -f calico.yaml

查看节点状态

 kubectl get no

查看Pod状态

 kubectl get pods -A

8.部署Nginx,检测集群是否正常

(1).在master 节点上创建一个deployment

 kubectl create deployment nginx --image=nginx

(2).创建一个service

 kubectl create service nodeport nginx --tcp 80:80

(3).查看Nginx服务

 oot@master:~# kubectl get svc
 NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
 kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP        4m52s
 nginx        NodePort    10.111.135.150   <none>        80:32608/TCP   2m31s

(4)访问Nginx服务

root@master:~# curl localhost:32608
 <!DOCTYPE html>
 <html>
 <head>
 <title>Welcome to nginx!</title>
 <style>
 html { color-scheme: light dark; }
 body { width: 35em; margin: 0 auto;
 font-family: Tahoma, Verdana, Arial, sans-serif; }
 </style>
 </head>
 <body>
 <h1>Welcome to nginx!</h1>
 <p>If you see this page, the nginx web server is successfully installed and
 working. Further configuration is required.</p>
 ​
 <p>For online documentation and support please refer to
 <a href="http://nginx.org/">nginx.org</a>.<br/>
 Commercial support is available at
 <a href="http://nginx.com/">nginx.com</a>.</p>
 ​
 <p><em>Thank you for using nginx.</em></p>
 </body>
 </html>

至此集群部署成功。

六、部署基准测试系统

1.配置helm

wget https://get.helm.sh/helm-v3.10.3-linux-amd64.tar.gz
 tar -zxvf helm-v3.10.3-linux-amd64.tar.gz
 mv linux-amd64/helm  /usr/local/bin/

执行helm list确认没有错误输出

 root@master:~# helm list
 NAME    NAMESPACE   REVISION    UPDATED STATUS  CHART   APP VERSION

2.安装OpenEBS

(1)先决条件

1). 确认 master 节点是否有 Taint，如下看到 master 节点有 Taint

 root@master:~# kubectl describe node gzdx-master | grep Taint
 Taints:             node-role.kubernetes.io/master:NoSchedule

2).暂时去掉 master 节点的 Taint

 kubectl taint nodes gzdx-master node-role.kubernetes.io/master:NoSchedule-

(2).使用 Helm 安装openebs

 root@master:~# helm repo add openebs https://openebs.github.io/charts
 "openebs" has been added to your repositories
 root@master:~# helm repo update
 Hang tight while we grab the latest from your chart repositories...
 ...Successfully got an update from the "openebs" chart repository
 Update Complete. ⎈Happy Helming!⎈

 root@master:~# helm repo list
 NAME    URL                             
 openebs https://openebs.github.io/charts

root@master:~# helm install openebs --namespace openebs openebs/openebs --create-namespace
 # 以下是成功后输出的内容
 NAME: openebs
 LAST DEPLOYED: Thu May 11 11:13:47 2023
 NAMESPACE: openebs
 STATUS: deployed
 REVISION: 1
 TEST SUITE: None
 NOTES:
 Successfully installed OpenEBS.
 ​
 Check the status by running: kubectl get pods -n openebs
 ​
 The default values will install NDM and enable OpenEBS hostpath and device
 storage engines along with their default StorageClasses. Use `kubectl get sc`
 to see the list of installed OpenEBS StorageClasses.
 ​
 **Note**: If you are upgrading from the older helm chart that was using cStor
 and Jiva (non-csi) volumes, you will have to run the following command to include
 the older provisioners:
 ​
 helm upgrade openebs openebs/openebs \
     --namespace openebs \
     --set legacy.enabled=true \
     --reuse-values
 ​
 For other engines, you will need to perform a few more additional steps to
 enable the engine, configure the engines (e.g. creating pools) and create 
 StorageClasses. 
 ​
 For example, cStor can be enabled using commands like:
 ​
 helm upgrade openebs openebs/openebs \
     --namespace openebs \
     --set cstor.enabled=true \
     --reuse-values
 ​
 For more information, 
 - view the online documentation at https://openebs.io/docs or
 - connect with an active community on Kubernetes slack #openebs channel

查看创建的 StorageClass

将设置为默认的 StorageClass openebs-hostpath

 kubectl patch storageclass openebs-hostpath -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

查看openebs状态

3.拉取系统

 git clone --depth=1 https://github.com/FudanSELab/train-ticket.git 
 cd train-ticket/

4.部署系统

 make deploy

七、引用参考连接

1.FudanSELab/train-ticket: Train Ticket - A Benchmark Microservice System (github.com)

2.kubeadm快速搭建K8s集群 基于v1.22.2版本 - 努力吧阿团 - 博客园 (cnblogs.com)

3.安装OpenEBS (timd.cn)

4.ubuntu 20.04 基于kubeadm部署kubernetes 1.22.4集群及部署集群管理工具 - 西瓜君~ - 博客园 (cnblogs.com)

5.使用kubeadm部署Kubernetes 1.26 - 架构小白|青蛙小白|关注程序开发、互联网技术、云原生 (frognew.com)

6.(18条消息) Kubernetes系列之三：部署你的第一个应用程序到k8s集群_wucong60的博客-CSDN博客