一、实验题目

题目二：用Kubernetes建立一个实验集群。支持pod的多个节点，在容器内部署了一个可访问的httpd示例服务。

二、实验环境

名称	版本
操作系统	Ubuntu 18.04 LTS
Docker	20.10.7, build f0df350
Kubernetes	v1.21.1

在VMware中设置三台主机，配置如下：

主机名	ipv4	节点
ubuntu-1	192.168.47.177	master
ubuntu-2	192.168.47.178	node-1
ubuntu-3	192.168.47.179	node-2

三、实验步骤

3.1 系统配置更改

1. 禁用swap

   swapoff -a
   

2. 关闭防火墙

   systemctl stop firewalld
   systemctl disable firewalld
   

3. 配置静态IP

   root@ubuntu-1:/home/zjy# vim /etc/netplan/00-installer-config.yaml 
   
   # This is the network config written by 'subiquity'
   network:
     ethernets:
       ens33:
         addresses: [192.168.47.177/24]
         dhcp4: false
         gateway4: 192.168.47.2
         nameservers:
                 addresses: [192.168.47.2]
         optional: true
     version: 2
   

4. 将/etc/hosts配置如下

   192.168.47.177 master
   192.168.47.178 node-1
   192.168.47.179 node-2
   

5. IP应⽤启动

   netplan apply
   

6. 查看配置结果

   

   

   

3.2 安装Docker

1. 先安装相关工具

   apt-get update && apt-get install -y apt-transport-https curl
   

2. 添加密钥

   curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
   

3. 使用官方安装脚本自动安装

   curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun
   

4. 查看docker版本

   root@ubuntu-1:/home/zjy# docker version
   Client: Docker Engine - Community
    Version:           20.10.7
    API version:       1.41
    Go version:        go1.13.15
    Git commit:        f0df350
    Built:             Wed Jun  2 11:56:40 2021
    OS/Arch:           linux/amd64
    Context:           default
    Experimental:      true
   
   Server: Docker Engine - Community
    Engine:
     Version:          20.10.7
     API version:      1.41 (minimum version 1.12)
     Go version:       go1.13.15
     Git commit:       b0f5bc3
     Built:            Wed Jun  2 11:54:48 2021
     OS/Arch:          linux/amd64
     Experimental:     false
    containerd:
     Version:          1.4.6
     GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
    runc:
     Version:          1.0.0-rc95
     GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
    docker-init:
     Version:          0.19.0
     GitCommit:        de40ad0
   

5. 启动docker service

   systemctl enable docker
   systemctl start docker
   systemctl status docker
   

   由于网络原因，我们在pull Image的时候，从Docker Hub上下载会很慢，使用阿里云加速器，修改文件：

   vim  /etc/docker/daemon.json
   
   {
       "registry-mirrors": ["https://alzgoonw.mirror.aliyuncs.com"],
       "live-restore": true
   }
   

   重启docker服务：

   systemctl daemon-reload
   systemctl restart docker
   

3.3 安装kubectl，kubelet，kubeadm

在Master和Node节点分别执行如下操作

1. 添加密钥

   curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
   

   由于服务器无法访问国外网站，因此先在本地下载好apt-key.gpg文件，再拷贝到虚拟机上通过apt-key add apt-key.gpg来加载。

2. 添加Kubernetes软件源

   cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
   deb http://apt.kubernetes.io/ kubernetes-xenial main
   EOF
   

   上面是官方的源，国内不通需要修改为如下：

   cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
   deb http://mirrors.ustc.edu.cn/kubernetes/apt kubernetes-xenial main
   EOF
   

3. 安装

   sudo apt-get update
   sudo apt-get install -y kubelet kubeadm kubectl
   sudo apt-mark hold kubelet kubeadm kubectl
   

   apt-get update 错误超时，需要修改apt-get的源，采用ustc源：

   vim /etc/apt/sources.list.d/kubernetes.list
   
   # deb http://apt.kubernetes.io/ kubernetes-xenial main
   deb http://mirrors.ustc.edu.cn/kubernetes/apt kubernetes-xenial main
   

   执行完成后，终端显示结果如下：

   

3.4 配置Master

1. 增加环境变量

   在/etc/profile下面增加如下环境变量：

   echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
   # 生效
   source ~/.bash_profile
   

   # 重启 kubelet
   systemctl daemon-reload
   systemctl restart kubelet
   

2. 初始化 kubeadm

   在master节点上执行：

   root@ubuntu-1:/home/zjy# kubeadm init --image-repository=registry.aliyuncs.com/google_containers --pod-network-cidr=10.244.0.0/16 --kubernetes-version=v1.21.1
   

   • –pod-network-cidr是指配置节点中的pod的可用IP地址，此为内部IP
   • –kubernetes-version 通过kubectl version 可以查看

   结果如下：

   

3. 安装网络插件

   在安装完Master节点后，查看节点信息会发现节点的状态为 NotReady。

   root@ubuntu-1:/home/zjy# kubectl get nodes
   
   NAME       STATUS     ROLES                  AGE   VERSION
   ubuntu-1   NotReady   control-plane,master   25m   v1.21.1
   

   原因是由于CNI插件没有配置，即还没有配置网络，可以配置多种网络，这里选用最常用的Fannel网络进行配置。

   root@ubuntu-1:/home/zjy# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
   
   Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
   podsecuritypolicy.policy/psp.flannel.unprivileged created
   clusterrole.rbac.authorization.k8s.io/flannel created
   clusterrolebinding.rbac.authorization.k8s.io/flannel created
   serviceaccount/flannel created
   configmap/kube-flannel-cfg created
   daemonset.apps/kube-flannel-ds created
   

3.5 配置Node

1. 在各个node节点执行如下命令（对应master配置返回的 kubeadm join命令），加入master集群

   root@ubuntu-2:/home/zjy# kubeadm join 192.168.47.177:6443 --token cnsmts.6w5gpiqxlqmhxnl0 --discovery-token-ca-cert-hash sha256:f90ca2cfc29707965cf35349e43945c32c2e4fe1db2cabd2183db9f9aef72f2d
   
   [preflight] Running pre-flight checks
           [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
   [preflight] Reading configuration from the cluster...
   [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
   [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
   [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
   [kubelet-start] Starting the kubelet
   [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
   
   This node has joined the cluster:
   * Certificate signing request was sent to apiserver and a response was received.
   * The Kubelet was informed of the new secure connection details.
   
   Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
   

2. 在master下查看nodes状态

   

四、部署服务

采用deployment方式部署 httpd 服务，这里采用 nginx。

4.1 命令模式

1. 命令如下：

   kubectl create deployment nginx --image=nginx
   

2. 查看部署结果

   

   该结果表明在当前集群上，已经成功自动地在 ubuntu-3（node-2） 节点上部署了一个 nginx 节点，并处于正常运行（running）的状态。

3. 互相通信

   根据给出的 Pod ipv4 地址，在 master 和 node 节点上对其进行 ping 通信，可见各个节点以及 Pod 已经可以互通，结果如下图：

   

   

   

4. 利用 curl 命令对其网页进行访问

   root@ubuntu-1:/home/zjy# curl 10.244.2.2
   

   

5. 创建 service

   部署的 Pod 内服务还只能在集群内部命名空间下访问，无法被外部用户访问和请求。因此，我们继续在控制节点上创建 service，将服务暴露给外部访问。

   root@ubuntu-1:/home/zjy# kubectl create service nodeport nginx --tcp 80:80
   service/nginx created
   root@ubuntu-1:/home/zjy# kubectl get svc
   

   

6. 外部访问

   在外部终端的浏览器中输入对应的地址和端口进行访问，可以看到此时外部用户也能访问到 nginx 的网页界面：

   

4.2 yaml模式

1. 创建 nginx-yaml.yaml 文件，内容如下：

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: nginx-yaml
   spec:
     replicas: 2
     selector:
       matchLabels:
         app: nginx
     template:
       metadata:
         labels:
           app: nginx
       spec:
         containers:
           - name: nginx
             image: nginx:latest
             ports:
               - containerPort: 80
   

2. 从上述 yaml 文件创建新的deployment：

   root@ubuntu-1:~# kubectl apply -f nginx-yaml.yaml
   deployment.apps/nginx-yaml created
   

3. 查看具体信息

   root@ubuntu-1:~#  kubectl describe deployment nginx-yaml
   Name:                   nginx-yaml
   Namespace:              default
   CreationTimestamp:      Thu, 17 Jun 2021 15:31:31 +0000
   Labels:                 <none>
   Annotations:            deployment.kubernetes.io/revision: 1
   Selector:               app=nginx
   Replicas:               2 desired | 2 updated | 2 total | 2 available | 0 unavailable
   StrategyType:           RollingUpdate
   MinReadySeconds:        0
   RollingUpdateStrategy:  25% max unavailable, 25% max surge
   Pod Template:
     Labels:  app=nginx
     Containers:
      nginx:
       Image:        nginx:latest
       Port:         80/TCP
       Host Port:    0/TCP
       Environment:  <none>
       Mounts:       <none>
     Volumes:        <none>
   Conditions:
     Type           Status  Reason
     ----           ------  ------
     Available      True    MinimumReplicasAvailable
     Progressing    True    NewReplicaSetAvailable
   OldReplicaSets:  <none>
   NewReplicaSet:   nginx-yaml-585449566 (2/2 replicas created)
   Events:
     Type    Reason             Age   From                   Message
     ----    ------             ----  ----                   -------
     Normal  ScalingReplicaSet  5m8s  deployment-controller  Scaled up replica set nginx-yaml-585449566 to 2
   

4. 查看集群中的 pod 情况

   root@ubuntu-1:~# kubectl get pods -o wide
   NAME                              READY   STATUS    RESTARTS   AGE     IP           NODE       NOMINATED NODE   READINESS GATES
   nginx-6799fc88d8-bkksv            1/1     Running   0          60m     10.244.2.2   ubuntu-3   <none>           <none>
   nginx-yaml-585449566-6z487   1/1     Running   0          3m44s   10.244.2.5   ubuntu-3   <none>           <none>
   nginx-yaml-585449566-g4fnn   1/1     Running   0          3m44s   10.244.2.6   ubuntu-3   <none>           <none>
   

5. 创建service

   root@ubuntu-1:~# kubectl create service nodeport nginx --tcp 80:80
   
   service/nginx created
   root@ubuntu-1:~# kubectl get svc
   NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
   kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP        15m
   nginx        NodePort    10.102.16.156   <none>        80:30831/TCP   5s
   

   

6. 外部访问