在Ubuntu 20.04 LTS上使用kubeadm部署K8s集群

一、环境信息

• MacBook Pro (13-inch, 2020, Four Thunderbolt 3 ports)
  macOS Big Sur Version 11.5.2
  2.3 GHz Quad-Core Intel Core i7
  16 GB 内存
• VirtualBox 6.1.22 r144080
• Ubuntu 20.04.2 LTS server

本次实验虽然在Mac上进行，但理论上仍然适用于在Windows主机上进行，主要是使用虚拟机软件VirtualBox 搭建Ubuntu 20.04.2 LTS 服务器。

最后，如果仍然发现有网络问题，看下是否是VirtualBox虚拟网络和Pod 网络发生冲突了。我这里由于NAT冲突了才切换到Bridged Adapter

1.1 虚拟机规划

1.1.1 机器配置

使用3台虚拟机

机器名称	cpu	内存	磁盘	网卡
master	2	3000M	动态100G	2个
node01	1	3000M	动态100G	2个
node02	1	3000M	动态100G	2个

1.1.2 网络配置

每台虚拟机配置两个网卡：Host-Only Adapter 和 Bridged Adapter 。Bridged Adapter网卡用来访问外网，采用动态IP。

机器名称	Host-Only Adapter	Bridged Adapter
master	192.168.56.130	DHCP
node01	192.168.56.131	DHCP
node02	192.168.56.132	DHCP

配置网络：
master节点配置，其他节点类似

sudo vim /etc/netplan/00-installer-config.yaml

# This is the network config written by 'subiquity'
network:
  ethernets:
    enp0s3:
      addresses:
        - 192.168.56.130/24
    enp0s8:
      dhcp4: true
  version: 2

应用配置

sudo netplan apply

应用网络配置后，ssh连接断开

更多VirtualBox网络模式介绍

1.2 系统配置

禁用swap

sudo swapoff -a
free -h
              total        used        free      shared  buff/cache   available
Mem:          2.4Gi       153Mi       1.8Gi       1.0Mi       416Mi       2.1Gi
Swap:            0B          0B          0B

持久化配置，注释掉swap那行即可
sudo vim /etc/fstab

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/ubuntu-vg/ubuntu-lv during curtin installation
/dev/disk/by-id/dm-uuid-LVM-L3Gvl91St70eyL2oJuu2BRgb2xWwZbjp67q0YZ6Vhef7M1EeUSWxeRocIegvMdz5 / ext4 defaults 0 0
# /boot was on /dev/sda2 during curtin installation
/dev/disk/by-uuid/418da6e3-3c15-49d3-bbc2-4355999de12e /boot ext4 defaults 0 0
# k8s 关闭swap
#/swap.img	none	swap	sw	0	0

修改时区

系统安装完成后默认时区是UTC，于当前时间差8个小时
要使用北京时间，需要将系统时区变更为CST

$ date
Tue Feb 22 06:57:53 UTC 2022
$ sudo timedatectl set-timezone Asia/Shanghai
$ date
Tue Feb 22 14:58:11 CST 2022

调整完时区后，如果想要系统日志的时间戳也立即生效，需要重启 rsyslog

$ sudo systemctl restart rsyslog

调整内核参数

如果系统没有加载br_netfilter模块，需要先加载该模块。
可通过lsmod | grep br_netfilter 来确认是否已经加载br_netfilter模块。
默认是没有加载的。可以先安装bridge-utils，后通过modprobe加载。

sudo apt-get install -y bridge-utils
sudo modprobe br_netfilter
lsmod | grep br_netfilter
br_netfilter           28672  0
bridge                176128  1 br_netfilter

此时lsmod可以看到br_netfilter

在Ubuntu 20.04 Server上，这个值就是1。如果你的系统上不一致，使用下面的命令来修改：

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system

二、正式安装

2.1 Docker

安装Docker

sudo apt install docker.io

启动Docker

sudo systemctl start docker

开机启动Docker

sudo systemctl enable docker

这种方式安装的Docker还是比较新的

$ docker --version
Docker version 20.10.7, build 20.10.7-0ubuntu5~20.04.2

2.2 kubeadm kubectl kubectl

由于谷歌源在国内是无法正常访问的，这里换成使用阿里源

# 配置阿里源
sudo apt-get install -y ca-certificates curl software-properties-common apt-transport-https
curl -s https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add -
sudo tee /etc/apt/sources.list.d/kubernetes.list <<EOF
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF

# 刷新软件列表后安装kubeadm kubectl kubectl
sudo apt-get update && sudo apt-get install -y kubelet kubeadm kubectl
# 锁定版本
sudo apt-mark hold kubelet kubeadm kubectl

三、构建集群

3.1 部署control-plane节点

control-plane节点机器上运行etcd 、 API Server

sudo kubeadm init --config kubeadm-config.yaml

kubeadm-config.yaml:

apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.56.130
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  imagePullPolicy: IfNotPresent
  name: master
  taints: null
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: 1.23.4
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
scheduler: {}
---
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: cgroupfs

• 指定仓库imageRepository，这里使用国内镜像仓库imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
• 指定IP地址advertiseAddress，advertiseAddress: 192.168.56.130

可通过kubeadm config print init-defaults打印默认配置

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.56.130:6443 --token abcdef.0123456789abcdef \
	--discovery-token-ca-cert-hash sha256:d4fef52d135fdcca5cfc75b5a07959cc923d17b189f87ba46e6aea61e47ca966

kubeadm init 完成后，提示需要做三件事情：
a.配置环境变量
b.安装网络插件
c.将节点加入集群

a.配置环境变量

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

如果是root用户，可以执行下面命令：

export KUBECONFIG=/etc/kubernetes/admin.conf

b.安装网络插件

K8S 的网络插件有很多，常见的有 flannel、calico等，更多查看Installing Addons
安装 Calico只需执行下面一条命令即可

kubectl apply -f https://docs.projectcalico.org/v3.21/manifests/calico.yaml

c.将节点加入集群

不是在control-plane上执行这条，在worker节点上执行

sudo kubeadm join 192.168.56.130:6443 --token abcdef.0123456789abcdef     --discovery-token-ca-cert-hash sha256:d4fef52d135fdcca5cfc75b5a07959cc923d17b189f87ba46e6aea61e47ca966

如果token过期,可以通过下面命令获取

kubeadm token create --print-join-command

构建control-plane节点的更多信息

• kubeadm reference guide
• Using kubeadm init with a configuration file
• 再次执行kubeadm init前，需要关闭集群
• 当然，这只是一个相对简单的集群，如果要考虑集群可用性，查看Options for Highly Available Topology
  

3.2 部署worker节点

安装Docker、kubelet 、kubeadm 、kubectl就够了,
然后，执行kubeadm join加入集群

3.3 验收集群

创建一个简单的Nginx pod

kubectl apply -f https://k8s.io/examples/pods/simple-pod.yaml

查看pod的IP：-o wide

michael@master:~$ kubectl get pods -o wide
NAME    READY   STATUS    RESTARTS   AGE   IP               NODE     NOMINATED NODE   READINESS GATES
nginx   1/1     Running   0          72s   172.16.196.129   node01   <none>           <none>

访问Nginx服务

curl 172.16.196.129

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

看到Nginx欢迎页代表集群搭建成功

四、参考文章

1. Getting started
2. Installing kubeadm
   
3. Creating a cluster with kubeadm
   
4. Kubernetes networking model
5. 图解 K8S（01）：基于 Ubuntu 20.04 部署 1.23版K8S 集群
6. Kubernetes 镜像
7. k8s——排错指南
8. Network Configuration
9. Introduction to Networking Modes