Ubuntu18.04 LTS docker-19.03 k8s-1.19 单主多从部署

安装 docker# 卸载老版本sudo apt-get remove docker docker-engine docker.io containerd runc# 卸载已安装的版本sudo apt-get purge docker-ce docker-ce-cli containerd.iosudo rm -rf /var/lib/dockersudo rm -rf /var/lib/cont

yidichaxiang

475人浏览 · 2021-06-17 15:34:31

yidichaxiang · 2021-06-17 15:34:31 发布

修改apt源

sudo mv /etc/apt/sources.list /etc/apt/sources.list.bak

sudo tee /etc/apt/sources.list <<-'EOF'
deb http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
EOF

sudo apt get

关闭swap

swapoff -a
sudo vim /etc/fstab
# 注释掉 
#/swap.img      none    swap    sw      0       0

安装 docker

卸载老版本或已安装版本

# 卸载老版本
sudo apt-get remove docker docker-engine docker.io containerd runc
# 卸载已安装的版本
sudo systemctl disable docker
sudo apt-get -y purge docker-ce docker-ce-cli containerd.io
sudo rm -rf /var/lib/docker
sudo rm -rf /var/lib/containerd
sudo rm -rf /etc/docker/

安装基础依赖

# 重新安装
sudo apt-get update
# 基础依赖
sudo apt-get install -y apt-transport-https ca-certificates curl gnupg lsb-release
# docker GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

安装指定版本

# 列出docker版本
sudo apt-cache madison docker-ce|grep '19.03'

# 安装指定版本
sudo apt-get install -y docker-ce=5:19.03.15~3-0~ubuntu-bionic docker-ce-cli=5:19.03.15~3-0~ubuntu-bionic containerd.io
sudo apt autoremove -y

# 调整执行权限
sudo systemctl status docker
sudo usermod -aG docker $(whoami)
docker version

# 测试
docker run hello-world

配置docker

sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json <<-'EOF'
{
    "registry-mirrors": ["https://i4jgg87u.mirror.aliyuncs.com"],
    "exec-opts": ["native.cgroupdriver=systemd"],
    "log-driver": "json-file",
    "log-opts": {
        "max-size": "100m"
    },
    "storage-driver": "overlay2"
}
EOF

sudo mkdir -p /etc/systemd/system/docker.service.d
sudo systemctl daemon-reload
sudo systemctl restart docker

sudo systemctl enable docker

使用kubeadm安装k8s

安装kubeadm

检查每个节点上 MAC 地址和 product_uuid 的唯一性

ip link 或 ifconfig -a
sudo cat /sys/class/dmi/id/product_uuid

确保 iptables 工具不使用 nftables 后端

update-alternatives --set iptables /usr/sbin/iptables-legacy
update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
update-alternatives --set arptables /usr/sbin/arptables-legacy
update-alternatives --set ebtables /usr/sbin/ebtables-legacy

启用br_netfilter 模块

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system

检查所需端口

控制平面节点

协议	方向	端口范围	作用	使用者
TCP	入站	6443*	Kubernetes API 服务器	所有组件
TCP	入站	2379-2380	etcd server client API	kube-apiserver, etcd
TCP	入站	10250	Kubelet API	kubelet 自身、控制平面组件
TCP	入站	10251	kube-scheduler	kube-scheduler 自身
TCP	入站	10252	kube-controller-manager	kube-controller-manager 自身

工作节点

协议	方向	端口范围	作用	使用者
TCP	入站	10250	Kubelet API	kubelet 自身、控制平面组件
TCP	入站	30000-32767	NodePort 服务**	所有组件

** NodePort 服务的默认端口范围。

使用 * 标记的任意端口号都可以被覆盖，所以您需要保证所定制的端口是开放的。

虽然控制平面节点已经包含了 etcd 的端口，您也可以使用自定义的外部 etcd 集群，或是指定自定义端口。

您使用的 pod 网络插件 (见下) 也可能需要某些特定端口开启。由于各个 pod 网络插件都有所不同，请参阅他们各自文档中对端口的要求

安装 kubeadm、kubelet 和 kubectl

您需要在每台机器上安装以下的软件包：

kubeadm：用来初始化集群的指令。
kubelet：在集群中的每个节点上用来启动 pod 和容器等。
kubectl：用来与集群通信的命令行工具。

kubeadm 不能帮您安装或者管理 kubelet 或 kubectl，所以您需要确保它们与通过 kubeadm 安装的控制平面的版本相匹配。如果不这样做，则存在发生版本偏差的风险，可能会导致一些预料之外的错误和问题。然而，控制平面与 kubelet 间的相差一个次要版本不一致是支持的，但 kubelet 的版本不可以超过 API 服务器的版本。例如，1.7.0 版本的 kubelet 可以完全兼容 1.8.0 版本的 API 服务器，反之则不可以。

有关安装 kubectl 的信息，请参阅安装和设置 kubectl文档。

sudo apt-get update
sudo apt-get install -y apt-transport-https curl
curl -s https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add -
cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
sudo apt-get update

 sudo apt list kubeadm -a | grep 1.19

sudo apt install -y kubeadm=1.19.11-00 kubectl=1.19.11-00 kubelet=1.19.11-00
sudo apt-mark hold kubelet kubeadm kubectl

安装k8s集群

配置hosts

sudo sed -i '/#.*IPv6/i 10.10.10.133\tk8s-master-133' /etc/hosts
sudo sed -i '/#.*IPv6/i 10.10.10.134\tk8s-master-134' /etc/hosts
sudo sed -i '/#.*IPv6/i 10.10.10.135\tk8s-master-135' /etc/hosts
sudo sed -i '/#.*IPv6/i 10.10.10.151\tk8s-master-151\n\n' /etc/hosts

初始化

sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=10.10.10.133 --kubernetes-version=v1.19.11 --image-repository registry.aliyuncs.com/google_containers

# 初始化成功

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

sudo tee /etc/profile<<-'EOF'
export KUBECONFIG=/etc/kubernetes/admin.conf
EOF

sudo kubectl get node
# 显示下面内容，表示初始化成功
NAME             STATUS     ROLES    AGE     VERSION
k8s-master-133   NotReady   master   4m10s   v1.19.11

加入其它节点

# 在 worker节点执行加入加点命令
sudo kubeadm join 10.10.10.133:6443 --token fwbfdk.fgtbund0rv39rjtn \
    --discovery-token-ca-cert-hash sha256:35fbc40067520fb7a1becb2fdea93c1292907637265708cc9709b7b0b1df1130

若是忘记token

sudo kubeadm token list
#默认情况下，令牌会在 24 小时后过期。
sudo kubeadm token create
# 获取 discovery-token-ca-cert-hash
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

# kubeadm join --token <token> <control-plane-host>:<control-plane-port> --discovery-token-ca-cert-hash sha256:<hash>

sudo kubeadm token list
TOKEN                     TTL         EXPIRES                USAGES                   DESCRIPTION                                                EXTRA GROUPS
fwbfdk.fgtbund0rv39rjtn   23h         2021-06-18T03:30:41Z   authentication,signing   The default bootstrap token generated by 'kubeadm init'.   system:bootstrappers:kubeadm:default-node-token

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
35fbc40067520fb7a1becb2fdea93c1292907637265708cc9709b7b0b1df1130
# 新join命令
sudo kubeadm join --token fwbfdk.fgtbund0rv39rjtn 10.10.10.133:6443 --discovery-token-ca-cert-hash sha256:35fbc40067520fb7a1becb2fdea93c1292907637265708cc9709b7b0b1df1130

部署网络插件

cni 没有手动配置，安装flannel后，master和worker都会自动创建相应配置

flannel

注意：需要 kubeadm init 时设置 --pod-network-cidr=10.244.0.0/16

# sudo kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
# 有可能访问失败,则手动下载kube-flannel.yml到本地，手动pull镜像
docker pull quay.io/coreos/flannel:v0.14.0

sudo kubectl apply -f kube-flannel.yml

参考https://kubernetes.feisky.xyz/setup/cluster/kubeadm

集群搭建完成

sudo kubectl get nodes
NAME             STATUS   ROLES    AGE    VERSION
k8s-master-133   Ready    master   141m   v1.19.11
k8s-master-134   Ready    <none>   126m   v1.19.11
k8s-master-135   Ready    <none>   126m   v1.19.11
k8s-worker-151   Ready    <none>   134m   v1.19.11

部署测试：

# 查看当前默认空间的pod
kubectl get pods
error: error loading config file "/etc/kubernetes/admin.conf": open /etc/kubernetes/admin.conf: permission denied
# 不想每次都是 sudo，这里看看/etc/kubernetes/admin.conf 文件权限
sudo ls -la /etc/kubernetes/admin.conf
-rw------- 1 root root 5568 Jun 17 03:30 /etc/kubernetes/admin.conf
#修改 /etc/kubernetes/admin.conf 文件权限 
sudo chmod 660 /etc/kubernetes/admin.conf
sudo ls -la /etc/kubernetes/admin.conf
-rw-rw---- 1 root root 5568 Jun 17 03:30 /etc/kubernetes/admin.conf
#把当前用户加到root组
sudo usermod -aG root $(whoami)

# 以后就不用再使用sudo了
kubectl get pod
NAME    READY   STATUS    RESTARTS   AGE
nginx   1/1     Running   0          43m

部署nginx

deployment方式

# 创建 pod
kubectl create deployment nginx --image=nginx:latest
deployment.apps/nginx created
# 查看 pod
kubectl get pod -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP           NODE             NOMINATED NODE   READINESS GATES
nginx-55649fd747-fdgrz   1/1     Running   0          26s   10.244.1.3   k8s-worker-151   <none>           <none>
# 暴露service
kubectl expose deployment nginx --port=80 --type=NodePort
service/nginx exposed

# 查看pod，svc
$ kubectl get pods,svc
I0617 07:08:30.725015   31566 request.go:645] Throttling request took 1.181185793s, request: GET:https://10.10.10.133:6443/apis/rbac.authorization.k8s.io/v1?timeout=32s
NAME                         READY   STATUS    RESTARTS   AGE
pod/nginx-55649fd747-fdgrz   1/1     Running   0          109s

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
service/kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP        3h37m
service/nginx        NodePort    10.108.163.16   <none>        80:32318/TCP   17s

# 查看pod，svc
kubectl get pods,svc -o wide
I0617 07:08:58.280594   31968 request.go:645] Throttling request took 1.178201528s, request: GET:https://10.10.10.133:6443/apis/batch/v1?timeout=32s
NAME                         READY   STATUS    RESTARTS   AGE     IP           NODE             NOMINATED NODE   READINESS GATES
pod/nginx-55649fd747-fdgrz   1/1     Running   0          2m17s   10.244.1.3   k8s-worker-151   <none>           <none>

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE     SELECTOR
service/kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP        3h38m   <none>
service/nginx        NodePort    10.108.163.16   <none>        80:32318/TCP   45s     app=nginx

成功访问:)

run方式

# 运行容器镜像
kubectl run nginx --image=nginx:latest
pod/nginx created
#查看状态
kubectl get pods,svc -o wide
I0617 07:23:44.599391   12225 request.go:645] Throttling request took 1.176132195s, request: GET:https://10.10.10.133:6443/apis/networking.k8s.io/v1beta1?timeout=32s
NAME        READY   STATUS    RESTARTS   AGE   IP           NODE             NOMINATED NODE   READINESS GATES
pod/nginx   1/1     Running   0          29s   10.244.3.4   k8s-master-135   <none>           <none>

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE     SELECTOR
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   3h53m   <none>
# 查看启动日志
kubectl logs pod/nginx
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2021/06/17 07:23:40 [notice] 1#1: using the "epoll" event method
2021/06/17 07:23:40 [notice] 1#1: nginx/1.21.0
2021/06/17 07:23:40 [notice] 1#1: built by gcc 8.3.0 (Debian 8.3.0-6)
2021/06/17 07:23:40 [notice] 1#1: OS: Linux 4.15.0-144-generic
2021/06/17 07:23:40 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2021/06/17 07:23:40 [notice] 1#1: start worker processes
2021/06/17 07:23:40 [notice] 1#1: start worker process 31
2021/06/17 07:23:40 [notice] 1#1: start worker process 32
2021/06/17 07:23:40 [notice] 1#1: start worker process 33
2021/06/17 07:23:40 [notice] 1#1: start worker process 34

# 暴露pod 服务
kubectl expose pod nginx --port=80 --type=NodePort
service/nginx exposed
# 查看状态
kubectl get pods,svc -o wide
I0617 07:27:55.163834   15956 request.go:645] Throttling request took 1.171045189s, request: GET:https://10.10.10.133:6443/apis/apiextensions.k8s.io/v1beta1?timeout=32s
NAME        READY   STATUS    RESTARTS   AGE     IP           NODE             NOMINATED NODE   READINESS GATES
pod/nginx   1/1     Running   0          4m40s   10.244.3.4   k8s-master-135   <none>           <none>

NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE     SELECTOR
service/kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP        3h57m   <none>
service/nginx        NodePort    10.106.231.160   <none>        80:30196/TCP   6s      run=nginx

成功访问:)

Kubernetes kubectl 命令表 http://docs.kubernetes.org.cn/683.html

Kubernetes kubectl 命令表https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands

K8S/Kubernetes

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐

【深度】阿里巴巴万级规模 K8s 集群全局高可用体系之美

作者 | 韩堂、柘远、沉醉来源 | 阿里巴巴云原生公众号前言台湾作家林清玄在接受记者采访的时候，如此评价自己 30 多年写作生涯：“第一个十年我才华横溢，‘贼光闪现’，令周边黯然失色；第二个十年，我终于‘宝光现形’，不再去抢风头，反而与身边的美丽相得益彰；进入第三个十年，繁华落尽见真醇，我进入了‘醇光初现’的阶段，真正体味到了境界之美”。长夜有穷，真水无香。领略过了 K8s“身在江

K8S/Kubernetes

如何基于 K8s 构建下一代 DevOps 平台？

作者 | 孙健波（天元）导读：当前云原生 DevOps 体系现状如何？面临哪些挑战？如何通过 OAM 解决云原生 DevOps 场景下的诸多问题？云原生开发应用模型 OAM(Open Application Model) 社区核心成员孙健波将为大家一一解答，并分享如何基于 OAM 和 Kubernetes 打造无限能力的下一代 DevOps 平台。什么是 DevOps？为什么基于 Kub