ubuntu离线安装k8s

如果报错：sysctl: cannot stat /proc/sys/net/netfilter/nf_conntrack_max: No such file or directory，可能是 conntrack没有加载，执行：lsmod | grep conntrack。docker-ce-linux安装包是阿里云官方提供的开源镜像免费下载服务，每天下载量过亿，阿里巴巴开源镜像站为包含docke

文章共6,525字 · 阅读需要大约22分钟

一键AI生成摘要，助你高效阅读

问答

雨欲语

1440人浏览 · 2024-02-04 10:28:14

雨欲语 · 2024-02-04 10:28:14 发布

六、整合kubectl与cri-dockerd

七、网络等插件安装

八、常见问题及解决方法

一、前期准备

①ubuntu系统

本地已安装ubuntu系统，lsb_release -a命令查看版本信息：

②安装包

二、安装前配置

①设置主机名hostname，管理节点设置主机名为master

# 需要设置其他主机名称时，可将 master 替换为正确的主机名node1、node2即可。
sudo hostnamectl set-hostname master

②编辑/etc/hosts 文件，添加域名解析

sudo vim /etc/hosts

cat <<EOF >>/etc/hosts
10.10.10.10 master
EOF

③关闭防火墙、selinux和swap

sudo systemctl stop firewalld
sudo systemctl disable firewalld
sudo setenforce 0
sudo sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config
sudo swapoff -a

④禁用selinux

sudo apt-get install selinux-utils

sudo apt --fix-broken install

sudo setenforce 0

⑤禁用swap分区

sudo swapoff -a

禁用后如需开启swap：

1、vim /etc/default/kubelet

#添加内容

KUBELET_EXTRA_ARGS="--fail-swap-on=false"

2、sudo vim /var/lib/kubelet/config.yaml

#请修改或追加以下内容

featureGates:

NodeSwap: true

memorySwap:

swapBehavior: UnlimitedSwap

3、sudo systemctl start kubelet

⑥关闭防火墙

查看当前的防火墙状态：sudo ufw status

关闭防火墙: sudo ufw disable

⑦设置服务启动参数

sudo vim /etc/sysctl.d/k8s.conf

参数内容如下

net.bridge.bridge-nf-call-iptables=1

net.bridge.bridge-nf-call-ip6tables=1

net.ipv4.ip_forward=1

vm.swappiness=0

vm.overcommit_memory=1

vm.panic_on_oom=0

fs.inotify.max_user_instances=8192

fs.inotify.max_user_watches=1048576

fs.file-max=52706963

fs.nr_open=52706963

net.ipv6.conf.all.disable_ipv6=1

net.netfilter.nf_conntrack_max=2310720

使配置生效

sudo sysctl -p /etc/sysctl.d/k8s.conf

如果执行报错见：八①

⑧安装ipvs内核模块

由于ubuntu系统默认已经加载ipvs内核模块，执行验证

lsmod | grep ip_vs

如果返回为空，表示没有加载，执行下面命令重新加载：

sudo modprobe ip_vs

安装 ipvsadm ipset

sudo dpkg -i ipset_7.5-1ubuntu0.20.04.1_amd64.deb

sudo dpkg -i ipvsadm_1:1.31-1_amd64.deb

sudo dpkg -i libipset13_7.5-1ubuntu0.20.04.1_amd64.deb

三、安装docker

①需要用到的离线包

containerd.io_1.6.22-1_amd64.deb

docker.io_24.0.5-0ubuntu1_20.04.1_amd64.deb

执行命令安装：

sudo dpkg -i containerd.io_1.6.22-1_amd64.deb

sudo dpkg -i docker.io_24.0.5-0ubuntu1_20.04.1_amd64.deb

sudo systemctl start containerd

sudo systemctl start docker

查看状态：

sudo systemctl status docker

②配置用户组

把需要使用docker命令的用户，添加到用户组中：

sudo groupadd docker

sudo usermod -aG docker $USER

$USER是环境变量，指当前用户

配置docker开机启动

sudo systemctl enable docker

如果配置开机启动报错，见：八②

③配置私有镜像仓库以及日志切分配置

修改daemon.json文件

sudo vim /etc/docker/daemon.json

{
"registry-mirrors": [

"https://registry.docker-cn.com"

],

"insecure-registry": [

"registry.docker-cn.com"

],

"log-driver":"json-file",

    "log-opts":{

      "max-size":"100m",

      "max-file":"3"

    }

}

四、安装cri-dockerd

cri-docker是一个支持CRI标准的shim（垫片），一边通过CRI跟kubelet交互，另一边跟docker api交互，从而间接的实现了kubernetes以docker作为容器运行。

需要用到的离线包：cri-dockerd_0.3.4.3-0.ubuntu-jammy_amd64.deb

执行命令：

sudo dpkg -i cri-dockerd_0.3.4.3-0.ubuntu-jammy_amd64.deb

安装完cri-docker后，对应的服务会自动启动，命令查看：

systemctl status cri-docker

五、部署k8s master节点

①需要用到的离线包

kubeadm_1.27.4-00_amd64.deb

kubelet_1.27.4-00_amd64.deb

kubectl_1.27.4-00_amd64.deb

执行命令：

sudo dpkg -i kubeadm_1.27.4-00_amd64.deb

sudo dpkg -i kubelet_1.27.4-00_amd64.deb

sudo dpkg -i kubectl_1.27.4-00_amd64.deb

如果报错，见：八③

导入k8s集群安装所需镜像

docker load -i k8s_images.tar

验证镜像是否存在：

docker image ls

六、整合kubectl与cri-dockerd

①修改配置文件

sudo vim /lib/systemd/system/cri-docker.service

内容如下：

ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --network-plugin=cni --cni-bin-dir=/opt/cni/bin --cni-cache-dir=/var/lib/cni/cache --cni-conf-dir=/etc/cni/net.d --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.9

修改后的cri-docker.service文件内容：

[Unit]

Description=CRI Interface for Docker Application Container Engine

Documentation=https://docs.mirantis.com

After=network-online.target firewalld.service docker.service

Wants=network-online.target

Requires=cri-docker.socket

[Service]

Type=notify

ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --network-plugin=cni --cni-bin-dir=/opt/cni/bin --cni-cache-dir=/var/lib/cni/cache --cni-conf-dir=/etc/cni/net.d --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.9

ExecReload=/bin/kill -s HUP $MAINPID

TimeoutSec=0

RestartSec=2

Restart=always

# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.

# Both the old, and new location are accepted by systemd 229 and up, so using the old location

# to make them work for either version of systemd.

StartLimitBurst=3

# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.

# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make

# this option work for either version of systemd.

StartLimitInterval=60s

# Having non-zero Limit*s causes performance problems due to accounting overhead

# in the kernel. We recommend using cgroups to do container-local accounting.

LimitNOFILE=infinity

LimitNPROC=infinity

LimitCORE=infinity

# Comment TasksMax if your systemd version does not support it.

# Only systemd 226 and above support this option.

TasksMax=infinity

Delegate=yes

KillMode=process

[Install]

WantedBy=multi-user.target

在/usr/lib/systemd/system/cri-docker.service文件中添加上如上配置；

--network-plugin：指定网络插件规范的类型，这里要使用CNI；

--cni-bin-dir：指定CNI插件二进制程序文件的搜索目录；

--cni-cache-dir：CNI插件使用的缓存目录；

--cni-conf-dir：CNI插件加载配置文件的目录；

--pod-infra-container-image:指定pause镜像这个一定要配置，不然systemctl status cri-docker会报错

②重启cri-dockerd服务

sudo systemctl daemon-reload && sudo systemctl restart cri-docker

七、网络等插件安装

①kubeadm初始化

sudo kubeadm init --kubernetes-version=v1.27.4 --apiserver-advertise-address=xxxxxx --apiserver-bind-port=6443 --image-repository=registry.aliyuncs.com/google_containers --service-cidr=10.96.0.0/12 --pod-network-cidr=10.244.0.0/16 --ignore-preflight-errors=Swap --cri-socket=unix:///run/cri-dockerd.sock --v=5

其中piserver-advertise-address是实际机器的ip地址

如果报错，见：八④

②配置kubectl工具

root用户：

sudo mkdir -p /root/.kube
sudo cp /etc/kubernetes/admin.conf /root/.kube/config

普通用户：

##创建自己的kube
mkdir -p $HOME/.kube
##复制root用户的kubectl配置到家目录下
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
##修改kubectl配置文件的权限
sudo chown $(id -u):$(id -g) $HOME/.kube/config

测试集群：

kubectl get nodes

kubectl get cs

kubectl get pods --all-namespaces

我这里是已经部署完毕并已经成功运行pod后的截图：

③安装calico

需要用到calico.yaml,详见：

https://docs.projectcalico.org/v3.20/manifests/calico.yamlhttps://docs.projectcalico.org/v3.20/manifests/calico.yaml执行命令：

kubectl apply -f calico.yml

部署完毕之后：coredns的两个pod变为runing状态

执行命令查看：

kubectl get pod -n kube-system

④设置允许master调度pod

先执行命令查看：

sudo kubectl describe node master | grep Taints

会显示：

Taints: node-role.kubernetes.io/control-plane:NoSchedule

执行命令：

kubectl taint node master node-role.kubernetes.io/control-plane:NoSchedule-

如果执行kubectl get nodes出现node没有ready，重启container和kubectl：

systemctl restart containerd

systemctl restart kubelet

如果报错见：八⑤

八、常见问题及解决方法

①sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No Such file or directory

如果出现这个错误说明没有先加载内核模块br_netfilter，bridge-nf 使 netfilter 可以对 Linux 网桥上的 IPv4/ARP/IPv6 包过滤。比如设置net.bridge.bridge-nf-call-iptables=1后，二层的网桥在转发包时也会被 iptables的 FORWARD 规则所过滤。

解决方案：

1.加载overlay和br_netfilter两个内核模块

sudo modprobe overlay && sudo modprobe br_netfilter

持久化加载上述两个模块，避免重启失效。

$ cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf

overlay

br_netfilter

EOF

通过运行 lsmod | grep br_netfilter 来验证 br_netfilter 模块是否已加载

通过运行 lsmod | grep overlay 来验证 overlay模块是否已加载

再次执行：sudo sysctl -p /etc/sysctl.d/k8s.conf

如果报错：sysctl: cannot stat /proc/sys/net/netfilter/nf_conntrack_max: No such file or directory，可能是 conntrack没有加载，执行：lsmod | grep conntrack

如果返回为空，表示没有加载，执行下面命令

重新加载：sudo modprobe ip_conntrack sudo sysctl -p /etc/sysctl.d/k8s.conf

②配置docker开机启动报错：Failed to enable unit: Unit file docker.service does not exist.

新建docker.service文件：sudo vim /lib/systemd/system/docker.service

[Unit]

Description=Docker Application Container Engine

Documentation=https://docs.docker.com

After=network-online.target docker.socket firewalld.service containerd.service time-set.target

Wants=network-online.target containerd.service

Requires=docker.socket

[Service]

Type=notify

# the default is not to use systemd for cgroups because the delegate issues still

# exists and systemd currently does not support the cgroup feature set required

# for containers run by docker

ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

ExecReload=/bin/kill -s HUP $MAINPID

TimeoutStartSec=0

RestartSec=2

Restart=always

# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.

# Both the old, and new location are accepted by systemd 229 and up, so using the old location

# to make them work for either version of systemd.

StartLimitBurst=3

# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.

# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make

# this option work for either version of systemd.

StartLimitInterval=60s

# Having non-zero Limit*s causes performance problems due to accounting overhead

# in the kernel. We recommend using cgroups to do container-local accounting.

LimitNOFILE=infinity

LimitNPROC=infinity

LimitCORE=infinity

# Comment TasksMax if your systemd version does not support it.

# Only systemd 226 and above support this option.

TasksMax=infinity

# set delegate yes so that systemd does not reset the cgroups of docker containers

Delegate=yes

# kill only the docker process, not all processes in the cgroup

KillMode=process

OOMScoreAdjust=-500

[Install]

WantedBy=multi-user.target

③安装kubeadm、kubelet、kubectl报错

如果报错：

Selecting previously unselected package kubeadm.

(Reading database ... 106973 files and directories currently installed.)

Preparing to unpack kubeadm_1.27.4-00_amd64.deb ...

Unpacking kubeadm (1.27.4-00) ...

Selecting previously unselected package kubelet.

Preparing to unpack kubelet_1.27.4-00_amd64.deb ...

Unpacking kubelet (1.27.4-00) ...

Selecting previously unselected package kubectl.

Preparing to unpack kubectl_1.27.4-00_amd64.deb ...

Unpacking kubectl (1.27.4-00) ...

dpkg: dependency problems prevent configuration of kubeadm:

kubeadm depends on kubernetes-cni (>= 1.1.1); however:

  Package kubernetes-cni is not installed.

kubeadm depends on cri-tools (>= 1.25.0); however:

  Package cri-tools is not installed.

dpkg: error processing package kubeadm (--install):

dependency problems - leaving unconfigured

dpkg: dependency problems prevent configuration of kubelet:

kubelet depends on kubernetes-cni (>= 1.1.1); however:

  Package kubernetes-cni is not installed.

kubelet depends on socat; however:

  Package socat is not installed.

kubelet depends on ebtables; however:

  Package ebtables is not installed.

kubelet depends on conntrack; however:

  Package conntrack is not installed.

dpkg: error processing package kubelet (--install):

dependency problems - leaving unconfigured

Setting up kubectl (1.27.4-00) ...

Errors were encountered while processing:

kubeadm

kubelet

需要安装依赖：kubernetes-cni cri-tools socat ebtables conntrack

sudo dpkg -i kubernetes-cni_1.2.0-00_s390x_86cdf4d82e3a59c3f6e12975b149a5e42afebff3fd342161abac520253237938.deb

sudo dpkg -i cri-tools_1.26.0-00_amd64.deb

sudo dpkg -i socat_1.7.3.3-2_amd64.deb

sudo dpkg -i ebtables_2.0.11-3build1_amd64.deb

sudo dpkg -i conntrack_1%3A1.4.5-2_amd64.deb

如果安装继续报错：

dpkg: error processing archive kubernetes-cni_1.2.0-00_arm64_5d61b8d04701612640667c1da13b616529ded1fed0b7405382d8d08eaa5b5af7.deb (--install):

package architecture (arm64) does not match system (amd64)

Errors were encountered while processing:

kubernetes-cni_1.2.0-00_arm64_5d61b8d04701612640667c1da13b616529ded1fed0b7405382d8d08eaa5b5af7.deb

halos@bgi:/mnt/test-halos/upgradeworkspace/deb$ sudo dpkg --add-architecture arm64

再次安装kubernetes-cni_1.2.0-00_arm64_5d61b8d04701612640667c1da13b616529ded1fed0b7405382d8d08eaa5b5af7.deb

sudo dpkg -i cri-tools_1.26.0-00_arm64_be3fa6bdc17ab229b45222887c442ae1a601b3b2bc3e011c9e7235767e7269c4.deb

④kubeadm初始化报错

如果报错：

[ERROR Port-10250]: Port 10250 is in use

要解除kubelet.service的mask状态，请使用以下命令：

sudo systemctl unmask kubelet.service

驱动一致还报错执行以下命令：

sudo kubeadm reset -f
sudo iptables -F && sudo iptables -t nat -F && sudo iptables -t mangle -F && sudo iptables -X

如果继续报错：

[init] Using Kubernetes version: v1.24.4

[preflight] Running pre-flight checks

error execution phase preflight: [preflight] Some fatal errors occurred:

[ERROR CRI]: container runtime is not running: output: E1107 11:17:24.937456 31126 remote_runtime.go:948] "Status from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"

time="2022-11-07T11:17:24+08:00" level=fatal msg="getting status of runtime: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"

, error: exit status 1

[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

To see the stack trace of this error execute with --v=5 or higher