K8s集群Containerd运行时离线标准化部署实操

项目	内容
适用版本	Kubernetes v1.28.15 + containerd 1.7.x + Rocky Linux 8.6 / RHEL 8.x
部署模式	单Master + 多Worker（可扩展高可用）
编写日期	2026-08-17
文档版本	v1.0

1. 操作环境与前置准备

1.1 硬件要求

角色	CPU	内存	硬盘	网卡
Worker	4核+	8GB+	200GB SSD	千兆×1
Harbor仓库（可选）	4核+	8GB+	500GB+	千兆×1

1.2 软件版本矩阵

组件	版本	说明
内核	4.18.0-372+	系统自带
Kubernetes	v1.28.15	稳定版
containerd	1.7.11+	CRI运行时
runc	1.1.10+	OCI运行时
CNI插件	1.3.0+	容器网络接口
Calico	v3.26.x	网络策略引擎
crictl	v1.28.0	CRI调试工具
Docker CLI（可选）	24.0.x	仅用于镜像打包导出

1.3 节点规划示例

主机名	IP地址	角色
k8s-worker01	192.168.10.21	Worker
k8s-worker02	192.168.10.22	Worker
harbor.local	192.168.10.30	私有镜像仓库

1.4 离线环境网络要求

• 所有节点之间网络互通（同VLAN或路由可达）
• Master节点6443端口对Worker开放
• 节点间10250（kubelet）、2379-2380（etcd）端口互通
• Calico BGP模式开放179端口；VXLAN模式开放4789/UDP

2. 对接原理：Containerd与K8s CRI架构

2.1 整体架构

┌─────────────────────────────────────────────────────┐
│                   Kubernetes Node                    │
│                                                      │
│  ┌──────────┐    CRI gRPC    ┌──────────────────┐   │
│  │  kubelet │◄──────────────►│   containerd     │   │
│  │          │  (unix socket) │   ┌────────────┐ │   │
│  └──────────┘                │   │  CRI Plugin│ │   │
│                              │   └────────────┘ │   │
│                              │   ┌────────────┐ │   │
│                              │   │  core API  │ │   │
│                              │   └────────────┘ │   │
│                              │   ┌────────────┐ │   │
│                              │   │  shim API  │ │   │
│                              │   └─────┬──────┘ │   │
│                              └─────────┼────────┘   │
│                                        │            │
│                              ┌─────────▼────────┐   │
│                              │  containerd-shim  │   │
│                              │  (runc v2)        │   │
│                              └─────────┬────────┘   │
│                                        │            │
│                              ┌─────────▼────────┐   │
│                              │      runc         │   │
│                              │  (OCI runtime)    │   │
│                              └─────────┬────────┘   │
│                                        │            │
│                              ┌─────────▼────────┐   │
│                              │   Linux Namespace│   │
│                              │   + Cgroups      │   │
│                              └──────────────────┘   │
└─────────────────────────────────────────────────────┘

2.2 核心对接点

CRI（Container Runtime Interface） 是Kubernetes定义的容器运行时gRPC接口规范，kubelet通过以下两个核心服务与containerd通信：

服务	职责
ImageService	镜像管理（拉取/推送/列出/删除镜像）

通信方式：Unix Domain Socket，默认路径 /run/containerd/containerd.sock

2.3 离线部署核心思路

公网环境（有网）                    离线环境（无网）
┌──────────────┐                  ┌──────────────┐
│ 1. 下载RPM包  │ ──── U盘/光盘 ──►│ 1. 本地YUM源  │
│ 2. 拉取镜像   │ ──── tar包 ─────►│ 2. ctr导入镜像│
│ 3. 下载二进制 │ ──── 压缩包 ────►│ 3. 解压部署   │
└──────────────┘                  └──────────────┘

离线部署三大介质：

1. RPM软件包 — 通过 yumdownloader 或 reposync 下载全量依赖
2. 容器镜像 — 通过 docker save / ctr images export 导出为tar包
3. 二进制文件 — 从GitHub官方Release下载预编译二进制

3. 离线介质包制作

以下操作在**有公网的跳板机**上执行，制作完成后拷贝至离线环境。

3.1 创建工作目录

mkdir -p /opt/k8s-offline/{rpms,images,binaries,cni,configs}
cd /opt/k8s-offline

3.2 下载RPM包（含完整依赖）

3.2.1 配置Kubernetes YUM源

cat > /etc/yum.repos.d/kubernetes.repo << 'EOF'
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.28/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.28/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

3.2.2 下载K8s组件及依赖

# 安装下载工具
yum install -y yum-utils

# 下载kubelet kubeadm kubectl及全部依赖
yumdownloader --resolve --destdir=/opt/k8s-offline/rpms \
  kubelet-1.28.15 \
  kubeadm-1.28.15 \
  kubectl-1.28.15 \
  cri-tools-1.28.0 \
  kubernetes-cni-1.3.0

# 下载containerd及依赖
yumdownloader --resolve --destdir=/opt/k8s-offline/rpms \
  containerd-1.7.11

# 下载系统依赖（离线环境通常缺失）
yumdownloader --resolve --destdir=/opt/k8s-offline/rpms \
  conntrack-tools socat ipvsadm ipset nfs-utils \
  bash-completion libnetfilter_cthelper \
  libnetfilter_cttimeout libnetfilter_queue

3.2.3 制作本地YUM仓库

yum install -y createrepo_c
createrepo_c /opt/k8s-offline/rpms/
cd /opt/k8s-offline && tar czf k8s-rpms-1.28.15.tar.gz rpms/

3.3 导出容器镜像

3.3.1 拉取K8s所需镜像

# 查看kubeadm所需镜像列表
kubeadm config images list --kubernetes-version=v1.28.15

# 拉取镜像
images=(
  registry.k8s.io/kube-apiserver:v1.28.15
  registry.k8s.io/kube-controller-manager:v1.28.15
  registry.k8s.io/kube-scheduler:v1.28.15
  registry.k8s.io/kube-proxy:v1.28.15
  registry.k8s.io/pause:3.9
  registry.k8s.io/etcd:3.5.9-0
  registry.k8s.io/coredns/coredns:v1.10.1
  docker.io/calico/cni:v3.26.1
  docker.io/calico/node:v3.26.1
  docker.io/calico/kube-controllers:v3.26.1
  docker.io/calico/typha:v3.26.1
)

for img in "${images[@]}"; do
  docker pull "$img"
done

3.3.2 导出镜像tar包

# K8s核心镜像
docker save \
  registry.k8s.io/kube-apiserver:v1.28.15 \
  registry.k8s.io/kube-controller-manager:v1.28.15 \
  registry.k8s.io/kube-scheduler:v1.28.15 \
  registry.k8s.io/kube-proxy:v1.28.15 \
  registry.k8s.io/pause:3.9 \
  registry.k8s.io/etcd:3.5.9-0 \
  registry.k8s.io/coredns/coredns:v1.10.1 \
  -o /opt/k8s-offline/images/k8s-core-images-v1.28.15.tar

# Calico镜像
docker save \
  calico/cni:v3.26.1 \
  calico/node:v3.26.1 \
  calico/kube-controllers:v3.26.1 \
  calico/typha:v3.26.1 \
  -o /opt/k8s-offline/images/calico-images-v3.26.1.tar

3.4 下载二进制工具

cd /opt/k8s-offline/binaries

# containerd官方二进制包（含runc）
wget https://github.com/containerd/containerd/releases/download/v1.7.11/containerd-1.7.11-linux-amd64.tar.gz

# runc独立二进制
wget https://github.com/opencontainers/runc/releases/download/v1.1.10/runc.amd64

# CNI插件
wget https://github.com/containernetworking/plugins/releases/download/v1.3.0/cni-plugins-linux-amd64-v1.3.0.tgz

# crictl
wget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.28.0/crictl-v1.28.0-linux-amd64.tar.gz

# etcdctl（可选）
wget https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz

3.5 介质包校验

cd /opt/k8s-offline

# 生成校验和
sha256sum rpms/*.rpm > rpms/SHA256SUMS
sha256sum images/*.tar > images/SHA256SUMS
sha256sum binaries/* > binaries/SHA256SUMS

# 最终打包
tar czf k8s-offline-full-v1.28.15.tar.gz rpms/ images/ binaries/ configs/

4. 系统初始化标准化配置

以下操作在**所有节点**（Master + Worker）上执行。

4.1 配置主机名与hosts

# Master节点
hostnamectl set-hostname k8s-master01

# Worker节点分别执行
hostnamectl set-hostname k8s-worker01
hostnamectl set-hostname k8s-worker02

# 所有节点配置hosts
cat >> /etc/hosts << 'EOF'
192.168.10.10  k8s-master01
192.168.10.21  k8s-worker01
192.168.10.22  k8s-worker02
192.168.10.30  harbor.local
EOF

4.2 关闭防火墙与SELinux

# 关闭firewalld
systemctl stop firewalld
systemctl disable firewalld

# 关闭SELinux
setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config
sed -i 's/^SELINUX=permissive$/SELINUX=disabled/' /etc/selinux/config

# 验证
getenforce  # 应输出 Disabled

4.3 关闭Swap

swapoff -a
sed -i '/swap/s/^\(.*\)$/#\1/g' /etc/fstab
free -h  # Swap行应全为0

4.4 内核参数优化

cat > /etc/sysctl.d/99-kubernetes-cri.conf << 'EOF'
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
net.ipv6.conf.all.disable_ipv6      = 1
net.ipv6.conf.default.disable_ipv6  = 1
fs.inotify.max_user_instances       = 8192
fs.inotify.max_user_watches         = 524288
fs.file-max                         = 52706963
fs.nr_open                          = 52706963
net.core.somaxconn                  = 32768
net.ipv4.tcp_max_syn_backlog        = 16384
vm.swappiness                       = 0
vm.overcommit_memory                = 1
EOF

# 加载内核模块
modprobe overlay
modprobe br_netfilter

cat > /etc/modules-load.d/k8s.conf << 'EOF'
overlay
br_netfilter
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
EOF

sysctl --system

4.5 配置时间同步

yum install -y chrony
systemctl enable --now chronyd

# 配置内网NTP服务器（如有）
sed -i 's/^pool .*/server 192.168.10.1 iburst/' /etc/chrony.conf
systemctl restart chronyd

chronyc sources -v
timedatectl

4.6 配置本地YUM源

# 上传介质包并解压
mkdir -p /opt/k8s-offline
tar xzf k8s-rpms-1.28.15.tar.gz -C /opt/k8s-offline/

# 备份原有repo
mkdir -p /etc/yum.repos.d/backup
mv /etc/yum.repos.d/*.repo /etc/yum.repos.d/backup/

# 配置本地源
cat > /etc/yum.repos.d/k8s-local.repo << 'EOF'
[k8s-local]
name=K8s Local Repo
baseurl=file:///opt/k8s-offline/rpms
enabled=1
gpgcheck=0
EOF

yum clean all && yum makecache

5. Containerd运行时离线部署

5.1 RPM包方式安装（推荐）

yum install -y containerd-1.7.11

# 验证
rpm -qa | grep containerd
containerd --version
runc --version

5.2 二进制方式安装

cd /opt/k8s-offline/binaries

# 解压containerd
tar Cxzf /usr/local containerd-1.7.11-linux-amd64.tar.gz

# 安装runc
install -m 755 runc.amd64 /usr/local/sbin/runc

# 安装CNI插件
mkdir -p /opt/cni/bin
tar Cxzf /opt/cni/bin cni-plugins-linux-amd64-v1.3.0.tgz

# 安装crictl
tar xzf crictl-v1.28.0-linux-amd64.tar.gz -C /usr/local/bin/

5.3 生成配置文件

mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml

5.4 关键配置项修改

编辑 /etc/containerd/config.toml，重点修改以下内容：

# 数据目录（建议单独挂载数据盘）
root = "/data/containerd"
state = "/run/containerd"

# 1. systemd cgroup驱动（K8s推荐，必须与kubelet一致）
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

# 2. pause镜像（离线环境使用私有仓库地址）
[plugins."io.containerd.grpc.v1.cri"]
  sandbox_image = "harbor.local/k8s/pause:3.9"

# 3. 私有镜像仓库（HTTP内网，跳过TLS验证）
[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/etc/containerd/certs.d"
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."harbor.local"]
      endpoint = ["http://harbor.local"]
  [plugins."io.containerd.grpc.v1.cri".registry.configs]
    [plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.local".tls]
      insecure_skip_verify = true

5.5 配置systemd服务

cat > /etc/systemd/system/containerd.service << 'EOF'
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target

[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/bin/containerd
Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=infinity
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now containerd
systemctl status containerd

5.6 配置crictl

cat > /etc/crictl.yaml << 'EOF'
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF

crictl version
crictl info

5.7 导入离线镜像

# 导入K8s核心镜像
ctr -n k8s.io images import /opt/k8s-offline/images/k8s-core-images-v1.28.15.tar

# 导入Calico镜像
ctr -n k8s.io images import /opt/k8s-offline/images/calico-images-v3.26.1.tar

# 验证
ctr -n k8s.io images ls
crictl images

**注意**：K8s使用的镜像必须导入到 `k8s.io` 命名空间，否则kubelet找不到镜像。

6. Kubernetes组件离线部署

6.1 安装K8s RPM包

# 所有节点执行
yum install -y kubelet-1.28.15 kubeadm-1.28.15 kubectl-1.28.15

# 启动kubelet（此时会反复重启，属正常现象，kubeadm init后恢复正常）
systemctl enable --now kubelet

6.2 配置kubelet cgroup驱动

cat > /etc/sysconfig/kubelet << 'EOF'
KUBELET_EXTRA_ARGS=--cgroup-driver=systemd --container-runtime-endpoint=unix:///run/containerd/containerd.sock
EOF

systemctl daemon-reload
systemctl restart kubelet

6.3 配置kubectl命令补全

yum install -y bash-completion
source /usr/share/bash-completion/bash_completion
echo 'source <(kubectl completion bash)' >> ~/.bashrc
echo 'alias k=kubectl' >> ~/.bashrc
echo 'complete -F __start_kubectl k' >> ~/.bashrc
source ~/.bashrc

7. 集群初始化与节点加入

7.1 准备kubeadm配置文件

在Master节点创建 /root/kubeadm-config.yaml：

apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.10.10
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///run/containerd/containerd.sock
  name: k8s-master01
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/control-plane
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: v1.28.15
controlPlaneEndpoint: "192.168.10.10:6443"
imageRepository: registry.k8s.io
networking:
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "10.96.0.0/12"
  dnsDomain: "cluster.local"
etcd:
  local:
    dataDir: /var/lib/etcd
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
clusterDomain: cluster.local
clusterDNS:
- 10.96.0.10
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
ipvs:
  scheduler: rr

7.2 预拉取镜像验证

# 查看所需镜像列表
kubeadm config images list --config=/root/kubeadm-config.yaml

# 对比已导入镜像
crictl images | grep -E 'kube-apiserver|kube-controller|kube-scheduler|kube-proxy|pause|etcd|coredns'

7.3 初始化Master节点

kubeadm init --config=/root/kubeadm-config.yaml --upload-certs

成功后输出示例：

Your Kubernetes control-plane has initialized successfully!

kubeadm join 192.168.10.10:6443 --token abcdef.0123456789abcdef \
  --discovery-token-ca-cert-hash sha256:xxxxxxxxxx

7.4 配置kubectl访问

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

kubectl get nodes
kubectl cluster-info

7.5 Worker节点加入集群

# 在Worker节点执行（token和hash从Master init输出获取）
kubeadm join 192.168.10.10:6443 \
  --token abcdef.0123456789abcdef \
  --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \
  --cri-socket unix:///run/containerd/containerd.sock

# 如token过期，在Master重新生成
kubeadm token create --print-join-command

7.6 移除Master污点（可选）

# 单节点集群或需在Master运行业务Pod时执行
kubectl taint nodes k8s-master01 node-role.kubernetes.io/control-plane:NoSchedule-

8. 容器网络插件部署（Calico）

8.1 下载并修改Calico YAML

# 有网环境下载
wget https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/calico.yaml -O calico.yaml

# 修改镜像地址为私有仓库
sed -i 's|docker.io/calico|harbor.local/k8s|g' calico.yaml

# 修改CIDR与kubeadm podSubnet一致
sed -i 's|# - name: CALICO_IPV4POOL_CIDR|- name: CALICO_IPV4POOL_CIDR|' calico.yaml
sed -i 's|#   value: "192.168.0.0/16"|  value: "10.244.0.0/16"|' calico.yaml

8.2 部署Calico

# 所有节点导入Calico镜像
ctr -n k8s.io images import /opt/k8s-offline/images/calico-images-v3.26.1.tar

# Master节点部署
kubectl apply -f calico.yaml

# 等待Pod就绪
kubectl get pods -n kube-system -w

8.3 验证网络

kubectl get nodes -o wide                    # 节点应为Ready
kubectl get pods -n kube-system -l k8s-app=calico-node
kubectl get ippools

9. 离线镜像仓库配置（Harbor对接）

9.1 Containerd配置Harbor认证

mkdir -p /etc/containerd/certs.d/harbor.local

# HTTP内网方式
cat > /etc/containerd/certs.d/harbor.local/hosts.toml << 'EOF'
server = "http://harbor.local"

[host."http://harbor.local"]
  capabilities = ["pull", "resolve", "push"]
  skip_verify = true
EOF

# HTTPS自签名证书方式（需拷贝CA证书）
# cp harbor-ca.crt /etc/containerd/certs.d/harbor.local/ca.crt

systemctl restart containerd

9.2 config.toml仓库配置

在 /etc/containerd/config.toml 中确认：

[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/etc/containerd/certs.d"

9.3 推送镜像到Harbor

# 登录Harbor
ctr -n k8s.io images login harbor.local -u admin -p Harbor12345

# 重新打标签
ctr -n k8s.io images tag \
  registry.k8s.io/pause:3.9 \
  harbor.local/k8s/pause:3.9

# 推送
ctr -n k8s.io images push harbor.local/k8s/pause:3.9

10. 验证流程

10.1 节点状态验证

kubectl get nodes -o wide

预期输出：

NAME            STATUS   ROLES           AGE   VERSION   INTERNAL-IP
k8s-master01    Ready    control-plane   10m   v1.28.15  192.168.10.10
k8s-worker01    Ready    <none>          5m    v1.28.15  192.168.10.21
k8s-worker02    Ready    <none>          5m    v1.28.15  192.168.10.22

10.2 系统Pod验证

kubectl get pods -n kube-system -o wide

核心组件必须全部Running：

• kube-apiserver-master01
• kube-controller-manager-master01
• kube-scheduler-master01
• etcd-master01
• kube-proxy-xxx（每节点一个）
• calico-node-xxx（每节点一个）
• calico-kube-controllers-xxx
• coredns-xxx（2副本）

10.3 容器运行时验证

systemctl is-active containerd    # 应为 active
containerd --version
crictl info | jq '.status'
crictl ps
crictl images

10.4 网络连通性验证

# 创建测试Pod
kubectl run net-test --image=harbor.local/k8s/pause:3.9 --restart=Never
kubectl get pod net-test -o wide

# 跨节点Ping
kubectl run ping-test --image=busybox:1.36 --rm -it --restart=Never -- ping -c 4 <对端PodIP>

# Service连通性
kubectl create deployment nginx --image=harbor.local/k8s/nginx:1.25 --replicas=2
kubectl expose deployment nginx --port=80 --type=ClusterIP
kubectl run curl-test --image=curlimages/curl --rm -it --restart=Never -- curl -s http://nginx

10.5 DNS验证

kubectl run dns-test --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default

预期输出包含：

Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name:      kubernetes.default
Address 1: 10.96.0.1

10.6 IPVS模式验证

ipvsadm -Ln
# 应看到ClusterIP对应的虚拟服务和后端Pod