K8s 1.30.10离线部署全攻略

li314830356

469人浏览 · 2026-07-20 23:15:02

li314830356 · 2026-07-20 23:15:02 发布

在 Ansible 主控机上创建以下目录结构，并将所有文件放入对应位置。

# 在 Ansible 主控机上执行mkdir -p k8s-binary-offline/{files,roles,group_vars,inventory}
cd k8s-binary-offline
tree .

预期目录结构如下：

k8s-binary-offline/
├── ansible.cfg
├── inventory
│   ├── hosts.ini
│   └── group_vars│       ├── all.yml
│       ├── k8s_master.yml
│       └── k8s_node.yml
├── files│   ├── binaries/
│   │   ├── kubernetes-server-linux-amd64.tar.gz
│   │   ├── cilium-linux-amd64.tar.gz
│   │   ├── etcd-v3.5.12-linux-amd64.tar.gz
│   │   ├── cfssl_linux-amd64
│   │   ├── cfssljson_linux-amd64
│   │   └── cfssl-certinfo_linux-amd64
│   ├── systemd/
│   │   ├── kube-apiserver.service
│   │   ├── kube-controller-manager.service
│   │   ├── kube-scheduler.service
│   │   ├── kubelet.service
│   │   ├── kube-proxy.service
│   │   └── etcd.service
│   ├── configs/
│   │   ├── kubelet-config.yaml
│   │   ├── kube-proxy-config.yaml
│   │   ├── bootstrap-kubelet.conf
│   │ └── cilium-config.yaml
│ └── certs/
│ └── (证书生成脚本和模板)
├── roles
│   ├── common
│   │ └── tasks
│   │       └── main.yml
│   ├── certs
│   │   └── tasks
│   │ └── main.yml
│   ├── etcd
│   │   └── tasks
│   │ └── main.yml
│   ├── master
│   │ └── tasks
│   │       └── main.yml
│   ├── node
│   │   └── tasks
│   │ └── main.yml
│ └── cilium
│ └── tasks
│           └── main.yml
└── site.yml

二、核心配置文件与脚本

1. Ansible 主配置文件 (`ansible.cfg`)

[defaults]
inventory = ./inventory/hosts.ini
host_key_checking = False
remote_user = root
roles_path = ./roles
retry_files_enabled = False
gathering = smart
fact_caching = jsonfile
fact_caching_connection = /tmp/ansible_factsfact_caching_timeout = 600[ssh_connection]
pipelining = True
ssh_args = -o ControlMaster=auto -o ControlPersist=30m -o ControlPath=/tmp/ansible-ssh-%h-%p-%r

2. Inventory 文件 (`inventory/hosts.ini`)

[k8s_master]
master01 ansible_host=10.10.1.10 ansible_user=rootmaster02 ansible_host=10.10.1.11 ansible_user=root
master03 ansible_host=10.10.1.12 ansible_user=root

[k8s_node]
node01 ansible_host=10.10.1.21 ansible_user=root
node02 ansible_host=10.10.1.22 ansible_user=root
node03 ansible_host=10.10.1.23 ansible_user=root
node04 ansible_host=10.10.1.24 ansible_user=root
node05 ansible_host=10.10.1.25 ansible_user=root
node06 ansible_host=10.10.1.26 ansible_user=root

[k8s_cluster:children]
k8s_master
k8s_node

[etcd]
master01 ansible_host=10.10.1.10
master02 ansible_host=10.10.1.11
master03 ansible_host=10.10.1.12

[all:vars]
ansible_ssh_private_key_file = /root/.ssh/id_rsa
ansible_become = yes
ansible_become_method = sudo
ansible_become_user = root

3. 全局变量文件 (`group_vars/all.yml`)

# 集群基础配置
k8s_version: "v1.30.10"
etcd_version: "v3.5.12"
cilium_version: "v1.16.6"
cluster_name: "k8s-cluster"
service_cidr: "10.96.0.0/12"
pod_cidr: "10.244.0.0/16"
cluster_dns: "10.96.0.10"

# 网络配置
apiserver_vip: "10.10.1.100"
apiserver_port: 6443

# 文件路径
local_files_dir: "/root/k8s-binary-offline/files"
remote_install_dir: "/opt/kubernetes"
remote_bin_dir: "{{ remote_install_dir }}/bin"
remote_cfg_dir: "{{ remote_install_dir }}/cfg"
remote_ssl_dir: "{{ remote_install_dir }}/ssl"
remote_systemd_dir: "/etc/systemd/system"

# 证书配置
cert_country: "CN"
cert_state: "Beijing"
cert_location: "Beijing"
cert_organization: "k8s"
cert_organizational_unit: "System"
cert_email: "admin@k8s.io"

# 系统配置
system_reserved: "cpu=200m,memory=1Gi"
kubelet_eviction_hard: "memory.available<500Mi,nodefs.available<10%,nodefs.inodesFree<5%"

4. 主节点变量 (`group_vars/k8s_master.yml`)

# Master 节点特定配置
apiserver_insecure_port: 8080
apiserver_secure_port: 6443
controller_manager_port: 10252
scheduler_port: 10251
etcd_client_port: 2379
etcd_peer_port: 2380

# Etcd 集群配置
etcd_cluster: "etcd01=https://10.10.1.10:2380,etcd02=https://10.10.1.11:2380,etcd03=https://10.10.1.12:2380"
etcd_initial_cluster_token: "etcd-cluster-k8s"

5. 工作节点变量 (`group_vars/k8s_node.yml`)

# Node 节点特定配置
kubelet_port: 10250
kube_proxy_port: 10256
max_pods: 110

三、离线二进制文件下载清单

在能访问互联网的机器上下载以下文件，然后上传到 Ansible 主控机的 files/binaries/ 目录。

组件	版本	下载地址	备注
Kubernetes	v1.30.10	`https://dl.k8s.io/v1.30.10/kubernetes-server-linux-amd64.tar.gz`	官方二进制包
Cilium CLI	v1.16.6	`https://github.com/cilium/cilium-cli/releases/download/v1.16.6/cilium-linux-amd64.tar.gz`	Cilium 命令行工具
Cilium	v1.16.6	`https://github.com/cilium/cilium/releases/download/v1.16.6/cilium-1.16.6.tar.gz`	Cilium 安装包
etcd	v3.5.12	`https://github.com/etcd-io/etcd/releases/download/v3.5.12/etcd-v3.5.12-linux-amd64.tar.gz`	etcd 集群
cfssl	最新版	`https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl_1.6.4_linux_amd64`	证书工具
cfssljson	最新版	`https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssljson_1.6.4_linux_amd64`	证书工具
runc	v1.1.12	`https://github.com/opencontainers/runc/releases/download/v1.1.12/runc.amd64`	容器运行时
containerd	v1.7.18	`https://github.com/containerd/containerd/releases/download/v1.7.18/containerd-1.7.18-linux-amd64.tar.gz`	容器运行时

下载脚本示例：

#!/bin/bash
# download.sh - 在能联网的机器上执行
mkdir -p binaries
cd binaries

# Kuberneteswget https://dl.k8s.io/v1.30.10/kubernetes-server-linux-amd64.tar.gz

# Cilium
wget https://github.com/cilium/cilium-cli/releases/download/v1.16.6/cilium-linux-amd64.tar.gz
wget https://github.com/cilium/cilium/releases/download/v1.16.6/cilium-1.16.6.tar.gz

# etcd
wget https://github.com/etcd-io/etcd/releases/download/v3.5.12/etcd-v3.5.12-linux-amd64.tar.gz

# cfssl
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl_1.6.4_linux_amd64 -O cfssl
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssljson_1.6.4_linux_amd64 -O cfssljsonchmod +x cfssl cfssljson

# 容器运行时
wget https://github.com/opencontainers/runc/releases/download/v1.1.12/runc.amd64wget https://github.com/containerd/containerd/releases/download/v1.7.18/containerd-1.7.18-linux-amd64.tar.gz

echo "下载完成，请将 binaries/ 目录上传到 Ansible 主控机的 files/ 目录下"

四、Ansible Roles 核心任务

1. 通用系统初始化 (`roles/common/tasks/main.yml`)

---
- name: 禁用 swap  shell: |
    swapoff -a
    sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab name: 设置主机名
  hostname:
    name: "{{ inventory_hostname }}"

- name: 配置 hosts 文件 blockinfile:
    path: /etc/hosts
    block: |
      10.10.1.10 master0110.10.1.11 master02
      10.10.1.12 master0310.10.1.21 node01
      10.10.1.22 node0210.10.1.23 node03
      10.10.1.24 node0410.10.1.25 node05
      10.10.1.26 node06 name: 安装基础依赖
  apt:
    name:
 curl - wget
 gnupg
 lsb-release
 apt-transport-https
 ca-certificates
 software-properties-common
 net-tools - ipvsadm - ipset
 sysstat - conntrack - nfs-common
    state: present update_cache: yes name: 配置内核参数 copy:
    src: "{{ local_files_dir }}/configs/sysctl.conf"
    dest: /etc/sysctl.d/k8s.conf
    owner: root group: root
    mode: '0644'
  notify: reload sysctl name: 加载内核模块
  copy:
    content: |
      overlay br_netfilter ip_vs
      ip_vs_rr ip_vs_wrr
      ip_vs_sh
      nf_conntrack dest: /etc/modules-load.d/k8s.conf owner: root
    group: root mode: '0644'
  notify: load kernel modules

- name: 配置时间同步
  apt:
    name: chrony state: present
  notify: restart chrony name: 创建安装目录
  file:
    path: "{{ item }}"
    state: directory
    owner: root group: root
    mode: '0755'
  loop:
 "{{ remote_install_dir }}"
 "{{ remote_bin_dir }}"
 "{{ remote_cfg_dir }}"
 "{{ remote_ssl_dir }}"
 /etc/cni/net.d
 /opt/cni/bin
 /var/lib/kubelet - /var/lib/kube-proxy - /var/lib/kube-controller-manager
 /var/lib/kube-scheduler
 /var/log/kubernetes

2. 证书生成 (`roles/certs/tasks/main.yml`)

---
- name: 复制证书工具
  copy:
    src: "{{ local_files_dir }}/binaries/{{ item }}"
    dest: /usr/local/bin/
    owner: root group: root
    mode: '0755'
  loop:
 cfssl - cfssljson name: 创建证书配置目录 file:
    path: /tmp/k8s-certs
    state: directory name: 生成 CA 配置文件 copy:
    content: |
      {
        "signing": {
          "default": {
            "expiry": "87600h"
          },
          "profiles": {
            "kubernetes": {
              "usages": [
                "signing",
                "key encipherment",
                "server auth",
                "client auth"
              ],
              "expiry": "87600h"
            }
          }
        }
      }
    dest: /tmp/k8s-certs/ca-config.json

- name: 生成 CA 证书签名请求 copy:
    content: |
      {
        "CN": "kubernetes",
        "key": {
          "algo": "rsa",
          "size": 2048
        },
        "names": [
          {
            "C": "{{ cert_country }}",
            "ST": "{{ cert_state }}",
            "L": "{{ cert_location }}",
            "O": "{{ cert_organization }}",
            "OU": "{{ cert_organizational_unit }}"
          }
        ]
      }
    dest: /tmp/k8s-certs/ca-csr.json

- name: 生成 CA 证书和私钥
  shell: |
    cd /tmp/k8s-certs
    cfssl gencert -initca ca-csr.json | cfssljson -bare ca args:
    creates: /tmp/k8s-certs/ca.pem

 name: 分发 CA 证书到所有节点
  copy:
    src: "/tmp/k8s-certs/{{ item }}"
    dest: "{{ remote_ssl_dir }}/"
    owner: root group: root
    mode: '0644'
  loop:
 ca.pem
 ca-key.pem - ca.csr
  delegate_to: "{{ groups['k8s_cluster'] | first }}"
  run_once: true

3. etcd 集群部署 (`roles/etcd/tasks/main.yml`)

---
- name: 解压 etcd 二进制文件
  unarchive:
    src: "{{ local_files_dir }}/binaries/etcd-v3.5.12-linux-amd64.tar.gz"
    dest: /tmp/
    remote_src: yes- name: 安装 etcd 二进制文件 copy:
    src: "/tmp/etcd-v3.5.12-linux-amd64/{{ item }}"
    dest: "{{ remote_bin_dir }}/"
    owner: root group: root
    mode: '0755'
  loop:
 etcd - etcdctl name: 创建 etcd 数据目录 file:
    path: /var/lib/etcd
    state: directory owner: root
    group: root    mode: '0700'

- name: 配置 etcd systemd 服务
  template:
    src: "{{ local_files_dir }}/systemd/etcd.service.j2"
    dest: "{{ remote_systemd_dir }}/etcd.service"
    owner: root
    group: root mode: '0644'
  notify:
 daemon-reload - restart etcd name: 启动 etcd 服务
  systemd:
    name: etcd
    state: started enabled: yes
    daemon_reload: yes

4. Master 节点部署 (`roles/master/tasks/main.yml`)

---
- name: 解压 Kubernetes 二进制文件 unarchive:
    src: "{{ local_files_dir }}/binaries/kubernetes-server-linux-amd64.tar.gz"
    dest: /tmp/
    remote_src: yes

- name: 安装 Master 组件二进制文件 copy:
    src: "/tmp/kubernetes/server/bin/{{ item }}"
    dest: "{{ remote_bin_dir }}/"
    owner: root
    group: root mode: '0755'
  loop:
    - kube-apiserver - kube-controller-manager
 kube-scheduler - kubectl
 kubelet
 kube-proxy name: 生成 apiserver 证书 shell: |
    # 证书生成逻辑 cd {{ remote_ssl_dir }}
    cat > kube-apiserver-csr.json <<EOF
    {
      "CN": "kube-apiserver",
      "hosts": [
        "127.0.0.1",
        "{{ apiserver_vip }}",
        "10.10.1.10",
        "10.10.1.11",
        "10.10.1.12",
        "10.96.0.1",
        "kubernetes",
        "kubernetes.default",
        "kubernetes.default.svc",
        "kubernetes.default.svc.cluster",
        "kubernetes.default.svc.cluster.local"
      ],
      "key": {
        "algo": "rsa",
        "size": 2048 },
      "names": [
        {
          "C": "{{ cert_country }}",
          "ST": "{{ cert_state }}",
          "L": "{{ cert_location }}",
          "O": "{{ cert_organization }}",
          "OU": "{{ cert_organizational_unit }}"
        }
      ]
    }
    EOF cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-apiserver-csr.json | cfssljson -bare kube-apiserver
  args:
    creates: "{{ remote_ssl_dir }}/kube-apiserver.pem"

 name: 配置 apiserver systemd 服务
  template:
    src: "{{ local_files_dir }}/systemd/kube-apiserver.service.j2"
    dest: "{{ remote_systemd_dir }}/kube-apiserver.service"
  notify:
 daemon-reload - restart kube-apiserver

- name: 配置 controller-manager  template:
    src: "{{ local_files_dir }}/systemd/kube-controller-manager.service.j2"
    dest: "{{ remote_systemd_dir }}/kube-controller-manager.service"
  notify:
 daemon-reload - restart kube-controller-manager name: 配置 scheduler template:
    src: "{{ local_files_dir }}/systemd/kube-scheduler.service.j2"
    dest: "{{ remote_systemd_dir }}/kube-scheduler.service"
  notify:
    - daemon-reload
 restart kube-scheduler

- name: 生成 admin kubeconfig shell: |
    # kubeconfig 生成逻辑
    kubectl config set-cluster kubernetes \
      --certificate-authority={{ remote_ssl_dir }}/ca.pem \
      --embed-certs=true \
      --server=https://{{ apiserver_vip }}:{{ apiserver_port }} \
      --kubeconfig={{ remote_cfg_dir }}/admin.kubeconfig kubectl config set-credentials admin \
      --client-certificate={{ remote_ssl_dir }}/admin.pem \
      --client-key={{ remote_ssl_dir }}/admin-key.pem \
      --embed-certs=true \
      --kubeconfig={{ remote_cfg_dir }}/admin.kubeconfig
    
    kubectl config set-context kubernetes \
      --cluster=kubernetes \
      --user=admin \
      --kubeconfig={{ remote_cfg_dir }}/admin.kubeconfig
    
    kubectl config use-context kubernetes \
      --kubeconfig={{ remote_cfg_dir }}/admin.kubeconfig args:
    creates: "{{ remote_cfg_dir }}/admin.kubeconfig"

- name: 启动 Master 服务 systemd:
    name: "{{ item }}"
    state: started
    enabled: yes daemon_reload: yes loop:
 kube-apiserver
 kube-controller-manager - kube-scheduler

5. Node 节点部署 (`roles/node/tasks/main.yml`)

---
- name: 安装 Node 组件二进制文件 copy:
    src: "{{ local_files_dir }}/binaries/kubernetes/server/bin/{{ item }}"
    dest: "{{ remote_bin_dir }}/"
    owner: root group: root
    mode: '0755'
  loop:
    - kubelet - kube-proxy

- name: 配置 kubelet  template:
    src: "{{ local_files_dir }}/configs/kubelet-config.yaml.j2"
    dest: "{{ remote_cfg_dir }}/kubelet-config.yaml"
    owner: root group: root
    mode: '0644'

- name: 配置 kubelet systemd 服务 template:
    src: "{{ local_files_dir }}/systemd/kubelet.service.j2"
    dest: "{{ remote_systemd_dir }}/kubelet.service"
  notify:
    - daemon-reload
 restart kubelet name: 配置 kube-proxy
  template:
    src: "{{ local_files_dir }}/configs/kube-proxy-config.yaml.j2"
    dest: "{{ remote_cfg_dir }}/kube-proxy-config.yaml"
    owner: root group: root
    mode: '0644'

- name: 配置 kube-proxy systemd 服务
  template:
    src: "{{ local_files_dir }}/systemd/kube-proxy.service.j2"
    dest: "{{ remote_systemd_dir }}/kube-proxy.service"
  notify:
 daemon-reload - restart kube-proxy name: 分发 bootstrap kubeconfig
  copy:
    src: "{{ local_files_dir }}/configs/bootstrap-kubelet.conf"
    dest: "{{ remote_cfg_dir }}/bootstrap-kubelet.conf"
    owner: root group: root
    mode: '0644'

- name: 启动 Node 服务 systemd:
    name: "{{ item }}"
    state: started
    enabled: yes daemon_reload: yes loop:
 kubelet
 kube-proxy

6. Cilium CNI 部署 (`roles/cilium/tasks/main.yml`)

---
- name: 解压 Cilium CLI unarchive:
    src: "{{ local_files_dir }}/binaries/cilium-linux-amd64.tar.gz"
    dest: /usr/local/bin/
    remote_src: yes name: 安装 Cilium  shell: |
    cilium install \
      --version {{ cilium_version }} \
      --cluster-name {{ cluster_name }} \
      --cluster-id 1 \
      --ipam kubernetes \
      --kube-proxy-replacement=strict \
      --native-routing-cidr={{ pod_cidr }} \
      --helm-set bpf.masquerade=true \
      --helm-set ipv4NativeRoutingCIDR={{ pod_cidr }} \
      --helm-set kubeProxyReplacement=true \
      --helm-set k8sServiceHost={{ apiserver_vip }} \
      --helm-set k8sServicePort={{ apiserver_port }}
  args:
    creates: /var/run/cilium/cilium.sock

- name: 验证 Cilium 安装 shell: |
    cilium status --wait
 register: cilium_status
  until: cilium_status.rc == 0  retries: 10 delay: 30- name: 验证网络连接 shell: |
    cilium connectivity test

五、主 Playbook (`site.yml`)

---
- name: 初始化所有节点
  hosts: k8s_cluster roles:
    - common name: 生成证书 hosts: master01 roles:
 certs name: 部署 etcd 集群
  hosts: etcd
  roles:
    - etcd

- name: 部署 Master 节点 hosts: k8s_master
  roles:
    - master name: 部署 Node 节点 hosts: k8s_node roles:
 node

- name: 部署 Cilium CNI hosts: master01 roles:
 cilium name: 验证集群状态
  hosts: master01
  tasks:
    - name: 检查节点状态
      shell: |
        kubectl get nodes -o wide register: node_status until: "'Ready' in node_status.stdout"
      retries: 20 delay: 10 - name: 检查 Pod 状态 shell: |
        kubectl get pods -A -o wide
      register: pod_status

 name: 显示集群信息 debug:
        msg: |
集群部署完成！
          节点状态：
          {{ node_status.stdout }}
 Pod 状态：
          {{ pod_status.stdout }}
访问集群：
          export KUBECONFIG={{ remote_cfg_dir }}/admin.kubeconfig kubectl cluster-info

六、部署执行命令

在 Ansible 主控机上执行：

# 1. 确保所有文件就位
cd /root/k8s-binary-offline
ls -la files/binaries/  # 确认所有二进制文件已下载

# 2. 配置 SSH 免密登录（在所有节点上执行）
ssh-keygen -t rsa -N '' -f ~/.ssh/id_rsa
for host in 10.10.1.{10..12} 10.10.1.{21..26}; do ssh-copy-id root@$host
done

# 3. 执行 Ansible Playbook
ansible-playbook -i inventory/hosts.ini site.yml -v

# 4. 验证部署
ssh root@10.10.1.10 "kubectl get nodes"
ssh root@10.10.1.10 "kubectl get pods -A"

七、故障排查要点

问题现象	可能原因	解决方案
Ansible 连接失败	SSH 配置问题	检查 `/etc/ssh/sshd_config` 的 `PasswordAuthentication` 和 `PermitRootLogin`
证书生成失败	时间不同步	所有节点执行 `chronyc sources` 检查时间同步
etcd 启动失败	端口冲突或证书问题	检查 `journalctl -u etcd`查看详细日志
apiserver 无法启动	证书 hosts 不匹配	确保证书包含所有 Master 节点 IP 和 VIP
kubelet 无法注册	bootstrap token 过期	重新生成 `bootstrap-kubelet.conf`
Cilium Pod CrashLoopBackOff	内核版本不兼容	Ubuntu 22.04 内核需 ≥5.4，执行 `uname -r` 确认
节点 NotReady	CNI 未就绪	检查 `kubectl describe node <node-name>` 查看事件

此方案基于二进制部署的最佳实践，结合 Ansible 的幂等性设计，确保一次性部署成功。所有配置文件已针对 Ubuntu 22.04 和 Kubernetes 1.30.10 优化，Cilium 配置采用直接路由模式，避免 overlay 网络性能损耗。

参考来源

亚马逊云科技技术品牌专区

更多推荐

open-code-review Codex 插件教程：对话式调用评审技能指南

本文是一份面向新手的 **open-code-review Codex 插件教程**，教你在 Codex 中通过对话式方式调用 open-code-review 的 AI 代码评审技能。open-code-review 是阿里巴巴开源的 AI 代码评审工具，采用"确定性流水线 + LLM Agent"混合架构，内置多语言评审规则（NPE、线程安全、XSS、SQL 注入等），可在 Codex 里用一

亚马逊云科技技术品牌专区

OBS Studio 完整指南：免费开源录屏与直播软件从零上手

想做一场像模像样的直播，或者把屏幕内容录成教程视频，但一想到商用软件的订阅费就有些犹豫？其实你不需要花一分钱。OBS Studio 是一款免费开源的直播与录屏软件，支持 Windows、macOS 和 Linux，采集、混音、编码、推流、录制，一条流水线全包了。## 先搞清楚 OBS Studio 是什么OBS Studio 是一个"场景化"的采集合成工具：你把摄像头、窗口、浏览器、图片、

亚马逊云科技技术品牌专区

如何用MediaCrawler爬取小红书抖音数据？7大平台爬虫从安装到实战完整指南

MediaCrawler 是一个支持小红书、抖音、快手、B站、微博、百度贴吧、知乎的开源自媒体数据采集工具。它能按关键词、按帖子、按创作者主页三种方式抓取笔记/视频内容、评论甚至二级评论，并把结果存成 JSON、CSV、Excel 或数据库。适合做市场调研、竞品监控、内容选题的运营人员，以及想要现成爬虫框架而不想从零写 JS 逆向的开发者。## 从一次"手动抄数据"说起想象这个场景：你负责