本文是这次升级的完整流程记录，将升级流程与遇到的问题都整理进来了。后续升级时如果遇到什么新问题，也都会更新到本文中。

升级前的集群状态：

# kubectl get node -o wide
NAME    STATUS   ROLES           AGE    VERSION    OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
x.x.x   Ready    control-plane   342d   v1.31.14   Ubuntu 24.04.4 LTS   6.8.0-124-generic   containerd://1.7.27
x.x.x   Ready    control-plane   342d   v1.31.14   Ubuntu 24.04.4 LTS   6.8.0-124-generic   containerd://1.7.27
x.x.x   Ready    control-plane   342d   v1.31.14   Ubuntu 24.04.4 LTS   6.8.0-124-generic   containerd://1.7.27

Kubernetes 升级

1.版本升级规划

K8s 官方文档明确说明：

Skipping MINOR versions when upgrading is unsupported

也就是说，从 v1.31.14 升级到 v1.35.6，不能一步到位，必须逐个版本升级，共经历四跳：

v1.31.14 -> v1.32.x -> v1.33.x -> v1.34.x -> v1.35.6

2.版本升级流程

由于不能跳版本，v1.31 -> v1.35 本质上是把单次升级流程重复执行四遍：

以下步骤针对每个控制平面节点：每完成一个版本升级后，所有节点同步升级，再进入下一个版本升级流程。

2.1.腾空节点

只对当前升级的节点进行：

kubectl drain xxx \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --timeout=300s

2.2.备份 ETCD

在所有控制平面节点上执行。

ETCDCTL_API=3
etcdctl --endpoints=https://127.0.0.1:2379 \
        --cacert=/etc/kubernetes/pki/etcd/ca.crt \
        --cert=/etc/kubernetes/pki/etcd/server.crt \
        --key=/etc/kubernetes/pki/etcd/server.key \
        snapshot save /root/backup/etcd-$(date +%F-%H%M).db

执行成功后输出一堆 info 日志，最终看到 Snapshot saved at ... 就备份成功了：

{"level":"info","ts":"xxxx-xx-xx","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/backup/etcd-xxxx-xx-xx-xxxx.db.part"}
{"level":"info","ts":"xxxx-xx-xx","logger":"client","caller":"v3@v3.5.21/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"xxxx-xx-xx","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"xx MB","took":"xx second ago"}
{"level":"info","ts":"xxxx-xx-xx","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/backup/etcd-xxxx-xx-xx-xxxx.db"}
Snapshot saved at /backup/etcd-x.x.x-xxxx.db

2.3.更新软件源

当前环境使用清华源。每次版本升级，都需要把旧版本源注释或清理，换成新版本源并下载对应的仓库公钥。

在所有控制平面节点更新：

# 下载前先将旧版本公钥改名备份一下,等升级结束后再清理
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.32/deb/Release.key \
  | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

# 配置新版本源
# $K8S_VERSION: 具体要升级的版本
cat /etc/apt/sources.list.d/kubernetes.list
deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] http://mirrors4.tuna.tsinghua.edu.cn/kubernetes/core:/stable:/$K8S_VERSION/deb/ /

# 更新源
apt-get update

# 列出当前可用版本
apt-cache madison kubeadm | head -n 1
   # 示例输出
   kubeadm | 1.32.13-1.1 | http://mirrors4.tuna.tsinghua.edu.cn/kubernetes/core:/stable:/v1.32/deb  Packages
apt-cache madison kubelet | head -n 1
apt-cache madison kubectl | head -n 1

# 确认组件未被 apt 锁定(hold)
apt-mark showhold | grep -Ei "kubeadm|kubelet|kubectl"

2.4.验证升级计划

检查可升级到哪些版本，并验证当前集群是否可升级。该命令只能在存在 kubeconfig 文件 admin.conf 的控制平面节点上运行。

# 安装目标版本的 kubeadm
apt-get install -y kubeadm='1.32.13-1.1'

# 确认版本
kubeadm version

# 验证升级计划: kubeadm 会列出当前版本/目标版本/各组件变更
kubeadm upgrade plan
# 输出内容(忽略输出的版本信息)
[preflight] Running pre-flight checks.
[upgrade/config] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade/config] Use 'kubeadm init phase upload-config --config your-config-file' to re-upload it.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: 1.32.13
[upgrade/versions] kubeadm version: v1.33.13
I0622 18:27:10.934338 3715038 version.go:261] remote version is much newer: v1.36.2; falling back to: stable-1.33
[upgrade/versions] Target version: v1.33.13
[upgrade/versions] Latest version in the v1.32 series: v1.32.13

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   NODE    CURRENT    TARGET
kubelet     x.x.x   v1.32.13   v1.33.13
kubelet     x.x.x   v1.32.13   v1.33.13
kubelet     x.x.x   v1.32.13   v1.33.13

Upgrade to the latest stable version:

COMPONENT                 NODE      CURRENT    TARGET
kube-apiserver            x.x.x     v1.32.13   v1.33.13
kube-apiserver            x.x.x     v1.32.13   v1.33.13
kube-apiserver            x.x.x     v1.32.13   v1.33.13
kube-controller-manager   x.x.x     v1.32.13   v1.33.13
kube-controller-manager   x.x.x     v1.32.13   v1.33.13
kube-controller-manager   x.x.x     v1.32.13   v1.33.13
kube-scheduler            x.x.x     v1.32.13   v1.33.13
kube-scheduler            x.x.x     v1.32.13   v1.33.13
kube-scheduler            x.x.x     v1.32.13   v1.33.13
kube-proxy                          1.32.13    v1.33.13
CoreDNS                             v1.11.3    v1.12.0
etcd                      x.x.x     3.5.24-0   3.5.24-0
etcd                      x.x.x     3.5.24-0   3.5.24-0
etcd                      x.x.x     3.5.24-0   3.5.24-0

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.33.13