问题

/usr/local/software/k8s$ sudo kubeadm init --config kubeadm-config.yaml

The API server is not healthy after 4m0.004632869s

Unfortunately, an error has occurred:
        context deadline exceeded

This error is likely caused by:
        - The kubelet is not running
        - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
        - 'systemctl status kubelet'
        - 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.
Here is one example how you may list all running Kubernetes containers by using crictl:
        - 'crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock ps -a | grep kube | grep -v pause'
        Once you have found the failing container, you can inspect its logs with:
        - 'crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock logs CONTAINERID'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher

排除

运行

journalctl -fu containerd

报错内容如下

-- Logs begin at Thu 2024-07-18 14:34:17 CST. --
7月 19 16:47:26 k8s-master containerd[877]: time="2024-07-19T16:47:26.457084029+08:00" level=info msg="trying next host" error="failed to do request: Head \"https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9\": dial tcp 74.125.23.82:443: connect: connection refused" host=registry.k8s.io
7月 19 16:47:26 k8s-master containerd[877]: time="2024-07-19T16:47:26.459027749+08:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:kube-apiserver-k8s-master,Uid:040f049f212bc464de4480954377b49d,Namespace:kube-system,Attempt:0,} failed, error" error="failed to get sandbox image \"registry.k8s.io/pause:3.9\": failed to pull image \"registry.k8s.io/pause:3.9\": failed to pull and unpack image \"registry.k8s.io/pause:3.9\": failed to resolve reference \"registry.k8s.io/pause:3.9\": failed to do request: Head \"https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9\": dial tcp 74.125.23.82:443: connect: connection refused"
7月 19 16:47:26 k8s-master containerd[877]: time="2024-07-19T16:47:26.459475414+08:00" level=info msg="stop pulling image registry.k8s.io/pause:3.9: active requests=0, bytes read=0"
7月 19 16:47:35 k8s-master containerd[877]: time="2024-07-19T16:47:35.443590860+08:00" level=info msg="trying next host" error="failed to do request: Head \"https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9\": dial tcp 74.125.23.82:443: connect: connection refused" host=registry.k8s.io
7月 19 16:47:35 k8s-master containerd[877]: time="2024-07-19T16:47:35.445345037+08:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:kube-scheduler-k8s-master,Uid:848918b6a55c77ff12f403b148f97751,Namespace:kube-system,Attempt:0,} failed, error" error="failed to get sandbox image \"registry.k8s.io/pause:3.9\": failed to pull image \"registry.k8s.io/pause:3.9\": failed to pull and unpack image \"registry.k8s.io/pause:3.9\": failed to resolve reference \"registry.k8s.io/pause:3.9\": failed to do request: Head \"https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9\": dial tcp 74.125.23.82:443: connect: connection refused"
7月 19 16:47:35 k8s-master containerd[877]: time="2024-07-19T16:47:35.445526564+08:00" level=info msg="stop pulling image registry.k8s.io/pause:3.9: active requests=0, bytes read=0"
7月 19 16:47:36 k8s-master containerd[877]: time="2024-07-19T16:47:36.589531271+08:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-k8s-master,Uid:7c8f2109063e96a79c8c6b7f134afcdb,Namespace:kube-system,Attempt:0,}"
7月 19 16:47:37 k8s-master containerd[877]: time="2024-07-19T16:47:37.462028459+08:00" level=info msg="trying next host" error="failed to do request: Head \"https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9\": dial tcp 74.125.23.82:443: connect: connection refused" host=registry.k8s.io
7月 19 16:47:37 k8s-master containerd[877]: time="2024-07-19T16:47:37.464171749+08:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:kube-controller-manager-k8s-master,Uid:827aa8e26beb088b734191c249f58aed,Namespace:kube-system,Attempt:0,} failed, error" error="failed to get sandbox image \"registry.k8s.io/pause:3.9\": failed to pull image \"registry.k8s.io/pause:3.9\": failed to pull and unpack image \"registry.k8s.io/pause:3.9\": failed to resolve reference \"registry.k8s.io/pause:3.9\": failed to do request: Head \"https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9\": dial tcp 74.125.23.82:443: connect: connection refused"
7月 19 16:47:37 k8s-master containerd[877]: time="2024-07-19T16:47:37.464335081+08:00" level=info msg="stop pulling image registry.k8s.io/pause:3.9: active requests=0, bytes read=0"
7月 19 16:47:40 k8s-master containerd[877]: time="2024-07-19T16:47:40.589231392+08:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:kube-apiserver-k8s-master,Uid:040f049f212bc464de4480954377b49d,Namespace:kube-system,Attempt:0,}"
7月 19 16:47:49 k8s-master containerd[877]: time="2024-07-19T16:47:49.589525773+08:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:kube-controller-manager-k8s-master,Uid:827aa8e26beb088b734191c249f58aed,Namespace:kube-system,Attempt:0,}"
7月 19 16:47:49 k8s-master containerd[877]: time="2024-07-19T16:47:49.590885115+08:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:kube-scheduler-k8s-master,Uid:848918b6a55c77ff12f403b148f97751,Namespace:kube-system,Attempt:0,}"
7月 19 16:47:58 k8s-master containerd[877]: time="2024-07-19T16:47:58.542361765+08:00" level=info msg="trying next host" error="failed to do request: Head \"https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9\": dial tcp 74.125.23.82:443: connect: connection refused" host=registry.k8s.io
7月 19 16:47:58 k8s-master containerd[877]: time="2024-07-19T16:47:58.544054030+08:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-k8s-master,Uid:7c8f2109063e96a79c8c6b7f134afcdb,Namespace:kube-system,Attempt:0,} failed, error" error="failed to get sandbox image \"registry.k8s.io/pause:3.9\": failed to pull image \"registry.k8s.io/pause:3.9\": failed to pull and unpack image \"registry.k8s.io/pause:3.9\": failed to resolve reference \"registry.k8s.io/pause:3.9\": failed to do request: Head \"https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9\": dial tcp 74.125.23.82:443: connect: connection refused"
7月 19 16:47:58 k8s-master containerd[877]: time="2024-07-19T16:47:58.544218745+08:00" level=info msg="stop pulling image registry.k8s.io/pause:3.9: active requests=0, bytes read=0"
7月 19 16:48:02 k8s-master containerd[877]: time="2024-07-19T16:48:02.538348935+08:00" level=info msg="trying next host" error="failed to do request: Head \"https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9\": dial tcp 74.125.23.82:443: connect: connection refused" host=registry.k8s.io
7月 19 16:48:02 k8s-master containerd[877]: time="2024-07-19T16:48:02.540741931+08:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:kube-apiserver-k8s-master,Uid:040f049f212bc464de4480954377b49d,Namespace:kube-system,Attempt:0,} failed, error" error="failed to get sandbox image \"registry.k8s.io/pause:3.9\": failed to pull image \"registry.k8s.io/pause:3.9\": failed to pull and unpack image \"registry.k8s.io/pause:3.9\": failed to resolve reference \"registry.k8s.io/pause:3.9\": failed to do request: Head \"https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9\": dial tcp 74.125.23.82:443: connect: connection refused"
7月 19 16:48:02 k8s-master containerd[877]: time="2024-07-19T16:48:02.541007538+08:00" level=info msg="stop pulling image registry.k8s.io/pause:3.9: active requests=0, bytes read=0"
7月 19 16:48:09 k8s-master containerd[877]: time="2024-07-19T16:48:09.581936202+08:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-k8s-master,Uid:7c8f2109063e96a79c8c6b7f134afcdb,Namespace:kube-system,Attempt:0,}"
7月 19 16:48:11 k8s-master containerd[877]: time="2024-07-19T16:48:11.455265832+08:00" level=info msg="trying next host" error="failed to do request: Head \"https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9\": dial tcp 74.125.23.82:443: connect: connection refused" host=registry.k8s.io
7月 19 16:48:11 k8s-master containerd[877]: time="2024-07-19T16:48:11.458479611+08:00" level=info msg="trying next host" error="failed to do request: Head \"https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9\": dial tcp 74.125.23.82:443: connect: connection refused" host=registry.k8s.io
7月 19 16:48:11 k8s-master containerd[877]: time="2024-07-19T16:48:11.459690367+08:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:kube-scheduler-k8s-master,Uid:848918b6a55c77ff12f403b148f97751,Namespace:kube-system,Attempt:0,} failed, error" error="failed to get sandbox image \"registry.k8s.io/pause:3.9\": failed to pull image \"registry.k8s.io/pause:3.9\": failed to pull and unpack image \"registry.k8s.io/pause:3.9\": failed to resolve reference \"registry.k8s.io/pause:3.9\": failed to do request: Head \"https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9\": dial tcp 74.125.23.82:443: connect: connection refused"
7月 19 16:48:11 k8s-master containerd[877]: time="2024-07-19T16:48:11.459912502+08:00" level=info msg="stop pulling image registry.k8s.io/pause:3.9: active requests=0, bytes read=0"
7月 19 16:48:11 k8s-master containerd[877]: time="2024-07-19T16:48:11.462530399+08:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:kube-controller-manager-k8s-master,Uid:827aa8e26beb088b734191c249f58aed,Namespace:kube-system,Attempt:0,} failed, error" error="failed to get sandbox image \"registry.k8s.io/pause:3.9\": failed to pull image \"registry.k8s.io/pause:3.9\": failed to pull and unpack image \"registry.k8s.io/pause:3.9\": failed to resolve reference \"registry.k8s.io/pause:3.9\": failed to do request: Head \"https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9\": dial tcp 74.125.23.82:443: connect: connection refused"
7月 19 16:48:11 k8s-master containerd[877]: time="2024-07-19T16:48:11.462683493+08:00" level=info msg="stop pulling image registry.k8s.io/pause:3.9: active requests=0, bytes read=0"
7月 19 16:48:16 k8s-master containerd[877]: time="2024-07-19T16:48:16.589948845+08:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:kube-apiserver-k8s-master,Uid:040f049f212bc464de4480954377b49d,Namespace:kube-system,Attempt:0,}"
7月 19 16:48:24 k8s-master containerd[877]: time="2024-07-19T16:48:24.584490833+08:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:kube-scheduler-k8s-master,Uid:848918b6a55c77ff12f403b148f97751,Namespace:kube-system,Attempt:0,}"
7月 19 16:48:26 k8s-master containerd[877]: time="2024-07-19T16:48:26.580524377+08:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:kube-controller-manager-k8s-master,Uid:827aa8e26beb088b734191c249f58aed,Namespace:kube-system,Attempt:0,}"
7月 19 16:48:31 k8s-master containerd[877]: time="2024-07-19T16:48:31.439020488+08:00" level=info msg="trying next host" error="failed to do request: Head \"https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9\": dial tcp 74.125.23.82:443: connect: connection refused" host=registry.k8s.io
7月 19 16:48:31 k8s-master containerd[877]: time="2024-07-19T16:48:31.441775988+08:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-k8s-master,Uid:7c8f2109063e96a79c8c6b7f134afcdb,Namespace:kube-system,Attempt:0,} failed, error" error="failed to get sandbox image \"registry.k8s.io/pause:3.9\": failed to pull image \"registry.k8s.io/pause:3.9\": failed to pull and unpack image \"registry.k8s.io/pause:3.9\": failed to resolve reference \"registry.k8s.io/pause:3.9\": failed to do request: Head \"https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9\": dial tcp 74.125.23.82:443: connect: connection refused"
7月 19 16:48:31 k8s-master containerd[877]: time="2024-07-19T16:48:31.441988378+08:00" level=info msg="stop pulling image registry.k8s.io/pause:3.9: active requests=0, bytes read=0"
7月 19 16:48:38 k8s-master containerd[877]: time="2024-07-19T16:48:38.452420339+08:00" level=info msg="trying next host" error="failed to do request: Head \"https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9\": dial tcp 74.125.23.82:443: connect: connection refused" host=registry.k8s.io
7月 19 16:48:38 k8s-master containerd[877]: time="2024-07-19T16:48:38.455780566+08:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:kube-apiserver-k8s-master,Uid:040f049f212bc464de4480954377b49d,Namespace:kube-system,Attempt:0,} failed, error" error="failed to get sandbox image \"registry.k8s.io/pause:3.9\": failed to pull image \"registry.k8s.io/pause:3.9\": failed to pull and unpack image \"registry.k8s.io/pause:3.9\": failed to resolve reference \"registry.k8s.io/pause:3.9\": failed to do request: Head \"https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9\": dial tcp 74.125.23.82:443: connect: connection refused"
7月 19 16:48:38 k8s-master containerd[877]: time="2024-07-19T16:48:38.456099172+08:00" level=info msg="stop pulling image registry.k8s.io/pause:3.9: active requests=0, bytes read=0"
7月 19 16:48:43 k8s-master containerd[877]: time="2024-07-19T16:48:43.577890069+08:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-k8s-master,Uid:7c8f2109063e96a79c8c6b7f134afcdb,Namespace:kube-system,Attempt:0,}"
7月 19 16:48:46 k8s-master containerd[877]: time="2024-07-19T16:48:46.433617680+08:00" level=info msg="trying next host" error="failed to do request: Head \"https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9\": dial tcp 74.125.23.82:443: connect: connection refused" host=registry.k8s.io
7月 19 16:48:46 k8s-master containerd[877]: time="2024-07-19T16:48:46.435805148+08:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:kube-scheduler-k8s-master,Uid:848918b6a55c77ff12f403b148f97751,Namespace:kube-system,Attempt:0,} failed, error" error="failed to get sandbox image \"registry.k8s.io/pause:3.9\": failed to pull image \"registry.k8s.io/pause:3.9\": failed to pull and unpack image \"registry.k8s.io/pause:3.9\": failed to resolve reference \"registry.k8s.io/pause:3.9\": failed to do request: Head \"https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9\": dial tcp 74.125.23.82:443: connect: connection refused"
7月 19 16:48:46 k8s-master containerd[877]: time="2024-07-19T16:48:46.435865964+08:00" level=info msg="stop pulling image registry.k8s.io/pause:3.9: active requests=0, bytes read=0"
7月 19 16:48:48 k8s-master containerd[877]: time="2024-07-19T16:48:48.452911773+08:00" level=info msg="trying next host" error="failed to do request: Head \"https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9\": dial tcp 74.125.23.82:443: connect: connection refused" host=registry.k8s.io
7月 19 16:48:48 k8s-master containerd[877]: time="2024-07-19T16:48:48.455955250+08:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:kube-controller-manager-k8s-master,Uid:827aa8e26beb088b734191c249f58aed,Namespace:kube-system,Attempt:0,} failed, error" error="failed to get sandbox image \"registry.k8s.io/pause:3.9\": failed to pull image \"registry.k8s.io/pause:3.9\": failed to pull and unpack image \"registry.k8s.io/pause:3.9\": failed to resolve reference \"registry.k8s.io/pause:3.9\": failed to do request: Head \"https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9\": dial tcp 74.125.23.82:443: connect: connection refused"
7月 19 16:48:48 k8s-master containerd[877]: time="2024-07-19T16:48:48.456161772+08:00" level=info msg="stop pulling image registry.k8s.io/pause:3.9: active requests=0, bytes read=0"

排查

 containerd config dump

 sudo  crictl images list

发现containerd 的镜像喝crictl不同

修改containerd 配置如图

sudo vim /etc/containerd/config.toml

重启containerd

sudo systemctl restart containerd

重置

 sudo kubeadm reset

 sudo rm -rf /etc/kubernetes/*

初始化

sudo kubeadm init --config kubeadm-config.yaml