【k8s】集群内 curl 访问 Connection refused。

---

---

问题。

[root@localhost k8s]# kubectl get pods -n dev -o wide --show-labels
NAME                             READY   STATUS    RESTARTS   AGE     IP            NODE                              NOMINATED NODE   READINESS GATES   LABELS
nginx-778cb5fb7b-dgntb           1/1     Running   0          2d21h   10.244.2.57   localhost.localdomain.k8s.node2   <none>           <none>            pod-template-hash=778cb5fb7b,run=nginx
pc-deployment-6696798b78-9s6pb   1/1     Running   0          19m     10.244.1.61   localhost.localdomain.k8s.node1   <none>           <none>            app=nginx-pod,pod-template-hash=6696798b78
pc-deployment-6696798b78-fz6xg   1/1     Running   0          19m     10.244.2.61   localhost.localdomain.k8s.node2   <none>           <none>            app=nginx-pod,pod-template-hash=6696798b78
pc-deployment-6696798b78-gp6fs   1/1     Running   0          19m     10.244.1.60   localhost.localdomain.k8s.node1   <none>           <none>            app=nginx-pod,pod-template-hash=6696798b78
[root@localhost k8s]# curl 10.244.2.61:80
curl: (7) Failed connect to 10.244.2.61:80; Connection refused

curl 请求总是 Connection refused。

发现 coredns 的状态是 CrashLoopBackOff。

[root@localhost k8s]# kubectl get pod -n kube-system
NAME                                                       READY   STATUS             RESTARTS   AGE
coredns-9d85f5447-5959v                                    0/1     Running            224        6d17h
coredns-9d85f5447-gvqxh                                    0/1     CrashLoopBackOff   222        6d17h
etcd-localhost.localdomain.k8s.master                      1/1     Running            1          6d17h
kube-apiserver-localhost.localdomain.k8s.master            1/1     Running            9          6d17h
kube-controller-manager-localhost.localdomain.k8s.master   1/1     Running            2          6d17h
kube-proxy-bqd7w                                           1/1     Running            0          31m
kube-proxy-nt48k                                           1/1     Running            0          32m
kube-proxy-wpt6n                                           1/1     Running            0          32m
kube-scheduler-localhost.localdomain.k8s.master            1/1     Running            2          6d17h
metrics-server-6b976979db-j5g6s                            1/1     Running            0          2d23h

[root@localhost k8s]# kubectl describe pod/coredns-9d85f5447-gvqxh -n kube-system
Name:                 coredns-9d85f5447-gvqxh
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 localhost.localdomain.k8s.master/192.168.142.150
Start Time:           Fri, 02 Dec 2022 01:22:31 +0800
Labels:               k8s-app=kube-dns
                      pod-template-hash=9d85f5447
Annotations:          <none>
Status:               Running
IP:                   10.244.0.5
IPs:
  IP:           10.244.0.5
Controlled By:  ReplicaSet/coredns-9d85f5447
Containers:
  coredns:
    Container ID:  docker://721345f5f22afd5d756bcba293b8a65e1796362f97b8a23d88d7f94fb603b3f2
    Image:         registry.aliyuncs.com/google_containers/coredns:1.6.5
    Image ID:      docker-pullable://registry.aliyuncs.com/google_containers/coredns@sha256:7ec975f167d815311a7136c32e70735f0d00b73781365df1befd46ed35bd4fe7
    Ports:         53/UDP, 53/TCP, 9153/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Running
      Started:      Thu, 08 Dec 2022 17:30:02 +0800
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 08 Dec 2022 17:22:54 +0800
      Finished:     Thu, 08 Dec 2022 17:24:54 +0800
    Ready:          False
    Restart Count:  221
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from coredns-token-hfjww (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  coredns-token-hfjww:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  coredns-token-hfjww
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  beta.kubernetes.io/os=linux
Tolerations:     CriticalAddonsOnly
                 node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                     From                                       Message
  ----     ------     ----                    ----                                       -------
  Normal   Pulled     45m (x124 over 3d18h)   kubelet, localhost.localdomain.k8s.master  Container image "registry.aliyuncs.com/google_containers/coredns:1.6.5" already present on machine
  Warning  Unhealthy  15m (x1515 over 3d3h)   kubelet, localhost.localdomain.k8s.master  Readiness probe failed: Get http://10.244.0.5:8181/ready: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  Warning  Unhealthy  5m50s (x664 over 3d3h)  kubelet, localhost.localdomain.k8s.master  Liveness probe failed: Get http://10.244.0.5:8080/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  Warning  BackOff    51s (x1553 over 3d3h)   kubelet, localhost.localdomain.k8s.master  Back-off restarting failed container

---

解决。

查看 host。

[root@localhost ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.142.150 localhost.localdomain.k8s.master
192.168.142.151 localhost.localdomain.k8s.node1
192.168.142.152 localhost.localdomain.k8s.node2

查看 /etc/resolv.conf。

[root@localhost ~]# cat /etc/resolv.conf
# Generated by NetworkManager
search localdomain.k8s.master
nameserver 192.168.142.2

发现 /etc/resolv/conf 中的 search 和 hosts 不一致。
修改 /etc/resolv/conf。

[root@localhost ~]# cat /etc/resolv.conf
# Generated by NetworkManager
search localhost.localdomain.k8s.master
nameserver 192.168.142.2

重启，解决。

NAME                                                       READY   STATUS    RESTARTS   AGE
coredns-9d85f5447-5959v                                    1/1     Running   227        6d17h
coredns-9d85f5447-gvqxh                                    1/1     Running   225        6d17h

[root@localhost ~]# curl 10.244.1.62:80
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>