问题背景与描述

这里是使用K8S集群启动的一个Jenkins的pod。

Failure -
java.net.UnknownHostException: updates.jenkins.io
	at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source)
	at java.base/java.net.SocksSocketImpl.connect(Unknown Source)
	at java.base/java.net.Socket.connect(Unknown Source)
	at java.base/sun.security.ssl.SSLSocketImpl.connect(Unknown Source)
	at java.base/sun.net.NetworkClient.doConnect(Unknown Source)
	at java.base/sun.net.www.http.HttpClient.openServer(Unknown Source)
	at java.base/sun.net.www.http.HttpClient.openServer(Unknown Source)
	at java.base/sun.net.www.protocol.https.HttpsClient.<init>(Unknown Source)
	at java.base/sun.net.www.protocol.https.HttpsClient.New(Unknown Source)
	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(Unknown Source)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(Unknown Source)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)
	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getHeaderField(Unknown Source)
	at java.base/java.net.URLConnection.getHeaderFieldLong(Unknown Source)
	at java.base/java.net.URLConnection.getContentLengthLong(Unknown Source)
	at java.base/java.net.URLConnection.getContentLength(Unknown Source)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getContentLength(Unknown Source)
	at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1261)
Caused: java.net.UnknownHostException: updates.jenkins.io
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
	at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
	at java.base/sun.net.www.protocol.http.HttpURLConnection$10.run(Unknown Source)
	at java.base/sun.net.www.protocol.http.HttpURLConnection$10.run(Unknown Source)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getChainedException(Unknown Source)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(Unknown Source)
	at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1277)
Caused: java.io.IOException: Failed to load https://updates.jenkins.io/download/plugins/sshd/3.1.0/sshd.hpi to /var/jenkins_home/plugins/sshd.jpi.tmp
	at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1284)
Caused: java.io.IOException: Failed to download from https://updates.jenkins.io/download/plugins/sshd/3.1.0/sshd.hpi
	at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1318)
	at hudson.model.UpdateCenter$DownloadJob._run(UpdateCenter.java:1872)
	at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:2167)
	at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1846)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:121)
	at java.base/java.lang.Thread.run(Unknown Source)

这是因为K8S集群中的这个Jenkins的pod无法ping通域名导致的

排查

使用命令查看的确无法ping通域名。
百度了一下解决方法。
可以根据自己的需要参看一下两个博客博客可以参考：https://www.cnblogs.com/sky-cheng/p/14254871.html
https://blog.csdn.net/hqing159/article/details/109530203?utm_medium=distribute.pc_relevant.none-task-blog-2%7Edefault%7EBlogCommendFromBaidu%7Edefault-5.no_search_link&depth_1-utm_source=distribute.pc_relevant.none-task-blog-2%7Edefault%7EBlogCommendFromBaidu%7Edefault-5.no_search_link

解决

查看/etc/resolv.conf文件

[root@k8s-master ~]# kubectl exec -it jenkins-0 -n kube-ops /bin/bash
bash-5.1$ cat /etc/resolv.conf 
nameserver 10.1.0.10
search kube-ops.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
bash-5.1$ 

可以看到dns服务器IP为10.1.0.10,我们查看下系统的coredns pod容器信息

[root@k8s-master ~]# kubectl get pods -n kube-system -o wide |grep coredns
coredns-9d85f5447-5jt6w                    1/1     Running   5          9d      10.244.36.68     k8s-node1    <none>           <none>
coredns-9d85f5447-ghkhm                    1/1     Running   5          9d      10.244.169.132   k8s-node2    <none>           <none>

可以看到两个coredns pod位于两个node节点上，并且状态是running，正常

我们再进一步查看dns service信息

[root@k8s-master ~]# kubectl get svc -n kube-system -o wide
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE   SELECTOR
kube-dns   ClusterIP   10.1.0.10    <none>        53/UDP,53/TCP,9153/TCP   18d   k8s-app=kube-dns

kube-dns服务的IP正是10.1.0.10，说明pod是通过kube-dns 服务来解析域名的，现在的问题是POD无法与kube-dns通信呢？还是coredns本身域名解析有问题呢，需要进一步来确认kube-dns 服务后端正确绑定了coredns容器，查看endpoint来确认

[root@k8s-master ~]# kubectl get endpoints -n kube-system -o wide|grep kube-dns
kube-dns                  10.244.169.132:53,10.244.36.68:53,10.244.169.132:53 + 3 more...   18d
[root@k8s-master ~]# 

以看到kube-dns后端正确的绑定了两个coredns pod的IP。

我们再将目标pod中的nameserver 的ip地址改为coredns pod的IP地址，绕过kube-dns服务，直接与coredns pod通信
需要将jenkins的pod中/etc/resolv.conf的内容修改成如下

nameserver 10.244.36.68
#nameserver 10.1.0.10
search kube-ops.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
bash-5.1$ 

10.244.36.68：为coredns pod其中一个的IP

但是改pod中的resolv.conf文件只有读取的权限。所以现在需要获取这个文件的可编辑权限。

查看这个pod在哪个node节点上。

[root@k8s-master ~]# kubectl get pods -n kube-ops -o wide
NAME        READY   STATUS    RESTARTS   AGE   IP             NODE        NOMINATED NODE   READINESS GATES
jenkins-0   1/1     Running   6          9d    10.244.36.70   k8s-node1   <none>           <none>
[root@k8s-master ~]# kubectl describe pod jenkins-0 -n kube-ops

获取到docker容器的ID。

在那个node节点服务器上使用docker命令以root用户权限进入pod

[root@k8s-node1 ~]# docker exec -it -u root '049f26d4075ed04b29475d191b2fb1ef8bee80d0487a8f7e99964de344b2cb17' /bin/bash
bash-5.1# vi /etc/resolv.conf 

修改完成后保存退出


重新安装插件，没问题。