k8s pod访问集群外域名原理以及使用了systemd-resolved的不同情况

1、不同情况下的linux主机访问外部域名原理

没有使用systemd-resolved的linux主机上访问外部域名一般是按照以下步骤来的：

  • 从dns缓存里查找域名与ip的映射关系

  • 从/etc/hosts里查找域名与ip的映射关系

  • 从/etc/resolv.conf里查找dns server，并发起解析请求

    /etc/resolv.conf的内容一般如下：
    nameserver 8.8.8.8

使用systemd-resolved的linux主机上访问外部域名一般是按照以下步骤来的：

  • 从dns缓存里查找域名与ip的映射关系

  • 从/etc/hosts里查找域名与ip的映射关系

  • 将dns解析请求发给本地systemd-resolved，由其去代理处理，因为systemd-resolved修改了 /etc/resolv.conf，使得本地解析请求全部发到127.0.0.1:53

    此时/etc/resolv.conf的内容一般如下：

# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0 trust-ad

  • 然后systemd-resolved再根据/run/systemd/resolve/resolv.conf里面的dns server去发起请求

    /run/systemd/resolve/resolv.conf记录的就是真正的后端dns server

cat /run/systemd/resolve/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 8.8.8.8
nameserver 4.4.4.4

2、pod内访问集群内service域名

当pod启动的时候，一般用的是dnsPolicy: ClusterFirst，此时就会将pod的/etc/resolv.conf改为集群内coredns的地址，此时将解析请求发给coredns，由其代理处理：

集群内coredns的service ip：

kubectl get svc -n kube-system

NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   14d

pod内的/etc/resolv.conf在启动的时候被指定dns server为coredns service ip：

# cat /etc/resolv.conf in pod
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

3、pod内访问集群外service域名

  • 使用dnsPolicy: ClusterFirst时：

    pod内的/etc/resolv.conf在启动的时候被指定dns server为coredns service ip。
    coredns的默认配置如下，此时如果用在集群内找不到这个service域名，就会用forward去转发请求，此时默认配置的是使用coredns内的 /etc/resolv.conf文件里的dns server

coredns default config：

    .:53 {
        log
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }

  • 使用dnsPolicy: Default时：

    这种方式其实是让 kubelet 来决定使用何种 DNS 策略。而 kubelet 默认的方式，就是使用宿主机的 /etc/resolv.conf

    简述： pod将dns代理到coredns，coredns使用kubelet的resolv指定的conf里面的内容来解析集群外的ip

4、coredns pod内的/etc/resolv.conf为什么有时跟主机/etc/resolv.conf不一致

场景：

当主机使用systemd-resolved来代理dns解析请求的时候，此时coredns pod内的/etc/resolv.conf跟主机/etc/resolv.conf不一致。

coredns pod内的/etc/resolv.conf：

cat /run/systemd/resolve/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 8.8.8.8
nameserver 4.4.4.4

主机/etc/resolv.conf：

# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0 trust-ad

coredns pod内的/etc/resolv.conf为什么有时跟主机/etc/resolv.conf不一致的原因：

coredns的pod使用的是dnsPolicy: Default，此时就会使用kubelet指定的resolvConf的地址，默认是/etc/resolv.conf，但是当主机使用了systemd-resolved，则kubelet的的resolvConf变成了：resolvConf: /run/systemd/resolve/resolv.conf，也就是systemd-resolved存放真实后端dns server的文件路径，所以其实coredns的pod使用的是/run/systemd/resolve/resolv.conf里的真实后端dns server

使用了systemd-resolved的主机，kubelet使用/run/systemd/resolve/resolv.conf，而不用/etc/resolv.conf的原因：

  • 如果coredns也是用/etc/resolv.conf，则集群里的dns解析请求都要代理到systemd-resolved，如果systemd-resolved挂了或者更新，那上层k8s集群里的dns解析也全部受到影响

  • 这里面会有循环依赖的问题，参考coredns的官方文档解析：

    Troubleshooting Loops In Kubernetes Clusters

    A common cause of forwarding loops in Kubernetes clusters is an interaction with a local DNS cache on the host node (e.g. systemd-resolved). For example, in certain configurations systemd-resolved will put the loopback address 127.0.0.53 as a nameserver into /etc/resolv.conf. Kubernetes (via kubelet) by default will pass this /etc/resolv.conf file to all Pods using the default dnsPolicy rendering them unable to make DNS lookups (this includes CoreDNS Pods). CoreDNS uses this /etc/resolv.conf as a list of upstreams to forward requests to. Since it contains a loopback address, CoreDNS ends up forwarding requests to itself.

    简述： coredns转到127.0.0.53，此时源目ip都是自己，自己在给自己转，就会有循环问题

5、在node上如何访问集群内的service域名

  • 通用方法：直接修改网卡interface的配置文件，在里面配上DNS的解析server

  • 不通用方法：linux没有使用systemd-resolved时：

    在/etc/resolv.conf里加入coredns的service ip

  • 不通用方法：linux使用systemd-resolved时：

    • 在/etc/systemd/resolved.conf里加入coredns的service ip
      [Resolve]
      DNS=10.96.0.10
      #FallbackDNS=
      #Domains=
      #LLMNR=no
      #MulticastDNS=no
      #DNSSEC=no
      #DNSOverTLS=no
      #Cache=no-negative
      #DNSStubListener=yes
      #ReadEtcHosts=yes
    • systemctl restart systemd-resolved.service
    • systemd-resolve --status查看结果
      Global
      LLMNR setting: no
      MulticastDNS setting: no
      DNSOverTLS setting: no
      DNSSEC setting: no
      DNSSEC supported: no
      DNS Servers: 10.96.0.10
      DNSSEC NTA: 10.in-addr.arpa
      16.172.in-addr.arpa
      168.192.in-addr.arpa
      17.172.in-addr.arpa
      18.172.in-addr.arpa
      19.172.in-addr.arpa
      20.172.in-addr.arpa
      21.172.in-addr.arpa
      …
      …

  • 注意：不可以直接改/etc/resolv.conf，否则重启后丢失配置，因为此时/etc/resolv.conf被systemd-resolved接管，每次重启由其来生成其中的内容

example：

# 指定使用coredns来解析集群内service：
root@:/home/ubuntu# nslookup vmselect-example-vmcluster-persistent.default.svc.cluster.local 10.96.0.10
Server:        10.96.0.10
Address:    10.96.0.10#53

Name:    vmselect-example-vmcluster-persistent.default.svc.cluster.local
Address: 10.244.0.5
Name:    vmselect-example-vmcluster-persistent.default.svc.cluster.local
Address: 10.244.1.3
Name:    vmselect-example-vmcluster-persistent.default.svc.cluster.local
Address: 10.244.1.4

# /etc/resolv.conf加入：
nameserver 10.96.0.10

# node上直接访问集群内service域名，此时会转到coredns去解析
root@:/home/ubuntu# curl vmselect-example-vmcluster-persistent.default.svc.cluster.local:8481/metrics
...
flag{name="promscrape.suppressScrapeErrors", value="false", is_set="false"} 1
flag{name="promscrape.suppressScrapeErrorsDelay", value="0s", is_set="false"} 1
flag{name="promscrape.yandexcloudSDCheckInterval", value="30s", is_set="false"} 1
flag{name="pushmetrics.extraLabel", value="", is_set="false"} 1
flag{name="pushmetrics.interval", value="10s", is_set="false"} 1
flag{name="pushmetrics.url", value="secret", is_set="false"} 1
flag{name="replicationFactor", value="1", is_set="false"} 1
flag{name="search.cacheTimestampOffset", value="5m0s", is_set="false"} 1
flag{name="search.denyPartialResponse", value="false", is_set="false"} 1
...
...

6、使用了systemd-resolved的主机，如果关闭systemd-resolved则机器的dns解析都会不同，即使主机能通后端dns server

ubuntu@:~$ sudo systemctl stop systemd-resolved.service
ubuntu@:~$ nslookup www.baidu.com
^C
ubuntu@:~$ dig www.baidu.com
^C
ubuntu@:~$ host www.baidu.com
^C

7、主机如何修改dns server

  • 没有systemd-resolved，直接修改/etc/resolv.conf
  • 如果机器装了systemd-resolved，那么就不可以直接改/etc/resolv.conf，则改法如下：

以加上8.8.8.8为例

root@:/home/ubuntu# cat /etc/systemd/resolved.conf
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See resolved.conf(5) for details

[Resolve]
DNS=8.8.8.8
#FallbackDNS=
#Domains=
#LLMNR=no
#MulticastDNS=no
#DNSSEC=no
#DNSOverTLS=no
#Cache=no-negative
#DNSStubListener=yes
#ReadEtcHosts=yes

systemctl restart systemd-resolved.service

root@:/home/ubuntu# systemd-resolve --status
Global
       LLMNR setting: no
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 8.8.8.8
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      ...
                      ...
# kubernetes # linux # docker
Logo
K8S/Kubernetes

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐

【深度】阿里巴巴万级规模 K8s 集群全局高可用体系之美

作者 |  韩堂、柘远、沉醉来源 | 阿里巴巴云原生公众号​前言台湾作家林清玄在接受记者采访的时候，如此评价自己 30 多年写作生涯：“第一个十年我才华横溢，‘贼光闪现’，令周边黯然失色；第二个十年，我终于‘宝光现形’，不再去抢风头，反而与身边的美丽相得益彰；进入第三个十年，繁华落尽见真醇，我进入了‘醇光初现’的阶段，真正体味到了境界之美”。​长夜有穷，真水无香。领略过了 K8s“身在江

avatar K8S/Kubernetes

如何基于 K8s 构建下一代 DevOps 平台？

作者 | 孙健波（天元）导读：当前云原生 DevOps 体系现状如何？面临哪些挑战？如何通过 OAM 解决云原生 DevOps 场景下的诸多问题？云原生开发应用模型 OAM(Open Application Model) 社区核心成员孙健波将为大家一一解答，并分享如何基于 OAM 和 Kubernetes 打造无限能力的下一代 DevOps 平台。什么是 DevOps？为什么基于 Kub

avatar K8S/Kubernetes

k8s 火了！

2020，上云之年，产品云端化成为一种趋势。在一线城市，很多公司都已经构建了自己的私有云环境，比如阿里云、网易云、华为云等。而Kubernetes 作为基于容器编排领域的王者，具备扩展...

avatar K8S/Kubernetes