定制 k8s 的 DNS 服务
深度定制kubernetes的DNS服务。
Table of Contents
Prepare the Dockerfile for dnsmasq image
Prepare our dnsmasq pod, i.e. custom name server
Integrate our name server with kube-dns: create the ConfigMap
Add entries to our DNS and test it out
Introduction
In some cases the k8s default DNS service does not match our needs, for example an Oracle RAC database has to resolve the public/private/vip/scan IPs for its nodes. This document describes how we can customize the DNS service including integrating our own DNS server.
This documents assumes you know the basic concepts of k8s and DNS.
This documents assumes you have read this page: multiple network interfaces for a pod, because we will touch this in later section.
k8s 1.9+ supports better DNS customization, but our environment is still 1.8 compatible, and uses kube-dns instead of CoreDNS.
Basics of k8s DNS
The default DNS record of a k8s pod is: a-b-c-d.sub-domain.my-namespace.pod.cluster.local
a-b-c-d is the IP string of the pod. For example, a pod my-pod with IP address 10.244.0.2 in default namespace, will have an A record of "10-244-0-2.default.pod.cluster.local". If the pod specifies a sub domain, such as ora-subdomain, it will be "10-244-0-2.ora-subdomain.default.pod.cluster.local". There is no pod name or hostname record in the DNS server. However, if we create a headless service (services without a clusterIP) with the name exactly same as the sub-domain, we will have a record "my-pod.ora-subdomin.default.svc.cluster.local" (note the suffix "pod.cluster.local" becomes "svc.cluster.local"). This is also where the StatefulSet's stable network identifier comes from, in that case, in addition to each pod that belongs to the StatefulSet, the service itself (i.e. "sub-domain.my-namespace.svc.cluster.local") will resolve to the IPs (a set) of the pods.
A service's DNS record is always <svc-name>.<namespace>.svc.cluster.local.
kube-dns customization
kube-dns can be extended to support additional DNS name servers. For instance, consider the following config map:
1 2 3 4 5 6 7 8 9 10 |
|
The custom configurations are stubDomains and upstreamNameservers.
First, let's see what is the resolve process when these configurations does NOT exist:
- The name resolves according to the default rule of k8s DNS, as stated in "Basics of k8s DNS" section.
- If the name is not found (e.g. names with example.com domain ), it is forwarded to upstream name servers that inherited from the node.
This means the k8s node's /etc/hosts and /etc/resolv.conf functions inside the pod containers.
Now what if stubDomains and/or upstramNameservers are specified? The process is:
- For names with the cluster suffix, i.e. ".cluster.local", the request is sent to kube-dns.
- For names with the stub domain suffix, i.e. ".example.com", the request is sent to the configured custom DNS server, listening at 1.2.3.4.
- For names without a matching suffix, for example "github.com", the request is forwarded to the upstream DNS, in this case 8.8.8.8 and 8.8.4.4 (These are public DNS servers of Google company).
Important: If upstreamNameservers is specified, we'll lost the node's resolving. A natural solution would be copying the node's resolve to the custom name server.
Setup our own name server
So what will be our custom name server? dnsmasq is an easy-to-configure, widely used one (kube-dns itself is using it). In this section, we will setup our own dnsmasq server from scratch.
Prepare the Dockerfile for dnsmasq image
First write the Dockerfile
Dockerfile
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
|
Note that we explicitly specify "resolv-file=resolv.dnsmasq.conf" to replace the default /etc/resolv.conf and thus bypass the k8s pod default resolving. We copied the node host's "nameserver"(s) and "search" domains, so that our custom name server can inherit the node's resolving ability. What we copied here is actually the kube-dns pod's /etc/resolv.conf, which in turn copied (partly) the file of the node:
|
The /etc/resolv.conf specified the "upstream" name servers of kube-dns, and explains how it inherited from the node's name resolving.
The run.sh specified in the CMD field of Dockerfile is:
|
put it to the same path of Dockerfile and build the image:
|
Prepare our dnsmasq pod, i.e. custom name server
Yaml work:
dnsmasq.yaml
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
|
Then create the pod and service:
|
Integrate our name server with kube-dns: create the ConfigMap
After the pod mydns starts, use "kubectl logs.." to see if it starts successfully, and then get its IP:
|
Now write our ConfigMap yaml file using this IP:
cm.yaml
1 2 3 4 5 6 7 8 9 10 |
|
Apply it
|
Add entries to our DNS and test it out
As we stated previously, we assume you have read Multiple Network Interfaces for a k8s pod, now we are about to test our DNS server with some simple pod that enables multiple network interfaces. The testing pod is an ole7 container doing nothing but sleeping:
ole7.yaml
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
|
Create it by:
|
The pod has two IP addresses, one is the flannel default (displays in "kubectl get pod ...") and one is in file /temp/PRIV_IP in the container.
Once the pod is initiated, we can test it out. In the following example, I got both public and private IPs for pod "ole7", added them to our custom name server (by redirecting the echo output to /dev/tcp/mydns-service/12345) dynamically, giving them names. Next, I pinged all the names with or without the domain suffix, I also ping some hosts that out of the k8s cluster, e.g. vm09xxl and home.cn.my.com, and all the commands are successful.
Test commands
|
We add two names for a single pod IP, e.g "ole.exmple.com" and "ole" to addn-hosts in our name server. When a name resolving request is sent to kube-dns, if the name has domain suffix i.e. example.com, the request is forwarded to the custom name server specified in the "stubDomains" attribute of the ConfigMap. If the name does not contain suffix, the request is firstly handled by kube-dns by adding the "...cluster.local" suffix, if the name could not be found in k8s cluster, kube-dns will forward it to "upstreamNameservers", in our case, this is the same custom name server, and the name is found in addn-hosts. On the other hand, a name such as "vm09xxl" follows the same resolving path, but cannot be found in addn-hosts, so the name server consults its /etc/resolv.dnsmasq.conf and forwards the request to its upstream name servers specified there.
Issues
Pod CIDR
Our current pod_network_cidr="10.244.0.0/16", the IP(s) allocated to pods conflict to those of our VMs'. For example, 10.244.0.25 is somevm1.cn.my.com, 10.244.0.26 is somevm2.cn.my.com. This may cause confusion in our name resolving. We may need to change pod_network_cidr to another value.
References
https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/
https://kubernetes.io/blog/2017/04/configuring-private-dns-zones-upstream-nameservers-kubernetes/
https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/
更多推荐
所有评论(0)