Merge pull request #9865 from pinewall/submit-tech20180609

submit tech20180609 Anatomy of a Linux DNS Lookup – Part I
这个 CI 没通过是因为原文的文件名中包含异常字符。
This commit is contained in:
Xingyu.Wang 2018-08-19 12:11:02 +08:00 committed by GitHub
commit bd34522f6e
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 279 additions and 285 deletions

View File

@ -1,285 +0,0 @@
pinewall is translating
Anatomy of a Linux DNS Lookup Part I
============================================================
Since I [work][3] [a][4] [lot][5] [with][6] [clustered][7] [VMs][8], Ive ended up spending a lot of time trying to figure out how [DNS lookups][9] work. I applied fixes to my problems from StackOverflow without really understanding why they work (or dont work) for some time.
Eventually I got fed up with this and decided to figure out how it all hangs together. I couldnt find a complete guide for this anywhere online, and talking to colleagues they didnt know of any (or really what happens in detail)
So Im writing the guide myself.
_If youre looking for Part II, click [here][1]_
Turns out theres quite a bit in the phrase Linux does a DNS lookup
* * *
![linux-dns-0](https://zwischenzugs.files.wordpress.com/2018/06/linux-dns-0.png?w=121)
_“How hard can it be?”_
* * *
These posts are intended to break down how a program decides how it gets an IP address on a Linux host, and the components that can get involved. Without understanding how these pieces fit together, debugging and fixing problems with (for example) `dnsmasq`, `vagrant landrush`, or `resolvconf` can be utterly bewildering.
Its also a valuable illustration of how something so simple can get so very complex over time. Ive looked at over a dozen different technologies and their archaeologies so far while trying to grok whats going on.
I even wrote some [automation code][10] to allow me to experiment in a VM. Contributions/corrections are welcome.
Note that this is not a post on how DNS works. This is about everything up to the call to the actual DNS server thats configured on a linux host (assuming it even calls a DNS server as youll see, it need not), and how it might find out which one to go to, or how it gets the IP some other way.
* * *
### 1) There is no such thing as a DNS Lookup call
* * *
![linux-dns-1](https://zwischenzugs.files.wordpress.com/2018/06/linux-dns-1.png?w=121)
_This is NOT how it works_
* * *
The first thing to grasp is that there is no single method of getting a DNS lookup done on Linux. Its not a core system call with a clean interface.
There is, however, a standard C library call called which many programs use: `[getaddrinfo][2]`. But not all applications use this!
Lets just take two simple standard programs: `ping` and `host`:
```
root@linuxdns1:~# ping -c1 bbc.co.uk | head -1
PING bbc.co.uk (151.101.192.81) 56(84) bytes of data.
```
```
root@linuxdns1:~# host bbc.co.uk | head -1
bbc.co.uk has address 151.101.192.81
```
They both get the same result, so they must be doing the same thing, right?
Wrong.
Heres the files that `ping` looks at on my host that are relevant to DNS:
```
root@linuxdns1:~# strace -e trace=open -f ping -c1 google.com
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib/x86_64-linux-gnu/libcap.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
open("/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 4
open("/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 4
open("/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 4
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 4
open("/lib/x86_64-linux-gnu/libnss_files.so.2", O_RDONLY|O_CLOEXEC) = 4
open("/etc/host.conf", O_RDONLY|O_CLOEXEC) = 4
open("/etc/hosts", O_RDONLY|O_CLOEXEC)  = 4
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 4
open("/lib/x86_64-linux-gnu/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = 4
open("/lib/x86_64-linux-gnu/libresolv.so.2", O_RDONLY|O_CLOEXEC) = 4
PING google.com (216.58.204.46) 56(84) bytes of data.
open("/etc/hosts", O_RDONLY|O_CLOEXEC)  = 4
64 bytes from lhr25s12-in-f14.1e100.net (216.58.204.46): icmp_seq=1 ttl=63 time=13.0 ms
[...]
```
and the same for `host`:
```
$ strace -e trace=open -f host google.com
[...]
[pid  9869] open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/libdst.cat", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid  9869] open("/usr/share/locale/en/libdst.cat", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid  9869] open("/usr/share/locale/en/LC_MESSAGES/libdst.cat", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid  9869] open("/usr/lib/ssl/openssl.cnf", O_RDONLY) = 6
[pid  9869] open("/usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines/libgost.so", O_RDONLY|O_CLOEXEC) = 6[pid  9869] open("/etc/resolv.conf", O_RDONLY) = 6
google.com has address 216.58.204.46
[...]
```
You can see that while my `ping` looks at `nsswitch.conf`, `host` does not. And they both look at `/etc/resolv.conf`.
Were going to take these two `.conf` files in turn.
* * *
### 2) NSSwitch, and `/etc/nsswitch.conf`
Weve established that applications can do what they like when they decide which DNS server to go to. Many apps (like `ping`) above can refer (depending on the implementation (*)) to NSSwitch via its config file `/etc/nsswitch.conf`.
###### (*) Theres a surprising degree of variation in
ping implementations. Thats a rabbit-hole I
_didnt_  want to get lost in.
NSSwitch is not just for DNS lookups. Its also used for passwords and user lookup information (for example).
NSSwitch was originally created as part of the Solaris OS to allow applications to not have to hard-code which file or service they look these things up on, but defer them to this other configurable centralised place they didnt have to worry about.
Heres my `nsswitch.conf`:
```
passwd:         compat
group:          compat
shadow:         compat
gshadow:        files
hosts: files dns myhostname
networks:       files
protocols:      db files
services:       db files
ethers:         db files
rpc:            db files
netgroup:       nis
```
The hosts line is the one were interested in. Weve shown that `ping` cares about `nsswitch.conf` so lets fiddle with it and see how we can mess with `ping`.
* ### Set `nsswitch.conf` to only look at files
If you set the `hosts` line in `nsswitch.conf` to be just `files`:
`hosts: files`
Then a `ping` to google.com will now fail:
```
$ ping -c1 google.com
ping: unknown host google.com
```
but `localhost` still works:
```
$ ping -c1 localhost
PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.039 ms
```
and using `host` still works fine:
```
$ host google.com
google.com has address 216.58.206.110
```
since, as we saw, it doesnt care about `nsswitch.conf`
* ### Set `nsswitch.conf` to only look at dns
If you set the `hosts` line in `nsswitch.conf` to be just dns:
`hosts: dns`
Then a `ping` to google.com will now succeed again:
```
$ ping -c1 google.com
PING google.com (216.58.198.174) 56(84) bytes of data.
64 bytes from lhr25s10-in-f174.1e100.net (216.58.198.174): icmp_seq=1 ttl=63 time=8.01 ms
```
But `localhost` is not found this time:
```
$ ping -c1 localhost
ping: unknown host localhost
```
Heres a diagram of whats going on with NSSwitch by default wrt `hosts` lookup:
* * *
![linux-dns-2 (1)](https://zwischenzugs.files.wordpress.com/2018/06/linux-dns-2-11.png?w=525)
_My default `hosts:` configuration in `nsswitch.conf`_
* * *
### 3) `/etc/resolv.conf`
Weve seen now that `host` and `ping` both look at this `/etc/resolv.conf` file.
Heres what my `/etc/resolv.conf` looks like:
```
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 10.0.2.3
```
Ignore the first two lines well come back to those (they are significant, but youre not ready for that ball of wool yet).
The `nameserver` lines specify the DNS servers to look up the host for.
If you hash out that line:
```
#nameserver 10.0.2.3
```
and run:
```
$ ping -c1 google.com
ping: unknown host google.com
```
it fails, because theres no nameserver to go to (*).
###### * Another rabbit hole: `host` appears to fall back to
127.0.0.1:53 if theres no nameserver specified.
This file takes other options too. For example, if you add this line to the `resolv.conf` file:
```
search com
```
and then `ping google` (sic)
```
$ ping google
PING google.com (216.58.204.14) 56(84) bytes of data.
```
it will try the `.com` domain automatically for you.
### End of Part I
Thats the end of Part I. The next part will start by looking at how that resolv.conf gets created and updated.
Heres what you covered above:
* Theres no DNS lookup call in the OS
* Different programs figure out the IP of an address in different ways
* For example, `ping` uses `nsswitch`, which in turn uses (or can use) `/etc/hosts`, `/etc/resolv.conf` and its own hostname to get the result
* `/etc/resolv.conf` helps decide:
* which addresses get called
* which DNS server to look up
If you thought that was complicated, buckle up…
--------------------------------------------------------------------------------
via: https://zwischenzugs.com/2018/06/08/anatomy-of-a-linux-dns-lookup-part-i/
作者:[dmatech][a]
译者:[译者ID](https://github.com/译者ID)
校对:[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译,[Linux中国](https://linux.cn/) 荣誉推出
[a]:https://twitter.com/dmatech2
[1]:https://zwischenzugs.com/2018/06/18/anatomy-of-a-linux-dns-lookup-part-ii/
[2]:http://man7.org/linux/man-pages/man3/getaddrinfo.3.html
[3]:https://zwischenzugs.com/2017/10/31/a-complete-chef-infrastructure-on-your-laptop/
[4]:https://zwischenzugs.com/2017/03/04/a-complete-openshift-cluster-on-vagrant-step-by-step/
[5]:https://zwischenzugs.com/2017/03/04/migrating-an-openshift-etcd-cluster/
[6]:https://zwischenzugs.com/2017/03/04/1-minute-multi-node-vm-setup/
[7]:https://zwischenzugs.com/2017/03/18/clustered-vm-testing-how-to/
[8]:https://zwischenzugs.com/2017/10/27/ten-things-i-wish-id-known-before-using-vagrant/
[9]:https://zwischenzugs.com/2017/10/21/openshift-3-6-dns-in-pictures/
[10]:https://github.com/ianmiell/shutit-linux-dns/blob/master/linux_dns.py

View File

@ -0,0 +1,279 @@
Linux DNS 查询剖析 第一部分
============================================================
我经常与虚拟机集群打交道([文1][3], [文2][4], [文3][5], [文4][6], [文5][7], [文6][8]),期间花费了大量时间试图掌握 [DNS 查询][9]的工作原理。遇到问题时,我有时只是不求甚解的使用 StackOverflow 上的“解决方案”;甚至那些“解决方案”有时并不工作。
最终我决定改变这种情况,决定一并找出所有问题的原因。我没有在网上找到完整手册或类似的其它东西,我问过一些同事,他们也是如此。
既然如此,我开始自己写这样的手册。
_如果你在找第二部分, 点击 [这里][1]_
结果发现“Linux 执行一次 DNS 查询”的背后有相当多的工作。
* * *
![linux-dns-0](https://zwischenzugs.files.wordpress.com/2018/06/linux-dns-0.png?w=121)
_“究竟有多难呢”_
* * *
本系列文章试图将 Linux 主机上程序获取(域名对应的) IP 地址的过程及期间涉及的组件进行分块剖析。如果不理解这些块的协同工作方式,调试并解决 `dnsmasq``vagrant landrush` 和 `resolvconf` 等相关的问题会让人感到眼花缭乱。
同时这也是一份有价值的说明,指出原本很简单的东西可以如何随着时间的推移变得相当复杂。在弄清楚 DNS 查询的原理的过程中,我了解了大量不同的技术及其发展历程。
我甚至编写了一些[自动化脚本][10],可以让我在虚拟机中进行实验。欢迎读者参与贡献或勘误。
请注意本系列主题并不是“DNS 工作原理”,而是与查询 Linux 主机配置的真实 DNS 服务器(这里假设查询了 DNS 服务器,但后面你会看到有时并不需要查询)相关的内容,以及如何确定使用哪个查询结果,或者何时使用其它方式确定 IP 地址。
* * *
### 1) 其实并没有名为“DNS 查询”的系统调用
* * *
![linux-dns-1](https://zwischenzugs.files.wordpress.com/2018/06/linux-dns-1.png?w=121)
_工作方式并非如此_
* * *
首先要了解的一点是Linux 上并没有一个单独的方法可以完成 DNS 查询工作;至少没有如此<ruby>明确接口<rt>clean interface</rt></ruby>的核心<ruby>系统调用<rt>system call</rt></ruby>
有一个标准 C 库函数调用 `[getaddrinfo][2]`,不少程序使用了该调用;但不是所有程序或应用都使用该调用!
我们只考虑两个简单的标准程序:`ping` 和 `host`
```
root@linuxdns1:~# ping -c1 bbc.co.uk | head -1
PING bbc.co.uk (151.101.192.81) 56(84) bytes of data.
```
```
root@linuxdns1:~# host bbc.co.uk | head -1
bbc.co.uk has address 151.101.192.81
```
对于同一个域名,两个程序得到的 IP 地址是相同的;那么它们是使用同样的方法得到结果的吧?
事实并非如此。
下面文件给出了我主机上 `ping` 对应的 DNS 相关的系统调用:
```
root@linuxdns1:~# strace -e trace=open -f ping -c1 google.com
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib/x86_64-linux-gnu/libcap.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
open("/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 4
open("/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 4
open("/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 4
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 4
open("/lib/x86_64-linux-gnu/libnss_files.so.2", O_RDONLY|O_CLOEXEC) = 4
open("/etc/host.conf", O_RDONLY|O_CLOEXEC) = 4
open("/etc/hosts", O_RDONLY|O_CLOEXEC)  = 4
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 4
open("/lib/x86_64-linux-gnu/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = 4
open("/lib/x86_64-linux-gnu/libresolv.so.2", O_RDONLY|O_CLOEXEC) = 4
PING google.com (216.58.204.46) 56(84) bytes of data.
open("/etc/hosts", O_RDONLY|O_CLOEXEC)  = 4
64 bytes from lhr25s12-in-f14.1e100.net (216.58.204.46): icmp_seq=1 ttl=63 time=13.0 ms
[...]
```
下面是 `host` 对应的系统调用:
```
$ strace -e trace=open -f host google.com
[...]
[pid  9869] open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/libdst.cat", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid  9869] open("/usr/share/locale/en/libdst.cat", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid  9869] open("/usr/share/locale/en/LC_MESSAGES/libdst.cat", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid  9869] open("/usr/lib/ssl/openssl.cnf", O_RDONLY) = 6
[pid  9869] open("/usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines/libgost.so", O_RDONLY|O_CLOEXEC) = 6[pid  9869] open("/etc/resolv.conf", O_RDONLY) = 6
google.com has address 216.58.204.46
[...]
```
可以看出 `ping` 打开了 `nsswitch.conf` 文件,但 `host` 没有;但两个程序都打开了 `/etc/resolv.conf` 文件。
下面我们依次查看这两个 `.conf` 扩展名的文件。
* * *
### 2) NSSwitch 与 `/etc/nsswitch.conf`
我们已经确认应用可以自主决定选用哪个 DNS 服务器。很多应用(例如 `ping`)通过配置文件 `/etc/nsswitch.conf` (根据具体实现 (*))参考 NSSwitch 完成选择。
###### (*) ping 实现的变种之多令人惊叹。我 _不_ 希望在这里讨论过多。
NSSwitch 不仅用于 DNS 查询,例如,还用于密码与用户信息查询。
NSSwitch 最初是 Solaris OS 的一部分,可以让应用无需将查询所需的文件或服务硬编码,而是在其它集中式的、无需应用开发人员管理的配置文件中找到。
下面是我的 `nsswitch.conf`
```
passwd:         compat
group:          compat
shadow:         compat
gshadow:        files
hosts: files dns myhostname
networks:       files
protocols:      db files
services:       db files
ethers:         db files
rpc:            db files
netgroup:       nis
```
我们需要关注的是 `hosts` 行。我们知道 `ping` 用到 `nsswitch.conf` 文件,那么我们修改这个文件(的 `hosts` 行),看看能够如何影响 `ping`
* ### 修改 `nsswitch.conf` `hosts` 行仅保留 `files`
如果你修改 `nsswitch.conf`,将 `hosts` 行仅保留 `files`
`hosts: files`
此时, `ping` 无法获取 google.com 对应的 IP 地址:
```
$ ping -c1 google.com
ping: unknown host google.com
```
但 `localhost` 的解析不受影响:
```
$ ping -c1 localhost
PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.039 ms
```
此外,`host` 命令正常返回:
```
$ host google.com
google.com has address 216.58.206.110
```
毕竟如我们之前看到的那样,`host` 不受 `nsswitch.conf` 影响。
* ### 修改 `nsswitch.conf` `hosts` 行仅保留 `dns`
如果你修改 `nsswitch.conf`,将 `hosts` 行仅保留 `dns`
`hosts: dns`
此时google.com 的解析恢复正常:
```
$ ping -c1 google.com
PING google.com (216.58.198.174) 56(84) bytes of data.
64 bytes from lhr25s10-in-f174.1e100.net (216.58.198.174): icmp_seq=1 ttl=63 time=8.01 ms
```
`localhost` 无法解析:
```
$ ping -c1 localhost
ping: unknown host localhost
```
下图给出默认 NSSwitch 中 `hosts` 行对应的查询逻辑:
* * *
![linux-dns-2 (1)](https://zwischenzugs.files.wordpress.com/2018/06/linux-dns-2-11.png?w=525)
_我的 `hosts:` 配置是 `nsswitch.conf` 给出的默认值_
* * *
### 3) `/etc/resolv.conf`
我们已经知道 `host``ping` 都使用 `/etc/resolv.conf` 文件。
下面给出我主机的 `/etc/resolv.conf` 文件内容:
```
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 10.0.2.3
```
先忽略前两行,后面我们会回过头来看这部分(它们很重要,但你还需要一些知识储备)。
其中 `nameserver` 行指定了查询用到的 DNS 服务器。
如果将该行注释掉:
```
#nameserver 10.0.2.3
```
再次运行:
```
$ ping -c1 google.com
ping: unknown host google.com
```
解析失败了,这是因为没有可用的 nameserver (*)。
###### * 另一个需要注意的地方: `host` 在没有指定 nameserver 的情况下会尝试 127.0.0.1:53。
该文件中还可以使用其它选项。例如,你可以在 `resolv.conf` 文件中增加如下行:
```
search com
```
然后执行 `ping google` (不写 `.com`
```
$ ping google
PING google.com (216.58.204.14) 56(84) bytes of data.
```
程序会自动为你尝试 `.com` 域。
### 第一部分总结
第一部分到此结束,下一部分我们会了解 `resolv.conf` 文件是如何创建和更新的。
下面总结第一部分涵盖的内容:
* 操作系统中并不存在“DNS 查询”这个系统调用
* 不同程序可能采用不同的策略获取名字对应的 IP 地址
* 例如, `ping` 使用 `nsswitch`,后者进而使用(或可以使用) `/etc/hosts``/etc/resolv.conf` 以及主机名得到解析结果
* `/etc/resolv.conf` 用于决定:
* 查询什么地址LCTT 译注:这里可能指 search 带来的影响)
* 使用什么 DNS 服务器执行查询
如果你曾认为 DNS 查询很复杂,请跟随这个系列学习吧。
--------------------------------------------------------------------------------
via: https://zwischenzugs.com/2018/06/08/anatomy-of-a-linux-dns-lookup-part-i/
作者:[dmatech][a]
译者:[pinewall](https://github.com/pinewall)
校对:[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译,[Linux中国](https://linux.cn/) 荣誉推出
[a]:https://twitter.com/dmatech2
[1]:https://zwischenzugs.com/2018/06/18/anatomy-of-a-linux-dns-lookup-part-ii/
[2]:http://man7.org/linux/man-pages/man3/getaddrinfo.3.html
[3]:https://zwischenzugs.com/2017/10/31/a-complete-chef-infrastructure-on-your-laptop/
[4]:https://zwischenzugs.com/2017/03/04/a-complete-openshift-cluster-on-vagrant-step-by-step/
[5]:https://zwischenzugs.com/2017/03/04/migrating-an-openshift-etcd-cluster/
[6]:https://zwischenzugs.com/2017/03/04/1-minute-multi-node-vm-setup/
[7]:https://zwischenzugs.com/2017/03/18/clustered-vm-testing-how-to/
[8]:https://zwischenzugs.com/2017/10/27/ten-things-i-wish-id-known-before-using-vagrant/
[9]:https://zwischenzugs.com/2017/10/21/openshift-3-6-dns-in-pictures/
[10]:https://github.com/ianmiell/shutit-linux-dns/blob/master/linux_dns.py