when you resolve a DNS name in a Python program, it checks /etc/hosts, but when you use dig, it doesn’t.
This one seems surprising at first but when you think about it, it makes sense: The python program (or any other program, including ping) is asked to resolve a name into an IP address, so it goes through the whole /etc/nsswitch.conf routine, usually starting with checking /etc/hosts.
dig and it’s older companions like host and nslookup are not actually designed to resolve a name into an IP, they are designed to query DNS services. What you’ve put on the local server is irrelevant to them.
Later edit after completing the article. nscd used to be far more popular on *BSD and Solaris 20 years ago. They may still be the solution there. For some reason Linux never incorporated it by default, so it was eventually re-invented on linux
The idea of DNS lookup on Linux being implemented by libc is weird to me — shouldn’t something that fundamental be at a lower level?
That said, I believe Chrome has long had its own DNS resolver, probably so it can do async lookups.
As for Darwin — libc is part of libSystem, not a separate entity. And mDNSResolver is the daemon that runs multicast DNS and DNS-SD, aka Bonjour … but IIRC it might also act as the resolver/cache for regular DNS too.
Apple OSs have a framework called SystemConfiguration that manages settings, replacing a lot of text files in traditional Unix. It stores a lot of network config like DNS resolvers. One of the benefit is it has an observer API that lets you receive notifications of when settings change, which is very useful in a mobile system that switches networks a lot.
Maybe a syscall, or at least a lower level library. Put it this way: since C standard libraries appear to be interchangeable components on Linux, I wouldn’t expect one of the tasks of building one to be implementing a DNS resolver. Just like I wouldn’t expect the stdio functions to implement a filesystem.
(For context, I’m coming from a world where the stdlib is part of the system library just like the syscall glue.)
Just like I wouldn’t expect the stdio functions to implement a filesystem.
I don’t think a file system and a DNS resolver are at the same level of “fundamental”ness, you need the kernel to multiplex access to the storage for correctness and security’s sake with a file system, the socket related syscalls already provide sufficient multiplexing to implement DNS same as any other protocol that lives on top of tcp or udp.
Put it this way: since C standard libraries appear to be interchangeable components on Linux, I wouldn’t expect one of the tasks of building one to be implementing a DNS resolver.
From a “Linux” perspective it’s not that it is designed to have interchangeable C libraries, it’s that it is agnostic to whether there is a C library at all(*). Any pressures that result in multiple C libraries would apply equally to any “lower level” library IMO, they would be just as interchangeable.
And the lower level C library that implemented a DNS resolver would need syscall wrappers for socket etc. to do its work, wrappers which libc already has. There’s not much benefit to breaking it off.
You can imagine a “liblinux” that was just syscall wrappers and then a “libdns” that used that for just the resolver bits and then libc consumed both of those and provided stdio, et al on top of that, and that’s a fine design but there’s no way to force glibc and musl say to both use “liblinux”/“libdns” so that the “lower level” isn’t interchangeable but libc still is.
For context, I’m coming from a world where the stdlib is part of the system library just like the syscall glue
This is generally true in Linux, it’s just that the “system” library is called “libc”. glibc comes with a pile of syscall glue, musl comes with its own pile of syscall glue.
* This is of course a bit of hyperbole there are definitely things in Linux that are the way they are because they’ve been informed by the needs of implementing a posixy/unixy libc.
I have also been surprised by getaddrinfo.
If you want to be able to resolve before a valid non-loopback interface is up on an IPv4 only system, but prefer IPv6 for IPv6 systems, the logic gets a bit tricky. You can get back IPv6 addresses even if IPv6 is disabled globally at the Linux kernel level.
It is also worth noting that getaddrinfo is a synchronous API. So if you are using async then you will either need a DNS threadpool or you will need go use a different resolver.
I am surprised that she writes off nscd, since I remember that being one of the widely accepted, low-effort ways to get caching on Linux. It used to be one of the first things I installed back when I used Linux, and was also common at one of my jobs.
On Linux with the GNU libc, /etc/nsswitch.conf is central on how getaddrinfo works. On some configurations, notably when running systemd-resolved, /etc/nsswitch.conf can instruct getaddrinfo to never look at /etc/resolv.conf. It also explains why /etc/hosts is checked first and how multicast DNS is entangled into that.
This one seems surprising at first but when you think about it, it makes sense: The python program (or any other program, including ping) is asked to resolve a name into an IP address, so it goes through the whole /etc/nsswitch.conf routine, usually starting with checking /etc/hosts.
digand it’s older companions likehostandnslookupare not actually designed to resolve a name into an IP, they are designed to query DNS services. What you’ve put on the local server is irrelevant to them.Later edit after completing the article.
nscdused to be far more popular on *BSD and Solaris 20 years ago. They may still be the solution there. For some reason Linux never incorporated it by default, so it was eventually re-invented on linuxThe idea of DNS lookup on Linux being implemented by libc is weird to me — shouldn’t something that fundamental be at a lower level?
That said, I believe Chrome has long had its own DNS resolver, probably so it can do async lookups.
As for Darwin — libc is part of libSystem, not a separate entity. And mDNSResolver is the daemon that runs multicast DNS and DNS-SD, aka Bonjour … but IIRC it might also act as the resolver/cache for regular DNS too.
Apple OSs have a framework called SystemConfiguration that manages settings, replacing a lot of text files in traditional Unix. It stores a lot of network config like DNS resolvers. One of the benefit is it has an observer API that lets you receive notifications of when settings change, which is very useful in a mobile system that switches networks a lot.
What lower level? Are you expecting DNS related syscalls or maybe some /dev/dns device?
Maybe a syscall, or at least a lower level library. Put it this way: since C standard libraries appear to be interchangeable components on Linux, I wouldn’t expect one of the tasks of building one to be implementing a DNS resolver. Just like I wouldn’t expect the stdio functions to implement a filesystem.
(For context, I’m coming from a world where the stdlib is part of the system library just like the syscall glue.)
I don’t think a file system and a DNS resolver are at the same level of “fundamental”ness, you need the kernel to multiplex access to the storage for correctness and security’s sake with a file system, the socket related syscalls already provide sufficient multiplexing to implement DNS same as any other protocol that lives on top of tcp or udp.
From a “Linux” perspective it’s not that it is designed to have interchangeable C libraries, it’s that it is agnostic to whether there is a C library at all(*). Any pressures that result in multiple C libraries would apply equally to any “lower level” library IMO, they would be just as interchangeable.
And the lower level C library that implemented a DNS resolver would need syscall wrappers for
socketetc. to do its work, wrappers which libc already has. There’s not much benefit to breaking it off.You can imagine a “liblinux” that was just syscall wrappers and then a “libdns” that used that for just the resolver bits and then libc consumed both of those and provided stdio, et al on top of that, and that’s a fine design but there’s no way to force glibc and musl say to both use “liblinux”/“libdns” so that the “lower level” isn’t interchangeable but libc still is.
This is generally true in Linux, it’s just that the “system” library is called “libc”. glibc comes with a pile of syscall glue, musl comes with its own pile of syscall glue.
* This is of course a bit of hyperbole there are definitely things in Linux that are the way they are because they’ve been informed by the needs of implementing a posixy/unixy libc.
It’s turtles all the way down.
I have also been surprised by getaddrinfo. If you want to be able to resolve before a valid non-loopback interface is up on an IPv4 only system, but prefer IPv6 for IPv6 systems, the logic gets a bit tricky. You can get back IPv6 addresses even if IPv6 is disabled globally at the Linux kernel level.
x-ref https://github.com/frc971/971-Robot-Code/commit/0a0a82766b91ccd4ab931093bdd6d9e61aa93797 (author/committer is not me because I don’t have an account on the correct system)
Once upon a time, I tried to make sense of all this. Personally, the historical perspective was what made it all clicked.
It is also worth noting that getaddrinfo is a synchronous API. So if you are using async then you will either need a DNS threadpool or you will need go use a different resolver.
Or use getaddrinfo_a, I guess.
I am surprised that she writes off
nscd, since I remember that being one of the widely accepted, low-effort ways to get caching on Linux. It used to be one of the first things I installed back when I used Linux, and was also common at one of my jobs.On Linux with the GNU libc,
/etc/nsswitch.confis central on howgetaddrinfoworks. On some configurations, notably when running systemd-resolved,/etc/nsswitch.confcan instructgetaddrinfoto never look at/etc/resolv.conf. It also explains why/etc/hostsis checked first and how multicast DNS is entangled into that.The thing that surprised me is that it can hang for large periods of time under certain conditions.
dscacheutil is unrelated to mDNSResponder.