Exposing Docker's internal DNS with CoreDNS

Whilst Docker is a containerisation technology, it's not just about running applications - there's also networking. When you add a container to a docker network, it magically becomes discoverable by other containers on the same network with DNS. All containers use Docker's magical internal DNS server to achieve this. However, whilst Docker networks are accessible to the host (unless you set internal: true), the same isn't true for the DNS server - which is only accessible from inside the Docker network.

For reasons I won't go into (at least in this post), I needed access to Docker's internal DNS server, to discover the address of specific containers by name - sort of like service discovery, but far more naive. There were 2 main approaches as I saw it - the first was by modifying Docker's networking setup to allow access to its DNS server from outside a docker network, and the second is by proxying requests through.

I'm not a networking guy, I much prefer living at layer 5 and up, so I don't want to go anywhere near Docker's networking rules. Docker has a hate-hate relationship with tools like iptables and UFW, which makes firewalling properly much much complex. I don't want to ever have to touch Docker's networking rules if I can possibly avoid it. So, proxying it is!

#Proxying DNS

When talking about proxying in the HTTP world, we tend to think of reverse proxies, which expose multiple services running behind it through a single "proxy" based on various incoming attributes (hostname, path etc). These services aren't accessible to clients directly, perhaps for security, additional functionality / configuration, or simply because there are multiple services on the same system. Common HTTP reverse proxies include nginx, Apache and Traefik.

To expose DNS externally, we need to do the same thing: Receive DNS requests with one tool, and relay them through to a backing service, in this case Docker's DNS server. Unlike a reverse proxy, we need to do very little processing in the middle. In fact, we don't need to do any processing, but there are some benefits to doing at least a little.

#Tooling

There are lots of different DNS servers out there: dnsmasq, BIND, unbound, even pihole (dnsmasq in a trenchcoat) and AdGuardHome to name a few popular ones. All of these are capable of being fully-fledged DNS servers - that you provide with DNS records and they'll reply. But that's not what we need right now. Additionally, all have the ability to delegate certain requests to other servers. For example, if you wanted everything to be handled by Cloudflare's DNS servers, but wanted mydomain.com to be handled by your own instance of BIND, you totally can.

#CoreDNS

For me, I didn't use any of these tools - I used CoreDNS. CoreDNS may be familiar to you if you're used to doing funky things with Kubernetes, but it's a highly configurable DNS server with a number of useful plugins and built-in functionality. I've used dnsmasq for a while, but CoreDNS is far easier to configure and work with. It's already running in a few different places in my infrastructure and I'm more than happy with it. There's a first-party Docker container too (unlike dnsmasq), which makes deploying it a breeze and updates even easier.

#No DNS?

All of these tools speak DNS natively. In the process, they parse and interpret the DNS request, and create a completely separate one to the backing DNS server. However, there's no reason we need to do that. DNS is just UDP, and it's possible just modify the raw packets a little to change their destination (known as DNAT). There are tools to do this, but it's easier to debug and introspect if the connection is completely terminated, and enables a little more functionality.

#Configuration

The premise is simple. When a DNS request comes in, forward it to Docker's DNS server, which is accessible to all containers at 127.0.0.11:53, and relay the response back. No additional processing necessary.

With CoreDNS, this is pretty simple:

Yes, that's really it - that's the whole configuration. CoreDNS will listen on port 53 (its default), and forward everything through to Docker. The errors and cancel lines are plugins to abort slow requests and show errors if they happen (there's no access logs by default).

If you take this configuration and deploy it (for example, with docker-compose), you'll be all set:

This exposes containers on the "coredns" network via a DNS server listening on port 5533. If you add the device to multiple networks, it should expose the devices on all the networks it's a part of.

#Extras

But, whilst this works perfectly fine, I wanted a little more out of this DNS server. And if you're still reading, you probably do too.

#Subdomain

Docker responds to requests for a container's "name", which in the case of my website is website-website-1. So dig website-website-1 @127.0.0.1 -p 5533 returns the correct IP address. But if someone sees website-website-1 in my configuration, it might not be clear what it is or where it's coming from. To deal with that, it'd be nice if instead it would respond to website-website-1.docker. Docker can't do that, but we can fake it.

To achieve this, we need to strip the .docker suffix from the incoming domain (if one exists), and stick it back on the reply. With CoreDNS, that's a single line with the rewrite plugin:

rewrite name suffix replaces all records ending with .docker with . (thus stripping it off). answer auto is the magic sauce which ensures the DNS response says the record is still for .docker, rather than the root. That's rarely an issue for clients, but it's still good practice to keep the query and response consistent.

And just like that, requests for *.docker work too.

#Restrictions

The purpose of this DNS server is to serve requests for Docker's DNS server only. Docker is smart enough to only reply to requests for services it knows about, but CoreDNS isn't. Whilst it's not an issue, it'd be nice if CoreDNS rejected any other kind of traffic, so it's easier to see a DNS leak and debug issues. To achieve that, we need to only accept requests for .docker, and block everything else. That's a few more lines of configuration, but not many.

#Only accepting .docker requests

CoreDNS borrows the concept of "views" from BIND. Views determine which requests a given server block (the top level thing in curly brackets) can respond to. CoreDNS's view plugin lets us define this:

Whichever server block defines this "view" will only respond do requests ending .docker.

#Rejecting anything else

The final step is to ensure that any request not serviced by our main "view" is rejected. By default, if no blocks are valid for the incoming request, CoreDNS will simply return no answer. For our use case, that's fine, but it'd be nicer to hint to the user the reason they got no answer.

For this, we can lean on CoreDNS's built-in ACL support from the acl plugin. Normally, ACLs allow restricting who can make queries to a given view. In our case, we want to define a catch-all view which allows no one to access it.

Much like everything with CoreDNS, that's just another few lines of configuration, this time right at the end of the file:

By not defining any allow statements, CoreDNS will "block" all requests which hit this view (ie which don't end .docker). It's also possible to drop the requests, that trades a little more security with a lot more debugging annoyance.

#Conclusion

Once CoreDNS is up and running, it's now possible to request the IP address of a device on a Docker network using just its name, on a server which won't reply to anything else.

Both CoreDNS and Docker's DNS are fast - I've never seen a request report more than 0ms with dig. Whilst you can add caching to CoreDNS, there's not much point, and you'd be better off adding caching to whatever is making the DNS request in the first place. Docker gives records a TTL of 10 minutes, but if a container restarts its IP address might change, so I'd recommend leaving the cache pretty low if the container might move.

If you take one thing away from this post, it's that CoreDNS is actually quite nice, and if you need to do something semi-complex with DNS, it's a great tool to know about.