Intermediary CNAME rewriting with AdGuardHome

As part of some recent infrastructure changes, I migrated my home DNS to AdGuardHome, from Pi-Hole. Being a single, self-contained binary, it's far easier to install and manage (it's in the AUR too), and it's got a few nice modern features like DNS-over-HTTPS and DNS-over-TLS. Sure, the ad blocking features are great, but I use it mostly for overriding DNS.

If I'm at home, and so is my server, there's no reason the traffic in between shouldn't stay on my LAN. It's faster, lower latency, and doesn't take a round trip via a datacentre in London. Before AdGuardHome, I did a simple IP mapping - rewriting any DNS record with the IP of my VPS to the IP of my home server. Pi-hole doesn't support this natively, but it's just a single line of plain dnsmasq config. For years, it worked just fine, although it does have the limitation of making it impossible to point a record actually at the VPS. AdGuardHome however doesn't support IP value rewriting, but it does have DNS rewriting.

AdGuardHome has a feature for "DNS rewrites", which is a poor name for an otherwise very useful feature. The "rewrites" don't actually rewrite, but instead are local records which are looked at before issuing a query upstream. If you want to add records, or override existing ones, this is the feature you want - and it works great. But, it doesn't work in 1 specific case: Recursively resolved CNAMEs.

#CNAMEs?

But first, a tangent - What is a CNAME?

A CNAME is a fairly crucial part of DNS. Whilst conventionally, when you query a record, you get back an IP address - it doesn't have to be that way. A CNAME record references another domain instead, for the client to look up again, which may be another CNAME, or the destination IP address.You might think this has performance implications, and you're not wrong, but most DNS providers will handle this "recursive" resolution for you, and return it all in a single query:

Here, we can see my Gitea server at git.theorangeone.net doesn't have an IP address, instead pointing to pve.sys.theorangeone.net, which itself has an IP address (213.219.38.11 in this case).

#Local DNS

And, here-in lies the problem. When I ran Pi-Hole, I had a custom dnsmasq config which rewrote any record with 213.219.38.11 to the LAN IP of my home server, and it worked great. Clients would never know the difference, but traffic would never leave my LAN. However, AdGuardHome doesn't support this - you have to specify custom records yourself. I have a lot of records, so that's a lot of records to add and maintain.

To get around this, I rewrote all my DNS records to use CNAMEs instead. By creating a single record in AdGuardHome for pve.sys.theorangeone.net , which points to my LAN IP address, it'd rewrite all the domains which pointed to pve.sys.theorangeone.net. That way, there'd only be a single record to maintain (which would rarely need to change) rather than needing to update the list for every service I deploy. Where a service is hosted is managed in 1 place, public DNS, and my home network knows how to find it in the most efficient way possible. Cloud services go directly to cloud services, and local ones stay on my LAN, nice and fast.

Or so I thought.

#The problem

Unfortunately, there's a problem, and I've kind-of already mentioned it in passing. Most DNS providers (in fact, all the ones I tried) will be helpful and resolve CNAMEs for you, avoiding the need for another round trip. Whilst they're nice enough to still show you the CNAME rather than flattening it entirely (likely because of DNSSEC), AdGuardHome doesn't operate on the intermediary value - it only looks at the final destination. I had, naively, assumed it was AdGuardHome doing the recursive resolution on CNAMEs, but I was wrong.

Therefore, whilst it's completely supported to add a rewrite for pve.sys.theorangeone.net, it'll only be triggered when I query that record directly, rather than if it's used as part of another query. There's an open issue about this support, but it sounds like it's considered "out of scope", so probably won't be coming any time soon.

So, what to do? I can't rewrite the IPs, because AdGuardHome doesn't support it (not to mention it has issues if I ever want to host something on my VPS directly - which at the time of writing I now do), and I can't rewrite an intermediary CNAME, because AdGuardHome doesn't support it.

I could restructure my domains, so ones hosted on my LAN are served at *.internal.theorangeone.net or alike, but that's quite messy, and makes moving services needlessly complex. All my subdomains are at 1 level (eg git., plausible. etc), and could be hosted in very different places, so routing certain subdomains also isn't an option. By keeping public DNS as the single source of truth, I can update it exactly once, and know traffic will be routed correctly.

Looks like I'll have to do this myself!

#CoreDNS

Fortunately for me, AdGuardHome does support customizing the upstreams quite extensively. I have a few upstreams configured, partly for availability reasons, and partly to help with privacy. Whilst the core configuration in AdGuardHome simply specifies a list of upstreams, you can specify specific upstreams for a specific domain (or domains). So instead of letting my usual upstreams handle my domains' traffic, I'll first proxy them through an intermediary DNS server, which can do what I need (mostly): CoreDNS.

Since I discovered CoreDNS, it's taken over any remotely custom DNS server use cases I've had. It's relatively lightweight, incredibly powerful, and has a flexible configuration language. Sure, it's yet another giant Go binary on my system, and I do have to compile it myself, but it works perfectly.

With a few lines of configuration in AdGuardHome, traffic for my domains is handled by CoreDNS running on the same system (on port 53053, because 5353 breaks mDNS):

Yes, this means I need to add an entry for any domain which my server handles traffic for, but that's a very small number (2 at the time of writing).

With domains being routed to CoreDNS, it's time to rewrite the records properly. Because CoreDNS is only intended to handle this rewriting functionality, rather than everything, I've configured it to block requests for any other domain:

As for the rewriting, this is where things get a little weird. I need the IP address for pve.sys.theorangeone.net to point to my LAN, rather than my VPS, where it does on public DNS. Conveniently, I can add hosts records to CoreDNS much like I would would in an /etc/hosts file:

Unfortunately, much like AdGuardHome, CoreDNS doesn't consider intermediary CNAMEs - just the initial request. or the destination IP.

What CoreDNS does support, using a not-especially-well-documented feature is rewriting these intermediary CNAMEs to somewhere else. You can change what the CNAME value is, and have the request flow back through CoreDNS to be resolved. If I wanted to change where git.theorangeone.net pointed, for example to a different CNAME rather than a resulting IP, this is exactly the feature I'd need. But, it's not. It's not that part of the chain I want to modify, it's what the intermediary CNAME points to, as opposed to changing the intermediary CNAME itself.

So, I can specify custom records, and I can change what the intermediary CNAME points to. I just need to put them together.

And so, the bodge.

When you rewrite a CNAME, it flows back through CoreDNS to be resolved, but there's nothing saying the rewrite has to change anything. By rewriting the CNAME to itself, we allow CoreDNS to modify what an intermediary CNAME points to, without needing to customize each domain explicitly, and even without an additional upstream lookup.

The final configuration looks a bit like this:

Sure, I could achieve similar by rewriting the CNAME to some other value instead, but this way only the final record is changed, and it means direct uses of pve.sys.theorangeone.net still work (I don't know why I'd need that, but it's nice to have anyway).

If you're going to attempt something like this yourself, you'll need to use CoreDNS 1.11.2+, as older versions would frequently result in mangled records being returned (IPv6 address for A records, for example), and caused me to spend hours debugging before eventually disabling AAAA lookups as a temporary solution. Long story short, concurrency is hard, but 1.11.2 works exactly as you'd expect.

#"What about DNS Filters?"

Alongside custom records, AdGuardHome also supports DNS filtering, which allows complex rewriting of records, including wildcards. If you're after rewriting records, this is what you probably want to use.

I however needed something far more complex. I don't want to change the structure of my records, and I don't want to list every subdomain in AdGuardHome. If the open issue gets closed, then I can drastically simplify this.

#Putting it all together

Requests for my domains (and no others) are passed from AdGuardHome to CoreDNS, which resolves them upstream (in my case, to Cloudflare via Quad9), rewrites the intermediary record for pve.sys.theorangeone.net to the LAN IP of my server, and returns them back to the client. Sure, this involves another cog in the DNS machine at my home, but that's ok. All this happens with no notable increase in latency when resolving domains, but conveniently means most of my self-hosted traffic doesn't leave my LAN, which does massively drops latency, increases throughput, and avoids a round-trip via London.

The biggest benefit of moving to CNAME rewriting is that I can now host things on that VPS should I want to (which I now am, but more on that later). There's a separate sys domain for services on the VPS directly, whilst pve.sys.theorangeone.net is used to route to things running on my home server, taking the shortest path it can.