When your gateway hijacks DNS
An internal site that wouldn't load, a resolver that was never asked, and the dead-IP query that proved a 'helpful' gateway was answering DNS behind my back.
An internal-only site stopped loading from every client on the network. The stack behind it was fine. I could reach it on port 443, and curl --resolve returned a clean HTTP 200 with a valid certificate. So the server wasn’t the problem. Nobody could find it. Classic DNS, except the obvious suspects were all innocent.
The resolver wasn’t the one being asked
Internal names live in a split-horizon zone on a resolver I run; public names go out to the internet. When I queried that resolver from a client, I got an empty answer. And the authority section came back signed by my domain’s public nameservers, not my internal one. That’s impossible if the query had actually reached my resolver.
So I pointed dig at a guaranteed-dead address (192.0.2.1, a documentation IP that can answer nothing) on port 53. It answered. A corpse picking up the phone. The call is coming from inside the house. That’s the whole tell: if a dead IP replies to a DNS query, something on the path is transparently intercepting port 53. The gateway was silently DNAT’ing every client’s DNS to its own resolver, which forwarded to the public internet, where internal-only names simply don’t exist. Not IPv6, not the resolver everyone blamed. A “helpful” gateway DNS feature, quietly breaking split-horizon by design.
# First the stack itself, DNS skipped: pin the hostname to the real server IP.
curl -I --resolve web.be.jit-c.eu:443:<server-ip> https://web.be.jit-c.eu
# -> HTTP/2 200. The server is healthy; the problem is finding it, not serving it.
# The dead-IP test: 192.0.2.1 (TEST-NET-1) is dead and should time out.
dig @192.0.2.1 -p 53 example.com +short
# -> You still get an A record back. A reply from a guaranteed-dead address means
# something on the path is intercepting port 53: the gateway is DNAT'ing all DNS.
192.0.2.1) that still picks up: something on your own path is intercepting port 53. The call is coming from inside the house.One resolver, every layer
The fix was two moves: stop the gateway intercepting DNS, and pull every layer of DNS security into the one resolver I actually control and can introspect.
Technitium now does the lot: authoritative answers for internal names, ad- and threat-blocking from OISD Big and Hagezi’s Threat Intelligence Feed (~2M domains, returned as NXDOMAIN), DNSSEC validation, QNAME minimization, EDNS Client Subnet stripped for privacy, and recursion for everything else over encrypted DNS-over-TLS to Quad9 and Mullvad. It’s all defined in an Ansible role, so it rebuilds from code instead of from memory.
Split-horizon DNS only works if your clients can actually reach your resolver. Anything on the path that “helpfully” answers port 53 is a silent single point of failure.
The lesson I’m keeping: when DNS misbehaves, don’t just check the server. Check whether your query is even getting there. A dead-IP test takes ten seconds and tells you who’s really answering.
(Earlier field notes from this corner of the lab: standing up the internal DNS in the first place, and teaching an AI agent to speak Technitium.)