When your gateway hijacks DNS
An internal site that wouldn't load, a resolver that was never asked, and the dead-IP query that proved a 'helpful' gateway was answering DNS behind my back.
An internal-only site stopped loading from every client on the network. The stack behind it was fine — I could reach it on port 443, and curl --resolve returned a clean HTTP 200 with a valid certificate. So the server wasn’t the problem. Nobody could find it. Classic DNS — except the obvious suspects were all innocent.
The resolver wasn’t the one being asked
Internal names live in a split-horizon zone on a resolver I run; public names go out to the internet. When I queried that resolver from a client, I got an empty answer — and the authority section came back signed by my domain’s public nameservers, not my internal one. That’s impossible if the query had actually reached my resolver.
So I pointed dig at a guaranteed-dead address — 192.0.2.1, a documentation IP that can answer nothing — on port 53. It answered. That’s the whole tell: if a dead IP replies to a DNS query, something on the path is transparently intercepting port 53. The gateway was silently DNAT’ing every client’s DNS to its own resolver, which forwarded to the public internet — where internal-only names simply don’t exist. Not IPv6, not the resolver everyone blamed. A “helpful” gateway DNS feature, quietly breaking split-horizon by design.
One resolver, every layer
The fix was two moves: stop the gateway intercepting DNS, and pull every layer of DNS security into the one resolver I actually control and can introspect.
<rect x="230" y="20" width="300" height="48" rx="12" fill="#eef0f8" stroke="#3b4eab" stroke-width="2" />
<text x="380" y="41" text-anchor="middle" font-size="15" font-weight="700" fill="#1f2752">Client query</text>
<text x="380" y="59" text-anchor="middle" font-size="11" fill="#64748b">any device → exactly one resolver</text>
<line x1="380" y1="68" x2="380" y2="88" stroke="#8b96cd" stroke-width="2" marker-end="url(#ah)" />
<rect x="180" y="90" width="400" height="72" rx="12" fill="#f5f6fc" stroke="#3b4eab" stroke-width="2" />
<text x="380" y="115" text-anchor="middle" font-size="14.5" font-weight="700" fill="#1f2752">Technitium — the resolver I control</text>
<text x="380" y="134" text-anchor="middle" font-size="11" font-family="ui-monospace, monospace" fill="#475569">split-horizon authority · filtering · recursive forwarder</text>
<text x="380" y="152" text-anchor="middle" font-size="10.5" fill="#64748b">the gateway no longer intercepts port 53</text>
<line x1="380" y1="162" x2="380" y2="186" stroke="#8b96cd" stroke-width="2" marker-end="url(#ah)" />
<!-- Stage A: split-horizon -->
<rect x="230" y="188" width="300" height="56" rx="10" fill="#fff" stroke="#3b4eab" stroke-width="1.5" />
<text x="380" y="212" text-anchor="middle" font-size="13" font-weight="700" fill="#1f2752">Split-horizon check</text>
<text x="380" y="230" text-anchor="middle" font-size="10.5" fill="#64748b">is this one of our internal names?</text>
<rect x="20" y="188" width="185" height="56" rx="10" fill="#eef0f8" stroke="#3b4eab" stroke-width="1.5" />
<text x="112" y="211" text-anchor="middle" font-size="11.5" font-weight="700" fill="#323f8a">Internal name</text>
<text x="112" y="228" text-anchor="middle" font-size="10" fill="#475569">→ authoritative answer</text>
<text x="221" y="210" text-anchor="middle" font-size="9" fill="#64748b">yes</text>
<line x1="230" y1="216" x2="208" y2="216" stroke="#8b96cd" stroke-width="1.5" marker-end="url(#ah)" />
<line x1="380" y1="244" x2="380" y2="268" stroke="#8b96cd" stroke-width="2" marker-end="url(#ah)" />
<!-- Stage B: blocklists -->
<rect x="230" y="270" width="300" height="56" rx="10" fill="#fff" stroke="#3b4eab" stroke-width="1.5" />
<text x="380" y="294" text-anchor="middle" font-size="13" font-weight="700" fill="#1f2752">Ad + threat blocklists</text>
<text x="380" y="312" text-anchor="middle" font-size="10.5" font-family="ui-monospace, monospace" fill="#475569">OISD Big · Hagezi TIF (~2M domains)</text>
<rect x="555" y="270" width="185" height="56" rx="10" fill="#fdecee" stroke="#dc2d3d" stroke-width="1.5" />
<text x="647" y="293" text-anchor="middle" font-size="11.5" font-weight="700" fill="#9d1926">On a list</text>
<text x="647" y="310" text-anchor="middle" font-size="10" fill="#9d1926">→ NXDOMAIN (blocked)</text>
<text x="543" y="292" text-anchor="middle" font-size="9" fill="#9d1926">match</text>
<line x1="530" y1="298" x2="553" y2="298" stroke="#dc2d3d" stroke-width="1.5" marker-end="url(#ar)" />
<line x1="380" y1="326" x2="380" y2="350" stroke="#8b96cd" stroke-width="2" marker-end="url(#ah)" />
<!-- Stage C: hardening -->
<rect x="230" y="352" width="300" height="56" rx="10" fill="#fafbfd" stroke="#3b4eab" stroke-width="1.5" />
<text x="380" y="375" text-anchor="middle" font-size="12" font-weight="700" fill="#1f2752">DNSSEC · QNAME-min · no EDNS-subnet</text>
<text x="380" y="393" text-anchor="middle" font-size="10" fill="#64748b">validate · minimise · don't leak the client</text>
<line x1="380" y1="408" x2="380" y2="432" stroke="#8b96cd" stroke-width="2" marker-end="url(#ah)" />
<!-- Stage D: encrypted recursion -->
<rect x="230" y="434" width="300" height="56" rx="10" fill="#f5f6fc" stroke="#3b4eab" stroke-width="1.5" />
<text x="380" y="457" text-anchor="middle" font-size="13" font-weight="700" fill="#1f2752">Encrypted DoT recursion</text>
<text x="380" y="475" text-anchor="middle" font-size="10.5" font-family="ui-monospace, monospace" fill="#475569">DNS-over-TLS → Quad9 · Mullvad</text>
<line x1="380" y1="490" x2="380" y2="514" stroke="#8b96cd" stroke-width="2" marker-end="url(#ah)" />
<rect x="230" y="516" width="300" height="48" rx="12" fill="#eef0f8" stroke="#323f8a" stroke-width="2" />
<text x="380" y="538" text-anchor="middle" font-size="13.5" font-weight="700" fill="#1f2752">Validated answer → client</text>
<text x="380" y="556" text-anchor="middle" font-size="10" fill="#64748b">public names, safely resolved</text>Technitium now does the lot: authoritative answers for internal names, ad- and threat-blocking from OISD Big and Hagezi’s Threat Intelligence Feed (~2M domains, returned as NXDOMAIN), DNSSEC validation, QNAME minimization, EDNS Client Subnet stripped for privacy, and recursion for everything else over encrypted DNS-over-TLS to Quad9 and Mullvad. It’s all defined in an Ansible role, so it rebuilds from code instead of from memory.
Split-horizon DNS only works if your clients can actually reach your resolver. Anything on the path that “helpfully” answers port 53 is a silent single point of failure.
The lesson I’m keeping: when DNS misbehaves, don’t just check the server — check whether your query is even getting there. A dead-IP test takes ten seconds and tells you who’s really answering.
(Earlier field notes from this corner of the lab: standing up the internal DNS in the first place, and teaching an AI agent to speak Technitium.)