DNS Security: DNSSEC, Response Policy Zones and Encrypted DNS

Four problems, four tools

DNS was designed in an era when the network was assumed friendly. A client asks a resolver for the address of a name, and the answer arrives in an unauthenticated UDP packet that anyone on the path can read or forge. Over the decades that produced several different problems. Answers can be forged, so a client goes to the wrong place. Queries can be read, so an observer learns every site a user visits. Genuine answers can point to malicious sites, because the attacker simply registered a name. And the infrastructure itself can be overwhelmed. Different tools address each problem, and mixing them up leads to either a false sense of security or conflicts between controls.

The tools are DNSSEC for authenticity, encrypted DNS for confidentiality in transit, response policy zones and protective DNS for blocking malicious destinations, and anycast and rate limiting for availability. This article covers the first three and points to our article on anycast DNS and edge load balancing for the last.

DNSSEC: proving answers are genuine

DNS Security Extensions are described in RFC 4033, with the record formats in RFC 4034 and the protocol changes in RFC 4035. RFC 4033 says DNSSEC adds data origin authentication and data integrity to DNS, together with a way to distribute public keys, and that the extensions do not provide confidentiality. Zones are signed. Each set of records of one type carries a signature in an RRSIG record, the zone's public keys are published in DNSKEY records, and a signed zone's parent holds a DS record, a digest of the child's key. A validating resolver can therefore check that an answer came from the holder of the zone's key and was not altered.

Trust flows down a chain. The resolver is configured with at least one trust anchor, a key or a DS value obtained through some trusted channel outside DNS; in practice the anchor is the root zone's key, which the resolver software ships with. A DNSKEY in the root authenticates the DS record for a top-level domain, which authenticates that domain's key, and so on to the name you asked for. Many zones separate a key-signing key, which signs the other keys, from a zone-signing key, which signs the records. RFC 4033 notes this split is an operational choice, not something validation enforces.

RFC 4035 describes what the validator concludes. The result is secure if a chain of signed records leads from a trust anchor to the data; insecure if it knows no trusted chain should exist, as for an unsigned zone; bogus if a chain should exist but signatures fail or required data is missing, which may mean an attack, misconfiguration or corruption; and indeterminate if the resolver cannot get the records needed to decide. A resolver returns an error for bogus data rather than the answer. Signed denial matters as much as signed answers: NSEC records prove that a name or type does not exist, so an attacker cannot simply claim it does not. NSEC links names in order, which also lets anyone walk a zone and list its names, a known side effect of the original design.

Three header bits matter operationally. A client sets DO to say it wants DNSSEC records, and CD to ask the server not to validate. The server sets AD to say it validated the answer. RFC 4035 warns that an attacker who can set AD in a response or CD in a query weakens protection for clients that rely on the resolver, so a stub resolver should trust AD only over a secure channel to a trusted server. In practice the validation should happen in your own recursive resolvers, which are the ones to configure.

What DNSSEC does not do matters just as much. It does not hide queries or answers. It does not protect against denial of service and adds a new kind, because validation costs computation. It does not protect zone transfers or dynamic updates, which need TSIG or other mechanisms. It does not stop an attacker from registering a convincing look-alike name and signing it perfectly. And it protects only DNS data: the web server behind the address still needs TLS, and a certificate authority validating a domain by DNS benefits from signed zones.

There are two sides. Validating is cheap and should be on by default in company resolvers; unbound, BIND and the cloud resolvers do it. Signing your own zones is a separate decision with more risk: a signature that expires or a botched key rollover makes your domain unreachable for every validating resolver, not just a few. Teams that sign should automate re-signing and rollover, monitor signature expiry independently, and rehearse the DS update at the registrar. Those who use a managed DNS provider can usually enable signing with a switch and a DS record. Ask who owns the rollover.

Response policy zones: your resolver's block list

A response policy zone is a way for a resolver to apply policy to the answers it returns. ISC describes RPZ as a method of filtering DNS responses so a resolver can block or redirect queries to known-malicious domains, IP addresses and name servers; it was invented at ISC and first implemented in BIND, but it is an open mechanism that other resolvers implement. The policy is written as a special zone that the resolver loads, usually transferred from a threat-intelligence provider by ordinary zone transfer.

A rule can trigger on the query name, on an address that would appear in a legitimate answer, on the name or address of the authoritative name server, or on the address of the client asking. A rule can answer with a synthetic NXDOMAIN or a NODATA response, redirect by CNAME to a warning page, substitute local data, force the client to retry over TCP, exempt the query from further processing, or drop it without answering. Commercial and public feeds exist for malware, phishing and command-and-control domains. The BIND configuration also has options such as break-dnssec, which controls whether a rewrite is allowed to break validation, and settings for waiting on recursion before applying a name trigger; check the documentation for the version you run.

RPZ is valuable because the DNS query is the earliest point at which an infected machine reveals itself. A host asking for a known command-and-control name can be blocked and logged, and the log points to the host to clean. Allow-list entries let you override a false positive. The limits are that policy is only as good as the feed, that malware can use IP addresses directly or encrypted DNS to a resolver of its choosing, and that rewriting answers conflicts with DNSSEC if not configured carefully, because a rewritten answer will not validate unless the resolver does the rewriting after its own validation.

Encrypted DNS: DoT and DoH

Plain DNS leaks every name a user looks up. Two standards encrypt the hop between client and resolver. DNS over TLS, RFC 7858, uses TCP port 853: the client connects, completes a TLS handshake and then exchanges ordinary DNS messages with a two-byte length prefix. The RFC defines an opportunistic profile, in which the client uses privacy when available but may not validate the server and is exposed to on-path attacks that remove privacy, and an out-of-band key-pinned profile, in which the client connects only to servers it can authenticate by pinned key fingerprints. It recommends clients reuse a single connection, and notes that handshakes add round trips compared to UDP, which session resumption and TCP Fast Open reduce.

DNS over HTTPS, RFC 8484, maps each query-response pair into an HTTP exchange. The payload is a DNS message in the application/dns-message media type, sent by GET with a base64url-encoded query or by POST with the raw message. HTTP/2 is the recommended minimum, answers can be cached under normal HTTP rules, and the client uses a configured URI template for its resolver. DoH runs on port 443 and blends with other web traffic, which makes interference and traffic analysis harder. The RFC itself says that filtering or inspection systems relying on plaintext DNS will not work under DoH, and that split-DNS and local policy can yield different answers from different servers.

Both protocols protect the path only. The RFC for DoH notes it provides no DNSSEC-style integrity: the client either validates itself or trusts the server's AD bit. Both move trust to the resolver operator, who now sees the queries. Residual risks listed for DoT include downgrade and middlebox interference, unprotected cleartext on port 53 and traffic analysis by size and timing; padding helps. DNSSEC and DoT are independent and complementary.

For a company this raises a governance question, not just a technical one. Browsers and operating systems may encrypt DNS to a public resolver by default, which bypasses your internal resolvers, your split-horizon names for internal systems and your RPZ. The usual answer is to run internal resolvers that support DoT or DoH toward clients where wanted, to block outbound port 853 and known DoH endpoints from the managed network, and to use enterprise configuration to point managed devices at the approved resolver. That keeps privacy from the local path while preserving the controls you are accountable for.

Protective DNS

Protective DNS is the commercial and government name for a filtering resolver. CISA describes its Protective DNS resolver as an upstream resolver that matches requests against threat-intelligence indicators and blocks, redirects or sinkholes the response on a match, alerting the affected agency and CISA. It covers mobile, roaming and cloud assets because their traffic goes straight to the resolver, and it is available at no cost to federal civilian agencies and, on a limited basis through a pilot, to critical-infrastructure entities. Mid-size companies can get the same function from commercial filtering resolvers or by running RPZ with licensed feeds.

A layered setup for a mid-size company

• Run two or more internal recursive resolvers with DNSSEC validation on, and make them the only resolvers managed devices may use.
• Apply an RPZ or protective DNS feed with logging, and send the logs to the same place as your firewall logs; alert on repeated blocked queries from one host.
• Block outbound DNS, DoT and known DoH destinations at the edge except from the resolvers, and deploy device policy that points to your resolver.
• Use encrypted DNS between roaming devices and your resolver or filtering service, so a laptop on hotel Wi-Fi is not exposed.
• Sign your public zones only if you can automate and monitor re-signing and key rollover. Otherwise use a provider that does.
• Restrict zone transfers and updates with TSIG, limit who can change registrar settings, and turn on registry locks for important domains.
• Watch for look-alike registrations of your brand and for dangling records that point to services you no longer use.
• Test the failure path: what happens to users if a resolver is down, if a signature expires, if the feed is empty.

Clayton Cybernetics designs resolver and DNS architecture for clients, including DNSSEC signing with managed rollover, RPZ and protective DNS, and the monitoring that makes a DNS change reviewable.

Sources

• RFC 4033 - DNS Security Introduction and Requirements
• RFC 4035 - Protocol Modifications for the DNS Security Extensions
• RFC 7858 - Specification for DNS over Transport Layer Security (TLS)
• RFC 8484 - DNS Queries over HTTPS (DoH)
• ISC - DNS Response Policy Zones (RPZ)
• CISA - Protective DNS Resolver
• BIND 9 Administrator Reference Manual (response-policy grammar)

This article is general technical information, not a DNS design for your network. We read the RFCs, the ISC page and the CISA page through a summarizing fetch tool, and RFC 4033 and 4035 (long documents) were read in part. RFC 4034 was not read; the record-format descriptions come from RFC 4033 and 4035. The BIND reference page was truncated before the RPZ section, so BIND option behavior is described only from the grammar fragment; check the documentation for your version. The layered setup, the operational advice on signing, rollover and blocking DoH, and the remarks on malware bypass are general engineering knowledge, not quoted from the sources listed. Last reviewed November 8, 2026.

Keep reading

Next · SecurityPost-Quantum Encryption in Practice: Hybrid ML-KEM and X25519, FIPS 203A large enough quantum computer would break the public-key cryptography that protects today's connections, and traffic recorded now could be decrypted later. In August 2024 NIST finalized the first post-quantum key-establishment standard, FIPS 203, called ML-KEM. In TLS the practical answer is a hybrid: combine ML-KEM with a classical X25519 exchange so that the connection stays safe if either holds. This article explains the standard, the hybrid design as published in an IETF RFC, what it costs, and how a company should inventory and migrate without panic.9 min read

• Penetration Testing Explained for Executives: Scope, Reports and How Often to Test8 min
• Ransomware Readiness and Incident Response: A Checklist Based on the CISA #StopRansomware Guide9 min