Networking Interview: How Do You Troubleshoot SERVFAIL After a DNSSEC Rollover?

Prompt and When It Applies

At 21:00 UTC, example.com performs a DNSSEC key rollover and changes the A record for api.example.com from 192.0.2.20 to 192.0.2.40. Ten minutes later, clients using some validating recursive resolvers receive SERVFAIL. Adding +cd to the same resolver query returns the new address and an RRSIG, and direct queries to the authoritative servers also receive responses. Other clients continue using the old address. The parent currently publishes a DS with key tag 18200, while the child zone publishes only a DNSKEY with key tag 51900. Diagnose, recover, and verify the incident without abandoning the DNSSEC validation boundary or distributing untrusted answers to users.

The domain, addresses, key tags, times, and changes are interview assumptions. 192.0.2.0/24 is a documentation address block. The central task is to derive the state of each DNS resolution layer and the DNSSEC chain of trust from observable symptoms. An application outage should not be inferred from SERVFAIL alone. The question fits SRE, infrastructure, networking, security, backend, and general software engineering roles, so its category is general.

A public 2025 discussion about senior-role interviews still treats DNS and DNSSEC knowledge as relevant interview material. That evidence establishes a current interview context; it does not establish a frequency or a fixed employer question. DENIC's 2026 report on the .de DNS outage documents a rollover defect that made most signatures unverifiable and caused validating resolvers to reject affected delegations. Key, signature, and validation failures therefore remain operationally real.

The existing “what happens when you type a URL” article covers the normal case in which a client delegates recursive work and caches shorten the path. The SSRF article focuses on authorizing resolved addresses and closing DNS-rebinding gaps. This question focuses on parent DS records, child DNSKEY/RRSIG records, validation state, safe recovery order, and verification across resolvers. Its failure layer and security invariants are distinct.

What the Interviewer Evaluates

The first signal is layered diagnosis. A name-resolution failure can originate in the local stub, a corporate or public recursive resolver, the parent delegation, authoritative servers, DNSSEC validation, or the final A, AAAA, or CNAME data. A strong answer holds the query name, type, time, and resolver constant, gathers the RCODE, answer, TTL, authority data, and DNSSEC records at each layer, and identifies the first point where observations diverge.

The second signal is accurate response-code interpretation. NXDOMAIN means the authoritative chain claims that the queried name does not exist. NOERROR with an empty answer can mean the name exists but has no record of that type. A timeout means no usable response arrived within the budget. REFUSED means the server declined the query. SERVFAIL is a generic server failure. Failed DNSSEC validation can produce SERVFAIL, but one status code cannot prove that cause.

The third signal is understanding the chain of trust. A DS record lives at the parent-side delegation point and connects the parent's trusted chain to a child DNSKEY. RRSIG records sign RRsets in the child zone, while the validator also checks algorithms, digests, signature time bounds, and a trust anchor. A key tag helps locate a candidate key; it is not a cryptographic proof. The answer must verify that the DS digest corresponds to an actual DNSKEY and that published keys validate the relevant signatures.

The fourth signal is controlled comparison. Against the same recursive resolver, a normal +dnssec query that fails while a +cd query returns raw records strongly implicates the validation path. CD asks the resolver to disable checking for that query and is useful for diagnosis. It does not make returned data trustworthy and is not a production bypass. An AD bit from a trusted validating resolver can claim authenticated data. Absence of AD can also reflect the client's request, resolver policy, or trust boundary, so it must be interpreted with the rest of the evidence.

Finally, the interviewer looks for recovery discipline. If the old key referenced by the current parent DS was retired too early, the fastest safe path is often to restore the DNSKEY and valid signatures that the current DS can authenticate, then resume a rollover with an overlap window. Globally disabling validation, flushing arbitrary caches, or repeatedly editing the A record expands risk. After recovery, existing TTLs still have to converge, and independent validators must prove both a sound chain and a working application.

Questions to Clarify Before Answering

• Does the impact affect one name, the whole zone, or many zones under one TLD? The scope determines whether to start with a child-zone change, the authoritative provider, or a parent/registry incident.
• Which recursive resolver does the failing client actually use? Browser encrypted DNS, corporate forwarders, VPNs, and operating-system settings can send clients on different paths from the same network.
• What are the RCODE, EDE, query type, and time of each failure? An A success with an AAAA failure, a broken CNAME hop, or a failure limited to validating resolvers produces different branches.
• What were the NS, DS, DNSKEY, RRSIG, and TTL values before and after the change? Rollover order, maximum cache lifetime, and recoverable old keys determine the mitigation.
• Do all authoritative servers return the same SOA serial and signed RRsets? Version skew among authorities can create intermittent results; one server is not representative of the zone.
• Are resolver and signer clocks correct? RRSIG records have inception and expiration times, so significant clock error can cause validation failure.
• Can the old IP safely continue serving traffic? If yes, retain compatibility during TTL convergence. If not, manage that connection risk separately because DNS cannot forcibly revoke cached answers.
• Who controls the parent DS and emergency changes? The registrar, registry, DNS provider, and internal on-call team have different permissions and response times.

30-Second Answer Framework

“I would pin down the failing client's recursive resolver and record the RCODE, EDE, A/AAAA/CNAME data, TTL, and timestamp. Against that same resolver, SERVFAIL normally and data with +cd directs me to the DNSSEC validation path. I would retrieve DS from the parent and DNSKEY, RRSIG, and SOA from every authoritative server, then compare the digest, algorithm, signature times, and node consistency. Here the parent DS has no matching child DNSKEY, consistent with retiring the old key too early. I would first restore the known-good key and signatures that match the current DS. After multiple validators return authenticated answers, I would monitor TTL convergence and add overlapping rollover, prepublication validation, signature-expiry alerts, and multi-vantage synthetic checks.”

Step-by-Step Deep Answer

Step 1: Establish the failure boundary and controls

Record the time, client network, actual recursive resolver, query name, and query type for one failure. Bypass application and browser caches and query the resolver that the client uses. Then choose an independent validating resolver as a control. Do not change the name, type, and resolver simultaneously, because the difference would no longer isolate a variable.

dig @<affected-resolver> api.example.com A +dnssec
dig @<affected-resolver> api.example.com AAAA +dnssec
dig @<control-resolver> api.example.com A +dnssec

Preserve status, flags, Answer, Authority, Additional, TTL, and any EDNS Extended DNS Error from the full response. If A validates while AAAA fails, inspect AAAA and its CNAME chain. If every type times out, first inspect network reachability, port 53, authoritative availability, and packet-size handling. If only one recursive resolver fails, its cache, policy, clock, or forwarding chain remains a candidate.

Clients still reaching the old IP are not contradictory evidence. An old A RRset can be reused until its TTL expires, and its accompanying signature may still be valid. A TTL cannot be retroactively shortened after publication, so keep the old endpoint safe and compatible through the known cache window.

Step 2: Use a CD comparison to enter the DNSSEC branch

Repeat the same name and type against the same failing resolver with CD set:

dig @<affected-resolver> api.example.com A +dnssec
dig @<affected-resolver> api.example.com A +dnssec +cd

If the normal query returns SERVFAIL but +cd returns A, DNSKEY, or RRSIG records, validation failure becomes a leading hypothesis. Google Public DNS domain troubleshooting uses the same contrast—Status 2 with success when validation is disabled—to identify likely DNSSEC problems, and Extended DNS Errors can provide a more specific reason. Continue inspecting the raw path because authoritative timeouts, delegation loops, and other server faults can also produce SERVFAIL.

+cd exposes data the resolver would otherwise reject. Pointing the application at a non-validating resolver or disabling DNSSEC globally turns an availability incident into an integrity risk. If an incident commander uses a validation exception during a broad upstream outage, it still needs a narrow domain scope, a time limit, recorded risk, and explicit exit criteria.

Step 3: Inspect delegation, authoritative data, and signatures separately

First use a trace to enumerate the actual path through root, parent, and child delegation. A trace is an observation of the query path, not a complete cryptographic validation result. Query DS directly from a parent authority, and query DNSKEY, SOA, and the business RRset from every child authority.

dig +trace example.com DS +dnssec
dig @<parent-authoritative> example.com DS +dnssec
dig @<child-authoritative-1> example.com DNSKEY +dnssec
dig @<child-authoritative-1> api.example.com A +dnssec
dig @<child-authoritative-1> example.com SOA +dnssec

Repeat the queries for every authority and compare the NS set, glue, SOA serial, DNSKEY RRset, and RRSIG records. An authoritative server returning A and RRSIG proves that it serves data. Authoritative servers generally do not validate the complete chain from the parent trust anchor on behalf of the requester. Direct-authority success and recursive-validation failure can therefore coexist.

Check that the parent DS owner, algorithm, digest type, and digest correspond to a child DNSKEY; that the DNSKEY RRset has a currently valid signature; that the business A RRset's RRSIG has a plausible key tag, algorithm, inception, and expiration; and that every signed zone in a CNAME chain can establish a chain. An independent validator can identify the failed node, after which the raw records should confirm it.

In this scenario, the parent's DS key tag 18200 has no candidate in the child DNSKEY RRset, and its digest does not match the current key 51900. Correlation with the change time shows that the old KSK/DNSKEY was removed before the parent DS completed a safe transition. The tag mismatch is a clue; digest comparison and signature validation complete the proof.

Step 4: Recover within the security boundary

Freeze automated rollover and unrelated DNS changes. Preserve the change log, zone snapshot, key identifiers, and failed responses. If the previous private key remains in the controlled key system, restore its DNSKEY and generate known-good signatures that establish a chain through the current parent DS. This is often faster than waiting for a parent change. Validate the complete chain in isolation before publishing the same state from all authorities.

If the old key is irrecoverable, DNS and security owners must coordinate a parent DS correction with the registrar and account for propagation. Removing DS returns the child to an unsigned state, weakens its authentication guarantee, and still has cache and parent-change latency. It is an approved, documented, scoped recovery option, not a casual shortcut. Generating a new key with only the same key tag cannot work—the digest will not match.

If the old A endpoint remains safe, continue serving it through the old RRset's TTL window. The old and new endpoints must use compatible critical configuration and authentication policies. Cache flushing can affect only resolvers under the operator's control, so recovery cannot depend on a nonexistent global flush.

Step 5: Prove that DNS and the application recovered

Query the fixed name from independent validating recursive resolvers and regions. Confirm that SERVFAIL is gone, the answer is expected, and a trusted validation result is present. Then perform a complete resolution with a fresh-cache resolver or an independent validation tool so a successful old cache does not hide the defect. Compare SOA serials, DNSKEY records, and signatures on every authority and record remaining signature lifetime.

Validate the application next: health checks, TLS certificates, authenticated requests, and critical APIs on the old and new addresses must work. DNS success with a misconfigured new endpoint is incomplete recovery. Observe failure rate, traffic to the old address, RCODES, lookup latency, and business errors for the original TTL and signature windows until old-cache traffic converges. Cover A, AAAA, CNAME, and any other record types used by the service.

Retain a timeline of publication, first validator failure, root-cause evidence, security decisions, parent/child recovery, and TTL convergence. Close the incident only after independent validation and user outcomes have both recovered.

Step 6: Turn rollover into a falsifiable release process

Prepublish the new DNSKEY so authoritative servers and caches can observe it. Update the parent DS while the old chain is still valid, wait for the relevant TTLs and registration workflow, continuously validate acceptable old and new chains, and only then retire the old DS, signatures, and key. The exact sequence must follow the provider and registry's supported procedure; one fixed number of waiting minutes is not a universal rule.

Release gates should prove that every production signer publishes mutually valid, supported DNSKEY and signature data; the parent DS reaches the intended key; RRSIG records have adequate remaining lifetime; sampled business RRsets validate; and alerts reach a human owner. DENIC's 2026 report also shows a defect that required multiple HSMs and escaped a single-HSM test, while existing validation alerts were not handled correctly. Prevention therefore needs production-topology coverage and an exercised alert loop.

High-Quality Sample Answer

“I would first identify the recursive resolver, query type, and timestamp used by a failing client and save the complete response. For the same resolver, api.example.com A +dnssec returns SERVFAIL, while adding +cd returns A and RRSIG. That controlled contrast prioritizes DNSSEC validation. The CD response is still untrusted and cannot be passed to the production application.

I would use +trace to observe the actual delegation, query DS directly from the parent, and query DNSKEY, SOA, A, and RRSIG from every child authority. I would test whether the parent DS algorithm and digest match a DNSKEY, whether all authorities share a serial, whether each RRSIG key tag and time range is plausible, and where an independent validator labels the chain bogus. A direct authority answer does not prove the chain, because that server supplies records while the recursive validator still has to connect the child key to the parent DS and a trust anchor.

The scenario's DS 18200 has no matching DNSKEY, and its digest does not match key 51900, showing that the old key was retired too early. I would freeze changes, restore the old DNSKEY and valid signatures that match the current DS from the controlled key system, validate them in isolation, and publish consistent state from all authorities. If the old key cannot be restored, the security owner and registrar must coordinate a DS correction. I would not disable validation globally or assume a global cache flush exists.

After recovery, I would run cold and warm queries from multiple regions and independent validating resolvers, checking RCODE, authenticated state, A/AAAA/CNAME, TTL, and authority serials. I would also test TLS and critical APIs on both old and new addresses. Clients may legitimately use the old IP during its TTL, so that endpoint stays compatible until traffic converges. Finally, I would add key prepublication, overlapping chains, a parent-DS release gate, signature-expiry monitoring, multi-authority consistency checks, production-topology exercises, and verified alert delivery to the rollover process.”

Common Mistakes

• Changing application servers immediately after seeing SERVFAIL → DNS has not produced a trusted address, so the application may receive no traffic → Pin the resolver and inspect the RCODE, EDE, and delegation first.
• Treating +cd success as trusted data → CD skips validation and can return broken data → Use it only as a control, then validate DS, DNSKEY, and RRSIG.
• Ruling out DNSSEC because direct authority queries work → An authority can serve data that does not validate through the parent chain → Test data availability and cryptographic validation separately.
• Comparing only key tags → A key tag selects candidates but does not replace algorithm and digest validation → Compute or independently validate the complete DS-to-DNSKEY relationship.
• Calling the old IP and SERVFAIL one cache bug → Valid old cache and failed new validation can coexist → Segment observations by resolver, cache age, and TTL.
• Deleting the old key immediately during rollover → Parent DS records and recursive caches can still depend on the old chain → Keep overlapping chains until validation and TTL gates pass.
• Disabling DNSSEC globally as mitigation → Resolution returns at the cost of authentication → Restore a known-good chain first; scope, time-box, approve, and reverse any exception.
• Lowering TTL after publication → Cached answers continue using the TTL they received → Plan TTL before migration and keep the old endpoint compatible during incidents.
• Testing only dig, not the product → The new IP can still have certificate, routing, or configuration failures → Verify resolution, TLS, critical APIs, and user error rates.
• Testing one signer only → Multi-node or multi-HSM production paths can generate different results → Exercise the real topology and compare every signing node.

Follow-Up Questions and How to Answer

Follow-up 1: How do you quickly distinguish NXDOMAIN, an empty answer, and SERVFAIL?

Inspect the RCODE and Authority section. NXDOMAIN claims the queried name does not exist. NOERROR with an empty Answer can mean that the name exists without the requested type, usually with an SOA. SERVFAIL means the server could not produce an acceptable result; DNSSEC bogus data, upstream timeout, delegation failure, and internal faults can all cause it. Determine which layer produced the response and whether a negative answer has authenticated denial evidence rather than relying on the browser message.

Follow-up 2: Why do some resolvers succeed while others fail?

Compare whether they validate DNSSEC, which RRset they cache, when it expires, their upstreams, clocks, and local policies. A successful resolver might still serve a valid pre-rollover cache or might not validate. A failing resolver may have refreshed and discovered the broken chain. Resolver diversity is a clue; CD/AD behavior, raw records, and cache age finish the attribution.

Follow-up 3: Can cache flushing restore every user immediately?

An operator can flush only browsers, operating systems, or recursive resolvers it controls. External resolvers and endpoints follow the TTL they already received, and a domain owner has no global flush API. Restoring a known-good signed chain while keeping the old endpoint compatible covers both cold queries and clients with old answers. Pre-migration TTL reduction works only when done early enough for the previous TTL to expire.

Follow-up 4: How do KSK and ZSK rollover boundaries differ?

A common deployment uses a KSK to sign the DNSKEY RRset and a ZSK to sign business RRsets such as A and AAAA. KSK rollover crosses the parent DS boundary and therefore organizations and caches. ZSK rollover is usually contained in the child zone but still requires overlapping DNSKEY and RRSIG validity. Providers can use different key models, so derive the answer from actual DNSKEY flags, signers, and the supported procedure instead of applying labels mechanically.

Follow-up 5: What release gates would you add to DNSSEC rollover?

Run the rollover in a preproduction environment with production-equivalent topology and prove that every signer produces mutually valid output. Before release, check the parent DS, child DNSKEY, supported algorithms, RRSIG inception/expiration, SOA serial, and authority consistency, then execute cold queries through at least two independent validators. Alerts need a named owner, an escalation path, and a rehearsed rollback. Inject expired signatures, missing keys, and node inconsistency to prove that the incident will be detected and handled.