Prompt and scope
Design a shadow-traffic system for a service receiving 20,000 requests per second. The primary request must keep its existing p99 latency, while the shadow version may lag. The system should sample per endpoint, persist enough evidence to replay a mismatch, compare meaningful response fields, and never let a mirrored request send a payment, email, or other external side effect.
The core distinction is request mirroring, not packet capture. A load balancer can forward a fire-and-forget copy to a mirror backend, while the primary response remains authoritative. Network packet mirroring has different targets and encapsulation, so it is not a substitute for an application-aware replay path.
What the interviewer is testing
The interviewer is looking for a safe comparison harness, not a second production service with copied traffic. Strong answers preserve the primary latency path, remove credentials, isolate writes, make sampling reproducible, and explain why a byte-for-byte diff is often wrong.
They will probe what happens when the capture queue is full, when the shadow is slower, when a response contains timestamps, and when the primary already performed a write. Treat every one of those as an explicit policy.
Clarifications that change the design
- Are requests read-only? Read-only requests can be replayed directly; mutating requests need a sandbox, stubs, or a synthetic identity.
- Is comparison exact or semantic? Exact comparison suits deterministic JSON; masks or invariant checks suit timestamps and generated IDs.
- Can captured bodies contain personal data? If yes, redact before storage, encrypt references, and define retention and access audit.
- Is the goal migration correctness, latency, or capacity? The metric and sampling policy change with the goal.
A 30-second answer framework
“I would capture an authenticated request after authorization but before external writes, sanitize secrets and personal data, and enqueue a deterministic sample asynchronously. The primary response remains authoritative. Isolated shadow workers replay against a sandbox with bounded rate and a separate credential. A diff service compares status, selected fields, invariants, and latency while ignoring declared nondeterminism. Durable captures support investigation, but queue overflow drops the mirror rather than slowing users. Promotion requires a defined divergence and error budget, plus checks that writes, privacy, and shadow capacity are safe.”
Step-by-step design
1. Capture without touching the critical path
Place a capture hook after authentication and authorization so the system knows the endpoint and tenant policy, but before any irreversible external call. Copy only the fields needed by the shadow. Strip authorization, cookies, user tokens, and payment material; replace them with a short-lived shadow credential or a fixed test identity.
Publish a MirroredRequest containing a request ID, trace ID, endpoint, sanitized headers, body reference, capture time, target version, and sampling configuration version. The enqueue operation must have a bounded timeout. If the local buffer or durable queue is full, record a drop metric and return the primary response.
2. Make sampling reproducible
Hash a stable request or trace ID and compare it with the configured sample rate. This keeps retries and investigations deterministic. Apply a second token bucket per endpoint and tenant to cap mirror RPS; the lower of the percentage and token budget wins.
Keep the sample configuration version with every capture. A later report can then distinguish a real divergence from a changed sampling rule.
3. Replay in an isolated environment
Workers read the durable queue and call the shadow target with a short timeout. The shadow must use a separate database or transactionless fixture, stub payment and email providers, and disable outbound callbacks. For read paths, a read-only replica or sanitized snapshot is usually safer than production storage.
At-least-once delivery is acceptable for the capture queue because replay is tagged by request ID. The worker records SHADOW_ERROR, DROPPED, or COMPLETED; a shadow timeout never retries indefinitely or blocks the primary.
4. Compare responses by meaning
Compare status codes first. For successful JSON responses, remove fields explicitly declared nondeterministic, then compare selected paths or canonicalized body hashes. Add domain invariants such as “total is nonnegative” or “the returned user belongs to the requested tenant.” Compare latency separately; a response can be correct but too slow for promotion.
Store both hashes, the diff paths, primary and shadow latency, deployment version, and a sample request reference. Do not store raw sensitive bodies in the report.
5. Define promotion gates
Create thresholds before replay begins: divergence rate, shadow error rate, latency regression, queue age, and redaction failures. Segment reports by endpoint, tenant tier, region, and response class; an aggregate green rate can hide a broken payment endpoint.
Use a sliding window with minimum sample counts. A 1% divergence threshold is an example policy, not a universal rule. Promotion also requires zero unsafe side effects, acceptable shadow capacity, and reviewed examples for every high-severity diff.
6. Operate and recover
Pause replay without disabling capture when the shadow is overloaded. Expire captures according to privacy policy, and audit access to stored request references. If a shadow version is rolled back, keep its reports linked to the deployment so later analysis does not mix versions.
Test queue loss, duplicate captures, stale configuration, credential leakage, side-effect attempts, nondeterministic fields, shadow timeouts, and a region failure. The success criterion is that the primary service remains within its SLO while the shadow produces actionable, reproducible evidence.
High-quality sample answer
“I would build an application-aware mirror hook after auth and before external writes. It strips real credentials and sensitive fields, stores a body reference, and samples deterministically by trace ID. The primary path never waits for the mirror; a bounded queue can drop mirror work under pressure.
“Isolated workers replay to a shadow version with a separate identity, database, and stubbed providers. Each replay has a request ID and timeout. The comparison service checks status, selected JSON paths, domain invariants, and latency, ignoring only fields declared nondeterministic. Reports retain hashes, diff paths, versions, and sanitized examples.
“Before promoting, I would set endpoint-level gates for divergence, errors, latency, queue age, and privacy failures, require a minimum sample, and inspect severe diffs. Mutating endpoints need a sandbox or are excluded. I would validate queue overflow, duplicates, credential stripping, side effects, and regional failure while proving that the primary SLO is unchanged.”
Common mistakes
- Mistake → Failure → Fix: Replaying production credentials → shadow calls can leak or mutate real systems → strip secrets and use isolated identities and providers.
- Mistake → Failure → Fix: Waiting for the shadow before responding → migration testing increases user latency → enqueue asynchronously and drop mirror work under pressure.
- Mistake → Failure → Fix: Comparing raw bodies only → timestamps and generated IDs create false positives → canonicalize and compare declared fields plus invariants.
- Mistake → Failure → Fix: Mirroring every request forever → storage, privacy, and shadow capacity grow without bound → use deterministic sampling, quotas, retention, and access audits.
- Mistake → Failure → Fix: Using one global divergence rate → a small critical endpoint can fail inside a green aggregate → gate by endpoint, tenant, region, and severity.
Follow-up questions and responses
What if the endpoint writes to a database?
Route the shadow to a disposable database or transaction stub, and replace external calls with fakes that record intent. If the write cannot be isolated, exclude the endpoint and state the coverage gap instead of pretending the result is safe.
What if the capture queue is full?
Fail open for the primary request: drop the mirrored copy, increment a labeled metric, and alert when the drop budget is exceeded. Backpressure on the primary would invalidate the main safety property.
How do you handle personally identifiable information?
Classify fields at capture time, redact or tokenize before durable storage, encrypt body references, limit retention, and audit access. A hash of a sensitive payload can still be identifying, so it is not automatically safe.
Can packet mirroring replace application mirroring?
No. Packet mirroring copies network traffic to an analysis target, while application mirroring knows endpoint semantics, credentials, tenant policy, and response comparison. Use packet mirroring for network inspection and an application-aware path for behavior validation.