Prompt and context
This question tests whether you can turn an intermittent symptom into a repeatable evidence chain. A concurrency bug may involve a data race, lock ordering, message interleaving, timeout, or external input; a single log usually records the outcome without preserving the ordering that caused it. Record/replay can preserve nondeterministic events from one execution and replay them, but it is constrained by platform support, system calls, overhead, and unrecorded external state. Cover safe sampling, layered diagnosis, replay boundaries, and falsification after the fix.
What the interviewer is evaluating
- Whether you define invariants, impact, and a minimal reproduction before enabling full tracing.
- Whether you distinguish data races, logical races, deadlocks, livelocks, and duplicate external delivery.
- Whether you can state what a race detector, ordinary logs, a timeline, and record/replay can actually prove.
- Whether you can design progressive verification under privacy, overhead, and production-risk constraints.
Clarifying questions to ask first
Clarify what “lost” means, affected requests, time window, version, machine architecture, and whether multiple processes or services are involved. Can you obtain request IDs, thread or goroutine IDs, lock metrics, and queue metrics? Does the symptom look like unsynchronized shared memory, a broken state machine, or duplicate and out-of-order messages? Is success defined by no race report, preserved invariants, or a business loss rate below a threshold?
A 30-second answer framework
I would preserve evidence and define data invariants first, then use low-cost logs and metrics to narrow the trigger. In a controlled test, run a race detector, thread checks, stress tests, and scheduler perturbation. If the failure remains difficult to reproduce and the platform is supported, record one failure and replay it repeatedly while inspecting locks, atomic operations, and message ordering to find the first broken invariant. Repair ownership, synchronization, or the state machine rather than adding sleeps. Replay the original failure, expand interleaving tests, and verify with business reconciliation and fault injection. Redact, limit, and delete recordings; replay is evidence within a boundary, not proof about every platform and dependency.
Step-by-step deep dive
1. Write invariants and evidence boundaries
Rewrite “a record was lost” as assertions such as unique sequence numbers, conserved account balance, or a message remaining retryable until acknowledgement. Mark shared state, owner, lock or atomic protocol, external side effects, and recovery path. Separate observed facts, plausible hypotheses, and branches still to verify so one error log does not become an unjustified root cause.
2. Narrow the trigger with low-cost signals
Add correlation IDs to requests, tasks, threads, or goroutines and record state transitions and key queue lengths instead of every instruction. Use load, latency injection, scheduler perturbation, and repeated runs to amplify interleavings. Document which code paths the race detector covers and which it cannot observe; a clean run does not prove the business logic is race-free.
3. Decide when to use record/replay
Use record/replay only when a failing execution can be captured within the same platform and input boundary and ordinary logs cannot preserve the decisive ordering. Tools may record system-call results, thread scheduling, or other nondeterministic inputs so a debugger can move forward, reverse, and inspect state. Confirm supported architectures, process model, sandbox restrictions, storage capacity, and overhead before enabling it on core production traffic.
4. Trace backward from the first broken invariant
During replay, set breakpoints or watchpoints and compare shared variables, lock owners, message sequence, and commit results between a correct and failed execution. Find the first divergence rather than guessing backward from the final missing record. For each candidate interleaving, write the happens-before relation: which write must precede a read and which acknowledgement must follow persistence. If an external service cannot be replayed, provide deterministic mocks or recorded inputs and state exactly what that evidence excludes.
5. Repair the design, not just the schedule
Prefer changing ownership, lock scope, atomic state transitions, or acknowledgement protocol so correctness follows an explicit invariant instead of a sleep or thread-priority change. The repair may need idempotency keys, a single writer, a transaction boundary, cancellation propagation, or sequence markers. Recheck exceptions, timeouts, retries, process crashes, and recovery paths so the defect is not moved to another branch.
6. Close with layered verification
Replay the original failure and confirm the invariant holds. Then run race detection, stress, and scheduler-perturbation tests across inputs and machine configurations. Add business reconciliation, duplicate submission, fault injection, and rollback drills; measure loss, duplication, latency, and resource overhead. Retain the smallest redacted trace with tool version, build flags, and conclusions for a limited period. If external coverage cannot be proven, limit the claim to “not reproduced within this boundary.”
Model high-quality answer
I would define invariants such as balance conservation, unique sequence numbers, and acknowledgement ordering, then protect and redact samples. I would narrow the trigger with correlation IDs, state transitions, queue metrics, load, and scheduler perturbation before running a race detector in a controlled environment. That detector covers races that actually execute, not every logical interleaving. If the platform supports it and reproduction remains hard, record one failure and use record/replay to move backward and forward in a debugger until the first invariant breaks and its happens-before relation is clear. Repair ownership, locking, or the state machine instead of adding sleeps, and review retries, timeouts, crashes, and recovery. Finally replay the failing trace, run race and interleaving stress tests, and verify with reconciliation and fault injection. Record the platform, dependency, overhead, and evidence boundary, and delete traces after a defined retention period.
Common mistakes
- Guessing from the final error without defining an invariant.
- Treating a clean race-detector run as proof that all concurrency logic is correct.
- Enabling full production recording without considering privacy, disk, CPU, and trace-storage risk.
- Recording timestamps but not the state or input needed to distinguish interleavings.
- Hiding a race with sleeps, more retries, or thread-priority changes.
- Replaying only the happy path and ignoring crashes, timeouts, cancellation, and external boundaries.
- Skipping business reconciliation, stress, fault injection, and a reproducible regression record after repair.
Follow-up questions and responses
Can record/replay prove there is no concurrency bug?
No. It proves that one recording can be replayed in a supported environment and input boundary and helps locate a problem in that execution. Other schedules, platforms, external services, and paths still require race detection, stress, and fault injection.
What if replay changes the timing?
Recording can add overhead or support only selected events. Compare trigger conditions and key metrics before and after recording, then cross-check with multiple traces, scheduler perturbation, and independent stress tests. If representativeness cannot be maintained, treat replay as a lead rather than sole proof.
How should sensitive failure traces be handled?
Use field allowlists, redaction, or tokenization at collection, restrict tenant and access scope, and apply encryption, retention, and deletion controls. Replay the smallest input in an isolated environment and never copy raw credentials or customer data into debugging infrastructure.
When should the issue become an architectural repair?
Escalate when multiple threads or services maintain the same invariant or when lock and retry patches cannot establish clear ownership. Move to a single writer, explicit state machine, transaction boundary, or verifiable message protocol, with migration, compatibility, and rollback plans. Replacing the debugger alone is not an architectural fix.