Prompt and Applicable Context
At 09:10, checkout degrades in one region: 5xx rises from 0.3% to 5.2%, p95 latency rises from 280 milliseconds to 2.6 seconds, and a release finished 10 minutes earlier. No data loss is confirmed. Explain how you would assess severity, contain impact, narrow the scope, test hypotheses, validate recovery, and preserve evidence for root-cause analysis during the first 30 minutes.
This is a cross-layer troubleshooting question for software engineers, SREs, DevOps engineers, platform engineers, and technical support engineers. It does not assign the fault to a client, service, database, network, or third-party dependency. The core skill is using evidence to converge from a symptom to a failure boundary while controlling operational risk.
The times, error rates, and latency are interview scenario inputs, not universal alert thresholds. Keep service restoration separate from root-cause proof. During a severe incident, a team may use a tested reversible mitigation while preserving logs, trace samples, and change history. A root-cause claim still needs evidence that distinguishes competing hypotheses.
What the Interviewer Evaluates
The first signal is an accurate problem statement. “The system is slow” cannot direct action. A strong answer establishes expected versus actual behavior, start time, affected users and regions, continuous versus intermittent failure, business loss, data integrity, and security risk. Those facts determine whether to declare an incident.
The second signal is prioritization. When users are actively failing, the immediate goal is to reduce impact safely. A rollback, feature disable, traffic shift, or degradation can also expand an incident when a database change is incompatible, fallback capacity is insufficient, or protocols have diverged. State the conditions, approver, reversal path, and recovery signals for the chosen mitigation.
The third signal is hypothesis-driven diagnosis. Logs, metrics, and traces are evidence types, not a sequence by themselves. A strong answer proposes a short list of falsifiable hypotheses, predicts the evidence each would produce, and chooses the lowest-risk check that separates them. A release near the start time raises one hypothesis's priority; temporal correlation does not prove causation.
The fourth signal is incident coordination. If several people change configuration, restart instances, or enable verbose logging simultaneously, they alter the scene and destroy attribution. A mature answer names an incident lead, an operator, and a communicator, freezes unrelated changes, and records every action and observed result. Only an explicitly authorized path changes production at a given time.
Finally, the interviewer looks for a verification loop. Error rate can fall because traffic subsided, and a restart can temporarily hide a leak. Compare recovery with a healthy control, inspect latency, errors, saturation, backlog, and business success, then separately reconcile duplicate charges and missing orders. Immediate recovery and long-term prevention need distinct exit criteria.
Questions to Clarify Before Answering
- Is this a methodology prompt or a past-experience prompt? Use the supplied scenario for a methodology prompt. If the interviewer asks for experience, switch to a verifiable incident, identify personal authority and team roles, and do not invent production work.
- What is the user impact and severity? A low-volume internal tool may permit observation. Broad checkout failure, data damage, or a security event requires immediate escalation, restricted changes, and a larger response team.
- Could data or security be compromised? With latency alone, partial traffic may remain safe. Possible duplicate charging, unauthorized access, or corruption calls for pausing writes or isolating the path and raises the approval bar for recovery.
- What did the release include? Code, configuration, database migrations, dependency versions, and infrastructure have different rollback risks. Rolling an application back after incompatible schema writes may worsen the incident.
- Where is the healthy control? Another region, an old-version instance, a tenant without the feature, or other endpoints in the same region can separate version, region, input, and dependency causes. Without a control, create a low-risk synthetic request or read-only check.
- What is my role and authority? The incident lead holds global state, the operator modifies the system, and the communicator updates stakeholders. An investigator should present evidence and escalate rather than exceed production authority.
- Which mitigations are known to be safe? Feature flags, traffic shifts, a stable stack, degradation, and rollback are options only after capacity, state compatibility, and reversal steps are confirmed.
30-Second Answer Framework
“I first establish scope, data risk, and severity, declaring an incident and freezing unrelated changes when needed. While users are affected, I choose a tested, reversible mitigation with sufficient capacity and preserve evidence. I compare time, region, version, request cohorts, and dependencies; write a few falsifiable hypotheses; use metrics for scope, traces for the slow boundary, and logs for the specific failure. I change one documented variable at a time. Recovery requires healthy SLIs, business success, drained backlogs, and data reconciliation, followed by root-cause and prevention work.”
This opening establishes order. A deep answer must also state the decision conditions and the evidence that would make you abandon a hypothesis.
Step-by-Step Deep Answer
Start with a symptom contract so the team investigates the same problem:
- Expected and actual: baseline 5xx is 0.3% and p95 is 280 milliseconds; current values are 5.2% and 2.6 seconds.
- Time: detected at 09:10 after a release finished 10 minutes earlier; find the true start and whether the failure is continuous.
- Scope: one region is currently confirmed; continue splitting by endpoint, version, tenant, device, and request input.
- Impact: count failed checkouts and business loss, and determine whether a user workaround exists.
- Integrity: check duplicate charges, charged-without-order cases, missing events, and unauthorized access separately. “Not confirmed” does not mean “proven absent.”
If the impact crosses the incident threshold, establish one coordination plane. The incident lead maintains priorities and the decision log; the operator performs production changes; the communicator posts user impact, known facts, current action, and next update time on a fixed cadence. Freeze unrelated releases. Record the timeline, dashboard snapshots, representative trace IDs, change versions, and each intervention. Preserving evidence should not block mitigation, but blind restarts erase in-memory evidence and change subsequent observations.
Decide whether to mitigate before diagnosing further. Use a clear comparison: is current harm greater than the worst credible risk of the mitigation? Stop writes or isolate the path when data damage is possible. Roll back or shift traffic when the recent change is safely reversible and the stable target has enough capacity. Prefer a feature flag, rate limit, or compatibility path when schemas, message formats, or external side effects have changed. Every mitigation needs an expected signal, observation window, and undo action.
Begin diagnosis with contrasts, not the longest log file. Find a boundary where only one side is unhealthy across five dimensions:
- Time: what code, configuration, certificates, traffic, and dependency state changed near onset?
- Region: is the same version healthy elsewhere, and is the old version also unhealthy in the affected region?
- Version: within one region, do old and new instances differ in error rate and latency?
- Request: is failure concentrated by endpoint, tenant, payment method, device version, or input size?
- Dependency: does slow time accumulate in the client, edge, application, database, cache, or third-party call?
Put candidate causes in a hypothesis ledger rather than leaving them in chat:
| Hypothesis | Prediction if true | Lowest-risk check | How the result changes direction | |---|---|---|---| | New release regression | New version fails in-region while old version is healthy | Split SLIs by version and compare traces for the same input | Both versions failing lowers its priority | | Regional configuration or dependency | The same version fails only in one region | Compare configuration digests and downstream span latency | Failure elsewhere redirects toward input or a global dependency | | Triggering request cohort | Errors cluster in a describable request group | Segment on non-sensitive fields and replay a redacted sample | Uniform distribution redirects toward shared resources | | Capacity or queuing | Saturation, queues, and tail latency rise together | Compare traffic, concurrency, pool waits, and resource percentiles | Idle resources redirect toward locks, timeouts, or dependencies |
Order checks by likelihood, user impact, time to answer, and production risk. Metrics locate when and where a problem begins. Distributed traces show where one request spends time. Structured logs explain specific errors, retries, and state transitions. Code, configuration, and deployment history describe what changed. One log line cannot represent the population, while an aggregate cannot explain one state machine; correlate them through a request ID and the same time window.
Change one variable per experiment and write down how the result would falsify the hypothesis. Sending a bounded synthetic request through an old configuration can distinguish configuration from input. Enabling verbose logs across production may add I/O and latency, expose sensitive data, and alter the failure. Run the most likely, safest, and most discriminating test first, using a healthy replica, staging, or limited traffic. When safe reproduction is impossible, retain “most likely cause” language rather than promoting correlation to certainty.
After a fix or mitigation, verify with signals independent of the action: 5xx, p50/p95/p99, successful checkout rate, resource saturation, connection-pool and queue backlog, downstream errors, retries, and alert state. Compare at least one healthy region or stable version and observe a predetermined window long enough to cover normal variation. Reconcile orders with payments for duplicate, missing, and stuck states. End mitigation only after user outcomes recover, backlogs drain, and integrity checks pass.
Complete root-cause analysis after stabilization. Separate the direct trigger, technical conditions that amplified impact, and process gaps that delayed detection. Prevention items need an owner, deadline, and verification method: canary regional configuration, synthetic dependency probes, end-to-end request IDs, automatic fallback criteria, or a rehearsed runbook. Two tested improvements are more useful than ten unowned recommendations.
High-Quality Sample Answer
All investigation findings and numbers below are fictional interview scenario data. They demonstrate a spoken answer and are not a real incident or universal thresholds.
“I would treat this as an active checkout incident. 5xx moved from 0.3% to 5.2%, p95 moved from 280 milliseconds to 2.6 seconds, and users are still failing. I would immediately check for duplicate charges, missing orders, and security risk; until those are ruled out, ‘no data loss confirmed’ is not a safety claim. I would freeze unrelated releases, assign one incident lead, one production operator, and one communicator, and preserve trace samples and change records.
The recent release is a high-priority hypothesis, not a conclusion. I would compare old and new instances in the affected region, then use a healthy region as a second control. Suppose affected-region v42 has 5.4% 5xx and v41 has 5.0%, while v42 in another region has 0.4%. That weakens a code-version-only explanation and raises a regional configuration or dependency problem.
I would then inspect traces for the same checkout cohort. Suppose 91% of failures time out in the payment-token service call, where p95 is 2.3 seconds in the affected region and 180 milliseconds in the healthy one. The change log shows regional configuration v17 was released at 09:00 and moved that call to a new proxy path. I would preserve representative evidence, confirm v17-to-v16 compatibility, rollback authority, and existing-connection behavior, then revert only that regional configuration. I would not simultaneously roll back code, restart, and scale.
One green dashboard after the revert is insufficient. Suppose over the next 15 minutes 5xx falls to 0.4%, p95 falls to 320 milliseconds, queues drain, and checkout volume recovers. Order-to-payment reconciliation also finds no duplicate charges, missing orders, or stuck states. The controlled configuration reversal plus those independent signals supports the proxy path as the direct trigger. I would still reproduce it in staging and ask why regional configuration lacked a canary and dependency-latency guardrail.
Prevention would include a small-traffic rollout for regional configuration, per-region synthetic probes and tail-latency alerts for payment tokenization, automatic fallback criteria covering both errors and latency, and a runbook for configuration rollback and data reconciliation. Each item gets an owner, due date, and an incident exercise as acceptance.”
Common Mistakes
- Declaring the recent release the root cause → Traffic, certificates, or downstream changes can coincide with it → Build version and region controls, then run a check whose prediction can fail.
- Opening every log first → Without time, scope, and hypotheses, more data creates more irrelevant anomalies → Quantify the symptom, then use metrics and traces to narrow the component and cohort.
- Rolling back unconditionally → An old version may be incompatible after schema, message, or state changes → Check state compatibility, capacity, and the reversal path before choosing mitigation.
- Letting everyone try a fix → Concurrent changes destroy attribution and may expand the incident → Assign roles and route production changes through one recorded authority.
- Calling a restart the root cause → Restarting resets queues, connections, or memory and only proves that state was cleared → Find what creates that state and watch whether it accumulates again.
- Checking only average latency → A severe failing tail can disappear in the mean → Inspect errors, latency percentiles, traffic, and saturation, segmented by affected cohort.
- Enabling verbose logs without a risk check → Logging can increase I/O and latency or expose sensitive fields → Limit instances, duration, and fields, with an automatic shutoff.
- Skipping data reconciliation after dashboards recover → Timeout retries may have caused duplicate charges, missing orders, or backlog → Reconcile business records and external side effects explicitly.
- Ending the review with “improve monitoring” → No metric, owner, threshold, or test means no verifiable improvement → Create executable actions and accept them through drills or fault injection.
Follow-Up Questions and Responses
Follow-up 1: The rollback finishes, but the incident does not improve. What next?
Record that rollback completed and verify traffic actually reaches the old version; otherwise “rollback failed” may itself be a deployment-system fault. If the old version remains unhealthy, lower the code-regression hypothesis and compare regional configuration, dependency spans, certificates, DNS, traffic mix, and shared resources. Do not redeploy the new version immediately unless restoring it has a clear benefit and assessed risk.
Follow-up 2: What changes if data corruption is suspected?
Containment takes higher priority. Pause relevant writes, isolate affected tenants, or enter read-only mode; preserve audit logs and raw events; and require data or security owners in recovery approval. Validate a repair on copies or isolated data, inventory the affected range, reconcile it, and use a reversible correction. Green service responses alone do not authorize reopening writes.
Follow-up 3: What if there are scattered logs but no distributed tracing?
Use gateway access logs, timestamps, instances, request types, and existing correlation fields to approximate the boundary. Send a low-risk synthetic request with a known identifier through each step. Do not start printing request bodies across production. After the incident, add an edge-to-dependency request ID plus latency, status, and version fields at critical boundaries; missing observability is an amplifying condition with an owner and acceptance test.
Follow-up 4: Several teams insist the fault is in someone else's system. How do you proceed?
The incident lead assigns component investigations by user impact, not organizational blame. Everyone works from one timeline and hypothesis ledger and submits testable predictions and evidence, such as “this request left our service successfully in 120 milliseconds, while the next span took two seconds.” Production authority stays singular, and the communicator consolidates facts so each team does not mutate the system independently.
Follow-up 5: The issue is intermittent and cannot be reproduced reliably. How do you state a root cause?
Expand safe passive evidence first: sample successes and failures, retain version and input dimensions, and capture resource and dependency state. Look for natural controls that separate hypotheses. Use fault injection or traffic replay in isolation, without creating risky production conditions for certainty. If evidence supports only probability, state the most likely factor and remaining unknowns, then add guardrails and observability that reduce impact on recurrence.