System design interview: How would you build an A/B platform that detects experiment-quality failures?

Prompt and context

A product organization runs hundreds of A/B tests each day. The platform assigns users, records exposure, computes metrics, and monitors quality. Past experiments suffered from uneven allocation, sample-ratio mismatch, and invalid conclusions caused by repeated peeking. Design the system from experiment configuration through decision review, including data delay, statistics, alerts, and permissions.

What the interviewer is testing

• Whether the candidate clarifies the unit of experimentation, randomization level, mutual-exclusion planes, and metric definitions.
• Whether they separate assignment, trigger, exposure, and outcome events.
• Whether they design randomization checks, SRM detection, and data-quality alerts.
• Whether they know that a fixed-horizon test cannot be peeked at freely and can propose sequential or anytime-valid methods.
• Whether alerts, audit, reruns, and release gates form one workflow.

Clarifying questions to ask first

1. Is randomization by user, device, session, or request? Can experiments run orthogonally?
2. How many experiments and users run daily, and what are metric latency and retention windows?
3. How are primary metrics, guardrails, and minimum detectable effects declared?
4. After SRM or missing exposure data, should traffic pause, the test be invalidated, or it be rerun?
5. Who can inspect assignments, change stopping rules, and approve a launch?

A 30-second answer

“I would split the platform into configuration and randomization, event collection, analysis, and a decision gate. Assignment uses a stable hash to map users to buckets and separates mutually exclusive from orthogonal experiments. The event path records assigned, triggered, exposed, and outcome states. Quality services continuously check bucket distribution and SRM, while analysis uses sequential or anytime-valid methods for monitoring and stopping. Every decision carries data, code, and audit versions; a quality alert blocks release instead of becoming only a chat message.”

Step-by-step deep answer

1. Define experiment and randomization boundaries

Configuration includes experiment ID, version, unit, traffic split, mutual-exclusion plane, target metrics, guardrails, minimum detectable effect, and stopping rules. The hash uses a stable unit key and experiment seed so one user does not switch variants per request. Mutually exclusive tests share a plane; orthogonal tests use separate planes and record collision policy.

2. Separate assignment from outcome events

Recording only conversion cannot prove that a user saw the variant. Events should include assignment, trigger, exposure, and outcome with experiment version, variant, time, an anonymous unit hash, and source. Version the event schema, allow late events into a recomputation window, and deduplicate with an idempotency key.

{
  "experiment": "checkout-copy-v3",
  "experimentVersion": 7,
  "unitHash": "u_8f2c",
  "variant": "treatment",
  "event": "exposure",
  "eventTime": "2026-08-01T12:00:03Z",
  "schemaVersion": 2,
  "idempotencyKey": "u_8f2c:checkout-copy-v3:7:exposure"
}

Immutable events and version identifiers connect assignment to analysis, so changing configuration cannot reinterpret historical data.

3. Monitor randomization and SRM

Randomization checks compare bucket distribution with expectation. SRM checks observed triggered samples against the configured ratio. Inspect both assigned and triggered levels because eligibility, client versions, or instrumentation can introduce bias only after assignment. Alerts should distinguish transient data delay, real allocation defects, and traffic changes, while retaining diagnostic slices.

4. Handle continuous observation and stopping

A fixed-horizon test that is checked daily and stopped when significant inflates Type I error. Use group sequential methods, SPRT, or anytime-valid confidence sequences, and record boundaries, alpha, beta, MDE, and observation times in the experiment version. A product owner seeing a lead cannot bypass the statistical gate; guardrail failure or a safety-boundary crossing should stop traffic first.

5. Design streaming and batch paths

A streaming path consumes Kafka-like events and updates allocation-quality and data-latency alerts within minutes. A batch path deduplicates, repairs late data, computes slices, and produces the final report. Both share schemas, experiment versions, and metric definitions. Streaming results are provisional; only a batch watermark makes them release evidence.

6. Connect decisions to audit and release gates

Experiment states include draft, running, paused, invalid, concluded, and archived. Changing a seed, metric, or stopping rule creates a new version and freezes the old result. The decision service emits data watermark, method, sample size, SRM state, guardrail state, and audit links. Release accepts only a concluded version whose quality checks pass.

High-quality sample answer

“I would first lock the randomization unit and experiment version, then separate assignment, trigger, exposure, and outcome events. Randomization uses a stable hash and mutual-exclusion or orthogonal planes. The event pipeline uses schema versions and idempotency keys for duplicates, late data, and replay. Quality services check bucket distribution and SRM at assigned and triggered levels and expose root-cause slices. Fixed-horizon tests cannot be damaged by arbitrary peeking, so the analysis uses sequential or anytime-valid methods and records boundaries and observation times. Streaming data drives alerts, while a batch watermark produces the report. Decisions carry configuration, code, data, and audit versions; SRM, missing exposure, or guardrail failure blocks release.”

Common mistakes

• Storing only final metrics → allocation or exposure loss is invisible → retain assignment, trigger, exposure, and outcome.
• Randomizing each request instead of each user → one user sees multiple variants → use a stable unit key and seed.
• Checking p-values daily and stopping early → the fixed-horizon assumption breaks → use sequential or anytime-valid methods.
• Calling SRM an outcome significance result → an abnormal ratio may invalidate the test → pause decisions and diagnose allocation and eligibility.
• Driving release directly from streaming data → late and duplicate events can rewrite conclusions → require a batch watermark and release gate.

Follow-ups and responses

Why check both assigned and triggered samples?

Assignment can be uniform while only eligible users who reach the page trigger the test. Eligibility logic, client versions, interceptors, or missing exposure events can distort the triggered ratio. Both levels distinguish a randomization defect from an exposure-path defect.

How do you support mutually exclusive and orthogonal tests?

Mutually exclusive tests share a traffic plane and seed, so a user enters only one test there. Orthogonal tests use separate planes and can combine, but the platform records a collision matrix and disables risky combinations.

What does an anytime-valid method solve?

It permits continuous monitoring and data-dependent stopping while providing time-uniform error control. The platform still declares the target effect, power, guardrails, and business loss; arbitrary metrics and stopping reasons do not become statistical guarantees.

Should an SRM failure trigger an automatic rerun?

Freeze the decision and retain raw data first. Slice by time, client, country, eligibility, and plane to find the cause. Rerun only after allocation or instrumentation is fixed and the product owner accepts a new analysis window; automatic reruns must not hide an invalid experiment.