How Do You Diagnose Sample Ratio Mismatch in an A/B Test?

Prompt and Applicable Context

A product experiment uses stable random assignment by user with a configured 50/50 split. After 100,000 eligible users, the analysis contains 51,000 control users and 49,000 treatment users. Treatment conversion is 1.8% higher, and the ordinary effect test reports p below 0.01. Decide whether the result is trustworthy and design a process to detect, diagnose, repair, and prevent Sample Ratio Mismatch, or SRM.

This question applies to data science, product analytics, business intelligence, and experimentation-platform roles. Amazon's current data-role interview guidance explicitly calls for statistics, working through ambiguous data, and turning analysis into actionable decisions. Microsoft experimentation research treats SRM as a core trustworthiness signal. The primary category is data: the central task is statistical testing and data-lineage diagnosis, not designing the entire experimentation platform.

The values 100,000, 51,000, 49,000, and 1.8% are interview assumptions, not industry benchmarks. The solution assumes the user is the randomization unit and each user should appear in exactly one variant. A real experiment randomized by device, account, workspace, household, or region must test the corresponding unit.

What the Interviewer Evaluates

The first signal is whether the candidate checks experiment integrity before reading business impact. A weak answer sees “1.8% lift with p below 0.01” and recommends launch. A strong answer first verifies that the observed population matches the configured assignment. The effect test can say that two observed outcomes differ; it cannot prove that the users remaining in those groups still form comparable randomized samples.

The second signal is whether a visible ratio difference becomes a statistical test. The same 51/49 split means something very different with 1,000 users than with 100,000 users. An SRM test compares observed variant counts with counts implied by the configured allocation. More data makes small systematic deviations easier to distinguish from chance.

The third signal is diagnostic coverage. SRM can enter through allocation configuration, unstable identities, variant execution, missing telemetry, bot filtering, data joins, or an analysis condition defined after treatment. Calling every SRM “bad randomness” misses many common execution and data-processing failures.

Finally, the interviewer looks for decision discipline. SRM is a symptom, not harmless noise that weighting automatically removes. In a published case, treatment changed engagement enough that a bot filter removed more highly engaged users; the business conclusion reversed after the defect was fixed. Until the root cause is understood, hold the ship decision instead of searching for another metric that remains significant.

Questions to Clarify Before Answering

• Does 50/50 describe assignment, exposure, triggering, or the final analysis population? Assignment can be clean

while exposure, eligibility, or downstream analysis develops SRM. Test each stage.

• What is the randomization unit? If assignment is by workspace, member counts can differ naturally. Test workspace

counts first and analyze outcomes with methods appropriate for clustered assignment.

• Is the identity stable and mutually exclusive? Cross-device login, cookie deletion, anonymous-to-account merging,

and ID migration can reassign or duplicate a person.

• Did the allocation change during the run? A ramp from 10/90 to 50/50 requires expected counts from each interval,

not the final split applied to the whole experiment.

• How is an “eligible user” defined? A condition affected by treatment, such as clicking a new treatment-only button,

is post-treatment selection and can destroy comparability.

• Can one variant redirect, fail, or log differently? Slower loads, crashes, and version-specific telemetry can remove

users during execution and collection.

• Which pipelines produce the final count? Separate assignment, exposure, event collection, deduplication, bot

filtering, dimension joins, and metric windows.

• Can the experiment be rerun? If unbiased reconstruction from raw logs is impossible, fixing the defect and rerunning

is more defensible than forcing a conclusion from the current sample.

30-Second Answer Framework

“I would not ship. With a 50/50 allocation, the expected counts are 50,000 per group. The chi-square statistic is (51,000 - 50,000)² / 50,000 + (49,000 - 50,000)² / 50,000 = 40; with one degree of freedom, p is about 2.54 × 10^-10. The final population therefore has a severe sample-ratio mismatch, so the business-effect p-value is not launch evidence. I would compare counts at assignment, exposure, trigger, log-processing, and final-analysis stages, then segment by time, platform, version, and identity type to find the first divergence. After repairing the cause, I would rerun SRM, telemetry, and A/A checks; if the sample cannot be reconstructed without bias, I would rerun the experiment.”

Step-by-Step Deep Answer

Start with an explicit decision gate: an experiment must pass randomization, data-integrity, and business-effect checks. Run SRM before effect analysis and surface it prominently in the experimentation platform. Microsoft's published practice uses p < 0.0005 as a conservative alert threshold to reduce false positives at platform scale. That is a documented example, not a universal constant. A team should choose its quality threshold in advance rather than after seeing results.

Step 1: Compute SRM correctly.

For two variants configured at 50/50 with 100,000 total users, each expected count is 50,000. The chi-square goodness-of-fit statistic is:

χ² = Σ (observed - expected)² / expected = 40

With two groups and no additional estimated parameter, there is one degree of freedom, giving p of approximately 2.54 × 10^-10. This is far below a conservative integrity threshold and is not plausibly ordinary allocation noise. The chi-square approximation needs adequate expected counts; a common NIST rule is roughly five expected observations per group. Very small samples, rare allocations, or many sparse variants call for an exact binomial or multinomial test.

The ratio must be interpreted with its sample size. If the total were 1,000 and the counts were 510/490, the expected counts would be 500/500, χ² = 0.4, and p would be about 0.527. The visible 51/49 ratio is identical, but the evidence is not.

Step 2: Find the first stage where counts diverge.

Build a stage funnel that preserves unique randomization-unit counts by variant:

If assignment is already imbalanced, inspect experiment configuration, bucketing code, and randomization IDs. If assignment is clean but exposure diverges, inspect variant-specific load failures, crashes, and redirects. If the untriggered population is clean but “users who visited checkout” is not, determine whether the trigger is affected by treatment or missing in control. If only the final fact table diverges, focus on filtering, deduplication, and joins.

Step 3: Localize the cause with time and segments.

Plot cumulative observed and expected counts by hour and locate when the gap begins. A sudden discontinuity points toward a configuration, deployment, or pipeline change. A steady deviation from experiment start is more consistent with a bucketing or trigger defect. Segment by platform, browser, app version, geography, login state, identity type, and traffic source, while showing segment size, direction, and first anomalous time.

Segmentation is a diagnostic tool, not a search for a subgroup that happens to pass SRM. Many slices naturally produce some small p-values. A useful slice supports a mechanism. If SRM is concentrated in an old iOS version and treatment exposure logs are missing in that version, inspect client delivery and telemetry; do not simply exclude iOS and declare the remaining experiment trustworthy.

Step 4: Verify randomization and identity semantics.

Bucketing should be a deterministic function of a stable randomization ID, experiment ID, and fixed salt. The same unit should remain in one variant throughout the experiment. Check:

1. Whether one randomization unit appears in multiple variants.
2. Whether anonymous IDs are reassigned when merged into account IDs.
3. Whether cookie clearing or cross-device use creates repeated “new users.”
4. Whether employee, bot, and exclusion rules are applied consistently around assignment.
5. Whether every ramp configuration has an auditable effective timestamp.

For workspace or household randomization, test cluster counts first. A treatment variant can randomly receive several large workspaces and show a 51/49 member-row ratio without a bucketing defect. Treating member rows as independent randomized samples would create a false alarm.

Step 5: Audit post-treatment selection and missingness.

A valid trigger should be determined from information available before treatment whenever possible. “Reached checkout” can be defined through a page exposure recorded in both variants. “Clicked the new treatment coupon button” has no equivalent observation in control and naturally selects more treatment users. Treatment performance can also change telemetry completion; bot, timeout, and error filters can remove users most affected by the change.

Do not assume missing users are random. Compare immutable assignment logs with each downstream table, sample missing IDs, and inspect their platform, time, version, and behavior. If treatment affects a user's probability of entering the analysis table, the observed conversion difference includes selection bias.

Step 6: Choose repair, reconstruction, or rerun.

• Wrong allocation or cross-variant users: stop, repair assignment, randomize again, and rerun.
• Complete assignment and exposure with a reversible ETL drop: repair the job, rebuild from unbiased raw events, and rerun SRM.
• Treatment-dependent trigger: use a pre-treatment condition or return to the untriggered intent-to-treat population.
• Missing required telemetry in a client version with no backfill: fix instrumentation and rerun for that target population.
• Planned allocation ramps: sum expected counts from each configuration interval and preserve the configuration history.

Weighting is only defensible when the selection mechanism is known, estimable, and independently validated. Most SRMs signal that the missingness mechanism is unknown. Multiplying 49,000 treatment users by a weight until the total looks like 50,000 does not restore the missing users' behavior or recover randomization.

Step 7: Build prevention into the experimentation platform.

Keep immutable assignment records, configured ratios, and effective timestamps. Compute SRM separately for assignment, exposure, triggered, and analyzed populations. Business results should not drive a ship decision until integrity checks pass. Run regular A/A tests to verify that bucketing, telemetry, and analysis remain clean when no product difference exists.

An alert should include total size, observed and expected counts, p-value, direction, first anomalous time, and major segments. Raise scrutiny after changes to deployment, identity, bot detection, or ETL. Retain enough lineage after an experiment ends to reproduce expected counts from historical configuration rather than preserving only a dashboard image.

High-Quality Sample Answer

“I would not ship because the reported 1.8% lift assumes comparable randomized samples. Under the configured 50/50 split, 51,000 control and 49,000 treatment users have expected counts of 50,000 each. The chi-square statistic is 40 with one degree of freedom, so p is about 2.54 × 10^-10. The experiment has SRM, and I would freeze the business conclusion.

First I would verify that user is the actual randomization unit and that 50/50 applied for the entire run. Then I would compare counts through assignment, actual exposure, trigger eligibility, event fact-table inclusion, and final analysis. The first divergent stage determines the investigation. Assignment SRM points to configuration, stable IDs, salts, or cross-variant membership. Clean assignment but imbalanced exposure points to deployment, performance, crashes, or redirects. Trigger-only SRM points to a treatment-dependent condition. Final-table SRM points to telemetry, bot filtering, deduplication, or joins.

I would segment by time, platform, app version, geography, login state, and ID type to find evidence for one mechanism, not to cherry-pick a clean subgroup. If unbiased raw events can reconstruct the population, I would repair the pipeline and rerun every integrity check. If users crossed variants, the trigger depends on treatment, or required telemetry cannot be recovered, I would repair the system and rerun the experiment.

For prevention, the platform should run SRM before revealing business results, retain immutable assignment and allocation history, and monitor assignment, exposure, trigger, and analysis counts separately. A/A tests, stable bucketing, and pre-treatment triggers verify the path. Only after the root cause is explained and the repaired population passes SRM would I reconsider the 1.8% lift and its guardrails.”

Common Mistakes

• Shipping because the outcome p-value is small → Effect inference assumes comparable samples → Pass SRM and data-integrity checks first.
• Looking only at the 51/49 percentage → The same ratio has different evidence at different sample sizes → Use expected counts and a statistical test.
• Treating SRM as only a random-number problem → Execution, logs, and filters also remove users → Trace the first divergence through the full path.
• Waiting for counts to look closer → More biased data does not restore randomization → Freeze the decision and diagnose immediately.
• Reweighting directly to 50/50 → Weights do not recover systematically missing behavior → Correct only under a validated missingness model.
• Checking only the final analysis table → It hides whether assignment or downstream processing failed → Retain counts at every stage.
• Triggering on a treatment-only behavior → Control lacks an equal chance to enter the sample → Use a symmetric pre-treatment condition.
• Testing member rows in a cluster experiment → Cluster-size variation creates false SRM → Test the actual randomization unit first.
• Dropping a bad platform after seeing results → Post-hoc population selection adds bias → Use slices for diagnosis and rerun the planned population.
• Reusing the old effect report after repair → It was computed from the defective sample → Regenerate integrity, effect, and guardrail analyses.

Follow-Up Questions and Responses

Follow-up 1: What if the split is still 51/49 but the total sample is only 1,000?

The expected counts are 500 each and the observed counts are 510 and 490. The chi-square statistic is (510 - 500)² / 500 + (490 - 500)² / 500 = 0.4, giving p of about 0.527 with one degree of freedom. That is insufficient evidence for SRM. This is why a fixed “more than one percentage point” rule cannot replace a statistical test, although a known engineering defect should still be investigated.

Follow-up 2: Overall assignment passes SRM, but iOS users fail. Can the experiment still be used?

Consider the number of slices, iOS sample size, and whether one mechanism explains the anomaly. If SRM is concentrated in a prespecified, material iOS version and exposure loss explains it, that population's result is untrustworthy. Whether other populations remain usable depends on the analysis plan and true isolation of the cause. Do not drop iOS only after seeing outcomes; repairing the client and rerunning the planned target population is safer.

Follow-up 3: The untriggered population is clean, but the triggered analysis has SRM. What is most likely wrong?

Inspect the trigger first. It may be logged in only one variant or depend on behavior caused by treatment. Replace it with a condition both groups can satisfy before treatment, such as reaching a page rather than clicking a new component. Also compare trigger telemetry coverage. Clean untriggered counts suggest base assignment may be sound, but they do not validate the triggered effect analysis.

Follow-up 4: Why not reweight the groups by their observed sizes?

SRM does not identify who is missing. If treatment lost the most active, highest-converting, or most crash-prone users, a count weight restores only the total, not the behavior distribution. Correction may be possible when selection probability is fully explained by pre-treatment variables and the model is validated, but that is an extra assumption, not the default remedy.

Follow-up 5: Workspaces were randomized 50/50, but user counts are 55/45. Is that SRM?

Check workspace counts first. If workspace assignment is 50/50 but treatment randomly received several large workspaces, member imbalance can be ordinary cluster-size variation rather than a bucketing failure. Outcome analysis must also account for within-workspace dependence. If balanced member traffic is required, use size stratification or matched randomization in the experiment design instead of treating member rows as independent assignments afterward.

Follow-up 6: The experiment ramped from 10/90 to 50/50. How do you compute expected counts?

Compute expected counts within each allocation interval and sum them. If 20,000 users arrived under 10/90 and 80,000 under 50/50, the total expected counts are 42,000 and 58,000, not 50,000 each. Use the configuration and effective time that were active at assignment; never apply the final split retroactively to the full run.