1. Prompt
A new onboarding flow is in an A/B test. After splitting by user type, treatment has a higher conversion rate for both new and returning users; after combining the strata, treatment has a lower overall conversion rate. The product owner wants a ship decision today.
Construct a concrete count example showing how the reversal can happen, then give an analysis flow from raw denominators and randomization balance to a stratified estimator and an interaction test. Finally, state the evidence required for a global rollout, a stratified rollout, pausing the experiment, or collecting more data.
2. Constraints and clarifications
- This is a randomized online experiment; “new” and “returning” are strata defined before the experiment starts.
- The primary metric is whether each user completes the core conversion in a fixed window. Repeated events from one user are not independent samples.
- The numbers in the prompt are practice data, not a real company result or a universal launch threshold.
- First align observation windows, deduplication, exposure definition, and data watermark. If a stratum is affected by treatment, it must not be used directly for a causal subgroup split.
3. Core reasoning: denominator mix can reverse the result
Overall conversion is a sample-size-weighted average of stratum conversion rates. If the stratum shares differ between arms, treatment can be better in every stratum while the aggregate reverses.
Consider an extreme but clear example. In a high-baseline stratum, control is 990/1000 = 99% and treatment is 100/100 = 100%. In a low-baseline stratum, control is 1/100 = 1% and treatment is 20/1000 = 2%. Treatment improves by one percentage point in both strata, yet aggregate control is 991/1100 = 90.1% and aggregate treatment is 120/1100 = 10.9%. The difference comes from placing most observations in different baseline strata, not from treatment making either stratum worse.
In a real experiment, such a large mix difference usually signals a randomization, exposure, sampling, or time-window problem. It does not prove that every overall average is wrong. When stratum shares are balanced by design, or a prespecified target-population standardization is used, the overall estimate can still represent the target population's average treatment effect.
4. Diagnosis and reference pseudocode
Aggregate user-level denominators and numerators by arm, stratum, and conversion flag. Compare stratum shares, effects, and confidence intervals. Do not start with a favorable overall percentage and then try arbitrary slices until an explanation appears.
for arm in [control, treatment]:
for stratum in [new, returning]:
n[arm, stratum] = count_distinct_users(arm, stratum)
y[arm, stratum] = count_users_with_conversion(arm, stratum)
rate[arm, stratum] = y[arm, stratum] / n[arm, stratum]
share[arm, stratum] = n[arm, stratum] / sum_s(n[arm, s])
check_exposure_and_assignment_balance()
check_missing_users_duplicates_and_window()
report_stratum_effects_and_confidence_intervals()
weights = preregistered_target_population_shares()
standardized_effect = sum_s(weights[s] * (rate[treatment, s] - rate[control, s]))
test_arm_by_stratum_interaction()If the strata are available before randomization, stratify randomization or at least monitor allocation within each stratum. The analysis plan should preregister the primary strata, target-population weights, primary metric, and stopping rule so the result does not dictate which slice to choose.
5. Correctness, statistics, and ship decisions
A stratified estimator puts each stratum effect back onto the same target-population weights, so a sample-mix difference is not mistaken for a treatment effect. The weights must come from a pre-experiment population definition or randomization design, not be inferred from the observed result.
Test the interaction between treatment and stratum to determine whether effect differences exceed sampling noise. If the interaction is material and stable, a different rollout for new and returning users can be justified. If it is not material, prefer the prespecified overall estimate instead of selecting the best-looking stratum.
Statistical significance alone does not justify shipping. Combine a minimum acceptable effect, guardrail metrics, experiment duration, sample size, delayed conversions, and data completeness. If exposure logging, cross-arm contamination, randomization, or window maturity is broken, pause the decision and repair the data rather than hiding the issue with more slicing.
6. Follow-ups and traps
- When can a stratum create a causal problem? If it is produced after exposure, conditioning on it can create collider bias; use a pre-exposure variable or a prespecified mechanism analysis instead.
- Why not slice without limit? Every unregistered slice adds opportunities for a chance significant result; control multiplicity and label exploratory findings as hypotheses.
- Is every aggregate difference Simpson's paradox? No. First check denominators, exposure eligibility, windows, deduplication, and missingness, then confirm that directions truly reverse within strata.
- What if treatment has a different stratum mix? Investigate randomization and routing first. If target-population weights are known, use the prespecified standardized estimate; never choose weights after seeing results.
7. References
- Google for Developers' Good Data Analysis and Analysis traps guides.
- Optimizely's explanation of Simpson's paradox and segmented experiment results.
- Public data-science interview prompts covering reversals in A/B tests.
8. Interview scoring points
Can calculate denominators and weights
The candidate should use counts to show “wins in every stratum, loses overall” and identify unequal stratum shares as the key clue.
Can separate diagnosis from decision
They should check randomization, exposure, windows, and data quality before discussing estimators, interactions, and rollout rules.
Can state causal boundaries
They should identify post-exposure strata, collider bias, and multiplicity risk instead of treating every slice as an independent conclusion.
Can give an executable plan
They should propose stratified randomization, preregistered weights, guardrails, and conditions for stratified rollout or continued testing rather than only saying “collect more data.”