Prompt and setting
This scenario fits data science, data analytics, and machine-learning interviews. The dataset contains user behavior, region, income, and a churn label; income is missing for 18% of rows. You must explain why it is missing before choosing a treatment. The percentage alone is not a decision rule.
What the interviewer tests
- Distinguishing MCAR, MAR, and MNAR instead of treating every null as the same problem.
- Explaining missingness indicators, imputation models, and train-validation boundaries.
- Detecting signal in missingness, selection bias, and information leakage.
Clarifying questions before answering
- Is income user-entered, lost by collection failure, or shown only in some regions? The mechanism changes what can be identified.
- Is the goal prediction or causal estimation? Prediction may retain a missingness signal; causal analysis needs stronger assumptions and sensitivity analysis.
- Is income unavailable at production inference time too? An offline backfill that will not exist online creates a feature mismatch.
- Is the split by user, time, or random row? The same user in multiple sets can leak information through imputation and evaluation.
30-second answer framework
I would map the missingness indicator against observed fields and the target, separating known causes, MAR explanations supported by observed data, and MNAR assumptions that observed data cannot verify. I would fit imputers only on training data, keep a missingness indicator, and evaluate by the original missingness pattern. If the mechanism is uncertain, I would compare deletion, simple imputation, model-based imputation, and sensitivity scenarios, then choose the option whose business cost and validation evidence are acceptable.
Step-by-step deep dive
1. Turn nulls into analyzable events
Create a missingness indicator M for each field and profile it by time, region, device, channel, and target label. Inspect logs, form flow, and upstream schema first. If missingness jumps after a release, fix collection rather than hiding the incident with imputation.
2. Explain the three mechanism boundaries
MCAR means missingness is unrelated to observed and unobserved values. Under that assumption and other required conditions, complete-case analysis is not biased by the mechanism, but it loses sample size. MAR means that after conditioning on observed variables, missingness is unrelated to the missing value itself, such as income missing in ways explained by region and device. MNAR means that even after observed variables are controlled, missingness remains related to the unobserved income or its cause, such as high-income users declining to answer.
These mechanisms are not labels proven by one plot. Logs and observed variables can support an explanation; MNAR often needs domain assumptions, external data, or sensitivity analysis.
3. Choose a treatment
- If a repairable collection failure caused the missingness, backfill the source or fix the pipeline and version the repair.
- For prediction when missingness carries signal, fit an imputer on training statistics or a training model, add a missingness indicator, and verify that production has the same signal.
- If the rate is small and MCAR is credible, deletion may be acceptable, but report sample loss and slice effects.
- When uncertainty matters or MNAR is plausible, compare multiple imputation, an explicit missingness model, bounds, and pattern-mixture sensitivity; one mean fill is not evidence.
scikit-learn's SimpleImputer, KNNImputer, and IterativeImputer are tools under different assumptions; they do not identify the mechanism automatically. Fit them on training data and apply them to validation, test, and production data.
4. Design validation and leakage defenses
Put imputation, scaling, and encoding in one training pipeline. Refit inside each cross-validation fold; never compute a global mean, median, or neighbor set before splitting. Split time data by time and user data by user. Compare original missingness, post-imputation distributions, indicator coefficients, calibration, and slice metrics. Inject extra missingness into validation to simulate production degradation.
5. Use business cost to accept a result
If misclassifying a high-value user costs more than missing an ordinary user, AUC is not enough. Report recall, calibration, threshold cost, review volume, and rejection risk under different missingness patterns. Freeze imputer version, schema, and policy before launch; monitor missingness drift and stop automated decisions or fall back to a safe rule when drift crosses a threshold.
High-quality sample answer
“I would not decide to delete or fill from the 18% rate alone. I would determine whether income is missing because of collection failure, user choice, or a channel-specific flow, then profile the missingness indicator by time, region, device, and churn. If observed fields explain it, I would use an MAR assumption, fit every imputer inside each training fold, and keep the indicator. If domain evidence suggests high-income users are less willing to answer, I would treat MNAR as an assumption that current data cannot prove and run pattern-mixture or bound sensitivity analyses. I would compare deletion, simple, and model-based imputation on a user- or time-isolated validation set, reporting calibration, slice cost, and production missingness. If production cannot obtain income, I would not rely on an offline backfill.”
Common mistakes
- Delete because the rate is 18% → sample loss and selection bias are unmeasured → analyze mechanism, slices, and business cost first.
- Compute a global mean before splitting → validation information enters training → fit imputation inside each training fold.
- Call a missingness indicator proof of MNAR → correlation does not reveal the unobserved cause → state the assumption and run sensitivity analysis.
- Compare only AUC → threshold cost, calibration, and drift stay hidden → report slices, decision cost, and production monitoring.
Follow-up questions and responses
What if income is completely unavailable in production?
Rebuild training and validation with production-available fields. Keep a missingness indicator only if it is a stable production signal, or remove the feature. Do not create an offline-only gain with backfilled income unless production obtains it at the same decision time.
How do you test MNAR sensitivity?
Define a plausible range for how missing values differ from observed values, vary that shift or missingness model, and recompute metrics and business cost. If the conclusion reverses in a reasonable range, label it assumption-dependent and collect external data or design additional sampling.
How do multiple and model-based imputation differ?
Model-based imputation often creates one completed dataset, which can work for prediction but understate uncertainty. Multiple imputation creates several plausible datasets and combines estimates, making uncertainty explicit. Both must fit within the training boundary and avoid target leakage.
What do you do first when missingness rises from 18% to 30%?
Pause normal scoring, inspect schema, client versions, instrumentation, and upstream jobs, and localize affected slices. If it is a collection incident, repair or backfill before retraining. If it is a business change, reassess the mechanism, threshold, and fallback policy.