Detect and Prevent Data Leakage in Machine Learning

Prompt and use cases

A fraud model performs exceptionally well on an offline random split but degrades substantially in production. Each row represents a transaction, and the model must decide whether to block it when it occurs. The label is whether a chargeback is confirmed within 30 days after the transaction. Candidate features come from transaction events, account-history aggregates, final chargeback outcomes, and manual-review states.

Explain data leakage, investigate target, temporal, entity-duplication, and preprocessing leakage, and redesign feature generation, train/validation/test splitting, cross-validation, and final evaluation. Also explain how to distinguish leakage from training-serving skew and genuine distribution drift.

This question appears in machine learning engineering, data science, risk, and recommender-system interviews. Interviewers rarely accept “split before preprocessing” as a complete answer. They will ask whether a feature existed at the real prediction time, whether the same entity may cross partitions, and whether the team repeatedly tuned against the test set.

What the interviewer evaluates

First, can the candidate give an operational definition? Data leakage occurs when model development or evaluation uses information unavailable at the real prediction time, or when information from a validation or test partition influences training, feature selection, or model selection. Leakage often makes offline metrics too optimistic, but a production drop alone does not prove leakage. Data drift, inconsistent labels, and faulty online feature computation can look similar.

Second, does the candidate define the prediction contract first? Every row needs a prediction_at, a label-observation window, a data-availability cutoff, and a description of the entities served in production. Without that boundary, even “transactions in the previous seven days” can contain events after prediction. Whether a business field leaks depends on when it becomes available, not on how reasonable its name sounds.

Third, does the split reproduce deployment? Ordinary random splitting is suitable only for independent and identically distributed samples. Ordered data needs forward validation. User, device, merchant, or other group structure calls for group isolation or at least a group-based stress test. Scaling, imputation, feature selection, and encoding must be fitted inside each training fold, and the final test set must not participate in tuning.

Finally, does the diagnosis form an evidence chain? Useful checks include point-in-time replay, a feature-availability ledger, cross-partition duplicate audits, suspicious-feature ablation, label-permutation negative controls, and comparison between random and deployment-faithful splits. A strong answer also admits the trade-off: stricter evaluation often lowers the score and leaves less training data, but produces a more credible estimate of generalization.

Questions to clarify before answering

• When does prediction happen? Assume the transaction is scored on arrival. Information produced afterward cannot enter that row's features.
• When does the label mature? A positive means a confirmed chargeback within 30 days. Rows near the dataset cutoff whose observation windows are incomplete cannot be treated as negatives.
• Does the model serve existing or unseen entities? If production repeatedly sees old accounts, temporal splitting is the primary evaluation. Generalization to new merchants or devices also requires entity-isolated testing.
• At what time are historical aggregates computed? Use event time and data actually visible then through point-in-time/as-of computation, not today's warehouse snapshot backfilled into history.
• Are there duplicates or near duplicates? Retries of one transaction, mirrored logs, multiple rows for one case, and highly similar text can all cross partitions.
• Which steps learn parameters from data? Imputation, normalization, vocabularies, feature selection, dimensionality reduction, target encoding, and threshold selection all count. Auditing only the final model is insufficient.
• How often has the test set been viewed? If its results guided feature or hyperparameter changes, it has participated in model selection and a new untouched holdout is required.
• What exactly degraded in production? Compare input distributions, feature missingness, label delay, offline replay, and serving logs instead of calling every failure leakage.

30-second answer framework

“I would define the prediction time and 30-day label window, then verify that every feature was truly available then. I would check four leakage paths: post-outcome fields, future data, cross-partition duplicates, and preprocessing fitted on all data. Development uses forward validation, with entity isolation where needed, and every learned transform fits only on training folds. After locking the model and threshold, I evaluate the final holdout once. Point-in-time replay, overlap audits, feature ablation, and label permutation help locate leakage; serving skew and drift are checked separately.”

Step-by-step deep dive

Write the prediction contract before choosing a splitter. At minimum, each row records a business entity, eventat, the time the system actually observed the event as availableat, predictionat, and labelreadyat. A feature is eligible only when availableat <= prediction_at and the same holds for every upstream input used to compute it.

The feature review for this scenario could look like this:

| Candidate feature | Available when the transaction occurs? | Rule | |---|---:|---| | Current transaction amount and channel | Yes | Use exactly what the serving request contains | | Account transaction count in the previous 7 days | Conditional | Count only earlier events that had arrived at that time | | Final chargeback reason | No | It is part of the label-formation process and must be removed | | Final manual-review state | No | It occurs after prediction and must be removed | | Device risk score | Conditional | Read the historical versioned snapshot, not a value recomputed today |

Next, classify leakage by the boundary crossed. Target leakage includes the label itself, a proxy for it, or a post-outcome action such as a chargeback reason or completed refund. Temporal leakage includes future events, rolling aggregates calculated over the full timeline, backfilled late data, and aggregating before a time split. Entity leakage includes copies of one transaction, multiple rows from one case, near-duplicate text, or a model memorizing validation accounts through an ID. Preprocessing leakage occurs when imputation values, scaling parameters, vocabularies, feature selection, or target encodings are fitted on all data. The full training process has then seen the evaluation partition even if the final classifier never saw its labels directly.

The split must follow the production question:

1. Sort by prediction_at. Use earlier periods for development and seal the latest contiguous period with mature labels as the final test set.
2. Run forward cross-validation inside development data: each fold trains on earlier data and validates on the next period. If label observation crosses a boundary, purge the end of training or insert a gap so training labels do not depend on outcomes from the validation period.
3. Build duplicate clusters and entity groups before assigning rows. If the goal is unseen-entity generalization, keep each group on one side. If production repeatedly serves old entities, temporal splitting remains the primary metric, but report an account-, device-, or merchant-isolated stress test as well.
4. Only a training fold may call fit. Validation folds and the test set receive transform from parameters learned on that training fold. Target encoding within training uses cross-fitting so each row's encoding comes from other folds that exclude its own label.
5. Use cross-validation to select features, hyperparameters, and the decision threshold. Once every decision is frozen, refit once on all development data and evaluate the final test once. Continuing to modify the model after reading that result requires a new test set.

The following pseudocode captures the workflow. forwardsplits enforces temporal order and label-window separation, while makepipeline binds every learned transform to the model:

dev, test = point_in_time_split(rows, test_period="latest_mature_period")

for train_idx, valid_idx in forward_splits(
    dev,
    time="prediction_at",
    purge="label_horizon",
):
    pipeline = make_pipeline(
        imputer="fit_on_train_fold",
        scaler="fit_on_train_fold",
        target_encoder="out_of_fold",
        model="candidate",
    )
    pipeline.fit(dev[train_idx].X, dev[train_idx].y)
    record(pipeline, dev[valid_idx])

locked_pipeline = select_and_lock()
locked_pipeline.fit(dev.X, dev.y)
final_result = evaluate_once(locked_pipeline, test)

Then diagnose. The first layer is a static lineage audit: record the source table, event time, availability time, aggregation window, update delay, and label dependency for every feature, and reject data past the cutoff automatically. The second is a partition audit: compare exact hashes, near-duplicate fingerprints, and entity-ID intersections. A duplicate cluster should be one splitting unit. The third is an experimental negative control: remove the most suspicious features, permute labels within valid groups, and replace the random split with temporal or group splits. A permuted result should return to the no-signal baseline. A large drop under a deployment-faithful split is a leakage alarm, not proof on its own.

The fourth layer is point-in-time replay. Choose historical transactions, freeze the feature service's clock at that time, and compare the offline training row with fields actually visible in serving logs. An offline value that incorporates later backfill is temporal leakage. Different computation rules, defaults, or versions between the two paths are training-serving skew. Even if both are correct, changing user populations or fraud tactics can cause distribution drift, so compare inputs, label rates, and segment metrics by time cohort.

Finally, turn the controls into product infrastructure: immutable dataset snapshots, reproducible split manifests, availability timestamps in feature definitions, versioned training pipelines, and access logs for the final holdout. Every new feature must answer one question: “For this row, could the serving system have computed exactly this value at prediction_at?” If the answer is unclear, it does not enter training.

High-quality sample answer

“Leakage means information crossed the prediction boundary we claim to enforce. This model scores a transaction when it occurs, so I would define prediction_at per row and wait for the 30-day chargeback window before treating its label as mature. Final chargeback reasons and final manual-review states are obvious outcome leakage. A seven-day account count looks legitimate, but it also leaks if today's warehouse snapshot adds late or post-prediction events to the historical row.

I would maintain event time, availability time, aggregation window, and label dependency for every feature, then reconstruct values with as-of joins. The latest mature time period is sealed as the test set, and development uses forward cross-validation. If the label window crosses fold boundaries, I purge the boundary. Duplicate transactions, cases, and near-duplicate records become clusters before splitting. Whether accounts must be fully isolated depends on whether production predicts returning or unseen accounts; I use the deployment-faithful case as the primary metric and entity isolation as a generalization stress test.

Imputation, scaling, feature selection, and encoding all live in the pipeline and fit only on each training fold. Target encoding is cross-fitted so a row's own label cannot help encode it. Validation results choose features, hyperparameters, and threshold. I then lock the process, refit on all development data, and inspect the final test once.

For diagnosis, I audit hashes, near duplicates, and entity overlap across partitions; run suspicious-feature ablations and label permutation within valid groups; and compare random with temporal or group splits. I also replay historical features against serving logs. A lower score under a temporal split only makes the original boundary suspicious. Point-in-time data crossing is leakage; matching point-in-time data but different online computation is training-serving skew; matching pipelines followed by degradation in later cohorts points toward genuine drift. The stricter process may lower the offline score, but it produces an estimate suitable for a launch decision.”

Common mistakes

• Calling only test-label exposure leakage → This misses future features, duplicate samples, and full-data preprocessing → Audit the complete information path from raw data through model selection.
• Declaring leakage whenever production degrades → Drift, label bias, and serving bugs also degrade performance → Check point-in-time lineage, offline replay, and time-cohort distributions separately.
• Scaling or selecting features on all data before splitting → Evaluation data influences learned transform parameters → Split first and fit the pipeline inside every training fold.
• Randomly splitting every dataset → Future samples or copies of the same entity can enter training → Choose a splitter from the deployment's time and entity structure.
• Filtering aggregates only by event_at → Late or backfilled records may not have been visible then → Constrain both event time and availability time.
• Deduplicating rows while ignoring near duplicates and case groups → The model can still memorize almost identical samples → Create duplicate clusters and business groups before assignment.
• Computing target encoding directly on all training rows → Each row's label can enter its own feature → Use out-of-fold encoding or an implementation with internal cross-fitting.
• Repeatedly viewing the test set and changing the model → The test gradually becomes a validation set → Keep an access-controlled final holdout and inspect it once after decisions are frozen.
• Expecting a fixed “random score” from label permutation → The no-signal baseline varies by metric and class distribution → Compare against the baseline under the same sampling and metric rules.
• Deleting all entity history to eliminate leakage → This may remove useful information genuinely available in production → Keep prediction-time information and evaluate it with the correct boundary.

Follow-up questions and responses

Follow-up 1: When is a random split appropriate?

It is reasonable when samples are approximately independent and identically distributed, production traffic and collected data share a generative process, and there is no meaningful time, user, device, experiment-batch, or duplicate-cluster structure. Duplicates and preprocessing boundaries still need auditing. If the production task predicts the future, a temporal holdout is usually more faithful.

Follow-up 2: Why does delayed labeling require a purge or gap?

A training transaction's label may be decided up to 30 days later. If training ends immediately before validation starts, that training label may depend on outcomes inside the validation period, information unavailable at the real validation start. The boundary must cover the label-observation horizon, or each training cutoff must include only labels already mature at that point.

Follow-up 3: May history from the same account be a feature?

Yes, if serving truly has that history at prediction time and the aggregate uses only past events visible then. Whether one account may cross training and test depends on the target: retain it across time when serving returning accounts, and isolate accounts when evaluating unseen-account generalization. Reporting both scenarios prevents one score from answering two different questions.

Follow-up 4: How does cross-fitting prevent target-encoding leakage?

Split training data into folds. Compute each fold's category statistics from labels in the other folds, then encode that held-out fold. A row's own label therefore cannot directly help construct its feature. Validation and test use mappings learned from the corresponding training data. Cross-fitting fixes leakage inside the encoder; it does not replace the correct outer temporal or group validation.

Follow-up 5: How do you distinguish data leakage from distribution drift?

First, use point-in-time replay to prove every offline feature was available at prediction. Then verify that online and offline systems compute the same values for the same rows. Failure of the first check is leakage; failure of the second is training-serving skew. After both pass, changes in inputs, label rates, and segment metrics across later cohorts provide drift evidence. Multiple failures can coexist.

Follow-up 6: What additional leakage risks come with pretrained or large language models?

Evaluation samples or near duplicates may already exist in pretraining data, retrieval indexes, or prompt examples, a path absent from the local train/test manifest. Use holdouts created after the training cutoff, privately generated sets, or sets audited for near duplicates, and version retrieval indexes and prompts. When complete non-overlap with pretraining data cannot be established, describe the result as an estimate subject to contamination risk rather than absolute generalization.