Data Engineering Interview: How Do You Design an ML Feature Pipeline Without Training-Serving Skew?

Question and When It Applies

Design an ML feature pipeline for real-time fraud scoring. The training set spans the previous 12 months, and the online system handles about 5,000 predictions per second. Each prediction needs roughly 50 features, with a 5 ms p99 budget for feature retrieval. These numbers are capacity assumptions for discussing trade-offs, not industry defaults.

The features include a one-hour transaction-amount aggregate from an event stream and account age updated daily. Some events arrive late, historical data may be backfilled, and a model update may change feature definitions. Explain how to build point-in-time-correct training data, serve online values with low latency, handle missing and stale data, roll out new versions, and prove that offline and online semantics match.

This question applies to data engineering, machine learning engineering, and ML platform interviews. The goal is not to recite “add a feature store.” It is to connect a feature's definition, computation, storage, serving, and verification into one traceable path.

What the Interviewer Is Evaluating

First, can the candidate define feature semantics before drawing a storage architecture? A strong answer puts the entity key, data type, source, transformation version, event time, availability time, aggregation window, freshness requirement, default policy, and owner into a feature contract. Both offline and online systems implement that contract instead of guessing meaning from a name.

Second, can the candidate reason correctly about time? eventat says when an event happened in the business domain; availableat says when the system actually knew about it. An event may have happened before a prediction but arrived afterward, so the service could not have used it. A training join that checks only eventat <= predictionat can still pull late or backfilled information into the past.

Third, can the candidate choose computation and storage for different access patterns? Offline training needs time history and point-in-time joins, while online serving usually needs the latest value for an entity. Window aggregates are good candidates for precomputation and materialization. Cheap features that depend only on the current request can be computed on demand. Offline and online execution engines may differ, but their semantics must match and the equivalence must be demonstrated with data.

Fourth, can the candidate design a safe version rollout and failure behavior? A model should pin the feature-set version it consumes, and old and new versions must coexist during canary and rollback windows. Missing does not mean zero. Stale data, store outages, and incompatible versions each need an explicit policy.

Finally, the interviewer wants executable verification: contract checks, golden records, historical point-in-time tests, sampled online feature vectors, offline replay, late and out-of-order event injection, backfill and rollback drills, plus latency, freshness, missingness, and parity metrics.

Questions to Clarify First

• What is the prediction time and label window? Assume scoring occurs when a transaction arrives and chargeback labels mature over the next 30 days. Label information must never flow back into features.
• What does the online latency budget include? Does 5 ms p99 cover only feature retrieval, or also networking, serialization, and inference? This answer treats it as the feature-retrieval budget.
• How fresh must each feature be? A one-hour transaction total may need minute-level freshness, while account age may tolerate daily updates. One TTL should not be applied to every feature.
• How late, duplicated, or out of order can events be? This determines watermarks, deduplication, recomputation, and online overwrite rules.
• Should a historical backfill reproduce “what was known then” or “the latest corrected truth”? Reproducing past serving needs the former; post-hoc analysis may need the latter. They cannot be mixed in one dataset.
• How should the business degrade on missing or stale online data? Fraud systems may use a baseline model, manual review, or a conservative decision instead of turning every missing value into zero.
• How are models and features deployed independently? We need to know whether multiple versions can be materialized in parallel, how long old versions remain, and what the rollback target is.
• What infrastructure already exists? For one low-traffic, batch-only model, shared versioned transformations and immutable snapshots may be sufficient. A full feature platform should not be the starting assumption.

30-Second Answer Framework

“I would first create a feature contract containing entity keys, two clocks, windows, versions, and freshness requirements. Raw events remain immutable, and the same definitions generate offline history and the latest online values. Training examples use a point-in-time join at predictionat, constrained by both eventat and availableat, so later-arriving data cannot travel into the past. The model pins featureset_version; old and new versions are materialized in parallel and shadow-compared before traffic moves. I sample the actual online feature vectors, replay them offline from raw events with the same definitions, compare values, missingness, and freshness feature by feature, and inject late events, backfills, store failures, and rollbacks.”

Step-by-Step Deep Dive

Step 1: Turn each feature definition into an executable contract.

Register at least the following fields for every feature:

Once a feature name is released, its definition must not change silently. If txnamountsum_1h changes from “authorized transactions” to “all attempted transactions,” publish a new version or feature name. An old model must not unknowingly consume the new meaning.

Step 2: Make offline and online paths start from the same facts.

Transactions, account changes, and other source events first enter an immutable log or replayable history layer. Stream processing computes high-freshness window features; batch processing computes slowly changing dimensions and historical backfills. Both paths write complete time history to the offline store and materialize the latest valid value per entity into the online store.

“Same definition” does not require batch and stream jobs to use the same language. Prefer shared declarative transformations or shared code. If two implementations are necessary, golden records and replay parity become release gates. A feature store organizes these constraints; it does not automatically remove divergence between two implementations.

Online writes must be idempotent. Events carry stable IDs so duplicates do not increment aggregates twice. When writing a latest value, compare feature timestamps and versions so an older late result cannot overwrite a newer value. Windowed features also need an explicit allowed-lateness interval, watermark, and correction rule.

Step 3: Build genuinely point-in-time-correct training data.

Every training example has a predictionat. For the same entity, a basic as-of join selects the latest feature version with eventat <= predictionat. When events can be late or backfilled, it must also satisfy availableat <= prediction_at:

eligible_feature = same_entity
  AND event_at <= prediction_at
  AND available_at <= prediction_at

selected_feature = latest eligible_feature by event_at, then available_at

The second condition matters. Suppose a transaction occurred on Monday, arrived on Wednesday, and the historical prediction happened on Tuesday. It is in the past according to business time but still in the future according to system knowledge. If the data platform cannot preserve available_at, use the immutable snapshot or online feature log from that time. Do not present today's corrected table as the state historical serving actually saw.

“As known then” and “latest corrected” should be explicit dataset modes. The former reproduces what a model could see at a historical moment; the latter supports reconciliation or post-hoc analysis. Labels are handled separately: only examples with mature observation windows enter training, and label-generation data never participates in feature joins.

Step 4: Choose between materialization and compute-on-read.

Window aggregates such as one-hour transaction amount or seven-day device count are expensive and freshness-sensitive, so incrementally compute and materialize them from the stream. Low-frequency features such as account age can be updated in batches. A cheap feature that depends only on the current request and needs no training history can be computed on demand, reducing storage and synchronization surface.

The online store retrieves the latest values by entity key and featuresetversion instead of scanning history at request time. The offline store preserves the time series for training, backfills, and audits. Along with values, the retrieval API should return or record feature timestamps, computation versions, and missingness states so serving can detect staleness and incompatibility.

Step 5: Define missing, stale, and outage behavior.

Each feature or feature group has its own freshness budget. At read time, calculate predictionat - featuretimestamp and mark values stale when they exceed that budget. Missing, stale, and the legitimate numeric value zero are three distinct states. Training data must represent missingness the same way as serving.

Risk determines degradation. A noncritical feature may use a default that was included in model training. A feature allowed to be briefly stale may use its previous value. If a critical fraud feature is unavailable, route to a model that does not depend on it, send the case to manual review, or make a more conservative decision. Record every degradation reason so the system cannot silently remain on defaults for an extended period.

Step 6: Roll out versions while preserving rollback.

The model artifact pins a featuresetversion containing feature names, schemas, and transformation versions. To release v2, materialize v1 and v2 in parallel. Shadow-read the same entities and replay historical cases. After coverage, freshness, value distributions, and per-feature differences meet their gates, deploy the model that consumes v2. Keep v1 until the rollback window has ended.

Schema compatibility rules must also be explicit. Adding an optional feature may be backward compatible; deleting a feature, changing its type, or changing its semantics usually requires a new version. Before model deployment, verify that the online store already has the required version and coverage. Do not deploy the model first and let feature backfill catch up later.

Step 7: Verify parity using facts from online serving.

Sample online requests and record the entity, predictionat, featureset_version, each feature value, feature timestamp, missing/stale state, and final model version. Recording only the model score makes it impossible to locate whether a mismatch came from a value, time boundary, or version.

The verifier starts from immutable source events, uses the same definition version, reconstructs the offline vector at the same prediction time, and compares feature by feature:

• values match, with declared tolerances for floating-point features;
• missing, default, and stale states match;
• event-window boundaries and time zones match;
• the online feature version matches the model declaration;
• late, duplicate, out-of-order, and backfilled events replay deterministically.

Before release, inject online-store unavailability, partially missing keys, data timeouts, and a v2-to-v1 rollback. At runtime, monitor p50/p95/p99 retrieval latency, materialization lag, missing rate, default rate, stale rate, version mismatch rate, and replay mismatch rate. Model-performance monitoring can expose the consequences, but it cannot replace this feature-level evidence.

Example of a Strong Answer

“I would define a feature contract before choosing a database. For each feature, I register the entity key, type, source, transformation version, event time, system availability time, window, freshness budget, and degradation rule. Raw events go to an immutable history layer. Stream processing handles high-freshness aggregates such as one-hour transaction amount; batch processing handles account dimensions and backfills. Both write time history offline and materialize the latest valid value per entity online.

Training examples use predictionat as the anchor for point-in-time joins. A feature must satisfy both eventat <= predictionat and availableat <= prediction_at. That excludes records that happened earlier but had not yet arrived. To reproduce historical serving, I use an as-known dataset; corrected data is kept separately for post-hoc analysis. Chargeback labels enter only after their 30-day window matures, and label sources never enter the feature pipeline.

Online serving retrieves roughly 50 values by entity and featuresetversion, together with timestamps and states. Window aggregates are precomputed; cheap request-only features are computed on demand. Stable event IDs prevent duplicate updates, and older late versions cannot overwrite newer online values. Each feature has its own freshness budget. Missing, stale, and zero remain distinct. If a critical fraud feature is unavailable, serving uses a validated baseline model or manual review instead of silently filling zero.

The model artifact pins its feature-set version. For v2, I materialize v1 and v2 in parallel, perform shadow reads and replay comparisons, move the model only after coverage passes its gate, and retain v1 for rollback. To prove parity, I sample actual online feature vectors with times and versions, rebuild the same vectors offline from immutable events, and compare every value and state. Release gates also cover late and out-of-order events, backfills, store failure, and version rollback. Runtime metrics include retrieval p99, materialization lag, missing/stale rates, version mismatches, and replay mismatches.

If there is only one low-traffic batch model, I would begin with one versioned transformation and immutable training snapshots. I would introduce a full feature platform only when multiple models genuinely share features and need both historical retrieval and low-latency serving.”

Common Mistakes

• Choosing an online database before defining semantics → low-latency storage cannot stop same-named features from having different meanings → create an executable feature contract first.
• Using only eventat in historical joins → records that arrived later travel into the past → also constrain availableat or reproduce the historical snapshot.
• Assuming a shared feature store guarantees parity → batch and stream paths may still use different windows, defaults, or time zones → prove equivalence with shared definitions, golden records, and replay.
• Allowing a late result to overwrite the online value → an older window can move entity state backward → compare feature timestamps and versions, and make writes idempotent.
• Turning every missing value into zero → zero may be legitimate, while outages become model signals → distinguish missing, default, stale, and real zero.
• Editing a released feature in place → old models consume new semantics without a version change → publish a new version and pin model dependencies.
• Deploying the model before feature backfill finishes → initial traffic sees missing or mixed versions → materialize first, validate coverage, then move the model.
• Comparing only offline and online distributions → similar distributions can hide wrong entity joins or window boundaries → replay and compare each feature for the same request.
• Logging only prediction scores → failures cannot be localized to values, times, or versions → sample actual vectors and their metadata.
• Building a full feature platform for every use case → a single batch model may absorb needless complexity → introduce capabilities according to real sharing, history, and latency needs.

Follow-up Questions

Follow-up 1: Why can event time and availability time not be merged?

Event time answers “when did this happen in the business domain?” Availability time answers “when did the system know?” Late events, manual corrections, and backfills make them diverge. Reproducing a historical prediction requires both time boundaries; otherwise the model uses information unavailable then. They may be equal if synchronous arrival is a real, auditable guarantee, but that assumption must not be treated as a fact by default.

Follow-up 2: Must batch and stream processing share exactly the same code?

Shared code reduces divergence and is worth preferring, but it is not the only valid design. Execution engines, state management, or performance requirements may require two implementations. In that case, share the contract and test data, verify equivalence with golden cases, boundaries, and historical-event replay, and make divergence tests release gates.

Follow-up 3: How should a late event correct an online window feature?

First define allowed lateness and window-closing rules. An event within that interval can be deduplicated and update the affected window, while only a newer feature version may overwrite the online value. Events outside the interval enter a correction or backfill job. Whether past training data changes depends on dataset mode: as-known reproduces past serving, while corrected reflects the latest truth. Store them separately.

Follow-up 4: How do you decide whether to materialize or compute a feature on demand?

Compare computation cost, reuse, freshness, retrieval latency, and parity risk. A window aggregate across many historical events that is heavily reused and latency-sensitive is a strong materialization candidate. A cheap request-only feature with no historical training need is a good compute-on-read candidate. Include backfill cost and failure surface, rather than looking only at CPU time for one request.

Follow-up 5: How can a feature definition change without interrupting serving?

Release a new feature-set version and materialize old and new versions in parallel. Validate schema and coverage, shadow-read online entities, and compare model impact before moving a small share of traffic to the new model. Old models remain pinned to the old version, and old data stays through the rollback window. Never replace semantics in place under the same name.

Follow-up 6: What if offline and online floating-point results are not identical?

Separate acceptable numeric error from semantic differences. Declare absolute or relative tolerances per feature and use the same null, time-zone, rounding, and window-boundary rules. A systematic difference by entity or boundary should be treated as an error. Do not use one broad global tolerance to hide wrong joins, precision truncation, or different aggregation orders.

Follow-up 7: Should serving fail or degrade when the online feature store is unavailable?

The decision depends on risk and recoverability. Low-risk recommendations may briefly use a cache or baseline ranking. High-risk fraud decisions may go to manual review, use a conservative rule, or reject requests that cannot be assessed safely. The strategy must appear in training and drills, log the reason, and have duration and traffic limits so a failure mode does not become normal operation.

Follow-up 8: How do you prove that the new pipeline reduced training-serving skew?

Select historical requests covering normal, missing, boundary-window, late, and backfilled cases, and save the actual online vectors and versions. Rebuild them offline at the same prediction_at from immutable events, then compare every value and state. Continue sampling the same reconciliation after launch and require mismatch rate to remain below a predefined gate. Distribution plots and model metrics supplement this evidence; they cannot replace same-request replay.