Prompt and context
You own a feature platform for ad-click prediction. Each training row contains entityid, labelts, and a label; feature values change as events arrive, while online requests need low-latency reads of current values. Explain the offline point-in-time join, and how you handle backfills, late events, and online serving. Assume an entity has multiple feature versions, each record has a computation time, and the label time is the business event time.
What the interviewer evaluates
The signal is whether you define “available at the time” before naming storage. A weak answer says warehouse plus Redis. A strong answer states the invariant featurets <= labelts, selects the latest eligible version per entity, returns null when no history exists, and separates a historical training view from the online latest value. It also explains which late events cause recomputation and how to detect training-serving skew.
Clarifying questions
- Is label time the business-event time or the time the label was written? The boundary changes the join.
- How long after the event may a feature arrive and still be usable? Strict real-time semantics require an availability timestamp.
- Must training reproduce every historical version, or only a recent window? The first requires full history; the second can use a lookback limit.
- Does online serving need the latest value now or a value as of an event time? The latter needs a timestamped API, not a single cached value.
A 30-second answer
“I store an entity key, feature version, and availability time for every feature record. While building training data, I perform an as-of join per entity and keep the latest record with featurets <= labelts; if none qualifies, the value stays null. An offline store keeps history, while an online store serves current values, both produced from the same feature definition and materialization flow. Late events enter a recomputation queue, affected windows are rebuilt into versioned snapshots, and I monitor join null rate, freshness, online/offline distributions, and model impact.”
Step-by-step solution
1. State the time invariant
For sample s=(e, tlabel) and feature history He, choose C={h∈He | h.featurets ≤ tlabel} and return arg max featurets(C). This prevents future values from entering training. If the business meaning is “data was available,” use available_ts as an additional constraint; event time alone can be too optimistic.
2. Model history and the join
Store the entity key, feature name or version, value, featurets, availablets, source batch, and quality state. Training rows carry label_ts. Partition by entity and order by time for an as-of join; use a source sequence or write batch as a deterministic tie-breaker for equal timestamps. An exact-time join drops most rows, while selecting the latest row leaks future information.
3. Separate offline and online paths
An offline store keeps full history for batch training; an online key-value store serves the current value quickly. A shared definition produces both paths and records its version, input snapshot, and materialization watermark. If bounded staleness is allowed, the online response may include the recent value and its feature_ts so the caller can decide whether to degrade; it must not pretend this is the training-time value.
4. Handle late events, backfills, and versions
Write late events to an immutable raw layer, then enqueue recomputation for the affected entities and time windows. Rebuild a new training snapshot instead of overwriting a snapshot used by a released model. Make backfills idempotent with input batch plus definition version as the job key. When feature logic changes, publish a new version and retain the old one so historical experiments remain reproducible.
5. Validate and monitor
Sample offline rows to assert featurets <= labelts, and track the null rate for entities with no eligible history. Monitor online latency, feature age, materialization delay, and errors. Compare training and serving distributions to detect different null handling or window boundaries. Store the sample snapshot, definition version, and input watermark in model metadata so training can be reproduced.
6. Know when not to build a full feature store
For a small batch-only project, a warehouse query with window ordering and an as-of join is simpler. Add an offline history layer, online key-value layer, registry, and materializer when millisecond serving, feature reuse, and continuous backfills justify them. Computing every feature in real time increases state, cost, and consistency risk; retaining only the latest value prevents historical training reconstruction.
High-quality sample answer
I treat “visible at that time” as a hard constraint. Every entity’s feature history carries featurets; when arrival can lag the event, I also store availablets. For (entityid, labelts), an as-of join selects the latest version whose timestamp is no later than labelts; when product semantics require actual availability, I require availablets <= label_ts as well. Joining the latest row directly is unsafe because it leaks future updates into historical examples.
The offline layer keeps full history for training and the online layer keeps current values for low-latency inference, both driven by the same versioned definition. Late events land in the raw layer and trigger idempotent recomputation of affected windows; released-model snapshots remain immutable and the result becomes a new version. I validate the time invariant, null rate, freshness, online/offline distributions, and model impact. If there is no low-latency serving requirement, I would omit the online layer and keep a batch design.
Common mistakes
- Mistake: joining the latest feature row → future updates enter historical examples and inflate offline metrics → use an entity-and-label-time as-of join.
- Mistake: storing event time but not availability time → an event can be early but still unseen when the label happened → record
available_tswhen latency or batching matters. - Mistake: overwriting feature history during a backfill → released models become impossible to reproduce → create immutable snapshots keyed by input batch and definition version.
- Mistake: maintaining separate online and offline transformations → null handling or window boundaries drift and create training-serving skew → share the definition or test both paths with golden examples.
Follow-ups and responses
A feature event arrives after the label, but its event time is earlier. Can training use it?
Not from event time alone. If online serving could not see it at the label time, require availablets <= labelts; otherwise training simulates information the serving path never had. Keep both timestamps and choose the strict or relaxed rule from the product contract.
The online path needs a feature as of an earlier event. Is the current-value cache enough?
No. A current-value cache answers “latest now,” not a historical timestamp. Expose timestamped history or materialize the required version ahead of the request; use the offline layer when the latency budget cannot support online historical reads.
Late events keep arriving. How do you keep recomputation bounded?
Coalesce requests by entity, time window, and feature version, then set a maximum lookback and priority. Route events beyond that window to a batch or manual process and record which model versions were not rebuilt. Monitor affected rows, recomputation time, and queue age instead of chasing all history indefinitely.