How Do You Evaluate a Recommendation System Offline and Online?

Question and when to use it

A feed shows 10 recommendations per request. A candidate ranker improves NDCG@10 on historical implicit-feedback logs. Design the offline evaluation and online experiment needed to decide whether to launch it. Cover the data split, candidate set, ranking and beyond-accuracy metrics, exposure bias, experiment design, guardrails, and the investigation when the offline winner loses online.

This is a representative machine learning engineering and data science interview question. It evaluates a model decision, not the architecture of the whole recommendation platform. The scenario assumes implicit feedback such as clicks, saves, or watch time. These logs record outcomes for items the old system exposed; silence on an unexposed item is unknown, not a verified negative.

What the interviewer is evaluating

A basic answer lists precision, recall, and NDCG. A strong answer first defines the user action and serving surface, because the same metric can mean different things for search results, a home feed, or “similar items.” It then makes the baseline and candidate comparable: the same time cutoff, users, eligibility rules, candidate pool, labels, and value of K.

The next signal is whether the candidate recognizes the limits of logged feedback. A random row split can leak later behavior into training. Negative sampling can change model ordering. Historical logs favor items chosen by the previous policy and confound preference with exposure and position. Offline evaluation can reject a weak or unsafe model, but it cannot by itself establish the causal product effect of changing the policy.

Finally, the interviewer wants a launch process. That includes stable randomized assignment, exposure logging, a primary outcome, system and user-harm guardrails, power and duration planning, rollback criteria, segment checks, and a methodical diagnosis when offline and online results disagree.

Questions to clarify before answering

• What user decision should improve? If the goal is finding at least one useful item quickly, MRR may matter; if every position in a ten-item feed contributes value, NDCG and cumulative engagement fit better.
• How are labels defined and matured? A click, qualified watch, save, purchase, or explicit dislike represents different utility. Delayed purchases require a mature observation window.
• Does the ranker score the full catalog or candidates from an upstream retriever? Evaluation must preserve the production candidate boundary. A ranker cannot recover a relevant item that retrieval never supplies.
• Which items were eligible at each historical time? Deleted, unavailable, or not-yet-created items must not appear in an earlier candidate set.
• Can the product run a randomized experiment? If not, the conclusion must be more cautious. Logged propensities may support counterfactual estimators, but only where the old policy gave the new policy's choices adequate support.
• What outcomes must not regress? Latency, errors, hides or reports, content concentration, supplier exposure, and downstream quality may constrain a model even when engagement rises.

30-second answer framework

“I would not launch from an NDCG@10 increase alone. First I would freeze the product objective, label, K, point-in-time split, eligible catalog, candidate-generation boundary, and comparison baselines. On the same examples I would report recall@10 and NDCG@10, add coverage or diversity where the product needs them, and slice new users, sparse users, new items, and key markets. I would treat unexposed items as unknown and document any negative sampling. If the candidate passes offline checks, I would run a user-level randomized experiment with assignment and exposure logs, one primary user or business outcome, latency and harm guardrails, a predefined minimum detectable effect and duration, and rollback thresholds. An offline win makes the model eligible for a test; the randomized online result decides whether it improves the product.”

Step-by-step solution

Step 1: Define the decision before selecting metrics

Write a one-line estimand: “For eligible feed requests from the target population, replacing the current ranker with the candidate changes the primary user outcome over the experiment horizon by how much?” This fixes the population, intervention, outcome, and time window. It also prevents choosing whichever metric later looks favorable.

Use at least the production ranker and a simple popularity or recency baseline. The simple baseline detects a pipeline that appears sophisticated but cannot beat a cheap policy. Keep eligibility filters and candidate generation fixed when the question is specifically about the ranker. If retrieval changes too, evaluate that as a separate treatment or explicitly call the result an end-to-end system comparison.

Step 2: Reconstruct point-in-time offline examples

For each evaluation request at time t, build features only from information available by t, use the catalog and eligibility state at t, and place later interactions in the label window. Split by time so training precedes validation and test. Group related events by user or session when crossing partitions would reveal the answer. Freeze feature definitions, filtering, deduplication, and label maturity before inspecting candidate results.

Run the baseline and candidate on the same users, requests, candidate pools, labels, cutoff, and K. Report the amount of traffic excluded by missing features or immature labels; otherwise a model can “improve” by silently dropping its hard cases. Evaluate important slices such as new versus returning users, sparse versus heavy users, new versus established items, locale, device, and catalog category.

Step 3: Match each metric to a product claim

Suppose the held-out relevant items are {A, C} and the top-five ranking is [A, B, D, C, E]. Precision@5 is 2 / 5, and recall@5 is 2 / 2 = 1. NDCG gives the item at rank 1 more weight than the relevant item at rank 4, so it tests ordering as well as membership. MRR is useful when the first relevant result dominates the experience.

These values answer different questions:

• Recall@K: how much of the labeled relevant set entered the top K.
• Precision@K: how many of the K displayed items are labeled relevant; implicit logs can contain false negatives.
• NDCG@K: whether higher-gain items appear earlier, with a rank discount.
• MRR@K: how early the first labeled relevant item appears.
• Coverage, diversity, novelty, or calibration: whether the policy serves enough of the catalog, avoids repetitive lists, exposes discovery value, or aligns the mix with user interests. Select only those tied to the product risk.

Add serving measures—latency, error and fallback rates, feature availability, and score distributions—because an offline model that cannot satisfy the online budget is not launchable. No single offline metric is a universal winner.

Step 4: Make logged-feedback bias visible

The old system chose what users could see. A click means both exposure and positive action; no click may mean disinterest, poor position, or no exposure. Do not turn every unobserved catalog item into a certain negative. Full-catalog ranking is the cleanest comparison when it is feasible. If evaluation samples negatives, use the same candidate pool, sampler, sample size, and random seeds for every model, report them, and do not compare sampled metric values with full-catalog values. Different negative samplers can even reverse the apparent model ordering.

Offline results remain conditional on the logged policy. If randomized exposure propensities were logged, inverse-propensity, self-normalized, or doubly robust estimates can reduce policy bias, but they do not create evidence where the old policy almost never exposed an item. Report overlap and weight variance, and clip or reject an estimator whose result is driven by a few extreme weights.

Step 5: Turn the offline survivor into an online experiment

Randomize by a stable unit, usually user or account, before recommendation exposure. Use a cluster such as a household or social group when one person's treatment can affect another person's outcome. Log assignment, eligibility, generated slate, position, exposure, action, model version, and fallback. Analyze by assigned treatment—intent to treat—so failures and fallbacks are not removed from the candidate arm.

Choose one primary user or business outcome that matches the stated objective. Predefine the minimum detectable effect, significance level, power, allocation, duration, and treatment horizon. Guardrails can include p95 latency, errors, hides, reports, abandonment, content concentration, and downstream quality. Before reading impact, verify sample-ratio balance, assignment persistence, exposure rates, telemetry completeness, and comparable label maturity.

Start with a small reversible ramp, then expand only while guardrails hold. Predefine stop and rollback conditions. Seasonality and delayed outcomes can require complete business cycles or a longer observation window; repeatedly peeking and stopping on a favorable day invalidates an ordinary fixed-horizon test.

Step 6: Diagnose an offline win that loses online

Investigate the boundary in order rather than immediately discarding the model:

1. Experiment integrity: sample-ratio mismatch, unstable assignment, missing exposures, or unequal fallback rates.
2. Serving parity: online features, filters, candidate generation, freshness, latency, and model version match the offline replay.
3. Evaluation construction: future leakage, an unrealistic random split, different candidate pools, or a favorable negative sampler.
4. Objective mismatch: NDCG optimized historical clicks while the product values satisfaction, retention, purchases, or diverse discovery.
5. Policy bias: the candidate explored items and positions not represented in old logs, so the offline labels undervalued or misvalued them.
6. Heterogeneous effects: a global gain hid losses for new users, a market, an item class, or a high-value segment.
7. Dynamics: novelty wore off, creators or suppliers adapted, or the new exposure distribution changed future training data.

The decision rule has three layers: offline evidence says the candidate is plausible and safe under known logged conditions; a randomized online experiment estimates its causal product effect; production monitoring checks whether that effect persists as users, inventory, and feedback loops change.

Example of a strong answer

“I would first make the two rankers comparable. The evaluation cutoff, eligible items, retrieval output, labels, users, and K=10 must be identical, and all features must exist at the request time. I would use an out-of-time test set and report exclusions and label maturity. Alongside the production model I would keep a popularity baseline, because a complex candidate that cannot beat it is not ready.

NDCG@10 is useful for order, but it is not the launch decision. I would add recall@10, then product-specific measures such as catalog coverage and within-list diversity, plus latency and fallback rate. Results need slices for cold users, sparse users, new items, and important markets. I would also state how negatives were formed. The historical policy determined exposure, so unexposed items are unknown; if negatives are sampled, every model must use the same sampler and the numbers are not comparable with full-catalog metrics.

If the candidate clears those checks, I would randomize persistently by user. Assignment and exposure are logged before outcomes, and the analysis keeps fallbacks in the assigned arm. I would predefine one primary outcome, the minimum detectable effect, power, horizon, guardrails, and rollback criteria. I would verify sample-ratio balance and telemetry before interpreting lift.

If offline NDCG rises but the online outcome falls, I would check experiment integrity and serving parity first. Then I would inspect time leakage, candidate and sampling mismatches, metric proxy failure, exposure bias, and segment effects. The offline result earns the candidate an experiment; only the online causal result, together with guardrails, earns a launch.”

Common mistakes

• Launching on higher NDCG alone → Offline ranking quality is conditional on historical labels and does not estimate the causal product effect → Use it as a screening signal, then require a randomized online result and guardrails.
• Randomly splitting interaction rows → Later user behavior or item state can leak into training → Use a point-in-time split and construct features, catalog eligibility, and labels as they existed then.
• Changing retrieval and ranking together without saying so → The source of the gain cannot be identified → Hold candidate generation fixed for a ranker comparison or label the treatment as end to end.
• Treating every unobserved item as negative → Many items were never exposed → Distinguish exposed non-actions from unknown items and document the evaluation candidate set.
• Using different negative samples for each model → The comparison changes both model and test difficulty → Share the pool, sampler, size, and seeds, or rank the full catalog.
• Reporting only one aggregate metric → A global average can hide cold-start or market failures → Report predefined slices and uncertainty alongside the aggregate.
• Optimizing clicks without a product objective → Position bias or clickbait can improve the proxy while harming satisfaction → Define the primary outcome and harm guardrails before model selection.
• Removing fallbacks from the treatment analysis → This hides a real serving failure and breaks randomization → Use intent-to-treat analysis and report fallback as a guardrail.
• Peeking until the experiment is positive → Repeated unplanned stopping inflates false positives → Fix the horizon and analysis plan or use a valid sequential design.
• Declaring an offline-online gap “model drift” immediately → Broken logging, parity, or assignment often produces the same symptom → Check experiment integrity and serving parity before model behavior.

Follow-up questions and responses

Follow-up 1: What if you cannot run an online experiment?

Use an out-of-time replay, multiple historical windows, strong baselines, slice tests, and a shadow or limited canary to reduce operational risk, but state that causal product impact remains unproven. If randomized propensities were logged and the new policy has sufficient support, add IPS, self-normalized IPS, or doubly robust estimates with overlap and variance diagnostics. Do not present them as permission to extrapolate beyond logged support.

Follow-up 2: How would you evaluate new users and new items?

Create explicit cold-start cohorts at the evaluation cutoff. Prevent later user history or item interactions from entering features. Compare fallback coverage, first-session outcomes, new-item exposure, and time to first meaningful action. A global metric dominated by established users and popular items does not answer the cold-start question.

Follow-up 3: What if watch time rises but diversity falls?

Return to the predefined objective and constraints. If diversity protects long-term satisfaction, catalog health, or user choice, treat it as a guardrail or a constrained optimization target rather than averaging unrelated metrics after the result. Examine concentration and repeat-exposure distributions by segment, then test a reranker or constraint as a new treatment.

Follow-up 4: When should you randomize by cluster instead of user?

Use clusters when treatment spills across units: household profiles share a screen, social recommendations affect friends, or marketplace exposure changes shared inventory. Randomizing individuals would violate independence and contaminate arms. Cluster assignment reduces the effective sample size, so power calculations must include within-cluster correlation.

Follow-up 5: Can counterfactual evaluation replace A/B testing?

It can screen policies when action propensities are known and overlap is adequate. IPS corrects exposure by weighting observed rewards, self-normalization trades some bias for variance control, and doubly robust methods combine a reward model with propensity weighting. All can fail with missing propensities, poor overlap, extreme weights, or an incorrect reward model, so a launch claim still benefits from a randomized online test.

Follow-up 6: How do you detect a harmful feedback loop after launch?

Monitor exposure concentration, creator or supplier reach, catalog coverage, repeated impressions, and outcome quality over successive cohorts—not only immediate engagement. Preserve model and policy versions in logs, compare training-data composition over time, and retain a small reference or exploration policy when feasible. Roll back or constrain the policy when concentration or harm guardrails breach their predefined limits.