1. Prompt and scenario
Your team is preparing an AI feature that summarizes customer-support conversations. Several summaries may be reasonable, human labeling is expensive, and one automated score cannot show whether the customer’s problem was actually resolved. Design an evaluation approach that decides whether to launch, for whom, and how to converge after failures.
2. What the interviewer is testing
- Whether you define the user task and unacceptable failures before naming a model score.
- Whether you combine offline tests, human review, online behavior, and safety monitoring.
- Whether you separate leading signals, lagging outcomes, and guardrails when they conflict.
- Whether you turn evaluation sets, risk tolerance, human escalation, and rollback into product mechanisms.
3. Clarifying questions to ask
- Is the output a draft, a recommendation, or an automatic irreversible action?
- Does success mean less handling time, higher first-contact resolution, or fewer complaints?
- Which errors could cause privacy, compliance, or customer harm and require interception?
- Do the target users, languages, industries, and data distribution match the evaluation sample?
4. A 30-second answer framework
I would define the task and risks first, then build a representative evaluation set. I would measure four layers: task outcome, output quality, trust and safety, and cost and latency. Without one ground-truth label, I would calibrate automated signals with expert pairwise preferences or pass/fail rules, then add online experiments, user feedback, and human escalation to continuous monitoring. Launch gates would be layered: a failed hard guardrail stops rollout, while quality and business value must both clear thresholds before expansion.
5. Step-by-step solution
Step one: Turn “good output” into an observable task
Write down the input, user action, and desired outcome. The summary task is not “look like a reference answer”; it is whether an agent can find the customer’s need, promise the correct next step, and avoid exposing sensitive information within the available time. Define an acceptable range and explicit failure examples for each outcome.
Step two: Build a layered evaluation set
Sample different languages, customer types, conversation lengths, and edge cases. Have domain experts use a rubric that permits multiple valid answers and scores key facts, omissions, tone, and privacy separately. Keep a frozen regression set and a recent sample for distribution change. When evidence is insufficient, record uncertainty instead of inventing a label.
Step three: Combine metrics and guardrails
A primary metric might be task completion, first-contact resolution, or post-edit adoption. The quality layer checks factual correctness, coverage, and citation completeness. Guardrails monitor harmful content, privacy leakage, bias, escalation, and complaints. Cost, latency, and review hours determine sustainability. Document each metric’s denominator, sampling window, and failure conditions.
Step four: Connect evaluation to the release decision
Run offline regression first, then expose a small, reversible cohort. A hard guardrail failure stops the feature; weak quality or business results trigger diagnosis at the prompt, retrieval, model, or interface layer. Feed online failures and human feedback into the evaluation set, and check whether the metrics still represent real risk. Keep human confirmation and a kill switch for high-risk actions.
6. Model answer
I would not treat one automated score as the answer. I would define the user task, tolerated errors, and irreversible risks, then freeze an evaluation set that matches the real distribution. Domain experts would score facts, coverage, tone, and privacy with a rubric, using pairwise comparison when several answers are valid.
>
Before launch I would measure task outcomes, output quality, trust and safety, cost, and latency. Task completion is the primary signal; privacy leakage, harmful content, and critical factual errors are hard guardrails. I would run offline regression and a small experiment with human confirmation for high-risk actions. If results miss the gate, I would classify failures across data, retrieval, model, and interface, update the set, and choose expansion, rollback, or shutdown.
7. Common mistakes
- Reporting only accuracy or an average score without defining the user task and failure impact.
- Treating one convenient sample as representative of every language, customer, and edge case.
- Calling higher clicks success while ignoring complaints or rework caused by wrong answers.
- Evaluating only before launch and ignoring production drift, feedback, and stale evaluation sets.
- Automatically executing uncertain output without human escalation, rollback, or a kill switch.
8. Follow-up questions and answers
Follow-up one: What if the human-labeling budget drops to one tenth?
Stratify by risk and spend the budget on high-impact and high-disagreement cases. Use sampled review, pairwise comparison, and user feedback for lower-risk cases. Keep an uncertain label and track sampling error; fewer labels do not provide the same certainty for free.
Follow-up two: What if task completion rises but privacy risk also rises?
Privacy is a higher-priority guardrail than the benefit metric. Stop expansion, isolate affected inputs and outputs, and fix redaction, retrieval permissions, or prompt policy. Relaunch only after the guardrail recovers and regression tests pass.
Follow-up three: How do you show that the metrics have not gone stale?
Feed production failures, human escalations, and user appeals back into the evaluation set, sliced by language, cohort, and version. Have domain experts review the rubric and record divergence between metrics and real outcomes. If divergence grows, change the metric or stop relying on the old gate.