Prompt and context
An enterprise search results page is being redesigned with clearer sources, filters, and next actions on each result card. The interviewer asks how you would define success, why those measures matter, how you would validate them, and what you would do when instrumentation is incomplete.
This is a product measurement design question. HEART names five user-experience dimensions: Happiness, Engagement, Adoption, Retention, and Task Success. GSM means Goals, Signals, and Metrics. They should create a decision chain, not a checklist that forces every dimension into the scorecard.
What the interviewer is evaluating
- Whether you define the user job before naming numbers.
- Whether you separate a product goal, observable signals, and operational metrics.
- Whether you select only the HEART dimensions relevant to the change and explain exclusions.
- Whether you specify numerator, denominator, window, cohorts, and guardrails.
- Whether you can choose between an experiment, staged rollout, and fixing data gaps.
Questions to clarify first
- Are the primary users employees, customers, or the public? Assume employees for this case.
- Is the goal faster discovery, better answer quality, or broader search coverage? Focus first on task success and time to find.
- Do stable search sessions, result-open events, and document events exist? If not, state the measurement risk.
- Are permissions, sensitive documents, or index lag involved? They affect cohorts and guardrails.
- Can traffic be randomized, or must rollout happen by team? Collaborative products need a contamination check.
A 30-second answer framework
“I would define the outcome as employees finding an authorized, useful document with less backtracking. I would use GSM to translate that outcome into task-success signals, then use HEART with Task Success as the primary dimension and Happiness and Retention as supporting checks. The primary metric would have an explicit unit and window; clicks would remain diagnostic. Before launch I would lock guardrails, instrumentation checks, and staged-rollout rules.”
Deep-dive answer
Define the user job and outcome first
Do not start with “we want more clicks.” Start with what the user must accomplish after search, such as opening the right document and resolving the current question. Task success could be a completed downstream action or an explicit helpfulness response; explain what each proxy can and cannot observe.
Turn the goal into a GSM chain
Write the Goal first: employees find an authorized, useful document faster. Then list Signals: opening a result, copying content, returning to search, reporting no help, or completing a related task soon after. Finally choose Metrics with explicit denominators and windows. A Goal is not a metric name, and a Signal is not automatically final success.
Select a small set of HEART dimensions
Prioritize Task Success because the redesign directly changes the finding task. Use Happiness through a short survey or feedback to check perceived quality, and Retention to see whether teams keep using search over four weeks. Adoption and Engagement can diagnose discovery and usage, but they should not become success criteria by default. Selecting fewer dimensions makes tradeoffs visible.
Write an auditable primary metric
One example is “the share of eligible search sessions that open an authorized result within 10 minutes and do not immediately return to search within 30 seconds.” The unit is a search session; the denominator excludes empty queries, bots, and internal tests; the numerator still requires a permission check. Dwell time is only a proxy and should be validated against reformulation, feedback, or downstream-task events.
Give diagnostics and guardrails separate jobs
Diagnostics can include first-result click-through, filter use, reformulation rate, zero-result rate, and p50/p90 time to find. Guardrails include permission errors, sensitive-document exposure, index latency, user reports, and support requests. If clicks rise while reformulation, reports, or permission errors rise, the primary metric does not justify a full rollout.
Handle bias in happiness and retention
Happiness feedback is affected by response self-selection, prompt timing, and language. Report response rate, survey version, and cohorts instead of treating a small high-scoring sample as the whole population. Match the retention window to search frequency: daily products may use 7 or 28 days, while infrequent workflows need longer observation. Report short-term task success separately from repeated use.
Design validation and decision rules
If users share links or collaborate on queries, randomize by team or workspace to reduce cross-version interference; otherwise user-level randomization may be adequate. Before launch, pre-register the primary metric, guardrail limits, minimum practical lift, and observation window. Check sample ratio, event loss, and permission validation first. Expand only when the primary threshold, guardrails, and data-quality checks all pass; pause when a critical guardrail is exceeded.
Model high-quality answer
“I would define success as employees completing a finding task faster, not clicking more results. The Goal is an authorized, useful answer in the enterprise knowledge base. Signals include opening a result, returning to search quickly, reporting no help, and completing a related task. In HEART, Task Success is primary, Happiness checks perceived quality, and Retention observes four-week reuse; Adoption and Engagement are diagnostics.
The primary metric could be the share of eligible search sessions that open an authorized result within 10 minutes without an immediate reformulation. Sessionization, numerator, denominator, bot exclusions, and permission failures belong in a metric contract before the experiment. First-result clicks, zero-result rate, and p90 time to find diagnose the funnel; sensitive exposure, index lag, reports, and support requests are guardrails.
I would check for team-level interference before choosing team randomization or staged rollout. Before reading outcomes, I would verify sample ratio, event loss, and permission checks. I would expand only when the preset minimum lift, guardrail intervals, and data quality pass. If clicks rise but reformulation or reports rise, I would pause and fix ranking or permissions. Happiness results would include response rate and cohorts so self-selected feedback is not presented as overall satisfaction.”
Common mistakes
- Making all five HEART dimensions primary: conflicts leave no release decision → choose one primary dimension and assign the rest explicit roles.
- Equating clicks with task success: users may return immediately → keep clicks diagnostic and add reformulation, feedback, or task-window signals.
- Writing Goals without Signals: the goal cannot be instrumented → list observable behavior before defining metrics.
- Reporting only average happiness: respondents and prompt timing bias the estimate → report response rate, version, and cohorts.
- Ignoring permissions and index lag: apparent success can damage trust → make exposure, latency, and reports guardrails.
- Randomizing collaborative users individually: teammates can cross-contaminate versions → choose a team-level unit when sharing is material.
- Leaving thresholds until after results: teams can change the definition later → lock lift, windows, and a decision matrix before launch.
Follow-up questions and answers
Follow-up 1: Why not make Adoption primary?
Adoption shows that users started using the redesign, not that search tasks succeeded. It is useful for discovery and migration diagnostics; the primary dimension should stay close to finding a usable answer.
Follow-up 2: Is a 30-second dwell rule arbitrary?
It is a proxy, not correctness. Calibrate it against historical dwell distributions, document types, and task research, then combine it with reformulation or feedback. Without calibration, downgrade it to a diagnostic signal.
Follow-up 3: What if there is no happiness survey?
Say that Happiness cannot be measured directly yet. Use low-friction feedback, no-help reasons, reports, and support text as proxies, and record their bias. Do not label behavioral proxies as subjective satisfaction.
Follow-up 4: How long should retention be measured for infrequent search?
The window should cover the expected interval before the task recurs. An infrequent internal workflow may need monthly or quarterly observation, while short-term task success can remain the rollout gate.
Follow-up 5: Clicks and task success rise, but time to find gets longer. What now?
Break the result down by rank position, filters, document type, and user experience. If task success improves while a time guardrail is exceeded, optimize ranking or interaction before expanding; do not select only the favorable metric.
Follow-up 6: How do you answer when instrumentation is missing?
Make data quality a release prerequisite. List missing events, replay or log-reconciliation checks, and coverage targets in the metric contract. Until coverage passes, the experiment is invalid; it cannot be interpreted as no effect.