Prompt and context
A core revenue metric needs a correction to its refund treatment, while hundreds of dashboards, alerts, and data products still use the old definition. Design a metric versioning and deprecation contract that makes the change explainable, migratable, and reversible.
Do not bind the answer to one catalog or semantic-layer vendor. Focus on definitions, dependencies, compatibility windows, release gates, consumer notices, and retirement evidence.
What the interviewer evaluates
Semantic boundaries
Can you turn a metric name into a contract containing formula, filters, time semantics, grain, unit, timezone, version, and owner instead of silently changing SQL?
Impact analysis
Can you enumerate dashboards, alerts, exports, models, and APIs, distinguishing direct from indirect dependencies?
Migration governance
Can you run old and new versions together with deadlines, approvers, and migration states instead of causing a hidden big-bang break?
Verifiable retirement
Can you prove retirement with usage, reconciliations, alert replays, and evidence that old calls are gone?
Clarifying questions to ask
- Is the change a bug fix, a business-definition change, or a source migration?
- Do finance, audit, historical recomputation, or legal retention require the old metric?
- Do consumers use SQL, BI, APIs, exports, or machine-learning features?
- Is there a cross-team or external-customer SLA?
- How long may both versions run, and who can extend the window?
- If values diverge, do we roll back the definition, data, or presentation layer?
30-second answer framework
“I would treat the metric definition as a versioned contract containing formula, filters, grain, time semantics, and owner. First I would build a dependency graph and usage snapshot, then publish a new version while retaining the old one; every response would expose version and effective time. Migration would prioritize high-risk consumers and use fixed samples, historical replays, and reconciliations. During deprecation I would notify owners and block new usage. I would retire only after old calls reach zero, critical consumers confirm, and audit records are complete, while retaining recoverable definitions and result snapshots.”
Step-by-step deep dive
Step 1: Freeze the current contract
Record the old version's name, formula, filters, grain, unit, timezone, source, freshness, owner, sensitivity, and effective time. Generate an immutable version for every change; never overwrite silently.
Step 2: Build dependency and risk graphs
Collect dependencies from the semantic layer, query logs, BI metadata, scheduled jobs, alert definitions, and API calls. Mark finance, customer-visible, near-real-time, and machine-learning consumers, then rank migration by impact and usage.
Step 3: Define compatibility
An alias or display-only change may use a compatibility alias. A formula, grain, or time-semantics change gets a new version. Responses, export metadata, and documentation return version, unit, and definition reference so consumers do not guess.
Step 4: Gate the new release
Run the new version in a sandbox and with a small consumer cohort. Gates check expression parsing, sample values, historical replay, nulls, units, permissions, latency, and cost. Owners and affected consumers approve before expansion.
Step 5: Migrate and notify
Assign an owner, deadline, and state to every dependency. Use catalog status, CI checks, query hints, and recurring reports to notify old-version users. Block new dashboards from referencing the old version; exceptions need an expiry.
Step 6: Accept, roll back, and retire
Compare versions on fixed samples, historical windows, and critical dashboards, explaining changes caused by refunds, late data, or timezones. Keep the old definition and result snapshots; pause retirement or switch back when anomalies appear. Retire only after old calls reach zero and audit, consumer confirmation, and rollback materials are complete.
Strong sample answer
“I would freeze the current revenue definition as v1, explicitly documenting recognition time, refund treatment, currency, timezone, grain, and owner. Query logs and catalog metadata would produce a dependency graph, with financial reports, customer invoices, and alerts marked high risk.
If correcting refunds changes the formula, I would publish v2 instead of overwriting v1. Both versions would run in parallel, and values would include version, unit, and freshness. CI would block new v1 references, while the migration list records owners and deadlines. I would reconcile known orders and fixed months, then replay alerts, exports, and APIs; every difference needs an explanation.
After high-risk consumers confirm, v1 usage is continuously zero, and documentation and audit records are complete, I would freeze v1 read-only and set a final shutdown date. I would retain its definition and result snapshots so historical reports remain traceable and anomalies can be recovered.”
Common mistakes
- Editing same-named SQL so historical reports silently change meaning.
- Inspecting only catalog references and missing query logs, alerts, exports, or APIs.
- Reusing one cache key or result table for old and new versions.
- Omitting unit, timezone, grain, or version fields and forcing consumers to guess.
- Sending an announcement without owners, deadlines, or enforcement.
- Using average difference to hide a critical month or high-risk customer's divergence.
- Deleting the old definition before historical explanation or rollback is possible.
- Treating one usage drop as proof while missing batch jobs and infrequent audit queries.
Follow-up questions and responses
Follow-up 1: What counts as a breaking change?
A formula, filter, grain, unit, timezone, source-reliability, or permission-semantics change is breaking. An alias or description edit is compatible only when the result contract remains unchanged.
Follow-up 2: How do you stop new dashboards from adopting the old version?
Mark the old version deprecated, and have CI and the semantic layer reject new references. Query hints show the replacement; exceptions require an owner, reason, and expiry.
Follow-up 3: How do you show a numeric difference is not a bug?
Replay fixed samples, historical windows, boundary orders, and reconciliation totals. Decompose differences by refunds, lateness, currency, and timezone, then obtain business-owner sign-off.
Follow-up 4: What if a low-frequency consumer never migrates?
Set a risk-based hard deadline and provide a migration report and replacement query. Allow controlled extensions for compliance or billing consumers, recording reason, approver, and new date.
Follow-up 5: Can you answer historical questions after retirement?
Retain immutable definitions, versions, input snapshots, or replayable materials, and record the version used by each historical report. Removing a live endpoint does not remove audit evidence.