1. Question and context
A SaaS product is moving from fixed subscriptions to usage billing. Usage events can be duplicated, late, or sent with the wrong dimension, and aggregation updates asynchronously. After a release, some customers see estimated usage above their real usage, while some invoices are already finalized. Design a product policy from discovery and verification through billing pause, correction, communication, and review. Explain the trade-off between revenue accuracy and customer trust.
2. What the interviewer evaluates
- Whether you distinguish event ingestion, aggregation, invoice estimation, and invoice finalization.
- Whether you define an actionable correction window, evidence standard, approval authority, and customer notice instead of only saying “refund manually.”
- Whether you consider idempotency identifiers, aggregation formulas, late events, negative adjustments, invoice cutoffs, and irreversible operations.
- Whether monitoring, dispute taxonomy, a compensation budget, and prevention form a product loop.
3. Clarifications to ask before answering
- Does the issue affect raw events, an aggregate, an estimated invoice, or a finalized invoice already sent?
- Is billing based on event count, a summed value, or the last value, and are there multiple dimensions?
- What corrections are allowed for the current and historical periods, who approves them, and when will customers see the result?
- Must all billing stop, or can the affected meter, tenant, or event window be isolated?
4. A 30-second answer framework
I would define an impact model and state machine: event ingestion, aggregation, estimated invoice, and finalized invoice. First isolate the affected meter or tenant, stop bad data from entering billing, and retain raw events and audit records. Correct the current period with traceable cancellation or negative adjustment; for a finalized invoice use credit, refund, or a future-period offset with an explicit irreversible boundary. Customers see correction status, evidence scope, and the next update time. After recovery, reduce recurrence with deduplication, versioned rules, reconciliation, and alerts.
5. Step-by-step deep answer
Step 1: Map the metering lifecycle and impact tiers
Separate event ingestion, aggregation, usage summary, estimated invoice, and finalized invoice into states. Successful ingestion does not mean the customer invoice is fixed; asynchronous aggregation may continue to change. Calculate impact by state, tenant, meter, and time window, prioritizing finalized invoices and the largest amounts.
Step 2: Define evidence and the freeze scope
Replay raw records using event ID, customer ID, event time, value, and dimensions, then compare production logs, meter summaries, and invoice snapshots. Minimize the freeze to an affected meter, version, or tenant and keep unaffected billing running. Add temporary deduplication for suspected duplicates, but never delete raw events that are needed for audit.
Step 3: Select a correction path by invoice state
For the current period, cancel an incorrect event, record a negative adjustment, or recompute the aggregate. A finalized invoice should not be silently rewritten; use a credit, refund, or next-period offset and show the reason and related event in the customer record. State the correction deadline, amount threshold, and finance approval so support agents apply one policy.
Step 4: Design communication and disputes
Tell customers the affected scope, whether the current invoice will change, what they need to do, and when the next update will arrive. Offer downloadable event detail, aggregation rules, and correction records while hiding unnecessary internal implementation detail. Collect the customer’s observation, time window, and business impact, then triage by evidence. When a metering error is confirmed, trigger compensation automatically instead of making the customer repeatedly prove loss.
Step 5: Verify recovery with metrics and guardrails
Monitor duplicate-event rate, late-event rate, aggregation delay, correction amount, dispute rate, refund time, and revenue variance. Set an automatic freeze threshold and a stop condition for expanding the rollout. Review idempotency, aggregation-formula changes, retry paths, and the boundary before and after invoice finalization; validate the new rules with a small replay first.
6. High-quality sample answer
I would separate event, aggregate, estimated-invoice, and finalized-invoice states and measure impact by meter, tenant, and billing period. For duplicate reporting, isolate the affected scope, stop bad data entering invoices, and preserve raw events and the audit trail. Recompute the current period with cancellation or a negative adjustment; use credit, refund, or a future-period offset for a finalized invoice, with clear approval and deadline rules. Customers receive scope, evidence, next update time, and a dispute entry point. After recovery, monitor duplicate rate, aggregation delay, disputes, and correction amount, then add idempotency, versioned rules, reconciliation, and automatic-freeze guardrails.
7. Common mistakes
- Freezing every customer after one anomaly → unrelated revenue and customers are affected → isolate by meter, tenant, and time window.
- Overwriting the aggregate → audit evidence is lost → preserve raw events and correct with traceable adjustments or compensating events.
- Treating an estimate as a finalized invoice → the wrong correction path is chosen → define the state machine and invoice cutoff.
- Promising a refund without policy → customers cannot predict outcomes → publish evidence, windows, approval, and update times.
- Fixing only the current data → the issue recurs → add idempotency, reconciliation, alerts, and replay validation.
8. Follow-up questions and responses
Follow-up 1: When can you cancel an incorrect event?
Cancel it while the event is in a correctable billing period and can be located by a stable ID. After finalization, use credit, refund, or a future-period offset and retain the linkage.
Follow-up 2: Why not edit the meter configuration directly?
The aggregation formula and event-field definitions affect the whole period. Editing them in place makes history hard to explain. Version the configuration, activate old and new meters separately, and replay and reconcile when needed.
Follow-up 3: How do you decide whether to freeze all billing?
Compare anomaly rate, financial risk, how precisely the error can be isolated, and correction speed. Freeze only the affected scope when it is provably safe; expand the freeze when other events cannot be trusted, with a maximum recovery time.
Follow-up 4: How do you measure whether the policy works?
Track duplicate-event rate, correction duration, dispute rate, automatic-compensation share, revenue variance, and retention. Sample customer cases to confirm that event detail and invoice changes are understandable.