Representative interview topic

System Design Interview: How Do You Balance Reliability and Release Speed with Error Budgets?

System designHard
Offer.cc Editorial TeamPublished Updated

Question

In a system design, how would you use error budgets to balance reliability and release speed?

Prompt and when it applies

An interviewer may ask, “In a system design, how would you use error budgets to balance reliability and release speed?” You must explain how SLIs, SLOs, and an error budget affect releases, rollbacks, capacity, and team coordination. This fits SRE, platform, backend, and senior system-design interviews involving high availability, frequent releases, or multiple team dependencies.

What the interviewer is assessing

The test is not whether you can recite 99.99%. It is whether you can turn a user-experience goal into an operational decision rule. Google SRE describes an error budget as the remaining space under an SLO, used to coordinate reliability work and innovation; when the budget is spent, ordinary changes pause while reliability is restored. Interviewers also look for a measurement window, data quality, progressive delivery, and auditable exceptions.

Clarifying questions to ask yourself

Clarify who the users are, which journey matters, where the service boundary lies, and whether the goal is availability, latency, freshness, or correctness. Confirm the SLO window, regions, ownership of dependencies, and rollback capability. If numbers are missing, state assumptions such as a four-week window, 99.9% availability, and measurement by valid user request.

A 30-second answer framework

Use five steps:

  1. Define a user-visible SLI and SLO.
  2. Calculate the window’s error budget and explain what consumes it.
  3. Connect the budget to progressive delivery, automatic rollback, and change gates.
  4. When the budget is spent, freeze ordinary changes and fund reliability work, with explicit exceptions for security and urgent fixes.
  5. Use postmortems, dependency attribution, and budget trends to adjust the next planning cycle.

Step-by-step deep answer

1. Define the SLI from user outcomes

Do not use server CPU or average latency as the reliability target by default. For a request service, choose the proportion of successful requests and requests below a latency threshold; for an asynchronous job, use on-time completion or result freshness. Separate user-impacting failures from internal retries and invalid traffic so noise does not spend the budget.

2. Choose an explainable SLO and window

For example, 99.9% successful valid requests over four weeks allows roughly 0.1% to fail. The window controls sensitivity to short incidents and long trends. With multiple SLOs, state the combination rule: a critical user journey can block release, while a secondary metric informs alerts and planning. Do not average percentages without explaining the population and weights.

3. Attribute budget consumption to changes

Record total budget, burn rate, and cause. Tag releases, configuration, dependency failures, capacity shortages, and false positives separately. Only trustworthy attribution tells the team whether to fix code, add capacity, change a dependency contract, or repair monitoring. Google’s example policy also distinguishes failures in the service, failures owned by another team, and traffic outside the SLO scope.

4. Design release gates and progressive rollback

Send an ordinary change to a small traffic fraction or one region, observe error rate, latency, and burn rate, then expand. The gate should check both remaining budget and short-window burn rate; a safe monthly average can hide a rapidly worsening incident. When behavior is unexpected, roll back before diagnosing to reduce recovery time. The rollback needs idempotency and data compatibility as well.

5. Define what happens after the budget is spent

Spending the budget does not mean development stops forever. Freeze ordinary features and nonessential data changes, then prioritize capacity, testing, dependency isolation, graceful degradation, and root-cause fixes. Security fixes and urgent defects that address the SLO miss can be exceptions, but record the reason, approver, and follow-up review so “urgent” does not become a permanent bypass.

6. Handle dependencies and cross-team ownership

Do not hide every external failure inside the service’s SLO. Track dependency, client, and service errors separately, and make repair and communication ownership explicit. When teams disagree about budget rules, align on the user journey and shared measurements, then escalate through the service owner. Transferring blame is not a reliability strategy.

High-quality sample answer

This fictional answer must be replaced with your own numbers and boundaries:

I would define success rate and latency SLIs for the most important user request. Suppose the valid-request success SLO is 99.9% over four weeks, so the error budget is 0.1% of valid requests. Monitoring shows remaining budget, one-hour burn rate, and failure attribution, excluding traffic outside the service boundary. Releases start in one region at a small fraction of traffic and expand gradually; crossing a burn threshold stops the rollout and triggers rollback. While the budget is healthy, product and SRE can ship within the risk envelope. After it is spent, ordinary changes freeze and the team prioritizes capacity, testing, dependency isolation, and root-cause fixes, with logged exceptions for security work. Every incident gets a blameless review and the fixes enter the next planning cycle. Release speed is therefore governed by remaining budget and observed risk rather than by preference.

Common mistakes

Treating the error budget as permission to cause failures

The budget represents user-tolerated failure space, not a quota for creating incidents. Explain user impact, window, burn rate, and repair ownership.

Giving only an availability number

Without an SLI, population, and window, 99.99% cannot guide a decision. Add valid requests, latency or freshness, and rules for excluding irrelevant traffic.

Freezing releases forever after a budget miss

That ignores security fixes, migrations, and recovery work. Define exceptions, approvals, rollback, and review so exceptions do not become the default path.

Ignoring progressive delivery and rollback compatibility

“Monitor and release” is not enough. Describe traffic stages, automatic stop conditions, rollback order, and compatibility between old and new reads and writes.

Follow-ups and advanced practice

The SLO is healthy, but the one-hour burn rate is high. Would you release?

Compare the long-window balance with the short-window trend. If consumption is sustained, stop expanding traffic, verify whether it is real user impact, a traffic spike, or a monitoring defect, and then decide whether to roll back.

An external dependency spent the budget. Should your team freeze too?

Check whether the service commitment includes that dependency failure, then decide from the user journey. Protect users and enable degradation even when ownership is external; the freeze rule should be agreed in advance with evidence-sharing and escalation.

Several SLOs fail at once. How do you prioritize?

Rank by critical user journey, blast radius, burn rate, and reversibility. Address the indicator that can expand the incident or block recovery first, then local performance issues, and state the tradeoff.

A product manager asks you to release after the budget is spent. How do you respond?

Turn the debate into data: show remaining budget, user impact, rollback cost, and repair time. Offer a small experiment or delay. If a genuine business emergency remains, use a fully recorded exception and schedule the reliability work and review.

Public sources

Related questions

Related interview tool

Use Solve for a system design answer

Clarify the requirements first, then move through scale, architecture, component choices, and trade-offs.

View the tool