System Design Interview: Design a Safe Feature Flag Service
Prompt and use cases
Design a feature flag service that enables teams to toggle features without redeploying, supports environment, user-segment, and percentage targeting, and remains safe when the configuration service is unavailable. Follow-ups may cover low-latency evaluation, auditability, approvals, rollback, and multi-region availability.
This fits platform engineering, backend, and system-design roles. Public interview libraries describe similar prompts with million-scale evaluations per second and very low latency; a published platform-engineering interview report emphasizes the control plane, data plane, and safe rollout boundary. Focus on those decisions rather than naming random components.
What the interviewer evaluates
- Whether you split a write-light, read-heavy configuration system into control and data planes.
- Whether percentage rollout is stable, so a user does not switch versions between requests.
- Whether defaults, expiry, kill switches, and rollback limit the blast radius of a bad configuration.
- Whether you explain consistency, latency, auditability, and tenant isolation trade-offs.
Clarifications before answering
Ask whether evaluation runs in an SDK, edge node, or central service; whether the target is one million evaluations per second; whether targeting uses users, organizations, regions, or sessions; how quickly changes must take effect; and what safe default each flag has. Also clarify whether experimentation, emergency disablement, and approvals are required.
A 30-second answer
I would define a control plane for flag creation, rules, versions, approvals, and audit logs, then a data plane where SDKs or edge caches fetch a published snapshot and evaluate locally. Rules match environment and target segments; percentage rollout uses a stable subject hash. Release gradually, watch alarms, and roll back automatically. If the control plane is down, use a bounded-stale snapshot or a risk-appropriate safe default.
Step-by-step solution
Core data model
A flag needs a key, environment, default value, ordered rules, version, publication time, expiry, and audit metadata. Rules may target organizations, regions, user attributes, or a percentage bucket. Rule order must be explicit: the first match wins. Validate syntax, types, and semantic conflicts before publication.
Control plane and data plane
The control plane handles writes, approvals, versioning, audits, and publication. The data plane reads only published snapshots and evaluates them. SDKs prefer an in-memory cache and refresh through polling, a stream, or notifications. A short control-plane outage therefore does not put a network call on every business request.
Stable rollout and safe release
Hash flagKey + stableSubjectId + salt and map it to buckets 0 through 9999. As exposure grows from 1% to 5%, 25%, 50%, and 100%, users already admitted stay on the same version. Validate rules and dependencies before release; watch errors, tail latency, and business metrics during release. Roll back to the last verified version when alarms fire.
~~~text evaluate(flag, context, snapshot): rules = snapshot[flag].rules[context.environment] for rule in rules: if matches(rule.targeting, context): if rule.percentage is absent: return rule.value bucket = hash(flag.key + context.stableSubject + rule.salt) % 10000 if bucket < rule.percentage * 100: return rule.value return snapshot[flag].safeDefault ~~~
Key trade-offs
| Option | Benefit | Cost | When to use |
|---|---|---|---|
| Central evaluation | One rule source and fast updates | Network dependency per request | Low throughput or strict consistency |
| Local SDK evaluation | Low latency and control-plane isolation | More complex distribution and secure storage | High QPS with bounded staleness |
| Edge evaluation | Local response and regional resilience | Harder distribution and invalidation | Global traffic and regional isolation |
AWS AppConfig documents versions, environments, deployment strategies, validators, gradual targeting, and alarm-based rollback. That supports treating release workflow as part of the system; it does not require copying every AWS component into another product.
Model answer
I would split the service into control and data planes. The control plane stores flag definitions, rules, versions, and audit records. After validation and approval, it produces an immutable published snapshot. SDKs or edge nodes cache that snapshot and evaluate locally, avoiding a central-service call on every request.
Percentage rollout uses a stable subject identifier, so a user stays on one version as exposure increases from 1% to 25%. Every flag has a safe default and an expiry policy. If an application cannot refresh, it either uses the last unexpired snapshot or disables the risky behavior according to the flag’s policy. Expand gradually while watching error rate, P99 latency, and business metrics, then roll back automatically on alarms. A short-path kill switch can provide emergency disablement, with explicit costs for cache invalidation, authorization, and auditability.
Common mistakes
- Calling the central configuration service on every request and turning it into a latency single point of failure.
- Using random numbers for percentage targeting, causing one user to flip versions between requests.
- Omitting version, expiry, and audit data, making it impossible to explain who published what and when.
- Designing only an on/off API without validation, rollback, or safe defaults.
- Mixing eventual consistency, cache-staleness windows, and emergency disablement without separate risk policies.
Follow-up questions and responses
How do you keep user experience consistent across instances?
All instances use the same stable subject identifier, flag key, and salt, and expose the snapshot version. Never bucket on a random request ID. For anonymous traffic, use a persistent session identifier and state its lifetime.
What happens when the configuration service is down?
The data plane keeps serving the last unexpired snapshot. After its TTL, it follows the flag’s risk policy and returns a safe default. SDKs should expose snapshot age, refresh failures, and evaluation source so stale behavior is visible.
How do you disable a dangerous feature?
Provide a permission-protected kill switch with a shorter path than ordinary rollout for high-risk flags. Record the operator, reason, version, and affected scope even during an emergency.
How do you prevent bad targeting rules?
Run schema, type, conflict, and coverage checks before publication. Evaluate rules against offline fixtures so each target reaches an expected branch. Test high-risk changes in a shadow or very small environment first.
How do you handle multiple regions?
Replicate published snapshots by region while keeping local data-plane reads. Include version and publication time in every snapshot, monitor regional version skew, and alert when a region cannot update. Continue using the previous safe snapshot during the gap.
When should you avoid feature flags?
Pure static configuration, one-time migrations, or authorization decisions requiring transactional consistency may not belong in a flag system. If flag count, rule complexity, and expiry debt grow, assign owners, expiry dates, and cleanup metrics so runtime rules do not replace normal release discipline.