How Would You Design a Feature Flag System?

Question and when to use it

Design a multi-tenant feature flag system used by 20,000 server instances across 3 regions. It stores 50,000 flags across projects, handles 5 million in-process evaluations per second with less than 1 millisecond of added p99 latency, propagates a published change within 5 seconds at p99, and keeps evaluating through a 15-minute control-plane outage. It must support typed variations, targeting, deterministic percentage rollouts, audit logs, approvals, and an emergency off switch.

The numbers are interview assumptions, not product benchmarks. Assume each runtime subscribes to at most 500 relevant flags, an average serialized flag definition is 2 KB, control-plane writes peak at 100 per second, and server-side evaluation rules may contain sensitive attributes. Client and mobile delivery, experimentation statistics, and a general-purpose configuration service are follow-ups rather than base requirements.

This question fits senior backend, platform, infrastructure, release-engineering, and system-design interviews. The reusable lesson is that a system can make the management path strongly governed while keeping the request path local and available. That separation determines nearly every later choice.

What the interviewer is evaluating

The first signal is whether the candidate separates deployment from release and control plane from data plane. A dashboard and database manage flag definitions. Application SDKs evaluate a versioned ruleset in process. A remote RPC for every flag check would put a management outage and network latency into every product request.

The second signal is semantic precision. A flag evaluation needs a flag key, environment, typed fallback, and evaluation context. Ordered rules, explicit targets, percentage allocation, and the default rule must have deterministic precedence. “Randomly give 10% of requests the feature” is wrong when a user must receive a stable experience across requests and regions.

The third signal is failure design. Last-known-good state preserves evaluation during a control-plane outage, but it also makes stale configuration an explicit risk. A strong answer defines startup behavior, maximum acceptable staleness, gap recovery, invalid-update rejection, emergency disable behavior, and different defaults for a cosmetic change versus a dangerous write path.

The fourth signal is operational ownership. Flag edits are production changes. Authentication, authorization, environment separation, optimistic concurrency, validation, approval policy, immutable audit records, staged publication, rollback, ownership, and retirement all belong in the design.

Questions to clarify before answering

• Where is evaluation performed? Server-side SDKs can receive the complete ruleset and evaluate locally. Browser and mobile clients cannot safely receive sensitive targeting rules, so they need a filtered or remotely evaluated model.
• What does the 5-second objective measure? Define it from a successful publish to 99% of healthy subscribed server instances applying that version. Offline instances and clients outside the supported freshness window need separate semantics.
• What is the fallback contract? If an SDK has never loaded a valid snapshot, it returns the typed default supplied by application code. After initialization, it may use the last-known-good version for the agreed stale window.
• Must the same subject stay in the same cohort? Percentage rollout requires a stable targeting key and deterministic hash inputs. Anonymous sessions need a durable identifier if stability matters.
• Can rules contain sensitive data? Prefer segment IDs and non-sensitive attributes. Server-side delivery can carry protected rules; browser delivery must not expose secrets, internal allowlists, or authorization logic.
• How critical is an emergency off switch? A five-second propagation SLO is useful but not instantaneous. Truly destructive operations still need server-side authorization and independent safety controls.

30-second answer framework

“I would split the platform into a governed control plane and a local evaluation plane. The control plane validates, versions, audits, and publishes immutable snapshots and patches. Server SDKs keep the last valid rules and evaluate targets, ordered rules, rollouts, and defaults in process; before initialization or on invalid state they return the code fallback. A deterministic rollout hashes a stable subject key with the environment, flag key, and salt. Streams push updates and polling repairs gaps, so a control-plane outage never enters the request path. I would test propagation, cross-SDK results, cohort stability, stale behavior, and rollback under faults.”

Step-by-step solution

Step 1: Turn the prompt into explicit contracts

Functional requirements are create, edit, approve, publish, disable, and retire flags; define boolean, string, number, or structured variations; target subjects or segments; allocate percentages; and return evaluation details for debugging. Non-functional requirements are local p99 below 1 millisecond, 5-second p99 propagation, deterministic results across regions, and continued evaluation for a 15-minute management outage.

Define three boundaries before drawing components:

1. A published version is immutable. A later edit creates another draft and version.
2. The propagation SLO applies to healthy connected server instances, not devices that are offline.
3. The application owns the final fallback value. The platform never invents a value when type checking, initialization, or evaluation fails.

For the base design, last-known-good state remains valid for at least the required 15 minutes. After a flag's published staleafter boundary, its staleaction chooses either continued last-known-good evaluation or the code fallback, and the SDK returns a stale reason. A security-sensitive or destructive path must choose the fallback rather than silently serving an old enablement rule.

This avoids hiding product policy inside infrastructure. A checkout migration may default to the old path, while a security-sensitive capability may default to disabled. Both use the same platform with different application contracts.

Step 2: Separate the control plane and evaluation data plane

The control plane contains the management API and UI, identity and role checks, schema and rule validator, approval workflow, relational source of truth, append-only audit log, snapshot builder, and publisher. Writes are low-volume compared with evaluations, so correctness, reviewability, and recoverability matter more than microsecond latency.

The data plane contains regional stream relays, snapshot storage, polling endpoints, and SDK-local stores and evaluators. Server SDKs open a regional stream, install an immutable snapshot, apply versioned patches, and evaluate without a network call. If the stream breaks, they retain the last valid state and poll with jitter. If a patch skips a version or fails checksum validation, the SDK discards the patch and requests a full snapshot.

Keep data-plane delivery independent of the primary control database. The publisher writes a durable versioned artifact before notifying relays. A database or dashboard outage can then stop new edits without stopping existing evaluations.

Step 3: Define the data model and APIs

A flag definition needs at least:

FlagDefinition {
  tenant_id, project_id, environment, flag_key
  version, value_type, variations[], off_variation
  ordered_rules[], default_rule, salt
  stale_after, stale_action
  state, owner, expires_at
}

Rule {
  rule_id, conditions[], outcome
}

Outcome = fixed_variation | weighted_variations[]

Segments are separately versioned because many flags may reference one cohort. An audit entry records actor, time, reason, expected previous version, before-and-after references, approval, and publish result. Keep drafts separate from published artifacts so an unapproved edit cannot leak into evaluation.

Representative APIs are:

PUT  /v1/projects/{project}/environments/{env}/flags/{key}
     body: draft definition, expectedVersion

POST /v1/projects/{project}/environments/{env}/flags/{key}:publish
     body: draftVersion, reason, approvalToken

GET  /v1/sdk/bootstrap?project={project}&env={env}&after={version}
GET  /v1/sdk/stream?project={project}&env={env}

The expected version prevents two editors from silently overwriting each other. Management credentials never double as SDK credentials. Tenant, project, environment, and allowed delivery mode are part of every authorization decision.

Step 4: Make evaluation and percentage rollout deterministic

Use one documented order across SDKs:

1. Check flag existence, environment, type, and whether targeting is enabled.
2. Apply an explicit subject or segment target.
3. Evaluate ordered rules; the first matching rule wins.
4. Resolve a fixed variation or a weighted rollout.
5. Use the default rule when nothing matches.
6. Return the application fallback with an error reason when evaluation cannot produce a valid typed result.

For a weighted rollout, derive a bucket from stable inputs:

bucket = H(tenant_id || environment || flag_key || salt || targeting_key) mod 100000

Map the bucket into cumulative variation ranges. The same inputs produce the same result on every instance and region. Keeping the salt and algorithm version in the published definition makes behavior reproducible. Expanding an existing contiguous range can preserve the subjects already inside it, but arbitrary reweighting or changing the hash inputs can move users; expose that consequence during review.

Two independent flags with the same percentages need not select the same subjects because the flag key participates in hashing. If several flags must move as one cohort, target a shared versioned segment or use an explicit experiment key. Never hash mutable fields such as email when a stable subject ID exists.

Step 5: Deliver snapshots and incremental changes safely

The snapshot builder resolves project and environment scope, validates referenced segments and prerequisites, sorts rules deterministically, serializes a canonical artifact, and attaches a version and checksum. A transaction commits the published metadata and an outbox record; an asynchronous publisher stores the artifact and announces it to regional relays. This avoids committing a flag without scheduling its delivery.

An SDK boot sequence is:

1. Load a valid persistent last-known-good snapshot when configured.
2. Fetch the latest full or delta artifact from the nearest relay.
3. Atomically replace the in-memory snapshot only after type, version, and checksum checks.
4. Mark the provider ready, then serve local evaluations.
5. Keep a stream open for updates and poll as a repair path.

Never mutate the active ruleset in place. Build a new immutable snapshot and swap one reference so concurrent requests see either the complete old version or the complete new version. Record the applied version and evaluation reason in diagnostic details.

Step 6: Design failure semantics before the happy path

The emergency off switch uses the same durable publish path with a priority lane, not an unaudited mutable side channel. It can skip ordinary waiting for approval only under a pre-authorized break-glass policy, but still records the actor, reason, previous version, and result. A five-second propagation target cannot guarantee that every disconnected instance has switched; destructive actions require independent enforcement.

Step 7: Protect the platform from configuration becoming authority

Separate management roles by tenant, project, and environment. Production edits may require a second approver, while development edits may publish directly. Encrypt credentials, rotate SDK keys, rate-limit management calls, and ensure a server key cannot edit flags. An append-only audit log and immutable artifacts make incident reconstruction possible.

Treat client delivery differently from server delivery. Send browsers or mobile devices only flags explicitly approved for client exposure, and preferably send already evaluated values for their context. A user can inspect or change client state, so a flag may alter presentation but cannot grant authorization, bypass payment, or replace a server-side entitlement check. Minimize personally identifiable fields in evaluation context and pseudonymize telemetry identifiers.

Give every temporary release flag an owner and retirement condition. After the rollout is complete, first make the winning path permanent, then stop evaluating the flag, then remove the definition after code references disappear. Otherwise old branches and interacting flags expand the test matrix indefinitely.

Step 8: Recheck capacity and prove the design

At 500 subscribed flags averaging 2 KB, one runtime snapshot is about 1 MB. Bootstrapping 20,000 instances at once transfers roughly 20 GB before protocol and replication overhead. Put canonical artifacts behind regional relays or object delivery, use conditional versions, jitter reconnects, and cap bootstrap concurrency. A single 2 KB change fanned to 20,000 instances is about 40 MB of logical payload, so incremental delivery is much cheaper than full refresh.

Five million evaluations per second must not synchronously emit five million network events. Even a 200-byte event would be about 1 GB per second, or 86.4 TB per day before replication. Buffer, batch, sample, or aggregate ordinary diagnostics. Preserve complete assignment events only when an experiment contract requires them, and put telemetry on a bounded queue that can drop noncritical events without delaying evaluation.

Validation covers more than throughput:

• Run the same golden rules and contexts against every SDK implementation and compare value, variant, reason, and error behavior.
• Property-test bucket determinism, approximate distribution, and stability when an existing rollout range expands.
• Measure local p50 and p99 by rule count and segment size; measure publish-to-apply lag separately.
• Disconnect streams, stop the control database, corrupt a patch, skip a version, expire credentials, and restart all instances together.
• Publish a bad rule to a small canary cohort, fire a health alarm, and verify that rollback creates and propagates a new audited version.
• Verify tenant isolation, production approval, client exposure filtering, audit completeness, and flag retirement.

Example of a strong answer

“I will scope the base system to server-side evaluation. There are 20,000 instances in three regions, but only 100 control writes per second, so the two traffic paths should not share an availability dependency. The management service stores drafts and immutable published versions in a relational database. Every publish validates types, rule references, and permissions, checks the expected prior version, writes an audit record and outbox entry, then builds a canonical project-and-environment artifact.

Regional relays distribute that artifact. Each SDK loads a last-known-good snapshot, receives updates over a stream, and polls to repair gaps. It installs a new immutable snapshot atomically and evaluates locally, keeping request latency below the 1-millisecond p99 target even when the control plane is down. Last-known-good is valid for at least 15 minutes; after the published stale boundary, the flag either keeps that value or uses the typed code fallback according to its risk policy. The local version, freshness, and evaluation reason are observable.

Evaluation order is off state, explicit targets, ordered rules, weighted outcome, then default. Percentage rollout hashes tenant, environment, flag, salt, and a stable targeting key into 100,000 buckets, so every instance makes the same choice. Changing hash inputs is a migration; multiple flags that require one cohort use a shared segment rather than coincidentally equal percentages.

The largest burst is fleet bootstrap: a 1 MB scoped snapshot times 20,000 instances is about 20 GB. I would serve snapshots regionally, send deltas, jitter reconnects, and bound retries. Evaluation telemetry is batched and sampled because recording 5 million synchronous events per second would threaten the product path. Finally, I would test cross-SDK conformance, propagation p99, restart storms, stale operation, corrupt and missing versions, canary rollback, break-glass audit, tenant isolation, and client exposure. The system is successful when request evaluation stays local and deterministic while every configuration change remains governed and recoverable.”

Common mistakes

• Calling a central service for every evaluation → product latency and availability now depend on the flag service → ship versioned rules to server SDKs and evaluate in process.
• Choosing users randomly on every request → one user moves between variants and experiment data is contaminated → hash a stable targeting key with documented flag-specific inputs.
• Treating last-known-good as permanently correct → a disconnected instance can serve an unsafe stale rule indefinitely → define staleness observability, reconnect and repair behavior, and an application safety fallback.
• Sending the server ruleset to a browser → users can inspect sensitive segments and credentials → filter client-safe flags or evaluate remotely, and enforce authorization on the server.
• Updating a shared rules object in place → concurrent requests can observe a partially applied configuration → validate a complete immutable snapshot and atomically swap references.
• Logging every evaluation synchronously → telemetry becomes the highest-volume dependency in the request path → batch, sample, aggregate, and shed noncritical events.
• Making emergency changes unaudited → the fastest recovery path becomes an untraceable production backdoor → use a priority publish lane with pre-authorized break-glass policy and immutable audit.
• Never retiring flags → obsolete branches and flag interactions multiply testing cost → assign an owner and remove the definition after the winning code path is permanent.

Follow-up questions and responses

Follow-up 1: How would you support browser and mobile SDKs?

Do not deliver the complete server ruleset. Mark which flags are client-exposable, authenticate the application rather than trusting it with a management credential, and return filtered values or context-specific evaluated results. Cache values for offline use with a visible version and freshness status. A client flag remains a user-experience hint; the server independently checks permissions, purchases, quotas, and other security decisions.

Follow-up 2: How do several flags keep exactly the same rollout cohort?

Identical percentages are insufficient when each flag key changes the hash input. Create a shared, versioned segment or an experiment assignment keyed by a common experiment ID, then reference it from every flag. Publish segment and dependent flag versions consistently, and test that cohort membership remains stable before increasing exposure.

Follow-up 3: What if a segment has ten million members?

Do not embed the full member list in every snapshot. Represent membership as a compact versioned artifact, shard it by subject hash, or precompute an attribute that the evaluator can consume. Bloom filters can reduce negative lookup traffic but cannot be the sole authorization mechanism because false positives exist. Measure memory, lookup latency, update amplification, and stale membership separately from ordinary rules.

Follow-up 4: Can an emergency off switch be instantaneous?

No distributed publish reaches disconnected processes instantly. Give emergency changes priority, keep regional streams warm, measure acknowledgments, and alert on lagging versions. For a destructive operation, combine the flag with a server-enforced guard such as disabling the write endpoint, revoking a capability, or blocking work at a gateway. The flag improves recovery speed but does not replace a hard safety boundary.

Follow-up 5: How would you migrate the hashing algorithm?

Store an algorithm version and salt with each published flag. Run the old and new evaluators in shadow mode and measure assignment movement. If movement is acceptable, publish a staged migration; if not, preserve existing subject assignments in a segment or migration table until the rollout completes. Never change the SDK hash implementation silently because different versions would disagree.

Follow-up 6: How do you prevent a flag dependency cycle?

Build the prerequisite graph during publish and reject a version when depth-first traversal finds a cycle. Bound maximum prerequisite depth and total evaluation work so an acyclic but pathological graph cannot violate latency. Include referenced flag versions in the artifact, test evaluation order across SDKs, and surface the dependency chain in diagnostic details.