System design interview: How would you build consistent, rollback-safe feature flag evaluation?

Prompt and context

The platform serves boolean, string, and structured feature flags to multi-tenant services. Rules may depend on tenant, user, region, and application version. After a release, some instances update immediately while others lag for minutes, and sensitive context must not enter logs. Design the control plane, data plane, evaluation context, cache, release, failure, and audit loop.

What the interviewer is testing

• Separating control-plane publishing from data-plane evaluation and defining version and consistency goals.
• Correctly merging, overriding, propagating, and protecting evaluation context.
• Designing cache invalidation, offline snapshots, defaults, and rollback without a live RPC per request.
• Making exposure, error, change, and rule-hit telemetry auditable without leaking personal data.

Questions to clarify first

1. Must evaluation be strongly consistent, or is stale data acceptable? What is the maximum staleness?
2. What is the rule precedence for tenant, user, device, region, and version?
3. How long must services run when the control plane is unavailable, and who approves defaults?
4. Which context fields are personal data, and which may enter exposure logs?
5. Are multi-language SDKs and local offline evaluation required, or is a remote evaluator acceptable?

A 30-second answer framework

The control plane would validate rules, publish immutable versions, approve, and roll back. Data-plane SDKs would evaluate versioned local snapshots for low latency and offline operation. Context would merge global, transaction, and invocation scopes with explicit override order and minimal fields. Instances would receive updates through streaming plus polling, with TTL and a version monotonicity check. Failures would use a typed default or last-known-good value and expose staleness; critical flags could fail closed. Audit events would contain versions and anonymous keys, never raw attributes.

Step-by-step deep answer

Step 1: Define objects and a release state machine

A flag contains type, default, rules, variants, environment, version, and activation time. Publishing moves through draft, validation, approval, canary, complete, or rollback; each change creates an immutable version. Compilation errors, type mismatches, or missing defaults block publication instead of becoming runtime failures for every service.

Step 2: Design evaluation context

Represent application, host, region, tenant, and user attributes as structured context. Merge global, transaction, and invocation context according to the specification with an explicit duplicate-key order. SDKs should not implicitly read arbitrary thread state. Apply field allowlists and redaction before context enters an SDK, transport, or log.

Step 3: Choose local or remote evaluation

Low latency and short control-plane outages favor local evaluation from a distributed rule snapshot. Remote evaluation centralizes complex logic but adds network and availability dependency to every call. A hybrid can keep simple evaluation in the SDK and use a provider for complex rules, while returning typed values, reasons, versions, and metadata.

Step 4: Establish cache and consistency goals

Key snapshot caches by environment, flag set, and version and protect them with checksums and expiry. Polling compensates for lost update notifications. An instance starts from its last trusted snapshot and catches up asynchronously. Define “never move backward,” maximum staleness seconds, and rollback propagation time as SLOs.

Step 5: Handle failures and safe defaults

On evaluation errors, return a type-compatible default or last-known-good value with reason, error code, and source. Payment, authorization, and deletion flags should not silently take a dangerous default; they may block or use an approved fail-closed policy. Guard against unknown flags, unsafe type conversion, and rule timeouts spreading through callers.

Step 6: Design audit and exposure telemetry

The control plane records who published, approved, canaried, or rolled back and when. The data plane records flag key, version, result, rule branch, SDK version, and an anonymous subject hash, never raw email, IP, or complete context. Isolate sampling, retention, and access by tenant and join outcomes to metrics to detect canary impact.

Step 7: Verify rollback and migration

Replay fixed contexts against each rule version and compare results across language SDKs. Exercise lost notifications, corrupt caches, control-plane outages, clock skew, partial rollback, and provider upgrades. Rollback creates a new version rather than rewriting history; compare instance-version distributions and business metrics after completion.

High-quality sample answer

The control plane owns type checks, compilation, approval, versions, and canaries; SDKs evaluate checksummed local snapshots for low latency and offline operation. Context merges global, transaction, and invocation scopes with a fixed override order and an allowlist for sensitive fields. Instances use notification plus polling, with a maximum staleness and monotonic-version rule. Errors return typed defaults or last-known-good values, with fail-closed options for critical flags. Audits preserve publish and rollback chains, exposure logs contain only versions, outcomes, and anonymous keys, and rollback is a new version verified by SDK replay and failure drills.

Common mistakes

• Calling the control plane synchronously for every evaluation and coupling business availability to it.
• Caching values without versions, so rollback and staleness cannot be explained.
• Leaving context merge order to language-SDK behavior and getting different results across services.
• Logging complete user attributes or emails.
• Overwriting old configuration during rollback and losing audit and replay history.

Follow-up questions and responses

Follow-up 1: Can evaluation continue if the configuration service is down?

Use the last trusted snapshot and expose its version, age, and source. Beyond the maximum staleness threshold, choose a risk-based default, block, or human action instead of running silently forever.

Follow-up 2: How do you keep language SDKs consistent?

Specify normalization, types, context merge, and error reasons and provide versioned cross-language input/output vectors. A provider can execute complex rules centrally while SDKs share the same protocol and lifecycle.

Follow-up 3: How do you keep a canary percentage stable?

Bucket a stable anonymous subject key with an explicit hash algorithm. A fixed rule version gives the same subject the same variant on every instance. Changing the algorithm or salt creates a new version with documented migration impact.

Follow-up 4: Why use hooks?

Hooks can add context before evaluation, validate a value after it, or emit telemetry, but they need explicit ordering, timeouts, and error handling. They must not silently change flag types or bypass audit.

Follow-up 5: How do you safely remove a flag?

Find code references, evaluation traffic, and default branches, publish a fixed-value version, observe it, then remove rules and SDK metadata. Keep historical versions and migration records so old instances do not receive an unknown type.