System design interview: How would you design a multi-tenant secret-rotation control plane?

Prompt and suitable context

This is a security system-design question. The focus is how a control plane coordinates secret versions, tenant policy, consumer confirmation, and rollout windows while the data plane only reads an authorized current version. AWS demonstrates alternating users to reduce database cutover disruption, while Google Secret Manager uses immutable versions and aliases for binding and gradual rollout. Abstract those patterns into an auditable multi-tenant workflow rather than copying one cloud API.

What the interviewer evaluates

• Whether you model tenants, secrets, versions, consumers, and policy while preventing cross-tenant access.
• Whether you design overlapping versions, health verification, idempotent retries, and reversible promotion.
• Whether you distinguish scheduled rotation, emergency revocation after exposure, and destruction.
• Whether you can explain tradeoffs among control-plane availability, caching, audit, and operations.

Clarifications to ask first

Confirm secret types, consumer forms, tenant count, rotation frequency, maximum overlap, and tolerated interruption. Ask whether the platform generates values, whether third-party APIs must be updated, whether consumers hot-reload, and whether regional isolation, retention, human approval, or emergency revocation is required. State assumptions when scale is missing, and separate secret values from metadata storage.

A 30-second answer framework

I would cover policy registration, new-version generation, staged distribution, verified promotion, then revocation and audit. Each tenant gets an isolated secret namespace and authorization policy; values exist only in a dedicated KMS or secret manager. The control plane creates a pending version, asks consumers to load it and report health, then moves an alias to current. The old version remains during a controlled overlap window. Failures pause the job and restore the alias; exposure uses a faster emergency path. Every step has idempotency, approval, and audit records.

Step-by-step deep answer

1. Model tenant, secret, and policy boundaries

Metadata stores tenant, purpose, algorithm, version, consumer, schedule, region, owner, and state. KMS or a dedicated secret manager holds values; the application database stores only references and hashes. Authorization is bounded by tenant, purpose, and consumer identity, and operators cannot read plaintext by default. Policy declares rotation period, minimum overlap, verification probes, retry limit, and approval requirements.

2. Generate and isolate a pending version

The scheduler creates an idempotent rotation job, generates a pending version, and records its reason, parent version, and expiry. Generators and distributors are separate, and logs never contain values. Third-party updates use least privilege and short-lived credentials. AWS's alternating-user strategy illustrates preparing a backup credential and validating it before cutover, but each dependency needs its own adapter.

3. Distribute gradually and verify consumers

Consumers obtain version references through short-lived authorization, never through logs or environment variables containing plaintext. Start with a small tenant or instance cohort, then expand. Verification checks authentication, business probes, error rate, and latency. Google warns that directly using a latest alias in production can immediately spread a bad value, so the control plane should support pinned aliases, partitioned rollout, and explicit promotion.

4. Switch atomically, overlap, and roll back

Move the current alias from the old to the verified version and keep previous for rollback. Promotion must be conditional so concurrent rotations cannot overwrite each other. The overlap window lets old credentials work briefly, but it needs an explicit expiry and revocation action. Consumer failures, probe regressions, or partial third-party success pause the job, restore the alias, and escalate.

5. Revoke urgently, recover, and observe

After exposure or suspected abuse, skip the normal schedule: freeze the old version, generate a replacement, update dependencies, and broaden verification. Deleting a version in the manager is insufficient if an external service still accepts the old credential. Track rotation success, verification time, overlap duration, rollbacks, expired versions, cross-tenant denials, plaintext-access alerts, and exposure response time. Backups retain encrypted material and recovery metadata only; drills verify tenant isolation and alias consistency.

High-quality sample answer

I would clarify secret types, consumer hot-reload capability, tenant scale, overlap window, and emergency-revocation objectives. Tenant, purpose, and consumer form an isolated authorization domain; values stay in KMS or a secret manager and the application database stores references. A scheduler creates an idempotent pending version. Distributors let a small cohort load it and report authentication, business-probe, and latency results. After approval, a conditional update moves current, while previous remains for a bounded rollback window. Failures or partial success pause and restore the alias. Exposure uses immediate revocation. Generation, access, promotion, rollback, and human approvals go to tamper-resistant audit logs, with metrics segmented by tenant and secret purpose.

Common mistakes

• Storing all values in one database or log and ignoring tenant and purpose isolation.
• Overwriting the old value immediately instead of modeling pending, current, and previous.
• Testing only a secret-manager write rather than real consumers and business probes.
• Relying on latest propagation with no partition, pause, or rollback.
• Treating scheduled rotation and emergency revocation as the same slow workflow.
• Deleting the platform record without revoking the old credential in external services or caches.

Follow-up questions and responses

What if a consumer cannot hot-reload?

Bind promotion to a reversible restart or deployment, validate a small instance cohort, then expand. Record instance version and confirmation; “notified” is not “effective.”

What if two rotation jobs run concurrently?

Use a tenant-and-secret lease or conditional generation number; only the current generation may promote an alias. Idempotency keys absorb duplicate requests, while expired jobs pause for human confirmation.

What if a third-party update succeeds but local persistence fails?

Treat the external update as retryable but not blindly replayable. Save request proof and an idempotency marker, query external state before compensating, and freeze promotion when state is uncertain so an operator can reconcile it.

How do you choose the overlap window?

Measure consumer cache time, deployment propagation, and maximum request duration rather than guessing. Longer overlap increases exposure; shorter overlap increases outage risk, so classify by secret purpose.

How does the data plane operate during control-plane failure?

Cache the last approved version reference and expiry policy. During an outage, reject new promotion but continue the last safe configuration. After recovery, reconcile missing state using generation numbers and audit events.