System Design Interview: Design a Multi-Tenant Secrets Management Service

Prompt and Applicable Context

Design a secrets management service for a multi-tenant platform. A secret may be a database credential, API token, private key, or certificate. The system stores 20 million active secret names with an average of three retained versions. Values average 2 KiB and cannot exceed 64 KiB. Peak traffic is 100,000 reads and 2,000 version writes per second. In the secret's home region, successful reads should complete below 50 milliseconds at p99.

Each region spans three availability zones. A write acknowledged by the home region must survive the loss of one zone. The disaster-recovery target for losing the whole region is an RTO of 15 minutes and an RPO of one minute. Those volumes and objectives are interview assumptions, not published limits of a commercial secrets product. If the business requires zero data loss after a regional failure, the candidate must explicitly pay for synchronous cross-region consensus or another equivalent durability boundary.

The threat model includes a stolen database snapshot, compromised backups, cross-tenant authorization mistakes, an operator with access to ordinary storage, accidental logging, and theft of one data-plane host. It does not claim to stop an already authorized workload from using a value it legitimately retrieved. The design must reduce that workload's privileges and credential lifetime, preserve attribution, and limit the blast radius of compromise.

The scope includes workload authentication, policy evaluation, static and dynamic secrets, encryption, immutable versions, staged rotation, revocation, leases, audit, replication, and recovery. Building the organization's identity provider, the database that consumes a generated credential, and a general-purpose cryptographic key-management service are external dependencies. Public security-engineering interview material available in 2026 includes the specific prompt to design a secrets management service and lists a microservices secrets manager as a security system-design exercise. This supports the topic's current representativeness without proving frequency at any particular company.

What the Interviewer Evaluates

The first signal is a precise protection boundary. Saying “encrypt the database” leaves the decryption key beside the ciphertext and does not explain which compromise is contained. A strong answer separates a hardware- or KMS-protected root, tenant-scoped key-encryption keys, per-version data-encryption keys, and ordinary ciphertext storage. It also identifies where plaintext can briefly exist.

The second signal is authorization at every request. Tenant isolation cannot depend on a path prefix supplied by the caller. The service derives tenant, workload, environment, and roles from authenticated identity, evaluates a versioned policy for the exact action and resource, and includes those facts in an audit event. Authentication, authorization, encryption, and auditing are distinct controls.

The third signal is lifecycle reasoning. A new value, a cryptographic wrapping-key change, and a credential change in an external database are three different operations. Rotation across an external system is not one atomic database transaction. It needs durable stages, overlap rules, idempotency, testing, compensation, and reconciliation.

The fourth signal is honest revocation semantics. Denying future reads does not erase a static password already copied into a process. Effective emergency revocation also disables or rotates the credential at the system that accepts it. A dynamic credential can offer stronger expiry and revocation because the target system participates in the lease.

Finally, the candidate should balance latency, security, and availability. Calling an HSM for every one of 100,000 reads per second is usually the wrong bottleneck. Caching plaintext in a shared distributed cache is the wrong cure. The answer should use a key hierarchy, narrowly scoped protected-memory caches, explicit stale-policy limits, and tested failure modes.

Questions to Clarify Before Answering

• Which secret types are in scope? Opaque static values need version storage. Database users and certificates may

also support dynamic issuance and target-side revocation.

• Who reads secrets? Prefer workload identities and short-lived sessions. Human viewing, if allowed at all,

requires stronger approval, step-up authentication, and separate audit policy.

• What does revocation promise? The service can immediately reject new reads. Invalidating a copied static value

requires the downstream system to rotate or disable it.

• What consistency does current require? This design makes promotion and revocation linearizable per secret in

its home region. A caller can also request an immutable version explicitly.

• Can a region fail with zero data loss? The stated DR RPO is one minute. RPO zero requires synchronous

cross-region acknowledgement and changes the latency and availability trade-off.

• Can applications use a local cache during an outage? Only an explicit policy may permit an encrypted,

time-bounded agent cache for selected static secrets. It weakens immediate-revocation guarantees.

• How large are values and versions? The prompt caps values at 64 KiB and assumes three retained versions on

average. Large documents belong in a different encrypted object-storage design.

• What must the audit record contain? Principal, workload, tenant, resource identifier, action, policy version,

decision, request ID, region, and time; never the plaintext value.

• Who can administer the root of trust? Separate duties, quorum-controlled break-glass access, and recovery drills

are required. A universal everyday administrator defeats tenant isolation.

30-Second Answer Framework

“I would separate a policy and authoring control plane from a low-latency read data plane. A workload authenticates with a short-lived platform identity; the service derives its tenant and evaluates a versioned least-privilege policy for the exact secret and action. Each immutable secret version is encrypted with a random data key using authenticated encryption. That data key is wrapped by a tenant key, while an HSM or KMS protects the root that unwraps tenant keys; ordinary storage sees only ciphertext and wrapped keys. A regional transactional quorum atomically appends a version and moves the current pointer with compare-and-set. Rotation is a durable create, install, test, promote, overlap, and retire workflow, because the external target cannot join one transaction. Dynamic credentials carry leases that the issuer renews or revokes. I would cache ciphertext and briefly cache unwrapped keys only inside hardened service memory, never in a shared plaintext cache. Audit goes to a separately protected append-only pipeline. I would verify cross-tenant denial, concurrent promotion, key loss, KMS outage, stuck revocation, log leakage, and full regional restore.”

Step-by-Step Deep Dive

Step 1: Turn the prompt into invariants and a threat boundary

Start with five invariants:

1. A request can address only a tenant derived from authenticated identity.
2. Ordinary databases, queues, logs, traces, and backups never receive plaintext values or plaintext keys.
3. A published version is immutable; promotion only changes a version pointer.
4. Every allow and deny decision emits an audit event without the value.
5. Acknowledged regional writes survive one availability-zone failure.

These invariants still leave residual risk. An authorized workload receives plaintext and can be compromised. A data-plane process briefly holds unwrapped key material and plaintext. A malicious root-of-trust administrator may exceed normal policy. Reduce those risks with short-lived identity, narrow policies, process isolation, disabled core dumps, protected memory, independent audit, separation of duties, and small key scopes. Do not claim that encryption at rest solves runtime compromise.

Step 2: Separate the control plane from the read data plane

The control plane manages tenants, policies, secret metadata, new versions, rotation jobs, approvals, and emergency actions. Its write rate is lower, and it favors strict validation and auditable workflows. The data plane authenticates workloads, evaluates policy, decrypts an allowed version, and returns it over mutually authenticated TLS. Its hot path must remain small.

Each tenant has a home region for authoritative secret and policy writes. A three-zone transactional quorum in that region commits metadata, ciphertext, version pointers, leases, idempotency records, and an audit outbox. Read nodes may scale horizontally, but current reads go through the home region's strongly consistent store or a replica with a proved read barrier. A stale replica must not resurrect a revoked version.

Identity comes from the platform identity provider through mechanisms such as workload certificates or signed service-account tokens. The secrets service validates issuer, audience, expiry, workload identity, and channel binding, then maps the trusted identity to tenant and role. The client cannot override that tenant with a header.

Step 3: Define a small API and immutable data model

One possible API is:

POST /v1/secrets/{path}/versions
  { value, idempotencyKey, expectedCurrentVersion? }
  -> { secretId, version, status: "PENDING" }

POST /v1/secrets/{path}/versions/{version}/promote
  { expectedCurrentVersion, idempotencyKey }
  -> { currentVersion }

GET /v1/secrets/{path}?version=current
  -> { value, version, expiresAt? }

POST /v1/dynamic/{role}/credentials
  { requestedTtl, idempotencyKey }
  -> { value, leaseId, expiresAt, renewable }

POST /v1/leases/{leaseId}/renew
POST /v1/leases/{leaseId}/revoke

The path is a resource name, not proof of authorization. The server normalizes it once, rejects ambiguous encodings, derives the tenant from identity, and checks action-specific policy.

Secret(secret_id, tenant_id, canonical_path, state,
       current_version, policy_ref, created_at)

SecretVersion(secret_id, version, status, ciphertext, nonce, auth_tag,
              wrapped_dek, tenant_kek_version, content_fingerprint_hmac,
              created_by, created_at)

Policy(policy_id, tenant_id, version, document, state, created_at)

Lease(lease_id, tenant_id, secret_id, principal_id, target_ref,
      status, expires_at, renewable_until, last_error)

Idempotency(tenant_id, principal_id, operation, key,
            request_fingerprint_hmac, result_ref, expires_at)

(tenantid, canonicalpath) and (secret_id, version) are unique. Version rows never change their encrypted payload. Promotion uses a conditional update on current_version; two administrators racing to promote from version 7 cannot silently overwrite each other. The audit outbox is committed with each state change and later exported to separately controlled storage. The fingerprints are keyed HMACs made with a separate tenant-scoped key, not raw hashes of low-entropy credentials that a stolen snapshot could test offline.

Step 4: Build an envelope-encryption hierarchy

Generate a random data-encryption key for every version and encrypt the value with an authenticated-encryption mode. Bind tenantid, secretid, version, and algorithm identifier as additional authenticated data, so ciphertext cannot be moved to another tenant or version without detection. Store the ciphertext, nonce, authentication tag, and wrapped data key together.

Each tenant has one or more versioned key-encryption keys. A root protected by an HSM or cloud KMS unwraps those tenant keys only for authorized data-plane nodes. The tenant key wraps each version's data key. An ordinary storage snapshot therefore lacks the material required to decrypt values. A stolen data-plane host exposes only the tenant keys and data keys present in that process, so placement and cache scope must limit how many tenants one node can serve.

HSM/KMS root
  -> unwrap tenant KEK version
      -> unwrap per-secret-version DEK
          -> AEAD-decrypt secret value with bound tenant/version metadata

Calling the HSM for every read would make the root service a latency and throughput bottleneck. Data-plane nodes may cache unwrapped tenant keys or data keys for a short, bounded period in hardened memory, partitioned by tenant risk. They clear caches on policy or key events and on process shutdown. Shared Redis, disk swap, crash dumps, logs, and traces never receive plaintext or unwrapped keys.

Changing the tenant wrapping key can rewrap data keys without decrypting every stored value. Changing a database password creates a new secret value and must coordinate with the database. These operations have different blast radii and must not share one vague “rotate” button.

Step 5: Make authorization and caching revocation-aware

Policies name actions such as create-version, promote, read, issue-dynamic, renew, revoke, and administer-policy. They can constrain tenant, project, environment, path, workload, network zone, time, and maximum TTL. Deny wins over allow. Human plaintext access is disabled by default; an approved exception records the ticket, approvers, reason, and short-lived grant.

Policy evaluation runs locally for latency, but cached policy has a version and a maximum age. A policy or principal revocation commits first, publishes invalidation, and advances a tenant authorization epoch. A read node must prove it has at least the required epoch before serving. If invalidation is unavailable and the maximum age expires, sensitive reads fail closed instead of trusting policy forever.

Ciphertext and immutable metadata are safe to cache broadly. Plaintext is not. A workload-side agent may hold an allowed value in its own memory until a declared TTL, reducing repeated reads. If the product permits an encrypted disk cache for selected availability-critical secrets, its device-bound key, expiry, and weaker revocation promise must be explicit. Returning an old secret merely because the control plane is down is not a universal fallback.

Step 6: Treat credential rotation as a recoverable workflow

Static credential rotation across a target database follows durable stages:

CREATED_PENDING
  -> INSTALLED_AT_TARGET
  -> VERIFIED
  -> PROMOTED_CURRENT
  -> OLD_VERSION_IN_OVERLAP
  -> OLD_VERSION_REVOKED
  -> COMPLETE

Each transition has an idempotent connector operation and recorded evidence. Create the new credential, install it in the target, test it through the intended path, promote the new version, allow a bounded overlap when the consumer requires it, then disable the old credential at the target. A reconciliation worker compares workflow state with the target and resumes after crashes. If installation succeeds but the response is lost, lookup must discover that fact rather than create another credential.

Emergency rotation may skip overlap because the old value is suspected compromised. That choice can create an application outage and should require an incident-scoped authorization path. Denying reads to the old stored version alone is insufficient while the target system still accepts it.

For dynamic secrets, a connector creates a unique short-lived credential for a workload and records a lease. Renewal returns the actual new expiry, which may be shorter than requested. Expiry or manual revocation tells the target to invalidate the credential. If the target is unreachable, the lease enters REVOCATION_PENDING; retries, alerts, and an operator runbook remain active. The service must not mark it revoked merely because a queue message was sent.

Step 7: Budget capacity and isolate hot tenants

At three retained versions per 20 million names and 2 KiB per value, raw encrypted value bytes are approximately:

20,000,000 × 3 × 2 KiB = 122,880,000,000 bytes ≈ 114 GiB

If encrypted metadata averages another 1 KiB per version, that adds roughly 57 GiB. Three regional replicas put the starting storage estimate near 513 GiB before indexes, audit, leases, idempotency, backups, and growth. This arithmetic comes from the prompt and is a sizing baseline, not a product benchmark.

At 100,000 reads per second and an average 2 KiB returned value, plaintext egress alone approaches 195 MiB per second, before TLS and response overhead. Shard tenants by a stable tenant identifier, then isolate unusually hot or regulated tenants onto dedicated cells. Apply per-tenant concurrency and rate budgets so one compromised workload cannot consume all decryption workers or audit capacity.

Read latency should be decomposed into identity validation, policy evaluation, metadata lookup, key unwrap/cache, decryption, audit enqueue, and network time. HSM cache misses and audit backpressure deserve independent budgets. Never drop audit silently to protect p99; either use a durable local/outbox boundary or reject sensitive requests when the required audit record cannot be preserved.

Step 8: Design regional recovery and root recovery separately

Replicate encrypted state and audit outbox records asynchronously to a warm disaster-recovery region, with measured replication lag. Promotion requires a fencing epoch so the old home region cannot resume writes as a second primary. At the stated one-minute RPO, callers must understand that the newest acknowledged versions may need recreation after whole-region loss. If that is unacceptable, require cross-region quorum before acknowledging a write.

The DR region also needs independent access to the root-of-trust service, working workload identity, policy state, and audit export. Copying only ciphertext is not a recovery plan. Test encrypted snapshots and point-in-time restore, but protect configuration, boot credentials, and auto-unseal material separately because they can be sensitive even when the data snapshot is encrypted.

Root recovery uses quorum-controlled break-glass procedures, separate administrators, immutable evidence, and regular drills. Losing a tenant key may destroy access to that tenant's values; leaking it may expose every value wrapped by that key. Key backup, rotation, placement, and destruction therefore need explicit lifecycle tests.

Step 9: Verify security properties and failure behavior

Build tests around invariants, not only happy-path latency:

• Generate cross-tenant path, encoded-path, wildcard-policy, and stale-identity attempts and prove every one is denied.
• Race version creation, promotion, policy revocation, and reads; confirm no stale node serves a newly denied version.
• Scan databases, queues, logs, traces, crash dumps, and backups for canary plaintext and unwrapped-key material.
• Inject KMS latency, key-cache eviction, storage quorum loss, audit backpressure, and full control-plane outage.
• Interrupt every rotation stage before and after the target-side action; prove reconciliation converges without

duplicate credentials or false completion.

• Make a dynamic-secret target unavailable during expiry; prove the lease stays visibly pending until target-side

invalidation succeeds.

• Restore the DR region from encrypted replication and snapshots; verify fencing, identity, policy, keys, audit, RTO,

and measured RPO.

• Load-test a highly skewed tenant distribution at 100,000 reads and 2,000 writes per second while measuring p99,

denial accuracy, HSM traffic, cache scope, and audit completeness.

High-Quality Sample Answer

“I would first state that the service protects against storage, backup, cross-tenant, logging, and limited host compromise, while an authorized workload remains able to misuse a value it receives. Every request starts with a short-lived workload identity. The service derives tenant and workload from that identity, then evaluates a versioned policy for the exact resource and action; a caller-supplied path cannot select another tenant.

I would use immutable versions. A create request writes a pending version, and promotion conditionally moves the secret's current pointer from the expected prior version. A three-zone quorum in the home region commits ciphertext, metadata, idempotency, and an audit outbox together. current reads use a read barrier so a stale replica cannot serve a revoked version.

For encryption, every version gets a random data key and authenticated encryption bound to tenant, secret, and version. A tenant key wraps that data key, and an HSM- or KMS-protected root unwraps tenant keys only on authorized data-plane nodes. Ordinary storage and backups contain ciphertext and wrapped keys. Read nodes may briefly cache unwrapped keys in hardened, tenant-partitioned memory; plaintext never enters shared caches, logs, traces, swap, or core dumps.

Credential rotation is a state machine: create, install at the target, verify, promote, overlap, revoke the old target credential, and complete. Every connector call is idempotent and reconciled, because an external database cannot join the metadata transaction. Emergency compromise may remove overlap. Dynamic credentials are issued per workload with a lease; expiry is complete only after the target confirms revocation, otherwise the system keeps a visible pending state and alerts.

The control plane owns policy, versions, approvals, and workflows; horizontally scaled data-plane cells serve reads. Ciphertext can be cached broadly, policy has a bounded versioned cache, and unwrapped-key caching is short and local. For a region loss, encrypted state replicates to a warm region with a one-minute RPO and a fencing epoch prevents split brain. If the requirement changes to RPO zero, I would synchronously commit across regions and accept higher latency or lower write availability.

Finally, I would test cross-tenant and encoded-path attacks, concurrent promotion and revocation, canary leakage into all observability and backup surfaces, KMS and audit outages, crashes at every rotation step, stuck lease revocation, and a complete DR restore. The design passes only when security decisions remain correct under those faults and the 100,000-read peak still meets its measured latency budget.”

Common Mistakes

• Using one database encryption key beside the database → a snapshot compromise can include both ciphertext and

its decryption path → separate the HSM/KMS root, tenant wrapping keys, per-version data keys, and ordinary storage.

• Trusting the tenant or path in the request → an object-name change becomes cross-tenant access → **derive tenant

from verified identity, canonicalize once, and authorize the exact action and resource.**

• Logging request or response bodies for debugging → the observability stack becomes a plaintext secret store →

use identifiers, decisions, versions, and canary leak tests without values.

• Calling the HSM on every read → root-of-trust latency and quota become the platform bottleneck → **use envelope

encryption and bounded protected-memory key caches with explicit blast radius.**

• Putting decrypted values in Redis → a latency optimization creates a broad plaintext compromise boundary →

cache ciphertext broadly and keep any plaintext only in the authorized consumer's bounded memory.

• Treating rotation as one update → target installation can succeed while metadata fails or vice versa → **use a

durable staged workflow, idempotent connector operations, overlap rules, and reconciliation.**

• Marking a lease revoked after publishing a message → the downstream credential may still work → **retain a

pending state until target-side invalidation is confirmed and alert on stuck revocation.**

• Promising immediate revocation of copied static values → future reads can be denied, but existing copies remain

usable → rotate or disable the credential at the accepting system and prefer short-lived dynamic credentials.

• Failing open on indefinitely stale policy → a removed principal can keep reading sensitive values → **version

policy caches, invalidate them, enforce a maximum age, and fail closed after the bound.**

• Copying ciphertext to a second region and calling it DR → identity, root keys, policies, fencing, and audit may

still be unavailable → test the complete recovery dependency graph and measure actual RTO and RPO.

Follow-Up Questions and Answers

Follow-up 1: How is a secrets manager different from a KMS?

A secrets manager stores and versions opaque credentials, authorizes retrieval, coordinates rotation, issues leases, and audits access. A KMS manages cryptographic keys and performs operations such as wrapping or signing. This design uses a KMS or HSM as its root of trust; it does not expose that root as an ordinary retrievable secret.

Follow-up 2: Can you cache a secret in the application?

Yes, only under an explicit contract. An agent can keep a permitted value in process memory until a bounded TTL and refresh before expiry. That reduces availability dependence and read volume, but extends the time during which policy revocation alone cannot remove an existing copy. High-risk or dynamic credentials should use shorter leases and target-side revocation rather than an unbounded cache.

Follow-up 3: What happens when the KMS or HSM is unavailable?

Nodes may continue only with already unwrapped keys inside their approved cache lifetime. New tenants, cache misses, and root-key operations fail closed. Track cache coverage, KMS error rate, and the remaining safe window; shed load before all nodes simultaneously expire. Extending key lifetime during an incident is a security decision requiring predefined policy, not an automatic retry loop.

Follow-up 4: How do you rotate the tenant key without decrypting every secret?

Unwrap each per-version data key with the old tenant key and wrap it with the new tenant key, preferably within the trusted key service so plaintext data keys do not leave that boundary. Store both tenant-key version and wrapped data key, checkpoint the migration, and retain the old tenant key until every reference and backup policy permits its retirement. The encrypted secret value itself does not change.

Follow-up 5: How do you prevent a privileged operator from reading every tenant's secrets?

Separate policy administration, key administration, infrastructure operation, and audit review. Disable routine human plaintext retrieval, require quorum-approved short-lived grants for exceptions, place regulated tenants in separate cells and key scopes, and send evidence to an independently controlled audit system. No single daily-use role should both grant itself access and decrypt values without external evidence.

Follow-up 6: Should reads remain available when the home region is down?

Only if the DR region has a fenced authoritative epoch, sufficiently current policy and encrypted data, access to its root of trust, and a clearly accepted RPO. Serving from an arbitrary stale replica can resurrect revoked access. At the stated one-minute RPO, promote the warm region through the tested failover process; RPO zero needs synchronous cross-region commits before the incident.

Follow-up 7: Why not store secrets only as environment variables?

An environment variable is a delivery mechanism, not lifecycle management. It may be exposed through process inspection, crash dumps, diagnostics, child processes, or deployment configuration; it also lacks central versions, target-side rotation, leases, and per-read attribution. If a deployment agent injects an environment variable, keep its scope and lifetime small and retain the secrets manager as the source of policy and lifecycle truth.