System design interview: How would you design federated SPIFFE workload identity?

Problem and context

A company runs multi-tenant microservices in two clouds and one private cluster. Different teams operate each environment, and production must remain separate from staging. A payments service calls a reporting service, but long-lived API secrets and cloud-specific roles cannot be copied into images. Design federated workload identity across trust domains.

Workloads need short-lived, verifiable identities. Recipients must trust only approved external trust domains, SPIFFE IDs, audiences, and business actions. Cover SPIFFE ID naming, SPIRE Server and Agent, node and workload attestation, X.509-SVID or JWT-SVID, foreign-bundle retrieval, authorization mapping, rotation, revocation, caching, disaster recovery, and migration from static credentials.

Preserve these invariants:

1. The identity source proves which controlled process is running, not merely that a machine has a network address.
2. A trust-domain bundle does not automatically grant local authority to every external workload.
3. A recipient stops accepting an expired or revoked SVID within the declared bound.
4. Private keys are generated by the workload or a controlled key manager; the control plane does not distribute long-lived private keys.
5. Federation exchanges identity and trust material; the resource service still performs business authorization.

What the interviewer is evaluating

A strong answer draws the boundaries between a trust domain, SPIFFE ID, SVID, SPIRE Server, Agent, and Workload API. SPIFFE defines identity naming and verifiable documents; SPIRE implements node and workload attestation and SVID issuance. Identity is not permission: spiffe://prod.example/ns/payments/sa/worker still needs an allow policy at the reporting service.

The second signal is federation. SPIFFE Federation exchanges public trust material through a configured and TLS-authenticated bundle endpoint. A recipient must know which trust domain the URL represents; it must not download a root for an arbitrary domain from a request. The Workload API can return foreign bundles, and a verifier selects the bundle matching the SVID’s trust domain.

The third signal is the proof chain. Node attestation proves the Agent’s node, and workload attestation matches trusted kernel, kubelet, or container-runtime attributes against registration selectors. A namespace, label, or self-asserted client identity alone is not enough against a stolen service credential.

Finally, discuss operations: an expired federation bundle, root-key rotation, a disconnected Agent, stale caches, a control-plane single point of failure, clock skew across clouds, legacy services that cannot load SVIDs, and rollback without disabling verification.

Questions to clarify first

• Which trust domains really need trust? If payments only calls reports, do not build a fully meshed trust graph; define a one-way or minimal edge.
• X.509-SVID or JWT-SVID? mTLS and connection identity fit long-lived service links; cross-cloud HTTP or external OIDC resources may need a JWT audience. Their validation caches and leak windows differ.
• Who operates the foreign bundle endpoint? One company can share governance; a different company needs explicit public material, TLS identity, and reviewed domain mapping.
• What is the revocation target? For example, a 15-minute maximum SVID lifetime, 60-second bundle polling, and five-minute blocking of new connections after emergency revocation.
• How does a workload prove itself? Kubernetes ServiceAccount, a cloud instance document, TPM, and a controlled one-time join token have different assumptions.
• Can legacy services consume SPIFFE? If not, can a sidecar or gateway translate in a restricted way? Its identity and permissions need their own boundary.

A 30-second answer

“I would create independent trust domains for production, staging, and on-prem and assign stable SPIFFE IDs that contain no tenant secret. Each domain’s SPIRE Server owns registration entries. An Agent proves its node, then uses local process attributes for workload attestation; the workload obtains a short-lived X.509-SVID or JWT-SVID from the Workload API.

The payments domain would configure only the reporting domain’s foreign bundle endpoint and pin the domain, TLS identity, and allowed mapping. Reporting validates the signature, SVID expiry, audience, and peer bundle, then authorizes the full SPIFFE ID, environment, tenant context, and action. Bundle and SVID updates use streaming APIs, short TTLs, and version metrics. If the control plane is unavailable, bounded previously verified material may serve low-risk traffic, but high-risk new connections fail closed. Migration runs old credentials and SVIDs in parallel, canaries by service pool, and deletes secrets only after audit and revocation targets are proven.”

Step-by-step design

Step 1: Partition trust domains and name identities

A trust domain is an identity namespace and root-of-trust boundary. Production, staging, PCI environments, or separately governed organizations should use separate domains instead of one global root. An ID might be:

spiffe://prod.example/ns/payments/sa/worker
spiffe://reports.partner/ns/analytics/sa/reader

The path should express a stable business principal and governance scope, not a pod name, IP, or short-lived deployment hash. Version, tenant, and region can be selectors, policy inputs, or token claims. Encoding every deployment into the ID turns rotation and authorization maintenance into a fleet migration.

Record federation as explicit edges: prod.example may validate reports.partner’s bundle, but only for audience reports-api and action ReportRead. A recipient must not allow a call merely because both domains use SPIFFE.

Step 2: Establish Server, Agent, and registration entries

The SPIRE Server stores registration entries, signing material, and node authorization. An Agent runs on each workload node and exposes the local Workload API. The Server sends an Agent only the entries that the Agent is authorized to manage, limiting the blast radius of one node compromise.

An entry should bind a SPIFFE ID, parent SPIFFE ID, selector set, permitted SVID profile, and audience. Selectors come from a trusted orchestrator or node property: Kubernetes namespace, service account, image digest, or cloud instance identity. Do not rely only on user-editable labels or treat an arbitrary configuration string as attestation evidence.

Step 3: Design two-stage attestation

At node startup, the Agent proves its identity with a cloud instance document, Kubernetes ServiceAccount, TPM, or one-time join token. The Server independently validates the proof and issues the Agent identity. The workload then calls the Workload API through a Unix domain socket or restricted endpoint. The Agent uses process ID, kernel, kubelet, or container-runtime facts to obtain selectors, matches registration entries, and returns an SVID.

This explains why the Workload API need not use ordinary network client authentication: the Agent identifies a local caller out of band. Socket permissions, namespace isolation, and trust in the host kernel or kubelet belong in the threat model. If the Agent cannot identify the caller, it should return PermissionDenied instead of a default high-privilege identity.

Step 4: Choose an SVID profile and handle keys

X.509-SVID fits mTLS and connection-level service identity. JWT-SVID fits HTTP or external OIDC exchange where an audience must travel with the assertion. A JWT-SVID verifier must use the bundle for the subject trust domain; an arbitrary JWKS is not the same root of trust.

SVIDs should be short-lived and streamed through the Workload API. The workload or an Agent key manager generates the private key and keeps it in restricted memory or a file descriptor. The Server signs the corresponding public key but never writes a long-lived private key into an image, environment variable, or ordinary configuration. A workload with multiple identities can use a hint or explicit selection for internal and external audiences so the default cannot be misused.

Step 5: Establish cross-domain bundle federation safely

The federation control plane reviews the foreign domain, bundle endpoint, endpoint profile, TLS certificate, and allowed SPIFFE ID prefix. A client knows in advance which trust domain the URL represents. TLS authentication proves the transport endpoint; the returned bundle still needs domain, version, signature, and expiry checks.

Store a foreign bundle in a versioned trust store. During peer validation, select the bundle by the SVID’s trust domain; reject a missing match. Do not make a foreign bundle a local issuing root, and do not permanently replace configuration because of a temporary redirect. Polling, ETags, expiry, and the last successful version should be observable.

Step 6: Map identity to resource authorization

The reporting service’s PEP verifies the mTLS peer or JWT-SVID, then sends the full SPIFFE ID, trust domain, audience, tenant, and action to a PDP. An example rule is:

allow if trust_domain == "prod.example"
   and spiffe_id == "spiffe://prod.example/ns/payments/sa/worker"
   and audience == "reports-api"
   and action == "ReportRead"
   and tenant == resource.tenant

Network reachability, valid SVIDs, and a federation edge do not constitute business permission. The resource service still checks tenant ownership, row-level access, approvals, and rate limits. If a workload identity must reach cloud IAM or an external OIDC service, use a restricted broker bound to audience, scope, SVID TTL, and purpose; never hand one broad cloud role to every workload.

Step 7: Coordinate rotation, revocation, and cache consistency

SVID expiry is handled through Workload API streams. Clients atomically replace certificates and keys and let new connections use the new material first. Trust-bundle rotation uses bounded overlap between old and new roots; remove the old root only after verifiers have loaded the new version and old SVIDs and connections are drained. An emergency compromise does not use normal overlap; publish a revocation or deny version and shorten the acceptance window.

Cache bundle version, issuer, trust domain, SVID expiry, and policy version. A claimed five-minute revocation objective requires connection establishment, JWT validation, and sidecar caches to observe the update within five minutes. Long connections need bounded certificate-age draining; changing a control-plane database without handling existing connections is not revocation completion.

Step 8: Plan scale, failure, and migration

SPIRE Server resource use grows with registration entries, and one instance is a failure point. Shard by region or trust domain, use multiple Servers for HA, limit which entries each Agent receives, and monitor issuance latency, Agent count, entry count, Workload API requests, and bundle lag. Avoid a fully meshed federation graph; use controlled edges or brokers.

When the control plane is unavailable, an unexpired SVID can support bounded low-risk existing connections, but the system must not issue identities forever. An expired bundle, failed node proof, key-manager failure, or unknown domain mapping should fail closed for high-risk new connections. During static-secret migration, run both paths, canary by service pool, verify audit and revocation, and then delete the old secret. Rollback returns to a still-controlled path, never to disabled authentication.

Model high-quality answer

“I would separate production, staging, and partner environments into trust domains and use stable SPIFFE IDs for workloads. Each SPIRE Server owns registration entries and an issuing root. An Agent proves its node, then the local Workload API uses process and orchestrator attributes for workload attestation. Workloads obtain short-lived X.509-SVIDs for mTLS or JWT-SVIDs for a fixed audience; private keys stay with the workload, Agent, or controlled key manager.

Payments and reports have one reviewed federation edge. The client pins the endpoint and domain and validates TLS, bundle version, and expiry. Peer validation selects the foreign bundle by the SVID trust domain and rejects a missing match. Reporting validates SVID, issuer, audience, and expiry, then authorizes the full SPIFFE ID, tenant, and action. Federation provides verification material; it does not grant business permission.

SVIDs and bundles rotate through streams, short TTLs, version metrics, and bounded stale material; long connections drain by certificate age. Proof or control-plane failure never issues an unknown identity, and high-risk calls fail closed. Static-secret migration uses parallel paths, canaries, revocation drills, and audit reconciliation until every service can demonstrate identity, authorization, and recovery boundaries.”

Common mistakes

• Sharing one trust domain across every cluster. One root or configuration leak reaches every environment; split domains by governance and risk.
• Treating a SPIFFE ID as business authorization. Identity answers who; the resource service still checks audience, tenant, action, and ownership.
• Doing only node attestation. A malicious process on the node may impersonate a service; use workload selectors as well.
• Using a mutable pod name or IP as a long-lived principal. Rebuilds and movement break authorization; use a stable ID and policy selectors.
• Treating a federation URL as a trust root. Preconfigure the domain and validate TLS, bundle content, version, and expiry.
• Allowing every external principal from a foreign bundle. Federation supplies verification material; policy must restrict ID prefixes, audience, action, and tenant.
• Putting long-lived private keys in images or environment variables. Copies expose them; generate and rotate at the workload or key-manager boundary.
• Accepting stale bundles or SVIDs forever. That violates revocation; track versions, TTLs, and connection age and reject after the bound.
• Default-allowing during a control-plane outage. Unknown identity must not gain authority; use bounded old material or fail closed by risk.
• Migrating mTLS without audit and authorization. Encryption does not prove business permission; record peer ID, policy version, tenant, and decision.

Follow-up questions and answers

If two trust domains need bidirectional communication, must both import each other’s bundle?

Not necessarily. If payments calls reports only, make a one-way edge: reports trusts the payments foreign bundle, while payments need not trust reports. Add the reverse edge only when reports also initiates calls, with separate audiences and policies. Mutual trust is not full permission.

Why not use the cloud provider’s workload identity directly?

Cloud identity is useful inside one cloud control plane, but multi-cloud, private, and partner environments have different issuers, SDKs, and policies. SPIFFE supplies portable naming, SVIDs, and bundle exchange. External cloud IAM can still use a restricted OIDC federation broker; SPIFFE does not replace business authorization.

Is an unauthenticated Workload API safe?

It relies on out-of-band process identification rather than treating a socket as an open network API. Restrict socket or endpoint permissions, isolate host and namespace, and match registration entries against kernel, kubelet, or container-runtime facts. An unidentifiable caller receives denial, never a default powerful SVID.

What if a foreign bundle endpoint is temporarily unavailable?

Keep the last trusted bundle with a version and expiry. It may validate existing low-risk connections while fresh; beyond the maximum stale age or for high-risk new connections, fail closed. Monitor bundle lag, last successful version, and denials instead of accepting an old root indefinitely.

How do long-lived connections handle SVID rotation and revocation?

The Workload API streams new material; clients atomically update and use the new certificate for new connections. Drain by maximum certificate age or revocation window, and recheck policy before high-risk commits. Replacing a file without handling established connections does not prove the old identity is gone.

How do you prove selectors cannot be forged?

Selectors come from trusted node or orchestrator APIs and are verified by Server or Agent. User-editable labels are only hints. Combine image digest, service account, namespace, process properties, and node proof, and audit and approve registration changes.

How do you roll back a static-secret migration?

Accept the old secret and SVID in parallel, enable SVID by service pool, and record success, authorization, and revocation for both paths. On failure stop new issuance and return to the still-controlled old secret, then continue the canary after fixing the issue. Define retirement evidence and a deletion gate; never disable authentication as rollback.