System design interview: Build a centralized authorization decision service

Prompt and context

Several business services need to answer the same question: who can perform which action on which resource? Design a shared decision service that accepts a subject, action, resource, and context, returns ALLOW or DENY, and supports users, service accounts, hierarchical resources, tenant boundaries, and policy changes.

Use these interview assumptions: 100,000 authorization checks per second at baseline, a peak of 3x average, a 20 ms p99 latency target, policy writes far less frequent than reads, and callers that must distinguish an explicit denial from temporary authorization-service unavailability. These are problem assumptions, not industry benchmarks. Payment capture, key issuance, and business-database writes remain outside this service.

What the interviewer is testing

Strong answers turn authorization into a four-tuple and a consistency contract instead of saying “add an RBAC service”:

• modeling boundaries for subject, resource, action, context, hierarchy, and cross-tenant references;
• when ACL, RBAC, ABAC, or relationship-based policies fit, and how to prevent policy logic from diverging across services;
• how the decision-read path, policy-write path, cache, and invalidation events work together;
• when a policy change becomes visible and whether stale cache can still allow access;
• whether an outage is fail-open or fail-closed for each risk class;
• how logs, explanations, replay, and fault injection prove correct denial without cross-tenant leakage.

Clarifying questions to ask

Ask constraints that change the design:

1. What is the consistency window? If revocation must take effect within 5 seconds, cache leases and version tokens must meet 5 seconds; a minute-scale allowance permits a longer TTL.
2. Is the check a single-resource lookup or a relationship query? “Can this user read a document?” may depend on membership, group inheritance, and parent resources; that determines whether a relation graph or batch checks are needed.
3. What is allowed during an outage? Public content may be allowed to fail open, while payment, personal data, and administration usually fail closed. Choose by risk, not habit.
4. Who writes and reviews policies? Direct business-team publishing requires versions, approval, static checks, and rollback; a security-only writer can reduce the control surface.
5. Do callers need explanations? Debugging may need a matched rule ID, but a client-facing explanation must not reveal another tenant’s resources or policies.

A 30-second answer framework

Start with this structure:

“I model a request as (principal, action, resource, context). A policy engine returns allow, deny, a policy version, and an internal rule ID when controlled debugging needs it. A control plane manages tenant-scoped policy and relation data; a data plane evaluates immutable versioned snapshots in a local read-only cache. Publishing emits replayable invalidation events, and callers send a minimum policy version so stale snapshots cannot silently override a newer decision. Sensitive paths fail closed when authorization is unavailable; low-risk reads degrade only under an explicit product policy. I validate revocation propagation, cross-tenant boundaries, cache races, and dependency failures with fault injection and replay.”

This gives the model, data flow, consistency, and failure behavior before a follow-up selects one area to expand.

Step-by-step deep dive

Define policy semantics first. A subject may be a user, service account, or group. A resource may be a tenant, project, or document in a hierarchy. Actions should be a bounded vocabulary such as read, write, and share. Context can include device state, source, or time, but an unverified client field is not a trusted fact. NIST’s ABAC guidance treats subject, object, and environment attributes as decision inputs; relationship policies represent membership as traversable edges.

Separate control and data planes. The control plane edits, validates, approves, versions, and rolls back policies. The data plane loads only published immutable snapshots and answers checks. Business services should not each ship a different policy parser. A publication creates a monotonic version and an invalidation event; the data plane atomically swaps the snapshot so a half-published rule set is never visible.

Design the check API. An internal interface could be:

Check(principal, action, resource, context, min_policy_version)
  -> decision, policy_version, rule_id, expires_at
BatchCheck(principal, [(action, resource, context)...])
  -> decisions[]

rule_id is for controlled logs and debugging. A cross-tenant caller must not discover resource existence through error detail. Batch checks reduce round trips but need limits on batch size and evaluation cost.

Handle consistency and caching. A local cache can meet a 20 ms target, but revocation cannot rely on TTL alone. A caller can send the last policy version it observed; if the local version is behind, the service reads a shared replica or returns “temporarily undecidable.” Invalidation events need a replay cursor and periodic reconciliation so a lost notification does not leave stale allows. For highly sensitive resources, include a resource version or short lease so permission and object versions are checked together.

Choose outage behavior. If the data plane cannot reach the control plane, an unexpired loaded snapshot may continue serving. After it expires, payment, personal-data, and administrative actions should fail closed; public static content may fail open only under an approved product rule. A timeout must be an explicit UNAVAILABLE, not a disguised DENY, or business logic will confuse infrastructure failure with a user permission failure.

Isolate tenants and abuse. Every policy object and cache key carries a tenant ID. The server derives the tenant from verified identity instead of trusting an arbitrary request-body field. Apply quotas per subject, tenant, and batch; cap relation depth and policy complexity so one tenant cannot exhaust evaluation resources.

Audit and verify. Record a request hash, subject, action, resource type, decision, policy version, latency, and failure class; keep sensitive values as irreversible identifiers. Test stale cache after revocation, group-inheritance cycles, identical resource IDs across tenants, a crash during publication, duplicate or lost invalidation events, data-plane restart, and dependency timeout. Acceptance requires replaying a policy version and explaining a decision, not merely returning DENY.

Estimate the bottleneck. At 100,000 checks per second baseline and 300,000 peak, 2 KB per request implies about 600 MB/s of peak ingress. Sending a full policy on every request amplifies latency and bandwidth, so prefer local snapshots, batch checks, and read replicas. If relation traversal becomes the bottleneck, shard by tenant or resource and cap depth and fan-out. State how load tests would replace these assumptions.

High-quality sample answer

“I would first confirm the revocation target and the safety boundary during failure. Assume 300,000 checks per second at peak, a 20 ms p99 target, and revocation effective within 5 seconds. I would separate control and data planes. The control plane stores tenant-scoped ACLs, group relations, and ABAC conditions; before publishing it checks syntax, cycles, and cross-tenant references, then creates an immutable snapshot and monotonic version. Each region’s data plane evaluates (subject, action, resource, context) from a local read-only cache. Invalidation events carry a cursor and are replayable; a caller can send minpolicyversion, so a lagging cache reads a shared replica or returns UNAVAILABLE.

For payment, personal-data, and administrative actions, an expired snapshot or unavailable dependency fails closed; public-content degradation requires explicit product approval. Results carry the policy version and an internal rule ID. Logs record decisions, latency, and error class without echoing another tenant’s data. Resource versions or short leases protect the most sensitive objects from an old allow after revocation. Load tests cover peak traffic and deepest relation traversal; fault injection covers lost invalidations, revocation races, cross-tenant IDs, and regional restarts. Reconciliation and versioned replay must show there are no unexplained allows.”

This connects assumptions, model, consistency, failure handling, and verification. In the interview, trim detail as constraints change; 300,000 checks per second and 5 seconds are not universal production facts.

Common mistakes

• Reducing every case to RBAC. Roles do not naturally express hierarchy, inherited relations, or environment conditions. Define the permission semantics before choosing the model.
• Using a fixed TTL as revocation. TTL bounds one staleness case but does not solve lost notifications or version races. Add versions, replayable invalidation, and reconciliation.
• Always failing open or always failing closed. Resource risk differs. Define behavior from the security floor, snapshot freshness, and business reversibility.
• Copying policy logic into every service. Multiple parsers drift and produce different denials. Version the policy and call one decision interface.
• Returning “resource missing” or detailed rules. That leaks cross-tenant information. Use stable external errors and controlled internal rule IDs.
• Testing only the normal allow path. Revocation, replay, cycles, and tenant boundaries are where authorization fails. Use fault injection and versioned replay.

Follow-up questions and responses

Can a local cache remain if revocation must be immediate?

Yes, but the cache cannot decide alone. Revocation must create a globally observable version or lease invalidation. Sensitive requests carry the minimum policy version; a lagging cache bypasses local data or waits for confirmation. If confirmation cannot fit the budget, return UNAVAILABLE or deny rather than silently using an old allow.

What if relation traversal finds a cycle through parent groups?

Reject cycles during publication, then still enforce maximum depth, node count, and time budget at runtime. The evaluator tracks visited nodes, stops the repeated branch, and returns a diagnosable internal error. A cycle must never turn one check into unbounded work.

How do you prove there is no cross-tenant authorization?

Make tenant scope mandatory in policy namespaces and cache-key prefixes, and derive the subject tenant from server-side identity. Generate identical resource IDs under different tenants with property tests; every allow must satisfy subject, resource, and policy tenant equality. Add log sampling and denial-path replay.

Must you copy Zanzibar’s external consistency model?

No. Choose version tokens, leases, or stronger consistency from the revocation target and business risk. If a product promises that a just-removed member cannot keep reading an old object, causal ordering and object versions are justified. For low-risk public content, weaker consistency and longer caching may be simpler. State the promise and its cost.