System design interview: How would you design a Kubernetes audit policy and reliable log pipeline?

Prompt and scope

A multi-tenant cluster must answer who changed what, when, and from where while controlling API-server memory, logging cost, and Secret leakage. Design the audit policy, file or webhook backend, sampling and alerts, failure handling, and forensic verification.

What the interviewer is testing

• Understanding ordered rules and the differences among None, Metadata, Request, and RequestResponse.
• Distinguishing RequestReceived, ResponseStarted, and ResponseComplete stages.
• Handling long-lived requests, sensitive bodies, backend blocking, and log integrity.
• Connecting audit goals to alerts, retention, access control, and drills.

Clarifying questions

1. Which high-risk resources, verbs, tenants, and principals must be audited?
2. Which bodies contain Secrets, tokens, or personal data, and how long must they be retained?
3. Should the backend be a file, webhook, or both, and what loss and latency window is acceptable?
4. Is tamper evidence, cross-region replication, and restricted raw-log access required?

A 30-second answer

Map investigation questions to fields and high-risk APIs, then order rules from specific to catch-all. Use Metadata for most traffic, Request for selected changes, and RequestResponse sparingly so Secret values do not enter ordinary logs. Omit early stages only when the reason is explicit. Protect file or TLS webhook backends with buffering, rate limits, encryption, integrity checks, access audits, and loss alerts. Drill backend failures to prove the chosen semantics.

Step-by-step design

1. Define the forensic question

Turn who, when, where, and what into fields and queries. High-risk objects include Secrets, RoleBindings, webhooks, nodes, and tenant quotas; routine health checks rarely need full bodies. The policy should answer investigations, not collect every possible field.

2. Write ordered rules

Order rules from specific to catch-all; the first match sets the level. Use Request or necessary RequestResponse for high-risk changes, Metadata for routine reads, None for explicit noise, and a low-level fallback to avoid silent gaps.

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - RequestReceived
rules:
  - level: Request
    resources:
      - group: ""
        resources: ["secrets"]
  - level: Metadata
    omitStages: ["RequestReceived"]
    resources:
      - group: ""
        resources: ["pods"]
  - level: None
    users: ["system:kube-probe"]

3. Control stages and sensitive fields

Long-lived requests can emit ResponseStarted and later ResponseComplete; stage omission can reduce duplication, but one request need not produce only one event. For Secrets and identity material, prefer metadata, hashes, or controlled summaries, never raw values in a normal log system.

4. Select and protect the backend

Rotate, compress, encrypt, and forward file logs reliably. A webhook needs TLS, authentication, queues, and backpressure. Partition the pipeline by tenant and risk, use append-only storage, and attach audit ID, policy version, and receive time. Downstream consumers must not synchronously block the API server.

5. Define failure semantics

Specify what happens when the backend is unreachable, disk is full, a queue overflows, or a network partitions: drop, block, or degrade. High-risk writes may use fail-closed only after quantifying control-plane impact; ordinary traffic can use bounded buffering and alerts. Record loss counters and the recovery scan range.

6. Verify and improve continuously

Generate events from known users, ServiceAccounts, proxies, and long-lived requests. Check rule matches, stages, tenant attribution, and redaction. Inject webhook delay, full disks, and policy updates, then verify alerts, recovery, and forensic queries. Track event volume, loss, latency, cost, and investigation coverage.

Model high-quality answer

I would map investigation questions to fields and high-risk resources, then order rules from specific to catch-all. Routine traffic gets Metadata, Secrets get controlled metadata, and selected changes get Request; long-lived stages are explicit. The backend uses encrypted files or a TLS webhook with buffering, backpressure, append-only integrity, and access auditing, without letting consumers block the API server. Failure behavior includes bounded buffering, loss counters, and alerts, with fail-closed reserved for quantified high-risk writes. Synthetic events, long-lived requests, and failure drills prove the policy and recovery path.

Common mistakes

• Recording every request at RequestResponse → sensitive leakage and runaway cost → tier by risk.
• Ordering rules casually → a catch-all hides high-risk events → specific-to-general ordering.
• Checking only that a log file exists → backend blockage or loss is invisible → monitor queue, latency, and loss.
• Ignoring stages → long-lived investigations miss timing → define stages and omission reasons.
• Synchronously blocking the API server on webhook consumers → control-plane collapse → isolate with buffers, timeouts, and backpressure.

Follow-up questions and responses

Why not record every Secret request at RequestResponse?

The body may contain plaintext Secrets. Most investigations need principal, object, verb, and result; deeper access requires redaction, isolated permissions, and short controlled retention.

Should the API request be blocked when the webhook is unavailable?

It depends on risk and availability goals. After quantifying impact, high-risk writes may fail closed while ordinary traffic uses bounded buffering and alerts. Either choice must expose loss and recovery scope.

How do you prove unknown resources are not silently omitted?

Keep a low-level fallback, compare discovery inventories with synthetic requests and policy versions, and trigger review when a new API appears instead of relying on manual detection.