Problem and context
Kubernetes ValidatingAdmissionPolicy evaluates CEL expressions against API requests. A policy definition is paired with a binding that selects resources and defines the enforcement action. This gives a cluster-native validation path, but a badly scoped or unavailable policy can reject legitimate recovery operations.
Assume production namespaces are labeled, image provenance is available as an admission field, and platform engineers own the policy. The design must distinguish hard safety rules from warnings, support dry runs, and preserve an audited emergency path.
What interviewers evaluate
Interviewers look for separation between policy definition and binding, explicit failure behavior, narrow resource matching, and a testable exemption model. Strong answers discuss CEL type checking, versioned rollout, audit metrics, and what happens when the policy engine or referenced data is unavailable.
An ordinary answer writes one expression that rejects bad YAML. A strong answer explains scope, action levels, compatibility, observability, and rollback.
Questions to clarify first
- Which API groups, versions, namespaces, and operations are in scope?
- Is the rule about image identity, digest pinning, host access, or all of them?
- Should a violation deny, warn, or only audit during the pilot?
- Which controllers or system namespaces need exemptions, and who approves them?
- What is the recovery path if a policy blocks an incident fix?
If the rule depends on external state that CEL cannot reliably access, keep the policy local and use a separate controller for richer checks. If exemptions are broad, the policy may provide false confidence.
A 30-second answer
“I would define small, typed CEL rules and bind them only to production workloads and relevant operations. I would start in audit or warning mode, measure violations by owner, then enforce one rule at a time. Exemptions would be narrow and audited, with a tested emergency path. I would monitor admission latency, rejection rate, and recovery success, and roll back the binding if policy behavior is unsafe.”
Step-by-step design
- Define the invariant. Write separate rules for approved image registry, digest requirements, and host privileges. Avoid one expression that is hard to test or explain.
- Type-check and test CEL. Validate expressions against the targeted resource schemas, including missing fields, lists, updates, and delete operations. Keep fixtures in the policy review workflow.
- Scope with a binding. Match only production namespaces, workload kinds, and operations that need protection. Separate bindings let teams adopt rules independently.
- Choose enforcement. Begin with audit or warning, promote to deny after owners remediate violations, and record the action in change history.
- Design exemptions. Exempt only named system identities or namespaces, require an expiry or approval record, and alert when exemptions are used.
- Roll back safely. Keep policy definitions versioned, disable the binding rather than deleting history, and verify that emergency Deployments can proceed under the documented path.
Alternatives include a validating webhook for external data or mutation plus validation for normalization. Use the built-in policy for deterministic, local invariants that must be enforced close to the API server.
Example answer
“I would create three policies: registry allow-list, digest required, and hostNetwork forbidden for production Deployments. Each binding would select only labeled production namespaces. The pilot runs in audit mode for one week; owners receive violation reports, and we fix false positives before enabling deny. Platform and break-glass identities are the only exemptions, with expiry and audit events. We alert on admission latency and rejected recovery changes, and rollback by disabling the binding if an incident reveals an unsafe rule.”
Common mistakes
- Error: Matching every namespace and operation → Why it fails: system controllers and recovery paths get blocked → Fix: scope bindings narrowly.
- Error: Treating warnings as enforcement → Why it fails: users assume safety is guaranteed → Fix: communicate action level and promotion criteria.
- Error: Using untyped CEL on missing fields → Why it fails: updates and older API versions behave differently → Fix: type-check and test absent-field cases.
- Error: Making exemptions permanent → Why it fails: bypasses become the real policy → Fix: require expiry, owner, and audit.
Follow-up questions and responses
What if a policy expression has a syntax error?
Kubernetes validates the policy definition and reports the error rather than accepting an invalid expression. Still run it through review fixtures and keep the binding in audit mode until behavior is confirmed.
How do you handle a rule that needs an external vulnerability database?
Do not pretend CEL can fetch arbitrary external state. Use a controller or webhook that owns freshness, timeout, and failure semantics, while keeping local invariants in the built-in policy.
How can an on-call engineer recover when a deny rule blocks a fix?
Provide a narrow, audited break-glass identity or namespace with expiry, document approval, and alert on use. Test the path before enforcement.
When would you remove the policy?
Remove or disable the binding when false positives remain high, admission latency threatens the API server, or the invariant has moved to a stronger control. Preserve history and metrics for the decision.