System Design Interview: How Would You Migrate Kubernetes Pods to User Namespaces?

Prompt and applicable context

A Kubernetes cluster contains legacy images that still run as UID 0 inside the container. Security wants user namespaces to reduce the host impact of a container escape, while the business is worried about volume ownership, hostPath, monitoring agents, privileged capabilities, and older nodes. Design the migration, validation, canary, observability, and rollback plan.

This is a system-design question. The signal is mapping isolation to identity, storage, scheduling, policy, and operations instead of only writing hostUsers: false.

What the interviewer assesses

• Explaining the difference between container-root and host UID, including capability scope.
• Finding boundaries across volumes, host namespaces, CRI/OCI runtimes, and Pod Security.
• Designing compatibility checks and staged migration without breaking existing workloads.
• Defining security benefit, performance cost, SLOs, metrics, and rollback conditions.
• Handling application, debugging, monitoring, and backup counterexamples after migration.

Kubernetes documents user namespaces as stable and enabled by default in v1.36; a Pod opts in with spec.hostUsers: false. Amazon’s SDE II material evaluates system design through reliability, efficiency, optimization, and scalability. This question adds the requirement to quantify security benefit and operational risk together.

Clarifying questions

1. Is the cluster Linux-only, and do kubelet, CRI, OCI runtime, and kernel versions meet the requirement?
2. Do workloads use hostNetwork, hostPID, hostIPC, hostPath, devices, privileged containers, or monitoring agents that need host UIDs?
3. Which volumes are shared across Pods, and are existing file UIDs/GIDs inside the mappable range?
4. Does the application really need host root, or only root semantics inside its container?
5. Is the goal escape containment, Pod Security compliance, or a specific namespaced capability such as container-local network administration?
6. Can you canary by namespace, node pool, or workload while retaining a hostUsers=true template for rollback?

30-second answer framework

Build an eligibility matrix for Linux, kernel, CRI/OCI runtime, host namespaces, volumes, and privileged capabilities. Set hostUsers: false only for eligible Pods. Verify that application UID semantics remain stable while the host sees a mapped, non-privileged UID. Test file access, monitoring, networking, and debugging, then canary by workload. Watch startup latency, permission errors, OOMs, escape controls, and business SLOs; switch back to the old template on threshold breaches.

Step-by-step deep answer

1. Explain the isolation model

A user namespace maps container users to different host UIDs and GIDs. Root inside the container can perform operations needed inside that namespace, but its capabilities are valid only there and do not grant host-root power. Kubernetes opts a Pod in with hostUsers: false and assigns non-overlapping host mappings on a node.

This changes a kernel identity boundary; it is not a complete sandbox. Continue using seccomp, AppArmor or SELinux, network policy, read-only filesystems, least privilege, and patched nodes. User namespaces are one control, not a solution for every escape path.

2. Check runtime and node eligibility

Confirm user-namespace support in kubelet, CRI, and OCI runtime on every target node. Kubernetes documentation lists support such as containerd 2.0, CRI-O 1.25, runc 1.2, and crun 1.9; deployments still need to verify kernel, idmapped mounts, and distribution settings.

Admission policy can reject nodes or capability combinations that are not eligible. CI renders the final Pod and checks hostUsers, security context, host namespaces, and volume types. Before rollout, a probe Pod validates creation, mounts, restarts, and node rescheduling.

3. Evaluate volumes and UID/GID

Pod runAsUser, runAsGroup, and fsGroup still describe the user inside the container. Volume permission semantics should remain usable as before, so applications normally do not need a blanket ownership rewrite just because user namespaces are enabled.

Files outside the mapped UID/GID range can appear as an overflow ID and may not be writable. Scan owners, init scripts, shared volumes, and backup/restore tools before migration. Repair images or data before enabling user namespaces for that workload.

4. Handle forbidden combinations and policy

With user namespaces enabled, a Pod cannot use certain host namespaces such as hostNetwork, hostPID, or hostIPC. Workloads that need host devices, privileged mode, or special proc mounts require separate review. Pod Security Standards may relax selected checks in a controlled way, but that is not permission to remove every other policy.

Mark host-namespace-dependent workloads as deferred and record exception approval, compensating controls, and an expiry. Do not automatically delete fields just to pass admission; that turns a functional failure into an invisible security downgrade.

5. Design performance, SLO, and observability

Track Pod startup and volume-mount latency, node CPU and memory, in-container permission errors, monitoring loss, backup/restore success, and restarts. Idmapped mounts can avoid recursive chown on large volumes, but measure with the real kernel, runtime, and filesystem rather than relying on a theoretical claim.

Tag logs and metrics with migration version, node pool, image, and volume type. Security signals include rejected escape tests, failed privileged operations, and Pod Security violations; business signals include request errors, latency, and data integrity.

6. Stage migration and rollback

Start with stateless workloads that use no host namespace and simple volumes, then expand to stateful services. Keep the hash of a hostUsers=true template for each batch. Stop automatically on permission errors, startup regression, business error rate, volume read/write failures, or node evictions.

Rollback changes more than one field: verify volume mounts, runAs users, monitoring agents, and admission output return to the old shape. Avoid upgrading runtime, kernel, and image in the same window so failures remain attributable.

7. State security benefit and residual risk

User namespaces reduce the host impact of a container-root escape and provide a narrower domain for workloads that need container-local administration. They do not protect application secrets, cross-container logic attacks, bad network policy, or an already compromised shared service.

The security review should list remaining hostPath, devices, capabilities, kernel interfaces, and service accounts, then combine seccomp, SELinux, node isolation, and vulnerability patching into defense in depth.

High-quality sample answer

I would build an eligibility matrix: target nodes must be Linux and meet user-namespace requirements for kubelet, CRI/OCI runtime, kernel, and filesystem. I would exclude Pods using hostNetwork, hostPID, hostIPC, privileged devices, or host UID assumptions. Eligible templates set hostUsers: false; tests confirm container UID/GID semantics remain correct while the host sees an unprivileged mapping.

Before rollout I would scan volume owners, init scripts, shared volumes, and backup/restore paths, including overflow-ID behavior. A probe Pod verifies creation, mounts, restarts, and rescheduling. Canaries measure startup and mount latency, permission errors, business SLO, evictions, monitoring, and backup signals. The old template hash remains available for an immediate rollback.

I would treat user namespaces as one defense-in-depth layer and keep seccomp, SELinux/AppArmor, network policy, least privilege, and node patching. Security benefit, performance cost, and deferred workloads belong in the migration checklist; setting one field is not the migration itself.

Common mistakes

• Writing only hostUsers: false without checking Linux, runtime, kernel, and volume conditions.
• Saying container root becomes an ordinary user and ignoring the two UID meanings.
• Treating user namespaces as an automatic fix for every escape, privilege, or network risk.
• Missing hostNetwork, hostPID, hostIPC, hostPath, device, and proc-mount limits.
• Recursively chowning every volume or failing to scan overflow UID/GID files.
• Upgrading runtime, kernel, and image together, making failures impossible to attribute.
• Having no old template, automatic stop condition, or verifiable rollback.

Follow-up questions and responses

Which Kubernetes version makes user namespaces stable?

The official documentation marks the feature stable and enabled by default in Kubernetes v1.36; Pods still opt in with spec.hostUsers: false. Verify the actual cluster and node versions before rollout.

Can container root still use capabilities?

It can use capabilities valid inside that user namespace, but those capabilities do not automatically become host privileges. Apply least privilege, seccomp, and LSM controls.

Will existing PVC permissions all break?

Container UID/GID semantics normally remain stable, so a blanket ownership rewrite is not required. Files outside the mapping range can become overflow IDs and need to be scanned and repaired.

Why cannot hostNetwork be combined with it?

User namespaces depend on isolated user and resource boundaries, and some host-namespace combinations would undermine that isolation, so Kubernetes disallows them. Keep host-network workloads as explicit exceptions with compensating controls.

How do you prove risk was reduced?

Run escape-isolation and privileged-operation tests, observe host UID, capabilities, and access scope, and compare business errors, startup latency, and volume integrity. Pod creation success alone is not proof.

Which workloads should wait?

Defer workloads needing host namespaces, special devices, privileged kernel interfaces, or unrepairable volume UID/GID layouts. Record the reason, compensating controls, and a reassessment date.