Prompt and scope
You operate a multi-tenant Kubernetes cluster on Linux nodes running SELinux in enforcing mode. The platform uses CSI volumes, and some workloads share a volume. The team plans to upgrade to v1.36. Design compatibility checks, a canary rollout, observability, and rollback without assuming that every node has SELinux enabled.
What the interviewer is testing
The interviewer wants you to separate SELinux nodes from ordinary nodes, explain the difference between a mount context and recursive relabeling, and identify risks when privileged and unprivileged Pods share a volume. A strong answer also covers CSI drivers, Pod securityContext, startup latency, data access, admission boundaries, and versioned rollback.
Questions to clarify first
- Which nodes are enforcing, and which CSI drivers and filesystems are in the supported scope?
- Is shared storage a hard business constraint, or can volumes be split or data exchanged explicitly?
- Is the migration optimizing startup latency, tenant isolation, workload continuity, or an explicit trade-off?
- Does the deployed version allow an explicit labeling policy, and what are the rollback window and evidence-retention requirements?
30-second answer
“I would inventory node SELinux modes, CSI drivers, volume types, and sharing relationships first. The v1.36 improvement uses mount contexts for supported volumes, reducing recursive relabeling cost but potentially changing access results for shared volumes. I would run equivalent workloads in an isolated node pool, checking mounts, labels, reads and writes, startup latency, and denial logs before expanding. If a guardrail fails, I would stop new workloads, preserve running Pods and evidence, and roll back using a policy supported by the deployed version.”
Step-by-step solution
1. Build the impact inventory
Record each node's kernel and SELinux state, enforcing or permissive mode, Kubernetes version, CSI driver version, and volume type. Export each Pod's seLinuxOptions, identity, privilege level, mount paths, and sharing relationships. Nodes without SELinux must not be included in the same behavioral conclusion.
2. Explain the behavior change
The Kubernetes change makes the SELinux volume-labeling improvement stable in v1.36. For supported volumes, the system can use a mount context instead of relabeling every file recursively. This usually reduces startup work for large volumes, but the assumption that a shared volume is relabeled before use can fail when different security domains access it. A successful mount does not prove that the application can read and write.
3. Check driver and policy boundaries
For every CSI driver, verify support for mount contexts, filesystem behavior, and mount-option restrictions. Check the deployed version's documentation for seLinuxChangePolicy, related feature gates, and defaults; do not copy a field or switch from an older release without verification. Admission policy should restrict unnecessary privilege, define shared-volume security domains, and audit changes.
4. Run reproducible comparisons
Prepare test Pods with the same image, UID/GID, securityContext, and volume data. Cover a single Pod, same-domain sharing, privileged and unprivileged sharing, an empty volume, and a volume containing many files. Verify mount contexts, file labels, reads and writes, restart recovery, expansion, and CSI reconnects. Measure each interval from Pod creation to Ready.
5. Add canary guardrails
Create a dedicated node pool and a small set of CSI StorageClasses, and start with retryable jobs. Do not give an unverified security-domain combination the same shared volume during the canary. Set thresholds for denial logs, mount failures, application permission errors, startup latency, and restart success. Stop expansion when a guardrail fires; do not hide the issue by broadening SELinux or privilege.
6. Handle shared-volume risk
When a volume is shared by privileged and unprivileged Pods, first split the volume or unify the security domain before continuing. For unavoidable sharing, define who owns labeling, when the volume is mounted, and how it is reclaimed, then validate an actual access matrix. Delete an old volume only after application recovery is proven, so a labeling problem is not mistaken for data loss.
7. Preserve rollback evidence
Rollback starts by stopping new Pods from entering the canary pool while preserving running jobs and events. Then select an explicitly supported recursive policy or an older node pool for the deployed version. Keep feature-gate settings, node labels, CSI configuration, Pod manifests, and SELinux AVC denial logs. Re-run the same comparison after rollback; a successful deployment command alone is not proof of recovery.
Model answer
I would structure the migration as inventory, comparison, canary, and rollback. Only Linux nodes with available SELinux in enforcing mode are in scope, and CSI driver support must be checked against the deployed Kubernetes version. v1.36 uses mount contexts for supported volumes, reducing recursive relabeling but requiring focused tests for shared volumes and different security domains. Tests cover empty and large volumes, one Pod, same-domain sharing, and privileged/unprivileged sharing; they compare labels, access, denial logs, startup latency, and restart recovery. The canary pool accepts retryable work only. A failed threshold stops expansion, preserves evidence, and returns new work to a documented supported path.
Common mistakes
- Treating every node as an SELinux node → the impact scope is wrong → partition by node mode and enforcing state.
- Testing only that mounting succeeds → runtime access is still denied → test the real identity, labels, reads, writes, and restart.
- Broadening privilege immediately → the security boundary expands → fix the security domain, sharing relationship, or driver configuration.
- Ignoring CSI differences → one volume class fails in the canary → matrix the driver, filesystem, and volume type.
- Deleting every Pod during rollback → evidence is lost and jobs stop → freeze new traffic and retain state and logs.
Follow-up questions and responses
Does permissive mode require the same migration?
It still deserves inventory and testing because permissive mode records denials without normally enforcing them. It does not represent enforcing behavior, so release gates must be reproduced on enforcing nodes.
How do you separate a labeling issue from a CSI issue?
Hold the node, image, and security context constant while swapping the volume type or driver. Compare mount events, kernel denials, file labels, and CSI logs. Attribute the issue to the security-context path only when it reproduces across drivers.
Is faster startup enough to prove success?
No. Performance is only one signal. The access matrix for different domains, restart recovery, expansion, reconnect behavior, and audit evidence must also meet the release gates.
What if shared volumes cannot use one security domain?
Prefer splitting the volume or using an explicit data-exchange path. If sharing is unavoidable, the platform should own the security-domain and lifecycle decisions, and the permitted Pod combinations should be enforced and regression-tested.