General interview: How would you use cgroup v2 memory.high and memory.max to control container risk?
Prompt and context
A platform team finds that one tenant's batch container can squeeze online services during a peak. The node uses cgroup v2, but the team set only one memory limit and cannot distinguish reclaim, throttling, and termination. Design a layered memory policy with protection boundaries, event monitoring, and rollback.
What the interviewer is testing
- Whether you separate memory.min/low protection from memory.high reclaim/throttling and memory.max enforcement.
- Whether you understand hierarchical memory.events and local memory.events.local.
- Whether you handle OOM group killing, child cgroups, bursts, and overcommit.
- Whether you can connect kernel files to orchestration, alerts, and drills.
Clarifying questions to ask first
- Are you protecting critical workloads, limiting a tenant, or protecting the whole node?
- Is the workload latency-sensitive, batch-oriented, or safe to reclaim and retry?
- What is the cgroup hierarchy, and do parent groups contain sidecars?
- Does Kubernetes manage the workload, and how do requests and limits map to cgroup v2?
- Should OOM kill one process, the entire cgroup, or cause the controller to rebuild the container?
A 30-second answer framework
Use memory.min and memory.low for protection levels, memory.high for an early reclaim and throttling boundary, and memory.max as the final hard limit. memory.oom.group determines whether OOM handling is group-wide. Allocate budgets from node to tenant to workload, monitor memory.current, memory.events, and PSI, and validate with canaries, pressure tests, and rollback. In Kubernetes, read the actual cgroup files after rendering instead of trusting YAML alone.
Step-by-step deep answer
Step 1: Build the semantic model
memory.min is a hard protection boundary whose usage is reclaimed only with strong pressure; memory.low is best-effort protection that can be reclaimed under severe pressure. memory.high triggers reclaim and throttling but does not directly invoke the OOM killer. memory.max is an uncrossable limit that can enter cgroup OOM when reclaim cannot satisfy it.
Step 2: Allocate parent and child budgets
Reserve node capacity for the system and critical services, assign a ceiling to each tenant parent, then set child groups for online, batch, and sidecar workloads. Child protection cannot unconditionally exceed the parent's available budget. Record each level's current usage, events, and policy version. Leave headroom below high for bursts so a short spike does not become an immediate max OOM.
Step 3: Choose the high-to-max relationship
memory.high lets the kernel apply reclaim and throttling first; the application can slow down after observing latency, throughput, and events. memory.max is the final safety valve and should reflect recoverability. A high that is too low causes chronic throttling, while a max that is too high exports pressure to the parent or node. Calibrate both with load tests.
Step 4: Decide the OOM group policy
For tightly coupled processes, memory.oom.group=1 makes OOM handling group-wide so a helper is not left behind after only the main process is killed. Independent batch jobs may prefer one process to exit and a queue to retry it. Record exit cause, restarts, and unfinished work for either choice; OOM is not an invisible retry.
Step 5: Monitor memory.events
memory.events is hierarchical, so child events can appear in a parent; memory.events.local reports only local events. Monitor deltas for high, max, oom, and oom_kill, correlated with memory.current, working set, PSI, latency, and queue age. Parse keys rather than relying on file order because new keys can appear.
Step 6: Map policy to orchestration
On Kubernetes, inspect requests, limits, QoS, and the node's cgroup mode together. After rendering, enter the container and read /sys/fs/cgroup to verify the actual values and that the runtime did not override them. Check sidecars, init containers, and shared emptyDir accounting; a child can look healthy while its parent reaches high or max.
Step 7: Drill, release, and roll back
On a single-node canary, lower high gradually and observe reclaim and throttling, then test max and oom.group termination. Alert separately on sustained high, first max, and actual oom_kill. Publish versioned policies with a previous version and event snapshot. Roll back the budget first, then reconcile killed jobs and queue retries so pressure is not left on the node.
High-quality sample answer
Use min and low for protection, high for observable reclaim and throttling, and max as the final hard boundary. Allocate node, tenant, and workload budgets through the parent-child hierarchy, then choose oom.group based on recoverability. Monitor hierarchical versus local memory.events with PSI, latency, queue age, and restarts. In Kubernetes, verify the rendered cgroup files. Roll out with a canary and pressure drill, and keep a versioned rollback policy.
Common mistakes
- Treating memory.high as an immediate OOM limit.
- Treating memory.low as an unbreakable guarantee.
- Trusting container YAML without reading runtime cgroup files.
- Mixing parent hierarchical events with local events.
- Restarting after OOM without checking idempotency, partial work, and retry storms.
Follow-up questions and responses
Follow-up 1: What happens after memory.high is crossed?
The kernel applies reclaim and throttling pressure. Processes can continue but latency may rise. It is not an OOM-killer switch; correlate the high counter with PSI and service latency.
Follow-up 2: Why keep memory.max if high exists?
High lets a workload yield under pressure, while max bounds growth that cannot be reclaimed and protects the parent or node. Max must be designed with recoverability and restart policy.
Follow-up 3: Why can memory.events appear to count an event twice?
Parent memory.events includes subtree events by default, while memory.events.local is local to that cgroup. Choose one level for each alert and deduplicate by hierarchy.
Follow-up 4: When should memory.oom.group be enabled?
Enable it when processes in one cgroup must live and restart together. Independent tasks can exit individually. Either choice needs cleanup, retry, and observability tests.
Follow-up 5: How do you prove Kubernetes mapping is correct?
On the target node, read memory.min, memory.low, memory.high, memory.max, and events for the container cgroup. Compare them with rendered requests, limits, QoS, and runtime settings, then run a pressure drill.