Linux Interview: How Do You Explain and Diagnose CPU Scheduling Latency?

Prompt and Applicable Roles

A 16-vCPU server running Linux 6.12 and cgroup v2 hosts a latency-sensitive API alongside an offline compression job. After the offline job starts, API p99 rises from 45ms to 600ms. Host monitoring reports about 78% aggregate CPU utilization, and the API accumulates little CPU time. The relevant API and offline-job threads all use SCHED_OTHER.

Explain how the modern Linux fair scheduler selects among runnable threads, then give a testable diagnostic process that distinguishes these causes:

• the API threads are runnable but wait too long for a CPU;
• the API cgroup is throttled by its cpu.max quota;
• affinity or a cpuset confines the work to a few busy CPUs;
• the API threads are actually sleeping on a mutex, I/O, or another event;
• higher-priority real-time threads prevent normal threads from running.

This question fits SRE, infrastructure, systems software, performance engineering, and backend roles that require Linux runtime knowledge. The values 16, 6.12, 45ms, 600ms, and 78% are fictional exercise inputs, not universal capacity or alert thresholds.

What the Interviewer Is Testing

The first boundary is task state. A scheduler can select only runnable threads. A thread sleeping on a mutex, network operation, disk operation, or timer is not waiting on a CPU run queue. That whole interval must not be labeled scheduling delay. The candidate should first answer whether the thread was already runnable.

The second layer is the fair-scheduling model. The historical CFS explanation uses virtual runtime, or vruntime, to approximate an “ideal multitasking CPU” and favors entities with smaller vruntime. The current EEVDF model continues to track the fairness debt called lag, admits only eligible entities, and selects the earliest virtual deadline among them. A nice value or cgroup weight affects long-run relative share. A request slice and virtual deadline affect how soon a task can get another execution opportunity. Neither mechanism promises a fixed p99 for an individual request.

The third layer is hierarchy and local constraints. A host with 22% idle capacity can still have a target cgroup that exhausted a hard quota or a thread confined to two saturated CPUs. cpu.weight is a relative weight among active sibling cgroups during contention, while cpu.max imposes a hard bandwidth limit per period. Task groups and cpusets can make an aggregate host value hide local starvation.

The fourth layer is direct evidence. Run-queue wait from /proc/12345/schedstat, runnable-to-running delay from perf sched timehist, and cgroup cpu.stat and cpu.pressure are closer to the mechanism than aggregate CPU or a context-switch count alone. A strong answer also bounds tracing privilege, overhead, and duration.

Finally, the candidate needs a mitigation boundary. Moving the API to SCHED_FIFO can make it preempt normal work, but an unbounded loop or lock holder could then starve the machine. The fix must follow the evidence: correct a bad quota, adjust sibling weights, repair a CPU set, or investigate a lock or I/O path when the thread was not runnable.

Questions to Clarify Before Answering

• Where does p99 timing begin and end? Separate ingress queueing, application execution, downstream waiting, and client network time, and align host samples to the same interval.
• When is the target thread runnable? Obtain thread states, off-CPU reasons, or stacks. Little CPU time can mean lack of scheduling or extended sleep.
• What cgroup hierarchy contains the API and offline job? Compare cpu.weight, cpu.max, parent limits, and cpu.stat for both. A parent cgroup also constrains its children.
• Which CPUs may run the work? Check thread affinity, cpuset.cpus.effective, per-CPU utilization, and the NUMA layout. A 16-vCPU host does not mean the thread is allowed to use all 16.
• Are there any real-time scheduling entities? Inspect priorities and CPU affinity for SCHEDFIFO and SCHEDRR work. The prompt says only that the relevant business threads use SCHED_OTHER; it does not exclude unrelated real-time threads on the host.
• Did the offline job alter a lock, memory, or I/O path? Compression can create CPU contention, but it may also intensify reclaim, file I/O, or queueing in a shared worker pool.
• Can a short trace be captured? perf sched commonly needs extra privilege and creates recording overhead. Confirm the direction with low-overhead counters first, then sample for 15 seconds in a controlled window.

30-Second Answer Framework

“I would first establish whether the API thread was already runnable during a slow request. Time sleeping on a lock or I/O is an off-CPU wait. Scheduling delay is the interval from runnable to actually running.

On Linux 6.12, I would explain the fair class through EEVDF: it tracks each entity's lag relative to ideal fair service, considers eligible entities, and selects the earliest virtual deadline. CFS's smallest-vruntime model remains useful historical intuition, but it does not fully describe the current selection rule. Nice and cpu.weight change relative share, cpu.max can throttle directly, and affinity or a cpuset controls which CPUs are available.

I would inspect per-CPU utilization, thread states, and cgroup configuration. Then I would compare the run-queue wait delta in /proc/12345/schedstat, throttling deltas in cpu.stat, and cpu.pressure. If the evidence still points to scheduling, I would capture a short perf sched trace to measure runnable-to-running delay directly. After a fix, I would replay the same load and verify API p99, the scheduling-delay distribution, throttling, CPU pressure, and offline throughput together instead of deriving a cause from 78% aggregate CPU.”

Step-by-Step Deep Dive

Step 1: Draw the Line Between Runnable and Sleeping

When a CPU needs another task, the Linux scheduler chooses only an entity runnable on that CPU or eligible to migrate there. Normal-policy threads have scheduling priority 0. Real-time SCHEDFIFO and SCHEDRR threads use a higher static-priority range and can preempt normal threads.

Little API CPU time therefore has two fundamentally different explanations:

1. The thread is runnable but does not run promptly because of run-queue contention, a quota, or CPU-set restrictions.
2. The thread sleeps on a futex, socket, disk, timer, or application queue and becomes runnable only when the event arrives.

Start by inspecting thread state and scheduling policy:

ps -eLo state,cls,rtprio,pri,ni,psr,pid,tid,comm --sort=-rtprio,-pri
pidstat -t -w -u -p "$pid" 1 10

ps is an instantaneous snapshot, so a single R or S does not settle the question. Context-switch and CPU data from pidstat help identify target threads, but a high context-switch count does not prove high scheduling delay. If the thread mostly sleeps, continue with an application trace, lock profiling, off-CPU stacks, I/O metrics, and downstream latency to locate the wake-up condition.

Step 2: Use CFS for Intuition, Then Update the Model to EEVDF

The classic CFS explanation scales each fair entity's actual execution time by its weight into vruntime. An entity with more weight accumulates vruntime more slowly. Selecting a smaller-vruntime entity moves long-run service toward the configured weights. That remains useful teaching intuition and explains the direction in which nice changes fair share.

Linux kernel documentation says the fair class began transitioning to EEVDF in 6.6. A concise answer can use two concepts:

• lag represents how much service an entity is owed or has received relative to ideal fair service; an entity with lag >= 0 is eligible;
• among eligible entities, the scheduler chooses the earliest virtual deadline. A shorter requested slice produces an earlier deadline and improves the response opportunity for latency-sensitive work.

EEVDF handles a fairness-versus-latency tradeoff, but no fixed millisecond SLA follows from a load weight. Actual wait still depends on available CPUs, entity count, the weight hierarchy, quotas, wake-up timing, and higher scheduling classes. In an interview, use CFS vruntime for historical intuition, then finish with the EEVDF selection rule required by the Linux 6.12 prompt.

Step 3: Separate Nice, Relative Weight, Hard Quota, and CPU Set

For SCHED_OTHER, the normal nice range is -20 to 19, and Linux keeps nice values per thread. Nice changes relative weight within the fair class. It reserves no CPU and cannot remove a cgroup limit. With task-group scheduling, entities in different cgroups first compete at the group level. Renicing one thread inside a group may do little to alter the share between two cgroups.

Three cgroup v2 controls answer different questions:

• cpu.weight defaults to 100 and ranges from 1 to 10000; it distributes contended CPU cycles proportionally among active siblings;
• cpu.max has the form $MAX $PERIOD and defaults to max 100000; a numeric maximum throttles the whole cgroup after it exhausts the period budget;
• cpuset.cpus.effective is the CPU set actually available after hierarchy and system state are applied.

Read host, thread, and target-cgroup constraints:

mpstat -P ALL 1 10
taskset -pc "$pid"
cat /sys/fs/cgroup/api/cpuset.cpus.effective
cat /sys/fs/cgroup/api/cpu.weight
cat /sys/fs/cgroup/api/cpu.max
cat /sys/fs/cgroup/api/cpu.stat
cat /sys/fs/cgroup/api/cpu.pressure

If nrthrottled and throttledusec in cpu.stat increase during the slow-request interval, the cgroup did hit its bandwidth limit. A hard group budget can stop that group even while other host CPUs are idle; the two metrics have different scopes. If quota counters remain flat but cpuset.cpus.effective is only 2-3 and CPUs 2 and 3 are saturated, the 78% host average is equally misleading.

Step 4: Measure the Wait from Runnable to Running Directly

After choosing a representative slow thread, read its scheduling counters twice:

cat /proc/"$tid"/schedstat
sleep 5
cat /proc/"$tid"/schedstat

The three fields are cumulative nanoseconds running on a CPU, cumulative nanoseconds waiting on a run queue, and the number of timeslices obtained. Use the delta between readings instead of the lifetime total. Fast growth in the second field with little runtime supports “runnable but not running promptly,” although quota, affinity, and higher-priority threads are still needed to explain why.

When a distribution and timeline are needed, take a brief system-wide trace:

sudo perf sched record -a -- sleep 15
sudo perf sched timehist --state --summary

perf sched timehist can show wait time, scheduling delay from runnable to running, and runtime, while placing wake-ups, migrations, and CPUs on a timeline. In production, first confirm kernel support, privilege, disk space, and acceptable overhead. Bound the collection window and remove trace data that is no longer needed. One 15-second sample that misses a tail event does not clear the scheduler; compare distributions across repeatable before-and-after runs.

Step 5: Converge on Five Causes with Discriminating Evidence

Use the following matrix to narrow the cause:

| Evidence combination | Direction | Next evidence | |---|---|---| | High run-queue wait and perf sched delay, high CPU pressure, no throttling | Fair-class run-queue contention | Per-CPU queues, weights, batch concurrency, and migrations | | nrthrottled and throttledusec rise with p99 | Exhausted cgroup hard quota | Parent and child cpu.max plus the capacity budget | | A few CPUs are full, others idle, and the effective cpuset is narrow | Affinity/cpuset hotspot | Pinning intent, NUMA placement, and migration range | | The thread sleeps most of the time and run-queue wait is low | Lock, I/O, timer, or application-queue wait | Futex/off-CPU stacks, I/O, and downstream traces | | A high-priority FIFO/RR thread runs for long intervals on the same CPU | Real-time scheduling starvation | Real-time priority, runtime, locks, and affinity |

cpu.pressure measures time during which runnable work cannot make progress because of CPU contention. It is useful evidence that CPU competition affects the workload, but it does not distinguish insufficient relative weight from quota throttling or bad pinning. cpu.stat, affinity, and scheduler traces complete that distinction.

Real-time work follows a separate priority path. A SCHEDFIFO thread runs until it blocks, is preempted by a higher-priority real-time thread, or yields. SCHEDRR adds a quantum only among threads at the same real-time priority. No fair-class weight lets a normal thread outrank a continuously runnable, higher-priority real-time thread.

Step 6: Fix the Proven Mechanism and Verify Both Business and Scheduler Metrics

Each mitigation should correspond to a confirmed mechanism:

• bad quota: raise or remove the incorrect cpu.max after evaluating capacity and neighbor impact;
• insufficient relative share: increase the API cgroup weight, reduce the offline cgroup weight, or lower the offline task's nice priority;
• bad CPU set: widen or rebalance the cpuset or affinity while checking NUMA and cache locality;
• fair-class contention: limit offline concurrency, split batches, or use SCHEDBATCH or SCHEDIDLE for appropriate offline work;
• sleeping wait: fix lock contention, worker pools, I/O, memory pressure, or a downstream service; scheduler tuning usually has no benefit;
• real-time starvation: remove an unnecessary real-time policy, bound runtime, and inspect priority inversion and lock dependencies.

Run a before-and-after comparison at the same request rate and offline load. Verify at least API p50/p95/p99 and errors, the per-thread scheduling-delay distribution, schedstat run-queue-wait deltas, cgroup throttling, CPU pressure, per-CPU utilization, and offline throughput. Improving API p99 must not permanently starve the offline job or transfer pressure to a downstream service or another tenant.

Strong Sample Answer

“A 78% host CPU average does not clear the scheduler. Linux schedules threads, and constraints can live at the thread, CPU, or cgroup level. I would first establish whether the thread handling a slow request is runnable. If it sleeps on a futex or I/O, its low CPU time mainly reflects waiting for a wake-up, so the scheduler could not select it.

The prompt specifies Linux 6.12. CFS vruntime explains the fairness intuition, but current selection should be described with EEVDF: the scheduler tracks lag relative to ideal service, lets entities with lag >= 0 compete, and selects the earliest virtual deadline. Nice and cpu.weight alter long-run relative share; cpu.max sets a hard periodic ceiling; affinity and cpusets restrict eligible CPUs. None of those mechanisms automatically guarantees a 45ms p99.

I would run mpstat -P ALL, then read cpuset.cpus.effective, cpu.weight, cpu.max, cpu.stat, and cpu.pressure for the API cgroup. If nrthrottled and throttledusec grow throughout the slow interval, quota throttling is directly observed; idle host capacity does not change that conclusion. If the work is confined to CPUs 2 and 3 and those CPUs are full, I would repair the local CPU set first.

With neither throttling nor pinning anomalies, I would take two readings of /proc/12345/schedstat for a representative TID and calculate the run-queue-wait delta. Under a reproducible load, I would briefly capture perf sched record and inspect timehist for runnable-to-running delay, the waker, and the CPU. Low run-queue wait combined with extended sleep would redirect the investigation to locks, I/O, application queues, and downstream traces.

Suppose the evidence shows the API cgroup is configured as 20000 100000, with throttling rising in lockstep with p99. I would correct the budget using measured capacity and limit offline concurrency. I would not simply move the API to SCHED_FIFO. The retest would use the same traffic and batch job and require API p99, scheduling delay, throttling, and CPU pressure to fall together while offline throughput remains acceptable and no task starves.”

Common Mistakes

• Treating 78% host CPU as proof there is no CPU problem → an average hides cgroup quotas and local CPU sets → inspect per-CPU, cgroup, and thread views together.
• Calling all off-CPU time scheduling delay → a sleeping thread is not runnable yet → use states, stacks, and traces to establish the wake-up point first.
• Reciting only “CFS picks the smallest vruntime” → the fair class began transitioning to EEVDF in Linux 6.6 → explain eligible lag and the earliest virtual deadline.
• Assuming cpu.weight=200 reserves two CPUs → weight expresses relative share among active siblings under contention → design capacity separately and use cpu.max for a hard ceiling.
• Assuming nice directly orders all containers → task groups first compete at the cgroup hierarchy → inspect group weights before deciding whether to change thread nice.
• Inferring scheduler latency from many context switches → normal I/O and concurrency can also create switches → measure schedstat run-queue wait and perf sched delay.
• Using SCHED_FIFO as a quick API fix → a high-priority real-time thread can starve normal work and amplify lock risk → consider real-time policy only for real deadlines, bounded runtime, and a complete safety analysis.
• Raising API weight without checking quota → the cgroup still throttles after reaching cpu.max → prove throttling with cpu.stat and correct the limit.
• Checking only API p99 → a change may starve the batch job or degrade another tenant → verify fairness, throughput, pressure, and errors too.

Follow-Up Questions and Responses

Follow-Up 1: Why Can a cgroup Be Throttled While the Host Still Has Idle CPUs?

cpu.max limits CPU bandwidth consumed by that cgroup during a period. After the budget is exhausted, the group waits for the next period even if CPUs unused by other groups are idle. Inspect cpu.max and compare deltas in nrthrottled and throttledusec from cpu.stat over the same interval. Those values answer whether the hard limit fired more directly than aggregate host utilization.

Follow-Up 2: Why Might Renicing the API Thread Have Little Effect?

Linux applies nice per thread, but with group scheduling, different cgroups first compete as scheduling entities using group weights. Renicing mainly changes a thread's share within its group. It cannot remove a parent cpu.max, and it may not alter the ratio between the API and offline groups. Map the cgroup hierarchy and weights before choosing thread nice or group cpu.weight.

Follow-Up 3: Would Moving the API to SCHED_FIFO Fix p99?

It may shorten the target thread's wait, but it introduces greater risk. A continuously runnable FIFO thread can suppress all normal fair-class work. If it spins, holds a lock, or depends on a normal thread that it starves, the system can stop making progress. Evaluate a real-time policy only when the work has a genuine deadline, runtime is strictly bounded, priority inversion is handled, and fallback protection exists.

Follow-Up 4: Does EEVDF Guarantee CPU Service Within One Slice?

No. A virtual deadline orders eligible scheduling entities, and a slice expresses a latency preference. Available CPUs, entity count, weights, quotas, affinity, and higher scheduling classes still affect real waiting time. EEVDF supplies a scheduler mechanism for balancing fairness and latency; it is not a business p99 SLA.

Follow-Up 5: How Do You Separate Mutex Wait from Run-Queue Wait?

A thread waiting for a mutex normally sleeps and becomes runnable after the lock is released. Run-queue wait starts after it is runnable. Align application or eBPF off-CPU stacks and futex or lock metrics with /proc/12345/schedstat and perf sched timehist. Extensive futex sleep with low run-queue wait points to a lock; extensive runnable-to-running delay points to scheduling or resource control.

Follow-Up 6: When Can CPU Pinning Actually Help?

Well-designed isolation can reduce migrations, cache disruption, and noisy neighbors, for example by reserving capacity-tested CPUs for latency-sensitive threads. The CPU set must be large enough, interrupts and background work need governance, and per-CPU queues require monitoring. Bad pinning creates local congestion while host capacity remains idle, so verify the result with per-CPU utilization and scheduling delay.