Linux Interview: Why Is Load Average High When CPU Usage Is Low, and How Do You Diagnose It?

Prompt and Applicable Roles

A Linux server with 16 logical CPUs has become slow. uptime reports load averages of 48 / 36 / 18, while mpstat still reports 72% aggregate CPU idle. Decide whether this proves CPU overload and explain how to identify the resource tasks are actually waiting for.

The answer must explain runnable and uninterruptible tasks in Linux load accounting, interpret the 1-, 5-, and 15-minute values correctly, and then use task states, Pressure Stall Information (PSI), and subsystem metrics to distinguish these cases:

• the CPU run queue is genuinely congested;
• local storage, a network filesystem, a driver, or another kernel wait is keeping tasks in state D;
• memory reclaim or swapping is creating indirect waits;
• host-level metrics and container- or service-level metrics cover different scopes.

This question fits SRE, DevOps, infrastructure, systems, and backend interviews. The values 16, 48, 36, 18, and 72% are fictional exercise inputs, not universal alert thresholds. The scenario assumes the metrics came from the same host and time window. If different collectors or cgroups produced them, align scope before interpreting them.

What the Interviewer Is Testing

First, the candidate must know that load average is not CPU utilization. Linux global load is an exponentially decaying average of runnable tasks plus tasks in uninterruptible sleep. Only the first group directly expresses CPU demand; the second can raise load while CPUs remain idle.

Second, the three numbers need a precise interpretation. They are not simple arithmetic averages of samples in the last 1, 5, and 15 minutes. They are exponentially decaying values with those time constants. 48 > 36 > 18 supports the directional conclusion that load has risen recently, but it cannot reconstruct an exact queue length at a past minute or prove a root cause.

Third, a strong diagnosis moves from a total to its composition and wait sites. It compares vmstat fields r and b, counts threads in R and D, and then uses wchan, kernel stacks, PSI, and storage or network-filesystem metrics to identify a resource. Merely listing top and iostat does not explain how evidence changes the conclusion.

Fourth, the candidate should handle counterexamples. Tools often label D as disk sleep, but an uninterruptible wait is not limited to a local disk. NFS, filesystems, device drivers, and some kernel-resource waits can also appear. %iowait is CPU-time accounting, so low iowait does not rule out blocked tasks.

Finally, the response needs safe mitigation and a verification loop. Adding CPUs, restarting, killing processes, or upgrading storage is appropriate only for particular evidence. After a fix, R/D task counts, PSI, subsystem latency, and business tail latency should recover together. Waiting only for load average to decay is insufficient.

Questions to Clarify Before Answering

• Do the metrics cover the same scope? Load average is generally host-wide, while application CPU may describe only one container or process. Compare host, cgroup, and service metrics first; otherwise “high load, low CPU” may be a scope mismatch.
• Is CPU idle an aggregate or per-CPU value? One CPU, NUMA node, or affinity-constrained workload may be queueing while the other CPUs are idle. Inspect mpstat -P ALL, affinity, and the cgroup CPU quota.
• Which requests slowed down, and when? Align the symptom with deployments, traffic, backups, storage latency, mount changes, and memory reclaim so the host signal can be tied to user impact.
• Do R or D tasks dominate? Many R tasks with busy CPUs support CPU contention. Many D tasks with idle CPUs support an uninterruptible wait. Both can coexist, so group by thread and workload.
• What storage path is involved? Local NVMe, cloud block storage, FUSE, NFS, and a remote database require different evidence. iostat cannot see an ordinary network RPC, and NFS needs mount and network-filesystem metrics.
• Is there memory or virtualization pressure? Swap-in/out, direct reclaim, major faults, hypervisor steal time, and cgroup limits change the investigation order.
• Can task stacks be read safely? /proc/12345/stack, some wchan values, and process I/O data may require extra permission. Start with low-overhead sampling and use only the minimum production privilege needed.

30-Second Answer Framework

“Load average is not a CPU percentage. Linux counts runnable tasks and tasks in uninterruptible sleep, then computes exponentially decaying averages with 1-, 5-, and 15-minute time constants. A load of 48 on 16 logical CPUs says many tasks are active or waiting uninterruptibly. With 72% CPU idle, I cannot conclude that the CPUs are saturated.

I would first align metric scope and check per-CPU utilization. Then I would use vmstat to compare runnable tasks in r with blocked tasks in b, and count threads in R and D. Many R tasks, high CPU PSI, and busy CPUs indicate CPU queueing. Many D tasks with high I/O PSI lead me to wchan, task stacks, pidstat, iostat, NFS metrics, and kernel logs. If memory PSI, swapping, and major faults rise together, I investigate memory pressure. After the fix, I verify business p99, R/D counts, PSI, and subsystem latency instead of waiting only for the load number to fall.”

Step-by-Step Deep Dive

Step 1: Split Load Average into Its Two Sources

The core input to Linux global load accounting can be summarized as:

nrrunning + nruninterruptible

nrrunning includes tasks currently executing on a CPU and tasks that are runnable but waiting to be scheduled. nruninterruptible represents tasks in uninterruptible sleep. The first three fields of /proc/loadavg are the 1-, 5-, and 15-minute load values for this active-task population. The fourth field looks like 7/1280: the first number is the currently runnable scheduling entities, and the second is all scheduling entities currently present. The fifth field is the PID most recently created.

Therefore, “load 48” does not mean “300% CPU utilization,” and it does not mean exactly 48 processes are queueing at this instant. It is a smoothed task-count signal. Dividing load by the number of logical CPUs is only a rough saturation hint for a CPU-bound workload. Once D-state tasks contribute materially, that ratio no longer describes the CPU queue.

The kernel updates an exponentially decaying average at fixed intervals. New samples have more weight, while older samples decay. Thus 48 / 36 / 18 indicates more active tasks recently than earlier. The 15-minute value can remain high after the cause disappears, so recovery decisions should prioritize current task states, PSI, and user-facing metrics.

Step 2: Align Scope, Per-CPU Utilization, and the Current Queue

Start with low-overhead, timestamped observations:

nproc
uptime
cat /proc/loadavg
vmstat 1 10
mpstat -P ALL 1 10

The first vmstat report is generally an average since boot, so use the later samples for the current incident. Important fields include:

• r: runnable tasks; a sustained value well above available CPUs together with busy CPUs supports CPU queueing;
• b: tasks in uninterruptible sleep; a sustained increase directs the investigation toward a waited-on resource;
• us, sy, id, wa, and st: user, system, idle, I/O-wait, and hypervisor steal time;
• si and so: swap-in and swap-out activity, to be interpreted with memory PSI and faults.

Seventy-two percent aggregate idle can still hide a hot CPU. A serial thread pinned to CPU 3 can saturate that CPU while the other 15 remain mostly idle. Per-CPU utilization, task affinity, container CPU quotas, and throttling distinguish this from a need for more CPUs across the whole host.

Step 3: Count R and D by Thread and Locate the Wait Site

Multiple threads within one process contribute to load, so use a thread-level view:

ps -eLo state,pid,tid,ppid,wchan:32,comm --sort=state
ps -eLo state= | sort | uniq -c

R means running or runnable. D means uninterruptible sleep. A signal normally cannot take effect until the task leaves that wait, so repeatedly issuing kill -9 neither releases the kernel resource immediately nor preserves useful evidence.

For a cluster of D-state threads, group by command, parent process, cgroup, and wchan. wchan shows where a thread is sleeping in the kernel and can narrow the search to block I/O, NFS, a filesystem, or a driver path. When permissions allow, inspect a representative task’s kernel stack:

cat /proc/12345/stack
cat /proc/12345/wchan

One function name is not a root cause. Many tasks clustered at the same wait site, aligned in time with subsystem latency and user impact, form stronger evidence. D state also does not prove that an application intentionally generated excessive I/O. A failed device, unreachable remote mount, or stuck kernel path can accumulate many waiting threads from a small request rate.

Step 4: Use PSI to Identify Which Resource Stops Progress

PSI quantifies time lost because of CPU, memory, or I/O contention:

for resource in cpu io memory; do
  echo "[$resource]"
  cat /proc/pressure/"$resource"
done

some is the share of time when at least some tasks are stalled on a resource. full is the share when all non-idle tasks are stalled simultaneously. System-level CPU full is retained as zero and should not be used to infer CPU saturation. avg10, avg60, and avg300 describe the recent 10-, 60-, and 300-second trends; total is cumulative stall time in microseconds.

PSI and load average answer different questions. Load average describes how many tasks are active or waiting uninterruptibly. PSI describes how much time a workload cannot make progress. Combining them prevents the reflex to “add CPUs because load is high.” With cgroup v2, also read cpu.pressure, io.pressure, and memory.pressure in the target cgroup to separate service pressure from host noise.

Use this matrix to organize evidence:

| Observation combination | First hypothesis | Next evidence | |---|---|---| | High CPU busy, high r, high CPU PSI | CPU run-queue contention | Per-CPU hotspots, CPU profile, affinity, and quota | | High CPU idle, many b/D tasks, high I/O PSI | I/O or another uninterruptible kernel wait | wchan/stacks, device or NFS latency, kernel logs | | High memory PSI, rising si/so or major faults | Reclaim or swapping pressure | Cgroup memory, working set, reclaim, and storage reads | | High host load, low target-cgroup PSI | Scope mismatch or another tenant | Attribute tasks and resources by cgroup/process |

This matrix is a starting point, not an automatic root-cause detector. CPU, memory, and I/O pressure can form a feedback loop. Memory reclaim can trigger file reads, for example, which can then put threads into D state.

Step 5: Enter the Relevant Subsystem Instead of Calling Every Wait “Slow Disk”

If the evidence points to block devices, examine throughput, latency, queueing, and errors:

pidstat -d -p ALL 1 10
iostat -xz 1 10
dmesg -T | tail -200

pidstat helps identify processes issuing I/O, while iostat reports device and partition activity. Interpret every field against the device type and its baseline. A %util value, for example, does not have one universal saturation threshold across storage architectures. Timeouts, resets, filesystem errors, or a detached device in kernel logs are closer to a failure cause than queue depth alone.

If wchan or the mount points indicate NFS, FUSE, or network storage, inspect mount state, nfsiostat, client retransmissions, network latency, and server health. Ordinary HTTP or database RPC waits are usually interruptible and may not enter Linux load average, so application traces, connection pools, and dependency latency still belong on the same timeline.

If memory PSI, swapping, or major faults rise, inspect cgroup memory events, anonymous and file working sets, direct reclaim, and the swap device. Faster storage may reduce the cost of swapping without fixing a working set that exceeds the memory budget.

Step 6: Mitigate from Evidence and Verify Causality

Mitigation should match the confirmed wait resource:

• CPU queue contention: shed admission, lower concurrency or batch priority, remove incorrect affinity, and add CPU only against a measured capacity model;
• storage or NFS failure: stop amplifying I/O, switch to a healthy replica or mount path, and repair the device, network, or server;
• memory pressure: bound the working set and concurrency, stop the thrashing workload, and add memory within a compatible capacity boundary;
• scope mismatch: isolate the noisy workload or correct cgroup quotas before deciding whether the target service needs scaling.

A restart may clear accumulated tasks, but it can also erase useful stacks and logs. If the underlying resource remains unavailable, new processes will block again. Before acting, preserve representative PIDs, task states, wchan, PSI, and subsystem snapshots, and state the expected metric change.

Acceptance should include recovered business p95/p99 and error rate, R/D counts back at baseline, falling pressure in the relevant PSI resource, recovered device/NFS/memory/CPU metrics, and safely drained backlog. Because load average decays, it is one recovery signal rather than the only incident-closure gate.

High-Quality Sample Answer

“I would not call this CPU overload just because load is 48 on 16 CPUs. Linux load average smooths the count of tasks running or waiting for CPU plus tasks in uninterruptible sleep. With 72% CPU idle, the load may contain many D-state waits, or the aggregate may hide a per-CPU, quota, or metric-scope problem.

48 / 36 / 18 are exponentially decaying averages. Directionally, they say active tasks increased recently. They are not arithmetic averages over three independent windows and do not identify the cause. I would first ensure load, CPU, and business latency came from the same host and time window, then inspect mpstat -P ALL for a hot CPU or steal time.

Next I would compare r and b with vmstat 1. Sustained high r, busy CPUs, and high CPU PSI would lead to a CPU profile, affinity, and cgroup throttling. High b and many D-state threads while CPUs remain idle would lead me to group threads by command and wchan, inspect /proc/12345/stack for representative tasks, and align the result with I/O PSI.

Suppose many threads are waiting in an NFS-related path, I/O PSI rises, and local block-device iostat remains normal. I would investigate mounts, client retransmissions, network latency, and the NFS server instead of upgrading the local cloud disk. Mitigation may be stopping the relevant batch job, switching reads to a healthy replica, or removing the affected node. Repeatedly killing D-state processes usually will not take effect immediately.

After repair, I would verify business tail latency and errors, R/D thread counts, host and cgroup PSI, NFS latency, and backlog. The 1-, 5-, and 15-minute load values decay over time, so a still-high 15-minute value should not trigger more risky changes when current waits and user outcomes are already stable.”

Common Mistakes

• Treating load average as CPU utilization → it counts tasks and includes uninterruptible sleep → split r from b and R from D before diagnosing CPU contention.
• Declaring overload whenever load exceeds CPU count → that rule is only a rough hint for CPU-bound demand → combine CPU busy, CPU PSI, and the runnable queue.
• Treating 1, 5, and 15 as simple window averages → Linux uses exponential decay and retains older samples → use the three values for direction and confirm the present with current metrics.
• Attributing every D task to a local disk → NFS, filesystems, drivers, and other kernel waits can be uninterruptible → trace wchan, stacks, and the matching subsystem.
• Ruling out I/O because iowait is low → iowait is CPU-time accounting, not a blocked-task count → inspect D tasks, I/O PSI, devices, and remote-storage latency.
• Looking only at host-average CPU → a hot CPU, affinity, quotas, or another cgroup can be hidden by the average → compare per-CPU, host, and target-cgroup metrics.
• Repeatedly issuing kill -9 to stuck tasks → the signal waits for the uninterruptible operation to return and evidence may be lost → preserve evidence and repair the waited-on resource.
• Calling one wchan value the root cause → it identifies the current kernel sleep site only → require clustering, time correlation, and subsystem evidence.
• Waiting for load average to reach zero before declaring recovery → decay lags and healthy systems do not require zero load → accept recovery from user outcomes, current tasks, PSI, and resource metrics.

Follow-Up Questions and Responses

Follow-up 1: Why can %iowait also be low when CPU idle is high?

%iowait is the fraction of CPU time that is idle while the system accounts for outstanding I/O. It is CPU-time accounting. Tasks can be blocked in NFS, a driver, or another uninterruptible kernel wait while CPUs run other work or remain ordinary idle. Aggregation across many CPUs can also dilute a local symptom. Use D-state counts, I/O PSI, wchan, and subsystem latency; low iowait alone cannot exclude I/O or a kernel wait.

Follow-up 2: Does load average observed inside a container represent that container?

Do not assume it does. The visible load and /proc scope depend on the host, PID namespace, runtime, and monitoring implementation, while application CPU may be cgroup-scoped. Read host and target-cgroup CPU, I/O, and memory PSI plus quota events, and attribute threads to cgroups. If host load is high but the target cgroup has no pressure, scaling that service may not help.

Follow-up 3: With load 48 and 72% CPU idle, must there be many D-state tasks?

No. The metrics may be unsynchronized, aggregate idle may hide a hot CPU, tasks may be constrained by affinity or a cgroup quota, or load may have just fallen while the exponential averages still retain the peak. Check the current runnable count in /proc/loadavg, vmstat r/b, per-CPU utilization, R/D thread counts, and collection timestamps to determine the composition.

Follow-up 4: Why does a D-state process sometimes survive kill -9?

Uninterruptible sleep lets some kernel operations pass a critical phase without ordinary signal interruption. SIGKILL can remain pending, but the task generally must return from the wait and reach a path that can exit before the process disappears. Repairing the device, mount, network, or driver wait is often more important than repeating the signal. A host restart, if necessary, must be evaluated against data integrity and service redundancy.

Follow-up 5: How should load-average alerts be configured?

Do not copy a fixed “load greater than CPU count” threshold. For CPU-bound services, combine load, runnable queue, CPU PSI, utilization, and cgroup throttling. For I/O-heavy services, combine I/O PSI, D states, device or remote-storage latency, and the business SLO. Set thresholds from the normal baseline, duration, scaling time, and user impact, and label every metric’s scope.