Linux Interview: Why Do df and du Show Different Disk Usage?

Prompt and Applicable Context

At 02:00, an application starts receiving ENOSPC while writing under /var. The host reports:

$ df -B1 /var
Filesystem          1B-blocks         Used  Available Use% Mounted on
/dev/nvme0n1p3    214748364800 210453397504          0 100% /var

$ sudo du -x -B1 -s /var
126701535232  /var

$ df -i /var
Filesystem          Inodes   IUsed    IFree IUse% Mounted on
/dev/nvme0n1p3    20000000 4200000 15800000   21% /var

The rounded values are 200 GiB total, 196 GiB used by df, and 118 GiB reachable through du, leaving a 78 GiB used-space gap. The inode output rules out inode exhaustion as the immediate cause. The service is unprivileged, the host cannot be rebooted, and no file may be removed until its owner and purpose are known.

This is a Linux production-troubleshooting question. The strongest answer does not start by deleting the largest pathname. It first proves that both tools observed the same filesystem, namespace, time, units, and access scope; then it explains which allocation can exist outside the visible directory tree and chooses a recovery action whose blast radius is understood.

What the Interviewer Evaluates

The first signal is a correct measurement model. df asks a mounted filesystem for its overall block statistics. du walks named files and directories and estimates the blocks represented by entries it can reach. They are measuring related but different sets. Large allocation held by an unlinked file or hidden below an over-mounted directory remains in the filesystem total even though the current walk cannot name it.

The second signal is scope control. Comparing df /var with an unrestricted du /var, running one command inside a container, ignoring permission errors, or comparing samples taken during rapid writes can manufacture a discrepancy. A candidate should align the mount target, mount namespace, filesystem boundary, byte units, privileges, and observation window before assigning a cause.

The third signal is safe incident response. lsof +aL1 /var can identify open files with link count below one on that filesystem, but its output is evidence rather than permission to kill a process or truncate a descriptor. A strong answer identifies the service owner, confirms the file descriptor and device, prefers an application-supported log reopen or graceful restart, and verifies both disk reclamation and application health.

The fourth signal is separating causes that look similar. ext4 reserved blocks reduce space available to unprivileged writers and can make Size - Used exceed Avail; they do not explain why 78 GiB is counted as used by df but absent from a same-filesystem du walk. Inode exhaustion can also return ENOSPC, but the prompt's 21% inode usage rejects that branch. Copy-on-write snapshots, compression, and quotas require filesystem-specific tools rather than an ext4 command copied blindly.

The final signal is closure. The answer should establish a before-and-after evidence set, explain the reclaimed byte range, confirm that the writer reopened the intended pathname, check for recurring deleted-open growth, and improve log rotation or mount-change monitoring. A lower percentage in df alone does not prove that the application is healthy or that data was preserved.

Questions to Clarify Before Answering

• Are both commands running in the same mount namespace? If one runs on the host and the other in a

container or service namespace, /var may resolve to different mounts. The investigation must enter the affected process's namespace or deliberately stay on the host.

• Which exact path is failing and which filesystem contains it? findmnt -T maps an arbitrary path

to its mount. A nested or bind mount changes what df and du should be compared against.

• Were the samples taken together with the same units and privileges? Fast-growing logs can change

between commands, human-readable rounding hides smaller differences, and unreadable directories make an unprivileged du undercount.

• What filesystem and storage layers are involved? ext4, XFS, Btrfs, ZFS, overlay filesystems, LVM

snapshots, and cloud volume snapshots expose different accounting and recovery controls.

• Is the shortage in data blocks, inodes, or a quota? df -i, quota tools, and the application error

distinguish branches. Deleting one large file does not cure inode exhaustion, and free global blocks do not override a user or project quota.

• Which recovery actions are operationally allowed? A log-reopen signal, graceful restart, failover,

or short maintenance window has different availability risk. Emergency descriptor truncation needs explicit ownership and rollback planning.

• How much headroom must be restored, and by when? The target determines whether to contain writes,

expand the volume, fail over traffic, or reclaim known allocation first. It does not justify deleting unknown data.

30-Second Answer Framework

“I would first freeze destructive actions and compare the same path, filesystem, mount namespace, time, privileges, and byte units using findmnt, df -B1, and sudo du -x -B1. With inode use at 21%, I would quantify the 78 GiB block gap and check lsof +aL1 /var for unlinked files that processes still hold open. If one explains the gap, I would verify its PID, descriptor, device, and service owner, then use the application's log-reopen path or a graceful restart and confirm the space returns. If not, I would inspect over-mounted directories, permission errors, and filesystem-specific metadata or snapshots. ext4 reserved blocks explain zero unprivileged availability, not the 78 GiB used-space gap. I would close by repeating the measurements and checking application writes, logs, and recurrence.”

Step-by-Step Deep Dive

Start by making the comparison reproducible. Capture the failing path, current namespace, mount source, filesystem type, block statistics, inode statistics, and the du total close together:

readlink -f /var
findmnt -T /var -o SOURCE,FSTYPE,OPTIONS,TARGET
df -B1 /var
df -i /var
sudo du -x -B1 -s /var

-x keeps du on the filesystem containing /var, so a nested filesystem is not added to the total. -B1 removes block-unit ambiguity. Run du with enough privilege and read its diagnostics; suppressing permission errors without inspecting them turns an access problem into a false storage theory. Capture commands from the same mount namespace. For a containerized writer, compare the host view with a deliberate nsenter view instead of assuming that identical path strings name identical objects.

The observed gap is a diagnostic quantity, not a file to search for:

same_filesystem_gap = df_used_blocks - du_reachable_allocated_blocks
                    = 196 GiB - 118 GiB
                    = 78 GiB

This equation is approximate because filesystem metadata, allocation granularity, concurrent writes, copy-on-write sharing, compression, and tool semantics do not form a perfect partition. It is still useful: a stable 78 GiB gap is too large to dismiss as output rounding. Search for causes that allocate blocks outside the visible, reachable tree before looking for another ordinary large pathname.

The highest-value check is an open file whose final directory link has been removed:

sudo lsof +aL1 /var

# After selecting a candidate from lsof, verify the live descriptor.
sudo readlink /proc/2481/fd/7
sudo stat -Lc 'device=%d inode=%i size=%s blocks=%b block_size=%B' /proc/2481/fd/7

Linux unlink removes a name. If it was the final link but a process still has the file open, the file and its blocks remain until the last referring descriptor closes. du cannot reach that inode through the directory tree; df still counts its allocated blocks. A common incident path is a logger that keeps writing to a file after rotation deleted or renamed it incorrectly.

Do not treat the apparent SIZE/OFF value as an exact reclaim forecast for every sparse or special file. Confirm that the descriptor is a regular file on the target device, that its link count is zero, which process owns it, whether it is still growing, and whether the application has a documented reopen signal. Correlate several large descriptors if one does not account for most of the 78 GiB.

The preferred recovery order is application-specific reopen, graceful reload, graceful restart or failover, then emergency intervention. A supported log-reopen action closes the stale descriptor and opens the current pathname without terminating the whole host. A controlled service restart also releases it, but must respect replicas, readiness, and in-flight work. Sending SIGKILL first discards the application's cleanup path. Removing more names does nothing to a file that is already unlinked.

Writing through /proc/PID/fd/FD can reclaim space without stopping the process, but it is a last resort. The writer may retain an old offset, create a sparse hole on its next write, corrupt an application format, or fail internal invariants. Use it only after the service owner confirms the descriptor and write semantics, traffic is contained, evidence is preserved, and a recovery plan exists. The generic runbook should never advertise descriptor truncation as the default fix.

If lsof cannot explain the gap, inspect mount topology. A filesystem mounted over a nonempty directory hides the underlying directory entries while their blocks remain allocated on the parent filesystem. findmnt reveals nested, bind, and over-mounted paths. During an approved maintenance action, a non-recursive bind of /var to an empty target outside /var can expose the underlying parent view without unmounting the production child mount:

sudo mkdir -p /mnt/var-underlay
sudo mount --bind /var /mnt/var-underlay
sudo du -x -B1 -s /mnt/var-underlay
sudo umount /mnt/var-underlay

Validate this approach against the actual mount tree first; namespace propagation and platform policy may change its effect. Do not unmount a busy production filesystem just to inspect it. If hidden files are found, identify their owner and retention requirements before moving or deleting them.

Next separate availability accounting from the used-space gap. On ext4, privileged-process reserved blocks can make raw free blocks unavailable to the application. Inspect rather than modify:

source=$(findmnt -n -o SOURCE -T /var)
sudo tune2fs -l "$source" | grep -E 'Block count|Reserved block count|Block size'
sudo du --inodes -x -d1 /var | sort -n

The prompt has 4 GiB between total and used but zero shown as available, which is consistent with some free blocks being unavailable to an unprivileged service. That explains the immediate write failure, not why df marks 78 GiB more as used than du can reach. Lowering the reserve during an incident can remove the headroom intended for privileged daemons and increase fragmentation risk; it requires a capacity decision, not a reflexive tune2fs -m 0.

Inode exhaustion is another independent ENOSPC path. Here df -i shows 21%, so it is not the trigger. If it were 100%, use du --inodes -x to locate high-file-count trees and remove only data covered by a retention policy. Block reclamation and inode reclamation are separate objectives.

Filesystem-specific accounting comes last. GNU du documents that copy-on-write sharing, compression, backup blocks, and network filesystems can make its estimate diverge from device consumption. On Btrfs or ZFS, inspect subvolumes, snapshots, quotas, and exclusive versus referenced bytes with that filesystem's tools. On overlay storage, inspect layers from the namespace and runtime that own them. Do not run ext4 tools against an XFS or Btrfs source.

If open files, hidden data, access errors, concurrent growth, reserved availability, and filesystem features still do not explain the evidence, inspect kernel logs and filesystem health. A repair tool is not an online diagnostic shortcut: preserve evidence, use the filesystem's documented procedure, and schedule any check that requires an unmounted volume.

Close the incident with the same evidence set. Repeat df -B1, df -i, and privileged du -x -B1; confirm that reclaimed bytes match the closed or removed allocation within expected overhead; verify a fresh application write; confirm the service now writes to the intended named file; and check latency, errors, replicas, and data integrity. Then alert on block and inode headroom, deleted-open growth, log-rotation failures, mount topology changes, and unexpected snapshot growth.

High-Quality Sample Answer

“The 78 GiB difference is plausible because df and du have different accounting scopes. df gets filesystem-wide block statistics, while du walks named entries it can reach. I would first prove this is a real gap by mapping /var with findmnt, running both tools in the affected service's mount namespace, using byte units, du -x, root privileges, and near-simultaneous samples. Inodes are only 21%, so I would set that branch aside.

My first hypothesis is deleted-open files. I would run lsof +aL1 /var, then verify each large result's PID, file descriptor, device, inode, owner, and growth. If a logger still owns roughly the missing space, I would use its supported reopen signal or a controlled graceful restart. I would avoid killing the process or truncating /proc until the service owner confirms the blast radius. After the descriptor closes, I would check that df reclaims the expected range and that the application writes to the new named log.

If that does not explain the gap, I would inspect findmnt output for a directory whose underlying files are hidden by another mount, review du permission errors, and then use filesystem-specific tools for snapshots, copy-on-write allocation, quotas, and metadata. Because this volume is ext4, I would inspect reserved blocks. They can explain why the service has zero available bytes even though total minus used is 4 GiB, but they cannot explain the 78 GiB used-space gap.

I would finish by repeating the original measurements, testing the failed write path, checking service health and data integrity, and recording exactly which allocation was released. The prevention work is to fix log rotation or mount lifecycle, alert on both blocks and inodes, and monitor unlinked-open growth before the filesystem reaches the emergency reserve.”

Common Mistakes

• Deleting the largest visible file immediately → it may be required data and may not explain an

invisible allocation → align the measurement scope and identify the owner before changing data.

• Comparing df /var with du / → nested filesystems and different roots invalidate the subtraction

→ map the failing path and use du -x on the same filesystem.

• Ignoring du permission errors → unreachable named files make the total artificially low → **run

with sufficient privilege and review every diagnostic.**

• Running commands in different mount namespaces → the same pathname can resolve to different

filesystems → collect both measurements in the affected process's namespace.

• Calling the 78 GiB gap “reserved blocks” → ext4 reserve affects unprivileged availability, while

the gap compares used blocks with reachable allocation → calculate each difference separately.

• Using rm on an already deleted-open file → its final name is already gone and the live descriptor

keeps the inode allocated → make the owning process close or safely reopen the descriptor.

• Sending kill -9 as the first fix → abrupt termination can lose work and bypass cleanup → **prefer

the supported reopen, reload, graceful restart, or failover path.**

• Truncating /proc/PID/fd/FD by habit → retained offsets and file-format assumptions can create new

corruption → reserve truncation for an approved emergency with verified semantics.

• Unmounting a busy child mount to reveal hidden files → dependent services can fail immediately →

inspect topology and use an approved alternate view or maintenance window.

• Treating inode usage as part of the byte gap → inode exhaustion is a separate ENOSPC mechanism →

check df -i and diagnose high file counts independently.

• Applying ext4 commands to every filesystem → snapshot and allocation semantics differ across XFS,

Btrfs, ZFS, and overlay layers → identify FSTYPE and use its supported tooling.

• Stopping after df drops → the writer may still be unhealthy or logging to the wrong target →

verify application writes, named files, data integrity, and recurrence signals.

Follow-Up Questions and Responses

Follow-up 1: What if lsof is unavailable or does not see the process?

Inspect /proc/PID/fd in the same PID and mount namespaces as the writer, looking for symlinks ending in (deleted), then verify device, inode, link count, and process ownership. Host lsof may miss a containerized process if permissions, PID namespaces, or security policy hide it. Do not make /proc truncation the next automatic step; the recovery decision remains service-specific.

Follow-up 2: What if du is larger than df instead?

First check whether du crossed into nested filesystems; du -x removes that source. Then check hard link argument semantics, apparent-size options, concurrent deletion, and copy-on-write or compression accounting. du estimates the selected directory hierarchy, while df reports one filesystem, so the direction of the discrepancy changes which scope error is plausible.

Follow-up 3: What changes on Btrfs or ZFS?

Visible file sizes are insufficient because snapshots and copy-on-write sharing can retain blocks after a pathname is deleted. Use the filesystem's own space, subvolume or dataset, snapshot, and quota views; distinguish referenced from exclusive allocation; and delete snapshots only under an explicit retention policy. ext4 reserved-block reasoning and tune2fs do not transfer.

Follow-up 4: Can we set the ext4 reserved percentage to zero to recover immediately?

Only after a capacity and reliability review. The reserve lets privileged processes continue and can reduce fragmentation. It may restore unprivileged availability, but it does not remove used blocks or explain deleted-open allocation. Reclaim or expand known capacity first, then tune reserve according to volume role and operational policy.

Follow-up 5: How would you reproduce deleted-open behavior safely?

Use a disposable filesystem or test host: open a large test file with a process, unlink its pathname while the descriptor remains open, compare df and du, observe it with lsof +L1, then close the descriptor and confirm the blocks return. Do not manufacture the experiment on a production volume or reuse a real service's descriptor.

Follow-up 6: What should the long-term alert measure?

Alert on time-to-exhaustion and minimum headroom for both blocks and inodes, segmented by filesystem and namespace. Add a gauge or periodic inventory for deleted-open regular files, log-rotation failures, snapshot growth, and mount topology changes. The alert should link to a runbook that starts with scope alignment and requires owner approval before destructive recovery.