Prompt and scope
Normal byte utilization does not prove that a filesystem can create another file. ext4 and similar filesystems manage data blocks and inodes; millions of small files, cache fragments, mail queues, or container layers can exhaust inodes first. The task is to find the cause while the service is running, without deleting an unknown directory or hiding evidence with a restart.
Public Linux and DevOps interview material often uses “disk full” as a diagnostic scenario. The df manual and Linux kernel ext4 documentation define the boundaries between blocks, inodes, and directory entries, so a strong answer derives actions from evidence instead of reciting cleanup commands.
What the interviewer is testing
- Distinguish byte blocks, inodes, user/project quotas, and container writable layers.
- Confirm the affected mount, time window, and write path before taking action.
- Find small-file hotspots, hidden mounts, rotation gaps, and files deleted while still open.
- Choose reversible and auditable recovery steps instead of
rm -rfor a blind restart. - Monitor inode use, file-count growth, and directory hotspots as capacity signals.
Questions to clarify first
- Which path and mount does the failing process write to? Is it the host, a container, or a temporary filesystem?
- What do
df -handdf -ireport? Are user or project quotas involved? - Is the failure creating a file, extending one, or writing to overlay, tmpfs, or a network filesystem?
- Is a deploy, log rotation, backup, or batch job running? Do retention rules restrict deletion?
- Can the service be briefly throttled, and is there a rollback or health-check window?
A 30-second answer
“I would pin down the failing process, mount, and time window, then compare df -h with df -i. If inode use is near 100%, I would locate file-count hotspots and inspect container writable layers, log rotation, and quotas. If inodes are healthy, I would check blocks, reserved space, quotas, and deleted-open files. Recovery would start with a controlled rotation, compression, cleanup of confirmed temporary data, or expansion while preserving evidence. Finally I would alert on bytes, inodes, file counts, growth rate, and time-to-remediate rather than one disk percentage.”
Deep-dive answer
Step 1: Establish the fault boundary
Preserve application and kernel logs, the failing path, and mount information. Determine whether one service, one container, or the host cannot create files. The same error text can represent inode exhaustion, blocks, quotas, or a read-only filesystem.
Step 2: Check blocks and inodes together
df -hT /
df -iT /
findmnt -T /var/lib/appdf -h reports data blocks and df -i reports inodes. Read both for the mount that owns the failing path. If inode use is near 100% while bytes remain, prioritize small-file analysis; if not, inspect blocks, quotas, read-only state, and container limits.
Step 3: Locate directory hotspots by count
Count directory entries before reading file contents to avoid unnecessary I/O. Narrow the search by directory and descend into the fastest-growing branch. Bound find to known paths, exclude other mounts, and give the traversal a resource budget during peak traffic. A directory full of tiny files can consume inodes while using little byte space.
Step 4: Separate files from mount boundaries
Overlay filesystems, bind mounts, tmpfs, and log volumes can make the host path differ from the layer where the process writes. Cross-check the process working directory, container configuration, and findmnt -T. Do not delete container-layer files from the host; use the runtime, volume policy, or application cleanup path.
Step 5: Inspect rotation, caches, and deleted-open files
Rotation may rename a log without making the process reopen it, and caches may create unbounded small files. lsof +L1 finds files with zero directory links that a process still holds. Releasing them normally requires the owner to close or reopen the file safely. A restart is not the default because it destroys evidence and may reproduce the write storm.
Step 6: Choose a recovery action
Order actions by risk: throttle noncritical producers, rotate or compress confirmed logs, clean caches covered by retention policy, then expand or migrate. Record each path, size, file count, owner, and rollback. Before deletion, verify that the data is not current configuration, queue state, database material, or audit evidence.
Step 7: Verify recovery and side effects
Run df -hT and df -iT again, then perform a real temporary-file create, log write, and critical request. Confirm inode use, error rate, and latency recover. For containers, verify that rebuilding or restarting does not immediately recreate the file-count spike.
Step 8: Build durable defenses
Monitor block and inode use, files per mount, directory growth, rotation delay, deleted-open files, and container-layer size. Set thresholds from growth rate and response time, not one universal 90% number. Put cleanup, expansion, rotation recovery, and verification into an executable runbook and rehearse it.
Trade-offs and boundaries
Cleanup versus expansion
Cleanup restores service quickly but may recur; expansion adds headroom without fixing the generator. Restore first, then use growth evidence to choose application changes, rotation, file granularity, or expansion.
Measurement accuracy versus online cost
A whole-disk find is precise but expensive. Directory counts and sampling suit continuous monitoring. During an incident, narrow the scope progressively instead of recursively reading the whole filesystem under I/O pressure.
Host versus container
Host metrics do not replace volume and overlay metrics. Every writable layer needs a quota, owner, and cleanup boundary; cross-layer deletion can create unpredictable image or volume behavior.
Failure drills and evolution
Failure: look only at df -h
Bytes can remain while inodes are zero. Include df -i in first-response diagnostics and alert history by mount.
Failure: rm -rf the largest directory
It may contain queues, evidence, or files being written. Confirm ownership, retention, open handles, and rollback, then clean in bounded batches.
Failure: restart instead of recovering
A restart may temporarily release deleted-open files while destroying evidence and hiding the generator. Let the owner close files safely and verify the next write.
Common mistakes and follow-ups
Mistake: inode use is about file size only
Inodes are primarily consumed by file count and filesystem format; zero-byte and tiny files still consume them.
Follow-up: why did deleting a file not free space?
The process still holds its file descriptor. The directory entry is gone, but blocks and the inode remain allocated until the descriptor closes.
Follow-up: how do you distinguish quotas?
Compare filesystem-wide metrics with user, project, and container quotas, then run a controlled create test as the same identity on the same path.
Follow-up: how do you validate log rotation?
Verify that the process opens the new file, closes the old handle, and that file count and inode use fall within budget; a new filename alone is not proof.
Follow-up: how do you prevent a small-file storm?
Batch records, partition by time, cap cache entries, set rotation limits, and monitor file-creation rate and directory-entry growth.
Follow-up: when does expansion not help?
If inode density is fixed and the new filesystem provides the same inode design, adding bytes does not solve inode exhaustion. Migrate, rebuild, or change file granularity.