Operating Systems Interview: How Do Virtual Memory and Page Faults Work?

Question and Applicable Scenario

A Linux service uses mmap to map a 20 GiB read-only index at startup, then forks 8 worker processes. Monitoring shows:

• each process's virtual memory grows by about 20 GiB soon after the mapping is created;
• RSS does not grow by the same amount immediately, but rises as queries touch more index regions;
• the first requests after a cold start have higher latency and more major page faults;
• repeating the same queries is faster and produces far fewer major faults.

Explain how virtual addresses become physical addresses, how a TLB miss differs from a page fault, what minor and major page faults mean on Linux, and why summing the RSS of all 8 workers can overstate physical memory consumption. Finish with a method for verifying the bottleneck and reducing cold-start latency.

This question fits backend, infrastructure, systems software, SRE, and performance engineering interviews. The 20 GiB mapping, 8 workers, and 4 KiB page size used below are hypothetical exercise assumptions, not production measurements from the sources. The index is a read-only file mapping. Anonymous memory, writable private mappings, container memory limits, and real-time systems would change the investigation.

What the Interviewer Is Evaluating

The first signal is whether the candidate separates four layers. A virtual address space describes what a process may address. Page tables store translations and permissions. The TLB caches recent translations. Physical pages contain the data currently resident in RAM. A basic answer says that virtual memory can exceed physical memory; a strong answer walks through a load instruction, including a TLB hit, a page-table walk, a repairable fault, and an illegal access.

The second signal is recognizing that a TLB miss is not a page fault. If the translation is absent from the TLB but the page-table entry is valid and resident, the processor can complete the page-table walk and fill the TLB. A page fault occurs when the current page-table state cannot satisfy the access, such as a nonresident page, a first write that requires copy-on-write, or a permission violation.

The third signal is correct interpretation of Linux counters. A minor fault does not require disk I/O. The page may already be in the page cache but not mapped into this process, or the kernel may be allocating an anonymous page or completing copy-on-write. A major fault requires disk I/O. It can load swapped anonymous memory, but it can also read a file-backed mapping that is absent from the page cache. Therefore, “major fault means swap” is incorrect.

Finally, the interviewer wants an evidence loop. A strong answer aligns request latency, fault deltas, file-page residency, block-device I/O, RSS/PSS, and memory pressure in the same time window before choosing warm-up, a different index layout, kernel access hints, or a smaller working set.

Clarifying Questions Before Answering

• Is this file-backed or anonymous memory, and is the mapping MAPSHARED or MAPPRIVATE? Read-only file pages can be shared through the page cache. Private writable pages may become process-specific after copy-on-write.
• How large is the hot working set, and is access sequential or random? If 95% of requests touch only a 2 GiB hot region, warming all 20 GiB adds startup and memory pressure. Sequential scans and random lookups also need different read-ahead choices.
• Does mmap happen before or after fork? Both approaches can map the same file, but mapping before fork makes it easier for workers to inherit one configuration. Each process still has its own page tables and TLB state.
• Do major faults rise at the same time as block reads and tail latency? That correlation is required before blaming demand paging for most of the cold-start cost. CPU saturation, locks, remote storage, and index initialization may coexist.
• Is swap enabled, are there container or cgroup memory limits, and has reclaim occurred recently? File-backed mappings can generate major faults without swap. Memory pressure can evict recently loaded file pages and cause repeated faults.
• What is the readiness SLO? A service that must accept traffic within 10 seconds cannot blindly scan 20 GiB. A longer initialization budget may allow selected faults to be moved before readiness.

30-Second Answer Framework

“mmap creates a file-backed virtual range without reading all 20 GiB, so VIRT jumps while RSS grows on first touch. The CPU checks the TLB; a miss can be resolved by a page-table walk when the entry is valid and resident, while a page fault needs kernel repair. Linux minor faults require no disk I/O; major faults do, so cold file pages raise major faults and request latency until the page cache warms. Read-only pages can be shared by 8 workers, so summed RSS double-counts them; use PSS. Correlate fault deltas, RssFile/PSS, read I/O, and p99, then warm only hot pages or test madvise and MAP_POPULATE within the startup budget.”

This framework explains the observations, separates the two common confusions, and closes with measurement and a decision. A complete answer should also explain the fault path and the conditions under which each optimization fails.

Step-by-Step Deep Dive

Step 1: Separate Address Space, Mapping, and Resident Pages

Virtual memory gives every process an independent address space. A virtual address can be viewed as a virtual page number plus an offset. Page tables map virtual page numbers to physical page frames and store state or permissions such as present, writable, and executable. Multi-level page tables allocate lower levels only for address ranges that are used, avoiding a huge linear table for a sparse address space.

The immediate result of mapping a 20 GiB file is a 20 GiB virtual memory area. The mapping describes how that address range corresponds to the file; it does not require the kernel to read the whole file immediately. Consequently:

• VIRT or VmSize can rise by about 20 GiB at once;
• untouched file pages do not need to be in RAM;
• when a query first reads a page, the kernel may load it into the page cache and install a page-table mapping;
• RSS counts the resident portion, so it grows as the working set is touched.

Assuming 4 KiB pages, touching the entire 20 GiB mapping means touching:

20 × 2^30 ÷ 4096 = 5,242,880 pages.

That calculation shows why “warm everything” is not free. If only a small region is hot, scanning more than 5.24 million pages brings low-value data into memory and may evict other useful cache entries.

Step 2: Walk Through the TLB and Page Tables

When the CPU executes a load, store, or instruction fetch, it must translate a virtual address:

1. look up the virtual page number in the TLB;
2. on a TLB hit, obtain the physical frame and combine it with the offset;
3. on a TLB miss, let hardware or the operating system consult the page table;
4. if the entry is valid, permitted, and resident, cache the translation in the TLB and retry or continue;
5. if the current entry cannot satisfy the access, enter the page-fault path.

A TLB miss means “the translation cache missed.” A page fault means “the current page-table state cannot complete this access.” The former may add only a page-table walk. The latter enters the kernel to decide whether the access is legal and repairable. Confusing the two leads to incorrect claims about what huge pages, prefetching, or faster storage can fix.

Step 3: Divide Page Faults into Repairable and Fatal Cases

A page fault is an exception, not automatically a program bug. The kernel first checks the address and permissions:

• Legal but nonresident file page: read the file page or map an existing page-cache page.
• First legal access to anonymous memory: a first read can map the shared zero page, while a first write allocates a real physical page.
• First write after fork to a private page: perform copy-on-write and give the writer a private copy.
• Unmapped address or forbidden permission: if the access cannot be repaired, deliver a signal such as SIGSEGV.

After repairing a fault, the kernel updates the page table and retries the instruction that faulted. If storage I/O is required, the process blocks while waiting and the scheduler may run another task. That blocking path is why major faults can directly increase per-request latency.

Step 4: Interpret Minor and Major Faults Correctly

Linux uses an operational distinction: did fault handling require disk I/O?

• Minor fault: no disk I/O was required. The file page may already be in the page cache but not yet mapped into this process. Anonymous memory may be materialized on demand, or copy-on-write may complete in memory.
• Major fault: disk I/O was required. In this scenario, the first cold query may touch an index page absent from the page cache, forcing the kernel to read it from the index file.

Two counterexamples strengthen the answer:

1. A major fault does not require swap. A cold file-backed mapping can require storage I/O.
2. A minor fault is not free. It may still enter the kernel, allocate a page, update page tables, and retry an instruction; it simply avoids the slower disk-I/O path.

Step 5: Explain Why Summed RSS Is Misleading

When 8 workers read the same read-only file mapping, the underlying file pages can be shared through the page cache. Each process's RSS includes the resident pages mapped into that process, so the same physical page can appear in multiple RSS values. Summing all 8 RSS values double-counts shared pages.

At minimum, distinguish:

• VmSize: virtual address-space size;
• VmRSS: all pages resident for this process;
• RssFile: resident file-backed mappings;
• RssAnon: resident anonymous memory;
• PSS: shared pages divided proportionally among the processes mapping them;
• VmPTE: memory consumed by page-table entries;
• VmSwap: swapped anonymous private data for the process, not a complete view of file mappings.

/proc/<pid>/smaps_rollup provides aggregate RSS and PSS for all mappings in a process. When estimating the 8 workers' attributable physical footprint, the sum of PSS is generally more useful than the sum of RSS, though system page-cache competition and unrelated processes still matter.

Step 6: Verify the Bottleneck with Reproducible Observations

Build one timeline instead of drawing conclusions from a single top screenshot. In a Linux test environment, collect:

grep -E 'VmSize|VmRSS|RssAnon|RssFile|VmPTE|VmSwap' /proc/$pid/status
cat /proc/$pid/smaps_rollup
perf stat -e page-faults,minor-faults,major-faults -p "$pid" -- sleep 30

Then run the exact same query set twice:

1. after a cold start, record request p50, p95, and p99, fault deltas, block reads, and RSS/PSS;
2. repeat immediately with the same data, concurrency, and code path;
3. if major faults, read I/O, and tail latency are all high in the first run and much lower in the second, demand-loaded file pages are a strong primary explanation;
4. if major faults are rare while latency stays high, inspect CPU, locks, remote calls, and internal index initialization;
5. repeat under controlled memory pressure to see whether reclaiming hot pages recreates the latency spikes.

Fault counters are cumulative, so compare deltas within the same time window rather than lifetime totals. Measure workers separately so a background scanner's faults are not incorrectly attributed to online requests.

Step 7: Choose an Optimization from the Working Set and SLO

The most aggressive option is not automatically the best:

1. Keep demand paging: fastest startup and memory proportional to the real working set. It fits workloads that tolerate cold traffic or have shifting hot regions, at the cost of first-touch latency.
2. Warm only hot pages: run representative queries or touch pages from a hot-set manifest before readiness. It moves bounded cost earlier and is usually more controlled than scanning 20 GiB, but the hot-set definition must be maintained.
3. Provide madvise access hints: MADV_WILLNEED says the range will be needed soon, allowing read-ahead. Sequential and random patterns also have distinct hints. These are performance hints, not residency guarantees.
4. Test MAP_POPULATE: it prefaults page tables and causes read-ahead for file mappings, reducing later blocking faults. The tradeoff is slower mmap and startup, concentrated I/O and memory pressure, and the fact that incomplete population does not necessarily fail the call.
5. Improve index layout and working-set size: separate hot metadata from cold data, improve locality, and reduce random cross-page access. This is more durable than blind prefetching.
6. Gate traffic: expose full readiness after a defined hot-set coverage or fault-rate target, with a timeout so a node cannot remain unready forever.

Huge pages may reduce TLB pressure, but they do not remove file I/O or correct a badly chosen working set. Evaluate them only after evidence identifies TLB misses as a primary bottleneck and the memory granularity, fragmentation, and deployment environment are acceptable.

High-Quality Sample Answer

“I would first confirm that the 20 GiB region is a read-only file mapping and that mmap happens before fork. mmap creates a relationship between a virtual address range and the file; it does not read the entire file into physical memory. Each worker's VmSize therefore rises by about 20 GiB immediately, while RSS grows only when queries touch pages.

An access checks the TLB first. A TLB miss only says that the recent translation cache lacks an entry. If the page-table entry is valid, permitted, and resident, a page-table walk and TLB fill are enough; there is no page fault. A fault occurs only when the page-table state cannot satisfy the access, such as a nonresident file page, a first copy-on-write write after fork, or an illegal permission.

For Linux counters, I define minor as no disk I/O required and major as disk I/O required. After a cold start, index pages used by the first requests may be absent from the page cache, so reading them causes major faults and higher tail latency. Repeating the same queries hits the page cache, so it may cause only minor faults or use existing mappings and latency falls. Major faults are not limited to swap; file-backed mappings can generate them even on a machine without swap.

The workers can share read-only file pages, but each worker's RSS counts shared pages that it maps. I would not add RSS values directly. I would inspect PSS, RssFile, RssAnon, and VmPTE in /proc/<pid>/smaps_rollup, then align them with block reads, minor and major fault deltas, and request p99 in the same 30-second window.

For verification, I would run the same query set twice. If major faults, read I/O, and p99 are high in the first run and fall together in the second, that supports demand loading as the primary cause. I would then measure the real hot working set. If only 2 GiB of the 20 GiB index is hot, I would warm that region before readiness and use an appropriate madvise hint for sequential or random access. If the SLO allows more startup cost, I would run an A/B test with MAP_POPULATE. I would not scan the whole file by default or treat huge pages as a generic page-fault fix. The final decision would compare startup time, first-minute p99, major-fault rate, PSS, and stability after memory reclaim.”

This answer connects the concepts, observations, and decision into one testable chain. If the interviewer changes the mapping type, working set, or readiness SLO, the same framework still produces a different, defensible choice.

Common Mistakes

• Treating VIRT as RAM already consumed → a file mapping can reserve an address range without making every page resident → inspect VmSize, RSS, PSS, and the mapping type together.
• Equating a TLB miss with a page fault → a valid resident page-table entry only needs translation → describe the TLB lookup, page-table walk, and fault condition separately.
• Claiming every page fault reads disk → page-cache hits, anonymous zero pages, and copy-on-write can produce minor faults → use the Linux minor/major I/O distinction.
• Claiming major faults only come from swap → an uncached file-backed page also requires storage I/O → identify whether the faulting page is anonymous or file-backed.
• Adding the RSS of all 8 workers → shared file pages are counted repeatedly → use PSS to apportion shared pages and compare RssFile with RssAnon.
• Reading lifetime fault totals → they do not prove a relationship to a particular slow request window → compare fault deltas, I/O, and latency in the same interval.
• Scanning all 20 GiB during startup → this can waste I/O, delay readiness, and evict more useful pages → measure the hot set and warm only what the SLO requires.
• Assuming MADVWILLNEED or MAPPOPULATE eliminates future faults → the former is a hint, the latter may be incomplete, and pages can later be reclaimed → test cold-start and memory-pressure behavior.
• Enabling huge pages immediately → they mainly change translation coverage and allocation granularity, not file I/O or working-set quality → first prove that TLB misses dominate.

Follow-Up Questions and Responses

Follow-up 1: There is no swap on the machine. Why are there still major page faults?

Major means that fault handling required disk I/O; the source does not have to be swap. The index in this scenario is file-backed. On the first access to a file page that is absent from the page cache, the kernel must read it from the file system, so the access can generate a major fault. Inspect RssFile, the mapped path, and block reads rather than checking only VmSwap.

Follow-up 2: Another worker already read the same file page. What happens on this worker's first access?

The page may already be in the system page cache while this worker lacks a page-table mapping for it. Establishing that mapping usually requires no disk I/O and can appear as a minor fault. A later access may still have a TLB miss, but if the page-table entry is valid, that is not a page fault.

Follow-up 3: Why can writing a small amount after fork increase PSS?

Private pages can initially be shared through copy-on-write. On the first write, the kernel creates a private copy for the writing worker and updates its page table. A shared page becomes attributable to one process, so PSS rises. The fault normally requires no disk I/O, but allocation and copying still cost time. A read-only index should avoid accidental writes to a private mapping.

Follow-up 4: What happens if access is random but the service uses MADV_SEQUENTIAL?

A mismatched hint can trigger useless read-ahead, loading pages that will not be used and increasing I/O. For random point lookups, test MADV_RANDOM or the default policy; if the hot set is known, targeted prefetching is preferable. Judge every hint by faults, I/O, PSS, and latency rather than by its name.

Follow-up 5: When is MAP_POPULATE appropriate?

It is worth an experiment when startup may spend more time and I/O, online first-touch latency must be stable, and the soon-to-be-used mapping range is reasonably predictable. It can be wasteful when the mapping is much larger than the hot set, nodes scale frequently, or memory pressure will reclaim pages soon. Population failure also does not necessarily fail mmap, so actual residency and later faults must be measured.

Follow-up 6: How would you distinguish page-fault churn from a real memory leak?

A leak usually shows private anonymous memory or unreclaimable objects growing continuously even under stable traffic. Growth in a file-backed working set appears more in RssFile and the page cache and may fall after reclaim. Compare RssAnon, RssFile, PSS, VmSwap, and heap or object profiles, then repeat under stable traffic and controlled memory pressure. RSS growth alone cannot distinguish the two.