Linux Interview: Why Does RSS Stay High After Memory Is Freed?

Prompt and Applicable Scenario

A 64-thread Linux service using glibc malloc starts at 800 MiB RSS. A periodic batch drives RSS to 6 GiB. Twenty minutes after the batch finishes, the application's heap profiler reports 1.1 GiB of live allocations, but RSS remains at 5.2 GiB. The process runs under an 8 GiB cgroup limit and has a 120 ms p99 latency SLO. Explain why free() does not guarantee that RSS falls, prove whether the gap is a leak, allocator retention, fragmentation, or another mapping, and choose a safe remediation.

The thread count, memory values, idle interval, limit, and SLO are exercise assumptions. Assume a native process, glibc malloc, cgroup v2, no child processes, and a repeatable batch. A managed runtime would add its own heap, garbage collector, and native-allocation layers. This question belongs to general because the central skill is Linux process accounting and allocator behavior, not application-language syntax.

Current interview material explicitly treats free lists, size classes, thread-local allocation, and fragmentation as systems-interview discussion points. Redis's production documentation describes the same observable symptom: deleting logical data may leave RSS near its prior peak because free chunks remain available to the allocator or pages still contain live objects. Linux and glibc documentation then provide the measurement and control surfaces needed for an evidence-based answer. These sources establish relevance and mechanics; they do not prove that a particular company uses this exact prompt or that it is asked at a particular frequency.

What the Interviewer Is Evaluating

The first signal is whether you separate ownership from residency. After free(p), the caller no longer owns that allocation and the allocator may reuse the block. The C allocation contract does not promise an munmap, a lower RSS, or immediate physical-page reclamation. Saying either “free always returns memory” or “free never returns memory” misses allocator-specific paths.

The second signal is whether you separate four measurement layers:

• application live allocations;
• allocator in-use, free, mapped, and releasable bytes;
• process mappings and resident pages;
• cgroup-wide charges and pressure.

RSS is not a live-heap counter. Linux defines VmRSS as RssAnon + RssFile + RssShmem. File mappings, shared memory, stacks, allocator metadata, and native libraries can all widen the gap. A heap profile and one top reading therefore cannot prove or disprove a leak.

The third signal is diagnostic discrimination. A leak leaves allocations reachable or otherwise live. Retention means free memory remains mapped for efficient reuse. Fragmentation means the allocator has free bytes but cannot form page-sized releasable regions, often because a few live objects pin pages or free space is split among arenas and size classes. These states can coexist, so the candidate needs controlled observations rather than a label inferred from one ratio.

Finally, the interviewer wants a production decision. Lowering arena counts, trimming aggressively, or replacing the allocator can reduce resident memory while adding locks, page faults, system calls, or CPU work. A strong answer protects both the 8 GiB limit and the 120 ms p99 SLO with a canary, a workload replay, acceptance thresholds, and rollback.

Questions to Clarify Before Answering

• Which allocator and version are active? glibc tunables and malloc_trim do not describe jemalloc, tcmalloc, mimalloc, a language runtime allocator, or a statically linked replacement. Confirm the loaded allocator and deployment image before using its counters.
• What exactly reports 1.1 GiB? A sampled heap profile, exact allocator counter, managed-runtime heap, and business cache metric cover different bytes. Confirm whether native libraries, stacks, direct mappings, and allocator metadata are included.
• Which RSS component stays high? RssAnon points toward heap, stacks, and anonymous mappings; RssFile toward file-backed mappings; RssShmem toward shared memory. If the increase is not anonymous, allocator tuning is the wrong first move.
• Does the plateau repeat or climb after each identical batch? A stable high-water plateau that serves the next batch without another 5 GiB rise suggests reuse. A staircase in live allocations or RSS needs a leak, workload, or mapping explanation.
• Did thread count, allocation sizes, or object lifetimes change? Many threads and cross-thread frees can spread blocks across arenas or caches. Mixed short- and long-lived objects can leave one survivor on many otherwise free pages.
• Is the cgroup actually under pressure? Compare memory.current, memory.events, PSI, swap, and neighboring processes. High RSS with ample headroom may be an efficiency issue; repeated memory.high events or proximity to 8 GiB makes release behavior operationally urgent.
• When will the next batch run? Keeping 4 GiB reusable for five minutes may be rational. Keeping it for twelve idle hours under a tight limit may justify a post-batch purge or a different allocation pattern.
• What regression budget is acceptable? The remedy changes if the service can spend 2% more CPU but cannot add 5 ms to p99, or if memory cost matters more than cold-allocation latency.

30-Second Answer Framework

“free() returns a block to the allocator; it does not promise to unmap its pages, so RSS can remain high without a leak. I would align one batch timeline and compare live-allocation profiles, allocator in-use and free bytes, RssAnon, per-mapping smaps, and cgroup usage. Then I would repeat the batch three times. Growing live bytes indicates a leak; stable live bytes plus a flat RSS that the next batch reuses indicates retention; abundant allocator-free bytes with little releasable memory and pages pinned by surviving objects indicates fragmentation. I would use malloc_trim(0) only as a glibc canary experiment, not a blanket fix. The final remedy could be ownership repair, lifetime separation, a bounded post-batch trim, tested arena tuning, or an allocator change, accepted only if a production-shaped replay stays below the memory target without breaking 120 ms p99.”

Step-by-Step Deep Dive

Step 1: Build One Memory Timeline

Record deployment version, PID, cgroup path, thread count, request and batch volume, allocation-rate distribution, live bytes, RSS components, memory.current, memory.peak, pressure, and OOM events. Sample before the batch, at the 6 GiB peak, immediately after deallocation, and through the 20-minute idle window. A different PID, cgroup, or workload invalidates the comparison.

The first question is operational: is RSS merely high, or is the service approaching a limit and reclaiming under pressure? At 5.2 GiB inside an 8 GiB limit, the apparent headroom is 2.8 GiB before other cgroup charges. That subtraction is only a scale check because the cgroup also accounts memory beyond this process's anonymous RSS. Use memory.current and memory.stat for the actual domain.

Step 2: Explain the Allocator-to-Kernel Boundary

An allocator requests larger regions from the operating system and divides them into chunks. glibc can extend ordinary arenas and can create separate anonymous mappings for sufficiently large allocations. When an application frees a chunk, the allocator first makes that chunk reusable. It can coalesce adjacent free chunks, place them in bins or caches, retain them to avoid future system calls, or release suitable pages.

Independently mapped large chunks can often be unmapped on free. Ordinary heap memory is harder. A free block in the middle of a region cannot shrink the end of the heap, and a page containing even one live raw-pointer object cannot be unmapped without relocating that object. C and C++ allocators generally cannot compact arbitrary live objects because application pointers would become invalid.

This produces three different kinds of overhead:

• Internal fragmentation: a 20-byte request may consume a larger aligned or size-class block.
• External or page-level fragmentation: free space exists, but it is split or pinned so whole pages cannot be released.
• Intentional retention: whole or partial pages remain mapped because the allocator expects reuse and release/reacquisition has a cost.

The labels describe mechanisms, not conclusions from a single RSS / live_bytes ratio.

Step 3: Reconcile the Four Measurement Layers

Start with Linux process accounting:

VmRSS = RssAnon + RssFile + RssShmem

Read /proc/PID/status for the broad split and /proc/PID/smaps_rollup for aggregate RSS, PSS, anonymous, file, shared, and lazy-free information. Use /proc/PID/smaps only when you need to identify the particular anonymous or file-backed mappings that grew. A one-time pmap or RSS total is less informative than synchronized deltas.

Then add allocator statistics. In glibc, mallinfo2 can expose bytes obtained through sbrk, bytes in mapped chunks, bytes handed to callers, free bytes, and the top releasable chunk. These fields do not cover every allocation source and must be sampled consistently, but they help answer whether the allocator owns most of the anonymous gap. Prefer the application's allocator-native profiling when it provides more complete mapped, active, resident, retained, and per-size-class data.

Finally, reconcile with cgroup v2. The cgroup includes all charged memory in its hierarchy, so it may exceed one process's RSS or move for a different reason. Compare memory.current and memory.stat with the process evidence instead of forcing them to equal.

Step 4: Use a Three-Cycle Reuse Experiment

Run the same batch three times on a production-shaped canary with the same 64-thread concurrency and input-size distribution. After each batch, wait the same 20 minutes and capture the same counters.

Interpret the shapes:

A plateau is not automatically harmless. If the next batch peaks above 8 GiB because old retained pages cannot serve its new size distribution, the service can still fail despite stable logical data. Conversely, a high plateau that efficiently satisfies the same workload may be preferable to forced release and repeated faults.

Step 5: Use Trim as a Bounded Diagnostic

On a glibc canary, call malloc_trim(0) once at the post-batch boundary and record its return value, RSS components, allocator live bytes, page faults, CPU, and subsequent request latency. The GNU interface attempts to release free heap memory and may use sbrk or madvise; it does not promise a particular RSS reduction.

If RssAnon falls substantially while live allocations remain at 1.1 GiB, some allocator-owned pages were releasable. That narrows the diagnosis, but it does not prove that calling trim in production is the best policy. If RSS barely changes, whole free pages may be unavailable, the growth may live outside glibc, or the metric may include different mappings. Failure to trim does not prove a leak.

Never place trim on every request. Releasing pages can trade resident memory for system calls, minor faults, zero filling, cache loss, and tail latency when the next batch arrives. A natural phase boundary with a long idle window is a more defensible candidate, and it still needs a canary.

Step 6: Match the Remedy to the Proven Cause

• Leak: fix the owning reference, cache bound, missing free, or library lifecycle. Trimming does not make live memory releasable.
• Intentional retention with near-term reuse: keep it, provision for the measured peak, and alert on a repeated staircase rather than forcing RSS to match live bytes.
• Lifetime-driven fragmentation: separate short-lived batch objects from long-lived service state, use a batch arena or region that can be released as a unit, and avoid interleaving one long-lived object across many transient pages.
• Arena or thread-cache dispersion: test fewer arenas, lower concurrency at the allocation hotspot, or a different ownership pattern. Fewer arenas may save memory but increase contention.
• Long idle phase under a tight limit: test one explicit post-batch trim or allocator-specific purge with rate limits and a rollback flag.
• Allocator mismatch: compare glibc with a suitable alternative under the exact trace. Microsoft Research's mimalloc design illustrates the core tradeoff: thread-local pages improve scalability and locality, while isolated ownership can retain memory that another thread cannot immediately reuse.

glibc's arenamax, trimthreshold, and mmap_threshold are experiments, not magic constants. Setting them makes behavior more static and can change contention, mapping count, release frequency, and syscall cost. Change one factor at a time and retain the original image as the rollback.

Step 7: Validate Memory and Latency Together

Replay 30 identical cycles on production-shaped hardware. The number 30 is an exercise test window, not a universal requirement. Track live allocations, allocator mapped/free/releasable bytes, RssAnon, total RSS, memory.current, pressure, page faults, allocation latency, CPU, throughput, and p50/p99 request latency.

Example acceptance criteria for this scenario could be: post-idle RssAnon at or below 2.2 GiB within 20 minutes, no upward staircase across 30 cycles, no OOM or sustained pressure, p99 at or below 120 ms, and no more than 3% CPU regression. These thresholds are practice assumptions to replace with the service's real budget. A remedy that reaches 2.2 GiB but pushes p99 to 145 ms fails; a remedy that preserves latency but still approaches the 8 GiB limit under the next valid size mix also fails.

High-Quality Sample Answer

“I would not call this a leak from RSS alone. free() ends the application's ownership of a block, but glibc may keep that block in an arena for reuse, and a few live objects can keep otherwise free pages resident. I would first verify that the 1.1 GiB number covers native live allocations, then align it with RssAnon, RssFile, RssShmem, smaps mappings, allocator statistics, and the cgroup charge through one complete batch.

I would replay the same batch three times. If live allocations increase after every 20-minute idle period, I would diff live-object profiles and fix the ownership path. If live bytes stay at 1.1 GiB, RSS stays near 5.2 GiB, and the next batch reuses that space without another rise, retention is the stronger explanation. If allocator-free bytes are high but releasable pages stay low and the plateau changes with object lifetimes or 64-thread concurrency, I would investigate fragmentation and arena dispersion.

As a glibc-only diagnostic, I would call malloc_trim(0) once on a canary after the batch. A lower RssAnon with unchanged live bytes shows that some allocator pages were releasable; it does not justify trimming every request. I would then choose the smallest cause-specific change: repair a leak, separate batch lifetimes into a releasable region, or canary a bounded post-batch trim or arena setting. I would replay 30 cycles and accept the change only if post-idle memory meets the agreed target, the RSS does not staircase, the cgroup stays safe, and p99 remains within 120 ms.”

Common Mistakes

• Calling the 4.1 GiB gap a leak → RSS includes allocator-free pages and non-heap mappings → prove growth in live allocations or retained ownership with synchronized profiles.
• Claiming free() always lowers RSS → a freed chunk may remain in an arena or share a page with live chunks → describe reuse, whole-page release, and independently mapped allocations.
• Claiming free() never returns memory → allocators can unmap large mappings, trim top chunks, or advise whole free pages away → state that release depends on allocator, layout, and policy.
• Using one fragmentation ratio as proof → a recent peak, file mapping, cache, or intentional retention can inflate it → compare live, free, mapped, resident, and releasable bytes over repeated cycles.
• Calling malloc_trim(0) on every request → forced release can add syscalls, faults, and tail latency → test it once at a natural idle boundary and measure the next allocation burst.
• Setting arena_max to one because there are 64 threads → memory may fall while lock contention rises → sweep candidate values under the same concurrency and protect p99.
• Switching allocators from a benchmark headline → allocation size, lifetime, and cross-thread-free patterns determine results → A/B the exact trace with rollback and both memory and latency criteria.
• Ignoring cgroup accounting → one process's RSS is not the complete memory domain → reconcile memory.current, memory.stat, descendants, and pressure with process metrics.

Follow-Up Questions and Responses

If malloc_trim(0) drops RSS from 5.2 GiB to 1.8 GiB, what have you proved?

You have shown that glibc owned a substantial amount of page-releasable memory at that moment and that the high RSS was not all live application data. You have not proved the absence of a smaller leak, the cause of retention, or the safety of frequent trimming. Re-run the workload and measure page faults, CPU, allocation latency, and p99 before selecting a policy.

If trim returns zero and RSS does not move, is it a leak?

No. The free space may be distributed across pages that still contain live chunks, memory may be held in another allocator or mapping, or no releasable glibc pages may exist. Compare live profiles, allocator free and releasable bytes, and smaps ownership. A leak requires evidence of live or retained allocations growing, not merely unsuccessful trimming.

What if RSS stays flat but memory.current continues to grow?

Investigate the cgroup delta instead of the heap. memory.stat can reveal file cache, shmem, socket, or kernel memory, and the cgroup may include other processes or descendants. Confirm the hierarchy and mapping ownership. Tuning glibc because one process's RSS is flat would target the wrong layer.

Why not force glibc to use one arena?

One arena can reduce dispersion but serializes more allocation work. With 64 threads, that may exchange resident memory for lock contention and p99 regressions. Test several bounded arena counts under the real allocation trace, capture allocator contention and latency, and select the smallest count that satisfies both budgets.

When would a batch arena or region allocator be better?

It is attractive when most objects share one clear lifetime: allocate them from a region during the batch and release the region together afterward. It is unsafe when references escape into long-lived service state, destructors or per-object cleanup are required, or one batch contains many unrelated lifetimes. Enforce the ownership boundary before relying on bulk release.

How would you evaluate a replacement allocator?

Use the same build except for allocator linkage, the same input trace, thread count, CPU placement, warm-up, and 30-cycle window. Compare peak and post-idle RSS, live-to-resident gap, allocation throughput, CPU, page faults, p50/p99, and failure behavior near the 8 GiB limit. Canary the winner behind a deployment rollback; a lower average RSS alone is not sufficient.