How Mmap Works for File-Backed Memory

File-backed mmap works by pointing a process's virtual memory straight at the kernel's page cache, so file data never gets copied between kernel space and user space. That single design choice explains almost everything about why mmap is fast, and it also explains every weird failure mode that occurs once real workloads start hitting it.

Compare it to the normal way of reading a file. With read() and write(), data has to travel from disk into the kernel's page cache, then get copied again into a buffer your program owns, and often back out again on the way to somewhere else. That's two copies and multiple context switches for what is, conceptually, one piece of information moving from point A to point B. mmap skips the second copy. The pages your process reads from are the same physical pages the kernel is already holding in cache. No buffer, no handoff, just a direct line.

What each mmap() parameter controls

The POSIX signature looks like this: void *mmap(void *addr, size_t length, int prot, int flags, int fd, off_t offset). Six parameters, and each one is doing real work, not just filling out a function signature for style points.

addr is the address you'd prefer the mapping to land at. In practice, almost nobody sets this to anything but NULL. Let the kernel pick. It knows what's free in the address space and you probably don't, so fighting it here buys you nothing.

length is how many bytes you want mapped, though the kernel doesn't actually deal in bytes internally, it deals in pages. So whatever number you pass gets rounded up to the nearest page boundary behind the scenes. If you ask for one extra byte past a page boundary, you've just mapped a whole additional page.

Then there's prot, which sets permissions on the mapping, and this is where things get genuinely useful:

PROT_READ Grants the process permission to read the pages.

• PROT_WRITE lets it write to them.
• PROT_EXEC lets it execute code from them (this is how shared libraries get loaded, incidentally).
• PROT_NONE allows nothing at all, which sounds useless until you realize it's the standard trick for building a guard page, a landmine you plant right next to a buffer so anything that overruns it faults immediately instead of quietly corrupting memory somewhere else.

How demand paging and the page fault lifecycle work

Calling mmap() doesn't actually load anything. The moment it returns, you've got a VMA (virtual memory area) registered in the process's address space, but there's no RAM committed to it. The page tables are empty. It's a promise, not a delivery.

The delivery happens on first touch, via a page fault. The CPU tries to translate the virtual address you just accessed, fails because there's no mapping yet, and traps into the kernel. From there, the kernel's fault handler runs through a fairly mechanical sequence:

1. Figures out which VMA the faulting address belongs to.
2. Maps that back to a specific file (the inode) and an offset inside it.
3. Checks the page cache, a per-inode structure that indexes cached pages by their offset.
4. If the page's already sitting in cache, that's a minor fault: the kernel just wires the existing physical frame into your process's page table. No disk involved.
5. If it's not cached, that's a major fault: the kernel grabs a fresh physical frame, reads the data off disk into it, drops it into the page cache, then maps it in.
6. The process resumes like nothing happened. As far as your code is concerned, the data was just always there.

The minor-versus-major distinction isn't academic, it's the whole ballgame for performance. Say half a file is already sitting in the page cache, maybe because another process read it earlier. Touching that half through mmap costs you minor faults only, essentially free. Touching the other, uncached half triggers major faults and real disk I/O. And because the page cache is a shared resource across every access path on the system, a file read via read() by one process and a file mapped via mmap by another are pulling from the exact same cached pages. There's no duplication happening anywhere.

Dirty pages, writeback, and what MAP_SHARED promises

Writing to a MAP_SHARED mapping edits a page cache page directly, and the CPU flips a dirty bit on that page table entry. That bit is just the kernel's way of saying: this memory is now newer than whatever's sitting on disk.

Writeback, the process of actually pushing that page to disk, doesn't happen on your schedule. It happens on the kernel's. Flusher threads periodically sweep for dirty pages and write them out, which is efficient for the disk but means your write returns long before the data is actually safe on the platter or the storage medium.

Your process never blocks waiting on disk I/O. That is why mmap writes feel instant. But instant and durable are two different promises. Until writeback runs, or until you force it, the only place your update definitely exists is RAM.

What "same physical pages" means in practice for cross-process coherence

MAP_SHARED's real guarantee is this: if two processes map the same file, their page tables point at the same physical frames in the page cache. Not copies of each other, not synchronized replicas. The same frames. So when process A writes, process B sees it immediately, with no copying step and no IPC channel required to ferry the update across.

The page cache only ever holds one version of a file's data at a time, which is precisely what makes this work. Map the same file from three different processes and all three are looking at the same physical memory through three different sets of page tables. It's the mechanism behind memory-mapped IPC: one process writes into the shared region, another reads it, and no system call gets involved in the handoff. Messaging systems built for low-latency throughput (Chronicle Queue is a known example) lean on exactly this property.

What the kernel does not do is order or synchronize those accesses for you. Sharing the physical pages is a plumbing guarantee, not a traffic-cop guarantee. If two processes are reading and writing the same region, coordinating who goes when is entirely on the application: locks, atomics, futexes, POSIX semaphores, or pthread mutexes set up with PTHREAD_PROCESS_SHARED. Without that coordination, a reader can catch a write mid-flight, half-old and half-new data stitched together into something that was never a valid state at any point in time.

The pitfalls that bite once the basic model is working

Everything above sounds clean in isolation. Then production happens.

The sharpest surprise is SIGBUS on truncation. If any process (including something outside your control, like a log rotation job) shrinks a file below the range you've got mapped, touching the pages past the new end of file doesn't return an error code you can check. It delivers SIGBUS, and by default that's fatal. There's no return value to inspect beforehand, no graceful failure. Prevention means either installing a signal handler for it or, better, just not letting the file shrink while anything holds a live mapping over it. The reliable pattern is to ftruncate() the file to its final size before you ever call mmap, and never touch that size again for the mapping's lifetime.

Durability is the second trap, and it catches people specifically because it looks so much like the write() world they already know. A store into a MAP_SHARED region means the page cache has the update. It does not mean the update would survive a crash. Code that gets ported over from a write() plus fsync() pattern and drops the equivalent msync() call loses the durability guarantee it thinks it still has, silently, with no error anywhere to flag the gap.

Then there's TLB pressure, which is the one that undercuts mmap's reputation for speed. Every page fault plants a new entry in the TLB, the CPU's small, fixed-size cache of virtual-to-physical address translations. Fine at low page counts. But under high-concurrency workloads doing random access across a huge number of distinct 4 KB pages, TLB misses stack up fast, and the translation overhead starts eating into the very performance advantage mmap was supposed to hand you, undercutting the assumption at its worst possible moment. This is the scenario where "mmap is always fast" quietly stops being true.

Consistency is the last piece, and it's less a bug than a mismatch in expectations. mmap gives you none of the ordering or isolation guarantees a database gives you by default. No ACID semantics ride along for free. Concurrent reads and writes into the same region are visible to each other instantly, sure, but the order they're visible in is entirely up to whatever synchronization you built yourself.

Tuning mmap with huge pages and madvise

Standard pages are 4 KB. Mapping a large region of a file requires a correspondingly large number of TLB entries to cover it, and the TLB, being small and fixed in size, runs out fast. That's the cause of the TLB pressure described above, and huge pages are the direct fix: instead of covering memory in 4 KB slices, a huge page typically covers 2 MB. This means dramatically fewer TLB entries are needed to map the same amount of data.

There are two ways to get there, and they trade off differently:

• MAP_HUGETLB reserves huge pages explicitly at mmap() time, guaranteed up front. Good fit for latency-sensitive workloads with memory needs you can predict in advance, but it depends on the system already having a huge page pool available to draw from.
• MADV_HUGEPAGE, set through madvise(), is a hint instead of a demand. It tells the kernel "promote this region to huge pages when you get the chance," and the kernel handles the actual promotion on its own schedule. Easier to bolt onto existing code, more flexible, but there's no guarantee on when, or whether, the promotion actually happens.

Kernel version matters here too. Transparent Huge Pages (THP) automatic scanning has been extended to cover file-backed pages, not just private anonymous memory and shmem like before. But when THP is running in per-process mode rather than system-wide, the kernel only promotes VMAs that were explicitly flagged with madvise(MADV_HUGEPAGE), it won't just do it to every mapping on a hunch.

None of this is a guaranteed win, either. THP can help a given workload or do nothing for it, and the only way to know which is to test it against the actual access pattern in question. Test it against the actual access pattern before treating it as a setting you flip and forget.