What a Write-Ahead Log Actually Does

Every database makes a promise it shouldn't be able to keep: your data survives a crash, and writes stay fast. Those two goals fight each other. Durability wants every change nailed to disk before you move on. Speed wants to avoid disk as much as possible, because disk is slow and memory is quick. The write-ahead log (WAL) is the trick that lets a database keep both promises without lying about either one.

The obvious way to guarantee durability is to flush every change to its actual home on disk before telling the user "yep, committed." That sounds safe. It's also brutal, because those changes land on data pages scattered all over the disk, and random writes to scattered locations are about the slowest thing a storage system can be asked to do. Multiplying that by thousands of transactions a second causes the database to grind to a halt waiting on disk heads (or, on SSDs, waiting on write amplification and garbage collection) instead of doing actual work.

So there's a real bind here. Users want crash-proof guarantees. Users also want fast writes. A system that just picks one loses either customers or data, and neither is a good look. WAL's answer is that you don't have to choose, because the log and the data files are allowed to do different jobs on different schedules. In ACID terms, WAL is the mechanism that buys atomicity and durability, the A and the D, without holding the rest of the system hostage to disk latency.

What the log-first rule means, step by step

The rule is simple to say and easy to underestimate: changes to data files can only happen after the WAL records describing those changes have been flushed to permanent storage. Not before. Not at the same time. After.

Break that into three steps and it stops sounding like a slogan and starts sounding like an engineering decision.

Step 1, log first. The change gets appended, sequentially, to the write-ahead log, and that log gets synced to stable storage. This is the only write that has to happen before the transaction can be called committed.

Step 2, apply later. The actual data page, sitting in the buffer pool in memory, gets updated whenever it's convenient. That write can be deferred for seconds, minutes, whatever, because the log already guarantees the change won't be lost. The urgency has already been handled.

Step 3, crash recovery. If the system falls over and comes back up, it replays the WAL to reconstruct any changes that never made it to the data files. This has a name: roll-forward recovery, or REDO, and it's the reason a crash doesn't mean data loss even though plenty of changes were sitting only in memory when the lights went out.

Each log record captures enough information to reconstruct the change it describes. Some designs also carry information needed to reverse changes, for rolling back transactions that never finished. A transaction isn't considered committed until its log records are safely on disk. That's the line, and the system enforces it strictly.

One structural detail governs whether sequential I/O is possible at all: the WAL is append-only. That's not a minor implementation choice, it's the whole reason sequential I/O is possible in the first place. Compare this to shadow paging, an alternative durability technique that copies whole pages instead of logging changes. WAL allows updates in place. It doesn't need to rewrite indexes and block lists every time something changes. Less rewriting, less overhead, simpler mental model.

Why sequential writes change the performance calculus

Storage hardware, spinning disk and SSD alike, is built to be fast at sequential I/O and comparatively sluggish at random I/O. Appending to a log is about as sequential as a write pattern gets, so WAL isn't fighting the hardware, it's working with the grain.

Only the WAL file needs a flush to guarantee the commit, saving the performance budget. Not every data file the transaction touched, just the log. This cuts the number of disk writes down significantly, because instead of forcing many scattered writes to be durable, the database forces one.

That single flush also stretches across many transactions at once. On a server handling a pile of small concurrent transactions, one fsync of the WAL file can commit a whole batch of them simultaneously. That's a real multiplier, not a rounding error.

Tuning WAL to match workload characteristics can yield 30 to 40 percent performance improvements without touching a single line of application code. Sector alignment matters too: matching log records to storage sector boundaries is a lower-level lever available to anyone tuning at that level.

Eventually you hit a wall that isn't really WAL's fault. At some point, the bottleneck shifts from the log abstraction to the hardware beneath it, the log design itself stops being the limiting factor.

How checkpointing closes the loop between the log and the data files

If the log just kept growing forever, recovery would eventually mean replaying years of history, which defeats the point. Checkpointing is what keeps that from happening.

A checkpoint flushes all the dirty pages sitting in the buffer pool out to the actual data files, then writes a checkpoint record into the WAL marking that it did so. After that record exists, everything in the log before it is no longer needed to reconstruct state, because the data files are already caught up to that point. Old WAL segments before the checkpoint can then be recycled or archived, which is what keeps log storage from growing without bound.

Checkpoint frequency is a genuine tradeoff, not a setting you pick once and forget.

• Checkpoint often: recovery after a crash is fast, because there's less log to replay. I/O spikes during normal operation can stall other work.
• Checkpoint rarely: normal write throughput stays smoother, with fewer disruptive spikes. A much longer stretch of log must be replayed at restart before the system is usable again.

Formally, a checkpoint is a record in the log marking the point where application state has been completely written to stable storage. Recovery algorithms use it as the starting line. Tuning it wrong in either direction causes jittery write latency during the day or a longer coffee break waiting for a restart at 3 a.m.

Where WAL shows up across the systems landscape

The pattern appears nearly everywhere data needs to survive a crash, which says something about how few good ideas there are in this business and how hard people lean on the ones that work.

PostgreSQL leans on WAL for crash recovery, online backup, and point-in-time recovery. Archive the WAL on top of a base backup, and the database can be rewound to any instant since that backup, as long as the archive is complete. Logical replication takes it further, decoding WAL records into logical operations and streaming them out to subscribers.

MySQL, running InnoDB, uses its own WAL called the InnoDB redo log for crash recovery, and keeps a separate server-level binary log (the binlog) for replication and point-in-time recovery to other servers. SQLite's WAL mode keeps a separate WAL file alongside the main database file, structurally doing the exact same thing: log first, apply later. MongoDB, via its WiredTiger storage engine, uses WAL as its durability layer too, proof that the pattern isn't a relational-database-only habit. Apache Cassandra calls its version the commit log, and every write hits it before touching any in-memory structure.

Apache Kafka bends the pattern in an interesting way. Messages get written to a durable, append-only log before consumers ever touch them, and Kafka's whole architecture is log-centric the same way WAL is. But there's a twist: in Kafka, the log itself is the data store. There's no secondary data file the log gets applied to later, unlike a classical WAL setup.

Even file systems get in on it. Modern file systems use a variant of WAL for metadata, called journaling, protecting directory structures and inodes with the same log-before-apply logic.

When the branding is stripped away, the pattern is identical everywhere it appears: log the intent durably, apply the change later, recover by replaying the log. The data structure changes. The principle doesn't.

WAL pushed to the limit: checkpointing in large-scale AI training

Distributed model training turns checkpointing into a WAL problem at a scale that would make a database administrator's eye twitch. The idea is structurally identical: write state durably before continuing, so a failure rolls back to a known-good point instead of forcing a restart from zero.

At this scale, hardware failure isn't the exception, it's the schedule. A 16,384-GPU cluster used to train LLaMA 3.1 405B reported a mean time between failures of around three hours, according to an arxiv paper. Meta's own reporting on the Llama 3 training run (cited in MLCommons's MLPerf Storage v2.0 announcement) put the run at 54 days with 419 failures along the way, and most of those failures scale up right along with cluster size. More GPUs, more chances for something to break.

Checkpoint size is the real bottleneck here, and it's a big one. LLaMA 70B, with its optimizer state included, needs roughly 700 GB when serialized. Even with a fast CPU-to-GPU bus and a 500 Mb/s connection, a single checkpoint can take upwards of 20 minutes, according to an arxiv paper. For the Llama 3.1 405B run specifically, checkpointing and failure recovery combined ate about 2.1 percent of total training time, with the optimal checkpointing cadence landing at every 4 minutes, each checkpoint taking about 2.5 seconds, per figures from epoch.ai.

How the checkpoint gets written depends on the parallelism strategy, and the two common approaches put very different pressure on storage:

• FSDP (Fully Sharded Data Parallel): every GPU process writes its own shard at the same time, which is a massive concurrent write burst hitting storage all at once.
• DDP (Distributed Data Parallel): a single rank writes the entire checkpoint while every other rank sits and waits, which is a serialization bottleneck baked right into the design.

Researchers are attacking this from the recovery side rather than the failure-prevention side. A framework called FFTrainer shifts the goal from stretching mean time between failures to shrinking recovery overhead instead, and an arxiv paper reports it cuts recovery time by up to 98 percent compared with prior checkpointing approaches, while cutting GPU utilization loss by up to 68 percent. A separate paper on a system called LMStor (Springer) reports shrinking checkpoint interruptions from minutes down to milliseconds, improving failure recovery efficiency by 3 to 24 times over traditional methods, and boosting dataset loading efficiency by 46.63 percent.

The storage pressure isn't theoretical. A 63-node NVIDIA B200 cluster (504 GPUs) saw NFS read traffic spike to roughly 230 GB/s across the cluster at session startup, driven by checkpoint and training data loading together, with about 200 GB pulled into page cache per node over roughly 25 minutes, according to an arxiv paper. Once training resumed from cache, NFS traffic dropped to almost nothing. That kind of burst-then-silence pattern caught MLCommons's attention: MLPerf Storage v2.0, launched in August 2025, added new checkpointing benchmarks for LLMs on scale-out systems, on top of the training-data-ingestion tests the earlier versions focused on exclusively.

None of this is a knock against log-first design. The bottleneck isn't the idea, it's the write path the idea has to travel through. Storage systems built for transactional row-and-page workloads were never built for the kind of burst write that a checkpoint event throws at them.

Building WAL on object storage instead of local disk

Object storage, S3, GCS, Azure Blob, is cheap, elastic, and durable in ways local disk simply isn't. It also wasn't built for WAL's append-only, sub-millisecond write path, which creates a real tension for anyone trying to put a WAL on top of a bucket.

A handful of systems have tackled this directly. Neon, a serverless Postgres provider, splits storage into two roles: Safekeepers receive WAL straight from Postgres, and Pageservers turn that WAL into actual data pages, serve reads, and upload storage layers to S3. A lot of the engineering effort has gone into cutting ingest latency, reducing read amplification, and tightening the WAL-to-S3 path specifically.

Chroma built something called wal3, designed to store data durably on object storage while keeping an ongoing cryptographic proof of correctness running the whole time. It leans on S3's conditional writes feature and a checksumming technique the team calls setsum, combined with a lock-free algorithm, to get correctness without needing coordination overhead between writers (per Chroma's engineering blog).

A separate design called OSWALD builds a WAL exclusively out of object storage primitives, needing only read-after-write consistency and compare-and-swap operations, both of which are available on S3, GCS, and Azure Blob. It supports checkpointing and garbage collection, and it's been formally specified and verified using the P programming language, according to published design documentation.

QuestDB appends incoming data to its WAL with very low latency, then ships that WAL asynchronously to object storage. New replicas bootstrap themselves by reading from the same WAL history, rather than needing a separate replication mechanism. And in the Kafka ecosystem, AutoMQ adds a shared streaming storage layer called S3Stream, with its own internal WAL component, sitting between the broker and object storage so Kafka-compatible behavior is preserved while object storage does the actual durability work underneath. A related proposal, KIP-1176, explored a per-broker S3-based WAL for Kafka, though it was withdrawn in December 2025 in favor of a different proposal, KIP-1150.

Different products, same underlying move. The WAL abstraction doesn't change, what changes is the durable target underneath it, swapping a local block device for object storage. The log-first principle survives the swap intact. What makes the swap safe, across every one of these systems, is conditional writes, the compare-and-swap primitive that keeps an append-only log from getting corrupted when more than one writer shows up at once. Without that primitive, this whole approach falls apart. With it, WAL semantics become something closer to a cloud-agnostic durability layer, available whether the bucket underneath happens to be S3, GCS, R2, or Azure Blob, without forcing anyone to migrate away from the storage that already holds their data.

What WAL's design choices mean for systems built on top of it

Recovery time isn't a fixed property of a database, it's a dial someone gets to turn. Checkpoint frequency, WAL buffer sizing, log segment management, all of these trade write-path smoothness against restart latency, and the right setting depends entirely on how much recovery time a given team can tolerate versus how much daytime jitter they're willing to eat.

WAL earns its keep well beyond crash recovery, too. Point-in-time recovery, online backup, logical replication, replica bootstrapping, all of it flows out of the same log-first architecture. The log isn't just a safety net sitting under the database in case something breaks, it's a running record of every state transition the system has ever made.

A boundary exists here, though. WAL guarantees that whatever got committed is durable. It does nothing to protect against an application bug that commits the wrong thing in the first place. The log is a faithful court reporter, not a fact-checker: it writes down what it was told, mistakes included.

The tuning numbers show a 30 to 40 percent performance swing from WAL buffer sizing alone, and warrant taking seriously rather than filing away as a footnote. A 30 to 40 percent performance swing from WAL buffer sizing alone means WAL isn't some sealed black box, it's a set of dials, and buffer sizing, sector alignment, and sync policy are all things a team can actually go adjust.

AI training sharpens this into something bigger. When the unit being written isn't a row anymore but a multi-hundred-gigabyte model shard, checkpoint design becomes a first-class piece of infrastructure because the scale demands it. The underlying principles haven't changed, but the engineering surface area sure has.

And once the WAL's home is a bucket instead of a local disk, the operational picture shifts too. Access control, cost, and retention policy all become part of the durability story now, not separate concerns bolted on afterward. The bucket stays the source of truth no matter what compute layer happens to be sitting in front of it that week.

When all of it is stripped down, the same idea keeps showing back up: WAL works because it splits durability (the log has to get written) from performance (the data pages can wait their turn). Relational databases rediscovered that split. Distributed message queues rediscovered it. AI training clusters, burning through hundreds of GPUs and hundreds of gigabytes per checkpoint, are rediscovering it right now. Anything that needs to be both safe and fast eventually lands in the same place.