Atomic Rename as a Crash-Safety Primitive

Atomic rename is the trick that makes crash recovery boring instead of terrifying. The pattern is simple: write your data to a temp file, call fsync() to force it out of memory and onto disk, then rename() that temp file over the real one. After a crash, exactly one of two things exists. Either the old file, untouched, or the new file, fully formed. No in-between. No torn half-write sitting there waiting to corrupt something downstream.

Arpit Bhayani put it well in February 2026: "Renaming of a file is an atomic operation, and this is one guarantee that makes so many database implementations simpler." He's right, and that single sentence explains why half the storage engines built have this pattern baked into them somewhere. But the guarantee is narrower than most people assume, and that gap is where a lot of "impossible" bugs actually live.

POSIX guarantees rename() is atomic with respect to other processes. POSIX guarantees rename() is atomic with respect to other processes, so a concurrent reader never sees a missing file mid-rename. That's the whole promise, and nothing more. That's the whole promise. It says nothing about what happens when the power cuts out mid-operation. Namespace atomicity and data durability are two completely different properties wearing the same trench coat, and mixing them up is how "correct" code loses data anyway.

Where the guarantee breaks: filesystem-by-filesystem reality

Borrill's March 2026 arXiv paper gave this problem a name that's almost too good: a "FITO category mistake." FITO, as in "failure is the only option," and the mistake is assuming atomic state transitions actually happen instantly at every layer of the stack. They don't happen instantly at every layer of the stack. Not in ext4's journaling, not in delayed allocation, not in fsync failure semantics, not in NVMe Flush/FUA behavior, not in Linux restartable sequences, and not down at the x86-64 level when an unexpected hardware signal occurs uninvited.

No syscall-based persistence primitive can define a commit boundary under failure. The return value from your syscall is consistent with multiple, materially different persistence states depending on which Linux filesystem you're standing on. Execution order does not imply persistence order. Namespace atomicity does not imply data durability. A syscall returning successfully does not mean the data actually made it anywhere permanent, because delayed allocation splits "the write finished" from "the blocks got allocated" from "the data is actually on disk" into three separate events that don't have to happen together.

Btrfs is the case everyone cites, and for good reason. Research has found that rename is crash-atomic on btrfs only under certain conditions, not universally, a distinction with real teeth where the guarantee can silently evaporate depending on the scenario. Empirical testing bears this out: a filesystem offering only minimal, in-order guarantees exposed 10 minor vulnerabilities. Btrfs exposed 31. Ext4 exposed 17. More features, more surface area, more ways to get burned.

Even the textbooks hedge. OSTEP's chapter on files says rename is "(usually)" atomic on crash. A GitHub issue filed against the book in May 2020 argues the word "usually" is doing an enormous amount of load-bearing work there, and that rename is, in practice, usually not crash-safe the way people assume. Each layer of the stack assumes the layer below it handles atomicity. None of them actually does. It's atomicity all the way down, until suddenly it isn't, and you find out at the worst possible time.

The stakes aren't academic. Borrill's paper ties this category mistake to AI training runs wasting somewhere between 12% and 43% of compute budgets, plus documented corruption incidents in PostgreSQL, etcd, and MySQL traced back to fsync failures. Financial systems lose billions annually to the same root cause. Not a fun sentence to write twice, but there it is.

The correct four-step protocol, and what each step is doing

The real safe-write sequence has four steps, not two, and most bugs live in the steps people skip.

1. Write your data to a temp file.
2. fsync() the temp file.
3. rename(temp, target).
4. fsync() the parent directory.

Step 2 exists because, without it, your data is sitting in OS buffers, not on disk. The rename can succeed perfectly and your new data can still vanish on crash, because rename() moved a name, not the bytes underneath it.

Step 4 is the one almost everyone forgets. It's the one that bites hardest. Without fsyncing the parent directory, the rename itself can be lost. The directory entry update sits in a buffer just like anything else, and a crash before that buffer flushes means the rename never happened as far as the disk is concerned. Someone in the Bhayani thread nailed it: "the real final boss is the parent directory fsync." Everyone remembers to protect their data. Almost nobody remembers the directory pointing at it needs the same protection.

Ext4's Fast Commit feature is basically a confession that this protocol matters. It exists specifically because applications need explicit barriers to get correct behavior out of the filesystem. If atomicity just happened automatically, Fast Commit wouldn't need to exist. Its presence is Borrill's argument made physical: correctness comes from executing a protocol, not from some built-in instant state transition.

Git's implementation is a clean example of doing this right. New objects get written to .git/objects before anything references them, so a crash between writing a blob and updating a ref just leaves a harmless orphan sitting around, unreferenced and inert. For ref updates, Git uses a lockfile pattern: create <ref>.lock with O_CREAT|O_EXCL (which also catches competing writers trying the same thing), write the new value, then atomically rename() it over the real ref file, with a signal handler standing by to clean up abandoned lockfiles.

Even that pattern has a ceiling, though. It only holds as long as every layer underneath behaves. Borrill points out that cloud-synchronized storage, iCloud being the obvious example, interposes an eventually-consistent replication layer that quietly defeats the mutual exclusion the lockfile was built to guarantee. The lockfile still runs. It just isn't protecting anything anymore.

Multi-file transactions and the risk of dual writes producing inconsistent results

Rename gives you exactly one atomic operation: swapping a single name. Real software rarely changes just one thing at a time. If you change five files and crash after three, your directory is now half old, half new, and nothing about a single rename() call is going to save you from that.

Zylos Research framed the sharpest version of this as the dual-write issue. Most "file" systems actually store two separate things in two separate places: a database row describing the file, and the bytes of the file itself. Those two stores don't share a transaction manager. A crash between updating one and updating the other isn't some rare edge case, it's a statistical certainty once you're operating at scale.

And the two ways this fails are not equally bad. "File copied, metadata not yet updated" is self-healing. A migration just looks incomplete and restarts. Fine. "Metadata says migrated, file missing" is the nightmare version: silent, user-facing, and it shows up as a permanent 404 for a record that swears the resource exists. Nobody's health check catches that. A customer does, usually at the worst time.

The open-source fstx library (2025 to 2026) tackles this by applying changes as no-replace renames in waves, each wave separated by an fsync barrier, then writing a durable COMMITTED marker at the end. Recovery finds entries by identity rather than by name, and unwinds anything unfinished in reverse, using the same barrier discipline going backward. The project validated this across roughly 1.3 million recovery runs, each one checked against a strict rule: state is exactly before the transaction or exactly after it, never caught in the middle.

SquirrelFS, published in ACM Transactions on Storage in November 2025 (LeBlanc, Taylor, Bornholt, Chidambaram), attacks the same problem from the filesystem side. It's a persistent-memory filesystem, and it uses Rust's typestate pattern to check update orderings at compile time, catching a whole category of bug before the code even runs. It also adds something clever: a "rename pointer" field on directory entries, where the destination entry's pointer references the physical location of the source. That's enough information to finish a rename cleanly after a crash. Without such guarantees, a crash mid-rename can leave the filesystem in an inconsistent state, which defeats the entire point of calling it atomic.

Zylos' recommended fix for the dual-store issue, more broadly, is to stop pretending it's a filesystem issue at all and model it as an explicit state machine instead: a persisted status field, gated by preconditions that are idempotent and safe to re-enter. The database row becomes the durable log of intent. It's the transactional outbox pattern, just applied to files instead of message queues.

A concrete example appears in the harmoniqs/amicode GitHub project (issue #1021, September 2026). Its extension updater used to write files without atomic guarantees, which goes sideways mid-crash. The fix treats the entire dist directory as one deployment unit, writes a pending-swap marker file before the swap starts, and removes it only once the swap completes successfully. If the process restarts and finds a stale marker sitting there, it can unwind the incomplete operation. The crash-recovery logic lives in that marker file on disk, not in the process's memory, which is the whole point: memory doesn't survive a crash, and marker files do.

Object storage's historic absence of atomic rename and the software-level workarounds it forced

S3, in its original form, only ever offered PUT, GET, and DELETE. No atomic move. No atomic rename of a directory, because there was never really a "directory" to begin with, just objects whose keys happened to share a prefix. Renaming a "folder" meant copying every object one at a time and deleting the old copies one at a time, with no atomicity tying the whole operation together.

Hadoop's entire commit model assumed rename and delete were atomic, so job output would show up all-or-nothing. That assumption held fine on HDFS. It fell apart on S3, where a rename could fail halfway through and leave callers with no safe way to treat it as a single commit step.

Delta Lake ran headfirst into this. On HDFS, atomic rename naturally resolves conflicts between competing writers. On S3, that primitive doesn't exist, so Delta Lake on S3 needs a DynamoDB lock service to stop two writers from committing the same transaction log version at the same time. That's a whole distributed locking system, built from scratch, purely to stand in for a filesystem feature that was never there.

Apache Iceberg solved the same problem differently, with optimistic concurrency instead of locks. A writer reads the current pointer, stages a new metadata file somewhere safe, and then attempts an atomic compare-and-swap on the pointer. Lose the race, and the writer just retries against whatever the latest snapshot turned out to be. The payoff is atomic commits and snapshot isolation on a single table: readers see the old state or the new state, and nothing in between, ever.

What makes this elegant is that the protocol doesn't care what's underneath it. It works with database transactions in a JDBC catalog, conditional PutObject calls in a REST catalog, or plain atomic rename on HDFS. Iceberg tracks the whole table through a tree of metadata files, manifest lists, and manifests pointing at the actual data, which is how it delivers atomic commits and snapshot isolation, readers see the old state or the new state, never anything in between.

None of that complexity is free, and none of it is really optional either. Every major lakehouse format ended up independently reinventing software-level atomic commit, because the storage layer underneath never gave them one. That's the tax charged for a missing primitive, paid separately by every format that needed the feature, not a design flourish.

How object storage gained atomic rename and what it still cannot guarantee

S3 Express One Zone shipped a RenameObject API in June 2025, and it's a genuinely different capability, not a workaround dressed up as one. It turns what used to be a multi-step, copy-then-delete dance into a single API call, inside the same directory bucket. No data actually moves. Renaming a 1-terabyte log file takes milliseconds instead of the hours a copy-based approach would need. It landed in AWS SDK version 2.31.66, and Hadoop filed HADOOP-19589 to wire in support. This only works inside S3 Express One Zone directory buckets, not on general-purpose S3.

Amazon S3 Files, announced in April 2026, goes further by pairing an EFS caching tier with S3 Standard as the durable backing store. Full POSIX semantics, rename included, along with chmod, symlink, and atomic writes, get handled immediately at the EFS tier, with changes written back to S3 Standard asynchronously behind the scenes. It speaks NFS v4.1/v4.2 (per the AWS launch), supports up to 25,000 concurrent NFS connections, and the published performance numbers from AWS are genuinely large: aggregate read throughput in the terabytes-per-second range, up to 250,000 read IOPS per filesystem, 1 to 5 GiB/s aggregate write throughput, up to 50,000 write IOPS, and a per-client read ceiling of 3 GiB/s.

When a file changes, S3 Files waits about a minute, batching up successive edits, before pushing anything to the S3 bucket. When a file changes, S3 Files waits about a minute, batching up successive edits, before pushing anything to the S3 bucket. Hammer the same file with rapid writes and they land as a single S3 PUT instead of spawning a fresh object version for every tiny change. Efficient, but it means "written" and "durable in S3" are not the same instant.

And here's the nuance that actually matters for crash-recovery thinking: S3 objects are immutable. They never supported atomic rename at the object layer, full stop. When a file gets renamed in S3 Files, the system writes a new object under the new key and deletes the old one, and for a directory rename, it repeats that dance for every object under that prefix. The atomic rename semantics are delivered at the POSIX/EFS layer. They do not propagate atomically down into the S3 backing store itself.

A Hacker News thread from launch day (April 8, 2026) summed up why this keeps happening across the entire industry: "The hardest part in building a distributed filesystem is atomic rename. It's always rename. Scalable metadata filesystems, Colossus, Tectonic, ADLSv2, HopsFS, are either designed around how to make rename work at scale or how to work around it at higher levels in the stack." That's just the shape of the problem, not a knock on any one system. That's just the shape of the problem.

There's a real cost argument for all this plumbing, too. For workloads where most of the data is cold, which describes a lot of training datasets outside their active research window, S3 Files runs around 40% cheaper than EFS alone, because the S3 backing tier bills at object-storage rates and only the hot working set pays filesystem prices.

The gap left standing at the end of all this is the exact same gap the whole piece started with, just relocated to cloud scale. POSIX atomicity at the EFS layer and atomicity at the S3 object layer are two different properties wearing the same word. Knowing that difference, and building around it instead of assuming it away, is what separates storage code that survives a crash from storage code that just gets lucky until it doesn't.

Atomic rename in AI training: checkpoint integrity at the scale where failure is expensive

A training checkpoint is model weights, optimizer state, the learning-rate schedule, RNG seeds, and step counters. It's model weights, optimizer state, the learning-rate schedule, RNG seeds, and step counters, everything required to resume a run, roll it back, or audit what happened at a given step. Lose any piece of that and "resume training" quietly turns into "start a forensic investigation."

Most frameworks write these checkpoints with plain file operations that don't guarantee durability. Crash mid-write and one of two bad things happens: silent corruption, where the checkpoint loads just fine and then produces NaN gradients because the data underneath is garbage, or a torn file that just crashes outright on load. Silent corruption is worse, honestly, because at least a crash on load tells you something's wrong.

Scale is what makes this expensive instead of merely annoying. MLPerf's checkpoint benchmark puts a 70-billion-parameter model's checkpoint at 912 gigabytes. A trillion-parameter model's checkpoint runs 15 terabytes. Those aren't rounding errors sitting on disk, waiting patiently for an atomic write protocol to protect them. They're enormous files, and every one of them is a single point of failure when the write protocol underneath is sloppy.

Run the math on what a failure actually costs. A single GPU preemption or NCCL timeout at hour 47 of a 60-hour fine-tuning run, with no checkpointing in place, torches the entire run and every dollar of compute spent getting there. Do checkpointing correctly, with a real crash-safe protocol, and that same failure costs a resume from the last known-good checkpoint and a loss measured in minutes, not days.

Atomic rename specifically guards against two failure modes here, no more, no less. Torn writes, where a partial checkpoint loads without complaint but holds corrupted data underneath. And lost checkpoints, where a crash lands after rename() succeeds but before the parent directory gets fsynced, and the checkpoint that should exist simply isn't there after reboot, gone as if it never happened.

CheckFreq, presented at FAST'21, is proof this doesn't have to be a tradeoff between safety and speed. It does per-iteration checkpointing with only about 3.5% overhead, showing that "crash-safe" and "fast enough to run constantly" aren't opposites fighting for the same budget. They're just two requirements that, done right, sit comfortably next to each other, the same way the temp-file-fsync-rename-fsync protocol was always meant to work in the first place: unglamorous, a little tedious to implement fully, and the entire reason the checkpoint is still there when everything else on the machine has already forgotten what happened.