Metadata Bottlenecks in Distributed Filesystems

You buy faster drives, add more nodes, and upgrade the network, but the training cluster still stalls. The reason has nothing to do with bandwidth and everything to do with metadata: the small records that tell a system where a file lives, who can touch it, and what state it's in. Before any data moves, a client has to clear a metadata check first. A client asking to read or write a file first has to contact the metadata service, confirm permissions, and get the file's location. None of the data bandwidth matters until that round trip finishes.

That round trip is where things slow down. Traces from Baidu AI Cloud, documented in the SwitchFS paper (EuroSys '26), show metadata operations making up anywhere from two-thirds to nearly all filesystem requests, and metadata dominates how long I/O takes to complete. Data bandwidth scales in a fairly forgiving way: add more data servers, get roughly proportional gains. Metadata doesn't scale that way, because the namespace itself is hierarchical. You can't just throw more servers at that dependency chain and expect it to untangle itself.

None of this is an engineering oversight. Filesystems promise two things: durability (an operation's effect is visible to the next operation immediately) and atomicity (updates never land half-finished). If a distributed system wants to keep those promises, it needs synchronous transactions, with full coordination before the operation returns. So every file create, every rename, every attribute change has to carry that coordination overhead on its critical path. More storage nodes don't lift that ceiling, because the ceiling was never about storage capacity. It was about how many servers have to agree with each other before one small operation is allowed to finish.

AI training workloads and the metadata bottleneck

AI training is close to the worst-case input you could design for a synchronous metadata system. Every one of those actions is a metadata operation, and AI workloads produce them by the truckload.

Small files make the problem worse in a specific, measurable way. When the actual data payload is tiny, the metadata round trip dominates the total time spent on I/O. Pile on concurrency: AI clusters run hundreds of nodes at once, many of them creating and opening files inside the same shared directories. That's the exact pattern that breaks synchronous directory-update systems hardest, a skewed, hot-spot workload hammering the same small set of metadata operations from every direction at once. FFSlim's research into training I/O points to the same culprits: storage redundancy, overwhelming small-file counts, layouts that fight the cache, and index structures heavy enough to become a workload of their own.

Checkpointing is where this becomes an expensive concern rather than an abstract one, because it starts costing real money. Every N training steps, every GPU rank writes out its weights, its optimizer state, its RNG state, and the entire cluster sits frozen until those writes land. Either way, the cluster just sat idle while the write finished. GPUs billing by the hour, idling while a filesystem finishes bookkeeping, is the clearest financial argument for taking metadata seriously that exists in this entire discussion.

The three structural mechanisms that produce metadata contention at scale

Metadata bottlenecks don't come from one cause. They come from three mechanisms that stack on top of each other, and no single fix addresses all three at once.

The first is synchronous transaction overhead. SwitchFS identifies this as the root cause: because every metadata write has to be immediately durable and atomic, the write has to wait for servers to reach consensus before it can return control to the caller. So that coordination tax applies to every create, every rename, every attribute update, no matter how small the operation is.

The second is directory hot-spot contention. Multi-tenancy makes it worse: when several training jobs share a directory namespace, you get exactly the skewed access pattern where contention peaks hardest.

The third is the partitioning dilemma, and it's the one with no clean answer. SwitchFS frames this honestly: choosing a partitioning granularity is a trade-off between load balance and operation overhead, and you can't maximize both at the same time.

POSIX semantics are the thread tying all three mechanisms together. Loosen the atomicity or visibility guarantees and coordination costs would drop. Applications are written assuming those guarantees hold, so when you break them quietly, you break the software built on top.

Production systems' response to the bottleneck at scale

Organizations that hit this wall in production didn't just tune a config file and move on. They redesigned where metadata lives, and in each case the answer involved pulling state out of the client.

DeepSeek took a different route when it open-sourced the Fire-Flyer File System, known as 3FS, in February 2025. Built to take advantage of modern SSDs and RDMA networking, 3FS pools storage from many nodes into one unified namespace reachable through a standard filesystem interface. Its answer to metadata contention: file creation, lookups, and attribute management run on stateless Meta nodes, and those nodes hand off durability and consistency to an external FoundationDB cluster. No metadata state lives on the compute nodes. In a production deployment running 200 Gbps InfiniBand across 180 storage servers, 3FS hit an aggregate read bandwidth of 6.6 TiB/s.

Huawei's FalconFS, deployed in 2024 and published at NSDI '26, faced one of the more demanding metadata environments on record: hundreds of petabytes in flight, over 300 billion small files, spread across billions of directories. FalconFS's answer was a stateless-client architecture, with all metadata state kept server-side, and its evaluations show throughput gains of up to 12.81× for deep learning model training compared with CephFS and Lustre.

Three different organizations, three different systems, one shared conclusion: metadata state doesn't belong on the compute node. Moving it into dedicated infrastructure, be it a stateless meta layer, an external transactional store, or a decoupled staging pipeline, eases the bottleneck. That conclusion is also exactly where the newest research picks up the argument.

What recent research proposes beyond distributing metadata servers

Distributing metadata servers more widely treats a symptom. The newer research argues you need to remove synchronous coordination from the server altogether, and in one case, push it into the network hardware itself.

SwitchFS, presented at EuroSys '26 (April 27 to 30, 2026, in Edinburgh, Scotland), starts from an observation that sounds obvious once it's stated: directories are rarely read the instant after they're updated. SwitchFS lets metadata operations return early and defers the actual directory update until a read comes in, hiding latency and batching work that used to happen one operation at a time.

The hard part is keeping POSIX semantics intact despite that asynchrony, and this is where SwitchFS does something genuinely different: it co-designs with a programmable network switch. The results back up the bet: up to 13.34× higher throughput and 61.6% lower latency than existing distributed filesystems under skewed workloads, and a 21.1× improvement in end-to-end throughput over CephFS on real-world workloads.

HMFS, published at SoCC '25, attacks a narrower but related target: the ordered indexing most metadata services rely on to support directory listings. HMFS decouples the directory listing interface applications expect from the underlying indexing structure, so a system no longer has to pay for strict ordering just to support a listing call.

Both projects land on the same deeper claim. The bottleneck was never the sheer volume of metadata; it's the overhead of the coordination protocols that enforce consistency, and you can rebuild those protocols without throwing POSIX semantics out the window.

Parallel filesystems: mounting S3, GCS, R2, and Azure Blob as one filesystem

Production parallel filesystems each make a different bet on that partitioning trade-off, and you pay for those bets once the workload is live.

Lustre splits metadata servers (MDS) from object storage servers (OSS), letting clients skip the MDS once they know where a file lives. It runs on most TOP500 supercomputers, and because it's open-source, licensing cost stays at zero. Its documented weak point is the classic one: the metadata server becomes the bottleneck at scale. Running it well takes dedicated expertise, though managed versions exist through AWS FSx for Lustre, Google Cloud Managed Lustre, and Azure Managed Lustre.

IBM Storage Scale, the system formerly called GPFS, distributes metadata across multiple nodes, eliminating any single central server. Among the four systems compared here, it shows the strongest small-file performance, and it comes with policy-based tiering to object storage built in.

BeeGFS runs metadata servers horizontally, with no single active MDS to create a choke point; scaling metadata out is as simple as adding more servers. Its vendor ecosystem is smaller than Lustre's at the extreme high end, though it's showing up more often in production AI clusters.

The systems built around dedicated MDS nodes inherit hot-spot risk, and the systems that spread metadata across every node trade that risk for a higher coordination cost per operation. Neither choice is free. When sizing any of these systems, the checkpoint write budget deserves attention before raw read bandwidth does, since checkpoint stalls are the most visible symptom of a metadata shortfall.

What object storage's metadata model adds

Object storage gave up the filesystem hierarchy entirely in exchange for flat, key-addressed objects reachable through HTTP-based APIs. That trade bought geographic replication and some of the lowest cost per gigabyte you can get at scale, and that's a big part of why object storage has become the default home for pre-training corpora.

Rich metadata comes at a latency cost. Object storage handles metadata in a way that typically carries too much latency, so you can't get full flash-level performance out of it. Metadata becomes a selection tool that filters what gets read.

At the lakehouse layer, Apache Iceberg tackles a related problem for tabular data, organizing metadata into a hierarchy (table metadata, manifest lists, manifest files) that keeps query planning fast even across billions of files, without scanning the entire namespace.

What object storage doesn't solve is POSIX compliance and random-write support. Any AI workload that needs shared, mutable access to a common directory tree, checkpointing, iterative training, agent workspaces that multiple processes read and write at once, can't run directly against object storage without something in between translating one model into the other.

Cloud-agnostic metadata unification as the next pressure point

Buyers phrase this problem as how to mount S3, GCS, R2, and Azure Blob as one filesystem without copying data between them. Modern AI pipelines rarely sit on a single storage system. Each of those has its own metadata namespace, and as pipelines spread across more backends, that fragmentation becomes a bottleneck no single system can fix by itself, no matter how well-optimized it is.

Moving data between these systems to force one unified view brings back the data-movement bottleneck distributed storage was supposed to eliminate. Unifying metadata across heterogeneous backends without copying the underlying data is the more useful answer. That means providing one consistent set of file and object semantics without inserting a gateway into the middle of every I/O path, across clouds and storage vendors.

That ideal isn't fully solved industry-wide. You end up building custom pipelines to bridge all these separate namespaces, and that's engineering effort spent moving data around instead of training models; if you stop building that kind of plumbing, and stop waiting on downloads before training or inference can even start, you tend to move noticeably faster.

Archil addresses this directly by mounting existing S3, GCS, R2, or Azure Blob buckets as an actual POSIX filesystem, with an NVMe cache layer absorbing the latency that would otherwise sit on the critical path. Agents and training jobs see one consistent metadata surface no matter which object store is actually holding the bytes underneath, and none of it requires an ETL pipeline or a copy of the data. The bucket stays the source of truth throughout.

Practical criteria for evaluating metadata architecture before a workload outgrows its storage

Checking metadata architecture before the bottleneck appears avoids costly rework later. By the time idle GPU time is measurable on a dashboard, the storage decision has already been made, and changing it gets expensive fast.

Start with file count and size distribution. A dataset with billions of small files, which is normal for LLM pre-training corpora and for autonomous driving datasets, will hit contention on any architecture with a centralized or coarse-grained metadata server well before compute ever saturates. Formats like FFSlim bundle small files into unified containers, so you cut the metadata operation count at the source, and that takes pressure off the filesystem before it ever sees the data.

Look at the concurrency pattern next. Multi-tenant setups, where several training jobs share one namespace, make it worse; giving each job its own namespace partition reduces contention without needing a different filesystem.

Set a checkpoint write budget. Define how long the cluster can sit idle during a checkpoint write and size storage to hit that number. It's the single clearest metric for whether metadata and data throughput can keep up, since idle time during a checkpoint write is a direct readout of inadequate storage.

Check whether clients are stateful or stateless. Check whether the system under evaluation can run stateless clients instead.

Confirm POSIX completeness. Archil's filesystem layer keeps full POSIX semantics on top of object storage backends for this exact reason, so existing training code, agent runtimes, and HPC tooling run without rewrites.

Finally, check backend agnosticism. Binding metadata management to one vendor's storage backend creates migration cost down the road and limits where cache nodes can sit relative to compute across clouds. Favor architectures where S3, GCS, R2, and Azure Blob all mount the same way, with no special cases and no separate code path depending on whose bucket is holding the data.