Checkpoint Storage for Distributed Training on S3
Optimizing checkpoint writes cuts GPU idle time and unlocks cheaper training infrastructure.
Optimizing checkpoint writes cuts GPU idle time and unlocks cheaper training infrastructure.
GPU stalls cost more than storage when data moves slowly between tiers.
Understanding why files are three separate layers, not one.
Skipping a costly data copy by pointing directly at kernel memory makes mmap blazingly fast.
Append-only logs let databases promise speed and safety without compromising either.
Working set size determines whether your reads and writes hit memory speed or disk speed.
Linux offers two incompatible locking mechanisms that don't see each other on local filesystems.
Atomicity at the filesystem level doesn't guarantee data survives crashes.
Understanding POSIX's three non-negotiable guarantees keeps your code from silently corrupting data.
Write-ahead logging lets filesystems recover from crashes without scanning the entire disk.
Systems use adaptive prefetching to predict what data you need before you ask for it.