What Is Checkpointing? How Large-Scale LLM Training Recovers from GPU Failures
"We ran a large-scale LLM training job for 54 days, and it was interrupted 466 times."
This actually happened during Meta's Llama 3 training. According to the published 54-day log, the training job was interrupted 466 times in total, 419 of which were unexpected failures. Of those unexpected interruptions, 58.7% were GPU-related, while software, network, and storage issues also brought training to a halt.
In an earlier article, What Is DiLoCo? Distributed LLM Training That Reduces Dependence on Ultra-Fast Networks, we looked at how changing the synchronization structure between workers can reduce the impact a single worker's failure has on the overall training run.
So in a typical distributed training environment, what determines the point from which training can resume after a failure?
The answer is checkpointing.
What Is Checkpointing?
Checkpointing is a technique for periodically saving the current progress of a training run. If a failure occurs, instead of starting training over from scratch, you can load the most recently saved checkpoint and resume from that point.
Why Checkpointing Is Necessary in Large-Scale GPU Training
A single GPU may have a low probability of failure. But when thousands of GPUs run simultaneously for months, the odds that something will go wrong somewhere in the cluster rise quickly.
Failures arise from a wide range of causes: not just GPU and server hardware, but software errors in CUDA and NCCL, network communication failures, storage and file system issues, and rack-level power or switch failures.
In synchronous distributed training involving thousands of GPUs, a single one of these issues can bring the entire job to a halt — and without checkpointing, days or weeks of training results could be lost to a single failure.
What Does a Checkpoint Actually Store?
Today, the training state of an LLM isn't stored on a single GPU or in a single file. Model and optimizer states are split across many GPUs depending on tensor parallelism, pipeline parallelism, data parallelism, ZeRO, and other strategies. This means the distributed state across GPUs must be saved at a consistent training step and restored as a single coherent state for training to resume properly.
A typical checkpoint includes the following information:
Model weights: parameters updated through training
Optimizer state: information needed for the next update, such as Adam's momentum and variance values
Training step and learning rate scheduler state: the current training stage and the learning rate in effect
Mixed-precision training state: loss scale, FP32 master weights, and similar values
Random number and data loader state: information needed to restore data order and processing position
A checkpoint isn't simply a model file — it's a complete bundle of state needed to resume training from a specific point in time.
Synchronous Checkpointing: The GPU Stops While Saving
The simplest approach is to pause training at fixed steps and write the entire state to disk or a network file system. This is called synchronous checkpointing.
In the synchronous approach, all GPUs reach the same step and then stop computation. The next training step can only begin once the save is complete.
As models grow larger, the amount of data that needs to be saved grows too. Adam-family optimizers in particular require momentum and variance values in addition to model weights, so for large models the total checkpoint size can range from hundreds of gigabytes to several terabytes.
Saving checkpoints more frequently reduces the amount of training lost to a failure, but increases GPU idle time. Saving less frequently reduces idle time, but means more training must be redone after a failure.
Asynchronous Checkpointing: Saving and Training Overlap
Asynchronous checkpointing emerged to reduce this idle time.
In the asynchronous approach, the GPU's training state is first snapshotted to CPU memory, and the work of writing it to disk or a network file system overlaps with the next training step. The GPU doesn't have to wait for the write to persistent storage to finish before continuing computation.
DataStates-LLM proposed copying model and optimizer state from GPU to CPU memory during windows in the forward and backward passes when that state isn't changing. The researchers reported that, in certain experimental settings, this sped up checkpointing by up to 48x and improved end-to-end training performance by up to 2.2x.
That said, the asynchronous approach doesn't eliminate save costs entirely. Because it shares GPU-CPU bandwidth, CPU memory, and network and storage resources with the training job, some performance interference remains.
The goal of asynchronous checkpointing isn't to eliminate save time — it's to minimize the amount of time the GPU stalls because of saving.
In-Memory and Tiered Checkpointing: Cutting Recovery Time Too
More recently, approaches that use CPU memory have been researched and adopted not just to reduce the training interruption caused by checkpointing, but to reduce recovery time after a failure as well.
In-memory checkpointing doesn't write every checkpoint to slow persistent storage. Instead, it first saves to the CPU memory of the host server. Because memory is faster than disk, state can be saved more frequently and loaded back more quickly.
However, if a checkpoint is only stored in the current node's memory, it disappears along with the checkpoint if that server itself fails. Tiered checkpointing was developed to address this.
Recent tiered architectures split storage location based on the scope of the failure:
Process failure: recover from the current node's CPU memory
Node failure: recover from a peer node's memory
Rack-level failure: recover from remote persistent storage
The idea is to save checkpoints frequently to the fast memory tier, and less frequently to the slower but safer persistent storage tier.
The TierCheck study proposed keeping lightweight checkpoints in local and peer memory while asynchronously writing full checkpoints to remote storage. According to the researchers' evaluation, this reduced checkpointing time to under 10 seconds for a 20-billion-parameter model. In experiments with a 40-billion-parameter model, it supported per-step checkpointing with roughly 10–15% overhead.
That said, this is early research published in 2026, so actual performance may vary depending on the model and infrastructure configuration.
How Often Should You Save a Checkpoint?
There's no single right answer for checkpoint frequency. Saving too often increases I/O and network load; saving too rarely increases the amount of training that must be recomputed after a failure.
Assuming failures occur evenly between checkpoints, saving every 30 minutes means, on average, about 15 minutes of training would need to be redone. Shortening the interval to 5 minutes reduces lost training time, but the save operation competes with training for resources more often.
The save interval should be set based on several factors together:
How often failures occur in the cluster
Checkpoint save and load time
Checkpoint size
Storage and network throughput
The acceptable window of training loss
The goal isn't the shortest possible interval — it's finding the interval that minimizes the combined cost of checkpointing overhead and post-failure recomputation. In ultra-large-scale environments, a tiered strategy — saving frequently to CPU memory and less frequently to persistent storage — tends to work best.
Comparing Checkpointing Methods at a Glance
Synchronous Checkpointing | Asynchronous Checkpointing | In-Memory / Tiered Checkpointing | |
|---|---|---|---|
Core approach | Halts training and writes to persistent storage | Overlaps saving with training | Saves hierarchically across memory and persistent storage |
Key trait | Simple to implement and manage consistency | Reduces GPU idle time during saves | Reduces rollback window and recovery time |
Storage location | Disk / network storage | CPU memory and persistent storage | Local / peer memory and persistent storage |
Advantage | Simple structure | Reduces training interruption | Fast recovery under frequent failures |
Caveat | GPU idles during saves | Memory and network resource contention | Requires managing memory usage and replication |
Checkpointing Alone Isn't Enough
Checkpointing solves the question of where to resume training after a failure. But actually reducing downtime requires failure detection and root-cause analysis, removing unhealthy nodes, and redeploying and restarting training on healthy resources together.
The TRANSOM case illustrates this well — it combined checkpointing with failure detection and automatic restart. The researchers reported that integrating checkpointing with anomaly detection and automatic recovery reduced total training time by 28% in a GPT-3 175B pretraining evaluation. In other words, no matter how fast checkpoints are saved, recovery keeps getting delayed if failures are detected late or if it's unclear which GPUs are healthy.
This is where checkpointing strategy in the training framework connects with GPU infrastructure operations. AIPub continuously monitors GPU utilization and resource state, and helps identify anomalies across GPUs and nodes in a cluster. Operators can quickly pinpoint the resources causing trouble, select healthy resources, and decide how to redeploy and restart the training job.
AIPub collects the real-time status of individual GPUs and nodes across the cluster, catching anomaly signals — such as temperature, memory errors, and communication latency — early. It automatically flags nodes with issues and separates them from healthy resources, so operators can check status instantly on a dashboard instead of tracing the root cause node by node.
This healthy-resource information ties directly back into checkpointing strategy. If tiered checkpointing answers "which memory tier to recover from," AIPub helps confirm "whether that tier is actually healthy," increasing confidence in the restart decision.
If checkpointing answers "where to resume from," the GPU infrastructure operations layer helps determine "what went wrong, and which resource to resume on."
Conclusion: If You Can't Prevent Failures, Recover Fast
Eliminating failure entirely in large-scale GPU training is close to impossible.
Checkpointing is a technique for preserving training state so that a failure doesn't force you to start over from scratch. The synchronous approach is simple to implement but stalls the GPU during saves. The asynchronous approach overlaps saving with training to reduce idle time. The in-memory and tiered approach splits storage location to cut recovery time after a failure as well.
That's why, in large-scale AI infrastructure, how often you save checkpoints matters just as much as how quickly you can detect failures and resume training on healthy resources.
The competitiveness of large-scale AI infrastructure doesn't come from building an environment where failures never happen. It comes from minimizing the training loss and downtime when failures do happen, and building a structure that gets back to a healthy state quickly.
Talk to TEN's experts about a checkpointing strategy and failure response system suited to your organization's GPU training environment.
References
Grattafiori et al., “The Llama 3 Herd of Models,” arXiv, 2024
Wu et al., “TRANSOM: An Efficient Fault-Tolerant System for Training LLMs,” arXiv, 2023
Maurya et al., “DataStates-LLM: Lazy Asynchronous Checkpointing for Large Language Models,” arXiv, 2024
Shen et al., “Fault-Tolerant Hybrid-Parallel Training at Scale with Reliable and Efficient In-memory Checkpointing,” arXiv, 2023
Han et al., “TierCheck: Tiered Checkpointing for Fault Tolerance in Large Language Model Training,” arXiv preprint, 2026