What Is DiLoCo? Distributed LLM Training Without Ultra-Fast Networks
"Can GPUs that aren't even in the same data center train one model together?"
The industry has spent years focused on building faster, lower-latency networks — NVLink, InfiniBand, RoCE, Spectrum-X.
Google DeepMind asked the opposite question.
What if, instead of making the network faster, you just drastically cut how often nodes need to communicate in the first place?
That question is where DiLoCo (Distributed Low-Communication training) started.
This article looks at how DiLoCo cuts communication requirements, how Streaming DiLoCo and Decoupled DiLoCo later improved on its limitations, and what this approach means for multi-cluster and neocloud GPU operations.
Why Distributed Training Needed Ultra-Fast Networks
In typical synchronous data-parallel training, each GPU processes a different data batch, then exchanges and synchronizes its computed gradients at the end of every training step.
The standard method for this is All-Reduce.
All-Reduce: a collective communication operation where multiple GPUs exchange and sum the values they each computed, then send the same combined result back to every GPU.
How long one training step takes depends on the model and batch size. But because GPU-to-GPU communication happens every single step, network latency and bandwidth have a direct impact on overall training performance. That's why large-scale synchronous training is typically built by connecting GPUs within a single data center over a high-bandwidth, low-latency network.
Conversely, GPUs spread across multiple data centers, or connected by a relatively slow network, are hard to use for a single synchronous training job.
DiLoCo changes exactly this assumption: that every worker has to synchronize at every single training step.
What Is DiLoCo? Distributed Training with Less Frequent Synchronization
In typical synchronous distributed training, GPUs share their computed results at the end of every single training step. In DiLoCo, multiple workers train independently for a period of time, and results are only gathered afterward.
Think of it like several teams starting from the same draft, each editing a portion independently, and then merging all the changes at once into a new shared version.
The DiLoCo training cycle works like this:
Deploy the global model to multiple workers.
Each worker runs several rounds of local training on its own data.
Once local training finishes, compute the pseudo-gradient — how much the model changed.
Each worker sends its pseudo-gradient to the outer optimizer.
The outer optimizer aggregates updates from all workers and updates the global model.
The updated model is redeployed to each worker.
Repeat.
Here, a worker is a unit that runs local training independently. It could be a single GPU or TPU, or a small cluster of several accelerators tightly connected internally.
Inside each worker, an inner optimizer handles the actual training. DiLoCo's flagship experiments used AdamW.
After local training finishes, each worker computes the difference between the model it started with and its current model. This delta is called the pseudo-gradient.
The outer optimizer aggregates pseudo-gradients from all workers and updates the global model. The flagship experiments used Nesterov Momentum.
Where conventional synchronous distributed training communicates every step, DiLoCo only communicates between workers after hundreds of local training steps.
In experiments on the C4 dataset with 8 workers, Google DeepMind reported performance comparable to fully synchronous training while cutting the number of worker-to-worker communications by 500x.
In other words, DiLoCo's core idea isn't making the network faster — it's cutting how often workers need to talk to each other in the first place.
Streaming DiLoCo: Cutting Peak Bandwidth and Communication Stalls
DiLoCo reduced communication frequency, but one problem remained. At the moment communication did happen, workers still had to exchange the model's entire set of parameters all at once — and sit idle, paused, while that happened.
Streaming DiLoCo, released in 2025, addressed this in three ways:
1. Synchronizing parameters in parts
Instead of sending one giant file all at once, send it in ordered pieces. Conventional DiLoCo exchanges the entire model's update at the moment of synchronization. Streaming DiLoCo splits the model's parameters into several parts and synchronizes each part sequentially. Because the whole model isn't transmitted at once, the peak network bandwidth required at any instant is lower.
2. Overlapping communication with training compute
Instead of waiting for a full transfer to finish, keep working while the transfer is in progress. In Streaming DiLoCo, a worker can keep training on the remaining parameters while some parameters are being synchronized. Overlapping communication and compute reduces the time workers spend waiting on synchronization and shortens overall training time.
3. Quantizing the transmitted data
Represent the same information in fewer bits to shrink the data actually transmitted. Representing the update values exchanged between workers at lower precision reduces the size of data that needs to be sent. The Streaming DiLoCo paper showed experimentally that combining these three improvements can cut required inter-worker bandwidth by up to roughly 100x on billion-parameter-scale models, while maintaining training quality comparable to the original approach.
Decoupled DiLoCo: Asynchronous Training That Doesn't Wait on Slow Workers
In April 2026, Google DeepMind released Decoupled DiLoCo, taking the idea one step further.
Both DiLoCo and Streaming DiLoCo cut communication volume significantly, but they were still fundamentally synchronous. At each synchronization point, every worker had to wait until all workers were ready. That meant if one worker slowed down or failed, it could delay training for everyone else too.
Decoupled DiLoCo solves this by splitting the training job into multiple independently running Learners.
Each Learner runs local training at its own pace, without waiting on any other Learner. Portions of the trained parameters are sent asynchronously to a central Synchronizer. The Synchronizer aggregates arriving updates into the global model, and routes around any Learner that's slow or has failed, so the rest of training doesn't stop.
Learner: a compute unit that runs local training independently.
Synchronizer: the central coordination layer that aggregates model updates from multiple Learners into the global model.
In a simulation Google DeepMind published across 8 data centers (5B model, 95% utilization), conventional data parallelism required roughly 198 Gbps of inter-data-center bandwidth, while Decoupled DiLoCo used roughly 0.84 Gbps.
In a large-scale simulation with artificially induced hardware failures (~1.2 million chips), Goodput — the share of compute effectively used for training — held at 88% for Decoupled DiLoCo. Conventional data parallelism dropped to 58% even with elasticity applied, and to 27% with no failure handling at all. In experiments on Gemma 4-family models, final average ML benchmark performance was comparable between the two — 64.4% versus 64.1%.
Decoupled DiLoCo goes beyond simply cutting communication volume — it evolves into a structure that isolates the impact of slow workers and hardware failures from the rest of training.
DiLoCo vs. Streaming DiLoCo vs. Decoupled DiLoCo
DiLoCo | Streaming DiLoCo | Decoupled DiLoCo | |
|---|---|---|---|
Core improvement | Sync workers after hundreds of local steps | Partial sync, overlapped communication/compute, quantization | Async training via independent Learners and a central Synchronizer |
Mainly reduces | Communication frequency | Peak bandwidth and communication stalls | Worker waiting, failure propagation, idle compute |
Sync model | Synchronous | Synchronous, streamed | Asynchronous |
Best-fit scenario | Multiple compute islands connected by low bandwidth | Environments where sending the full model at once is impractical | Heterogeneous, multi-region environments with meaningful failure risk |
※ The three approaches aren't mutually exclusive — Streaming DiLoCo and Decoupled DiLoCo each build a different improvement on top of DiLoCo's basic structure.
What DiLoCo Means for GPU Infrastructure Operations
Until now, the industry has pursued faster network fabric as the way to raise GPU cluster scaling efficiency. The DiLoCo family proposes a different answer: instead of making the network faster, redesign the training algorithm to need the network less.
That said, DiLoCo doesn't eliminate the need for network design or GPU operations. Even with less frequent inter-worker communication, you still need periodic synchronization, model state management, failure handling, and data placement. And if you're using multiple data centers and compute islands together, you still have to place training jobs based on each site's GPU availability, performance differences, and network conditions.
In particular, in environments where multiple workloads — training and inference alike — share a single GPU infrastructure, overall cost efficiency comes down to which resources get assigned to which job and how idle resources are used.
Ultimately, even as DiLoCo-family techniques cut inter-worker communication requirements, you still need an operations layer that places GPU resources and tracks usage across multiple sites and workloads. AIPub is built to manage GPU resources across sites in a unified way, allocate resources and permissions by team and project, and monitor infrastructure usage.
Start GPU Infrastructure Operations with AIPub
Even as DiLoCo-family techniques reduce communication requirements and open up the possibility of combining multiple compute islands into a single training job, the problem of actually placing and managing GPU resources across your whole organization remains.
AIPub partitions GPU resources into fine-grained units so multiple training and inference workloads can efficiently share physical resources, and separates resource and access permissions by team and project to reduce interference between workloads.
It also tracks infrastructure metrics like GPU utilization, memory, and interconnect status in real time, so you can see from an infrastructure standpoint which resources are idle and where bottlenecks are occurring.
Conclusion: Cutting the Bottleneck with Algorithms, Not the Network
DiLoCo, Streaming DiLoCo, and Decoupled DiLoCo are three attempts at solving the same problem from different angles. DiLoCo cuts communication frequency. Streaming DiLoCo lowers peak bandwidth and communication stalls. Decoupled DiLoCo makes worker synchronization asynchronous to isolate the impact of failures.
This approach won't immediately change every organization's GPU infrastructure strategy. But if GPUs are still hard to acquire and building a single ultra-fast cluster isn't practical, the mere possibility of combining resources scattered across multiple sites into one training job becomes an important option to have.
Let's work through how to design and operate your organization's GPU infrastructure across multiple sites and workloads, together with TEN's experts.
References
Douillard et al., "DiLoCo: Distributed Low-Communication Training of Language Models," arXiv, 2023
Douillard et al., "Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch," arXiv, 2025
"Decoupled DiLoCo for Resilient Distributed Pre-training," arXiv, 2026
Google DeepMind, Decoupled DiLoCo announcement (April 23, 2026)