Decentralized Training of Foundation Models in Heterogeneous Environments

1 Introduction

• Foundation-model training is costly, while decentralized GPUs may be underused and cheaper to operate than equivalent cloud devices.The introduction gives a GPT3-175B training-cost estimate and describes volunteer GPU resources as a potential alternative.Source: §1 · pp. 1–2
• Prior decentralized work focuses on data parallelism, insufficient when model parameters exceed one device’s capacity, or on alternative architectures evaluated at smaller computational scales.The paper instead focuses on a standard GPT-style architecture without changing model architecture or convergence behavior.Source: §1 · p. 2
• The central scheduling problem is assigning micro-batch-and-layer tasklets to devices connected by slow, heterogeneous networks.Heterogeneity greatly expands the search space; even finding the best pipeline strategy corresponds to a hard open-loop traveling-salesman problem.Source: §1 · p. 2
• The proposed communication-cost model separates data-parallel cost into balanced graph partitioning and pipeline-parallel cost into joint graph matching and a traveling-salesman problem.An efficient evolutionary algorithm searches for an allocation under this model.Source: §1 · p. 3
• 3.8–4.8× faster end-to-end than state-of-the-art systems when training GPT3-1.3B across 64 GPUs in eight regions.The reported computation and convergence dynamics do not differ across regimes; the prototype is also 1.7–3.5× slower than data-center Megatron/DeepSpeed despite a network up to 100× slower.Source: §1 · p. 3
• The system also uses optimizations to hide communication within computation and reduce the impact of slow connections.The paper reports ablations examining the scheduler and system optimizations individually.Source: §1 · p. 3

2 Decentralized Training of Foundation Models: Problem Formulation

• Devices may have different pairwise delays and bandwidths; model layers are divided into consecutive stages, and training data into parallel micro-batches.The setting uses both pipeline and data parallelism: pipeline parallelism processes stages across devices, while data parallelism requires gradient synchronization.Source: §2 · pp. 3–4
• A tasklet is the forward and backward computation for one training-data micro-batch at one pipeline stage in an iteration.The scheduler assigns each tasklet to a device to maximize decentralized training throughput.Source: §2 · p. 3
• The scheduler seeks an assignment of distinct tasklets to devices that maximizes throughput by accelerating communication among tasklets with high communication volumes.The formulation represents the communication network with device nodes and edges labeled by average latency and bandwidth.Source: §2 · pp. 3–4
• The optimization goal is to minimize communication cost, with separate costs for pipeline-parallel activation transfers and data-parallel parameter or gradient transfers.The formulation uses pipeline and data parallel degrees whose product equals the total number of devices.Source: §2 · p. 4

3 Scheduling in Heterogeneous Environments

• Distinct communication delays and bandwidths in fully heterogeneous networks can make the scheduling search space grow exponentially.The paper contrasts this with homogeneous data-center assumptions that constrain the search space.Source: §3 · p. 4

3.1 Overview of the scheduler

• The scheduler minimizes data- and pipeline-parallel communication costs by forming balanced data-parallel groups before optimizing their pipeline layout.The objective is a modified balanced graph-partition problem, for which the paper uses an evolutionary algorithm.Source: §3.1 · pp. 4–5

3.2 Modelling data parallel communication cost

• Data-parallel groups exchange sharded gradients under a cost model bounded by the slowest device, while parallel groups make total cost depend on the slowest group.The stated model assumes a colocated sharded parameter server for within-group communication.Source: §3.2 · pp. 5–6

3.3 Modeling pipeline parallel communication cost

• Pipeline cost combines matching same-micro-batch devices across groups with choosing a group order, modeled as an open-loop traveling-salesman problem.The coarsened graph represents communication costs between groups; edge costs use the maximum cost over a perfect matching.Source: §3.3 · p. 6

3.4 Searching via hybrid genetic algorithm

• A hybrid genetic algorithm creates offspring from candidate balanced partitions, applies local search, and retains improved partitions in the population.The paper uses this approach because the balanced graph-partition problem is NP-hard and the objective is complex.Source: §3.4 · pp. 6–7
• Greedily applying the traditional Kernighan–Lin gain does not reduce training communication cost, so the scheduler uses a tailored local search focused on fast links.Its rationale is that removing a fast within-group link tends not to change data-parallel cost, while the link may be useful for pipeline matching after a move.Source: §3.4 · p. 7
• The local-search swaps can be extended from pairs to circular sequences using the new gain function.The paper describes this extension as following the circular Kernighan–Lin procedure.Source: §3.4 · p. 7

3.5 Other System Optimizations

• Three CUDA streams assign receiving, computation, and sending slots so communication overlaps computation, which can be fully hidden in the communication-bound decentralized scenario.The slots handle activations and backward gradients between neighboring pipeline stages.Source: §3.5 · p. 7

4.1 Experimental Setup.

• The five-case evaluation uses 64 V100 GPUs with measured AWS region-pair bandwidth and latency; cases include on-demand and spot data centers and two-campus links.The on-demand case uses 8 nodes with NVLink and 25 Gbps inter-node bandwidth; the spot case mixes 4- and 1-GPU nodes with 10 Gbps inter-node bandwidth.Source: §4.1 · p. 8
• Geo-distributed cases model four US regions and eight worldwide regions, with cross-region links spanning 10–250 ms latency and 0.3–1.3 Gbps bandwidth worldwide.Within-region links are set to 5 ms and 2 Gbps in both cases.Source: §4.1 · p. 8
• Evaluation uses GPT3-1.3B, varying layers across 24, 32, and 40 and batch sizes across 1024, 2048, and 4096; Megatron and Deepspeed settings are grid-searched.Systems are compared by throughput, expressed as PFLOPS; the optimal baseline settings vary by case.Source: §4.1 · p. 8

4.2 Results

• In worldwide geo-distributed cases, the system achieves a 4.8× speedup over Megatron and 3.6× over Deepspeed; in all other cases, it is 1.2–2.5× faster.Source: §4.2 · p. 8
• With larger batches and deeper models, the performance gap to centralized systems narrows because these changes do not increase the respective communication costs.The passage attributes this pattern to data-parallel communication not increasing with batch size and pipeline-parallel communication not increasing with layers per device.Source: §4.2 · pp. 9–10
• The scheduler provides up to 2.7× speedup over random assignment, and its local-search strategy significantly outperforms Kernighan–Lin.Both local-search strategies outperform random assignment.Source: Figure 3 · p. 9, §4.2 · p. 10, Figure 4 · p. 10

Foundation models.

• Megatron and ZeRO distribute foundation-model training in data centers, but their collective communication paradigms can perform poorly over slow, heterogeneous connections.Megatron uses AllReduce for tensor-parallel activations; ZeRO uses ScatterGather to dispatch sharded parameters for layer-wise data parallelism.Source: §5 · p. 10

Decentralized optimization.

• Decentralized optimization commonly synchronizes gradients among neighboring workers, while Learning@Home and DeDLOC adapt training implementations for open collaborative environments.The passage also describes Varuna as supporting GPT training on cloud-provider spot instances, with limited attention to heterogeneous connections.Source: §5 · p. 10

Volunteer computing.

• Volunteer computing has long distributed intensive tasks across open environments, including Folding@home protein-dynamics simulations and Learning@Home mixture-of-experts training.BOINC is cited as an early example of the broader volunteer-computing approach.Source: §5 · p. 10

• The paper studies foundation-model training on decentralized devices connected over heterogeneous networks and proposes a scheduler to assign pre-training tasklets.Source: §6 · p. 10
• In the worldwide geo-distributed scenario, the proposed scheduling algorithm achieves a 4.8× speedup over prior state-of-the-art training systems.Source: §6 · p. 10
• The authors argue that decentralization may shift power positively, while also necessitating new governance structures for responsible foundation-model development and deployment.Source: §6 · p. 10

• Cheaper computational resources could reduce foundation-model training budgets and increase accessibility for small research and commercial institutions.Source: §7 · p. 15
• The authors also warn that decentralization and democratization may reduce control of cheaper computing resources and accelerate foundation-model risks.Source: §7 · p. 15
• The authors call for community engagement on governance questions around decentralized foundation-model training.Source: §7 · p. 15

8 Limitation and Future Work

• The scheduler assumes homogeneous GPU resources, although devices with different memory budgets and compute power may participate.The authors suggest splitting tasklets into smaller pieces and assigning different numbers of pieces to different GPU types.Source: §8 · p. 15
• The scheduling algorithm has room for improvement, including exploration of reinforcement learning as a future direction.The authors also say they want to strengthen their argument about end-to-end speedup.Source: §8 · p. 15
• Handling dynamic GPU joins and departures and providing fault tolerance remain open system questions.Periodic model checkpointing is suggested as a possible approach to fault tolerance.Source: §8 · p. 15

9 Anatomy of the Current ML Systems for Foundation Model Training

• Megatron combines tensor model, pipeline, and data parallelism, while ZeRO stage-3 shards parameters and gathers them as needed for forward and backward computation.ZeRO stage-3 also uses ReduceScatter to synchronize gradients after backward computation.Source: §9 · pp. 15–16
• Megatron and ZeRO rely on collective communication, which demands high bandwidth and leaves them sensitive to stragglers on heterogeneous links.The paper notes that tensor model parallelism is recommended only within a DGX server with high-bandwidth NVLinks.Source: §9 · p. 16
• Pipeline parallelism uses point-to-point communication and can improve compute utilization by placing more computation within each stage.Pipeline parallelism alone scales to only a limited number of stages, so it must be combined with data parallelism to scale across many GPUs.Source: §9 · p. 16
• The paper frames careful scheduling of pipeline tasks as a way to use fast links and avoid slow links in decentralized environments.The communication pattern creates opportunities for assigning work across devices with heterogeneous connections.Source: §9 · p. 16

10.1 Multiple Execution of the Benchmark

• Across the five scenarios, the study repeated benchmarks three times, and execution variance in each setting stayed within 5%.The scheduler used the same scheduled layout across its three executions; unscheduled runs used three random layouts.Source: §10.1 · p. 16

10.2 Tuning of Megatron and Deepspeed

• Megatron’s optimal configuration used tensor model parallelism only in Case 1; the other scenarios selected DTP = 1, DPP = 8, and DDP = 8.In Case 1, the selected values were DTP = 2, DPP = 4, and DDP = 8.Source: §10.2 · p. 17
• ZeRO-S3 ran much longer than Megatron and the proposed system; combining Deepspeed pipeline parallelism with ZeRO-S1 outperformed ZeRO-S3 in most Case 1 settings and all Case 2–5 settings.The reported Deepspeed results use the better of ZeRO-S3 and the pipeline-plus-ZeRO-S1 combination.Source: §10.2 · p. 17, Figure 5 · p. 17

10.3 Network Benchmark.

• For the regional and worldwide scenarios, the authors measured cross-region NCCL delay and bandwidth using AWS instances connected through a VPN.The measurements were summarized for Case 4 and Case 5 in Tables 1 and 2.Source: §10.3 · p. 17

10.4 Other Presentation of Experimental Results

• Figure 6 supplements Figure 3 with per-iteration runtimes, while Tables 1 and 2 present delay and bandwidth for the regional and worldwide scenarios.The table captions specify NCCL measurements for Case 4 regional and Case 5 worldwide distribution.Source: §10.4 · p. 17, Table 1 · p. 18, Table 2 · p. 18

10.5 Deployment on FluidStack

• The FluidStack deployment used 32 A40 GPUs across US Mid and US East, with measured inter-cluster bandwidth of 3.73–3.87 Gbps.The cross-region delay was 21.8±0.3 ms in each direction.Source: §10.5 · p. 18
• 27.4%, 26.4%, and 29.7% of cluster peak FLOPS were obtained for GPT3-1.3B, GPT3-6.7B, and GPT3-13B, respectively.The GPT3-1.3B run used 40 layers and a 4K batch size; the GPT3-6.7B and GPT3-13B runs used batch size 1024.Source: §10.5 · p. 19, Figure 7 · p. 20