FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression

• FusionLLM enables individuals or entities to collaboratively train LLMs on geo-distributed GPUs while communicating activations and gradients rather than raw data.This design aims to mitigate hardware scarcity and protect data privacy to some degree.Source: §1 · pp. 1–2
• Decentralized training must address remote automatic differentiation, heterogeneous software and hardware, stragglers, and low-bandwidth communication.The reported network range is 10 Mbps∼10 Gbps, making communication a major throughput bottleneck.Source: §1 · p. 2
• OP-DAG represents operators and dependencies independently of low-level frameworks, while a runtime executor partitions sub-DAGs and implements remote automatic differentiation.This abstraction allows local devices to use different ML frameworks as execution engines.Source: §1 · p. 2
• OP-Fence uses workload and communication estimates to cluster similarly connected devices and partition the DAG to reduce iteration time.The estimator models each OP-Node’s computational workload and each OP-Data’s communication time.Source: §1 · p. 2
• FusionLLM real-world experiments across three clusters with 48 heterogeneous GPUs connected by 8 Mbps–10 Gbps Internet and Ethernet outperformed baseline methods by 1.45–9.39×.Source: §1 · p. 2

2.1 LLM Training

• GPT-3’s 175B parameters create extreme computational and memory demands, requiring 13.17 years on H100 or 60.28 years on RTX 4090 for pre-training.The cited estimates motivate more feasible approaches to training large models.Source: §2.1 · p. 3

2.2 Distributed Training

• Data, pipeline, and tensor parallelism each face scalability constraints involving batch optimization, communication, dependency, or homogeneous bandwidth requirements.Pipeline parallelism communicates intermediate results between stages, while tensor parallelism requires frequent communication and high bandwidth homogeneity.Source: §2.2 · p. 3

2.3 Motivation, Challenges and Opportunities

• Decentralized GPUs offer an underused computational reservoir, and collectively harnessing them may expand access to large-model training.The motivation includes potentially improving utilization, reducing costs, and broadening participation.Source: §2.3 · p. 3
• Software heterogeneity and hardware variability create remote differentiation, partitioning, and straggler challenges across participating devices.Differences in GPU and CPU architectures, memory, bandwidth, and compute capacity can produce unequal task completion times and extended waiting.Source: §2.3 · pp. 3–4
• Low Internet bandwidth can dominate throughput, motivating heterogeneous inter-layer partitioning and compression of activations or gradients.Inter-layer partitioning can reduce communication touchpoints, while sparsification can reduce transferred data volumes.Source: §2.3 · p. 4

3.1 Design Goals

• FusionLLM targets customized models, heterogeneous software, flexible scheduling, and selective communication compression in decentralized GPU training.Its design goals include supporting different ML frameworks and model architectures while avoiding indiscriminate compression and manual workload estimation.Source: §3.1 · pp. 4–5

3.2 System Overview

• The broker converts user-provided model and training definitions into an FP/BP operator-DAG, partitions it into sub-DAGs, and assigns them to computing devices.CompNodes execute operators, communication, and training procedures through interfaces that accommodate heterogeneous devices and ML frameworks.Source: §3.2 · p. 5

3.3 DAG Partition

• DAG edges encode operator dependencies, while the executor transfers activations during forward propagation, gradients during backpropagation, and applies configured optimizers after gradients are computed.Sub-graphs identify required inputs and communicate outputs or gradients when dependent operators reside on external CompNodes.Source: §3.3 · p. 6

3.4 OP Data Structure

• OP-Data standardizes operator messages with fields for provenance, users, gradient requirements, iteration and micro-batch state, and compression metadata.These attributes support dependency tracking, backpropagation, pipeline synchronization, and compression handling.Source: §3.4 · p. 6

3.5 Estimation of Computation and Communication

• FusionLLM estimates operation time as computation, parent-data retrieval, and memory-writing costs, using FLOPs divided by actual device speed and an alpha-beta communication model.Actual computation speed is adjusted through warmup profiling, while cross-device retrieval uses communication latency and message size.Source: §3.5 · pp. 6–7

3.6 Estimation of Workloads and Througput

• The workload model assigns sub-DAGs to CompNodes and estimates sequential or pipelined throughput from computation and cross-device communication costs.The analysis considers operator parallelism, sub-graph execution ranges, pipelined batches, and the simplifying assumption that local read/write costs are negligible.Source: §3.6 · p. 7

4 OP-Fence

• The scheduler addresses computation, memory, and communication balance because each affects utilization, bottlenecks, and throughput.Source: §4 · pp. 7–8
• Scheduling and partitioning are formulated as a throughput optimization problem, with forward-DAG scheduling standing in for the structurally reversed backward DAG.The approach assumes each device has sufficient CPU and disk memory to load and run its assigned sub-DAG.Source: §4 · p. 8
• DNN operator DAGs usually have low degree, often forming sequential layer chains, which provides structure for partitioning.Source: §4 · p. 8, Figure 4 · p. 8
• OP-Fence detects high-bandwidth device clusters and assigns each cluster a connected operator sub-graph to reduce low-bandwidth transfers.It uses the Louvain algorithm to identify clusters and partitions the OP-DAG accordingly.Source: §4 · p. 8

5.1 Basic Communication Compression

• Top-K sparsification transmits the largest-magnitude gradient values while leaving other entries zero after decompression.This reduces transmitted data by communicating selected values and their indexes.Source: §5.1 · p. 9, Figure 7 · p. 9

5.2 Adaptive Communication Compression

• Adaptive compression assigns stronger compression to lower-bandwidth links to trade communication time against convergence more selectively.The method computes a new inter-node ratio from the user-specified ratio and estimated original communication time.Source: §5.2 · p. 9
• The adaptive ratio depends on estimated original communication time, and its coefficient 3 accounts for transmitting float32 values and int64 indexes.Source: §5.2 · p. 9
• With large n_b, adaptive compression does not change the decrease rate of the bottleneck term with r, supporting high-n_b pipeline-parallel training.Source: §5.2 · p. 9

6 Implementation

• The current system uses Python, PyTorch, torchtext, and HuggingFace Transformers, with 11.3K Python lines and about 0.5K C++ lines in its core.Source: §6 · p. 9
• GPU peer-to-peer communication uses N2N and MPI collectives, while future work targets direct connections and faster-link selection.Source: §6 · p. 9
• A CUDA-level TopK sparsification library is used to reduce compression time relative to the PyTorch TopK operation.Source: §6 · p. 9
• FusionLLM supports customized model definitions, allowing users to compose models from framework and custom operators such as PyTorch Conv and ReLU layers.The implementation illustrates this interface with a two-layer CNN.Source: §6 · p. 9

7.1 Experimental Setup

• The evaluation uses heterogeneous GPU testbeds and measures averaged per-iteration latency together with training-loss convergence for ResNet18, ResNet101, and GPT2-XL.The experiments include two GPU clusters and four testbeds, with communication heterogeneity characterized across devices.Source: §7.1 · p. 9, §7.2 · pp. 9–10, Figure 9 · p. 10

7.2 End-to-End System Comparisons

• Baselines include no compression, naive Top-k compression, equal-number partitioning, and equal-computation-cost partitioning.Source: §7.2 · p. 10

7.3 Convergence of Training

• Adaptive TopK preserves convergence more closely than uniform TopK for ResNet-18 and ResNet-101, while both compression methods do not harm GPT2-XL convergence.GPT2-XL with both compression methods even shows improved convergence in the reported experiment.Source: §7.3 · p. 10, Figure 8 · p. 10

7.4 System performance

• Communication, rather than computation, is the bottleneck for GPT2-XL: less than 0.5 seconds of propagation contrasts with about 20 seconds to transmit 20 MB at 1 MB/s.Equal-computation partitioning reduces latency only slightly, while equal-number partitioning has the highest latency.Source: §7.4 · p. 11, Figure 10 · p. 11
• Both uniform and adaptive TopK substantially reduce latency relative to no compression, but uniform TopK can harm convergence and therefore increase total training time.Because bandwidth is heterogeneous, uniform TopK does not always outperform adaptive TopK by a large margin.Source: §7.3 · p. 10, §7.4 · p. 11

7.5 Effect of Compression Ratio

• Increasing the compression ratio to 1000 may fail to reduce latency proportionally because message latency and scheduling overhead can become bottlenecks.Source: §7.5 · p. 11

• The current communication implementation routes all peer-to-peer traffic through a remote N2N server, limiting available bandwidth utilization.Source: §8 · p. 11
• The system assumes network stability, but packet loss, latency variation, connection interruptions, and hardware faults may severely affect training.Source: §8 · p. 11
• The paper does not thoroughly analyze geographically distributed data-transmission costs or incentives for multiple entities to contribute resources.Source: §8 · p. 11
• FusionLLM establishes a proof-of-concept for geo-distributed training of DNNs and LLMs despite unresolved framework challenges.Source: §8 · p. 11

9.2 Edge DL

• Edge-learning research emphasizes inference or small-model settings, whereas FusionLLM partitions large-model training across devices with differing topologies.Source: §9.1 · pp. 11–12, §9.2 · p. 12

9.1 Decentralized DL

• Existing decentralized systems train larger models such as GPT-2, but do not jointly address heterogeneous GPUs, collaboration variability, software and hardware compatibility, and general DL tasks.Source: §9.1 · p. 12

9.3 Incentive Mechanism

• Decentralized-training incentives must handle unknown client arrival and departure, competing participation choices, and malicious clients seeking disproportionate rewards.Source: §9.3 · p. 12

9.4 Graph Partition

• Graph partitioning jointly balances load and communication cuts, an NP-hard objective addressed by prior heuristic and bandwidth-aware methods.Source: §9.4 · p. 12

• FusionLLM uses OP-DAG to support model customization, remote automatic differentiation, and diverse software environments across geo-distributed GPUs.Source: §10 · p. 12
• The decentralized approach is presented as a proof-of-concept that could support more inclusive, scalable, and efficient machine learning.Source: §10 · p. 12