PaperlayerView exact revision on arXiv

Source-linked AI summary

Democratizing LLM Training with the Edge: Challenges, Systems, and the Road Ahead

Ziyue Luo, Jiaxuan Cai, Cedric LeDenmat, Srijith Nair, Fatemeh Nourzad, Rohith Krishnan, Qinhang Wu, Jifan Zhang, Sungjae Lee, Zhe Li, Peiwen Qiu, Siddharth Shah, Rishabh Sharma, Sundararajan Srinivasan, Yinglun Xia, Xue Zheng, Zidong Liu, Bicheng Ying, Kaushik Chowdhury, Gauri Joshi, Yingbin Liang, Robert Nowak, Srinivasan Parthasarathy, Saurav Prakash, Balaraman Ravindran, Sanjay Shakkottai, Ness B. Shroff, Haibo Yang, Aylin Yener, Jia Liu

arXiv:2610.32215v2cs.DCcs.LG

TL;DR

Centralized LLM development depends on large-scale compute and data, while relevant private data and infrastructure are unevenly available. The paper presents a shared cloud-parameter-server architecture for distributed pretraining and fine-tuning, and reports stable convergence in geo-distributed pretraining experiments.

  • Problem

    Centralized LLM training relies on massive data and compute, while private data governance and infrastructure costs constrain participation by smaller institutions.

    Source: §I · p. 1

  • Method

    A cloud-hosted parameter server coordinates heterogeneous clients and reuses one synchronization, compression, and fault-tolerance stack across pretraining and fine-tuning.

    Source: §I · p. 2

  • Results

    The design converges stably and reduces wall-clock loss faster than baselines, while DiLoCo diverges quickly under its recommended hyperparameters.

    Source: 2) Paradigm 1: Federated Pretraining · p. 9

  • Takeaways & Limitations

    The reported trajectories indicate that distributed pretraining designs need to account explicitly for heterogeneous, geo-distributed conditions.

    Source: 2) Paradigm 1: Federated Pretraining · p. 9

    Supporting elastic membership still requires WAN-aware failure detection and state reconciliation for clients that leave, rejoin, or arrive during training.

    Source: B. Fault Tolerance under Elastic… · p. 11

Abstract

from arXiv · show

The growing computational power of modern GPU data centers has let a few industrial giants train and deploy large language models (LLMs) on massive amounts of data. Yet the sheer scale of these models and their training data makes centralized development increasingly hard to sustain, and at the same time shuts out smaller institutions that hold relevant private data but lack comparable infrastructure. In this paper, we organize our positions on distributed LLM training around the pretraining and fine-tuning paradigms, covering federated learning, split learning, and decentralized approaches, and we assess their readiness for geo-distributed settings with heterogeneous hardware and unreliable connectivity. We then present a {\em unified} system architecture built around a cloud-hosted parameter server that coordinates heterogeneous clients over outbound gRPC connections, reusing one communication, compression, and fault-tolerance stack for both pretraining and a family of fine-tuning methods. We validate the design with real geo-distributed experiments on GPT-2 Medium and Llama3-1B pretraining across multiple university sites, showing stable convergence under realistic conditions. Finally, we chart the road ahead, identifying open problems that span frontier model support, elastic fault tolerance, adaptive compression, federated alignment, privacy and trust, incentive design, and system--algorithm co-design, all of which must be addressed to make democratized LLM training a reality.

I. INTRODUCTION

The paper frames democratized LLM training as a response to centralized compute and data constraints, then identifies the system challenges of training across institutions. It proposes a shared cloud-parameter-server architecture and surveys methods and open problems for distributed pretraining and fine-tuning.

I. INTRODUCTION

  • Centralized LLM development concentrates compute, limits independent review, raises trust and security concerns, and carries environmental costs.Private data governance and the expense of hundreds of high-end GPUs also constrain participation by smaller institutions.Source: §I · p. 1
  • Geo-distributed training must handle heterogeneous GPUs and software stacks, variable WAN links, and institutional firewalls, NATs, and network policies.Identical work can make faster GPUs wait for stragglers, while incompatible numerical formats risk divergence; WAN variability and restrictive network policies further complicate coordination.Source: §I · p. 2
  • The paper surveys distributed LLM training by pretraining versus fine-tuning and by synchronous, asynchronous, and infrequent-synchronization coordination.It assesses how existing methods handle compute, network, and statistical heterogeneity in geo-distributed, WAN-constrained settings.Source: §I · p. 3
  • A cloud-hosted parameter server and heterogeneous clients reuse periodic delta synchronization, quantization-based compression, and fault-tolerant rejoin across pretraining and fine-tuning.Clients connect over outbound gRPC; the shared architecture also uses asynchronous aggregation, bounded staleness, and fault-tolerant communication to address system challenges.Source: §I · pp. 2–3
  • Preliminary GPT-2 Medium and Llama3-1B pretraining results show stable convergence and competitive wall-clock efficiency under realistic WAN conditions.The paper reports that the architecture’s building blocks are reused by federated pretraining and fine-tuning paradigms without modification.Source: §I · p. 3
  • The paper identifies open problems in frontier and multimodal models, elastic fault tolerance, adaptive compression, federated alignment, privacy and trust, incentive design, and system-algorithm co-design.It also makes the unified training framework implementation publicly available.Source: §I · p. 3

II. BACKGROUND: DISTRIBUTED LLM TRAINING

Distributed LLM training spans pretraining and fine-tuning, which share optimization techniques but differ in their computational and communication requirements. Those differences affect system choices such as compression, pipelining, and exchanged messages.

II. BACKGROUND: DISTRIBUTED LLM TRAINING

  • The section groups existing distributed LLM training methods into pretraining and fine-tuning, while noting that both use techniques such as federated and split learning.Their optimization processes and computation and communication requirements differ.Source: §II · p. 3
  • Federated and split learning techniques recur across both categories, but their differing requirements mean the same system choices do not automatically apply to each.Source: §II · p. 3
  • Training category shapes system choices, including compression, whether models are pipelined or split, and which network messages are exchanged.Source: §II · p. 3

A. Preliminaries

The preliminaries establish parameter-server coordination and synchronization models, then explain how compute, network, and statistical heterogeneity interact in WAN-scale learning. They situate prior methods by the forms of heterogeneity they address and their communication assumptions.

A. Preliminaries

  • A parameter server maintains model parameters while workers pull them, compute updates, and push them for aggregation.Its coordinator-based pattern fits dynamic membership, heterogeneous workers, and restricted or asymmetric connectivity.Source: A. Preliminaries · p. 3
  • Synchronous training uses a barrier and common parameter snapshot, but wall-clock progress is limited by the slowest worker and communication path.Asynchronous updates improve utilization under stragglers but can be stale; bounded-staleness models limit that lag, while infrequent synchronization lowers coordination cost but risks greater drift.Source: A. Preliminaries · pp. 3–4
  • Federated learning uses synchronous rounds or asynchronous updates, while infrequent synchronization amortizes communication through more local computation between exchanges.These choices trade tighter synchronization and reduced staleness against waiting, utilization, and drift-prone updates.Source: A. Preliminaries · p. 4
  • Heterogeneity spans compute, network, and non-i.i.d. client data, and the forms interact because slower devices and weaker links create stale updates whose effects depend on data differences.The section distinguishes systems-focused methods such as FedProx, TiFL, Oort, and FedScale from methods targeting statistical heterogeneity, including SCAFFOLD, FedNova, and MIME.Source: A. Preliminaries · p. 4

B. Distributed LLM Pretraining

The section describes pretraining as next-token learning over large text corpora and reviews collaborative alternatives to centralized training. Federated and decentralized approaches keep data local, but current systems still need to address heterogeneous resources and geography.

B. Distributed LLM Pretraining

  • Pretraining teaches an LLM to generate text autoregressively from large text corpora using cross-entropy loss.The formulation averages next-token loss over passages and their token positions.Source: B. Distributed LLM Pretraining · p. 4
  • Centralized pretraining requires large datasets and model parameters, often distributed across hundreds of high-performance devices for hours or days.Data collection and curation add overhead to the centralized pipeline.Source: B. Distributed LLM Pretraining · p. 4
  • Federated and decentralized federated learning let participating devices train locally in parallel and aggregate models through a parameter server or decentralized coordination.DiLoCo is identified as a close point of comparison, using Nesterov momentum as its outer optimizer and time-based synchronization.Source: B. Distributed LLM Pretraining · p. 5
  • Federated and decentralized training avoid collecting and centralizing data because the data remain locally held.Federated learning is also motivated by privacy, and some algorithms have theoretical convergence rates that improve with more participating devices.Source: B. Distributed LLM Pretraining · p. 5
  • Split learning can let lower-compute, lower-memory clients train only the model’s initial layers while the server handles the remaining processing.Source: B. Distributed LLM Pretraining · p. 5
  • Many systems address a single concern, while an ideal system would adapt to system and data heterogeneity and geographic layout through automatic model splitting or pipelining and network-aware load balancing.Source: B. Distributed LLM Pretraining · p. 5

C. Distributed LLM Fine-tuning

Fine-tuning adapts pretrained LLMs to downstream tasks, with parameter-efficient methods updating a smaller representation while freezing the rest. The section describes LoRA and highlights additional constraints for federated and reinforcement-learning fine-tuning.

C. Distributed LLM Fine-tuning

  • Fine-tuning adapts a pretrained model to downstream tasks through supervised learning or reinforcement learning, updating either all parameters or a parameter-efficient subset.In parameter-efficient fine-tuning (PEFT), a lower-dimensional representation is trainable while the remaining parameters stay frozen, reducing gradient size and memory needs.Source: C. Distributed LLM Fine-tuning · p. 5
  • LoRA represents weight updates as low-rank factors, with lifting matrices B_i and projection matrices A_i forming the trainable parameters.The rank r is typically much smaller than the dimensions m and n of the weight matrix.Source: C. Distributed LLM Fine-tuning · p. 5
  • Federated fine-tuning may require model splitting or pipeline parallelism when edge devices cannot store or process an entire LLM.Designs must also account for system, data, and model heterogeneity and random client dropouts.Source: C. Distributed LLM Fine-tuning · p. 5
  • Reinforcement-learning fine-tuning supports multi-turn generation tasks, including human-value alignment, reasoning, tool-calling agents, and retrieval-augmented generation.Such systems often require distributed RL across multiple devices.Source: C. Distributed LLM Fine-tuning · p. 6

III. A UNIFIED SYSTEM FOR GEO-DISTRIBUTED LLM TRAINING

The system provides shared infrastructure for geo-distributed pretraining and fine-tuning, with design goals addressing heterogeneous GPUs, variable WAN links, and client-side firewall or NAT constraints. Its architecture centers on a cloud parameter server coordinating clients through outbound connections.

III. A UNIFIED SYSTEM FOR GEO-DISTRIBUTED LLM TRAINING

  • The system's shared infrastructure provides communication, synchronization, compression, and fault tolerance for both pretraining and fine-tuning, while each paradigm configures it for its own objectives.The section presents preliminary evaluation results as validation under realistic geo-distributed conditions.Source: §III · p. 6

A. Design Goals

  • The design goals are to keep heterogeneous GPU clusters working without stragglers, handle WAN latency and bandwidth variation, and let clients connect outbound and rejoin quickly after preemption.The rejoin goal calls for a single remote procedure call to the parameter server.Source: A. Design Goals · p. 6

1) Architecture Overview:

  • The architecture uses a cloud-hosted parameter server to hold global parameters and optimizer state while a master client bootstraps the model and campus or edge clients perform local updates.Figure 1 depicts the master upload and worker synchronization and recovery with the parameter server.Source: 1) Architecture Overview · p. 6, Figure 1 · p. 6
  • The design moved from pipeline parallelism, which required direct peer-to-peer links between clusters, to a centralized parameter server because institutional firewalls and NAT make those links difficult to implement reliably.In the pipeline design, each stage used data parallelism with AllReduce within its cluster.Source: 1) Architecture Overview · p. 6, Figure 2 · p. 6
  • The parameter server aggregates client deltas with FedAvg and sends updated global snapshots over outbound client connections, avoiding complex inbound firewall configuration.Centralization reduces connectivity from O(N^2) peer links to O(N) client-to-server links and supplies a single point for TLS encryption and access control.Source: 1) Architecture Overview · pp. 6–7
  • An optional multi-PS mode groups clients, typically by geographic proximity, and synchronizes the cluster servers with ring AllReduce to balance load and mitigate geography-related latency.Each cluster PS coordinates local training and periodically aggregates client updates.Source: 1) Architecture Overview · p. 7
  • Clients use FSDP workers with micro-batch sizes and mixed precision adjusted to local compute and memory; they send model deltas after configurable wall-clock intervals rather than every training step.Clients keep optimizer states locally, and the synchronization interval controls WAN load and update staleness.Source: 1) Architecture Overview · p. 7

2) Communication Protocol:

The system uses gRPC to accommodate institutional network constraints, compresses gradient payloads to reduce traffic, and supports recovery through live parameter-server state and local disk checkpoints.

2) Communication Protocol:

  • The system replaced mesh-oriented PyTorch RPC with gRPC because dynamic ports and many client connections conflicted with institutional firewall policies.Persistent bidirectional streaming over a single fixed, outbound client port supports ongoing model-update exchange without repeatedly establishing connections.Source: 2) Communication Protocol · p. 7

3) Network Compression:

  • Gradient compression quantizes 32-bit values to 8-bit or 16-bit integers, then applies Huffman encoding; flattening tensors reduces quantization-pair overhead and supplies more values for encoding.The respective quantization compression factors are 4 and 2; the receiver decodes and dequantizes the payload.Source: 3) Network Compression · p. 7

4) Fault Tolerance and Rejoin Logic:

  • After a dropped stream or client relaunch, the client authenticates and fetches the latest global model and saved optimizer state from the parameter server, typically resuming in roughly 10 seconds.Disk checkpoints are written after each synchronization; cold recovery takes about one minute, and the manager retains the ten checkpoints with the lowest evaluation loss, refreshed every 1000 steps.Source: 4) Fault Tolerance and Rejoin Logic · p. 7

5) Protocol Walk-through:

The protocol initializes and distributes a model through the parameter server, then synchronizes client updates and supports client recovery. A shared meta-algorithm accommodates pretraining and several fine-tuning methods by distributing and aggregating trainable parameters.

5) Protocol Walk-through:

  • A master client initializes the model, uploads it to the parameter server, and the server distributes it to workers before periodic client training and update synchronization.Clients send quantized model deltas; the parameter server dequantizes them, applies FedAvg, and returns the updated global model.Source: 5) Protocol Walk-through · p. 7
  • After preemption or a crash, a client reauthenticates with a one-time token, retrieves the latest global model and its saved optimizer states, and resumes training.This recovery flow is identified as part of the system protocol.Source: 5) Protocol Walk-through · pp. 7–8

C. Level 2: Application Paradigms

  • Pretraining and fine-tuning build on the same Level 1 infrastructure while configuring it for their respective objectives.Figure 3 presents the system as shared building blocks at Level 1 and application paradigms at Level 2.Source: C. Level 2: Application Paradigms · p. 8, Figure 3 · p. 8

1) A meta-algorithm for distributed LLM training:

  • The meta-algorithm applies to distributed pretraining and fine-tuning with LoRA, Hybrid LoRA, RAVAN, or zeroth-order optimization, following a FedAvg-style process.The parameter server reparameterizes the model and distributes trainable parameters; clients train locally and return their deltas for aggregation and a global-learning-rate update.Source: 1) A meta-algorithm for distributed… · p. 8
  • The meta-algorithm repeats local updates until a step limit or timeout, aggregates client deltas at the parameter server, and restores and checkpoints the global model.Interrupted clients request the current parameter-server parameters and restart local training.Source: 1) A meta-algorithm for distributed… · p. 8

2) Paradigm 1: Federated Pretraining:

The pretraining system combines heterogeneous GPU clients with shared coordination and communication mechanisms. In geo-distributed experiments, GPT-2 Medium converges stably and faster in wall-clock time than the baselines, while Llama3-1B loss decreases without divergence.

Pretraining system

  • Each client uses gradient accumulation and FSDP, choosing hardware-fitting micro-batches while maintaining a consistent effective mini-batch size across heterogeneous clients.Gradients are quantized to BF16 or INT8 based on link speed; communication failures trigger retransmission, and a PS timeout handles clients that miss H local steps.Source: 2) Paradigm 1: Federated Pretraining · p. 8

Preliminary results on GPT-2 Medium pretraining

  • The GPT-2 Medium experiment uses OpenWebText, two university sites, and updates synchronized every 10 minutes; comparisons include FedAvg, DiLoCo, and FedNova.The sites use one OSU worker with 2× H100 GPUs and two UW–Madison workers with 2× RTX 3090 GPUs each, coordinated through AWS.Source: 2) Paradigm 1: Federated Pretraining · pp. 8–9
  • The design converges stably and reduces GPT-2 Medium loss faster in wall-clock time than the baselines; DiLoCo diverges quickly under its recommended hyperparameters.The authors conclude that geo-distributed designs must account explicitly for latency variance and compute heterogeneity.Source: 2) Paradigm 1: Federated Pretraining · p. 9

Preliminary results on cross-site communication latency

  • Cross-site gradient latency fluctuates over time and differs across sites, producing high average latency and variance that create stragglers and destabilize synchronization.These conditions motivate a centralized relay for firewall-constrained connectivity and compression to reduce WAN payloads.Source: 2) Paradigm 1: Federated Pretraining · p. 9

Preliminary results on Llama3-1B pretraining

  • For Llama3-1B, loss decreases steadily without divergence despite the larger model and heavier communication load.The experiment uses two university sites, two AWS instances, 20-minute synchronization, and gradient quantization from FP32 to UINT8.Source: 2) Paradigm 1: Federated Pretraining · p. 9

3) Paradigm 2: Federated Fine-tuning:

The paper implements five distributed fine-tuning algorithms with different communication, computation, and privacy trade-offs. They reuse the system backbone through different reparametrizations, except for federated split learning, which uses specific system changes.

Fine-tuning evaluation

  • We implement and evaluate five distributed fine-tuning algorithms, each with different trade-offs in communication efficiency, computation efficiency, and privacy; evaluation fine-tunes GPT2-medium for question answering on SQuAD.Figure 8 shows the geographical locations of GPUs used for LoRA fine-tuning and a real-time Grafana dashboard tracking training, validation, and network metrics.Source: 3) Paradigm 2: Federated Fine-tuning · p. 9

Federated LoRA-based fine-tuning

  • Federated LoRA exchanges only the low-rank adapter matrices A and B, rather than updating the full model; the described FedIT aggregation is known to be inconsistent.The system can also be extended to other federated fine-tuning variants.Source: 3) Paradigm 2: Federated Fine-tuning · p. 9

Hybrid LoRA fine-tuning

  • Hybrid LoRA keeps each client’s LoRA adapters private while exchanging only base parameters with the parameter server for aggregation.Clients update base parameters for Hbase steps and private adapters for Hlora steps, enabling site-specific personalization without exposing the adapters.Source: 3) Paradigm 2: Federated Fine-tuning · p. 9

Multi-headed LoRA Adapters (RAVAN)

  • RAVAN reparametrizes weights with trainable heads Hi, while W, A, and B are frozen and shared only once at the beginning.The reparametrization uses the set of heads Hi within the shared system backbone.Source: 3) Paradigm 2: Federated Fine-tuning · p. 9

Zeroth-order federated fine-tuning

  • Zeroth-order fine-tuning sends a random-number-generator seed to clients, which compute directional gradients for server aggregation, reducing communication from O(d) to O(1).Here d is the number of parameters; clients share the scalar directional gradient with the server.Source: 3) Paradigm 2: Federated Fine-tuning · pp. 9–10

Federated split learning (FSL-SAGE)

  • FSL-SAGE partitions the model between clients and server, using separate F-server and S-server processes and infrequent auxiliary-model alignment to reduce communication and allow parallel clients.The system applies it to GPT-2 Medium LoRA question answering on SQuAD across four university sites; the paper calls this its first geo-distributed framework for scalable federated split learning.Source: 3) Paradigm 2: Federated Fine-tuning · p. 10

IV. THE ROAD AHEAD

The paper identifies open problems that remain after demonstrating stable geo-distributed training, spanning model scale, membership churn, communication, and alignment. Its current experiments reach 1B parameters, and frontier-scale and multimodal training require strategies the present design does not yet address.

A. Scaling to Frontier and Multimodal Models

  • Experiments reach 1B parameters; scaling to 70B and beyond needs new strategies, while WAN placement and routing of sparse experts can inflate communication, unbalance load, and destabilize routing.Multimodal models also introduce modality-specific encoder, data-availability, and communication heterogeneity not addressed by the current design.Source: A. Scaling to Frontier and Multimodal… · p. 10

B. Fault Tolerance under Elastic Membership

  • Managed-cluster fault-tolerance systems assume reliable, high-bandwidth interconnects and homogeneous hardware—conditions that do not hold at the open edge.The cited examples include Bamboo, Oobleck, and Gemini.Source: B. Fault Tolerance under Elastic… · pp. 10–11
  • Open-edge membership makes disruption routine: clients leave, rejoin, or arrive with unfamiliar hardware, requiring WAN-aware work reassignment and state reconciliation without stalling the federation.The paper identifies lightweight WAN-aware failure detection and state reconciliation as missing pieces for open, heterogeneous membership.Source: B. Fault Tolerance under Elastic… · p. 11

C. Adaptive Communication Compression

  • Adaptive compression remains open: bit-widths, sparsification, and synchronization frequency should respond to network measurements and be co-designed with the outer optimizer.The paper also notes that larger models can be more communication-efficient in some regimes because computation grows faster than communication volume.Source: C. Adaptive Communication Compression · p. 11

D. Federated Alignment and Reinforcement Learning

  • Federated alignment is beyond current supervised fine-tuning; DPO may reduce shared components and communication, but private, heterogeneous preferences remain difficult to orchestrate without averaging away their diversity.WAN-distributed RLHF rollouts must also keep sampled trajectories fresh enough for stable policy optimization, calling for a different execution model from co-located systems.Source: D. Federated Alignment and… · p. 11

E. Privacy, Security, and Trust

The road ahead includes privacy and trust risks, incentives that account for contribution timing and strategic behavior, and closer coordination between system and algorithm choices. The paper calls for cross-disciplinary collaboration and shared infrastructure to support accessible, reproducible research.

E. Privacy, Security, and Trust

  • At LLM scale, the utility–privacy tradeoff for differential privacy is not well understood, and whether robust aggregation retains its guarantees remains open.The paper also identifies update leakage and the need for Byzantine robustness against clients submitting corrupted updates.Source: E. Privacy, Security, and Trust · p. 11

F. Incentive Design and Participation Economics

  • Compute-hour and Shapley-value rewards overlook timing and staleness; incentive rules must also address data quality and strategic gaming to limit free-riding and adverse selection.The paper notes that Shapley scores are unstable across aggregation strategies.Source: F. Incentive Design and Participation… · p. 11

G. System-Algorithm Co-Design

  • Synchronization cadence, compression, and fault tolerance interact with optimizer and aggregation choices, motivating an auto-tuning layer that adjusts system and algorithm parameters together.Longer synchronization saves communication but increases staleness, which may call for a different outer optimizer.Source: G. System-Algorithm Co-Design · p. 11

A Call to Arms

  • The paper calls for coordinated work across disciplines and shared open-source platforms, standardized benchmarks, and reproducible evaluation to lower barriers and enable fair comparisons.Without concerted effort, foundation-model training may remain concentrated among well-resourced actors; shared infrastructure can help resource-constrained communities contribute to and benefit from advances.Source: A Call to Arms · p. 12

Related papers

Loading 2610.32215v2…