PaperlayerView exact revision on arXiv

Source-linked AI summary

Distributed Inference and Fine-tuning of Large Language Models Over The Internet

Alexander Borzunov, Max Ryabinin, Artem Chumachenko, Dmitry Baranchuk, Tim Dettmers, Younes Belkada, Pavel Samygin, Colin Raffel

arXiv:2312.08361v1cs.LGcs.DC

TL;DR

Large language models are difficult to use because their scale requires high-end hardware, motivating cost-efficient distributed inference and fine-tuning. The paper develops fault-tolerant inference and decentralized load balancing for unreliable, heterogeneous Internet-connected devices. Petals runs Llama 2 (70B) and BLOOM (176B) over the Internet, with distributed generation at least 10× faster than local offloading under realistic network speeds.

  • Problem

    50B+ language models require multiple high-end GPUs or multi-node clusters, limiting access to researchers without such hardware.

    Source: §1 · p. 1

  • Method

    The paper combines fault-tolerant distributed inference with decentralized load balancing that assigns transformer blocks across unreliable devices as they join or leave.

    Source: §1 · p. 2, §3.2 · p. 4, §3.3 · p. 7

  • Results

    ≥10× faster autoregressive generation than local offloading is achieved with realistic network speeds on Llama 2 (70B) and BLOOM (176B).

    Source: §1 · p. 2

  • Takeaways & Limitations

    Petals supports inference and fine-tuning over unreliable Internet-connected devices with the same correctness guarantees as local execution.

    Source: §1 · p. 2

    The approach leaves privacy of data processed by outside peers and broader impacts of increased LLM accessibility as limitations for discussion and future work.

    Source: §5 · p. 10

Abstract

from arXiv · show

Large language models (LLMs) are useful in many NLP tasks and become more capable with size, with the best open-source models having over 50 billion parameters. However, using these 50B+ models requires high-end hardware, making them inaccessible to most researchers. In this work, we investigate methods for cost-efficient inference and fine-tuning of LLMs, comparing local and distributed strategies. We observe that a large enough model (50B+) can run efficiently even on geodistributed devices in a consumer-grade network. This could allow running LLM efficiently by pooling together idle compute resources of multiple research groups and volunteers. We address two open problems: (1) how to perform inference and fine-tuning reliably if any device can disconnect abruptly and (2) how to partition LLMs between devices with uneven hardware, joining and leaving at will. In order to do that, we develop special fault-tolerant inference algorithms and load-balancing protocols that automatically assign devices to maximize the total system throughput. We showcase these algorithms in Petals - a decentralized system that runs Llama 2 (70B) and BLOOM (176B) over the Internet up to 10x faster than offloading for interactive generation. We evaluate the performance of our system in simulated conditions and a real-world setup spanning two continents.

1 Introduction

Large language models are increasingly capable but difficult to use because their scale demands substantial hardware. This work develops Petals, a fault-tolerant decentralized system that pools unreliable Internet-connected devices for cost-efficient inference and fine-tuning.

  • 50B+ language models require multiple high-end GPUs or multi-node clusters because OPT-175B and BLOOM-176B need over 350 GB accelerator memory for inference.Source: §1 · p. 1
  • For models over 50B parameters, communicating activations over a slow network can be faster than swapping layers from local RAM or SSD.Source: §1 · p. 2
  • Dual attention caches enable recovery from failed servers, while decentralized load balancing assigns transformer blocks to maximize throughput as devices join or leave.Source: §1 · p. 2
  • Petals runs inference and fine-tuning over unreliable devices while preserving the correctness guarantees of local execution and operating persistently with volunteer support.Source: §1 · p. 2
  • With realistic network speeds, the distributed algorithms perform autoregressive generation ≥10× faster than local offloading on Llama 2 (70B) and BLOOM (176B).Source: §1 · p. 2

2 Background: efficient training and inference

This section reviews model parallelism and parameter offloading as approaches to efficient training and inference. Model parallelism distributes model computation across devices, whereas offloading trades cheaper storage for repeated parameter transfers and can be slow for large models.

2.1 Model parallelism

  • Model parallelism assigns subsets of model parameters and computations to devices, communicating activations between them.Source: §2.1 · p. 2, §2 · p. 2
  • Tensor parallelism splits each layer across devices but incurs all-to-all communication and synchronization after every layer.Source: §2.1 · pp. 2–3
  • Pipeline parallelism assigns full layers to stages and processes multiple microbatches, although some stages remain idle even with optimal execution.Source: §2.1 · p. 3
  • Real-world systems combine parallelism forms according to hardware and network type, using tensor parallelism within closely connected devices and pipeline parallelism across servers.Source: §2.1 · p. 3

2.2 Offloading

  • Parameter offloading loads layers from RAM or SSD to an accelerator just in time, enabling large models on a single low-end accelerator with sufficient storage.Source: §2.2 · p. 3
  • 11 seconds per forward pass is required for one OPT-175B token on one GPU even under best-case hardware and bandwidth, with training taking twice that time.Source: §2.2 · p. 3
  • Offloading overhead can be reduced by large batches or overlapping communication with computation, but every device still computes the full model.Source: §2.2 · p. 3

3 Method

The method analyzes why very large LLM inference is latency-sensitive and develops a fault-tolerant, load-balanced pipeline for unreliable Internet-connected devices, alongside parameter-efficient fine-tuning support.

3.1 Performance bottlenecks of LLM inference

  • Autoregressive generation requires O(n · t) communication rounds, making it more latency-sensitive than training, while offloading loads every layer t times.Attention-cache storage also creates substantial memory demands, including 9.6 GB per 2048-token sequence for GPT-3- or OPT-175B-sized models.Source: §3.1 · p. 4
  • Interactive applications expose offloading's weakness because bulk parallelism preserves throughput but does not provide fast responses or support long in-context examples.The authors therefore explore pipeline parallelism for inexpensive, preemptible, Internet-connected resources with node failures, network errors, and high latency.Source: §3.1 · p. 4

3.2 Distributed generation with fault tolerance

  • Dual server- and client-side caches let replacement servers restore attention state after failures, while cached activations limit normal transfers to one token's activations per round.The client maintains alternative servers and intermediate activations, enabling failed pipeline stages to be replaced during generation.Source: §3.2 · p. 5
  • After failures, the algorithm sends O(t) data per failed server and recomputes only failed stages, interpolating between cached and cache-less inference.With no failures it recovers O(n · t) communication; if all servers fail after one step, it effectively performs non-caching generation.Source: §3.2 · p. 5

3.3 Automatic load balancing

  • Servers periodically reassign contiguous transformer-block intervals to maximize total throughput as heterogeneous peers join, leave, or fail.Rebalancing also redistributes resources when all peers serving particular blocks disappear.Source: §3.3 · pp. 6–7

3.4 Parameter-efficient fine-tuning

  • Parameter-efficient fine-tuning keeps trainable parameters on clients while servers backpropagate through fixed layers, allowing concurrent tasks and custom adapters without server-side updates.Quantization reduces device count and communication rounds, while dynamic blockwise quantization halves activation bandwidth without noticeable generation-quality effects.Source: §3.4 · p. 7, §3.5 · p. 7

4 Experiments

Experiments show that the fault-tolerant distributed algorithms remain effective under unreliable servers, heterogeneous hardware, network variation, and concurrent clients. Across Llama 2 and BLOOM evaluations, the system generally outperforms parameter offloading, including in real-world cross-continent conditions.

4.1 Inference with unreliable servers

  • Our fault-tolerant algorithm maintains reasonable performance across tested failure conditions, especially at higher failure rates.Caching with restarts is fastest without failures, while cache-less inference degrades substantially for longer sequences.Source: §4.1 · p. 8

4.2 Experiments for Llama 2 (70B) and BLOOM (176B)

  • The evaluation covers Llama 2 (70B) and BLOOM (176B) on controlled networks, heterogeneous devices, and a real-world deployment spanning Europe and North America.The real-world setup uses 14 smaller servers connected at 100–1000 Mbit/s, while controlled experiments include three- and ten-server configurations.Source: §4.2 · pp. 8, 10
  • Across experiments, sequential inference measures generation steps per second, while parallel forward passes measure tokens per second for batch processing and fine-tuning.Backward-pass performance is reported separately because it depends on trainable weights, batch size, and other hyperparameters.Source: §4.2 · p. 9
  • The experiments additionally test concurrent clients and evaluate load balancing, model compression, and correctness after randomly shutting down servers.These tests assess system behavior beyond the principal throughput comparisons.Source: §4.2 · pp. 9–10
  • The system outperforms offloading for inference by about an order of magnitude, while training throughput is higher than offloading except under 100 Mbit/s or high-latency networking.Inference performance is mainly affected by latency, whereas large-batch fine-tuning forward passes depend on both bandwidth and latency.Source: §4.2 · p. 10
  • The distributed system significantly outperforms offloading for inference, including when each GPU node concurrently runs its own single-batch client.This comparison holds for equal hardware and supports collaborative use of the system over independently run offloading.Source: §4.2 · p. 10

5 Conclusion

The paper concludes that its fault-tolerant algorithm and decentralized system enable LLM inference on unreliable Internet-connected devices. It also identifies privacy of data processed by outside peers and broader impact as limitations requiring consideration.

5 Conclusion

  • The proposed system significantly outperforms alternative approaches for running inference on consumer-grade hardware and scales to models with hundreds of billions of trainable parameters.The conclusion attributes this result to a fault-tolerant inference algorithm and decentralized execution over distributed unreliable devices.Source: §5 · p. 10
  • Users processing sensitive data may need trusted servers or an isolated network because first-block servers can use their inputs to recover client data.The paper identifies privacy as a key limitation of its approach.Source: §5 · p. 10

A Quality and efficiency of BLOOM with 8-bit quantization

The paper reports that 8-bit quantization has little effect on benchmark quality while adding about 5% inference-time overhead for batch size 1.

A Quality and efficiency of BLOOM with 8-bit quantization

B Estimating theoretical best throughput with RAM offloading

The paper estimates a theoretical upper bound for RAM offloading and explains how the system supports beam-search sequence reordering. These analyses connect offloading limits with the system’s cache-management design.

B Estimating theoretical best throughput with RAM offloading

  • With best-possible CPU RAM offloading via PCIe 4.0 using 16 lanes per GPU and zero offloading latency, the inference-speed upper bound is 1/5.5 ≈0.18 tokens/s.Source: B Estimating theoretical best… · p. 15

C Extension to beam search algorithms

  • Beam-search algorithms maintain k candidate sequences and may reorder or duplicate candidates between inference steps.This differs from ordinary batched inference because sequence ancestry can change across steps.Source: C Extension to beam search algorithms · p. 15
  • The system supports arbitrary beam-search selection criteria by letting clients send indices that reorder server-side attention caches before the next inference step.For example, repeated indices can make one prior sequence produce multiple next-beam sequences.Source: C Extension to beam search algorithms · p. 15
  • Tables 4 and 5 report zero-shot accuracy and generation throughput for BLOOM-176B and OPT-175B under 8-bit and 16-bit weights.The supplied table captions identify the compared models, precisions, and throughput setting.Source: Table 4 · p. 15, Table 5 · p. 15

D Details of the server load balancing algorithms

The server load-balancing algorithms assign contiguous transformer-block segments using throughput bottlenecks, then rebalance when departures or new assignments make the current allocation inadequate. The greedy procedure approaches optimal throughput while limiting disruptive block replacements.

Initial block assignment

  • Each server measures the minimum of its Internet and GPU throughputs, which becomes the cached overall server throughput used for assignment.Source: D Details of the server load balancing… · p. 16
  • The next server targets blocks with the smallest current throughput, covering as many bottlenecks as possible before considering higher-throughput blocks.Candidate segments are compared lexicographically by their sorted throughput values.Source: D Details of the server load balancing… · p. 16
  • 90–100% of optimal throughput is usually achieved by the polynomial-time greedy assignment under realistic throughput values.The optimum is found by trying all possible assignments in exponential time.Source: D Details of the server load balancing… · p. 16

Rebalancing

  • Servers periodically simulate reassignment and commit only when the eventual throughput improves by at least p%.The protocol predicts how other servers would react before making the change.Source: D Details of the server load balancing… · p. 16
  • With a threshold p = 20%, the algorithm avoids oscillations while closing gaps after up to 100% of servers holding certain blocks leave.Block replacements stop once throughput is good enough unless an essential number of servers joins or leaves.Source: D Details of the server load balancing… · p. 16

E Evaluation of the server load balancing algorithms

The evaluation simulates heterogeneous servers whose availability follows daily activity cycles and compares increasingly adaptive load-balancing strategies. Full load balancing stays near the practical upper bound, whereas static or random assignment loses throughput under departures.

Evaluation setup and results

  • The study uses 206 virtual instances with sampled throughputs and memory capacities, following schedules that produce roughly 100–110 active servers at peaks and 15–25 at troughs.Source: E Evaluation of the server load… · p. 17
  • The comparison includes no balancing, join-time balancing only, full per-minute balancing, and an optimal per-minute upper bound.Source: E Evaluation of the server load… · p. 17
  • Full load balancing stays within 10–15% of the upper-bound throughput most of the time while maintaining connectivity.The upper bound is impractical because it requires solving an expensive optimization and reloading different layers every minute.Source: E Evaluation of the server load… · p. 17
  • Random contiguous assignment has zero throughput most of the time because it cannot form a complete pipeline.Source: E Evaluation of the server load… · p. 17
  • Balancing only when servers join performs reasonably during arrivals but quickly loses throughput when random departures create pipeline bottlenecks.Existing peers must rebalance to repair those bottlenecks.Source: E Evaluation of the server load… · p. 17

F Experiments with a wider range of failure rates

Additional experiments test fault-tolerant inference across failure rates up to 5% and sequences up to 2048 tokens. The proposed algorithm maintains reasonable performance across all tested conditions, especially at higher failure rates.

Failure-rate evaluation

  • Reasonable performance is maintained across all tested failure rates up to 5% and sequence lengths up to 2048 tokens, unlike the baselines.Higher failure rates represent conditions common with Internet communication, spot instances, and unreliable hardware.Source: F Experiments with a wider range of… · p. 18, Figure 3 · p. 18
  • Figure 3 reports sequential BLOOM 7.1B inference speed in steps/s as activation-transfer failure probability varies.A zero-speed baseline did not finish within one hour.Source: Figure 3 · p. 18

G Performance of training-time forward and backward passes

Training-time forward and backward performance is evaluated for BLOOM-176B sequence classification and causal language modeling under prompt and prefix-tuning settings. Prefix tuning, longer prompts, smaller batches, and CPU-only client computation reduce throughput, while GPU client computation usually adds little.

Evaluation scope

Sequence classification

Language modeling

  • GPU-enabled causal-language-modeling throughput is within 10% of sequence-classification throughput because the client performs only a small share of forward and backward computations.The model head and loss function do not decisively affect performance in this setup.Source: G Performance of training-time forward… · p. 19
  • CPU-only causal-language-modeling throughput is 5–10 times worse because multiplying the output embedding matrix by all batch-token hidden states is inefficient on CPU.Source: G Performance of training-time forward… · p. 19

H Limitations and broader impact

The system broadens access to large-model inference while exposing privacy, security, and participation constraints that require operational safeguards.

Privacy

  • Sensitive-data users should restrict clients to trusted servers or deploy an isolated network because first-block servers may recover client inputs.Secure multi-party computing and privacy-preserving hardware are identified as future ways to address this limitation.Source: H Limitations and broader impact · pp. 19–20

Motivating contributors

  • Because clients need not serve model blocks, the system may face an imbalance between available server capacity and inference demand.Reward points are proposed to encourage peers to host blocks, with validators issuing rewards to server owners.Source: H Limitations and broader impact · p. 20

Security

  • Faulty servers may return incorrect computations maliciously or to earn rewards without performing the work.Random validator requests can eventually expose and penalize such servers, although incorrect outputs remain possible.Source: H Limitations and broader impact · p. 20
  • Running client data through multiple disjoint server chains and comparing outputs can reduce the probability of faulty results.Source: H Limitations and broader impact · p. 20

Broader impact

Related papers

Loading 2312.08361v1…