PaperlayerView exact revision on arXiv

Source-linked AI summary

Petals: Collaborative Inference and Fine-tuning of Large Models

Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Max Ryabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, Colin Raffel

arXiv:2209.01188v2cs.LGcs.DC

TL;DR

Large language models remain difficult to use because their hardware demands exceed what many researchers can access, while offloading is slow and hosted APIs limit research flexibility. PETALS distributes inference and fine-tuning across participants’ resources. It runs very large models more efficiently than offloading in several stated settings, while retaining support for model adaptation and sharing.

  • Problem

    Very large language models require costly high-end hardware, while offloading can be slow and hosted APIs often lack access to model control flow or internal states.

    Source: §1 · p. 1

  • Method

    PETALS distributes model layers across participant-run servers and lets clients collaborate on inference and fine-tuning, with trained modules shareable through a model hub.

    Source: §1 · p. 2

  • Results

    Offloading is about an order of magnitude slower than PETALS for single-batch inference; for fine-tuning forward passes it is competitive with multiple GPUs and constrained PETALS networking, otherwise PETALS offers higher throughput.

    Source: §3.3 · p. 6

  • Takeaways & Limitations

    Sharing trained modules supports collaborative improvement of the base model as users contribute task-specific training updates.

    Source: §4 · p. 7

    Peers serving the first model layers can recover input tokens, so sensitive-data users should restrict clients to trusted servers or use an isolated swarm.

    Source: §4 · p. 6

Abstract

from arXiv · show

Many NLP tasks benefit from using large language models (LLMs) that often have more than 100 billion parameters. With the release of BLOOM-176B and OPT-175B, everyone can download pretrained models of this scale. Still, using these models requires high-end hardware unavailable to many researchers. In some cases, LLMs can be used more affordably via RAM offloading or hosted APIs. However, these techniques have innate limitations: offloading is too slow for interactive inference, while APIs are not flexible enough for research that requires access to weights, attention or logits. In this work, we propose Petals - a system for inference and fine-tuning of large models collaboratively by joining the resources of multiple parties. We demonstrate that this strategy outperforms offloading for very large models, running inference of BLOOM-176B on consumer GPUs with $\approx$ 1 step per second, which is enough for many interactive LLM applications. Unlike most inference APIs, Petals also natively exposes hidden states of served models, allowing to train and share custom model extensions based on efficient fine-tuning methods.

1 Introduction

Large language models have become powerful and publicly available, but their hardware demands keep them difficult for most researchers to use. PETALS proposes collaborative inference and fine-tuning across participants’ resources.

1 Introduction

  • OPT-175B and BLOOM-176B need over 350 GB of accelerator memory for inference and significantly more for fine-tuning, usually requiring multiple high-end GPUs or multi-node clusters to run.These options are extremely expensive, limiting research and potential applications of large language models.Source: §1 · p. 1
  • RAM or SSD offloading can run large models on low-end accelerators, but BLOOM-176B generation takes at least 5.5 seconds per token with RAM and 22 seconds with SSD.Many computers also lack enough RAM to offload 175B parameters.Source: §1 · p. 1
  • Hosted APIs are user-friendly but often lack the flexibility researchers need to change model control flow or access internal states.Current API pricing can also make some research projects prohibitively expensive.Source: §1 · pp. 1–2
  • PETALS lets participants host model layers on servers and form chains for inference or fine-tuning, with optimizations for running 100B+ models efficiently.The system also supports sharing trained submodules for inference or further training.Source: §1 · p. 2, Figure 1 · p. 2

2 Design and use cases

PETALS divides large-model inference and adaptation across networked participants: servers host pretrained layers while clients route tasks through them. The design also supports reusable trained modules.

2.1 Inference of billion-scale models

  • Clients keep model embeddings locally, route inputs through a chain of servers holding Transformer blocks, and receive representations to compute next-token probabilities.Servers cache attention keys and values during active sessions; clients can replace a failed or offline server.Source: §2.1 · p. 2

2.2 Training for downstream tasks

  • In distributed fine-tuning, clients own trained parameters while servers compute forward and backward passes through pretrained layers without updating them.This lets clients run different training tasks simultaneously on the same servers without interfering.Source: §2.2 · p. 3
  • For soft prompt tuning, clients train task-specific input embeddings and a classification head, routing batches through remote servers and updating both from returned gradients.PETALS supports other parameter-efficient methods, including LoRA and prefix tuning, and allows custom local modules after existing blocks.Source: §2.2 · pp. 3–4

2.3 Sharing and reusing trained modules

  • Users can share trained modules through the Hugging Face Hub, where task and base-model tags help others find suitable adapters.Weights and code are uploaded by committing them to a Git repository.Source: §2.3 · p. 4

3 Internal structure and optimizations

PETALS combines quantization, server selection, and rebalancing to make large-model inference practical across distributed GPUs. Benchmarks compare it with offloading and show that performance depends on workload and network conditions.

3 Internal structure and optimizations

  • A single BLOOM-176B inference step takes less than a second in raw FLOPs on an RTX 3070, but naïve execution needs 44 GPUs and communication rounds; quantization and nearby-server selection reduce these costs.The system targets computation, communication-round, and network-latency constraints.Source: §3 · p. 4

3.1 Large model inference on consumer GPUs

  • 8-bit weight quantization reduces the BLOOM model's required nodes from 44 to 22, while hidden-state quantization halves bandwidth requirements without noticeable generation-quality effects.Weight quantization has about 5% overhead at batch size 1, becoming negligible for larger batches.Source: §3.1 · p. 5

3.2 Collaborating over the Internet

  • Nodes periodically rebalance their assigned layers when doing so can improve throughput, quickly redistributing resources if peers serving blocks leave.Clients route inference through low-latency server paths and reroute to replacements after server failure.Source: §3.2 · p. 5

3.3 Benchmarks

  • Offloading is about an order of magnitude slower for single-batch inference; training forward-pass offloading is competitive with multiple GPUs and PETALS networking at 100 Mbit/s or high latency.In other cases, PETALS offers higher throughput for training.Source: Table 3 · p. 6, §3.3 · p. 6
  • Inference depends little on bandwidth or sequence length but degrades with latency; large-batch parallel forward passes are affected by both bandwidth and latency.Table 3 reports sequential inference steps and parallel forward passes, with RTT denoting round-trip latency.Source: §3.3 · p. 6, Table 3 · p. 6

4 Discussion and future work

PETALS faces privacy and server-correctness risks in a multi-party network, while the discussion considers incentives, safeguards, and ways to manage model improvements over time.

Incentives for peers to contribute

  • Because clients can use servers without serving layers, the resulting supply-demand imbalance could be addressed by rewarding server peers with points redeemable for priority or other rewards.Source: §4 · p. 6

Privacy

  • Peers serving the first model layers can recover input tokens, so sensitive-data users should choose trusted servers or an isolated PETALS swarm.Secure multi-party computing or privacy-preserving hardware may address this limitation in the future.Source: §4 · p. 6

Security

  • Independent servers may return incorrect forward- or backward-pass outputs, whether through faults or attempts to influence results or earn rewards without computing.The security discussion assumes servers are run by many independent parties.Source: §4 · p. 7
  • A pledge-and-hash scheme could make cheating costly: servers pledge points, announce tensor hashes, and risk losing the pledge if a trusted third party confirms a mismatch.The approach still leaves a chance of wrong outputs, but is intended to incentivize exposing malicious servers quickly.Source: §4 · p. 7

Making changes to the main model

  • Shared parameter-efficient updates could let users collaboratively improve the base model as more users train it.The paper describes this as a meaningful step toward collaborative improvement, with the model becoming more capable over time.Source: §4 · p. 7
  • Because task-best parameters may change over time, the discussion calls for tracking fine-tuned versions and testing them on living benchmarks.Such testing would help ensure subsequent versions improve the desired capabilities.Source: §4 · p. 7
  • Model updates could be versioned by adding groups of layers, while adapters could carry tags identifying the model versions they support.The paper says this fine-grained versioning would be straightforward to add, though uncommon at the time.Source: §4 · p. 7

5 Conclusion

PETALS offers collaborative inference and fine-tuning of large models through an Internet-served system. It combines reduced resource requirements with routing and load-balancing algorithms, with the aim of broadening access to LLMs.

  • PETALS provides collaborative inference and fine-tuning of large language models, with a user-friendly generation interface and a flexible API for Internet-served models.Source: §5 · p. 7
  • 8-bit compression reduces the resources required to run very large models, and the system includes algorithms for reliable routing and load balancing.Source: §5 · p. 7
  • The authors hope PETALS will broaden LLM access and make previously impossible or overly expensive applications, studies, or research questions more feasible.Source: §5 · p. 7

Ethics Statement

The authors do not envision direct negative impacts beyond enabling broader public interaction with LLMs trained on uncurated web-crawled data; those models are already open-access.

  • The paper frames its ethical concern as broader public interaction with LLMs trained on uncurated web-crawled data.Source: Ethics Statement · p. 7
  • The authors do not envision direct negative impacts beyond granting the public this ability to interact with such models.Source: Ethics Statement · p. 7
  • The models are already open-access and can be exposed through APIs or other means.Source: Ethics Statement · p. 7
Loading 2209.01188v2…