PaperlayerView exact revision on arXiv

Source-linked AI summary

TPI-LLM: Serving 70B-scale LLMs Efficiently on Low-resource Edge Devices

Zonghang Li, Wenjiao Feng, Mohsen Guizani, Hongfang Yu

arXiv:2410.00531v1cs.DCcs.AI

TL;DR

Edge devices cannot readily host large LLMs, while cloud inference exposes private prompts and single-user workloads make pipeline parallelism inefficient. TPI-LLM uses tensor parallelism with a sliding window memory scheduler and star-based allreduce while keeping raw data local. It reduces latency substantially against established systems and runs 70B-scale models with a 3.1 GB per-device footprint.

  • Problem

    Edge devices have limited memory and compute, cloud inference exposes user prompts, and single-user workloads make pipeline parallelism inefficient.

    Source: §1 · pp. 1–2

  • Method

    TPI-LLM combines tensor parallel inference, sliding window weight scheduling with overlapped disk I/O, and star-based allreduce while keeping sensitive data local.

    Source: §1 · p. 2, §5 · p. 10

  • Results

    Over 80% less time-to-first-token and token latency than Accelerate and over 90% less than Transformers and Galaxy, with Llama 2-70B using 3.1 GB per device.

    Source: §1 · p. 2

  • Takeaways & Limitations

    TPI-LLM enables 70B-scale LLM inference on memory-limited edge devices while retaining user prompts and generated sequences locally.

    Source: §3 · p. 3, §4.1 · p. 8, §5 · p. 10

    The framework assumes a specific physical edge-network topology with equal link latency and equal device computing power.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Large model inference is shifting from cloud to edge due to concerns about the privacy of user interaction data. However, edge devices often struggle with limited computing power, memory, and bandwidth, requiring collaboration across multiple devices to run and speed up LLM inference. Pipeline parallelism, the mainstream solution, is inefficient for single-user scenarios, while tensor parallelism struggles with frequent communications. In this paper, we argue that tensor parallelism can be more effective than pipeline on low-resource devices, and present a compute- and memory-efficient tensor parallel inference system, named TPI-LLM, to serve 70B-scale models. TPI-LLM keeps sensitive raw data local in the users' devices and introduces a sliding window memory scheduler to dynamically manage layer weights during inference, with disk I/O latency overlapped with the computation and communication. This allows larger models to run smoothly on memory-limited devices. We analyze the communication bottleneck and find that link latency, not bandwidth, emerges as the main issue, so a star-based allreduce algorithm is implemented. Through extensive experiments on both emulated and real testbeds, TPI-LLM demonstrated over 80% less time-to-first-token and token latency compared to Accelerate, and over 90% compared to Transformers and Galaxy, while cutting the peak memory footprint of Llama 2-70B by 90%, requiring only 3.1 GB of memory for 70B-scale models.

1 INTRODUCTION

Edge LLM serving must address privacy, severe device constraints, and inefficient single-request parallelism. TPI-LLM combines tensor parallelism with memory and communication optimizations to serve larger models efficiently.

  • Edge devices typically provide only 4–16 GB of memory and limited computing power, while Llama 3.1-70B requires at least 40 GB even with quantization.A single Nano-M device also runs Bert-L at 120× the latency of an A100 GPU.Source: §1 · p. 1
  • Single-user edge workloads make pipeline parallelism inefficient because devices remain idle, motivating tensor parallelism for better utilization.Prior edge approaches adaptively partition models or restructure Transformer layers for this setting.Source: §1 · pp. 1–2
  • Blocking CPU-to-disk offloading can raise Llama 3.1-8B token latency to 30 seconds per token, exposing the need for overlapped memory management.The sliding window scheduler asynchronously loads and unloads layer weights while computation and communication proceed.Source: §1 · p. 2
  • TPI-LLM keeps prompts and generated sequences local while coordinating limited edge devices through compute- and memory-efficient tensor parallel inference.The prototype was tested across multiple Llama and Yi model sizes against Transformers, Accelerate, and Galaxy.Source: §1 · p. 2
  • Star-based allreduce addresses link latency as the communication bottleneck and outperforms ring- and tree-based methods.The design treats bandwidth as no longer the primary issue.Source: §1 · p. 2
  • Over 80% less time-to-first-token and token latency than Accelerate, and over 90% less than Transformers and Galaxy, with Llama 2-70B using 3.1 GB per device.TPI-LLM was evaluated as a prototype for low-resource edge devices.Source: §1 · p. 2

2 OBSERVATIONS AND MOTIVATIONS

On low-resource edge devices, tensor parallelism can reduce overall inference time despite greater communication because it keeps devices computing concurrently. It still leaves substantial memory pressure, motivating additional scheduling.

  • The central design questions are whether computation or communication dominates and whether tensor or model parallelism is more efficient on low-resource edge devices.These questions guide the empirical comparison before introducing TPI-LLM.Source: §2 · p. 2
  • Tensor parallelism is preferred because simultaneous computation outweighs its 70% communication share, whereas model parallelism leaves devices idle despite only 2% communication.The comparison uses Llama 3.1-8B on four 8-core laptops connected over a 178 Mbps LAN.Source: §2 · p. 2
  • Figure 1 compares tensor and model parallelism by computation time, communication time, and per-device memory footprint as tensor-parallel nodes increase.The figure motivates examining both performance balance and memory scaling.Source: Figure 1 · p. 3
  • Even with four tensor-parallel nodes, memory remains 12 GB per device because intermediate results, KV cache, libraries, and competing applications add to model-parameter storage.A memory scheduler is therefore needed to avoid out-of-memory issues.Source: §2 · p. 3

3 TPI-LLM FRAMEWORK WITH SLIDING WINDOW MEMORY SCHEDULING

TPI-LLM combines tensor-parallel edge inference with privacy-preserving prompt locality, latency-aware allreduce, and sliding-window weight scheduling. The framework targets single-prompt, low-resource devices by distributing computation while overlapping disk I/O with computation and communication.

3 TPI-LLM FRAMEWORK WITH SLIDING WINDOW MEMORY SCHEDULING

  • Single-prompt edge workloads favor tensor parallelism because pipeline scheduling can leave devices underutilized when only one prompt is processed at a time.The framework is designed for devices with limited memory, low computing power, and often no CUDA-capable GPU.Source: §3 · p. 3

3.1 THE PARALLEL FRAMEWORK DESIGN OF TPI-LLM SYSTEM

  • Distributed attention heads, FFN weights, and KVCache reduce per-device computation and memory costs while allreduce aggregates intermediate results.Each device caches its local key-value pairs, partitioning KVCache across devices rather than replicating it.Source: §3.1 · pp. 3–4
  • TPI-LLM keeps prompts and generated sequences on the master device while workers perform partitioned tensor computations across Transformer layers.The master broadcasts embeddings and control data, workers compute attention and FFN blocks, and aggregated outputs return to the master for next-token sampling.Source: §3.1 · pp. 3–4

3.2 ALLREDUCE LATENCY ANALYSIS

  • TPI-LLM identifies link latency, rather than bandwidth, as the allreduce bottleneck; for Llama 2-70B, data transfer takes only 3.4 ms at 300 Mbps.Increasing bandwidth does not significantly reduce token latency, while multi-hop ring and tree methods amplify link-latency costs.Source: §3.2 · pp. 5–6

3.3 SLIDING WINDOW MEMORY SCHEDULING

  • The sliding-window scheduler dynamically loads and unloads only needed weights, preloading upcoming layers and overlapping disk I/O with computation and communication.A daemon thread maintains the window, while processed weights are unloaded to control peak memory usage.Source: §3.3 · p. 6, Figure 4 · p. 7
  • The scheduler reaches steady state when attention, FFN, and allreduce time cover weight-loading time, with retained FFN blocks providing a speed–memory tradeoff when disk I/O is slow.In a four-laptop Llama 2-7B example, the tight condition is unmet but the loose condition permits steady-state operation.Source: §3.3 · pp. 6–7

4 EXPERIMENTS

Experiments evaluate TPI-LLM across emulated and real edge testbeds, measuring latency, memory, scaling, and benchmark performance. The results show that sliding-window scheduling and tensor-parallel collaboration enable larger models with lower resource use.

4.1 OVERVIEW OF TPI-LLM PERFORMANCE

  • 3.1 GB per device lets TPI-LLM serve Llama 2-70B, while the scheduler also fits Llama 3.1-70B within device limits.The evaluation focuses on TTFT, token latency, and peak memory footprint per device.Source: §4.1 · p. 8
  • With the memory scheduler enabled, two devices can run 70B models, while without it, two devices are limited to smaller models between 3B and 7B.A memory window size of 4 requires slightly more memory but provides faster speed.Source: §4.1 · pp. 8–9

4.2 SCALING OVER VARYING EDGE CONDITIONS

  • More devices and CPU cores reduce token latency, while increasing bandwidth from 300 Mbps to 1 Gbps has little effect because each allreduce transfers only 256 KB.Under these tested conditions, computation remains the bottleneck rather than network bandwidth.Source: §4.2 · p. 9, Figure 5 · p. 9

4.3 COMPARISON WITH BENCHMARKS

  • TPI-LLM achieves lower token latency and memory footprint than standalone Transformers, model parallelism, and Galaxy, including for larger 70B models.Model parallelism is limited by pipeline bubbles, while Galaxy’s ring reductions traverse each link at least 14 times; TPI-LLM instead minimizes hops with star-based allreduce and overlaps I/O.Source: §4.3 · pp. 9–10, Figure 6 · p. 9, Table 3 · p. 10

4.4 REAL CASE STUDY

  • On four real laptops, Transformers incurs swap-related latency and OOM failures, while Accelerate fails at 13B because of disk-offloading implementation limits.Transformers+MS improves TTFT and token latency across model sizes by asynchronously preloading weights and overlapping I/O with computation and communication.Source: §4.4 · p. 10
  • Distributing computation across four laptops further reduces TTFT and token latency because the computation savings exceed communication delays.Using three laptops produces slightly higher latency because of reduced parallelism.Source: §4.4 · p. 10

5 CONCLUSION

The conclusion presents TPI-LLM as a compute- and memory-efficient tensor-parallel system for serving 70B-scale LLMs on low-memory edge devices. It combines local prompt handling, sliding-window weight management, and star-based allreduce to reduce resource and latency barriers.

  • TPI-LLM keeps sensitive prompts and generated sequences local while serving 70B-scale LLMs through tensor parallelism.The system is intended for low-resource edge devices.Source: §5 · p. 10
  • Its sliding-window scheduler overlaps disk I/O with computation and communication, allowing larger models to run on devices with very limited memory.Source: §5 · p. 10
  • Because link latency rather than bandwidth is the main communication issue, TPI-LLM uses star-based instead of ring- or tree-based allreduce.Source: §5 · p. 10
  • Experiments on emulated and real testbeds show lower TTFT, token latency, and peak memory footprint than Transformers, Accelerate, and Galaxy.The system also serves Yi-34B and Llama 2/3/3.1-70B on low-memory devices.Source: §5 · p. 10

REPRODUCIBILITY

The paper supports reproducibility by providing source code, usage guidance, and a prebuilt Docker image.

  • Source code, a detailed README, and a prebuilt Docker image are provided to support reproducing the experiments.Key experimental setups are described in Section 4.Source: REPRODUCIBILITY · p. 11

A.1 PROOF OF PROPOSITION 2

The proof shows that small messages make link latency, rather than bandwidth or data transfer, the dominant allreduce bottleneck, favoring star-based communication.

A.1 PROOF OF PROPOSITION 2

  • In TPI-LLM, each device sends only tens of kilobytes, so data-transfer latency is minimal and link latency dominates.This makes conventional tree- and ring-based algorithms less effective on high-latency edge networks.Source: §A.1 · p. 13
  • Star allreduce sends worker data directly to the master through the router, while tree and ring methods require additional communication hops or steps.The star model includes reduction and broadcast, whereas tree aggregation forwards data through another worker and ring communication circulates data among neighbors.Source: §A.1 · p. 13, Figure 7 · p. 13
  • Star-based allreduce is most efficient because it minimizes link latency.Under negligible data-transfer, barrier, and aggregation costs, star latency is 2tlink versus 4tlink for tree and ring methods.Source: §A.1 · p. 13

A.2 A SIMPLE-TO-USE MEMORY SCHEDULER

The memory scheduler uses a context manager to load required block weights and release them afterward, simplifying large-model deployment on low-memory devices.

A.2 SIMPLE-TO-USE MEMORY SCHEDULER

  • A context manager loads the required block weights and unloads used weights to free memory for subsequent blocks.The implementation requires only one additional line of code around the relevant computation.Source: §A.2 · pp. 13–14

A.3 PROOF OF PROPOSITION 3

The proof analyzes a repeating schedule that overlaps attention and FFN computation, allreduce communication, and weight loading, deriving conditions for nonblocking steady-state execution.

A.3 PROOF OF PROPOSITION 3

  • The scheduler repeatedly overlaps weight loading with computation and allreduce across attention and FFN blocks.The process continues until generation finishes, with each time slot tracking the next block’s loading progress.Source: §A.3 · p. 14
  • The first FFN computation proceeds without blocking when attention computation plus allreduce covers the next FFN weight-loading time.Otherwise, computation waits for the FFN weights to finish loading.Source: §A.3 · p. 14
  • Steady-state execution requires inequalities that bound cumulative attention, FFN, and allreduce times against cumulative FFN and attention loading times.The derived general condition is l · tattn + (l −1) · tffn + (2l −1) · tall reduce ≥l · τffn + (l −1) · τattn.Source: §A.3 · p. 14
  • A complementary set of inequalities covers schedules where FFN loading and attention loading alternate in the critical path.Its general pattern is (l −1) · tattn + l · tffn + (2l −1) · tall reduce ≥(l −1) · τffn + l · τattn.Source: §A.3 · pp. 14–15

A.4 PROOF OF PROPOSITION 4

The proof derives Proposition 4 by introducing intermediate slack variables and transforming the steady-state inequality into two sufficient conditions for avoiding weight-loading stalls.

A.4 PROOF OF PROPOSITION 4

  • The proof defines α as the available slack across attention, FFN, allreduce, and weight-loading times, then derives a first condition.It uses α > 0 together with τattn −tffn −tall reduce < 0.Source: §A.4 · p. 15
  • It defines β = tffn +tall reduce −τattn and derives a second condition ensuring the next attention weights do not stall execution.The condition follows from β > 0 and τffn −tattn −tall reduce < 0.Source: §A.4 · p. 15

A.5 PROOF OF PROPOSITION 5

The appendix formalizes memory-footprint reasoning for the sliding-window scheduler and describes emulated and real edge testbeds, model configurations, and a three-laptop case study.

A.5 PROOF OF PROPOSITION 5

  • The peak-memory analysis models parameter counts, activation storage, temporary tensors, memory management, and caching, using a scaling factor γ to estimate memory.The analysis derives worker-node footprint expressions from the memory window and excludes preprocess and postprocess blocks from worker memory.Source: §A.5 · pp. 15–16

A.6 PROOF OF PROPOSITION 6

  • The scheduler overlaps computation, communication, and disk I/O, retaining selected FFN blocks when disk latency disrupts steady-state preloading.Preliminary measurements report t_attn = 11 ms and t_ffn = 17 ms, with FFN blocks generally having higher computation and weight-loading latency.Source: §A.6 · p. 16

A.7 KLONET TESTBED

  • The Klonet testbed emulates edge networks with controllable bandwidth, delay, jitter, packet loss, device resources, and network topology.Its topology includes eight user devices, eight home gateways, and one core router, with settings based on measurements from the authors’ edge network.Source: §A.7 · p. 17, Figure 9 · p. 17

A.8 CONFIGURATIONS OF THE USED MODELS

  • The appendix includes configurations for the Llama models used in evaluation, as summarized in Table 5.The supplied table caption identifies the model-configuration scope but does not provide individual configuration values.Source: Table 5 · p. 17

A.9 PEAK MEMORY FOOTPRINT WITH MEMORY WINDOW SIZE 4

  • Table 6 reports peak memory footprint per device when the memory-window size is set to 4.The supplied caption identifies the evaluation condition without exposing the table’s individual values.Source: Table 6 · p. 18

A.10 REAL TESTBED AND CONFIGURATIONS

  • The real testbed uses four laptops connected through local Wi-Fi, with one master generating the output sequence and three workers connected to it.The generated sequence is identical to single-server inference.Source: §A.10 · p. 18, Figure 10 · p. 18, Table 7 · p. 18

A.11 CASE STUDY WITH 3 LAPTOPS

  • Using three laptops—one MacBook Pro, one MacBook Air, and one Dell—produces slightly higher latency because parallelism is reduced.Table 8 compares Transformers, Accelerate, Transformers+MS, and TPI-LLM in this three-laptop case study.Source: §A.11 · p. 19, Table 8 · p. 19

Related papers

Loading 2410.00531v1…