PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMCad: Fast and Scalable On-device Large Language Model Inference

Daliang Xu, Wangsong Yin, Xin Jin, Ying Zhang, Shiyun Wei, Mengwei Xu, Xuanzhe Liu

arXiv:2309.04255v1cs.NIcs.AI

TL;DR

Mobile generative NLP requires private, local inference, but memory limits hinder the use of larger, more capable LLMs. LLMCad uses a memory-resident small model to generate tokens and a target model to verify them through token trees, adaptive fallback, and speculative generation. It reduces average per-token generation time by 2.9–9.3× on IoT devices and 3.5–4.7× on smartphones without compromising accuracy.

  • Problem

    Mobile devices need private local generative NLP inference, but memory limits cause weight swapping, severely slowing larger LLMs and restricting scalability.

    Source: §1 · p. 1, §2.2 · p. 4

  • Method

    LLMCad uses generate-then-verify collaboration: a memory-resident smaller LLM generates candidate tokens, while a target LLM verifies and corrects them using token trees, adaptive fallback, and speculative generation.

    Source: §3.1 · p. 5

  • Results

    2.9–9.3× and 3.5–4.7× lower average per-token generation time on IoT devices and smartphones, respectively, without compromising accuracy.

    Source: §1 · p. 3

  • Takeaways & Limitations

    LLMCad mitigates the mobile memory wall while retaining the accuracy of the target LLM and enabling larger models such as LLaMA2-13B to run on smartphones.

    Source: §1 · p. 3, §6 · p. 13

    Speculative generation consumes relatively more energy because memory-resident and target LLM execution run in parallel.

    Source: §4.4 · p. 12

Abstract

from arXiv · show

Generative tasks, such as text generation and question answering, hold a crucial position in the realm of mobile applications. Due to their sensitivity to privacy concerns, there is a growing demand for their execution directly on mobile devices. Currently, the execution of these generative tasks heavily depends on Large Language Models (LLMs). Nevertheless, the limited memory capacity of these devices presents a formidable challenge to the scalability of such models. In our research, we introduce LLMCad, an innovative on-device inference engine specifically designed for efficient generative Natural Language Processing (NLP) tasks. The core idea behind LLMCad revolves around model collaboration: a compact LLM, residing in memory, takes charge of generating the most straightforward tokens, while a high-precision LLM steps in to validate these tokens and rectify any identified errors. LLMCad incorporates three novel techniques: (1) Instead of generating candidate tokens in a sequential manner, LLMCad employs the smaller LLM to construct a token tree, encompassing a wider range of plausible token pathways. Subsequently, the larger LLM can efficiently validate all of these pathways simultaneously. (2) It employs a self-adjusting fallback strategy, swiftly initiating the verification process whenever the smaller LLM generates an erroneous token. (3) To ensure a continuous flow of token generation, LLMCad speculatively generates tokens during the verification process by implementing a compute-IO pipeline. Through an extensive series of experiments, LLMCad showcases an impressive token generation speed, achieving rates up to 9.3x faster than existing inference engines.

1 INTRODUCTION

Mobile generative NLP needs private, local inference, but limited memory prevents mobile devices from scaling to larger, more capable LLMs. LLMCad addresses this memory wall through generate-then-verify model collaboration and reports substantial speedups without accuracy loss.

Motivation

  • Mobile memory limits force weight swapping, increasing inference latency by 59–224× and creating a trade-off between real-time generation and emergent abilities.For LLaMA, 10B parameters are described as both the minimum for arithmetic reasoning and the maximum for real-time smartphone inference.Source: §1 · pp. 1–2

LLMCad: breaking memory wall through model collaboration

  • LLMCad delegates easy tokens to a memory-resident smaller LLM while a target LLM verifies and corrects them, preserving accuracy and accelerating multi-token verification.The target model can verify N tokens in one inference rather than generate them sequentially.Source: §1 · p. 2

Token tree generation and verification

  • Token tree generation expands candidate pathways, allowing the target LLM to validate branches in batch rather than checking a single linear sequence.Confidence-based branch pacing, tree decoding, and non-autoregressive verification support this design.Source: §1 · p. 2

Self-adaptive fallback strategy

  • Self-adaptive fallback uses cumulative uncertainty and verified-token history to trigger verification promptly while reducing unnecessary checks.The strategy verifies more frequently when generation capability is weaker and less frequently when it is stronger.Source: §1 · p. 2

Speculative generation pipeline

  • A speculative generation pipeline continues small-model generation during verification, but schedules it below the memory upper bound to avoid processor and memory contention.This pipeline exploits cases where verification detects no errors and speculative tokens remain usable.Source: §1 · p. 3

Implementation and evaluation

  • 2.9–9.3× and 3.5–4.7× lower average per-token generation time on IoT devices and smartphones, respectively, without compromising accuracy.LLMCad also achieves up to 5.6× speedup over competitive baselines and generates more than one token per second for LLaMA2-13B on smartphones.Source: §1 · p. 3

2 BACKGROUND AND MOTIVATION

Decoder-based LLMs generate tokens autoregressively, making on-device inference difficult when model weights exceed memory. The background motivates LLMCad by connecting this memory bottleneck with model collaboration and the need to prevent delegated-token errors.

2.1 Decoder-based Generative LLMs

  • Decoder-based LLMs combine embeddings with stacked decoder layers, while encoder-decoder models such as T5 and mT5 add encoder layers before decoding.The described decoder layers include masked self-attention, LayerNorm, and Linear operations.Source: §2.1 · p. 3
  • Autoregressive inference generates one token at a time and feeds each new token back into the model for the next prediction.This sequential dependency continues until the end-of-sequence token is produced.Source: §2.1 · p. 4

2.2 On-device LLM is Memory-bounded

  • 189–224× higher inference latency on TX2 occurs with only 5.8–12.2× larger models, while disk I/O consumes 95.9–98.8% of inference time after memory exceeds the device budget.Table 1 reports corresponding accuracy, peak-memory, and latency changes across LLM variants.Source: §2.2 · p. 4, Table 1 · p. 4
  • Traditional batching and compute-I/O overlap provide limited benefit for autoregressive on-device inference, while aggressive low-bit quantization can lose model capability.The passage attributes these limits to sequential token generation, I/O dominating computation, and insufficient precision at very low bit widths.Source: §2.2 · p. 4

2.3 Opportunities and Challenges of LLM collaboration

  • mT5-Small correctly generates more than 80% of the tokens generated by mT5-Large on IWSLT2017 translation, motivating delegation of easy tokens to a smaller model.The smaller model can fit within the memory budget while approaching the larger model on easier data points.Source: §2.3 · p. 5
  • Wrong-token delegation can propagate through autoregressive generation and produce catastrophic downstream errors, so LLMCad uses generate-then-verify to inspect and rectify them.The target LLM supplies the ground truth for verification, preventing error propagation and preserving accuracy.Source: §2.3 · p. 5

3 DESIGN

LLMCad combines a memory-resident smaller LLM with a larger target LLM to generate tokens quickly while verifying and correcting them. Its design uses token-tree generation and non-autoregressive verification, self-adaptive fallback, and speculative generation under memory and compute constraints.

3.1 Overview

  • LLMCad uses a lightweight memory-resident model as generator and a larger target model as verifier to preserve target-model accuracy during on-device generation.The target model verifies generated tokens and rectifies errors, while the smaller model handles most token generation.Source: §3.1 · p. 5, §3.2 · p. 7

3.2 Token tree generation and verification

  • Token trees retain multiple candidate continuations, allowing LLMCad to include sub-optimal tokens and verify all branches in one non-autoregressive target-model inference.Branches are batched for verification; correct sequences are constructed, searched depth-first, and rolled back at detected errors.Source: §3.1 · p. 5, §3.2 · p. 8
  • Confidence-based branch pacing allocates more resources to branches more likely to match the ground truth while controlling branch lengths with max-min fairness.Cumulative token confidence is used to estimate each branch’s likelihood.Source: §3.2 · p. 6
  • 8.5–9.9× lower verification time is achieved by non-autoregressive verification at 10 input tokens for mT5-Large and LLaMa-13B, compared with autoregressive verification.The improvement comes from one weight swap rather than repeated per-token swapping, reducing I/O overhead.Source: Figure 9 · p. 7, §3.2 · p. 8

3.3 Self-adaptive fallback strategy

  • Tree-cumulative confidence and a self-adaptive threshold trigger verification when global uncertainty is high and adjust verification frequency using verified-token accuracy history.The threshold decreases after error-free verification and increases when verification detects errors.Source: §3.3 · p. 8

3.4 Speculative Generation Pipeline

  • Speculative generation continues memory-resident decoding during verification, but LLMCad limits parallel execution to avoid processor and memory contention.Target-model loading stops at a memory upper bound, and speculative execution runs only when loading remains below the budget.Source: §3.4 · pp. 8–9

4 EVALUATION

LLMCad is evaluated across smartphones and IoT devices, tasks, memory budgets, energy consumption, and component ablations. It consistently improves generation speed while preserving accuracy, with stronger benefits under constrained memory and measurable contributions from all three techniques.

4.2 Generation Speed

  • LLMCad consistently outperforms all baselines in per-token generation time without compromising accuracy across the tested devices.The evaluation reports results using generation accuracy and per-token generation time across four devices.Source: §4.1 · pp. 9–10, §4.2 · p. 10
  • 2.9–9.3× and 3.47–4.67× speedups over Std are achieved on IoT and smartphone devices, respectively, with 0.86 s/token for question answering on Xiaomi 11.The reported speed enables real-time generation with over-10B LLMs on a commercial off-the-shelf device.Source: §4.2 · p. 10
  • 2.9–9.3× and 1.83–2.45× reductions in per-token generation time are reported against StdPL and STI, while accuracy improves 11.1–19.0 percentage points over STI.The improvement is attributed to memory-resident generation and tree non-autoregressive verification.Source: §4.2 · p. 10
  • Against collaboration baselines, LLMCad improves accuracy by 4.5–94.5 and 9.8–96.7 percentage points over BLD while achieving 1.1–1.4× and 1.1–1.3× speedups on IoT and smartphones.Compared with SP, it reduces per-token generation time by 1.93–2.00× on IoT and 1.34–1.77× on smartphones.Source: §4.2 · p. 10

4.3 Memory Sensitivity Analysis

  • Under lower memory budgets, LLMCad maintains the highest speedup, including 6.12× over StdPL, 1.91× over SP, and 1.25× over BLD at 4GB on Jetson TX2.Its advantage increases as memory becomes more constrained because delegation to the memory-resident model becomes more beneficial.Source: §4.3 · pp. 10, 12

4.4 Energy Consumption Analysis

  • LLMCad reduces per-token energy consumption relative to the evaluated baselines, with reductions up to 7.96× on Jetson Orin NX and 3.59× on Xiaomi 11.The reductions are attributed to delegating more tokens to the memory-resident LLM while preserving accuracy.Source: Table 5 · p. 11, §4.4 · p. 12
  • LLMCad’s energy advantage is smaller than its latency advantage because speculative generation runs the memory-resident and target LLMs in parallel.This parallel execution increases energy consumption relative to the latency benefit.Source: §4.4 · p. 12

4.5 Ablation Study

  • All three techniques contribute to improvement: token-tree generation and verification provide 2.6–4.3× speedups, speculative execution reaches 1.51×, and self-adaptive fallback reaches 1.08–1.20×.The ablation evaluates TGV, SF, and SPP on mT5, T5, and LLaMA2 workloads.Source: §4.5 · p. 12, Figure 15 · p. 12

5 RELATED WORK

Prior optimization work reduces inference latency through model collaboration, algorithm-level techniques, and pipelining. LLMCad builds on these directions while targeting memory-bound on-device LLM inference.

Model collaboration

  • Model collaboration delegates workloads to lightweight models to reduce inference latency while preserving relatively high accuracy, but prior approaches often target non-LLM architectures or require retraining.Speculative decoding is the closest related direction because it uses smaller LLMs for generation and larger LLMs for verification.Source: §5 · p. 12

Mobile ML optimization

  • Quantization, knowledge distillation, caching, and token pruning reduce model size or computation, and LLMCad is orthogonal to and compatible with these optimizations.These methods respectively reduce model size, reuse existing results, or prune unnecessary tokens.Source: §5 · p. 13

Pipeline optimization

  • Pipeline optimization overlaps computation with model transmission or shard loading, motivating LLMCad’s speculative generation pipeline for I/O blocking and limited parallelism.Prior systems pipeline model transmission with GPU execution or model-shard loading with computation.Source: §5 · p. 13

6 CONCLUSIONS

LLMCad is presented as an efficient on-device inference engine that combines three techniques to address mobile memory constraints. Compared with state-of-the-art engines, it substantially reduces per-token generation time without compromising accuracy.

6 CONCLUSIONS

  • 2.9–9.3× lower average per-token generation time on IoT devices and 3.5–4.7× lower time on smartphones, without compromising accuracy.The reported improvements come from token tree generation and verification, self-adaptive fallback, and a speculative generation pipeline.Source: §6 · p. 13
  • LLMCad addresses on-device generative NLP inference with token tree generation and verification, self-adaptive fallback, and speculative generation that uses hardware resources during verification.The work is described as the first efficient inference engine for on-device generative NLP tasks.Source: §6 · p. 13

Related papers

Loading 2309.04255v1…