PaperlayerView exact revision on arXiv

Source-linked AI summary

SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices

Ruslan Svirschevski, Avner May, Zhuoming Chen, Beidi Chen, Zhihao Jia, Max Ryabinin

arXiv:2406.02532v3cs.CL

TL;DR

Running 50B+ LLMs interactively on consumer GPUs is difficult because limited memory requires RAM or SSD offloading, making generation bandwidth-bound. SpecExec builds a large cache tree of likely draft continuations and validates it in one target-model pass. It achieves 4–6 tokens per second with 4-bit weights or 2–3 tokens per second with 16-bit weights, with 10–18x speedups over sequential inference on the same hardware.

  • Problem

    Consumer GPUs cannot fit many 50B+ LLMs, while parameter offloading makes interactive inference bandwidth-bound because weights must be transferred for each generated token.

  • Method

    SpecExec deterministically constructs a large tree of likely continuations with a powerful draft model and validates the tree as a cache in one target-model pass.

  • Results

    4–6 tokens per second with 4-bit quantization and 2–3 tokens per second with 16-bit weights, with 10–18x speedups over sequential inference on the same hardware.

  • Takeaways & Limitations

    SpecExec demonstrates interactive inference of large models on consumer GPUs with RAM offloading.

  • Takeaways & Limitations

    The study does not aim to create the fastest possible local-inference implementation and leaves combination with orthogonal performance improvements beyond its scope.

Abstract

from arXiv · show

As large language models gain widespread adoption, running them efficiently becomes crucial. Recent works on LLM inference use speculative decoding to achieve extreme speedups. However, most of these works implicitly design their algorithms for high-end datacenter hardware. In this work, we ask the opposite question: how fast can we run LLMs on consumer machines? Consumer GPUs can no longer fit the largest available models (50B+ parameters) and must offload them to RAM or SSD. When running with offloaded parameters, the inference engine can process batches of hundreds or thousands of tokens at the same time as just one token, making it a natural fit for speculative decoding. We propose SpecExec (Speculative Execution), a simple parallel decoding method that can generate up to 20 tokens per target model iteration for popular LLM families. It utilizes the high spikiness of the token probabilities distribution in modern LLMs and a high degree of alignment between model output probabilities. SpecExec takes the most probable tokens continuation from the draft model to build a "cache" tree for the target model, which then gets validated in a single pass. Using SpecExec, we demonstrate inference of 50B+ parameter LLMs on consumer GPUs with RAM offloading at 4-6 tokens per second with 4-bit quantization or 2-3 tokens per second with 16-bit weights.

1 Introduction

SpecExec targets bandwidth-bound inference of oversized LLMs on consumer GPUs by combining large-budget speculative decoding with a draft-tree cache and single-pass verification. The method is reported to achieve interactive speeds while accepting 10–20 tokens in favorable settings.

  • Motivation: Consumer GPUs cannot fit many 50B+ models, so CPU-RAM offloading makes generation bandwidth-bound because parameters must be transferred for each new token.A single offloaded Llama 2-70B token takes at least 4.5 seconds in 16-bit on an RTX 3090 under stated hardware assumptions.
  • Limitations of prior methods: Prior speculative-decoding methods can saturate near ten accepted tokens as speculation grows, while static tree methods require reoptimization across domains, sampling settings, and hardware.The saturation is reported for the Llama-2 7B draft and the associated model and dataset combination.
  • SpecExec: SpecExec deterministically builds a large tree of likely continuations with a powerful draft model, then validates the tree as a continuation cache in one target-model pass.The design addresses performance, flexibility, and scalability issues identified for prior methods.
  • Contributions: SpecExec analyzes speculative-decoding behavior at large draft budgets and identifies ways to improve acceptance when scaling to thousands of draft tokens.This analysis motivates the method’s large-budget design.
  • Contributions: The proposed draft-tree technique can produce trees yielding 10–20 accepted tokens when the draft budget is sufficiently large.This is presented as a principal contribution of SpecExec.
  • Results: 4–6 tokens/second with 4-bit quantization and 2–3 tokens/second with 16-bit weights, yielding 10–18x speedups over sequential inference on the same consumer hardware.The reported setup uses RAM offloading for Llama 2-70B or comparable models.

2 Background

The paper reviews speculative decoding and parameter offloading as complementary techniques for large-model inference. Its target setting is stochastic generation of modern instruction-tuned models on consumer hardware where memory capacity and bandwidth are limiting constraints.

  • Speculative decoding: Speculative decoding drafts candidate tokens with a small model and verifies them in parallel with a larger target model while preserving the target model’s sampling distribution.The framework has drafting and verification phases, with acceptance probabilities chosen to match sequential generation.
  • Scope: SpecExec focuses on speculative-decoding algorithms that support stochastic target-model sampling and guarantee sample probabilities identical to standard generation.This excludes methods limited to greedy inference because chat applications commonly require response variability.
  • Parameter offloading: Parameter offloading loads model weights from RAM or other storage onto the GPU as needed, but interactive inference is slow because it processes only one or a few tokens while waiting for parameters.Offloading is more effective for large batches, where each loaded layer processes many tokens.
  • Consumer-device setting: The study targets modern instruction-tuned models such as Llama-2-70B-chat and Mixtral 8x7B on consumer setups with limited VRAM, CPU memory, and PCIe bandwidth.The surveyed configurations include single consumer GPUs with 12–24 GB VRAM and 32–64 GB RAM, though the survey may not be fully representative.
  • Motivation: Running 50B+ models on consumer hardware generally requires offloading or aggressive compression, while 1.5–2-bit compression can cause significant accuracy losses.The paper therefore motivates mild quantization together with offloading.

3 Preliminary analysis

The preliminary analysis shows why offloaded inference is well matched to large-budget speculation: thousands of tokens can be processed nearly as cheaply as one, but ordinary speculative trees saturate. Sharp target distributions motivate searching for likely continuations with a capable draft model.

  • Offloading bottleneck: A 70B offloaded model can process thousands of tokens in nearly the same time as a single token because parameter loading, rather than computation, dominates the pass.This creates the hardware condition that makes large speculative batches attractive.
  • Acceptance saturation: Accepted-token counts saturate as the speculation budget grows, including with the more powerful Llama-2 7B draft model.The paper presents this as an observed limitation of single- or multi-sequence speculative decoding.
  • Draft-model choice: Large draft models are practical in the offloading setting because each offloaded forward pass takes significantly more than a second, reversing the usual preference for smaller drafts.This differs from regular GPU inference, where drafting with a 7B model would be impractical.
  • Draft-tree motivation: Beam search with a capable draft model can recover many high-probability target tokens, but deterministic search is incompatible with stochastic speculative sampling without an additional cache-based mechanism.This observation motivates SpecExec’s approach to reusing target probabilities over a draft tree.

4 Method

SpecExec uses a draft model to build a deterministic tree of likely continuations, precomputes target probabilities for those tokens in parallel, and samples from a cache while preserving sequential target-model sampling. Its key trade-off is lower individual-token acceptance than traditional speculative decoding, offset by better tree structure as the draft budget grows.

  • Speculative Execution: SpecExec reuses cached target probabilities until a sampled prefix leaves the cache, then recomputes a new draft tree and target-model batch.This loop combines cache lookup, sampling, cache exhaustion detection, and precomputation of future-token probabilities.
  • Speculative Execution: SpecExec deterministically builds a tree of likely continuations, processes its tokens with the target model in parallel, and caches probabilities for subsequent sampling.The method searches over draft continuations, runs a target-model forward pass on the selected tree, and stores next-token probabilities keyed by tree prefixes.
  • Speculative Execution: Unlike traditional speculative decoding, SpecExec may use a deterministic draft tree because its verification procedure remains correct for any tree.SpecExec samples using the target distribution at every prefix, so it is locally equivalent to sequential target-model sampling and preserves outputs for the same random seed.
  • Speculative Execution: SpecExec has lower acceptance probability for individual tokens than traditional speculative decoding, but becomes more effective as the draft-token budget increases.For small budgets it can underperform traditional speculative decoding; with larger budgets, its better-structured trees lead to more accepted tokens for the same draft size.
  • Search for the Optimal Draft Tree: The draft tree selects high-probability continuations through a parallel search that retains the K best tokens by cumulative probability.The search is implemented as a special case of single-source shortest path over the tree of token sequences, with priority-queue expansion and pruning to the token budget.
  • Speculative Execution: The implementation summarizes the method as selecting K draft tokens, processing them with the offloaded target model, and determining accepted tokens.The approach is intended to exploit parallel processing during offloaded inference, where large batches improve utilization relative to sequential execution.

5 Experiments

Experiments show that SpecExec exploits sharp target-model distributions and capable draft models to scale speculative decoding, outperforming SpecInfer at large draft budgets and accelerating offloaded inference across hardware settings.

  • 5.2 Draft Acceptance Rates: SpecExec performs on par with SpecInfer at small budgets but eventually outscales it as the draft budget grows.The authors attribute this to SpecExec selecting likely-token continuations rather than sampling draft tokens, with larger gains on MTBench and at lower temperature.
  • 5.3 Inference Speed: SpecExec consistently speeds up RAM-offloaded generation by several times on A100 across chat, instruct, and base-model evaluations.Tables 1 and 2 report inference speed and generation rate for SpecExec and SpecInfer under these settings.
  • 5.3 Inference Speed: Consumer-GPU experiments use a 4-bit GPTQ Llama-2-70B target, and even the RTX 2080Ti generates quickly enough for interactive use.The fastest inference occurs on the RTX 4090, while the 2080Ti setup uses a Sheared-Llama-1.3B draft model and consumes just over 7 GB of VRAM.
  • 5.3 Inference Speed: The optimal A100 draft-tree size is 128–512 tokens for SpecInfer versus 1024–2048 for SpecExec.Larger trees increase accepted tokens, but beyond hundreds or thousands of tokens, drafting time accelerates and the speed-optimal size becomes smaller than the acceptance-maximizing size.
  • 5.3 Inference Speed: SpecExec also delivers competitive speedups without offloading and remains robust under generation penalties.These evaluations are secondary to the offloading experiments and are reported in appendices.

6 Conclusion and Future Work

SpecExec combines offloading with speculative decoding to enable interactive local inference of large models, while the study leaves fastest-possible implementation to future work.

  • 6 Conclusion and Future Work: SpecExec unites offloading and speculative decoding to demonstrate interactive-speed inference of large models on consumer GPUs.The method is presented as a practical approach for large-budget speculative decoding.
  • 6 Conclusion and Future Work: The study does not aim to provide the fastest possible local-LLM implementation, which depends on orthogonal performance improvements outside its scope.The authors expect memory-bandwidth constraints to remain increasingly important for large-model inference.
  • 6 Conclusion and Future Work: SpecExec constructs a large draft tree covering likely continuations, then validates the tree as a cache in the target model.The draft tree is built using a parallel search procedure and target-model verification.
  • 6 Conclusion and Future Work: The draft-tree search can be reformulated as a shortest-path problem and solved with a modified parallel Dijkstra algorithm for large drafts.Non-negative negative-log draft probabilities define the edge weights, while expansion of multiple nodes improves parallel execution.
  • 6 Conclusion and Future Work: A maximum depth parameter limits inefficient sequential drafts when the draft model is highly confident in a single continuation.Without this limit, a long single stem may later be discarded after one draft-model mistake.

B SpecExec Algorithm Diagram

The SpecExec diagram summarizes the algorithm’s high-level flow, while the surrounding discussion situates its parallel shortest-path search among prior GPU and distributed graph-search work.

  • B SpecExec Algorithm Diagram: Figure 6 provides a block diagram of the key steps in the SpecExec algorithm.The supplied material identifies the visual as a high-level overview rather than reporting a numerical comparison.
  • B SpecExec Algorithm Diagram: SpecExec’s search formulation belongs to a broader literature on GPU-efficient, distributed, approximate, and theoretical parallel shortest-path algorithms.The cited examples include GPU SSSP implementations, distributed graph search, and theoretical analyses.

D Additional Implementation Details

The implementation drafts on the GPU, evaluates all draft tokens with the offloaded main model in parallel, and then selects accepted tokens using verification.

  • D Additional Implementation Details: The runtime loop generates a draft tree, computes main-model probabilities for its tokens, and selects accepted tokens through verification.The draft model is loaded first, the main model is loaded layerwise when offloading, and verification completes the cycle.
  • D Additional Implementation Details: All draft tokens are processed in parallel with a merged attention mask, while model-layer prefetching and overlapped loading hide offloading costs.The implementation loads subsequent layers while previous layers compute activations.
  • D Additional Implementation Details: The system keeps draft and main-model key/value caches on the GPU, with RAM offloading available for longer prompts or smaller GPU memory.Grouped-query attention keeps caches relatively small for short prompts, according to the authors.
  • D Additional Implementation Details: The best draft-token budget depends on hardware: RAM offloading favors fewer tokens, whereas SSD offloading or float16 weights favor larger budgets.This reflects different bottlenecks across hardware configurations.
  • D Additional Implementation Details: The experiments use GPTQ quantization and require substantial host RAM, including at least 140 GB for Llama-2-70B in 16-bit precision.The authors recommend 192 GB in practice to accommodate the draft model, caches, and other processes.
  • D Additional Implementation Details: Although experiments ran mainly on A100 GPUs, no model layers were kept in VRAM, yielding 12–22 GB observed VRAM use for 2048-token draft trees.This setup was intended to emulate consumer-GPU offloading conditions.

E Ablation: Acceptance with Different Draft Models

The ablation compares draft models by accepted-token count across draft sizes for Llama-2-70B Chat, showing a small acceptance advantage for the unquantized 13B model despite its memory cost.

  • E Ablation: Acceptance with Different Draft Models: The unquantized 13B draft model accepts slightly more tokens, but its 26 GB memory footprint is inaccessible to modern consumer GPUs.The comparison is reported for different SpecExec batch sizes using the OpenAssistant dataset and recommended sampling settings.
  • E Ablation: Acceptance with Different Draft Models: Figure 7 plots accepted-token count against draft size B for Llama 2-70B Chat with different draft models.The visual compares how acceptance changes as the SpecExec draft budget increases.

F On The Suboptimality of Beam Search

Standard beam search is suboptimal for SpecExec because retaining only final-beam hypotheses can reduce accepted-token coverage. Algorithm 2 generalizes a less-pruned search and expands the graph to maximize coverage probability without manual length or width tuning.

  • F On The Suboptimality of Beam Search: Initial top-k beam-search experiments performed worse than expected and often plateaued as maximum beam-search length increased.These observations motivated the analysis leading to Algorithm 2.
  • F On The Suboptimality of Beam Search: The experiment varies beam size and maximum within-beam length, measuring accepted tokens for standard and modified beam-search variants.Figure 8 reports accepted-token counts in a grid indexed by beam size and maximum length.
  • F On The Suboptimality of Beam Search: Algorithm 2 expands the graph optimally to maximize coverage probability without manual tuning for beam length or width.It generalizes the modified search that avoids pruning partial hypotheses excluded from the final beam.
  • F On The Suboptimality of Beam Search: Standard beam search can be outperformed by a trivial modification that retains partial hypotheses outside the final beam.The modified search begins like beam search but does not prune those partial hypotheses.

G Application to in-memory inference

SpecExec also provides measurable speedups for inference without parameter offloading. Although smaller than its offload-setting gains, these speedups remain competitive with recent work such as Chen et al. [2024a].

  • G Application to in-memory inference: SpecExec delivers measurable inference speedups without offloading, though they are less impressive than in offload settings.The paper reports these results for an A100 GPU in Table 4.
  • G Application to in-memory inference: Without offloading, SpecExec remains competitive with recent work such as Chen et al. [2024a].

H Drafting penalty effects

SpecExec remains stable under lightweight token-exclusion penalties but loses absolute speedups under heavier penalties. Heavier restrictions also degrade generated-text quality as tokens are skipped or replaced.

  • H Drafting penalty effects: The robustness experiments apply the same token penalties to both draft and target models and compare exclusions affecting fewer or more tokens.
  • H Drafting penalty effects: Lightweight token-exclusion penalties preserve SpecExec’s accepted tokens per iteration, whereas heavier penalties reduce absolute speedups.The experiments penalize tokens beginning with “r” or containing the letter “r”.
  • H Drafting penalty effects: Heavier penalties deteriorate generated-text quality as the model skips or replaces tokens to satisfy the restrictions.With lighter penalties, the model can work around the restrictions and produce reasonable text.
Loading 2406.02532v3…