Source-linked AI summary
Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, Zhifang Sui
TL;DR
Autoregressive decoding makes LLM inference latency costly because it generates tokens sequentially and is constrained by memory operations. This paper surveys Speculative Decoding, formalizes its drafting and verification framework, categorizes methods, and evaluates leading approaches with Spec-Bench. Across tested settings, leading methods provide substantial speedups, while the evaluation does not cover batched Speculative Decoding across batch sizes.
Problem
Autoregressive token-by-token generation creates substantial LLM inference latency, motivating methods that improve decoding efficiency.
Method
The paper provides a formal formulation, taxonomy, review of drafting and verification strategies, Spec-Bench, and comparative evaluation of leading methods.
Results
Leading methods achieve substantial speedups across tested settings, with EAGLE reaching 1.8×–2.4× over autoregressive decoding across most greedy subtasks.
Takeaways & Limitations
The survey clarifies the Speculative Decoding research landscape and provides comparative evidence to inform understanding of method advantages and limitations.
Takeaways & Limitations
The evaluations could not cover batched Speculative Decoding because no implementation was available, leaving speedup across batch sizes for future experiments.
Abstract
from arXiv · showhide
To mitigate the high inference latency stemming from autoregressive decoding in Large Language Models (LLMs), Speculative Decoding has emerged as a novel decoding paradigm for LLM inference. In each decoding step, this method first drafts several future tokens efficiently and then verifies them in parallel. Unlike autoregressive decoding, Speculative Decoding facilitates the simultaneous decoding of multiple tokens per step, thereby accelerating inference. This paper presents a comprehensive overview and analysis of this promising decoding paradigm. We begin by providing a formal definition and formulation of Speculative Decoding. Then, we organize in-depth discussions on its key facets, such as drafter selection and verification strategies. Furthermore, we present a comparative analysis of leading methods under third-party testing environments. We aim for this work to serve as a catalyst for further research on Speculative Decoding, ultimately contributing to more efficient LLM inference.
1 Introduction
Speculative Decoding addresses autoregressive inference latency by drafting multiple tokens efficiently and verifying them in parallel with the target LLM. The survey formalizes and organizes this paradigm while introducing a benchmark for comparing methods.
- Motivation: Autoregressive decoding generates one token per step, causing latency to grow with generated-sequence length and model scale.The bottleneck primarily comes from transferring LLM parameters between memory and accelerator cache.
- Core paradigm: Speculative Decoding drafts multiple future tokens and verifies them in parallel, accepting only tokens that satisfy the target LLM’s criterion.Tokens after the bifurcation position are discarded to preserve generation quality.
- Motivation: The paradigm exploits easy-token prediction and memory-bandwidth-bound inference to reduce repeated LLM parameter memory operations.A smaller model or other efficient drafter predicts candidate tokens before target-model validation.
- Research questions: Key open questions concern balancing drafter accuracy with drafting efficiency, preserving quality and parallelism during verification, and establishing unified evaluation conditions.Existing methods were evaluated under disparate testing conditions, motivating a common benchmark.
- Contributions: The paper presents the first comprehensive survey, a formal definition and formulation, a systematic taxonomy, and Spec-Bench for comparative evaluation.The authors aim to guide newcomers and motivate future research.
2 Overview
The paper surveys Speculative Decoding comprehensively, covering its evolution, formal foundations, major design choices, and benchmark-based evaluation.
- Overview: The survey covers Speculative Decoding’s evolution, formal definition and formulation, draft-model selection, verification strategies, and drafter–target alignment.It also reviews leading techniques and introduces Spec-Bench to assess acceleration across methods.
3 Evolution of Speculative Decoding
Speculative Decoding evolved from early draft-then-verify approaches into a general paradigm for accelerating LLM inference. Its development introduced independent drafters, relaxed verification, and lossless acceleration for sampling.
- Motivation: Autoregressive decoding is latency-limited because each step transfers LLM parameters from HBM to accelerator cache while producing only one token.This memory-bound process underutilizes modern accelerators.
- Early attempts: Blockwise Decoding introduced draft-then-verify by using additional FFN heads to draft multiple tokens and verify them in parallel with the original LLM.The approach targets greater inference parallelism while preserving the original model’s outputs.
- SpecDec: SpecDec used an independent specialized non-autoregressive Transformer for efficient drafting and relaxed verification to increase token acceptance.It achieved around 5× speedup over autoregressive decoding with comparable quality.
- Speculative Sampling: Speculative Sampling extended the paradigm to losslessly accelerate various sampling methods using smaller off-the-shelf language models from the same model series.These smaller models did not require additional training.
- Evolution: Together, pioneering methods established the Draft-then-Verify paradigm as a promising approach to LLM acceleration.The paper organizes these studies and subsequent research into a detailed categorization.
4 Formulation and Definition
Speculative Decoding formalizes inference as repeated draft-and-verify steps: an efficient drafter proposes multiple tokens, and the target LLM verifies them in parallel. Its effectiveness depends on accepted tokens, drafting efficiency, verification, and drafter–target alignment.
- 4.1 Autoregressive Decoding: Autoregressive decoding generates one token per memory-bound LLM call, making generation inefficient and time-consuming.
- 4.2 Speculative Decoding: Speculative Decoding drafts multiple future tokens, verifies them in parallel with the target LLM, and iterates until EOS or maximum length.
- 4.2 Speculative Decoding: The paradigm’s acceleration primarily depends on token acceptance, which reflects draft quality, verification criteria, drafter–target alignment, and drafting efficiency.
- 5.1 Independent Drafting: Independent drafting uses a separate model, including specialized non-autoregressive models or smaller same-series LMs, to propose tokens efficiently.SpecDec requires training a draft model, whereas off-the-shelf same-series models can avoid additional training or architectural modification.
- 5.2 Self-Drafting: Self-drafting avoids an external draft model but requires finding or training an aligned drafter and can add computational complexity when two models are integrated.These costs become more pronounced when no smaller counterpart exists or when deployment is distributed.
- 6.1 Greedy Decoding: Greedy verification is simple but may reject high-quality drafted tokens that differ from the target LLM’s top-1 predictions, limiting speedup.
- 6.2 Speculative Sampling: Speculative sampling uses a verification criterion proved to preserve the target LLM’s output distribution, while approximate variants relax strictness to improve acceptance.One example multiplies the drafted-token probability by a lenience parameter l ∈[0, 1].
- 6.3 Token Tree Verification: Token tree verification merges candidate sequences through shared prefixes and uses a tree attention mask to verify the structure in parallel.
7 Alignment
Alignment methods aim to make drafter behavior more similar to the target LLM, improving speculation accuracy while balancing drafting efficiency. The survey compares alignment strategies and notes that probability-distribution mismatch can hurt sampling performance.
- Alignment: Speculation accuracy depends partly on behavioral similarity between the drafter and target LLM.The survey describes alignment as a way to improve this similarity.
- Alignment: Sequence-level knowledge distillation trains drafters on sentences generated by the target LLM.Blockwise Decoding uses this alignment strategy.
- Alignment: Collective boost-tuning fine-tunes multiple small language models and aggregates their outputs as drafts.The strategy applies sequence-level distillation to improve speculation accuracy.
- Alignment: Sequence-level distillation can degrade sampling performance because it ignores the target LLM’s probability distributions.Recent work therefore explores alternative distillation strategies, including online knowledge distillation.
- Alignment: Table 3 summarizes drafter types, drafting strategies, alignment approaches, verification strategies, and reported speedups.The table provides a structured comparison of existing methods.
8 Spec-Bench
Spec-Bench evaluates speculative decoding across six application scenarios and supports third-party comparisons under shared testing conditions. In the reported Vicuna-7B experiments, EAGLE generally leads, while speedups decline as sampling temperature increases.
- Benchmark Construction: Spec-Bench enables fairer comparisons by testing open-source approaches on the same device and in the same environment.The benchmark addresses disparate benchmarks, devices, and evaluation settings.
- Benchmark Construction: Spec-Bench compares methods across six subtasks: conversation, translation, summarization, question answering, mathematical reasoning, and retrieval-augmented generation.It samples 80 instances from each of six widely used datasets.
- Comparative Evaluation: Figure 5 reports mean speedups over three runs on Vicuna-7B with batch size 1 under greedy settings.The main evaluations use FP16 precision on a single consumer-grade 3090 GPU.
- Comparative Evaluation: 1.8×–2.4× speedup: EAGLE outperforms autoregressive decoding across most subtasks under greedy settings.The strongest reported EAGLE result is approximately 2.4× on mathematical reasoning.
- Comparative Evaluation: 2.4× speedup: PLD performs especially well on summarization, where input and output have high similarity.Its speedup falls to 1.1×–1.3× on translation and question answering.
9 Challenges and Future Directions
The survey identifies unresolved challenges in balancing speculation accuracy with drafting efficiency, extending speculative decoding to batched inference, and integrating it with other efficient techniques. It frames these issues as directions for further research.
- Drafter Efficiency and Accuracy: Scaling up the drafter can improve speculation accuracy while reducing drafting efficiency and overall speedup.The survey highlights behavior alignment as a promising way to improve accuracy without increasing latency.
- Drafter Efficiency and Accuracy: Drafted tokens after the bifurcation position are discarded, motivating alignment methods that prioritize early-position token quality.Drafting quality and speculation length also remain open factors affecting accuracy.
- Batched Inference: Only a few implementations, including EAGLE and SpS, currently support batched inference.The survey identifies batched inference as important for managing real-time LLM-service inputs.
- Batched Inference: In batched speculative decoding, latency depends on the slowest sample because samples may require different numbers of decoding steps.Additional computational complexity also increases with batch size, especially during sampling.
- Integration with Other Techniques: Future work includes combining speculative decoding with contrastive decoding, multimodal generation, vLLM, non-autoregressive generation, and FlashAttention.The survey presents these integrations as directions for further improving inference efficiency or extending applicability.
10 Conclusion
The paper surveys Speculative Decoding comprehensively and introduces Spec-Bench for comparative evaluation. It aims to clarify the field’s research landscape and future directions.
- The paper reviews Speculative Decoding’s evolution, formal formulation, existing-method taxonomy, and leading techniques.
- It introduces Spec-Bench as an extensive benchmark for evaluating Speculative Decoding methods.
- The paper presents comparative evaluations of prominent methods to provide insights into future research directions.
Limitations
The paper’s comparative analysis does not yet cover speedup across different batch sizes. Subsequent experiments are planned to address this boundary.
- The study examines current Speculative Decoding methodologies and emerging trends through comparative analysis of leading open-source methods.
- The analysis offers insights into the advantages and limitations of different open-source Speculative Decoding models.
- Future experiments are planned to assess Speculative Decoding speedup across various batch sizes.
Ethics Statement
The paper evaluates Speculative Decoding across diverse tasks and compares representative open-source methods under specified hardware and software settings. The benchmark includes six subtasks and multiple drafting strategies.
- Spec-Bench: Spec-Bench evaluates speedup across six subtasks, including conversation, summarization, retrieval-augmented generation, translation, question answering, and mathematical reasoning.
- Spec-Bench: The benchmark uses MT-bench, WMT14 DE-EN, Natural Questions, and GSM8K, with 80 randomly selected test instances for each of three listed subtasks.
- Compared methods: The comparison covers six open-source methods using smaller language models, additional lightweight heads, special tokens, input spans, or retrieved drafts.
- Experimental setup: Experiments use Vicuna-v1.3 models, primarily a single RTX 3090 GPU with batch size 1, alongside further analysis on an A100 GPU.
C Details of Main Experimental Results
EAGLE performs strongly across Spec-Bench, while PLD benefits when inputs and outputs overlap substantially. Most methods show weaker speedup on translation, and hardware is examined as a performance factor.
- EAGLE achieves an overall speedup ranging from 1.6× to 2.4× across Spec-Bench subtasks.
- PLD’s speedup rises from 1.27× on question answering to 1.66× on retrieval-augmented generation.
- Most methods obtain suboptimal speedup on translation, which the authors suspect may reflect limited multilingual data in pretraining corpora.
- The analysis compares the influence of computational hardware, model scale, and computational precision under a fixed A100 experimental environment.
D.1 Computational Devices
Speculative Decoding benefits from stronger computational hardware, but its measured acceleration varies with model scale and numerical precision. Under the reported A100 evaluations, leading methods reach about 2.4× speedup, while FP32 reduces acceleration.
- Hardware effects: Medusa and Lookahead gain more on advanced GPUs, with speedups increasing from 1.48× to 2.42× and 1.11× to 1.77×, respectively.The paper attributes this improvement to greater idle computational resources on high-performance devices.
- Hardware effects: Medusa and EAGLE achieve an overall 2.4× speedup on a single A100 GPU, reaching approximately 2.8× on multi-turn conversation and mathematical reasoning subtasks.The evaluation uses Spec-Bench with Vicuna-7B at FP16 precision and greedy settings.
- Model scale: EAGLE maintains 2.4×–2.5× speedup across model scales, whereas Medusa declines from 2.4× to 2.0× as model scale increases.Medusa performs better on Vicuna-7B but loses acceleration at larger scales.
- Numerical precision: FP32 precision reduces speedup for all evaluated methods; PLD reaches 1.01×, while EAGLE falls from 2.39× to 1.74×.Most approaches are primarily evaluated at FP16, whose outputs may diverge from autoregressive decoding because of accumulated floating-point errors.