PaperlayerView exact revision on arXiv

Source-linked AI summary

Decoding Speculative Decoding

Minghao Yan, Saurabh Agarwal, Shivaram Venkataraman

arXiv:2402.01528v4cs.LGcs.CL

TL;DR

Speculative decoding can improve LLM inference throughput without sacrificing output quality, but its performance depends on draft-model choice. The paper benchmarks this choice and redesigns drafts around latency, reporting up to 111% higher throughput than existing models.

  • Problem

    Speculative decoding’s speedup depends on draft-model choice, but the factors that determine throughput gains require systematic study.

    Source: §1 · p. 1

  • Method

    The authors benchmark speculative decoding across more than 350 experiments, analyze draft latency and language-modeling accuracy, then design pruned drafts that trade depth for width.

    Source: §1 · pp. 1–2

  • Results

    Up to 111% higher throughput than existing models is reported, while draft-model latency is a key bottleneck and language-modeling accuracy does not correlate strongly with speculative-decoding performance.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The pruned drafts can be used without modification for supervised fine-tuned targets, and the approach generalizes to state-of-the-art models.

    Source: §5 · p. 9

    The study focuses on empirically improving lossless speculative-decoding throughput; broader studies of approximate efficient inference algorithms are left for future work.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Speculative Decoding is a widely used technique to speed up inference for Large Language Models (LLMs) without sacrificing quality. When performing inference, speculative decoding uses a smaller draft model to generate speculative tokens and then uses the target LLM to verify those draft tokens. The speedup provided by speculative decoding heavily depends on the choice of the draft model. In this work, we perform a detailed study comprising over 350 experiments with LLaMA-65B and OPT-66B using speculative decoding and delineate the factors that affect the performance gain provided by speculative decoding. Our experiments indicate that the performance of speculative decoding depends heavily on the latency of the draft model, and the draft model's capability in language modeling does not correlate strongly with its performance in speculative decoding. Based on these insights we explore a new design space for draft models and design hardware-efficient draft models for speculative decoding. Our newly designed draft model can provide 111% higher throughput than existing draft models and our approach generalizes further to all LLaMA models (1/2/3.1) and supervised fine-tuned models.

1 Introduction

The paper studies what determines speculative-decoding throughput and argues that draft-model latency, not language-modeling accuracy, is the key design concern. It proposes hardware-oriented draft redesigns that trade depth for width, reporting throughput gains and reduced KV-cache use.

1 Introduction

  • Across more than 350 experiments, draft-model latency emerges as the key speculative-decoding throughput bottleneck.The study includes LLaMA-65B, OPT-66B, fine-tuned chat targets, and draft models ranging from about 5× to 528× fewer parameters than their targets.Source: §1 · p. 1
  • Draft-model language-modeling accuracy does not correlate strongly with speculative-decoding performance, and greater depth increases latency even at equal parameter count.The authors say existing drafts were designed primarily for accuracy under a parameter budget and are suboptimal for maximizing speculative-decoding throughput.Source: §1 · p. 2
  • Trading depth for width while retaining the parameter count boosts speculative-decoding throughput by over 60%.The paper also describes using pruning to create favorable draft configurations for OPT, LLaMA, and LLaMA-3.1.Source: §1 · p. 2
  • Up to 111% higher throughput than existing models is reported for the redesigned draft models.The conclusion also says the pruned models can be used without modification for supervised fine-tuned targets and that the approach generalizes to state-of-the-art models.Source: §1 · p. 2
  • The redesigned drafts reduce KV caches by 37%, enabling larger batch sizes, and outperform self-speculative decoding in several setups.The paper identifies self-speculative decoding as Zhang et al. (2023).Source: §1 · p. 2

2 Background and Related Work

The background explains speculative decoding as parallel verification of tokens proposed by a smaller draft model, while related work situates this paper among other LLM-serving and decoding approaches. The paper focuses on measuring how draft-model choice affects throughput and using those observations to design drafts.

2.1 Background

  • Decoder-only LLM inference has parallel prefill followed by sequential autoregressive decoding, which is memory-bandwidth bound on modern GPUs.Speculative decoding has a smaller draft generate multiple tokens for parallel verification by the target; it can potentially speed decoding when more than one token is accepted on average.Source: §2.1 · p. 2
  • For OPT drafts, the figure caption reports that throughput falls as model size increases, despite consistent increases in TAR.The caption attributes the decline to higher inference latency.Source: Figure 2 · p. 3

2.2 Related Work

  • The paper distinguishes its focus on speculative decoding from other LLM-serving approaches aimed at better hardware utilization, context-based pruning, or shallower, wider translation networks.The related-work discussion also connects speculative decoding to speculative execution in hardware.Source: §2.2 · p. 3
  • Prior speculative-decoding work includes draft training and early-exit methods, while this paper benchmarks draft choices and uses the findings to design throughput-oriented drafts.Early-exit approaches are described as having capped latency benefits; other directions include multiple generation heads, encoder-decoder drafts, multiple draft tokens, distillation, and long-context scenarios.Source: §2.2 · p. 3

3 Understanding Speculative Decoding

The study finds that draft latency limits speculative decoding, with model depth driving latency more strongly than width and language-task accuracy tracking TAR only weakly. Its experiments compare target and draft models across several tasks and decoding configurations.

3 Understanding Speculative Decoding

  • The evaluation uses OPT-66B and LLaMA-65B targets with OPT and LLaMA draft variants on MMLU, Hellaswag, and Chatbot Arena.Experiments use batch size 1 and greedy decoding; MMLU is 5-shot, while the remaining datasets use zero-shot evaluation.Source: §3 · p. 4

3.1 Bottlenecks in Speculative Decoding

  • Draft-model latency remains a key bottleneck because draft tokens are generated autoregressively, while target verification processes the candidate sequence in one prefill-like operation.The authors report that verification time stays constant even when look-ahead is scaled to hundreds of tokens.Source: §3.1 · p. 5

3.2 Understanding Draft Model Latency

  • At similar parameter counts, autoregressive decoding latency scales linearly with layer depth, while increasing attention heads from 16 to 36 leaves latency almost constant even when width doubles.The experiments support the existence of more latency-efficient architectures at the same parameter budget, though changing depth and width also affects prediction quality and TAR.Source: §3.2 · p. 5

3.3 Understanding Draft Model TAR

  • TAR correlates little with task accuracy, so a draft with higher task accuracy can have similar TAR to one with lower accuracy.The authors suggest that most sentence tokens are not content words and therefore do not affect task accuracy.Source: §3.3 · p. 6

4 Draft Model Design for Speculative Decoding

The design section trades draft-model depth for width to reduce latency while preserving useful token acceptance, then tests throughput, memory, and generalization across models and decoding settings.

4.1 Draft Model Design

  • The wide 1.3B draft reduced latency by 49% and improved throughput by 41% versus the deep variant, despite 3% lower token acceptance rate on MMLU.Results were reported as similar on Chatbot Arena and Hellaswag.Source: §4.1 · p. 6, Table 3 · p. 7
  • 97.7% to 111.8% higher throughput came from NoFT-Wide-796M than Sheared-LLaMA-1.3B under temperature sampling, using 0.8% of its training tokens.The reported results span various datasets and models.Source: Table 2 · p. 7, §4.1 · p. 7
  • Reducing draft models below five layers sharply lowered alignment, while target-model latency exceeded 80% of each decoding cycle at five layers.Further draft-latency reductions then yielded only marginal overall latency gains, while lower token acceptance reduced throughput.Source: §4.1 · pp. 6–7
  • The wider 1.3B design reduced KV-cache requirements by more than 37%, enabling larger batch sizes with minimal effect on autoregressive decoding latency.The reported comparison is against a same-sized 1.3B model.Source: Table 4 · p. 7, §4.1 · p. 8, Table 5 · p. 8
  • 42.6% higher throughput than LLAMA-3.2-1B and 51.8% higher than self-speculative decoding were achieved by Wide-829M with LLAMA-3.1-8B as target.The pruned model was distilled over 1 million tokens to recover token acceptance after pruning.Source: §4.1 · p. 8, Table 6 · p. 8

4.2 Ablation Studies

  • NoFT-Wide-796M achieved similar token acceptance when the target changed to LLAMA-2-70B, supporting its use with models based on similar training recipes and tokenizers.The draft was distilled from LLAMA-7B.Source: §4.2 · p. 8, Table 7 · p. 8
  • For Vicuna 33B targets, NoFT-Wide-796M outperformed Tiny-LLAMA-1.1B by up to 45%, despite lower token acceptance rates on MMLU and Hellaswag.Tiny-LLAMA-1.1B had 4x higher latency and 22 layers, compared with five layers for NoFT-Wide-796M.Source: §4.2 · p. 8, Table 8 · p. 9
  • On CNNDM and XSum with LLAMA-2-70B targets, LLAMA-NoFT-Wide-796M outperformed self-speculative decoding by up to 53%.The draft has five attention and MLP layers each; self-speculative decoding computes 33 attention and 43 MLP layers in the described setup.Source: §4.2 · p. 9

5 Conclusion

The conclusion identifies draft-model depth and language-model accuracy as important design considerations for speculative decoding. It reports higher throughput from shallower, wider pruned drafts and generalization across target-model types.

5 Conclusion

  • Draft-model latency is bottlenecked by depth, while language-model accuracy does not correlate strongly with speculative-decoding performance.The paper proposes pruned drafts that trade depth for width and align with the target model.Source: §5 · p. 9
  • The proposed draft models increase throughput by up to 111% over existing models.Source: §5 · p. 9
  • The pruned drafts can be used with supervised fine-tuned target models without modification, and the approach generalizes to state-of-the-art models.Source: §5 · p. 9

Limitations

The work studies lossless speculative-decoding throughput, leaving approximate efficient inference algorithms for future work.

Limitations

  • The study focuses on lossless speculative-decoding throughput, where lossless means preserving the target LLM’s output distribution.Broader studies of approximate efficient inference algorithms are left for future work.Source: Limitations · p. 9

A More Experiment Results

The supplied passage identifies a table reporting draft-model autoregressive generation latency.

A More Experiment Results

  • Table 11 reports the latency of each autoregressive generation step of the draft model.Source: Table 11 · p. 11

A.1 OPT analysis

On Chatbot Arena, larger OPT and LLAMA models generally have lower throughput despite higher TAR; on MMLU, task accuracy does not track TAR.

A.1 OPT analysis

  • As OPT and LLAMA model size increases, Chatbot Arena throughput generally decreases despite consistent increases in TAR.The passage attributes the throughput decline to higher inference latency.Source: §A.1 · p. 11
  • Task accuracy on MMLU is reported as irrelevant to TAR for OPT and LLAMA models.The accuracy values come from the OpenLLM Leaderboard (HuggingFace, 2023).Source: §A.1 · pp. 11–12, Figure 8 · p. 12

B Discussion

  • The discussion considers how changes in models or underlying hardware may affect the study's insights.Source: B Discussion · p. 12

B.1 Future Draft Model Design

The analysis weighs the TAR advantage and latency cost of larger draft models, finding that substantial TAR or latency improvements may be needed for throughput parity with OPT-125M.

B.1 Future Draft Model Design

  • The simplified performance model removes the mutual-independence assumption and almost perfectly captures real speculative-decoding throughput.The model is used to examine the TAR gain or latency improvement needed to change draft-model choice.Source: §B.1 · p. 12
  • Up to 56% latency reduction is needed for larger OPT drafts to match OPT-125M throughput; OPT-1.3B requires 32.9%.Source: Table 12 · p. 13, §B.1 · p. 13
  • At high throughput requirements, OPT-6.7B may fail to meet the target, and the TAR gap between drafts at a given throughput is larger than in Figure 2.Figure 10 presents the required TAR to achieve a given throughput.Source: §B.1 · p. 13, Figure 10 · p. 13
  • A 1.3B OPT draft needs a TAR advantage greater than 2 over OPT-125M to match throughput on MMLU; OPT-2.7B and OPT-6.7B cannot surpass it under the TAR-8 cap.The cap follows from the length of draft-token generation, and the required TAR gain for OPT-6.7B exceeds it.Source: §B.1 · p. 13, Figure 11 · p. 13

C OPT-350M Configurations

The OPT-350M variants explore width and depth tradeoffs while keeping parameter count close to the original model.

C OPT-350M Configurations

D Simplifying Analytical model

The paper simplifies the speculative-decoding cost model to avoid assuming token independence, then tests its throughput predictions against real measurements.

D Simplifying Analytical model

  • Figure 12 compares predicted and measured throughput for LLAMA and OPT draft models serving LLAMA-65B and OPT-66B on MMLU.Measurements used four 80GB A100 GPUs and 100 real-server iterations; error bars show measurement standard deviation.Source: Figure 12 · p. 14, D Simplifying Analytical model · p. 14
  • The simplified analytical model removes the original model's assumption that generated tokens are mutually independent.The original model uses expected token acceptance rate and look-ahead length to describe speculative-decoding speedup.Source: D Simplifying Analytical model · p. 14
  • The model decomposes throughput assuming draft execution and target verification happen sequentially.TAR denotes the average number of matched tokens across a query or dataset.Source: D Simplifying Analytical model · p. 14
  • The maximum deviation between the analytical model and real measurements was 3.5%, supporting its accuracy in predicting speculative-decoding throughput.Source: D Simplifying Analytical model · p. 14
Loading 2402.01528v4…