PaperlayerView exact revision on arXiv

Source-linked AI summary

GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding

Cunxiao Du, Jing Jiang, Xu Yuanchen, Jiawei Wu, Sicheng Yu, Yongqi Li, Shenggui Li, Kai Xu, Liqiang Nie, Zhaopeng Tu, Yang You

arXiv:2402.02082v1cs.CL

TL;DR

Speculative decoding reduces LLM latency by using a small draft model, but its speed depends on useful proposals accepted by the target. This paper introduces GLIDE, which reuses the target KV cache, and CAPE, which expands proposals using confidence scores. Experiments report higher acceptance and walltime speedups on Vicuna models.

  • Problem

    Speculative decoding uses a small draft model to reduce LLM latency, but acceleration depends heavily on the acceptance rate of its proposed tokens.

    Source: §1 · p. 1

  • Method

    GLIDE reuses the target model’s KV cache through draft-model cross-attention, while CAPE uses confidence scores to add candidate tokens for verification.

    Source: §1 · p. 2

  • Results

    Across four datasets, GLIDE improves acceptance rate by up to 23.5%, while CAPE reaches 2.50x and 2.61x walltime speedups on different Vicuna models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support GLIDE and CAPE as effective ways to accelerate speculative decoding, with their combination reporting a 2.5x speedup on Vicuna models.

    Source: §6 · p. 8

    Because the method does not change LLM outputs, it could also speed the creation of harmful or biased content, including hate speech or misinformation.

    Source: §7 · p. 9

Abstract

from arXiv · show

Speculative decoding is a relatively new decoding framework that leverages small and efficient draft models to reduce the latency of LLMs. In this study, we introduce GliDe and CaPE, two low-hassle modifications to vanilla speculative decoding to further improve the decoding speed of a frozen LLM. Specifically, GliDe is a modified draft model architecture that reuses the cached keys and values from the target LLM, while CaPE is a proposal expansion method that uses the draft model's confidence scores to help select additional candidate tokens for verification. Extensive experiments on different benchmarks demonstrate that our proposed GliDe draft model significantly reduces the expected decoding latency. Additional evaluation using walltime reveals that GliDe can accelerate Vicuna models up to 2.17x and further extend the improvement to 2.61x with CaPE. We will release our code, data, and the trained draft models.

1. Introduction

Autoregressive generation makes large language models slow, and speculative decoding depends on a draft model that proposes tokens the target accepts. GLIDE reuses the target’s KV cache, while CAPE expands proposals according to confidence; experiments report higher acceptance and faster decoding.

1. Introduction

  • Autoregressive token-by-token generation makes large language models particularly slow at larger model sizes, despite real-time application needs.Speculative decoding uses a smaller draft model to propose tokens for target-model verification.Source: §1 · p. 1
  • Prior acceleration approaches align draft and target models or propose multiple candidate sequences, sometimes requiring multiple models or sacrificing fluency.The paper positions its methods as alternatives that add little computational or memory overhead.Source: §1 · pp. 1–2
  • GLIDE adds cross-attention so the draft model can reuse target-model KV-cache entries from verified tokens.The authors argue this reuse makes the draft more likely to behave like the target.Source: §1 · p. 2
  • CAPE dynamically selects additional candidate tokens using the observed positive correlation between draft confidence and target acceptance.It expands a proposal sequence rather than using multiple draft models or non-autoregressive candidate sequences.Source: §1 · p. 2
  • GLIDE and CAPE together reach 2.50x and 2.61x walltime speedups on different Vicuna models.GLIDE improves acceptance rate by up to 23.5% across four datasets using Vicuna and Mistral as target models.Source: §1 · p. 2

2. Related Work

Prior speculative-decoding work improves draft efficiency or acceptance through alternative heads, multiple candidates, and distillation. This work focuses on acceptance by reusing the target KV cache and expanding proposals.

2. Related Work

  • Earlier speculative decoding includes independent draft models as well as approaches without a draft model.Block-wise parallel decoding was also proposed using auxiliary models to predict future tokens.Source: §2 · p. 2
  • Prior methods improve draft efficiency with multiple prediction heads or acceptance with multiple candidate sequences and distillation.The cited examples include multiple heads, tree verification, and sequence-level or online distillation.Source: §2 · p. 2
  • GLIDE reuses the target model’s KV cache and CAPE adds highly ranked candidates to proposals, both targeting acceptance rate.The paper identifies acceptance rate as one of the main factors in speculative-decoding speedup.Source: §2 · p. 2

3. Background: Speculative Decoding (SD)

Speculative decoding uses a fast draft model to propose tokens and a slower target model to verify them in parallel. Accepted tokens and one additional token form the prefix for the next round.

3. Background: Speculative Decoding (SD)

  • The draft model autoregressively proposes γ tokens, which the target model verifies in parallel and accepts up to the first rejected token.Verification also returns one additional token for free.Source: §3 · p. 3
  • The accepted tokens and the additional target-generated token are appended to the prefix for the next speculation round.The accepted count n can range from zero to γ.Source: §3 · p. 3
  • Speculative decoding and speculative sampling preserve the target model’s outputs under greedy search and random sampling, respectively.The passage attributes this equivalence result to Leviathan et al. (2023).Source: §3 · p. 3

4. Our Method: GLIDE with a CAPE

GLIDE lets a draft model reuse the target’s cached keys and values, while CAPE adds confidence-selected alternatives to draft proposals. Training simulates delayed KV access, and a modified verification mask handles expanded proposals.

4.1. GLIDE: Glimpse Draft Model

  • After verification, GLIDE retains target KV entries for accepted tokens and discards entries for rejected tokens before draft generation resumes.The draft then attends to the retained target cache while proposing the next tokens.Source: §4.1 · p. 3, Figure 2 · p. 4
  • GLIDE inserts cross-attention into each draft-model layer so its queries attend to the target’s cached keys and values for the verified prefix.Draft hidden vectors are projected into one query matrix per target-model head, and draft layers attend to corresponding upper target-model layers.Source: §4.1 · p. 3
  • Training uses a block-wise attention mask to simulate the delayed target KV cache available during inference.Training sequences are divided into blocks of length 5 in the experiments; the mask is not used at testing.Source: §4.1 · p. 4

4.2. CAPE: Confidence-Aware Proposal Expansion

  • CAPE associates each proposed token with an expansion set of other high-probability tokens for that position.The expansion-set size is chosen from the top token’s confidence score: 7, 5, 3, or 1 for confidence ranges (0, 0.3], (0.3, 0.6], (0.6, 0.8], or (0.8, 1], respectively.Source: §4.2 · p. 5
  • CAPE’s additional candidates are identified while autoregressive drafting continues and are not used to predict later tokens.The authors contrast this with beam-search expansion, which branches candidates and is more expensive.Source: §4.2 · p. 5
  • CAPE linearizes the expanded proposal for parallel target verification and applies a causal mask that restricts each token to preceding original-proposal tokens and itself.The mask uses candidate positions to preserve the original proposal’s ordering.Source: §4.2 · p. 5

5. Experiments

On MT-Bench, GLIDE+CAPE outperforms the tested baselines in walltime speedup and improves on GLIDE alone. GLIDE’s target-cache reuse raises acceptance, while further analyses examine cache-layer choice, beam-search overhead, and confidence-guided expansion.

5.1. Settings

  • Experiments use Vicuna 7b/13b/33b and Mistral 7b targets, with GLIDE evaluated on four task datasets and CAPE on MT-Bench.GLIDE is assessed using acceptance rate and expected speedup, with actual decoding speed and walltime speedup also reported.Source: §5.1 · pp. 5–6

5.2. Evaluation of GLIDE

  • GLIDE’s target-cache reuse raises acceptance by 5.4–23.5 percentage points over Vanilla Drafter and significantly beats baseline draft models across settings.GLIDE’s acceptance rate generally falls between 55% and 70%, and its expected speedup ranges from 1.67 to 2.24.Source: Figure 4 · p. 6, Table 1 · p. 6, §5.2 · p. 7
  • GLIDE accelerates decoding on MT-Bench, with accelerated Vicuna-33b decoding faster than Vicuna-7b without speculative decoding.Source: §5.2 · p. 7, Figure 5 · p. 7

5.3. Evaluation of CAPE

  • GLIDE+CAPE outperforms all tested baselines in MT-Bench walltime speedup, and adding CAPE improves GLIDE’s speedup.Walltime speedup is measured against standard decoding with only the target model.Source: §5.3 · p. 7, Figure 6 · p. 7
  • GLIDE+CAPE verifies at most 32 proposed tokens per step versus Medusa’s 64, and may therefore support larger-batch inference.Source: §5.3 · p. 8

5.4. Further Analysis

  • Using KV cache from higher target-model layers produces higher acceptance rates than using lower-layer cache, though lower-layer cache still helps versus none.Source: §5.4 · p. 8, Figure 7 · p. 8
  • Beam search raises speculation time from 5.5 ms to 10.9 ms per step on average, while CAPE adds 0.2 ms; copying and sorting drive the beam-search cost.Source: §5.4 · p. 8, §5.3 · p. 8
  • Confidence-aware expansion is useful: fixed-size expansion reduces MT-Bench decoding speed from 70.6 to 68.4 tokens/sec.Source: §5.4 · p. 8

6. Conclusions

The paper proposes GLIDE, which uses the target model’s KV cache to improve draft proposals, and CAPE, which adds candidate tokens for verification. Their integration yields a 2.5x speedup on Vicuna models; batch serving and extremely long contexts remain future work.

6. Conclusions

  • GLIDE reuses the target model’s KV cache to improve proposal generation, while CAPE expands proposals with additional candidate tokens for verification.Source: §6 · p. 8
  • 2.5x speedup on Vicuna models is reported for the integration of GLIDE and CAPE.Source: §6 · p. 8
  • Future work will explore batch serving GLIDE and its effectiveness with extremely long contexts.Source: §6 · p. 9

7. Impact Statement

The method speeds LLM inference without changing generated outputs, which may also accelerate harmful or biased content creation. The authors call for careful use and strong safeguards.

7. Impact Statement

  • Faster inference may make AI technologies more accessible and open possibilities for personal AI applications.Source: §7 · p. 9
  • Because the method does not change LLM outputs, it could speed the creation of harmful or biased content, including hate speech or misinformation.Source: §7 · p. 9
  • The authors identify careful use and strong safeguarding measures as necessary to reduce risks from faster processing.Source: §7 · p. 9

A. Experiment Details

Experiment details vary GLIDE’s architecture and training setup by target-model size. The paper also reports that distillation further improves GLIDE but doubles vanilla GLIDE’s training time, and points to additional walltime results in Table 3.

A. Experiment Details

  • 7B and 13B targets use single-layer GLIDE, while the 33B target uses two layers; both configurations have hidden dimension 4096.Source: A. Experiment Details · p. 11
  • GLIDE training took approximately 7 hours for 7B, 10 hours for 13B, and 100 hours for 33B target models.Training used zero2 with eight H800 GPUs for 7B and 13B, and zero3 with 16 H800 GPUs for 33B; the draft model was trained for one epoch on pretrain and SFT datasets.Source: A. Experiment Details · p. 11

B. Impact of Distillation

  • Seq-level distillation further improves GLIDE, but its training cost is twice that of vanilla GLIDE, so it is omitted from other experiments.Source: B. Impact of Distillation · p. 11

C. Walltime for Speculation

D. Batch Serving

Batch-serving tests duplicate input IDs and use one H800 with fp16. Small batch sizes do not substantially reduce decoding speed, but long prompts frequently cause out-of-memory errors, motivating future work on serving integration.

D. Batch Serving

  • The tests duplicate input IDs across batch entries and use one H800 with fp16; the authors identify combining speculative decoding with batch-serving systems as an important future topic.Source: D. Batch Serving · p. 11
  • Small batch sizes do not substantially reduce decoding speed because inference is memory-bounded in that setting.Source: D. Batch Serving · p. 11
  • Out-of-memory errors frequently occur with long prompts, even when running the target model alone.Source: D. Batch Serving · p. 11

E. Configurations for GLIDE

Draft-model depth and hidden size were tuned against target-model size. Smaller targets favored shallower drafts, while increasing hidden size could offset reducing depth.

  • A 12-layer llama-160m draft could reduce decoding speed because sequential layer computation outweighs the acceptance gains from added layers.Source: E. Configurations for GLIDE · p. 11
  • For targets smaller than 33B, the optimal draft has one hidden layer; for 33B targets, two, selected by grid search from llama-68m’s 2-layer, 768-dimension structure.Source: E. Configurations for GLIDE · p. 11

F. Limitation

GLIDE requires a corresponding draft model for each new LLM, although the reported training cost is described as manageable. The configuration findings also note trade-offs between draft depth and hidden size.

  • GLIDE is not plug-and-play: each new LLM requires retraining a corresponding draft, though training a 68MB draft takes about 7 hours on 8 H800 GPUs.Source: F. Limitation · pp. 11–12
  • With fewer target-model layers, adding draft layers can reduce theoretical speedup because the added time cost exceeds the acceptance-rate gain.Source: Figure 8 · p. 12
  • Increasing draft hidden size can accelerate theoretical speedup because hidden-layer elements compute in parallel, whereas layers run autoregressively.Source: Figure 9 · p. 12

G. Case Study

An MT-bench example contrasts accepted tokens from the vanilla draft, GLIDE, and GLIDE with CaPE. GLIDE predicts terms such as “bacteria” and “germs,” and CaPE further improves token acceptance.

Loading 2402.02082v1…