Source-linked AI summary
Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation
Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, Zhifang Sui
TL;DR
Autoregressive decoding’s low parallelism limits inference efficiency and real-time deployment. SpecDec addresses this with an independent Spec-Drafter and Spec-Verification within a draft-then-verify framework, reporting around 5× speedup with comparable generation quality to beam search and additional practical advantages.
Problem
Autoregressive decoding has low parallelism, causing poor inference efficiency and limiting advanced AR models’ use in real-time scenarios.
Method
SpecDec uses an independent Spec-Drafter and Spec-Verification to draft multiple tokens and verify them efficiently for seq2seq generation.
Results
Around 5× speedup is achieved on seq2seq tasks with comparable generation quality to beam search decoding.
Takeaways & Limitations
SpecDec demonstrates practical value for real-world generative-model acceleration through additional deployment advantages beyond speedup.
Takeaways & Limitations
SpecDec adds test-time memory cost and is particularly suitable when GPU memory is abundant but latency improvement is urgent.
Abstract
from arXiv · showhide
We propose Speculative Decoding (SpecDec), for the first time ever, to formally study exploiting the idea of speculative execution to accelerate autoregressive (AR) decoding. Speculative Decoding has two innovations: Spec-Drafter -- an independent model specially optimized for efficient and accurate drafting -- and Spec-Verification -- a reliable method for verifying the drafted tokens efficiently in the decoding paradigm. Experimental results on various seq2seq tasks including machine translation and abstractive summarization show our approach can achieve around $5\times$ speedup for the popular Transformer architectures with comparable generation quality to beam search decoding, refreshing the impression that the draft-then-verify paradigm introduces only $1.4\times$$\sim$$2\times$ speedup. In addition to the remarkable speedup, we also demonstrate 3 additional advantages of SpecDec, revealing its practical value for accelerating generative models in real-world applications. Our models and codes are available at https://github.com/hemingkx/SpecDec.
1 Introduction
Speculative Decoding studies draft-then-verify acceleration for seq2seq generation, introducing Spec-Drafter and Spec-Verification to address AR decoding’s low parallelism. Experiments report around 5× speedup with comparable generation quality to beam search, alongside practical deployment advantages.
- Motivation: Autoregressive decoding generates tokens sequentially, limiting parallelism and inference efficiency on modern devices and restricting real-time deployment of advanced AR models.The paper frames low parallelism as a source of high deployment costs.
- Approach: SpecDec applies the draft-then-verify paradigm to seq2seq generation by drafting multiple tokens and verifying them with the existing AR model.The work draws inspiration from speculative execution in computer architecture.
- Approach: Spec-Drafter is an independent model specialized for accurate and efficient drafting, designed according to Capability and Latency principles.These principles target the drafting process rather than the existing AR model itself.
- Approach: Spec-Verification relaxes vanilla verification so more drafted tokens can be accepted without sacrificing generation quality, improving decoding efficiency.Its output may differ slightly from AR greedy decoding while retaining the stated quality guarantee.
- Results: Around 5× speedup is reported across machine translation and abstractive summarization for popular Transformer architectures, with comparable generation quality to beam search.The paper contrasts this result with prior draft-then-verify work reporting 1.4×∼2.0× speedup.
- Practical advantages: SpecDec additionally offers a better latency-throughput trade-off, easy adaptation to existing models, and retention of the original model’s behavior.These three advantages are presented as evidence of practical value for real-world acceleration.
2 Background: draft-then-verify decoding
Draft-then-verify decoding drafts several tokens efficiently, verifies them in parallel with the original AR model, and retains only the matching prefix before repeating the process.
- Overview: The draft-then-verify paradigm drafts multiple tokens as speculative AR results and then verifies them in parallel.The process is described as an implicit implementation of speculative execution in Transformer inference.
- Draft: Blockwise Decoding adds k −1 FFN heads to an existing AR model to predict the next k drafted tokens in parallel.This is presented as a representative model-based drafting approach.
- Verify: The original AR model verifies the drafted tokens in parallel and identifies bifurcation position c as the largest index preserving agreement before that position.All previous c −1 drafted and corresponding AR-decoded tokens must be identical.
- Output: Drafted tokens after position c are discarded, while the final decoded tokens combine the accepted drafted prefix with the AR-decoded token at the bifurcation point.The supplied passages identify both the discard rule and the resulting-token construction.
- Iteration: The drafting and verification steps repeat until the termination condition is met.This iteration produces the complete decoded sequence.
3 Speculative Decoding
SpecDec accelerates seq2seq autoregressive decoding through a specialized drafter and a relaxed verification strategy. Its design targets both drafting accuracy and latency while accepting suitable drafted tokens efficiently.
- SpecDec overview: SpecDec combines Spec-Drafter and Spec-Verification to improve drafting and verification in the draft-then-verify paradigm.Spec-Drafter addresses drafting, while Spec-Verification relaxes verification criteria.
- Design principles: Drafting accuracy affects accepted tokens per iteration and therefore both total drafting and verification latency.More capable drafting can increase accepted tokens per iteration and reduce the number of decoding iterations.
- Spec-Drafter: Spec-Drafter uses an independent encoder-decoder with distinct attention queries for drafted positions, unlike Blockwise Decoding’s shared attention mechanism.The design is intended to preserve drafting capability while modeling different target positions separately.
- Spec-Drafter: Spec-Drafter reduces decoder layers and reallocates capacity to the encoder because the decoder is repeatedly forwarded during iterative decoding.This encoder-favored design aims to lower drafting latency with little generation-quality degradation.
- Spec-Verification: Spec-Verification accepts drafted tokens among the top-β AR candidates when their log-likelihood gap remains within τ, rather than requiring top-1 matches.The relaxed criterion can accept drafts that differ slightly from AR top-1 results.
4 Experiments
Experiments show SpecDec accelerates Transformer seq2seq decoding substantially through its drafter and verification design, while preserving or improving generation quality and offering practical deployment advantages.
- Results: 4.6×∼5.5× speedup across translation benchmarks is achieved by SpecDec, with improved BLEU quality compared with AR greedy decoding.The comparison uses Transformer-base models and contrasts SpecDec with Blockwise Decoding.
- Drafting: Spec-Drafter outperforms the head-based drafter in end-to-end generation quality and efficiency.Reducing drafter capacity or using a balanced encoder-decoder architecture substantially harms acceleration performance through lower acceptance or higher iteration latency.
- Drafting: SpecDec performs best at block size k = 25 with 7.89 mean accepted tokens per iteration; larger blocks reduce performance because drafting too many tokens is difficult.Blockwise Decoding instead performs best at k = 10.
- Verification: Moderately relaxed Spec-Verification increases accepted tokens, speed, and quality, but over-relaxation lowers BLEU from 26.97 to 26.58.The selected hyperparameters are β = 3 and τ = 1.0; β = 5 and τ = 5 reaches almost 7× speedup but lowers BLEU.
- Practical Value: SpecDec offers better latency-throughput trade-offs, adapts to existing models, and retains the original model’s behavior.Its reported consistency exceeds 85%, compared with around 55% for a newly built fast NAR model.
5 Related Work
Related work includes early draft-then-verify methods, later speculative-decoding studies, and non-autoregressive decoding approaches. The paper positions SpecDec as a more thorough formal investigation of drafting and verification for Transformer inference.
- Speculative Decoding: Later studies acknowledge, explore, or adopt speculative decoding to accelerate Transformer inference, including approaches using small autoregressive drafters and advanced sampling.The paper identifies several subsequent studies and notes that one uses the same name, Speculative Decoding.
- Early Draft-then-verify attempts: Earlier draft-then-verify methods used input-guided verification or additional decoder heads but did not fully investigate the paradigm’s potential.The paper describes Blockwise Decoding as generating k positions in parallel and verifying them with the original head.
- Non-autoregressive Decoding: Non-autoregressive decoding generates multiple tokens in parallel and has motivated methods for improving quality through alignment objectives and target-token dependency modeling.The paper presents NAR decoding as a separate line of efficient generation research.
6 Conclusion
SpecDec applies speculative execution to seq2seq generation through dedicated drafting and verification phases. The paper reports substantial Transformer acceleration and argues that the method is practical for real-world deployment.
- Conclusion: SpecDec explicitly embraces speculative execution for seq2seq acceleration and formally studies both drafting and verification phases.The paper presents this as its first formal treatment of the approach for seq2seq generation.
- Conclusion: An appropriately invested auxiliary drafter increases computational parallelism and substantially speeds up Transformer inference.The paper attributes the speedup to better utilization of computing resources.
- Conclusion: SpecDec’s acceleration and experimentally demonstrated advantages support its practical value for model deployment in real-world applications.The paper suggests the paradigm may evolve into a standard for efficient Transformer decoding.
Limitations
SpecDec adds a separate Spec-Drafter module, increasing test-time memory consumption. The method is therefore especially suited to settings with abundant GPU memory and urgent latency requirements.
- Limitations: Extra Spec-Drafter memory cost at test time constrains SpecDec’s deployment setting.The paper frames the method as trading surplus GPU memory for speed improvements.
- Limitations: SpecDec is particularly suitable when GPU memory is abundant but latency improvement is urgently needed.The authors contrast memory as a manageable constraint with latency as a persistent deployment bottleneck.
B.1 Additional Memory Cost by SpecDec
SpecDec’s additional memory cost is dominated by Spec-Drafter weights, while most intermediate storage is negligible or shared with autoregressive decoding. The main sequence-dependent intermediate costs are either batch-dividable or already present in AR inference.
- 8MB (fp32) / 4MB (fp16) is the memory cost of retaining the Spec-Drafter encoder representation when B = 32, S = 128, and d = 512.This representation has size B · S · d and remains allocated until decoding finishes.
- For short sequences, vocabulary-projection outputs scale as B · |V| · k but can be divided into small batches to avoid massive memory cost.This intermediate is usually smaller than the memory required for Spec-Drafter weights.
- For long sequences, self-attention storage grows quadratically with S and dominates intermediate-result memory in both AR and SpecDec.Because this cost is present in both decoding strategies, it does not introduce additional memory cost compared with AR.
- SpecDec memory comparisons use peak GPU memory during inference on WMT14 EN-DE and CNN-DM scenarios.The reported measurements use torch.cuda.max_memory_allocated().
- Spec-Drafter weights account for most of SpecDec’s additional memory cost, while intermediate-variable storage is negligible because drafting and verification alternate.The additional memory cost is not very likely to increase significantly with batch size or sequence length.
B.2 Memory Is Rarely the Bottleneck
Experiments with T5-large indicate that online seq2seq deployment is generally constrained by latency rather than GPU memory. This supports accepting SpecDec’s additional memory cost in exchange for lossless acceleration.
- Over 1 second of MT latency is too slow for engines that typically require less than 100ms, despite memory use below 2GB on a 48GB A40.The latency and memory measurements use T5-large with fp16 on one Nvidia A40 GPU.
- T5-large is rarely deployed for online service because its size and serving cost are high.
- Nearly 5 seconds of summarization latency is too long for online service even at batch size 32, while memory remains below 50% of one A40 GPU.The memory capacity is 48GB, and the experiment uses greedy decoding.
- Latency is the bottleneck of seq2seq models for online deployment in most cases, rather than memory.The authors therefore frame SpecDec as a time–memory trade-off favoring significant lossless acceleration.
C SacreBLEU and COMET Scores
SpecDec achieves evaluation scores on par with AR under SacreBLEU and COMET, while its speed advantage remains substantial across WMT14 EN-DE examples. The authors recommend SacreBLEU for future comparisons because tokenized BLEU can vary with preprocessing.
- SpecDec achieves performance on par with the AR model under SacreBLEU and COMET evaluation.The scores are reported for WMT14 EN-DE and supplied alongside tokenized BLEU results.
- Up to 1.8 BLEU points of deviation can result from inconsistent tokenized-BLEU usage, motivating the recommendation to use SacreBLEU.
- 3×∼7× speedup is achieved for most WMT14 EN→DE sentences, with rare cases exceeding 10×∼11× versus AR beam search.The distribution covers 3,003 test sentences with k = 25 compared with Transformer-base beam size 5.
- Knowledge distillation can make greedy decoding comparable to beam search, while greedy decoding is more common in cost-sensitive online deployment.The cited discussion presents this as a practical reason SpecDec need not be limited to beam-search settings.
- 4× ∼6× faster performance with comparable beam-search quality is reported for Spec-Verification in Table 1.The paper characterizes Spec-Verification as an approximate and heuristic solution similar to beam search.
F Carbon Emission
SpecDec uses more instantaneous GPU power but substantially shortens GPU runtime, reducing total energy use and carbon emissions for the evaluated translation workload. The paper computes energy and emissions under a fixed device and carbon-intensity setting.
- E = P × t × PUE calculates total energy, with PUE set to 1.1; CO2eq uses 0.385g CO2eq/Wh as the carbon-intensity factor.The carbon-intensity factor is based on the US national average.
- 28% higher GPU power consumption accompanies 540% shorter GPU hours for SpecDec than autoregressive decoding.The comparison translates 3,000 sentences using the same P100 GPU device.
- 23.7% of autoregressive decoding’s total energy consumption is used by SpecDec, yielding a 4.2× reduction in carbon emission.The authors report this as a reduction rather than an increase in carbon emissions.
G.1 Speculative Decoding with an Autoregressive Drafter
For seq2seq generation, autoregressive drafting achieves limited acceleration compared with SpecDec. SpecDec combines a specially trained drafter with efficient non-autoregressive drafting to reach nearly 5× speedup without quality degradation.
- 4.9× speedup: SpecDec substantially exceeds the 1.6×∼2.8× end-to-end speedup reported for autoregressive drafting in seq2seq generation.The comparison concerns acceleration of seq2seq generation, where AR drafting is described as severely limited.
- Almost 5× speedup without quality degradation is achieved on T5-XXL (11B) translation using a 0.5B Spec-Drafter.The comparison uses WMT14 EN-DE translation and a 24-layer encoder, 6-layer decoder Spec-Drafter.
- SpecDec's drafter is specially learned to align its draft results with the target model's results.
- A deep-encoder-shallow-decoder architecture and non-autoregressive draft generation make SpecDec more efficient than autoregressive drafting.
- During verification, accepted tokens are retained, an inappropriate token can be replaced, and tokens after the bifurcation position are discarded.An example shows “Nach den” accepted, “Angaben” replaced with “vorliegenden,” and later drafted tokens discarded.