Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding

1 Introduction

• Autoregressive decoding makes many memory-bandwidth-bound calls, while draft-model selection and extra memory burden speculative methods, and compression can require retraining and alter outputs.On an A100, decoding 128 tokens can take up to 100× longer than a sequence-level forward pass on the same number of tokens.Source: §1 · p. 1
• Self-speculative decoding uses one LLM to draft by skipping intermediate layers, then verifies draft tokens with the original model in one forward pass.The drafting and verification paths share the same model, avoiding a standalone draft model and its additional memory overhead.Source: §1 · p. 2
• Bayesian optimization searches offline for layer combinations to skip, with the objective of minimizing average inference time per token.The search is performed at the model level.Source: §1 · p. 2
• When acceptance is below 80%, K = 1 is optimal; above 80%, a larger K is necessary, motivating adaptive draft exiting.Draft exiting stops generation when confidence falls below a threshold, avoiding unnecessary computation and potential discard of additional draft tokens.Source: §1 · p. 2
• Up to 1.99× end-to-end speedup is reported on text summarization and code generation tasks.The evaluation uses LLaMA-2 and its variants.Source: §1 · p. 2

Transformer-based LLM inference

• Inference systems optimize LLM execution, yet the section identifies a remaining gap in algorithm–systems co-design for hardware efficiency.It notes that token generation can take most end-to-end inference time compared with prompt encoding.Source: §2 · pp. 2–3

Model Compression

• Quantization, pruning, and distillation can speed inference, but often require architecture or training changes and do not preserve identical outputs.The related-work discussion cites Han et al. (2015), Molchanov et al. (2016), and Hinton et al. (2015), among others.Source: §2 · p. 3

Speculative Execution

• Prior speculative decoding methods use or train high-quality draft models, increasing memory overhead; copying reference text tokens instead relies on repetitiveness.The paper contrasts these approaches with a method that does not require additional memory or explicit assumptions about data distribution.Source: §2 · p. 3

Early Exit

• Early-exit work varies calculation paths at inference, and the paper draws inspiration from skipping intermediate layers during drafting.The related work mentions encoder-only Transformers and an encoder-decoder LLM.Source: §2 · p. 3

Autoregressive Decoding (Greedy)

• The greedy autoregressive baseline repeatedly selects the highest-probability next token until the target sequence length is reached.The algorithm takes the model, prompt, and target length as inputs.Source: §2 · p. 3

3.1 Standard Autoregressive Decoding

• Autoregressive decoding is memory-bandwidth-bound because each token requires the model parameters to pass through the accelerator.The model size divided by memory bandwidth sets a ceiling on decoding speed.Source: §3.1 · p. 3

3.2 Self-Speculative Decoding

• The method uses one LLM to draft tokens with selected intermediate layers skipped, then verifies them with the original LLM.Verification uses a single forward pass; the method faces trade-offs in how many draft tokens to generate and when to stop.Source: §3.2 · p. 4

3.3 Selection of Skipped Layers

• Skip-layer selection balances acceptance and drafting speed: skipping too many layers can lower acceptance, while skipping fewer limits potential speedup.Optimizing acceptance alone could select no skipped layers and yield no speedup; skipping all layers can lower acceptance enough to negate speed gains.Source: §3.3 · pp. 4–5
• Bayesian optimization searches skip-layer combinations to minimize average inference time per verified token, with Figure 3 illustrating the search.It iteratively evaluates layer choices using a Gaussian-process surrogate and an acquisition function; the chosen configuration is fixed per model, while task-level optimization is suggested when tasks vary significantly. The scheme can also be combined with quantization and sparsification.Source: §3.3 · p. 5, Figure 3 · p. 5

3.4 Adaptive Draft-Exiting Mechanism

• Drafting stops when a token’s predicted probability falls below threshold γ, avoiding computation on later draft tokens likely to be discarded.The threshold adapts using acceptance rate after each verification stage to keep that rate near target α.Source: §3.4 · pp. 5–6

4.1 Setup

• Evaluation covered LLaMA-2-13B, LLaMA-2-13B-Chat, CodeLLaMA-13B, and LLaMA-2-70B; BO selected skipped layers over 1,000 iterations, with text and code benchmarks.CNN/DM and XSum used 1-shot evaluation and ROUGE-2; HumanEval used pass@1 and pass@10.Source: §4.1 · p. 6, §4.2 · p. 6

4.2 Main Results

• 1.210×–1.992× speedups over Autoregressive on text tasks at temperatures 0.0/0.2 came with no ROUGE-2 loss; CodeLLaMA-13B reached 1.345×/1.456× with similar pass@1/pass@10.Source: §4.2 · p. 6, Table 1 · p. 7, Table 2 · p. 7

4.3 Impact of Skipped Layer Selection

• Skipping about half the layers produced peak end-to-end speedup, but layer combinations mattered and skipping more than 42 layers reduced speedup.Some inappropriate layer combinations decreased end-to-end inference speed.Source: §4.3 · p. 6, Figure 4 · p. 7

4.4 Effectiveness of Draft-Exiting

• With fixed K, speedup first rose and then fell; K=8 wasted computation on draft tokens likely to fail verification, while K=4 reached only 1.44×.K=2 had a 0.924 acceptance rate but only 1.37× speedup, so acceptance rate and speedup were not directly proportional.Source: §4.4 · p. 7, Table 3 · p. 8
• Static thresholds yielded 1.38×–1.58× acceleration; γ=0.8 had AR=0.935 but 1.55× speedup, while γ=0.2 had AR=0.749 and 1.38× speedup.Source: §4.4 · pp. 7–8, Table 4 · p. 8
• Adaptive threshold updating achieved 1.57× speedup, comparable to or better than carefully tuned static thresholds, by adjusting γ toward a target acceptance rate across instance difficulties.The adaptive draft-exiting mechanism was reported as insensitive to changes in K.Source: §4.4 · p. 8

4.5 Evolution of Draft-Exiting Threshold

• Across roughly 5,000 drafting and verification iterations, γ varied substantially across models and datasets, supporting adaptive updates over static thresholds.Source: §4.5 · p. 8, Figure 5 · p. 8

4.6 Breakdown of Computation

• Skipped-layer selection and adaptive draft-exiting contributed to faster average per-token inference, while drafting consumed most latency and γ updates added negligible computational cost.Source: §4.6 · p. 8, Table 5 · p. 8

• Self-speculative decoding accelerated LLaMA-2 and fine-tuned models by up to 1.99× without additional neural network training or extra device memory.Source: §5 · p. 8

• The research used open-source datasets, models, and tools and focused exclusively on improving model inference efficiency.Source: §6 · p. 9

7 Limitations

• No-training drafting limits how many layers can be skipped: excessive reduction can lower acceptance rate and diminish speedup.Fine-tuning may mitigate this but adds memory overhead because the draft no longer shares the original model’s parameters.Source: §7 · p. 9

A Data

• Evaluation covers CNN/DM and XSum summarization plus HumanEval code generation, using ROUGE-2 and pass@10, respectively.Efficiency measurements randomly sample 1,000 CNN/DM and XSum test instances.Source: A Data · p. 12

B Setup

• Offline Bayesian optimization uses eight training instances; 1,000 iterations take about 2.5 hours for 13B models and 6 hours for LLaMA-2-70B.The search is performed once offline at the model level.Source: §7 · p. 9, B Setup · pp. 12–13

C Which layers are skipped?

• Skipped layers favor attention and cluster in the model’s latter half, consistent with attention latency and reported redundancy in later layers.The section links attention skips to attention’s contribution to per-token latency and suggests the first half predicts most tokens accurately.Source: C Which layers are skipped? · p. 13

D Effect of Skip Strategy

• Against BO’s 24 attention and 12 MLP skips, fixed first, middle, last, or random choices yielded minimal acceleration; last-layer skips were slightly better.The authors attribute the last-layer result tentatively to greater redundancy in the model’s final portion.Source: D Effect of Skip Strategy · p. 13

E Number of Iterations of BO

• 1.35× speedup is achieved with 200 Bayesian-optimization iterations on LLaMA-2-13B for CNN/DM; 800 and 1,000 iterations give consistent performance.The 800- and 1,000-iteration searches select the same skipped layers.Source: E Number of Iterations of BO · p. 13

F Aggressive Skip

• The 75-million-token aggressive-skip run has reduced acceleration, which the authors attribute tentatively to overfitting; training also requires extra time and draft-model storage.They consider this trade-off acceptable for users with ample resources.Source: F Aggressive Skip · pp. 13–14
• At 60M training tokens, aggressive skipping yields AR 0.900 and speedup 1.96×, versus 1.57× in Table 1; without training, AR is 0.695 and speedup 1.15×.The strategy skips top-K layers according to Bayesian-optimization probabilities.Source: F Aggressive Skip · p. 13, Table 9 · p. 14

G Effect of Max # of Draft Tokens

• With adaptive draft exiting, speedup stays relatively stable as maximum draft-token K increases because most tokens exit early; the default K is 12.The authors report that a relatively large K lets adaptive exiting operate optimally.Source: Table 10 · p. 14, G Effect of Max # of Draft Tokens · p. 15

H Adaptation

• Layer skipping is selected to avoid extra parameter storage, and the approach can be combined with quantization or sparsification for users with limited computing resources.The section describes these combinations as adaptations for resource-constrained settings.Source: H Adaptation · p. 15

H.2 Sparsification

• Sparsification reduces average inference latency per token by varying amounts but does not dramatically reduce VRAM; speedup is slightly lower because the base model is also accelerated.The experiments include unstructured and structured N:M sparsity.Source: Table 12 · p. 15, §H.2 · p. 15

H.1 Quantization

• Fp4 quantization reduces video-memory demand to 19.6 GB, nearly half, although dequantization may increase average inference latency per token.The evaluated schemes are fp8, fp4, and nf4 against default bf16.Source: §H.1 · p. 15

I Exploration of Diverse Tasks

• The authors state that the method’s lossless characteristic is task-agnostic, while noting that evaluation on more diverse tasks remains open.The section extends exploration to GSM8K and MT-bench.Source: I Exploration of Diverse Tasks · pp. 15–16
• GSM8K evaluation reports problem-solving accuracy on 1,318 test questions, while MT-bench averages GPT-4 scores across 160 answers to 80 two-turn questions.The MT-bench setup uses LLaMA-2-13B-Chat and evaluates eight prompt categories.Source: I Exploration of Diverse Tasks · p. 16
• On GSM8K, the approach achieves 1.35× acceleration without compromising task performance; on MT-bench, it achieves about 1.27× speedup with lossless task performance across diverse tasks.Source: I Exploration of Diverse Tasks · p. 16

J.1 Quantitative Analysis

• The non-finetuned, one-shot evaluation reports significant acceleration without compromising output quality, with conclusions consistent across ROUGE-2 and recorded ROUGE-1/ROUGE-L results.Source: §J.1 · pp. 16–17

J.2 Qualitative Analysis

• Greedy self-speculative decoding matches autoregressive outputs, while sampling produces the same distribution; task performance relies on the base model’s capabilities.The case study covers CNN/DM and XSum on LLaMA-2-13B.Source: §J.2 · p. 17, Table 16 · p. 18

K Algorithm with Sampling

• The random-sampling variant adds adaptive draft exiting; higher temperature or topp increases token diversity, while lower values make selection more deterministic.The authors note that diverse decoding paths may benefit scenarios such as code generation.Source: K Algorithm with Sampling · p. 17