Source-linked AI summary
Margins, Not Windows: Training-Free Per-Step Lossy Speculative Decoding
Oszkár Urbán, Young D. Kwon, Stylianos I. Venieris, Cecilia Mascolo
TL;DR
Speculative decoding reduces autoregressive inference cost, but common systems keep verification strict and draft-tree shape fixed, while prior relaxations rely on limiting assumptions. AdaptiveSpec adapts both decisions per step from internal signals, achieving up to 56% higher throughput than EAGLE-3 while retaining 93% to fully lossless task accuracy.
Problem
Autoregressive decoding is sequential, while speculative decoding systems commonly retain strict token matching and static or constrained draft-tree decisions.
Method
AdaptiveSpec combines a target-probability margin rule for mismatches with per-step tree-shape adaptation using draft confidence and rolling acceptance history.
Results
Up to 56% throughput improvement over EAGLE-3 is reported with 93% to fully lossless downstream task-accuracy retention across the evaluated tasks and target models.
Takeaways & Limitations
The two adaptive axes are empirically orthogonal and their gains compound in the implemented SGLang system.
Takeaways & Limitations
The method is instantiated on EAGLE-3, with transfer to diffusion-based drafters left for future empirical confirmation.
Abstract
from arXiv · showhide
Speculative decoding accelerates LLM inference by drafting candidate tokens and verifying them in parallel. Tree-attention drafters such as EAGLE-3 are widely adopted, yet typically hold two decisions fixed: (1) a strict token-match verification rule and (2) a static draft-tree shape. Prior work relaxes each in isolation under limiting assumptions: long draft chains for training-free lossy verification, and adaptive tree shaping under a fixed token budget. We introduce AdaptiveSpec, a training-free per-step speculative decoding method that adapts both decisions from internal signals already produced during decoding. A per-step margin rule promotes a mismatched draft-proposed token when the ratio of the target's probability on the drafted token to its top-1 probability exceeds a threshold with no dependence on draft length or underlying drafter architecture. A per-step tree policy adjusts the draft tree's depth, width, and node count directly from a fused signal of draft top-1 confidence and a rolling acceptance history capturing recent draft-target agreement, allowing the total draft count to vary rather than only be redistributed. The two adaptations operate on orthogonal axes and compound in effect. Implemented on the SGLang production-grade serving engine, AdaptiveSpec improves throughput over the state-of-the-art autoregressive speculative decoding method EAGLE-3 by up to 56%, recovering 93% to fully lossless task accuracy across GSM8K, MATH-500, and HumanEval on three target models (DeepSeek-R1-Distill-Llama-8B, Llama-3.1-8B-Instruct, Qwen3-8B).
1 Introduction
AdaptiveSpec targets speculative decoding’s fixed verification rule and static tree shape with two training-free, per-step adaptations driven by internal decoding signals. The combined method improves throughput while retaining high downstream accuracy.
- Autoregressive generation limits inference efficiency because each token requires a forward pass after its predecessor, especially for long reasoning traces.
- Speculative decoding drafts candidate tokens and verifies them in parallel, reducing the target-model passes needed for correctly predicted tokens.
- AdaptiveSpec promotes mismatched draft tokens when their target probability-to-target-top-1 ratio exceeds a threshold, without auxiliary training or drafter-specific length requirements.
- Its dynamic tree policy adjusts tree-shape hyperparameters per step using draft confidence and rolling acceptance history, rather than redistributing nodes within a fixed budget.
- 18-44% average throughput improvement per target model over EAGLE-3 accompanies 93% to fully lossless downstream task-accuracy retention.
- 56% peak throughput improvement is reported over the relevant state-of-the-art speculative-decoding regime.
2 Related Work
Related work relaxes verification rules or adapts draft-tree shapes, but each approach has structural or deployment limitations. AdaptiveSpec instead uses direct probability signals and per-step tree adaptation without lookahead or a fixed total budget.
- Speculative Decoding: Speculative decoding verifies multiple drafted candidates in one target forward pass, while tree drafters expose multiple continuations to that verification pass.
- Adaptive Draft Trees: TALON adapts depth and width online but only redistributes nodes within a fixed global token budget and lacks integration into a production-grade engine such as SGLang.
- Adaptive Draft Trees: AdaptiveSpec jointly adjusts nsteps, top-k, and ndt per decoding step, allowing draft compute to contract on weak-draft steps.
- Lossy Verification: Training-based lossy-verification methods rely on external trained verifiers, while FLy uses an entropy gate and lookahead window to accept mismatches.
- Lossy Verification: FLy’s lookahead window does not transfer cleanly when acceptance is low and is structurally incompatible with EAGLE-3’s shorter draft chains.
- Lossy Verification: AdaptiveSpec’s margin rule uses a single-position target-distribution measurement with no lookahead or trained module.
3 Method
AdaptiveSpec addresses sequential decoding by adapting both draft-tree construction and verification per step using signals already produced during decoding. Its margin rule relaxes exact matching, while its confidence-driven tree policy varies depth, width, and total draft computation.
- Overview: AdaptiveSpec uses two per-step adaptive axes: tree shaping during drafting and margin-based token promotion during verification.Both axes use internal decode-loop signals without auxiliary training or extra computation.
- Margin-based Lossy Verification: At the first mismatch, the margin rule promotes the drafted token when its target probability divided by the target top-1 probability reaches threshold κ.This single-position rule provides tunable relaxation and avoids lookahead, chain-length dependence, and external semantic-verification training.
- Dynamic Draft Tree-Shaping: The Draft Confidence Score combines draft top-1 probability with rolling acceptance history to select (nsteps, top-k, ndt) each decoding step.The rolling acceptance rate is an EMA with α = 0.3; a divisor d controls when the score saturates and biases the policy toward deep-narrow trees.
- Dynamic Draft Tree-Shaping: Confident drafts receive deep, narrow chains, whereas uncertain drafts receive wide, shallow trees across the configurable shape range.Depth interpolates with DCS and branching with 1−DCS over (3, 4, 4) ↔ (7, 1, 8).
- Dynamic Draft Tree-Shaping: Unlike fixed-budget redistribution, the tree policy changes total draft computation, shrinking compute on weak-draft steps and expanding it on confident steps.A circuit breaker restores the starting configuration after five consecutive zero-accept steps.
- Implementation: Static CUDA-graph constraints are handled by pre-capturing one graph for every selectable (nsteps, top-k, ndt) triplet and swapping among them during decoding.This preserves production-engine acceleration while enabling per-step tree selection.
4 Experiments
AdaptiveSpec is evaluated on three target models and three benchmarks in SGLang against EAGLE-3, TALON, and FLy. Results show that dynamic tree shaping, lossy verification, and their combination improve throughput while trading off or preserving accuracy depending on configuration.
- Experimental Setup: AdaptiveSpec is evaluated on Llama-3.1-8B-Instruct, DeepSeek-R1-Distill-Llama-8B, and Qwen3-8B across GSM8K, MATH-500, and HumanEval.Experiments use greedy decoding at batch size 1 on one NVIDIA A100 with SGLang and SpecForge.
- Ablation Study: Dynamic tree shaping alone improves average speedup over EAGLE-3 by +30%, +11%, and +23% on Llama-3.1-8B, Qwen3-8B, and DeepSeek-R1-8B, respectively, at full accuracy.It directly adapts tree structure per step rather than merely redistributing a fixed node budget.
- Ablation Study: Lossy verification alone matches or exceeds FLy on every target, with average speedups of 2.22× versus 2.20×, 1.47× versus 1.40×, and 2.90× versus 2.65×.On DeepSeek-R1-8B MATH-500, it reaches 2.93× versus 2.57× (+14%) with 97% versus 92% recovery.
- Main Results: Combined AdaptiveSpec achieves the best speedup in every model-benchmark cell, improving over static EAGLE-3 by 17-56%.The largest gain is 2.82× versus 1.81× (+56%) on Llama-3.1-8B HumanEval, with 103% accuracy retention.
- Main Results: On average, Combined improves over EAGLE-3 by +44% on Llama-3.1-8B, +18% on Qwen3-8B, and +38% on DeepSeek-R1-8B.The corresponding speedups are 2.46× versus 1.71×, 1.87× versus 1.58×, and 2.98× versus 2.16×.
- Ablation Study: The ablation shows each axis independently improves the EAGLE-3 baseline, while their combination reaches 2.44× average throughput and 2.46× (+44%) on Llama-3.1-8B at 98% recovery.Dynamic shaping raises average speedup from 1.82× to 2.21× without accuracy cost, while static lossy verification reaches 2.20× with an average 8% accuracy cost.
- Hyperparameter Analysis: Lower κ increases throughput but decreases recovery, so κ is selected per model-benchmark cell to balance the trade-off.Different models find different tasks difficult, causing the preferred threshold to vary.
- Limitation Analysis of FLy’s Verifier: FLy’s lookahead window is structurally mismatched with EAGLE-3: accepted lengths of 2.08-3.27 are shorter than FLy’s required w=6.AdaptiveSpec avoids this dependency by reading the target distribution at the mismatch position.
5 Conclusion
AdaptiveSpec is a training-free, per-step speculative decoder combining lossy verification with dynamic tree shaping. Its two orthogonal components compound to improve throughput while retaining high downstream task accuracy.
- AdaptiveSpec converts near-miss draft tokens into accepted tokens using only the target distribution at verification time, without an external verifier.
- Its dynamic tree-shape policy adjusts all three EAGLE-3 hyperparameters independently at each decoding step.
- 18-44% average throughput improvement per target model, with 93% to fully lossless downstream task accuracy retention.The peak single-cell improvement is +56%; evaluation covers three target models inside SGLang.
- The lossy verification and dynamic tree-shaping components act as orthogonal throughput axes whose gains compound.
6 Future Work
The paper identifies future applications of AdaptiveSpec beyond language generation and with retrieval-based drafting for repetitive agentic workloads. These directions target sequential-decoding bottlenecks where per-step adaptation may improve inference efficiency.
- AdaptiveSpec could extend to autoregressive units beyond language tokens because the sequential-dependency bottleneck is not specific to language generation.The paper highlights embodied vision-language-action models, where latency affects effective coordination.
- In physical domains, reducing decoding latency matters for agents that must coordinate perception and action without waiting seconds between steps.
- Agentic workloads produce repetitive traces, allowing previously generated sequences to be reused through retrieval-based drafting.
- A proposed direction is combining retrieval-based drafters with AdaptiveSpec’s per-step signals for repetitive agentic settings.
7 Limitations
The evaluation has three stated boundaries: empirical rather than formal distribution preservation, batch-size-one measurement, and exclusive use of EAGLE-3 drafting. These limitations leave broader task, batching, and drafter transfer unverified.
- The per-step margin rule does not formally preserve the target distribution, and its accuracy retention may vary outside the evaluated tasks.
- Evaluation uses only batch size 1, leaving AdaptiveSpec’s performance at higher batch sizes for future work.The authors expect gains to persist but do not report multibatch results.
- AdaptiveSpec is instantiated only with EAGLE-3, with no reported results for diffusion-based drafters such as DFlash or DDTree.Transfer to DDTree is expected by the authors but lacks empirical confirmation.
A EAGLE3 target–draft model pairings
Table 6 documents the target–draft model pairings used in the experiments. Each pairing lists the target model first and its EAGLE-3 draft checkpoint second.
- Table 6 lists each target model on the first line of a pairing and the corresponding EAGLE-3 draft checkpoint on the second.
B Experiment configurations
The experiment configurations define tree-shape settings and verifier hyperparameters, including per-cell margin thresholds κ for AdaptiveSpec.
- Table 7 specifies each method’s tree shape and relevant hyperparameters, including κ for AdaptiveSpec’s lossy margin rule.Tree shape is represented by (nsteps, top-k, ndt) or a dynamic range A↔B.
- Table 8 reports the per-(model, benchmark) κ values used by AdaptiveSpec’s Combined configuration.For two DeepSeek cells, Lossy-only κ values are additionally shown in parentheses when they differ from Combined.
C SpecForge Benchmarking Implementation Details
Benchmarking uses greedy, batch-size-one decoding on a single NVIDIA A100 through SGLang and the SpecForge harness, with task-specific maximum generation lengths.
- Experiments use greedy decoding at batch size 1 on a single NVIDIA A100 through SGLang and SpecForge.
- Maximum generation lengths are 2048 tokens for GSM8K and 4096 tokens for MATH-500 and HumanEval.