Source-linked AI summary
ValuSpec: Plug-and-Play Candidate Valuation before Target Verification for Tree-Based Speculative Decoding
Liang He, SiYuan Ma, Qishi Zhan, Yongqi Fan, Mingyu Cao, Qizhen Lan, Zhaolu Kang, Xilu Wang
TL;DR
Larger candidate sets can include off-trajectory candidates, increasing rejected tokens and reducing effective progress per iteration, so deciding which candidates merit verification becomes key. ValuSpec filters branches from a supplied candidate tree using a threshold calibrated on target-model greedy trajectories, then verifies the filtered tree under output-identical greedy decoding. ValuSpec achieves the highest throughput across three workloads and both target sizes, with retrieval-based speedups over AR of 2.18×–2.31× on 7B and 2.24×–2.38× on 13B.
Problem
Larger candidate sets can include off-trajectory candidates, increasing rejected tokens and reducing effective progress per iteration, so deciding which candidates merit verification becomes key.
Source: §1 · p. 2Method
ValuSpec filters branches from a supplied candidate tree using a threshold calibrated on target-model greedy trajectories, then verifies the filtered tree under output-identical greedy decoding.
Source: §1 · p. 3, §3 · pp. 4–5Results
ValuSpec achieves the highest throughput across three workloads and both target sizes, with retrieval-based speedups over AR of 2.18×–2.31× on 7B and 2.24×–2.38× on 13B.
Source: §5.1 · p. 7Takeaways & Limitations
Candidate supply and verifier-side valuation are separate optimization stages, and ValuSpec can reduce verification work without modifying candidate generation or retrieval.
Source: §5.1 · p. 8, §7 · p. 10The study focuses on greedy decoding with output-identical verification; extending candidate valuation to stochastic decoding remains future work.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Tree-based speculative decoding accelerates large language model inference by verifying multiple candidate branches in parallel. However, in practical serving scenarios, similar requests repeatedly induce overlapping candidate branches. While retrieval-based methods mitigate this by reusing historical drafts, the resulting candidate trees often contain many low-quality branches as the candidate tree grows, which reduces verification efficiency and wastes target-side computation. To address this, we propose ValuSpec, a plug-and-play candidate valuation mechanism that filters low-quality branches from a supplied candidate tree before expensive target-side verification. ValuSpec introduces transfer-based supervision to align filtering with the target model's behavior. A filtering threshold is estimated from the target model's greedy trajectories on a calibration split, without any parameter training. The target model then verifies the filtered tree under the output-identical greedy rule, so accepted tokens are guaranteed to match standard autoregressive decoding exactly. Experimental results on HumanEval, GSM8K, Dolly, and SpecBench show that, in our main evaluation, ValuSpec achieves up to 2.38x end-to-end speedup in retrieval-based tree settings. Furthermore, integrating ValuSpec into the EAGLE-3 dynamic tree framework yields up to 2.78x end-to-end speedup without modifying its tree construction strategy, demonstrating that ValuSpec serves as a generally applicable and plug-and-play candidate valuation module.
1 Introduction
ValuSpec addresses wasted target-side verification in speculative decoding by filtering low-quality branches from an already constructed candidate tree. It uses target-model greedy trajectories to calibrate filtering and preserves output identity under deterministic decoding.
1 Introduction
- Retrieval reuses historical drafts to reduce repeated online generation, but growing candidate trees can still contain branches that waste target-side verification.Such branches may match the current context superficially yet diverge early from the target model’s behavior.Source: §1 · p. 2
- Acceptance rate decreases monotonically as candidate draft length grows, indicating greater redundancy and a higher proportion of low-quality candidates.This shifts the bottleneck toward deciding which candidates are worth verifying after reuse is introduced.Source: §1 · p. 2
- ValuSpec filters low-value branches after a candidate tree is constructed and before target verification, rather than changing candidate construction.This verifier-side stage can be applied sequentially with generation-side tree optimization.Source: §1 · p. 3
- ValuSpec calibrates its filtering threshold from deterministic target-model trajectories without additional parameter training, then uses output-identical greedy verification.Accepted tokens therefore match deterministic autoregressive decoding.Source: §1 · p. 3
- ValuSpec improves acceptance-related efficiency, reduces verified tree size, and increases end-to-end throughput across different candidate suppliers.The reported gains also extend to generation-side optimized methods, supporting complementarity between candidate supply and verifier-side valuation.Source: §1 · p. 3
2 Related Work
Prior speculative-decoding work improves candidate supply through online drafting, retrieval, or dynamic tree construction. ValuSpec instead values branches in an already supplied tree before target verification.
Speculative Decoding and Candidate Supply
- Retrieval-based methods reuse historical branches to reduce repeated online generation, but may leave large, redundant trees and verifier-side inefficiency.ValuSpec complements candidate-supply methods by filtering low-quality branches from supplied trees before verification.Source: §2 · p. 3
- The paradigm comparison distinguishes conventional online generation, retrieval reuse with redundant trees, and ValuSpec’s filtering of supplied candidates.ValuSpec reduces verification redundancy without modifying candidate construction.Source: Figure 2 · p. 4
Dynamic Tree Construction and Generation-Side Pruning
- Dynamic construction and generation-side pruning shape or reduce trees during candidate supply, while ValuSpec filters an already materialized tree before verification.The two stages are described as complementary, and ValuSpec provides additional acceleration atop generation-side optimized suppliers.Source: §2 · pp. 3–4
Verifier-Side Candidate Valuation
- Verifier-side candidate valuation targets low-quality branches that retrieval-based and dynamic-tree methods can still send to the target model.ValuSpec uses transfer-based supervision to filter candidates before verification while preserving exact output equivalence.Source: §2 · p. 4
3 ValuSpec: Plug-and-Play Candidate Valuation before Target Verification
ValuSpec separates candidate supply from valuation: it scores branches in a supplied tree, filters them using a threshold calibrated from target greedy trajectories, and verifies the remaining subtree. Deterministic greedy verification preserves the target model’s output.
Overview
- ValuSpec accepts an already materialized candidate tree from an upstream supplier and applies the same valuation mechanism before target-side verification.The paper studies retrieval-based trees as its primary setting and also considers dynamic trees supplied by EAGLE-3.Source: §3 · p. 5
Transfer-based supervision and threshold calibration
- On calibration contexts, the fixed lightweight valuation model scores target-generated greedy trajectories using their mean log-probability, without parameter updates.These scores provide the basis for calibrating the transfer threshold.Source: §3 · p. 6
- The transfer threshold is a lower quantile of calibration-trajectory scores; its quantile level controls filtering conservativeness.At inference, the threshold is estimated from calibration data rather than manually tuned, and evaluation uses a separate split.Source: §3 · p. 6
Candidate valuation and target verification
- At inference, ValuSpec retains supplied branches whose valuation scores meet or exceed the calibrated threshold, forming a filtered subtree for target verification.The mechanism concentrates target-side computation on branches more consistent with target greedy behavior.Source: §3 · p. 6
- The target model verifies the filtered tree in one forward pass and accepts the longest greedy-consistent prefix, preserving output identity with greedy autoregressive decoding.If filtering leaves no tree, the procedure falls back to one-step greedy decoding.Source: §3 · pp. 4–6
4 Experimental Setup
The experiments evaluate ValuSpec on generation, reasoning, and instruction-following benchmarks, using separate calibration and evaluation data. They compare retrieval and dynamic-tree suppliers under output-identical greedy decoding and report throughput, speedup, and verifier-side metrics.
Benchmarks
- HumanEval, GSM8K, and Dolly evaluate code generation, arithmetic reasoning, and instruction following; SpecBench is added for EAGLE-3 integration.Experiments use greedy decoding, with batch size 1 as the default and additional batch sizes 4 and 8.Source: §4 · p. 6
Recurring-workload protocol
- A non-overlapping 70% calibration and 30% evaluation split estimates the threshold and measures throughput, while evaluation contexts are disjoint from those used to build candidate trees.The protocol represents recurring workloads in which calibration requests and future evaluation requests share local continuation patterns.Source: §4 · p. 6
Cross-dataset transfer
- Cross-dataset experiments calibrate the threshold on one dataset and apply it to another without recalibration to test generalization beyond the calibration workload.The evaluated models include Llama-2 targets with a lightweight LLaMA valuation model, plus Llama-3.1-8B-Instruct with EAGLE-3 for dynamic-tree experiments.Source: §4 · p. 7
Candidate tree construction
- The evaluation compares retrieval-based trees under a 64-token budget with EAGLE-3 dynamically constructed trees at an approximately 256-token effective budget, reporting throughput, speedup over AR, acceptance rate, and filtered tree size.Methods are compared under output-identical constraints, with main experiments on one NVIDIA A100 GPU and additional RTX 4090 results in Appendix C.Source: §4 · p. 7
5 Experimental Results
ValuSpec filters already-constructed candidate trees before target verification, leaving upstream generation procedures unchanged. Across recurring workloads, cross-dataset transfer, and EAGLE-3 integration, it improves throughput; analysis also shows substantial residual redundancy in supplied trees.
5.1 Main Results: Recurring-Workload Evaluation
- 2.18×–2.31× speedup over AR on 7B targets and 2.24×–2.38× on 13B targets; ValuSpec has the highest throughput across recurring workloads while preserving exact output equivalence.Retrieval SD constructs the tree first; ValuSpec then filters low-value branches before verification. Table 1 reports throughput in tokens/s with speedup over AR in parentheses.Source: §5.1 · p. 7, Table 1 · p. 8
5.2 Cross-Dataset Threshold Transfer
- 2.05×–2.29× speedup on 7B targets and 2.07×–2.35× on 13B targets across transfer directions, with thresholds applied to another dataset without re-calibration.The threshold is calibrated on one dataset’s 70% split; the upstream candidate-supply procedure remains unchanged.Source: §5.2 · pp. 8–9, Table 2 · p. 8
5.3 Integration with Dynamically Constructed Trees
- 1.35×–1.52× additional speedup over EAGLE-3 and up to 2.78× over AR, with ValuSpec applied after tree construction and without changing EAGLE-3’s generation procedure.Table 3 reports throughput on Meta-Llama/Llama-3.1-8B-Instruct.Source: §5.3 · p. 9, Table 3 · p. 9
- 1.38×–2.14× additional speedup across Standard SD, Retrieval-SD, and EAGLE-3 on Qwen3.5, with gains persisting under cross-dataset transfer and batched inference.For every supplier, ValuSpec exceeds the best fixed likelihood cutoff in [0.3, 0.8]; Table 4 covers HumanEval, GSM8K, and Dolly.Source: §5.3 · pp. 9–10, Table 4 · p. 9
5.4 Acceptance, Filtered Tree Size, and Verification Efficiency
- After ValuSpec, retained EAGLE-3 candidates exceed 86% acceptance, versus 48.2%–71.4% without it; Qwen3.5 analysis retains approximately 73%–88% of branches with empty-tree rates below 1.4% and false-discard rates below 2%.These analyses fix upstream candidates before ValuSpec; output identity comes from greedy verification and autoregressive fallback, not the filtering decision itself.Source: §5.4 · p. 10
6 Discussion and Limitations
ValuSpec’s benefit depends on residual verifier-side redundancy in the upstream tree, and its current study is limited to greedy output-identical verification.
6 Discussion and Limitations
- ValuSpec’s gains depend on how much redundant verification remains after candidate generation; highly selective upstream trees may leave less opportunity.The amount of residual redundancy varies across candidate suppliers, model families, and workloads.Source: §6 · p. 10
- The study focuses on greedy decoding with output-identical verification; extending candidate valuation to stochastic decoding remains future work.Source: §6 · p. 10
- ValuSpec adds a lightweight scoring step before verification, while deeper production-serving integration and profiling remain future work.The reported benefits persist in batch-size experiments.Source: §6 · p. 10
7 Conclusion
ValuSpec values branches in an already materialized candidate tree using a threshold calibrated from target-model greedy trajectories. Experiments report efficiency gains across candidate suppliers while preserving exact output equivalence under deterministic decoding.
7 Conclusion
- ValuSpec operates after candidate generation and before target verification, deciding which supplied branches are worth forwarding rather than altering upstream tree construction.Source: §7 · p. 10
- A threshold calibrated from target-model greedy trajectories provides training-free, plug-and-play valuation and removes low-value branches while preserving exact output equivalence under deterministic decoding.Source: §7 · p. 10
- 1.38×–2.14× speedup over Standard SD, Retrieval-SD, and EAGLE-3 on Qwen3.5, alongside reported effectiveness under cross-dataset transfer and batched inference.The conclusion presents verifier-side valuation as complementary to generation-side candidate supply and tree optimization.Source: §7 · p. 10
A Sensitivity Analysis
Draft length affects acceptance and speedup differently: acceptance falls as n grows, while speedup can peak at an intermediate length. ValuSpec’s best tested setting is n=4 across the reported workloads and hardware/model configurations.
A.1 Draft Length Sensitivity
- n=4 is ValuSpec’s best setting on Dolly across A100/7B, A100/13B, and RTX 4090/7B, and is also optimal on HumanEval and GSM8K.These ValuSpec evaluations use fixed tree hyperparameters with 64 draft tokens.Source: §A.1 · p. 13
- For SD on Dolly, acceptance rate decreases monotonically as draft length n increases, while speedup peaks at n=3.The reported pattern reflects a trade-off between verification gain and verification overhead.Source: §A.1 · p. 13
- RTX 4090 shows smaller relative SD gains than A100, as its higher autoregressive throughput makes SD overheads harder to amortize.Source: §A.1 · p. 13
A.2 Quantile Setting and Overhead
ValuSpec uses a conservative quantile setting for output-identical greedy decoding and reuses an offline-calibrated threshold to limit deployment overhead. The algorithm separates calibration from online tree filtering and verification.
A.2 Quantile Setting and Overhead
- δ = 0.05 is fixed in the main experiments to avoid overly aggressive filtering while preserving the tree structure needed for output-identical verification.The quantile controls filtering conservativeness.Source: §A.2 · p. 13
- Threshold τtr is calibrated offline and reused across future requests, including related workloads, without recalibration.The main deployment cost is lightweight branch filtering rather than repeated threshold estimation; the valuation model is substantially smaller than the target model.Source: §A.2 · p. 13
B Full Algorithm of ValuSpec
- Algorithm 1 separates offline calibration on Dcal from online candidate-tree filtering and output-identical verification on trees constructed from Deval.The pseudocode gives the complete decoding procedure.Source: B Full Algorithm of ValuSpec · p. 14
C Cross-hardware evaluation on RTX 4090.
On RTX 4090, ValuSpec achieves the highest throughput across the reported databases and outperforms SD and Retrieval SD. The results extend its reported gains to consumer hardware with different compute–memory trade-offs.
C Cross-hardware evaluation on RTX 4090.
- 1.67×–1.84× speedup over AR on RTX 4090, with ValuSpec achieving the highest throughput across the reported databases and outperforming SD and Retrieval SD.The comparison covers GSM8K, HumanEval, and Dolly using AR, SD (n = 3), and Retrieval SD (draft tokens = 64).Source: C Cross-hardware evaluation on RTX 4090 · pp. 14–15
- RTX 4090 evaluation tests whether ValuSpec’s gains extend beyond the primary A100 hardware setting.The paper describes the RTX 4090 as a consumer GPU with different compute–memory trade-offs.Source: C Cross-hardware evaluation on RTX 4090 · p. 14
- 1.84× vs. 1.45× speedup on HumanEval for ValuSpec and Retrieval SD, respectively.The paper uses this comparison to note that retrieving more candidate branches alone is insufficient.Source: C Cross-hardware evaluation on RTX 4090 · p. 15
NeurIPS Paper Checklist
The checklist reports the paper’s experimental reproducibility details alongside limitations in statistical reporting, compute accounting, and code availability. It also summarizes the authors’ stated scope boundaries for ValuSpec.
NeurIPS Paper Checklist
- The authors identify smaller potential gains when upstream candidate trees are highly selective or workloads contain little reusable structure.They also note dependence on residual verifier-side redundancy, an additional scoring step, and a need for deeper production-system integration and profiling.Source: NeurIPS Paper Checklist · p. 16
- The checklist says the work proposes an inference-time algorithmic mechanism and does not claim formal theoretical results.It also reports no crowdsourcing or human-subject experiments.Source: NeurIPS Paper Checklist · pp. 16, 18
- The paper specifies datasets, model configurations, calibration/evaluation protocol, threshold procedure, candidate-tree budgets, metrics, hardware, and the full ValuSpec algorithm.The checklist says these details provide a reproducible description of the method and main experimental settings.Source: NeurIPS Paper Checklist · p. 16
- The ValuSpec implementation code has not yet been publicly released, although the datasets, pretrained models, and benchmark resources are publicly available.The checklist also notes that the manuscript does not enumerate license names and terms for every existing asset.Source: NeurIPS Paper Checklist · pp. 16–17
- The paper reports throughput and speedup under fixed configurations but provides no error bars, confidence intervals, or formal significance tests.It also does not report per-experiment runtime or total GPU compute used throughout the project.Source: NeurIPS Paper Checklist · p. 17
- The paper says it does not include a separate societal-impact analysis and describes broader effects as indirect and inherited from deployed models and applications.It states that the work changes computational efficiency rather than underlying model capabilities or generated content.Source: NeurIPS Paper Checklist · p. 17