PaperlayerView exact revision on arXiv

Source-linked AI summary

SpecBlock: Block-Iterative Speculative Decoding with Dynamic Tree Drafting

Weijie Shi, Qiang Xu, Fan Deng, Yaguang Wu, Jiarun Liu, Yehong Xu, Hao Chen, Jia Zhu, Jiajie Xu, Xiangjun Huang, Jian Yang, Xiaofang Zhou

arXiv:2605.07243v2cs.CL

TL;DR

Speculative decoding must balance path coherence against drafting cost, since autoregressive drafters are expensive while parallel drafters produce incoherent candidate paths. SpecBlock uses block-iterative dependent drafting with adaptive tree branching and cost-aware serving updates, improving mean speedup over EAGLE-3 by 8–13% and extending the gain to 11–19%.

  • Problem

    Speculative decoding lacks a drafter that combines few sequential calls with coherent dependence along each draft path.

  • Method

    SpecBlock generates dependent multi-token blocks, expands branches iteratively from cached hidden states, and allocates branching with a co-trained rank head.

  • Results

    Mean speedup improves over EAGLE-3 by 8–13% across three target models, while cost-aware adaptation extends the gain to 11–19%.

  • Takeaways & Limitations

    SpecBlock offers a lower-cost alternative to EAGLE-3 while retaining path-dependent drafting, and selective adaptation further improves serving speedup.

  • Takeaways & Limitations

    The rank head is imperfect, sometimes misallocating sibling branches, so finer-grained or more accurate predictions could improve verifier-budget allocation.

Abstract

from arXiv · show

Speculative decoding accelerates LLM inference by drafting a tree of candidate continuations and verifying it in one target forward. Existing drafters fall into two camps with opposite weaknesses. Autoregressive drafters such as EAGLE-3 preserve dependence along each draft path but call the drafter once per tree depth, making drafting a non-trivial share of per-iteration latency. Parallel drafters cut drafter calls by predicting multiple future positions in one forward, but each position is predicted without seeing the others, producing paths the verifier rejects. In this paper, we propose SpecBlock, a block-iterative drafter that combines path dependence with cheap drafting. Each drafter forward produces K dependent positions and we call this a block. The draft tree grows through repeated block expansions. Two mechanisms explicitly carry path dependence to keep later draft positions accurate. Within each block, a layer-wise shift carries the previous position's hidden state into every decoder layer. Across blocks, each new block can start from any position of the previous block, inheriting its hidden state to extend the path. To spend verifier budget where acceptance is likely, a co-trained rank head replaces the fixed top-k tree by allocating per-position branching during drafting. To avoid training the drafter on prefixes it never produces at inference, a valid-prefix mask drops the loss at later positions once an earlier one is wrong. Beyond static drafting, a cost-aware bandit at deployment uses free verifier feedback to update the drafter selectively, only when the expected throughput gain exceeds the update cost. Experiments show that SpecBlock improves mean speedup by 8-13% over EAGLE-3 at 44-52% of its drafting cost, and cost-aware adaptation extends this lead to 11-19%.

1 Introduction

SpecBlock is a block-iterative speculative drafter that preserves path dependence while reducing sequential drafting overhead through dependent multi-token blocks and batched branch extensions. Rank-guided branching, valid-prefix training, and cost-aware adaptation target acceptance quality, training validity, and deployment efficiency.

  • Drafting paradigms: 30% of per-iteration latency is consumed by autoregressive drafting on 8B-level targets, versus roughly 7% for parallel drafters.Autoregressive drafters preserve path dependence but require sequential rounds; parallel drafters reduce calls while predicting positions independently.
  • SpecBlock: SpecBlock produces K dependent positions per forward and repeatedly expands blocks from prior tree nodes, extending multiple branches in parallel instead of adding one depth per call.This design explicitly carries dependence within and across blocks.
  • Rank-guided tree construction: A co-trained rank head predicts each position’s target-rank bucket, which sets sibling branching and allocates alternatives unevenly across the draft tree.The mechanism gives more verifier budget to positions where the target token ranks lower in the draft distribution.
  • Valid-prefix curriculum learning: Valid-prefix curriculum learning drops later-position supervision after an earlier prediction is wrong, matching training contexts to the jointly generated prefixes used at inference.Later predictions consume earlier model predictions rather than fresh ground-truth prefixes, and a wrong earlier prediction causes the verifier to reject the path.
  • Results: 8–13% mean speedup over EAGLE-3 is achieved at 44–52% of its drafting cost, increasing to 11–19% with cost-aware serving-time adaptation.The adaptation bandit uses free verifier feedback and updates selectively when expected throughput gain exceeds update cost.

2 Related Work

Related work contrasts autoregressive drafters, which preserve token dependence at higher drafting cost, with parallel or blockwise drafters, which reduce calls but lose adjacent-token dependence. Prior work also externalizes tree construction or adapts serving-time behavior, whereas SpecBlock integrates branching into drafting and supports verifier-feedback adaptation.

  • Autoregressive drafters: Autoregressive drafters preserve dependence by proposing token chains that target models verify in parallel, while token trees such as SpecInfer accommodate uncertainty through multiple candidate paths.The EAGLE family trades drafter capacity for fidelity by autoregressing in the target model’s representation space.
  • Parallel and blockwise drafters: Parallel and blockwise drafters use one forward to predict several future positions, but independent predictions lose adjacent-token dependence and diverge from target continuations after early depths.Draft-head methods attach independent heads at fixed offsets, exemplifying this trade-off.
  • Tree construction: Prior methods shape draft trees externally using offline optimization or drafter signals, whereas SpecBlock uses a rank head to set per-position branching and block starts.Sequoia uses offline dynamic programming, while C2T, OPT-Tree, DySpec, and TALON adapt trees from probability, confidence, or budget signals.
  • Serving-time adaptation: Serving-time adaptation either tunes speculation hyperparameters while freezing drafter weights or updates the drafter using verifier feedback to address distribution shifts that reduce acceptance length.Examples include bandits over proposal configurations, learned acceptance-probability thresholds, and verifier-feedback distillation.

3 SpecBlock

SpecBlock is a block-iterative speculative drafter that predicts K dependent positions per forward, then extends trees by reusing the drafter from selected earlier positions. It preserves path dependence with within-block state shifts, adapts branching through a rank head, and trains with inference-consistent valid-prefix masking.

  • Block-iterative drafting: Each drafter forward predicts K dependent positions in parallel, while repeated forwards grow the tree beyond depth K from selected earlier positions.This places SpecBlock between autoregressive drafting, which preserves dependence but incurs depth-proportional cost, and parallel drafting, which reduces calls but loses dependence.
  • Within-block dependence: A layer-wise shift explicitly carries position k−1’s state into position k, recovering dependence lost to diluted cross-position attention within one forward.Cross-position causal attention also restricts each position to earlier positions within the block and cached preceding blocks.
  • Bucket-driven branching: The co-trained rank head assigns positions to four branching buckets, attaching bucket-specific top-b siblings so verifier budget follows predicted rank.It uses detached hidden-state and draft-distribution features, shaping the rank head without changing token prediction.
  • Cross-block iteration: Cross-block iteration batches scheduled starting positions and uses their cached drafter states to produce K further positions, reaching depth M·K with M drafter forwards.Because target hidden states are unavailable before verification, later blocks bypass the target-conditioning projection and use self-produced drafter features.
  • Valid-prefix training: The valid-prefix mask trains draft and rank losses only on paths that remain consistent after earlier predictions, matching the drafter’s inference-time prefix distribution.Rank-head labels are derived from the target token’s rank in the draft distribution and are masked by the same valid-prefix mask.

4 Experiments

Experiments across three target models and six benchmarks show that SpecBlock improves speedup over EAGLE-3 while substantially reducing drafting cost. Ablations and deployment results attribute these gains to block drafting, path-dependent mechanisms, and selective cost-aware adaptation.

  • Evaluation setup: SpecBlock is evaluated on Llama-3.1-8B-Instruct, Qwen3-8B, and Qwen3-32B across six benchmarks spanning conversation, code, mathematics, instruction following, question answering, and translation.Experiments use an NVIDIA A100-80GB GPU at batch size 1, with temperature 0 and 1.0.
  • Main results: 8–13%: SpecBlock improves mean speedup over EAGLE-3 across all six target-model configurations.Cost-aware adaptation adds 2–4% over always-update SpecBlock+OSD on the four benchmarks where the bandit engages.
  • Drafting efficiency: 7 ms: SpecBlock reduces drafter time from 17 ms using two block forwards, while retaining τ=4.41 on Llama-3.1-8B.EAGLE-3 reaches τ=5.89 through seven sequential calls and incurs a 31% drafter share; SpecBlock is only 1.48 tokens lower.
  • Scaling and sampling: 46%: On Qwen3-32B, SpecBlock’s drafting cost is 46% of EAGLE-3’s, compared with 52% on Llama-3.1-8B.The drafter-share comparison is 11% versus 24% on Qwen3-32B and 16% versus 31% on Llama-3.1-8B; sampling at T=1.0 lowers τ while preserving ordering.
  • Ablations: 0.10×: Removing the cost-aware bandit reduces mean speedup while leaving τ at 4.54, because always-update pays a backward on every query.Removing the head-only action reduces mean speedup by 0.24×, raises τ slightly to 4.58, and incurs an order-of-magnitude longer backward.

5 Conclusion · A Implementation Details

SpecBlock combines block-iterative dependent drafting, hidden-state mechanisms, adaptive branching, and valid-prefix training with cost-aware deployment updates. It improves mean speedup over EAGLE-3 by 8–13%, rising to 11–19% with cost-aware adaptation.

  • 5 Conclusion: SpecBlock produces K dependent positions per forward and extends draft paths through repeated block expansions from earlier hidden states.This block-iterative design combines dependent drafting with repeated path extension.
  • A Implementation Details: A layer-wise shift carries each previous position’s hidden state into every decoder layer to preserve within-block dependence.The mechanism explicitly transfers prior-position information throughout the decoder.
  • A Implementation Details: A co-trained rank head sets per-position branching to allocate verifier budget where the drafter is uncertain.Branching is dynamically allocated rather than fixed across positions.
  • A Implementation Details: The valid-prefix curriculum masks the loss after an earlier draft position is wrong, aligning training with produced prefixes.This prevents later-position supervision from relying on invalid prefixes.
  • 5 Conclusion: The deployment bandit selectively updates the drafter when expected throughput gain exceeds update cost, using free verifier feedback.Adaptation is cost-aware rather than applied indiscriminately.
  • 5 Conclusion: 8–13%: SpecBlock improves mean speedup over EAGLE-3 across three target models.This is the reported static-drafting improvement.
  • 5 Conclusion: 11–19%: cost-aware adaptation extends SpecBlock’s mean-speedup gain over EAGLE-3.The larger gain follows from selective deployment-time updates.

A.1 Drafter setup and training configuration · A.2 Attention Mask for SpecBlock · A.3 Cost-aware adaptation

The appendix specifies SpecBlock’s drafter configuration and attention pattern, then details cost-aware adaptation’s bandit lifecycle, scheduling, synchronization, and drift control. Together, these procedures define how drafting is trained, masked, adapted, and deployed across single- and dual-GPU settings.

  • A.1 Drafter setup and training configuration: The drafter matches the target’s decoder design and hidden dimension, trains its decoder and LM head from scratch, and freezes target-initialized token embeddings.It uses a reduced vocabulary of 32,000 frequent tokens covering 98.7% of training tokens.
  • A.1 Drafter setup and training configuration: Training uses AdamW defaults with batch size 96 and a cosine schedule with linear warmup over the first 1.5% of updates.Each drafter requires approximately 3,000 A100-80GB GPU-hours; inference defaults to one A100-80GB GPU, with a dual-GPU adaptation variant.
  • A.2 Attention Mask for SpecBlock: Each block forward attends to the verified prefix, preceding current-iteration blocks along the branch path, and its own block-causal context.The verified prefix supplies K keys and values from previously committed positions, while cross-block visibility follows the selected branch.
  • A.3 Cost-aware adaptation: The bandit begins with a 10-query warmup, followed by 8 cold-start events using ε-greedy exploration as ε decays from 0.30 to 0.10.After each training event, the next 2 queries cannot trigger updates, and EWMA reward updates are skipped when strig < 5.
  • A.2 Attention Mask for SpecBlock: In the illustrated K=4 case, block 2 branches from B1,1 and attends to on-path positions B1,0 and B1,1 while masking off-path positions B1,2 and B1,3.Prefix tokens attend causally to themselves, and each block sees the verified prefix plus its own block-causal positions.
  • A.3 Cost-aware adaptation: Verifier signals route updates to head or full training buffers only when those buffers saturate, with backward passes and weight synchronization scheduled around target verification.The schedule supports single-GPU separate-stream execution and dual-GPU deployment with periodic in-place weight copies.
  • A.3 Cost-aware adaptation: Per-query control selects skip when both predicted throughput gains are nonpositive, otherwise choosing the action with the largest predicted gain before buffering rejected positions and target distributions.The trigger records the signal and action, then applies a KL-regularized training objective before syncing weights during the next target-verify window.
  • A.3 Cost-aware adaptation: The adaptation loop opens an N-query measurement interval after synchronization and reverts to the last good checkpoint if accepted length decreases monotonically across 3 windows.EWMA updates are gated by the trigger threshold strig ≥ smin.

A.4 Baselines and benchmarks · A.5 Inference procedure

The evaluation compares SpecBlock with six drafting baselines across benchmarks ranging from 80 to 4,000 prompts, then verifies up to M·K drafted positions in one parallel forward per iteration. The longest accepted prefix is committed before drafting resumes.

  • A.4 Baselines and benchmarks: Six drafting baselines are used for comparison.The passage identifies the baseline set but does not enumerate all six in the supplied text.
  • A.4 Baselines and benchmarks: Standard speculative sampling autoregressively samples future tokens from a smaller drafter for one target verification forward.Llama-3.1-8B is paired with Llama-3.2-1B, while Qwen3 targets are paired with Qwen3-0.6B.
  • A.4 Baselines and benchmarks: Medusa uses K independent fixed-offset decoding heads on the target’s last hidden state without cross-position attention or layer-wise dependence.This distinguishes Medusa from drafters that explicitly model dependence between future positions.
  • A.4 Baselines and benchmarks: 80 multi-turn dialogues in MT-Bench and 164 code prompts in HumanEval form the smallest benchmark sets.The benchmark suite also includes larger sets, but these two provide the lower prompt-count range stated in the passage.
  • A.4 Baselines and benchmarks: 4,000 instruction prompts in Alpaca and 3,610 open-domain questions in Natural Questions form the largest benchmark sets.The suite also contains 500 MATH-500 competition problems and 549 WMT-23 translation pairs.
  • A.5 Inference procedure: SpecBlock invokes the drafter at most M times to grow a tree up to M·K depth, while the verifier scores all candidates in one parallel forward.The first block conditions on target multi-layer features; later blocks use the drafter’s cached last-layer state and bypass Wcond.
  • A.5 Inference procedure: The longest accepted prefix is committed before the next verifier iteration begins.This is the iteration-level commitment rule described in Algorithm 2.

B Rank head

The rank head uses a 15-dimensional summary of each draft distribution alongside the hidden state to classify positions into four buckets for verifier-budget allocation. Its classification is strongest on the extreme buckets, including the dominant rank-1 bucket and the high-rank give-up bucket.

  • Distribution summary features: The rank head reads a 15-dimensional summary ψ(p_t,k) of the draft distribution alongside the hidden state.The summary includes ten dimensions describing the log-probability profile of the top-10 tokens and three dimensions capturing distribution shape signals.
  • Classification quality: The rank head is supervised as a four-way classifier whose predictions determine how verifier budget is allocated.Evaluation covers approximately 72,000 validation positions with valid-prefix mask equal to one, using the target token’s rank within p_t to define ground-truth buckets.
  • Classification quality: 75.2% of positions belong to b0, while b2 and b3 each account for under 6%.The class frequencies are sharply imbalanced.
  • Classification quality: 0.978 precision is achieved by b0, while b3 reaches 0.822 F1.The two extreme buckets are the easiest to classify; b0 corresponds to target rank 1, whereas b3 represents high-rank positions.

C Per-position acceptance rate · D Adaptation deployment

SpecBlock’s acceptance rates decline smoothly with chain position, while block boundaries improve recovery by restarting from cached hidden states. Adaptation deployment is evaluated across single- and dual-GPU settings using speedup and accepted-length metrics across three target models.

  • C Per-position acceptance rate: αk measures the conditional probability that the drafter’s greedy token at chain position k matches the target, across K·M=8 positions and two cross-block iterations.The chain position is not tied to a fixed within-block index because the block boundary depends on how many positions are taken before the next block starts.
  • C Per-position acceptance rate: α1 exceeds 0.80 on both targets, while α8 falls to 0.369 for Qwen3-8B and 0.544 for Llama-3.1-8B.Qwen3-8B exhibits the sharper decline across positions.
  • C Per-position acceptance rate: Within each block, αm,j decreases monotonically across its four positions under the condition that earlier positions in that block match.This diagnostic measures per-block, per-position acceptance rather than chain position alone.
  • C Per-position acceptance rate: Position 1 of block 1 reaches 0.784 on Llama-3.1-8B and 0.748 on Qwen3-8B, exceeding block 0’s final-position rates of 0.465 and 0.407.Block 1 starts from the drafter’s cached state rather than the target’s state.
  • D Adaptation deployment: Table 4 evaluates SpecBlock+adapt under single-GPU and dual-GPU deployment across three target models, reporting speedup over vanilla decoding and average accepted length per verifier call.Subscripts show the absolute gain over SpecBlock without adaptation.
  • C Per-position acceptance rate: The block boundary acts as a recovery mechanism, supporting block-iterative drafting over a single longer block of length K·M.The recovery effect persists even though block 1 begins from the drafter’s own cached hidden state.

D.1 Single-GPU and dual-GPU deployment

Cost-aware adaptation supports both single-GPU and dual-GPU deployment, with comparable gains across benchmarks and target models. Single-GPU avoids cross-device synchronization, while dual-GPU remains an option when an extra device is available.

  • Deployment regimes: Cost-aware adaptation supports single-GPU training on a separate CUDA stream and dual-GPU training with periodic in-place synchronization between drafter copies.The single-GPU regime shares the inference device, whereas the dual-GPU regime uses a second device for the training stream.
  • Comparative results: Both deployment regimes deliver comparable gains across all benchmarks and three target models, with neither consistently dominating.Single-GPU matches or exceeds dual-GPU on Spd for MATH-500, Alpaca, and NQ at both temperatures, while dual-GPU wins on WMT-23 at T=0.
  • Comparative results: The τ gap remains within 0.05 on most benchmarks, and both regimes recover most of NQ’s out-of-distribution acceptance loss.These results indicate that the deployment choice does not materially change accepted length on most benchmarks.
  • Deployment choice: Single-GPU deployment is viable because it avoids cross-device weight-transfer latency, while dual-GPU deployment is an option when an extra training device is available.The dual-GPU regime additionally incurs synchronization latency, making sync frequency a deployment consideration.

D.2 Mixed-task adaptation · E Case Study · PROMPT

The paper evaluates SpecBlock under heterogeneous repeated traffic and contrasts its tree construction with EAGLE-3 on a shared prompt. Adaptation improves acceptance and speedup, while SpecBlock reaches comparable responses with fewer drafter forwards.

  • D.2 Mixed-task adaptation: The mixed stream samples equal proportions from MATH-500, WMT-23, Alpaca, and NQ across 2K queries.The evaluation targets heterogeneous, repeated production-like traffic by sweeping full adaptation passes.
  • D.2 Mixed-task adaptation: 0.22–0.30: After N=8 adaptation epochs, τ rises over the no-adapt baseline on both target models and deployment regimes.Both Spd and τ increase monotonically with the number of adaptation epochs.
  • D.2 Mixed-task adaptation: 3.9–10.0%: After N=8 adaptation epochs, Spd improves over the no-adapt baseline, with larger gains on Qwen3-8B.Dual-GPU remains ahead of single-GPU, but the gap stays below 0.05 in τ and below 1% in Spd.
  • E Case Study: On the same Qwen3-8B prompt under greedy decoding, SpecBlock and EAGLE-3 commit nearly the same response content.The case study marks drafted-and-accepted tokens in green and target-sampled bonus tokens in red across the first 30 committed tokens.
  • E Case Study: 7: EAGLE-3 requires seven sequential drafter forwards to reach depth 7, while SpecBlock uses two forwards for the same prompt.SpecBlock’s first block emits K=4 chain positions in one forward, and its second block batches additional chains from rank-head–selected starts.
  • PROMPT: The illustrative prompt asks the model to write a Python function computing the Fibonacci sequence.The prompt is the shared task used for the case-study response and tree visualization.
  • E Case Study: SpecBlock trades a slightly shorter accepted run per iteration for an iteration that finishes in two drafter calls instead of seven.This summarizes the visible structural trade-off in the case study tree comparison.

F Limitations

SpecBlock’s performance depends on imperfect rank-head predictions and workload-sensitive drafting choices. Its block width is fixed during training, while block depth can extend at inference, but deployment settings may not suit every target-cost or acceptance distribution.

  • Rank-head accuracy: The verifier tree’s shape depends on rank-head accuracy, and the four-bucket classifier is not perfectly accurate.It nevertheless outperforms a uniform tree with the same node budget in the reported ablation.
  • Inference flexibility: Block width K=4 is fixed at training because layer-wise shifting is built around a specific K, whereas block count M can extend naturally at inference.A drafter trained with M=3 continues to work at M=4; deployments use M=2 without searching per workload.
  • Workload sensitivity: Larger targets may favor larger K or deeper block stacking, while closer drafter-target costs favor smaller values and different acceptance distributions may require untested settings.The preferred configuration depends on the relative verifier and drafter costs and on how closely workloads match the training mix.
Loading 2605.07243v2…