PaperlayerView exact revision on arXiv

Source-linked AI summary

Nucleus Speculative Decoding: Plausibility-Aware Verification Beyond Exact Distribution

Shuhao Li, Fanghua Ye, Wanyu Lin, Tianyu Yuan, Xiaoyu Shen

arXiv:2610.07822v1cs.CL

TL;DR

Speculative decoding accelerates autoregressive generation, but standard verification can reject target-plausible proposals and limits retained draft tokens. NSD adds target-nucleus acceptance and characterizes its distributional deviation; across five benchmarks, it reports throughput improvements of up to 416% over AR and 215% over SD.

  • Problem

    Standard speculative verification can reject proposals that remain highly ranked or lie in the target's high-probability region, limiting the draft tokens retained after a verification pass.

    Source: §1 · pp. 1–2

  • Method

    NSD accepts a draft token if it passes standard speculative verification or belongs to the target model's nucleus.

    Source: §1 · p. 2

  • Results

    Across five benchmarks with Nemotron and Qwen backbones, NSD improves throughput by up to 416% over AR and 215% over SD.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results suggest plausibility-aware verification is a promising direction for efficient speculative decoding while retaining additional draft tokens and maintaining competitive task performance.

    Source: §5 · p. 11

    Applying the sequence-level bound to a speculative backend requires establishing its conditional-kernel assumption across proposal selection, recovery, bonus sampling, and decoder state.

    Source: §3 · p. 5

Abstract

from arXiv · show

Speculative decoding accelerates autoregressive generation by using a lightweight draft model to propose multiple tokens that are verified by a target model in parallel. However, the standard acceptance rule focuses on exact distribution correction and rejects tokens that remain highly plausible under the target model when the draft model assigns excess probability. This conservative verification limits the number of draft tokens retained after each verification forward pass. We introduce Nucleus Speculative Decoding (NSD), a relaxed verification method that incorporates target-model plausibility into speculative decoding. NSD accepts a draft token if it satisfies the standard acceptance rule or belongs to the target model's nucleus. We theoretically characterize the distributional deviation introduced by our method and show that the single-step error is exactly determined by the draft model's excess probability within the target nucleus. We further derive sequence-level fidelity bounds that quantify how local deviations accumulate over autoregressive decoding. Experiments across multiple target models and proposal mechanisms demonstrate that NSD consistently improves speculative decoding efficiency while maintaining competitive task performance. Our method achieves throughput speedups of up to $5.16\times$ over autoregressive decoding and up to $3.15\times$ over standard speculative decoding. These improvements coincide with longer accepted lengths, allowing more output tokens to share the cost of each target verification pass. Analysis shows that plausibility-aware verification provides an effective approach for relaxed verification and speculative decoding efficiency. Our code is available at https://github.com/EIT-NLP/Nucleus-Speculative-Decoding.

1 Introduction

Speculative decoding reduces autoregressive generation cost, but target verification remains expensive and can reject draft tokens that are still plausible under the target model. NSD adds target-nucleus membership as an acceptance condition and characterizes its distributional deviation.

1 Introduction

  • Target verification consumes 57.1%–78.1% of speculative decoding wall-clock time, and verification stops at the first rejected proposal.All later proposals in that block are discarded after an early rejection.Source: §1 · p. 1
  • At Top-P=0.95, 52.6% of rejected tokens remain in the target nucleus, while 81.5% rank within the target's top ten.These results show that rejection under the standard rule does not necessarily mean a proposal is implausible under the target model.Source: §1 · p. 2
  • NSD accepts a draft token if it passes standard speculative verification or belongs to the target nucleus.The nucleus is the highest-probability token set whose cumulative probability reaches a threshold [16].Source: §1 · p. 2
  • Across five benchmarks with Nemotron and Qwen backbones, NSD improves throughput by up to 416% over AR and 215% over SD.Its single-step distributional deviation is exactly determined by draft excess probability within the target nucleus, and the paper establishes sequence-level fidelity guarantees.Source: §1 · p. 2

2 Related Work

Prior speculative-decoding work improves drafting or relaxes verification in several ways. NSD instead adds target-nucleus membership to verification while retaining existing residual correction and bonus sampling.

Speculative Decoding

  • Speculative decoding uses a lightweight draft model to propose tokens for parallel target verification, accepting with min{1, p(x)/q(x)} and using residual correction on rejection [4] [17].SpecInfer uses token trees [21], while Draft & Verify skips target-model layers [28].Source: §2 · pp. 2–3
  • Recent draft-model advances improve proposal quality or efficiency, whereas NSD modifies verification and is complementary to those drafting mechanisms.The cited approaches include feature-based drafters [18] [19] [20], multi-token prediction [9] [12], parallel diffusion drafting [5], semi-autoregressive proposals [7], and unified autoregressive diffusion models [11].Source: §2 · p. 3

Relaxed Verification

  • Relaxed-verification methods use likelihood, probability, ranking, adaptive, or learned criteria [2] [3] [10] [13] [15] [17] [23] [26], while NSD adds target-nucleus acceptance [16].NSD retains the original residual correction and bonus sampling mechanisms, using target probabilities already computed during verification.Source: §2 · p. 3

3 Nucleus Speculative Decoding

NSD augments speculative decoding with target-nucleus membership as an additional acceptance condition. Its analysis relates extra acceptance to local distributional deviation and gives conditional sequence-level fidelity guarantees.

3 Nucleus Speculative Decoding

  • NSD accepts a draft proposal through standard speculative verification or target-nucleus membership, using target probabilities already computed during verification.The method leaves the proposal mechanism, residual correction, and bonus sampling unchanged.Source: §3 · p. 3
  • Standard speculative decoding verifies draft tokens in parallel, stops at the first rejection, samples a replacement from the residual distribution, and adds a bonus token if all proposals pass.At each position its acceptance probability is min{1, p_i(x_i)/q_i(x_i)}.Source: §3 · p. 4
  • For each verification position, NSD constructs a target-distribution nucleus using threshold θ and accepts a proposal in that set even when standard verification would reject it.NSD does not renormalize the target or draft distributions over the nucleus; the actual conditional proposal distribution includes sampling transformations, and nucleus ties use a fixed ordering.Source: §3 · p. 4
  • The one-cycle algorithm drafts K tokens, evaluates target distributions for K+1 positions in one pass, then emits accepted proposals, a residual replacement, or a bonus token.The output block contains at most K+1 tokens.Source: §3 · p. 4
  • NSD's additional acceptance is exactly the draft's excess probability mass within the target nucleus, ES, so average acceptance rises from 1−R to 1−R+ES.R is the rejection probability of standard speculative decoding; ES is also the probability that NSD accepts a proposal rejected by standard SD under a shared proposal.Source: §3 · p. 5
  • When R>0, NSD residual resampling decreases from R to R−ES because it rejects only draft excess outside the nucleus.The single-step deviation equals the extra acceptance, and ES≤R.Source: §3 · p. 5
  • The sequence-level guarantee assumes each conditional next-emitted-token distribution stays within ϵ_t total variation of the target distribution for every relevant history.The decoder's internal state is marginalized conditional on emitted history.Source: §3 · p. 5
  • Applying the bound to a backend requires establishing that conditional-kernel assumption across proposal selection, recovery, bonus sampling, and decoder state.For illustration, ϵ≤10^-3 and T=32 imply coupling agreement of at least (0.999)^32≈0.9685.Source: §3 · p. 5

4 Experiments

Across the evaluated models and proposal mechanisms, NSD improves throughput over autoregressive and standard speculative decoding, while task-performance changes vary. Ablations show that efficiency and quality depend on draft block length and nucleus threshold, and analysis links savings to retaining additional plausible proposals and reducing verification calls.

4.1 Experimental Setup

  • The setup evaluates five proposal mechanisms across Nemotron and Qwen model sizes on mathematical reasoning and code generation, measuring task quality and throughput.Main experiments use Nemotron linear self-speculation with 32-token blocks and Qwen3 with DSpark and 7-token blocks; metrics include accuracy or pass@1, speedup relative to AR, and mean accepted length τ.Source: §4.1 · pp. 5–6

4.2 Main Results

  • 323.0% over AR and 84.98% over SD are the mean throughput improvements across 30 paired comparisons; NSD reports the highest speedup in every comparison.The evaluations cover Nemotron with linear self-speculation and Qwen3 with DSpark drafting.Source: §4.2 · p. 6, Table 1 · p. 7
  • NSD increases mean accepted length τ across every model and benchmark group, from 6.29, 4.98, and 4.67 to 9.54, 11.82, and 11.54 for Nemotron-3B, -8B, and -14B.For Qwen3-4B, -8B, and -14B, τ increases from 5.36, 5.41, and 5.38 to 6.60, 6.69, and 6.68.Source: §4.2 · p. 6
  • Task quality varies: NSD exceeds SD in 15 of 30 accuracy comparisons, ties one, and falls below SD in 14, with an average difference of +0.40 percentage points.Qwen3-4B is an exception to higher overall accuracy: it drops from 68.65% to 67.66% as overall speedup rises from 3.13× to 4.00×.Source: §4.2 · p. 6

4.3 Ablation Study

  • Increasing draft block length K from 8 to 32 improves speedup and mean accepted length at every tested threshold, but gains can diminish or reverse on MBPP.For Nemotron-14B on MBPP at θ = 0.925, increasing K from 24 to 32 decreases speedup from 3.81× to 3.59× and τ from 9.53 to 8.87; the joint K and θ configurations therefore involve trade-offs.Source: §4.3 · pp. 8–9, Table 2 · p. 8
  • Threshold effects depend on model and task: lowering θ can improve speed and accuracy together on one setting but trade task performance for speed on another.For Nemotron-14B at K = 32, lowering θ from 0.95 to 0.85 improves GSM8K accuracy from 92.87% to 93.86% and MBPP pass@1 from 59.92% to 62.26%, while speedups fall from 5.16× to 4.86× and 3.92× to 3.58×. For Nemotron-8B at K = 32, lowering θ from 0.95 to 0.90 improves GSM8K speedup and accuracy, while MBPP pass@1 decreases from 62.26% to 59.92%.Source: §4.3 · pp. 8–9

4.4 Analysis

  • Among additional acceptances, 86.65% lie in the nucleus core and 94.1% rank within the target’s top five; draft entropy is lower than target entropy in the reported mean comparison.Mean draft and target entropies are 0.865 and 1.101, respectively; 70.30% of these tokens have higher target entropy.Source: §4.4 · pp. 9–10
  • On Nemotron models, NSD reduces verification time from 5.26 to 2.72 milliseconds per output token and verification calls per output token from 0.1600 to 0.0835.Target execution time per call changes by less than 1%, supporting fewer verification calls per emitted token as the primary source of savings.Source: §4.4 · p. 10

5 Conclusion

The paper introduces Nucleus Speculative Decoding (NSD), which adds target-nucleus membership to stochastic acceptance without changing proposal generation, residual correction, or bonus sampling. Its analysis characterizes single-step distributional deviation and conditional sequence-level fidelity, while experiments report higher throughput and competitive task performance.

5 Conclusion

  • NSD supplements stochastic acceptance with target-nucleus membership while leaving proposal generation, residual correction, and bonus sampling unchanged.The method is presented as compatible with diverse speculative-decoding paradigms.Source: §5 · p. 10
  • The single-step deviation from the target distribution equals the draft model’s excess probability inside the target nucleus, with conditional sequence-level fidelity bounds.Source: §5 · p. 10
  • Experiments across Qwen and Nvidia-Nemotron model families and proposal mechanisms report improved throughput while retaining additional draft tokens and maintaining competitive task performance.The paper characterizes plausibility-aware verification as a promising direction for efficient speculative decoding.Source: §5 · pp. 10–11

AI use statement

The authors report using AI-assisted tools for writing-related tasks and retain responsibility for the manuscript’s research and final content.

AI use statement

  • AI-assisted tools supported writing tasks, while the authors remain responsible for the research contributions, results, interpretations, and final manuscript.Source: AI use statement · p. 11

Reproducibility Statement

The appendix describes reproducibility measures and analyzes NSD’s emitted-token distribution under stated proposal and recovery assumptions. It gives exact single-step error and acceptance identities, threshold monotonicity results, and a conditional sequence-level coupling bound.

Reproducibility Statement

  • The paper reports that code and datasets are publicly available and describes its experimental setup, including models, decoding parameters, and hardware.Experiments use NVIDIA Pro5000 GPUs, with latency and throughput measured under the same hardware configuration.Source: Reproducibility Statement · p. 11

A.1 Assumptions and Notation

  • The single-step analysis assumes proposals are sampled from the actual transformed proposal distribution and that rejection recovery uses the specified residual distribution.The result concerns the emitted token, including recovery, rather than conditioning only on acceptance.Source: §A.1 · p. 15

A.2 Single-Step Output Distribution and Exact Error

  • Under the stated assumptions, one-step total variation error equals ES, the excess draft probability mass in the target nucleus, and also equals NSD’s proposal-acceptance gain over standard SD.If R = 0, the emitted law equals p = q and ES = 0.Source: §A.2 · p. 15, It is normalized and satisfies · pp. 15–16

It is normalized and satisfies

  • The emitted distribution is normalized because accepted proposal mass and residual recovery mass sum to one.The proof constructs the accepted mass from the standard SD mass plus nucleus excess, then assigns the remaining rejection probability to the residual distribution.Source: It is normalized and satisfies · p. 15

A.3 Exactness and Threshold Monotonicity

  • NSD is exact if and only if ES = 0, equivalently q(x) ≤ p(x) for every token in the selected set; expanding nested sets raises both acceptance and one-step error.These monotonicity claims hold at fixed p, q, and prefix, and do not imply monotonic accuracy or throughput across independently generated trajectories.Source: §A.3 · p. 16

A.4 Sequence-Level Coupling Bound

  • A sequence-level coupling guarantee requires a uniform bound on the decoder’s next-output kernel relative to the target kernel; the theorem gives complete-sequence agreement probability at least Q_T.For speculative blocks, proposal selection, recovery, bonus sampling, and decoder state must be included in that kernel.Source: §A.4 · pp. 16–17

A.5 Uniform-Error Corollaries and Interpretation

The appendix gives a sufficient output-horizon condition for a coupling agreement bound and states when exact sequence sampling holds. It cautions that these guarantees require justified sequence-level error bounds, not diagnostic averages alone.

A.5 Uniform-Error Corollaries and Interpretation

  • For 0 < ϵ < 1, the product bound certifies agreement of at least 1 − δ whenever T ≤ log(1 − δ) / log(1 − ϵ).This is a sufficient output-horizon bound, not a prescription for choosing speculative block size.Source: §A.5 · p. 17
  • If the decoder’s next-output kernel equals the target kernel at every relevant history and position, the sequence distribution is exact for every finite horizon.The stated NSD case requires complete one-step kernels satisfying the proposal and recovery assumptions with ES = 0 everywhere.Source: §A.5 · p. 17
  • The coupling statement is not a guarantee of unchanged task accuracy, and observed diagnostic averages do not establish the required uniform sequence-level bounds.Applying the bound to a correctness event requires a justified sequence-level bound.Source: §A.5 · p. 17

B Method conventions and implementation correspondence

Implementation correspondence depends on how each backend defines proposal membership and sampling, while the mechanism comparison distinguishes retaining a proposal from generating a token after failure. Additional proposal mechanisms are evaluated under specified settings, but the supplied passages do not give their Table 4 cell values.

B Method conventions and implementation correspondence

  • Cumulative-mass membership includes all equally scored tokens and need not be the smallest sorted set reaching the threshold; the nucleus gate does not truncate replacement or bonus sampling.Under an unrestricted positive-temperature softmax, the ordering agrees with Equation 1.Source: B Method conventions and… · p. 17
  • DSpark's optional confidence gate can condition the proposal law, while Nemotron's reported SD path may use an external routine whose equivalence is not established here.For DSpark, the gate must be disabled or its conditioning accounted for when interpreting qi as the actual proposal law; Nemotron equivalence depends on unavailable external source and historical configuration.Source: B Method conventions and… · p. 18
  • The Qwen3.5-MTP adapter gates on processed target logits, delegates several sampling and stopping functions to vLLM, and does not certify all backend branches.The main-text correspondence assumes temperature one and an unrestricted policy.Source: B Method conventions and… · p. 18
  • The exact acceptance–deviation identity excludes denominator and residual-mass safeguards and assumes residual recovery independent of the rejected token's identity.Symbolic trace checks do not execute backend caches or GPU kernels.Source: B Method conventions and… · p. 18

C Mechanism comparison with closely related work

  • The mechanism comparison distinguishes proposal retention from next-token generation after failure, and its entries are not interchangeable formulations or a controlled performance comparison.The table caption identifies p and q as the target and actual proposal laws.Source: C Mechanism comparison with closely… · p. 18, Table 3 · p. 19

D Additional Proposal Mechanisms

  • Additional proposal-mechanism comparisons use DFlash-b16 and EAGLE-3-TTT7 with Qwen3-4B, while Table 4 reports task performance and mean accepted length over five seeds.Both comparisons use temperature 1.0, target sampling top-p = 1.0, and an NSD acceptance threshold of 0.95.Source: D Additional Proposal Mechanisms · p. 19, Table 4 · p. 19

D.1 MTP with Two Proposal Tokens

The Qwen3.5-4B MTP comparison uses two proposal tokens per block; NSD has higher reported throughput than SD in the supplied result.

D.1 MTP with Two Proposal Tokens

  • The comparison reports Qwen3.5-4B MTP with two proposal tokens per block using the vLLM backend.Temperature is 1.0, target sampling top-p is 1.0, and the NSD acceptance threshold is 0.95.Source: §D.1 · p. 19
  • 179.90 tokens/s is the NSD throughput, 1.70× the SD throughput of 105.90 tokens/s.The MTP results use block size K = 2; the maximum committed length is three, including the continuation token.Source: §D.1 · p. 19, Table 6 · p. 20
  • Table 6 reports mean accuracy or pass@1 and verification-block-weighted τ for the Qwen3.5-4B MTP comparison.The caption defines τ as weighted by verification-block counts.Source: Table 6 · p. 20

E Limitations

The study's evaluation is bounded by the range of model sizes available and evaluated.

E Limitations

  • The study identifies the range of evaluated model sizes as a limitation.The evaluation focuses primarily on relatively compact target and corresponding draft models.Source: E Limitations · p. 20
  • Hardware constraints and checkpoint availability for Nemotron and DSpark constrain the evaluation's model-size range.The authors name both factors as reasons for the study's scope.Source: E Limitations · p. 20
  • The reported findings primarily cover relatively compact target models and their corresponding draft models.This is the nearest stated boundary on the evaluation's scope.Source: E Limitations · p. 20

Related papers

Loading 2610.07822v1…