SpecDiff-2: Scaling Diffusion Drafter Alignment For Faster Speculative Decoding

1 INTRODUCTION

• Inference-time reasoning can improve task performance but raises wall-time latency because autoregressive models generate tokens sequentially.The resulting wall-time budgets constrain practical problem-solving depth.Source: §1 · p. 1
• On Math500 with 14B and 72B Qwen2.5-Instruct models, Figure 1 reports over 40% higher throughput for SpecDiff-2’s aligned drafter than for base SpecDiff.The figure compares throughput increases relative to vanilla inference across acceleration algorithms.Source: Figure 1 · p. 1
• Speculative decoding reduces latency by having a small drafter propose multiple tokens for a heavier verifier to check in parallel, while preserving vanilla-decoding outputs.Realized throughput depends on keeping drafter cost low and increasing expected acceptance per cycle.Source: §1 · pp. 1–2
• SpecDiff-2 uses discrete diffusion for non-autoregressive drafting and train- and test-time mechanisms to align diffusion drafters with autoregressive verifiers.The approach targets drafting latency and draft-verifier misalignment together.Source: §1 · p. 2
• Up to +55% average tokens-per-second over previous baselines and up to 5.5× average speed-up over standard decoding, without any loss of accuracy.The paper reports state-of-the-art throughput across reasoning, coding, and mathematical benchmarks.Source: §1 · p. 2

2 PRELIM: SPECULATIVE DECODING

• A drafter Q proposes a block of γ tokens from prefix s, and verifier P scores all positions in parallel.The models expose next-token posteriors q(· | s) and p(· | s).Source: §2 · p. 2
• Tokens are committed left-to-right using the standard lossless acceptance rule, with the prefix before token x_i denoted s = x_1:i−1.For i > 1, s is the sequence preceding x_i.Source: §2 · p. 2
• At the first rejection, the remaining draft is discarded and a replacement token is drawn from the normalized residual distribution.The rule accepts immediately whenever q(x_i | s) ≤ p(x_i | s).Source: §2 · p. 2
• The acceptance and replacement rules preserve the verifier’s output distribution, enabling parallel proposal and verification without changing vanilla-decoding outputs.The paper cites Leviathan et al. (2023) for this result.Source: §2 · p. 2

3 ACCEPTANCE & DISTILLATION

• The position-wise acceptance α_j(s) is the probability that draft token j is accepted given that the first j−1 drafted tokens were accepted.This conditions each position’s acceptance on the preceding accepted prefix.Source: Figure 2 · p. 3
• Improvements in position-wise acceptance translate monotonically into longer accepted streaks and higher throughput at fixed drafter cost.Acceptance rate governs realized speed-up in draft-then-verify decoding.Source: §3 · pp. 2–3
• Prior distillation methods commonly treat per-position acceptances as exchangeable and optimize alignment immediately after the prefix, whereas diffusion drafters vary substantially across positions.The full window of draft predictions therefore needs careful optimization to obtain expected speed-up.Source: §3 · p. 3
• Streak-distillation aligns later draft positions at an average of 3.2× greater acceptance than autoregressive distillation at those positions.Autoregressive alignment concentrates gains early and degrades rapidly with draft index, motivating alignment across the full window.Source: §3 · p. 3

4 SPECULATIVE DIFFUSION DECODING

• A diffusion drafter proposes a window of γ tokens simultaneously, so drafting cost depends primarily on denoising steps rather than window length.The paper distinguishes diffusion drafters Qdiff from autoregressive drafters Q.Source: §4 · p. 3
• Masked discrete diffusion corrupts some sequence positions and trains a denoiser to predict token distributions at all positions in parallel.The masked cross-entropy training objective is evaluated over the currently masked positions M(t).Source: §4 · p. 3
• At prefix s, drafting initializes the γ-token window with [MASK] tokens and applies a small number of denoising steps to produce a joint proposal in parallel.The paper notes that a single denoising step is often used in practice.Source: §4 · p. 3
• Low and predictable drafting latency does not by itself solve alignment: block-level diffusion distributions can be miscalibrated against the verifier’s prefix-conditional token decisions.This miscalibration can reduce acceptance even when joint samples appear fluent.Source: §4 · p. 4

5 ALIGNING DIFFUSION DRAFTERS FOR ACCELERATED INFERENCE

• The framework combines train-time and test-time alignment mechanisms on the diffusion drafter while keeping the verifier frozen.The reported evaluations use datasets disjoint from finetuning.Source: §5 · p. 4

5.1 Train-time alignment: Streak-distillation

• Streak-distillation optimizes expected accepted streaks using a tractable surrogate evaluated along verifier-generated trajectories.It replaces the product of acceptance terms with greedy acceptance terms evaluated at verifier tokens.Source: §5.1 · pp. 4–5

5.2 Test-time alignment: Self-selection acceptance

• One diffusion denoising pass exposes position-wise marginals from which K joint drafts can be sampled with negligible sequential overhead.The candidates are ranked by a streak-oriented verifier score before lossless verification.Source: return y · p. 5

return y

• Self-selection chooses the candidate with the greatest expected throughput under the verifier before applying the acceptance rule.The ranking scores can be computed efficiently with tree-style attention.Source: return y · p. 6
• Greedy acceptance verifies the selected draft without using its drafting probabilities, supporting cross-tokenizer use.The verifier probabilities are cached from the candidate-ranking phase.Source: return y · p. 6

6 EXPERIMENTAL SETTINGS

• The study evaluates four drafter–verifier pairs across three datasets using wall-clock speedups, token acceptance, and average accepted streak length.All methods use identical prompts, stopping criteria, and verifier decoding parameters within each setting.Source: §6 · p. 6
• The baselines include Speculative Sampling, EAGLE, EAGLE-2, and the unaligned-drafter SpecDiff method.The paper uses SpecDiff to isolate the impact of the proposed alignment methods.Source: §6 · p. 6
• The tasks cover open-ended question answering, mathematical reasoning, and code generation, with benchmarks outside the reported finetuning distribution.Verifiers include Qwen2.5-72B-Instruct and LLaMA-2-70B-Chat; diffusion drafters include DiffuCoder-7B and DiffuLlama-7B.Source: §6 · pp. 6–7
• The original EAGLE-3 results could not be reproduced in the authors’ environment.Wall-clock measurements use A100 80GB GPUs.Source: §6 · pp. 6–7

7.1 Wall-Clock Speedups and Accepted Streaks

• Larger-capacity drafters yield longer accepted streaks, while parallel diffusion keeps draft latency comparable to smaller autoregressive drafters requiring multiple forward passes.The reported comparison is between 7B drafters and EAGLE-2’s approximately 1B drafters.Source: §7.1 · p. 7
• 4.71× average speed-up on math and coding versus EAGLE-2’s 3.43×, narrowing on open-ended QA to 3.24× versus 2.80×.Relative gains taper from 37% on reasoning to 16% on Q&A, where semantic diversity is higher.Source: §7.1 · pp. 7–8
• 4.22× average speed-up across settings, with SpecDiff-2 improving over EAGLE-2 by over 30% and exceeding 5× speed-up on specialized coding questions.The reported comparison also finds the highest speed-ups and longest accepted streaks across the evaluated speculative decoders.Source: §7.1 · p. 7, Table 2 · p. 8

7.2 Scaling Efficient Reasoning With SpecDiff-2

• More streak-distillation compute, test-time self-selection, and speculative diffusion each raise acceptance and effective parallelism, improving accuracy within a fixed wall-time budget.The paper presents acceleration compute as an additional scaling knob for time-limited reasoning.Source: §7.2 · pp. 8–9
• +63% accuracy over vanilla Qwen2.5-72B and +11% over unaligned SpecDiff at the same 15-second Math500 reasoning budget.The evaluation uses chain-of-thought prompting and varies the wall-time reasoning budget.Source: §7.2 · p. 8, Figure 6 · p. 9

7.3 Ablations: Train-time and Test-time Scaling

• ∼30% higher speed-up for both Qwen models follows streak-distillation through 60,000 steps, with up to +32% over EAGLE-2 baseline throughput.The scaling curve uses at most 75 GPU-hours; Figure 7 reports gains on out-of-distribution Math500 prompts that surpass EAGLE-2.Source: Figure 7 · p. 9, §7.3 · p. 9
• At T = 2.0, scaling to K = 8 drafts adds up to +20% throughput, while T = 0.1 shows little scaling.The tests vary drafting temperature and draft count on Math500 across both Qwen2.5 models.Source: §7.3 · pp. 9–10
• +30% average speed-up from train-time distillation and a further +15% from the test-time algorithm yield 40–50% improvement over unaligned SpecDiff.Source: §7.3 · p. 10

• Early acceleration work used pruning and quantization, while speculative decoding accelerates generation without degrading output quality [Leviathan et al., 2023; Chen et al., 2023].Pruning and quantization improved efficiency at a cost to overall generation quality [Frantar et al., 2023].Source: §8 · p. 10
• Training objectives based on distillation and optimal transport align autoregressive drafter distributions with verifier posteriors, while draft-tree methods improve throughput per verification step [Zhou et al., 2023; Sun et al., 2023; Li et al., 2024a;b; 2025].Source: §8 · p. 10
• Parallel draft heads and diffusion language models address drafter latency by reducing sequential drafting dependencies [Ankner et al., 2024; Christopher et al., 2025].The paper reports substantial improvements against these proposals.Source: §8 · p. 10

9 FUTURE WORK AND LIMITATIONS

• Formal analysis of greedy-acceptance trade-offs and systematic cross-family and cross-tokenizer drafting remain open directions.The paper notes that acceptance rules are typically fixed, while greedy acceptance can support heterogeneous drafters and mismatched tokenizers.Source: §9 · p. 10
• Using K specialized drafters raises unresolved diversity–alignment, cost-aware selection, and compute-allocation questions.The proposed extension would select among parallel drafters specialized to sub-distributions.Source: §9 · p. 10
• Scaling laws and compute–latency frontiers for diffusion drafters remain to be derived as functions of γ, acceptance, and verifier size.The paper also flags optimal diffusion-drafter size as an open question.Source: §9 · p. 10
• Efficient semi-autoregressive inference kernels and KV-cache support for γ-length diffusion drafts are still lacking.The paper identifies both as hardware-optimization needs for efficient inference.Source: §9 · p. 10

10 CONCLUSION

• Up to +55% average faster generation over previous baselines and up to 5.5× average speed-up over standard decoding, without accuracy loss.The conclusion reports state-of-the-art performance across reasoning, coding, and mathematical benchmarks.Source: §10 · p. 11

A ADDITIONAL RESULTS

• DIFFUCODER outperforms EAGLE-2 in nearly all settings for QWEN-2.5-14B-INSTRUCT, while DIFFULLAMA does not consistently outperform it.For these smaller verifiers, the diffusion drafter is over 20× larger than EAGLE/EAGLE-2, making drafting latency uncompetitive; higher tokens per draft drive SpecDiff-2's advantage on select tasks.Source: A ADDITIONAL RESULTS · p. 13

A.1 Llama Model Temperature Ablation

• Temperature = 1.0 gives the strongest performance when scaling the number of DIFFULLAMA drafts.This result supplements the main-text drafter-temperature analysis.Source: §A.1 · p. 13

A.2 Diffusion Steps Ablation

• With T = 1, speed-up degrades as diffusion steps increase because draft latency rises linearly while accepted tokens per draft remain relatively stagnant.The paper sets the number of diffusion steps to T = 1.Source: §A.2 · p. 13

A.3 Gamma Ablation

• DIFFUCODER uses γ = 32 and DIFFULLAMA γ = 16, balancing small windows that limit long accepted drafts against large windows that lower draft quality.The reported figures illustrate speed-up trade-offs across diffusion steps and window sizes.Source: §A.3 · p. 13, Figure 10 · p. 13, Figure 9 · p. 13
• DIFFUCODER handles larger drafts more effectively than DIFFULLAMA, likely reflecting its 130B specialized-data tokens versus 60B tokens mostly from SlimPajama.Source: §A.3 · pp. 13–14

A.4 Joint Alignment Scaling

• Combining streak-distillation with test-time parallel drafting improves throughput by +39% over the base diffusion drafter and EAGLE-2, reaching 4.3× throughput over QWEN2.5-72B on Math500.Figure 11 illustrates the joint impact of both alignment techniques.Source: §A.4 · p. 14, Figure 11 · p. 14

A.5 Streak-distillation versus DistillSpec

• A naive DistillSpec variant averages position-wise acceptances across the γ window but remains insensitive to positional dependencies, yielding lower expected throughput than streak-distillation.The reported throughput gap follows from the variant's mismatch with the theoretical throughput definition.Source: §A.5 · p. 14

B EXPERIMENTAL DETAILS

• Streak-Distill achieved a +35% alignment/throughput increase over the base model versus DistillSpec for Qwen2.5-14B-Instruct on Math500.Both methods trained for 40k steps (∼50 GPU-hours).Source: Figure 12 · p. 15

B.1 Hardware Setup

• The verifier was evaluated in bfloat16 on GPU, with 75-GiB max-memory limits for devices 0 and 1 passed during large-model loading; runs used PyTorch 2.3+ with CUDA and Transformers 4.46.x.The drafter was loaded through a CPU-first path, and the procedure avoided runtime adapter merges during speculative runs and timing.Source: §B.3 · pp. 15–16, §B.2 · p. 15

B.4 Acceptance and Distribution Collection

• A DraftProbCollector records drafted-token positions and probabilities, optionally capturing full distributions for verifier-side reweighting; verifier sampling accepts or replaces each token and logs accepted streak lengths.Acceptance compares a uniform draw with the verifier probability; rejected tokens are resampled from the verifier distribution with the rejected token zeroed out.Source: §B.4 · p. 16

B.5 Evaluation Protocol

• Evaluation logs drafter and verifier timings, cache operations, pass counts, and accepted streaks, then compares speculative throughput with verifier-only generation on the same prompt and hardware.Benchmarks are streamed from Hugging Face and include MATH-500, LiveCodeBench, MT-Bench, and GSM8K.Source: §B.5 · p. 16, §B.6 · p. 16

B.7 Datasets

• Distillation was robust to different datasets and exhibited strong transfer across tasks.The tested corpora included Qwen 14B/72B completions, filtered Llama 13B completions, and Llama 70B instruction responses.Source: §B.7 · p. 16

B.8 Test-time verification.

• Multi-draft verification flattens draft trees for batched verifier scoring with a custom branch-aware attention mask, reuses KV-cache along valid paths, and logs parallel acceptance decisions.The implementation also profiles mask construction and tree flattening to compare their overhead with the single-branch loop.Source: §B.8 · pp. 16–17

Problem 7: Counting and Probability

• The Problem 7 counting result is 6.Source: Problem 7: Counting and Probability · p. 19

Problem 8: Counting and Probability

• There are 6 distinct arrangements of two indistinguishable IMO and two indistinguishable IPhO gold medals.Source: Problem 8: Counting and Probability · p. 19

Problem 9: Number Theory

• Rosa is 49 years old: the congruences imply R = 84m + 49, and the under-75 condition gives m = 0.Source: Problem 9: Number Theory · p. 19