PaperlayerView exact revision on arXiv

Source-linked AI summary

Speculative Diffusion is a Tale of Two Tradeoffs

Aaron Zorel, Guy Van den Broeck, Aditya Grover, Daniel Israel

arXiv:2610.31819v1cs.LG

TL;DR

Autoregressive decoding is sequential, while lossless speculative decoding limits speedup to what draft-verifier alignment permits. This paper jointly varies diffusion effort and acceptance leniency in speculative diffusion. It reports that the joint variation often extends the speed-quality Pareto frontier.

  • Problem

    Autoregressive decoding is bottlenecked by sequential generation, and lossless speculative decoding limits speedup by how closely the drafter approximates the verifier.

    Source: §1 · p. 1

  • Method

    The paper studies speculative diffusion by jointly varying a diffusion drafter's denoising effort and the verifier's acceptance leniency.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    Joint variation of diffusion effort and acceptance leniency often extends the speed-quality Pareto frontier of speculative diffusion.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Neither diffusion effort nor acceptance leniency alone captures the full speculative-diffusion Pareto frontier.

    Source: §6 · p. 9

    The optimality bounds concern next-token distributions conditioned on a prefix and do not trivially extend to joint distributions across multiple positions.

    Source: §2 · p. 4

Abstract

from arXiv · show

Speculative decoding is a technique that accelerates the inference speed of autoregressive large language models (LLMs) by sampling from a faster, lower quality draft model. Traditionally, a fundamental characteristic of speculative decoding is that it is lossless with respect to the verifier distribution, guaranteeing speed without sacrificing quality. This differs from recently developed diffusion LLMs (dLLMs), which are notably able to achieve substantial speedups by degrading the quality of their outputs with less diffusion effort. Because diffusion models are able to sample faster than autoregressive models by generating tokens in parallel, they are considered strong candidates for draft models in speculative diffusion. While it is widely understood that diffusion models can sacrifice quality for speed, speculative decoding itself can also be generalized to generate lossy samples faster using more lenient acceptance criteria with respect to the verifier distribution. Presented with the two parameters of diffusion effort and speculative decoding leniency, this paper explores the full achievable Pareto frontier via joint variations of these parameters. Our experiments show that jointly varying diffusion effort and acceptance leniency can extend the speed-quality Pareto frontier over single step diffusion drafting.

1 INTRODUCTION

Autoregressive decoding is slowed by sequential token generation, while traditional speculative decoding preserves the verifier distribution but limits speedup by drafter alignment. This paper studies speculative diffusion by jointly varying diffusion effort and acceptance leniency, and reports that their combinations can extend the speed-accuracy Pareto frontier.

1 INTRODUCTION

  • Autoregressive inference is bottlenecked by sequential token generation and repeated transfer of model parameters and the growing KV cache.Each decoding step transfers these for a single token.Source: §1 · p. 1
  • Traditional speculative decoding uses a faster drafter and parallel verification while preserving the verifier's output distribution.It accepts an initial proposal prefix through rejection sampling, giving acceleration without degrading generation quality.Source: §1 · p. 1
  • Speculative diffusion combines a diffusion drafter's denoising-effort tradeoff with the verifier's acceptance-leniency tradeoff.Mapping their joint configuration space reveals interactions between the two parameters.Source: §1 · p. 2
  • The paper's theoretical contributions include an optimal linear-interpolation target under a total-variation bound and conditions for preserving the verifier distribution with multistep drafts.It also provides a throughput model for the interaction between speed and quality.Source: §1 · p. 2
  • The empirical study spans masked and uniform diffusion drafters on reasoning and coding tasks.The introduction identifies these as the paper's empirical settings.Source: §1 · p. 2
  • Jointly varying diffusion effort and acceptance leniency often extends the speed-accuracy Pareto frontier of speculative diffusion.The paper reports that multistep denoising with more lenient acceptance can generate faster at comparable accuracy.Source: §1 · p. 2

2 BACKGROUND

This section defines diffusion drafting and speculative decoding, then introduces lossy acceptance as a way to trade deviation from the verifier distribution for higher proposal acceptance. It characterizes the optimal target under a total-variation constraint and states the scope of that result.

Diffusion Language Models

  • Diffusion language models denoise a noisy token block in parallel, with diffusion effort K denoting the number of denoising steps.More effort typically improves quality at the cost of inference speed.Source: §2 · p. 3

Lossless Speculative Decoding

  • Lossless speculative decoding accepts proposals by rejection sampling and uses a verifier-based replacement after rejection, preserving the verifier distribution.Its acceptance depends on alignment between the draft and verifier distributions.Source: §2 · p. 3

Lossy Speculative Decoding

  • Lossy speculative decoding relaxes exact sampling from the verifier, and this section studies linear interpolation under a total-variation constraint.The distance from the modified target to the draft determines acceptance, while its distance from the verifier measures deviation.Source: §2 · p. 3

Optimality of Linear Acceptance

  • Under a total-variation deviation budget, the optimal target linearly interpolates between verifier and draft distributions.The theorem gives maximum acceptance probability as 1 − DTV(p, q) + ϵ.Source: §2 · p. 4
  • The interpolation parameter λ shifts the target toward the draft, ranging from verifier sampling at λ = 0 to draft sampling at λ = 1.Figure 2 illustrates the theorem and a motivating example of separate Pareto-frontier regions.Source: §2 · p. 4, Figure 2 · p. 4

Scope of the Result

  • The optimality result concerns a target chosen before the proposed token is observed and a next-token distribution conditioned on prefix h.The bounds do not trivially extend to the joint distribution across multiple token positions.Source: §2 · p. 4

3 SPECULATIVE DIFFUSION

Speculative diffusion combines the speed–quality tradeoffs of diffusion drafting and lossy speculative decoding by varying diffusion effort and acceptance leniency. The section motivates multi-step diffusion drafting, explains how to use its proposals in rejection sampling, and models the resulting throughput–quality tradeoff.

3.1 A MOTIVATING EXAMPLE

  • Diffusion can draft over a wider speed range than a small autoregressive drafter, while verifier guidance can address a diffusion model’s accuracy plateau; their complementary tradeoffs may extend the Pareto frontier.The section motivates combining diffusion’s higher speeds with speculative decoding’s higher quality.Source: §3.1 · p. 5

3.2 MULTI-STEP DIFFUSION DRAFTING

  • Multi-step diffusion drafting varies denoising effort, often through step count K, and can use the final-step probabilities as proposal probabilities in rejection sampling.Other effort controls can include confidence thresholds or entropy bounds.Source: §3.2 · pp. 5–6
  • Unlike autoregressive drafting, multi-step diffusion does not condition each token on previously accepted tokens, and its marginal probabilities may require summing over intractably many denoising trajectories or an ELBO approximation.Source: §3.2 · p. 5

3.3 A SIMPLE THROUGHPUT MODEL

  • The throughput model uses expected acceptance rate and verifier agreement as proxies for speed and quality, with diffusion effort K and leniency λ controlling their tradeoff.It estimates drafting time as cd(K) ≈ a + bK and assumes verifier-call time is constant.Source: §3.3 · p. 6
  • The model captures an approximately linear acceptance–agreement tradeoff, but no single denoising budget reaches the full throughput–quality frontier; measured discrepancies reflect changing prefixes and EOS-shortened blocks.The setup does not establish how total variation quality maps to downstream reasoning accuracy, and other acceptance criteria may behave differently.Source: §3.3 · pp. 6–7

4 EXPERIMENTS

The experiments explore diffusion effort and acceptance leniency across diffusion–verifier pairs and compare drafters under a shared verifier. Jointly varying both controls extends the speed–quality frontier for multi-step-capable SDAR, while several specialized one-step drafters remain strong in throughput.

4.1 EXPERIMENTAL SETUP

  • The experiments first examine Dream-7B with Qwen2.5-32B and DiffusionGemma-26B-A4B with Gemma4-31B, then compare recent drafters under a common Qwen3-8B verifier.Source: §4.1 · p. 7, §4 · p. 7
  • The experiments use GSM8K, HumanEval, and GPQA Diamond, excluding model–benchmark pairs when standalone accuracies are nearly equal or the drafter performs worse than random.Dream and SDAR are evaluated on GSM8K and HumanEval; DiffusionGemma is evaluated on GPQA Diamond.Source: §4.1 · p. 7

4.2 JOINT EXPLORATION OVER TWO TRADEOFFS

  • Across both model pairs, several diffusion-step values contribute to the speed–accuracy frontier; low effort with strict acceptance tends toward higher accuracy, while higher effort with lenient acceptance tends toward higher speed.The multi-effort frontier pattern persists across drafting block sizes.Source: §4.2 · p. 8

4.3 COMPARING DRAFTERS WITH A SHARED VERIFIER

  • Jointly varying diffusion effort and acceptance leniency substantially improves SDAR’s speed–accuracy tradeoff and places it on the frontier despite no specialized speculative-diffusion training or architecture.Specialized one-step drafters remain strong, especially in throughput.Source: Figure 5 · p. 8, §4.3 · p. 9

5 RELATED WORK

Prior work develops diffusion drafting and lossy speculative acceptance separately. The paper positions its analysis at their intersection, while noting that the linear acceptance criterion itself is not new.

Diffusion drafters

  • Diffusion-drafting work reduces sequential proposal evaluations, but typically finds little benefit from multiple denoising steps; tree-based verification is outside this work’s scope.Prior approaches include causal-consistency path search, target-feature conditioning, and hybrid diffusion-drafting/autoregressive-verification models.Source: §5 · p. 9

Lossy speculative decoding

  • Prior work introduced lossy acceptance and analyzed its trade-off, but did not investigate leniency with diffusion drafters.Nemotron-Labs-Diffusion mentioned interpolating autoregressive and diffusion distributions as a possible direction, without experimental evaluation.Source: §5 · p. 9
  • The paper analyzes the previously named linear acceptance criterion as a constrained optimization problem and claims optimality under a total variation constraint.The criterion has appeared under names including collaborative decoding, linear interpolation, and dynamic ensemble verification.Source: §5 · p. 9

6 CONCLUSION

The conclusion identifies denoising effort and verifier acceptance leniency as complementary speed-quality mechanisms in speculative diffusion. The authors also disclose several uses of LLM assistance.

6 CONCLUSION

  • Speculative diffusion’s speed-quality frontier depends on both drafter denoising effort and verifier acceptance leniency; varying either alone does not capture the full frontier.The authors suggest routing requests to frontier configurations by task difficulty and error tolerance, and dynamically adjusting denoising budgets and verification thresholds.Source: §6 · p. 9

AI USAGE DISCLOSURE

  • The authors used LLMs to find relevant papers, plot results, polish LaTeX, and draft code, while retaining responsibility for the final content.Source: AI USAGE DISCLOSURE · p. 10

A PROOF OF THEOREM 1

The proof uses a total-variation triangle-inequality bound and shows that a constructed distribution attains it. The supplied passages do not state the resulting acceptance-rate expression.

A PROOF OF THEOREM 1

  • The proof bounds total variation from the verifier below by the original distance minus the distance between the constructed and original distributions.It then establishes that the bound is attained by a constructed distribution.Source: A PROOF OF THEOREM 1 · p. 13
  • The proof concludes that the constructed distribution attains the lower bound, yielding the stated acceptance-rate result.Source: A PROOF OF THEOREM 1 · p. 13

B ALTERNATIVE PROOF OF THEOREM 1

The appendix gives a geometric alternative proof, states assumptions for an expected-token proposition, and describes rejection sampling and a modified Cactus procedure. The experiments used one H200 GPU.

B ALTERNATIVE PROOF OF THEOREM 1

  • The alternative proof interprets the interpolated signed measure as the closest point to the verifier among measures within a specified norm-distance of the draft.The argument uses norm properties and notes that the closest-point statement applies to probability measures as well as signed measures.Source: B ALTERNATIVE PROOF OF THEOREM 1 · pp. 13–14

C EXPECTED NUMBER OF GENERATED TOKENS PER DRAFTING ROUND

D REJECTION SAMPLING FOR MULTI-STEP DIFFUSION DRAFTING

  • For multi-step diffusion drafting, rejection sampling preserves the fixed target distribution even when the proposal probability vector is random and unknown in distribution.The proof conditions on the realized proposal and then applies the law of total expectation.Source: D REJECTION SAMPLING FOR MULTI-STEP… · p. 15

E MODIFIED CACTUS

  • Modified Cactus raises the drafted token’s target probability under a divergence constraint, rescales other tokens, and accepts according to the adjusted probability.The Taylor approximation is only justified when the desired modified probability is close to the original verifier probability; otherwise the experiments solve for the optimum with Newton’s method.Source: E MODIFIED CACTUS · p. 15

F EXPERIMENTAL DETAILS

F.1 BENCHMARK SELECTION

The benchmarks use GSM8K, GPQA Diamond, and HumanEval, with accuracy baselines guiding task selection. The experiments focus on settings where drafters have useful task accuracy and verifiers have room to improve.

F.1 BENCHMARK SELECTION

  • The benchmark set uses 500 random GSM8K questions and all GPQA Diamond and HumanEval problems, with one completion per problem.GSM8K and GPQA score extracted final answers; HumanEval uses pass@1 from executing generated code against tests.Source: §F.1 · pp. 15–16
  • Table 1 reports baseline task accuracies with one standard error and defines the gap as verifier minus drafter accuracy in percentage points.Source: Table 1 · p. 16
  • Dream and SDAR meet the task-accuracy and verifier-improvement criteria on GSM8K and HumanEval, but perform worse than random on GPQA.DiffusionGemma is run only on GPQA Diamond because its accuracy is almost the same as the verifier's on GSM8K and HumanEval.Source: §F.1 · p. 16

F.2 MODELS, DATA, AND SAMPLING

The section specifies model, sampling, and sweep configurations, then calibrates a simple throughput model and presents linear-acceptance comparisons. Calibration uses empirical drafter-verifier distances to set lossless acceptance rates, while Figures 7 and 8 provide corresponding benchmark plots.

F.2 MODELS, DATA, AND SAMPLING

  • The model configurations, sampling settings, and completion limits are documented in Tables 2–4; runs disable thinking and use generation limits of 2048 for GPQA and 512 for GSM8K and HumanEval.Dream-7B uses entropy-based unmasking, while DiffusionGemma's drafter runs use a fixed number of diffusion steps because its native termination criteria complicate effort control.Source: §F.2 · pp. 16–17, Table 2 · p. 16, Table 3 · p. 17

F.3 SWEEP GRIDS AND MEASUREMENT PROTOCOL

  • The sweep grids vary diffusion effort and acceptance leniency across model pairs, and Table 5 lists the evaluated Cactus budgets at each effort.For the linear modifier, λ ranges from 0 to 1 in increments of 0.1, with K grids varying by model pair and benchmark.Source: §F.3 · p. 18

F.4 SIMPLE THROUGHPUT MODEL CALIBRATION

  • For Dream-7B and Qwen2.5-32B, dK is 0.566, 0.517, 0.424, and 0.360 for K = 1, 2, 3, and 4, yielding lossless acceptance rates 0.434, 0.483, 0.576, and 0.640.The measured distances determine the starting points of Figure 3a's lines; Figure 6 shows the drafting-cost linear fit and verifier-cost distribution.Source: §F.4 · p. 18, Figure 6 · p. 19

G.1 LINEAR ACCEPTANCE

G.2 DRAFTING BLOCK SIZE ABLATION

The appendix tests block size, extends comparisons to HumanEval, and reports additional shared-verifier and multi-step Uno sweeps. Uno's extra drafting steps generally do not improve its measured speed-quality frontiers, while qualitative examples illustrate high-leniency failure modes.

G.2 DRAFTING BLOCK SIZE ABLATION

  • Changing Dream–Qwen's drafting block size affects latency and throughput but preserves the main observation that different diffusion efforts contribute to different frontier regions.The ablation uses B ∈ {32, 64, 128} and remaining-canvas drafting, sweeping K and λ for each configuration.Source: §G.2 · p. 20, Figure 9 · p. 20

G.3 HUMANEVAL

G.4 SHARED-VERIFIER COMPARISON

  • Figure 12 presents complete Figure 5 sweeps, distinguishing SDAR effort settings by color and marking joint and fixed-K = 1 frontiers with thick curves.Hollow markers include measured configurations that are dominated.Source: Figure 12 · p. 22

G.5 UNO AT MULTIPLE DIFFUSION STEPS

  • K = 1 captures Uno's entire latency frontier on both benchmarks and its entire throughput frontier on HumanEval; K = 4 contributes to neither frontier.On GSM8K, one low-accuracy K = 2 setting also contributes to measured throughput, and the pattern is consistent with Uno's one-step distillation objective.Source: §G.5 · p. 22, Figure 13 · p. 22

H QUALITATIVE EXAMPLES

  • Selected Dream–Qwen examples under linear acceptance illustrate failure modes at high acceptance leniency, with correct calculations or code and errors or repetition highlighted.Source: H QUALITATIVE EXAMPLES · p. 23

H.1 MATH

The qualitative examples include math responses to a 20% enrollment increase and code responses to a score-comparison task. They range from a correct answer or implementation to visible errors and malformed output.

H.1 MATH

  • Another math response gives 600 instead of 60 after stating that 50 + 10 = 60.Source: §H.1 · p. 23
  • A math response calculates 20% of 50 as 10 and adds it to reach 60 students.Source: §H.1 · pp. 23, 25

H.2 CODE

  • The code example returns the absolute difference between each actual match score and its guess, producing a result array.Source: §H.2 · p. 26
Loading 2610.31819v1…