PaperlayerView exact revision on arXiv

Source-linked AI summary

Factorization-Guided Training and Inference for Diffusion-Drafted Speculative Decoding

Jiakun Fan, Min Xiao, Qiang Wang, Neiwen Ling, Zhichao Liu, Zhihao Jia, Dimitrios S. Nikolopoulos, Zijun Wei

arXiv:2610.23935v1cs.LGcs.AI

TL;DR

Diffusion drafters parallelize proposals, but prefix rejections and draft–verify latency can limit the throughput benefit. DScout uses factorizations to guide training and inference choices, and improves end-to-end performance across models, tasks, and serving loads.

  • Problem

    In diffusion-drafter decoding, correct predictions after an early rejection may not increase accepted length, and higher MAT may not improve throughput when draft–verify latency also rises.

    Source: §1 · p. 1

  • Method

    DScout uses position-wise loss and MAT–latency throughput factorizations to guide training supervision and inference configuration selection.

    Source: §1 · pp. 1–2

  • Results

    DScout’s training recipe and inference configurations improve end-to-end performance across models, tasks, and serving loads.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The factorizations provide a framework for diffusion-drafter training strategy design and inference configuration selection.

    Source: §6 · p. 10

    Inference selection currently uses a shared block size per target model and concurrency setting; whether online adaptation yields sustained gains under changing workloads remains to be evaluated.

    Source: §6 · p. 10

Abstract

from arXiv · show

Speculative decoding accelerates large language model inference through parallel verification of tokens proposed by a drafter. Diffusion drafters further parallelize proposal generation, but greater proposal capacity or higher mean accepted tokens (MAT) need not yield higher throughput. Because verification accepts only a consecutive prefix, early rejections can render later correct predictions ineffective, limiting MAT gains, while even higher MAT may not improve throughput if per-round latency increases. We introduce DScout, which uses two complementary factorizations to guide diffusion drafter training and inference for higher throughput. For training, DScout factorizes position-wise loss into inclusion, weighting, and prediction loss. This guides the design and combination of mixed-horizon training, intra-block unmasking, and acceptance-weighted loss to align supervision with prefix acceptance and improve MAT. For inference, DScout factorizes throughput as the ratio of MAT to per-round latency, guiding target-conditioning depth, drafter depth, and inference block size selection to identify a high-throughput configuration for each workload. Across six benchmarks, DScout achieves $3.84\times$ and $4.14\times$ mean speedups over autoregressive decoding on Qwen3-4B and Qwen3-8B, respectively, improving mean speedup over the state-of-the-art baseline by 9.1% and 7.1%. Across models and concurrency settings, our inference configuration policy provides up to 9.75% additional mean speedup over DScout's fixed configuration.

1 Introduction

DScout addresses two bottlenecks in diffusion-drafter decoding: prefix-dependent token acceptance during training and the MAT–latency trade-off during inference. It uses complementary factorizations to guide supervision and configuration selection.

1 Introduction

  • Prefix-dependent acceptance means correct predictions after the first rejection do not contribute to accepted length, while higher MAT improves throughput only if it compensates for added draft–verify latency.These constraints motivate accounting for prefix dependence in training and balancing MAT against latency at inference.Source: §1 · p. 1
  • DScout factorizes position-wise loss into inclusion, weighting, and prediction loss to guide supervision allocation and combine training interventions around prefix acceptance.The resulting recipe combines mixed-horizon training, intra-block unmasking, and acceptance-weighted loss to improve MAT.Source: §1 · p. 2
  • DScout factorizes throughput as MAT divided by draft–verify round latency, guiding selection of target-conditioning depth, drafter depth, and inference block size.The selection balances MAT and latency across workloads.Source: §1 · p. 2
  • DScout evaluates its training recipe and inference configuration selection across target models, benchmarks, and concurrency settings using end-to-end comparisons and ablations.The paper positions these as complementary ways to assess acceleration and the contributions of the training and inference choices.Source: §1 · p. 2

2 Preliminaries

Speculative decoding uses parallel target verification of draft tokens, accepting only a consecutive prefix. Block-diffusion drafters generate multiple future tokens jointly, conditioned on target-model features.

2.1 Speculative Decoding

  • Each speculative-decoding round proposes bs −1 tokens for parallel target verification, accepts the consecutive prefix before the first rejection, and emits a replacement or bonus token.Appropriate acceptance and replacement rules preserve the target model’s output distribution [7] [16].Source: §2.1 · p. 2
  • Prefix survival captures the dependence of accepted length on earlier tokens: a draft token contributes only if all preceding draft tokens are accepted.The drafter proposes bs −1 tokens, and the target verifies them in one parallel forward pass.Source: §2.1 · pp. 2–3

2.2 Block-Diffusion Drafting

  • Block-diffusion drafters predict multiple future tokens jointly by unmasking positions within a block using available context.The target model verifies the resulting proposal as in speculative decoding.Source: §2.1 · p. 2, §2.2 · p. 3
  • DFlash’s drafter uses target features from selected layers, fused through a linear projection, to condition its block-diffusion predictions.The target remains frozen; drafter layers map fused features into cached keys and values reused across rounds.Source: §2.2 · p. 3

3 Method

DScout uses a position-wise loss factorization to align diffusion-drafter supervision with prefix acceptance, and a throughput factorization to select inference settings that balance mean accepted tokens (MAT) against latency. Its training strategies address supervision inclusion, weighting, and prediction context; its inference policy jointly considers target-conditioning depth, drafter depth, and block size.

3 Method

  • DScout combines training-recipe design with inference configuration selection, guided by position-wise loss and throughput factorizations.The throughput factorization balances MAT against draft–verify latency.Source: §3 · p. 3, §3.2 · p. 6

3.1.1 Position-Wise Loss Factorization

  • Position-wise loss factorization separates supervision inclusion, weighting, and prediction loss, exposing how horizon and mask choices allocate training signal across positions.For a position k, π(k) is its expected inclusion in the loss across training horizons and mask patterns; this work uses token-level cross-entropy and does not explore alternatives.Source: §3.1.1 · pp. 3–4

3.1.2 Factorization-Guided Training Strategies

  • Mixed-horizon training with {8, 16, 24, 32} reduces early rejections versus fixed-32 while retaining continuations beyond 16 tokens at the same inference block size; with unmasking, it outperforms either strategy alone, with non-additive gains.Unmasking reveals target tokens that provide context without contributing to prediction loss. AWL weights each position’s loss based on the likelihood that verification reaches it.Source: §3.1.2 · pp. 4–5

3.2 Inference Configuration Selection

  • Inference configuration selection evaluates target-conditioning depth, drafter depth, and inference block size jointly through their effects on MAT and draft–verify latency.Full target conditioning is motivated by a measured throughput change below 1% across tested depths; mixed-horizon checkpoints support block-size evaluation without retraining for each candidate size.Source: §3.2 · pp. 6–7
  • Throughput rises and then falls with drafter depth; the highest measured throughput occurs at d = 6 for bs = 16, 24, 32, and d = 4 for bs = 8.The depth-selection analysis uses diminishing MAT gains and approximately linear latency growth to motivate locating the throughput peak efficiently.Source: §3.2 · pp. 6–7

4 Experimental Evaluation

The evaluation compares DScout with speculative-decoding baselines across six tasks and studies end-to-end performance, training-recipe ablations, and drafter-depth selection. Results show that performance depends on workload and the balance between accepted tokens and draft–verify latency.

4 Experimental Evaluation

  • The evaluation covers six tasks, compares against EAGLE-3, DFlash, and PARD-2, and repeats evaluations five times.Speedup is computed against a matched AR baseline per dataset and averaged equally across datasets.Source: §4 · p. 7

4.1 End-to-End Performance.

  • Up to 4.14× speedup over AR and about 9% higher mean speedup than DFlash on Qwen3 targets at both temperatures.The comparison uses matched AR baselines across the benchmark tasks.Source: §4.1 · p. 7, Table 3 · p. 8
  • Shorter blocks can improve throughput despite lower MAT when draft–verify rounds finish sufficiently faster.The preferred block size varies by task and temperature; on Qwen3-8B at temperature 0, bs = 32 favors Math while bs = 16 favors MBPP, LiveCodeBench, and MT-Bench.Source: §4.1 · pp. 7–8

4.2 Design Analysis

  • The highest measured throughput occurs at drafter depth d = 6 for bs = 16, 24, and 32, but at d = 4 for bs = 8.The preferred depth therefore depends on inference block size; the results motivate the Qwen3-8B 35t6d configuration.Source: §4.2 · p. 8

4.1 End-to-End Performance.

  • 2.45–4.14× mean speedup over AR, with DScout (best) exceeding native-block-size DFlash in all tested settings.Block-size selection adds up to 9.75% mean speedup over DScout (16) at fixed checkpoints.Source: §4.1 · p. 7, Table 4 · p. 9

4.2 Design Analysis

  • At bs = 16, the cumulative training ablation improves MAT by 16.1%, with every added strategy contributing and AWL providing the largest conditional increment.These gains require no additional inference-time operations.Source: §4.2 · p. 8, Figure 4 · p. 9

5 Related Work

Related work spans autoregressive and parallel speculative drafters, block-diffusion approaches, and serving-efficiency systems. DScout builds on block-diffusion drafting while focusing on training supervision and inference configuration.

Autoregressive and parallel drafters

  • Early speculative methods draft autoregressively, while EAGLE-3 combines multi-layer target features with direct token prediction and Medusa, Hydra, and PARD reduce sequential drafting.These approaches use multiple prediction heads or parallel masked-token prediction.Source: §5 · p. 9

Block-diffusion drafters

  • DiffuSpec, SpecDiff-2, and DART explore diffusion drafting; DFlash generates a target-conditioned block-diffusion proposal in one pass, which DScout builds on.DScout studies how supervision and inference configuration jointly affect decoding efficiency.Source: §5 · p. 9

Serving efficiency and inference configuration

  • Serving systems improve efficiency through memory management, scheduling, or token-tree verification, while DScout selects drafting configurations using MAT–latency trade-offs.Mixed-horizon training supports block-size choices across workloads and concurrency settings without retraining.Source: §5 · pp. 9–10

6 Discussion and Conclusion

DScout uses loss and throughput factorizations to guide training strategy design and inference configuration selection. The paper identifies online block-size adaptation as a next step beyond its current fixed selection.

Discussion

  • The current inference policy selects one shared block size per target model and concurrency setting; online adaptation under changing workloads remains to be evaluated, including its overhead.Mixed-horizon checkpoints support the proposed flexibility without retraining.Source: §6 · p. 10

Conclusion

  • Across models, tasks, and serving loads, the resulting training recipe and inference configurations improve end-to-end performance.The authors present the two factorizations as a framework for future training-strategy design and inference-configuration selection.Source: §6 · p. 10

A Evaluation Details and Supplementary Results

The appendix details evaluation protocols and aggregation, then documents the serving block-size grid used to select configurations. Grid selection holds each checkpoint fixed and optimizes equal-weight mean speedup across six tasks.

A.1 Additional Setup and Aggregation

  • Complete-system comparisons use unequal training budgets, matched AR baselines, and separate platforms for training-control studies and system benchmarks.Speedups across SGLang and PARD-2 runtimes do not establish an absolute throughput ranking.Source: §A.1 · p. 13
  • Repeat variability is the sample SD of five paired macro-speedup repeats on fixed prompts, not uncertainty over unseen tasks.Point estimates instead use task-level ratios of means.Source: §A.1 · p. 13

A.2 Serving Configuration Grid

  • At fixed checkpoints, each target–concurrency setting selects one block size by maximizing equal-weight mean speedup across six tasks; this is not a joint optimum over target-conditioning and drafter depth.The evaluated candidates are bs ∈ {8, 10, 12, 14, 16, 24, 32}; Figure 6 records the candidates and the selected block size in each row.Source: §A.2 · p. 13, Figure 6 · p. 14
  • Table 5 reports sample standard deviations of serving speedup over five repeats.Its entries are ordered DFlash / DScout (best).Source: §A.2 · p. 13, Table 5 · p. 14

A.3 Acceptance and Depth Study Protocols

These protocols compare position-wise acceptance and drafter-depth throughput, and estimate the arithmetic cost of deeper target conditioning. They specify the aggregation and measurement conditions for interpreting those comparisons.

A.3 Acceptance and Depth Study Protocols

  • The training-horizon controls use Qwen3-8B hidden-9t9d checkpoints trained for six epochs, with mixed horizons sampling B ∈ {8, 16, 24, 32} equally.Fixed-16 uses γ = 7; fixed-32 and mixed horizons use γ = 15, and Figure 2 averages dataset-normalized histograms across six system-benchmark tasks.Source: §A.3 · p. 14
  • Position-wise acceptance compares final DScout (35t6d) with DFlash on Qwen3-8B using conditional acceptance derived from accepted-length histograms.Results are averaged equally across five repeats and datasets within each domain; conditioning on surviving prefixes means acceptance need not decrease with position, and the comparison does not isolate architecture, recipe, or training-budget effects.Source: §A.3 · p. 14, Figure 7 · p. 15
  • Drafter-depth measurements use the six-epoch Qwen3-8B 35t hidden-conditioned family across d ∈ {2, 3, 4, 5, 6, 7, 8, 12} and bs ∈ {8, 16, 24, 32}.For each dataset, the minimum and maximum of five repeats are discarded; the middle three are averaged, then averaged equally across six datasets, with no error bars because repeat-level dispersion is unavailable for aggregates.Source: §A.3 · p. 14

A.4 Target-Conditioning Cost

  • Throughput profiling sweeps target-conditioning depth t = 1–35 at d = 6, bs = 16, and c = 1, holding acceptance at 16 tokens per round to isolate execution cost.The measurements are end-to-end throughput rather than isolated feature-fusion latency; bars represent run variability, not confidence intervals for the throughput ratio.Source: §A.4 · p. 15
  • For Qwen3-8B at bs = 16 and about seven newly fused context tokens, the estimated fusion and target-dense costs are 6.11 and 222.26 GFLOPs, respectively (2.75%).This cached-context arithmetic excludes prefill, attention matrix products, the output head, drafter operations, and memory traffic; it illustrates scale, not measured latency or a latency bound.Source: §A.4 · p. 15

A.5 Controlled Depth–Latency Profiling

The depth analysis profiles latency under fixed acceptance, then gives a conditional unimodality result for throughput across consecutive drafter depths. The training algorithm excerpt shows horizon sampling and parameter updates.

A.5 Controlled Depth–Latency Profiling

  • The fixed-acceptance profile across depths 1–35 fits relative round latency as 0.9777 + 0.0190d, with R2 = 0.9975 and maximum relative deviation of 1.86%.This supports approximately linear latency over the profiled range; Figure 8 plots the measured ratio and affine fit.Source: §A.5 · p. 15, Figure 8 · p. 16

A.6 Conditional Unimodality over Drafter Depth

  • With fixed conditioning, block size, workload, and execution conditions, throughput is unimodal over consecutive depths if MAT increments diminish and latency increments do not decrease.Plateaus and boundary optima are allowed, and MAT need not increase monotonically; the result does not establish a global optimum across configurations.Source: §A.6 · p. 16
  • The proposition is a search rationale under explicit assumptions, not an unconditional guarantee for measured serving curves or a global optimum across block sizes.Measurements do not establish exact unimodality, and the proposition applies separately at each fixed block size.Source: §A.6 · p. 16

B DScout algorithm

  • The training algorithm samples B ∈ {8, 16, 24, 32} equally, obtains frozen-target outputs and context features, predicts draft tokens, then updates drafter and adapter parameters using the accumulated loss.The excerpt also shows sampled masks and argmax token predictions.Source: B DScout algorithm · p. 17

C Training hyperparameters

The supplied passage identifies a table of DScout training configurations.

C Training hyperparameters

  • Table 6 is titled “DScout training configurations.”The passage gives the table title but no configuration values.Source: Table 6 · p. 18
Loading 2610.23935v1…