PaperlayerView exact revision on arXiv

Source-linked AI summary

When Parallel Drafter Meets Parallel Speculative Decoding

Fuliang Liu, Xue Li, Kun Qian, Zhibin Wang, Wanchun Dou, Wenyuan Yu, Chen Tian

arXiv:2609.27396v1cs.CL

TL;DR

Parallel drafters still leave drafting on the critical path, and existing parallel speculative decoding can revert to serial drafting when its predicted outcome is wrong. DPara precomputes representations for every acceptance boundary during verification and conditions the next draft on the revealed outcome. Across seven benchmarks, it reports average speedups of 3.21× and 3.52× over autoregressive decoding while outperforming the evaluated serial and parallel baselines.

  • Problem

    Parallel drafters leave drafting on the critical path, while existing parallel speculative decoding must predict verification outcomes and can fall back to serial drafting when those predictions miss.

    Source: §1 · pp. 1–2

  • Method

    DPara precomputes draft representations for every acceptance boundary while verification runs, then uses a lightweight autoregressive head to condition on the revealed outcome.

    Source: §1 · p. 2, §3.3 · p. 6

  • Results

    3.21× and 3.52× average speedups over autoregressive decoding on Qwen3-8B and Qwen3-14B, respectively, with DPara leading the evaluated serial and parallel baselines across seven benchmarks.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    DPara’s outcome-independent precomputation hides the dominant backbone cost within verification and achieves state-of-the-art end-to-end decoding latency across the evaluated workloads.

    Source: §5 · p. 9

Abstract

from arXiv · show

DSpark-style parallel drafters have made speculative decoding highly effective, yet their draft phase remains serialized on the critical path of every round. Parallel speculative decoding (PSD) overlaps drafting with verification, yet existing methods must guess the accepted prefix and bonus token in advance: a wrong guess reverts the whole batch to serial drafting. We present DPara, a PSD framework that reuses effective parallel drafters yet guarantees backbone--verification overlap in every round, thereby eliminating this probabilistic fallback altogether. While the target verifies, DPara's diffusion backbone precomputes draft representations for every acceptance boundary with the bonus left unspecified; a lightweight autoregressive head then combines the revealed verification outcome with the matching precomputed representation to emit the next round's draft tokens almost instantly---fully parallelizing the dominant backbone forward with verification and leaving only the negligible head cost serial. Experiments on Qwen3-8B and Qwen3-14B across seven math, coding, and chat benchmarks show that DPara achieves average speedups of $3.21\times$ and $3.52\times$ over autoregressive decoding, surpassing the strongest serial and parallel speculative decoding baselines alike.

1 INTRODUCTION

Parallel drafters reduce drafting cost but leave drafting on the critical path, while existing parallel speculative decoding can revert to serial drafting when predicted outcomes miss. DPara guarantees backbone–verification overlap independent of the verification outcome and reports higher speedups than serial and parallel baselines.

1 INTRODUCTION

  • Existing parallel drafters leave drafting on the critical path, while outcome-dependent parallel speculation can revert the whole batch to serial drafting after a mismatch.At batch size 16, up to 96% of SSD steps fall back.Source: §1 · pp. 1–2
  • DPara precomputes draft representations for every accepted-prefix boundary while verification runs, leaving bonus-token conditioning to a lightweight autoregressive head.This makes backbone–verification overlap independent of the verification outcome and removes prediction-induced fallback.Source: §1 · p. 2
  • M-DFlash adapts the diffusion backbone to multiple feature-less anchors by finetuning from the published DSpark checkpoint, with no new trainable parameters and the AR head frozen.This addresses a regime the standard DFlash model does not satisfy.Source: §1 · pp. 2–3
  • 3.21× and 3.52× average speedups over autoregressive decoding on Qwen3-8B and Qwen3-14B, respectively, with DPara leading the evaluated serial and parallel baselines across seven benchmarks.On Qwen3-14B, its average speedup exceeds PEARL and SSD by 31% and 64%, respectively.Source: §1 · p. 2
  • DPara-A10 retains 98% of DPara’s speedup while cutting drafting-device cost by ∼90%, and DPara-single retains 90%–95% at small batches.These variants explore lower drafting budgets.Source: §1 · pp. 2–3
  • Finetuning only the diffusion backbone raises average acceptance length by 30.7%.Source: §1 · p. 2

2 BACKGROUND AND RELATED WORK

Speculative decoding amortizes target-model work by drafting tokens for verification, and parallel drafters reduce the time needed to generate a draft block. Parallel speculative decoding overlaps drafting with verification, but existing approaches can still fall back to serial speculation when outcomes differ from their predictions.

2.1 SPECULATIVE DECODING

  • Speculative decoding drafts a block for the target model to score, then uses distribution-preserving rejection sampling to retain an accepted prefix and append a bonus token.The bonus is sampled at the first rejection or after full acceptance.Source: §2.1 · p. 3

2.2 PARALLEL DRAFTER

  • Diffusion-inspired parallel drafters predict multiple future positions in one forward pass, while lightweight autoregressive heads in Domino and DSpark restore local causal dependence and longer acceptance lengths.Drafting nevertheless remains sequentially coupled to verification across decoding steps.Source: §2.2 · p. 3

2.3 PARALLEL SPECULATIVE DECODING

  • Parallel speculative decoding overlaps next-step drafting with current verification, but an unanticipated outcome can invalidate the precomputed continuation and trigger serial speculation.DPara precomputes logits for every acceptance boundary and uses the lightweight head with the revealed bonus token to generate the next draft.Source: §2.3 · pp. 3–4

3 DPARA DESIGN

DPara prepares draft predictions for every possible acceptance boundary so its diffusion backbone can run while the target verifies. M-DFlash supplies the required multiple-anchor inputs, and a lightweight autoregressive head uses the actual verification outcome to produce the next draft.

3.1 BACKBONE REQUIREMENT OF DPARA

  • DPara requires predictions from multiple feature-less anchors for every possible accepted-prefix length, which is unknown until verification finishes.The standard DFlash instead predicts from one forwarded token appended to a fully featured prefix; M-DFlash generalizes that single-anchor behavior.Source: §3.1 · p. 4, Figure 5 · p. 5

3.2 FINETUNING DFLASH TO M-DFLASH

  • M-DFlash adapts only the DSpark-initialized backbone to multiple feature-less anchors; the autoregressive head remains frozen and no new trainable parameters are introduced.The target model is also frozen and supplies online features and supervision.Source: §3.2 · pp. 4–5
  • Training minimizes a position-weighted mixture of cross-entropy and full-vocabulary probability distance, with decay γ = 4 and supervision masked to the assistant span.The confidence loss is disabled, so the draft budget d is not dynamically truncated.Source: §3.2 · pp. 5–6

3.3 DPARA ALGORITHM

  • DPara runs M-DFlash alongside verification, precomputes logits for all acceptance boundaries without guessing the accepted prefix or bonus, then uses the matching branch.Only the lightweight autoregressive head remains exposed between rounds; the backbone cost is hidden when it fits within the verification window.Source: §3.3 · p. 6

3.2 FINETUNING DFLASH TO M-DFLASH

  • For d = 3, a shared causal spine and four isolated bidirectional branches compute future logits for every acceptance boundary in one backbone pass.Each branch reads only its corresponding anchor prefix, so its predictions remain valid for that acceptance boundary.Source: §3.2 · p. 5, Figure 6 · p. 6

4 EXPERIMENTS

Across seven math, coding, and chat benchmarks, DPara outperforms the evaluated serial and parallel baselines. Explorations show how its performance and resource requirements behave under batching, heterogeneous drafting, single-GPU colocation, and backbone finetuning.

4.1 EXPERIMENTS SETUP

  • Experiments cover Qwen3-8B and Qwen3-14B across seven math, chat, and coding datasets, with main results measured using batch size 1, greedy decoding, up to 512 new tokens, and draft length 7.Training uses GB200 GPUs; main results use an H800 node, with an additional GPU dedicated to drafting for PSD methods.Source: §4.1 · pp. 6–7, Table 1 · p. 7

4.2 MAIN RESULT

  • DPara averages 3.21× and 3.52× speedups over autoregressive decoding on Qwen3-8B and Qwen3-14B, respectively, and leads every model–dataset pair, tying DSpark on Qwen3-14B Alpaca.On Qwen3-14B, its average speedup exceeds PEARL and SSD by 31% and 64%, respectively.Source: §4.2 · p. 7, Table 1 · p. 7
  • DPara retains average acceptance lengths of 4.43 and 4.41 on Qwen3-8B and Qwen3-14B, below DSpark’s 5.36, yet improves average speedup over DSpark by 11.1% and 5.1%, respectively.Its advantage over DSpark persists at batch sizes 1–16 on both target models and all evaluated benchmarks; at batch 16, speedups over autoregressive decoding remain above 2× in the reported settings.Source: §4.2 · pp. 7–8

4.3 FURTHER EXPLORATIONS OF RESOURCE-CONSTRAINED DRAFTING

  • DPara-A10 retains about 98% of DPara’s speedup across four benchmarks and stays well ahead of DSpark when drafting uses an A10 and the target uses an H800.Figure 7 profiles the round anatomy and compares Qwen3-8B speedups over autoregressive decoding for A10 drafting.Source: Figure 7 · p. 8, §4.3.1 · p. 9

4.3.1 HETEROGENEOUS DRAFTING

  • For heterogeneous drafting, moving the LM head to the target lets the A10 transfer hidden states—roughly 1/30 the size of full-vocabulary logits—while only target-side LM-head execution remains on the critical path.This trades some target compute for reduced network transfer while preserving backbone and hidden-state-transfer overlap with verification.Source: §4.3.1 · p. 9

4.3.2 SINGLE-GPU COLOCATION

  • DPara-single reaches 2.62× speedup over autoregressive decoding at batch size 1 on Qwen3-8B with GB200, retaining 90%–95% of DPara’s speedup at batches 1–4 while halving GPU count.Its advantage over DSpark disappears at batches 8 and 16, where colocation makes decoding compute-bound and the backbone competes for resources and bandwidth.Source: §4.3.2 · p. 9, Figure 8 · p. 9

4.4 ABLATION STUDY

  • With the AR head and decoding configuration unchanged, backbone finetuning raises mean acceptance length τ from 3.62 to 4.73 (30.7%) across four Qwen3-8B benchmarks.τ increases on each benchmark: GSM8K 4.07 to 5.82, MATH-500 4.06 to 5.41, HumanEval 3.81 to 4.78, and Alpaca 2.55 to 2.92.Source: §4.4 · p. 9, Figure 8 · p. 9

5 CONCLUSION

DPara replaces outcome-dependent speculation with outcome-independent precomputation, eliminating prediction-induced serial backbone fallback while hiding the dominant drafting cost during verification. This yields state-of-the-art end-to-end decoding latency across the evaluated math, coding, and chat workloads.

  • DPara decouples backbone execution from bonus-conditioned token generation, eliminating prediction-induced serial backbone fallback and achieving state-of-the-art end-to-end decoding latency across math, coding, and chat workloads.Source: §5 · p. 9

A LATENCY BREAKDOWN

Profiling shows that overlapping the diffusion backbone with verification shortens DPara’s critical path relative to serial drafting. Across batch sizes, DPara remains faster than DSpark, whose higher acceptance length does not offset its longer step at batch size 1.

  • At batch size 1, DPara exposes 1.3 ms (10%) beyond a 12.5 ms verification window, versus DSpark’s 4.4 ms serial draft stage (25% of its step).DPara’s total step is 13.8 ms, compared with DSpark’s 17.7 ms; the DPara step is 1.21× the autoregressive per-token forward.Source: A LATENCY BREAKDOWN · p. 12
  • Across both target models and seven benchmarks, serial draft stages account for 24%–32% of DFlash/DSpark-style drafters’ per-step latency.DFlash’s profiled draft stage is 4.1 ms, or 24% of a 17.3 ms step.Source: A LATENCY BREAKDOWN · p. 12
  • DPara’s step grows from 13.8 to 18.0 ms (+30%) across batch sizes 1–16 and remains fastest; DSpark’s step grows 32% and stays 28%–30% longer.Source: A LATENCY BREAKDOWN · p. 12, Figure 9 · p. 12

B SSD FALLBACK RATE UNDER BATCHING

SSD’s serial re-drafting fallback becomes more frequent as batch size grows because one request’s bonus-token mismatch invalidates the batch’s precomputed drafts. DPara records no such fallback events at any measured batch size.

  • SSD fallback occurs when any request’s revealed bonus token misses its speculation cache, forcing the whole batch to be re-drafted serially.The measured fraction is the share of speculation steps with at least one such call, on Qwen3-14B using the Appendix A configuration.Source: B SSD FALLBACK RATE UNDER BATCHING · p. 12, Figure 10 · p. 13
  • At batch size 16, SSD falls back to serial re-drafting in 57%–96% of steps, while DPara has no such events at any batch size.Source: B SSD FALLBACK RATE UNDER BATCHING · p. 13, Figure 10 · p. 13
  • The per-step fallback probability grows as 1 − (1 − p)^B, where p is per-request bonus-mismatch probability and B is batch size; chat reaches 91%–96% fallback at batch 16.Even math workloads, with 8% mismatch at batch size 1, fall back in over half of steps at batch size 16.Source: B SSD FALLBACK RATE UNDER BATCHING · pp. 12–13

C RESULTS UNDER SAMPLING

Under sampling, DPara retains the highest average speedup on both models and remains fastest on every dataset. Sampling lowers acceptance and average speedups relative to greedy decoding.

C RESULTS UNDER SAMPLING

  • The sampling comparison uses batch size 1, temperature 1.0 for target and drafter, top-p 1.0, draft length 7, and up to 512 new tokens.It uses the same 80 requests per benchmark; speculative-sampling rejection can reject a draft token even when it matches the target’s most likely continuation.Source: C RESULTS UNDER SAMPLING · p. 13
  • Sampling reduces DPara’s average speedups from 3.21× to 2.72× on Qwen3-8B and from 3.52× to 3.13× on Qwen3-14B.Average acceptance also falls, from 4.43 to 4.11 on Qwen3-8B and from 4.41 to 4.09 on Qwen3-14B.Source: C RESULTS UNDER SAMPLING · pp. 13–14
  • 2.72× on Qwen3-8B and 3.13× on Qwen3-14B are DPara’s average speedups under sampling, the highest on both models.DSpark reaches 2.54× and 3.05×, respectively; DPara is fastest on every dataset of both models.Source: C RESULTS UNDER SAMPLING · p. 14

D DEPLOYMENT DETAILS FOR RESOURCE-CONSTRAINED DRAFTING

The resource-constrained deployment moves head and verification work to the target GPU while the A10 computes backbone representations. Its speedup remains close to DPara’s, while single-GPU colocation can lose ground as batch compute demand grows.

D DEPLOYMENT DETAILS FOR RESOURCE-CONSTRAINED DRAFTING

  • DPara-A10 puts only the diffusion backbone on the A10; the H800 handles verification, branch selection, the LM head, and AR-head sampling.The A10 transfers final hidden states rather than full-vocabulary logits, keeping network delay off the critical path.Source: D DEPLOYMENT DETAILS FOR… · p. 14
  • During verification, the A10 runs the next backbone pass and asynchronously sends branch hidden states for the H800 to use when verification returns.The H800 sends the accepted length and target features back so the A10 can advance its drafting context.Source: D DEPLOYMENT DETAILS FOR… · p. 14
  • DPara-A10 retains about 98% of DPara’s speedup on all four benchmarks, averaging 3.14× on Qwen3-8B and 3.43× on Qwen3-14B.For Qwen3-14B GSM8K at batch 1, its step is 14.3 ms versus 13.8 ms for DPara; the backbone and hidden-state push fit within the 12.5 ms verification window.Source: D DEPLOYMENT DETAILS FOR… · p. 14
  • On one GPU, verification and backbone precomputation use separate CUDA streams, but shared compute and memory bandwidth mean logical concurrency may not fully hide backbone latency.As batch size grows, the colocated advantage over DSpark disappears at sizes 8 and 16; separate GPUs avoid direct resource competition.Source: D DEPLOYMENT DETAILS FOR… · pp. 14–15

E TRAINING RECIPE

M-DFlash backbones are finetuned from DSpark checkpoints using target-generated features and supervision. Training uses a shared response set and balances legal anchor counts.

E TRAINING RECIPE

  • M-DFlash backbones are finetuned for one epoch with AdamW, a linear-warmup cosine schedule, learning rate 3e-5, and 4% warmup.The effective global batch is 128, with micro-batch 1 per GPU and gradient accumulation.Source: E TRAINING RECIPE · p. 15
  • Training texts are truncated to 4096 tokens and sampled with up to 256 prediction windows that balance all legal anchor counts k ∈ {1, . . . , d + 1}.The loss follows §3.2.Source: E TRAINING RECIPE · p. 15
  • A shared set of responses generated by Qwen3-8B on Open-PerfectBlend prompts is used to finetune M-DFlash drafters for Qwen3-8B and Qwen3-14B.The corresponding official DSpark checkpoints initialize the backbones; AR heads remain frozen, and target features and supervision come online from the corresponding frozen target model.Source: E TRAINING RECIPE · p. 15

F LLM USAGE

LLM-based tools were used during manuscript preparation for language refinement, while the authors developed the research and remain responsible for the manuscript.

F LLM USAGE

  • LLM-based tools were used solely during manuscript preparation to refine language and improve readability.The authors reviewed the complete manuscript and remain responsible for its content and accuracy.Source: F LLM USAGE · p. 15
  • The authors independently developed the research questions, technical approach, experimental methodology, interpretation of results, and conclusions.Source: F LLM USAGE · p. 15
  • The authors reviewed the complete manuscript and remain fully responsible for its content and accuracy.Source: F LLM USAGE · p. 15
Loading 2609.27396v1…