Source-linked AI summary
DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation
Xin Cheng, Xingkai Yu, Chenze Shao, Jiashi Li, Yunfan Xiong, Yi Qian, Jiaqi Zhu, Shirong Ma, Xiaokang Zhang, Jiasheng Ye, Qinyu Chen, Chengqi Deng, Jiping Yu, Damai Dai, Zhengyan Zhang, Yixuan Wei, Yixuan Tan, Wenkai Yang, Runxin Xu, Yu Wu, Zhean Xu, Xuanyu Wang, Muyang Chen, Rui Tian, Xiao Bi, Zhewen Hao, Shaoyuan Chen, Huanqi Cao, Wentao Zhang, Anyi Xu, Huishuai Zhang, Dongyan Zhao, Wenfeng Liang
TL;DR
Parallel speculative drafters generate long blocks efficiently but lose acceptance at later positions, while verifying every token can waste throughput under high concurrency. DSpark combines semi-autoregressive drafting with confidence-scheduled, hardware-aware verification, improving accepted length across evaluated models and accelerating per-user generation by 60%–85% versus MTP-1 at matched throughput.
Problem
Parallel drafters lack intra-block dependencies, causing later-position acceptance decay, while full-block verification can degrade throughput under high-concurrency workloads.
Method
DSpark couples a parallel backbone with a lightweight sequential module and dynamically schedules verification length using survival probabilities and real-time engine load.
Results
DSpark consistently outperforms Eagle3 and DFlash across evaluated models and domains, improving macro-average accepted length over Eagle3 by 30.9% on Qwen3-4B.
Takeaways & Limitations
In live DeepSeek-V4 traffic, DSpark accelerates per-user generation by 60%–85% versus MTP-1 at matched throughput and unlocks previously unattainable strict-interactivity tiers.
Takeaways & Limitations
Production scheduling must reconcile the algorithm’s smooth-capacity assumption and dynamic-token requirements with discrete hardware capacity and continuous CUDA graph replay.
Abstract
from arXiv · showhide
Speculative decoding accelerates Large Language Model (LLM) inference by decoupling draft generation from target verification. While recent parallel drafters efficiently propose long token sequences in a single forward pass, they suffer from rapid acceptance decay due to a lack of inter-token dependencies. Furthermore, indiscriminately verifying these extended blocks wastes critical batch capacity on tokens with high rejection risks, severely degrading throughput in high-concurrency serving systems. We introduce DSpark, a speculative decoding framework that unifies high-throughput parallel generation with adaptive, load-aware verification. To maintain draft quality, DSpark utilizes a semi-autoregressive architecture, coupling a parallel backbone with a lightweight sequential module, to introduce intra-block dependency modeling and mitigate suffix decay. To optimize system efficiency, DSpark employs confidence-scheduled verification, dynamically tailoring the verification length for each request based on estimated prefix survival probabilities and engine-specific throughput profiles. On offline benchmarks across diverse domains, DSpark substantially improves the accepted length over state-of-the-art autoregressive and parallel drafters. When deployed within the DeepSeek-V4 serving system under live user traffic, DSpark successfully mitigates verification waste. Compared to the established production baseline (MTP-1), DSpark accelerates per-user generation speeds by 60 to 85 percent at matched throughput levels. More importantly, by preventing severe throughput degradation under strict interactivity constraints, it enables performance tiers that were previously unattainable, shifting the Pareto frontier of our serving system.
1. Introduction
DSpark addresses speculative decoding’s draft-quality and verification-efficiency bottlenecks with semi-autoregressive drafting and confidence-scheduled, load-aware verification. It improves accepted length over autoregressive and parallel baselines offline and accelerates per-user generation under matched production throughput.
- Motivation: Autoregressive LLM inference requires a full forward pass for each token, creating latency, low GPU utilization, and high waiting time in latency-sensitive production workloads.Speculative decoding instead drafts candidate blocks and verifies them in one target-model forward pass, accepting the longest consistent prefix.
- Motivation: Parallel drafters reduce sequential drafting overhead but suffer acceptance decay because independent position predictions cannot model within-block dependencies, while verification length remains difficult to optimize.These limitations create coupled generation-quality and system-efficiency bottlenecks.
- Method: DSpark combines a fully parallel draft backbone with a lightweight serial output head that injects local transition information and mitigates suffix decay.The semi-autoregressive design preserves the drafting speed of parallel models while addressing their lack of within-block inter-token dependencies.
- Method: Confidence-scheduled verification uses per-position prefix-survival estimates and real-time engine throughput profiles to tailor each request’s verification length.The hardware-aware scheduler directs target-model verification budget toward tokens with the highest expected return.
- Evaluation: 30.9%, 26.7%, and 30.0% higher macro-average accepted length than Eagle3, and 16.3%, 18.4%, and 18.3% higher than DFlash, respectively, across Qwen3-4B, 8B, and 14B targets.The offline benchmarks span mathematical reasoning, code generation, and daily chat.
- Evaluation: 60%–85% faster per-user generation with V4-Flash and 57%–78% faster with V4-Pro than MTP-1 at matched aggregate throughput capacities.The deployment ran within DeepSeek-V4 under live user traffic.
2. Background
Speculative decoding accelerates autoregressive inference by drafting multiple tokens and verifying them in one target-model pass, but its efficiency depends on the draft–verification tradeoff. Existing drafters are either sequential and dependency-aware or parallel and low-latency, with parallel designs risking verification waste from rejected suffixes.
- Speculative decoding: Speculative decoding uses a lightweight draft model to propose γ tokens, while the target model verifies them in one forward pass and accepts the longest consistent prefix (Chen et al., 2023; Ge et al., 2022; Leviathan et al., 2023).Verification proceeds left to right, so the first rejection discards all subsequent proposed tokens.
- Speculative decoding: Speedup depends on drafting faster, increasing the accepted-token count τ, or reducing effective verification time.The relevant cycle costs are drafting time T_draft, verification time T_verify, and the accepted tokens per cycle τ.
- Autoregressive drafters: Autoregressive drafters condition each token on previous samples, providing strong modeling capacity but imposing T_draft ∝ γ and therefore favoring small blocks and shallow architectures (DeepSeek-AI, 2024; Li et al., 2024b,c, 2026b; Zhang et al., 2025).Tree-based verification is used to compensate for their short blocks.
- Parallel drafters: Parallel drafters generate all γ tokens in one forward pass, making T_draft nearly block-size-independent and enabling larger blocks such as γ=16 (Cai et al., 2024; Chen et al., 2026; Li et al., 2025a; Liu et al., 2026a; Sandler et al., 2026).DFlash uses target-model KV-injected context, bidirectional within-block attention, and a single-pass masked-token design to support deeper architectures and larger blocks.
3. Architecture
DSpark combines semi-autoregressive drafting with confidence-scheduled verification: a parallel backbone supplies efficient block generation, while lightweight sequential conditioning improves dependencies and scheduling avoids low-value verification under load.
- 3.1 Semi-autoregressive generation: DSpark couples a parallel backbone with a lightweight sequential module to preserve efficient draft generation while modeling dependencies among tokens within each block.The parallel stage produces all draft logits in one pass, and the sequential stage conditions each position on previously sampled tokens while remaining much cheaper than the parallel computation.
- 3.1 Semi-autoregressive generation: The sequential module samples drafts left to right using prefix-dependent transition biases, with Markov and recurrent heads providing increasingly rich within-block context.The Markov head uses a low-rank transition matrix, while the RNN head maintains recurrent state summarizing the full prefix history.
- 3.2 Confidence-scheduled verification: Full-block verification can reduce throughput because acceptance varies across domains and unnecessary verification consumes batch capacity, especially in high-concurrency deployments.Structured text tends to have higher acceptance than open-ended chat, and the cost of an extra verification token increases with engine load.
- 3.2 Confidence-scheduled verification: DSpark schedules verification by combining per-position confidence estimates with a hardware-aware prefix scheduler that selects verification lengths according to expected survival and current system load.The confidence head estimates conditional survival probabilities for draft tokens, while the scheduler avoids spending target-model batch capacity on low-confidence suffixes.
- 3.2 Confidence-scheduled verification: Sequential temperature scaling calibrates cumulative prefix-survival probabilities while preserving their ordering, enabling confidence scores to support verification-length decisions.Calibration proceeds from left to right by minimizing Expected Calibration Error for the cumulative product of per-position scores.
4. Experiments
DSpark consistently improves accepted length over autoregressive and parallel drafters across model scales and domains, while semi-autoregressive coupling preserves strong initial predictions and limits suffix decay. Experiments further show that confidence-based pruning addresses domain-dependent verification waste and motivates load-aware scheduling.
- 4.1 Experimental Setup: DSpark is evaluated against DFlash and Eagle3 across Qwen3-{4B, 8B, 14B} and Gemma4-12B on mathematical reasoning, code generation, and daily chat benchmarks.All drafters are retrained under the same training framework, using Open-PerfectBlend’s 1.3 million samples and regenerated target responses.
- 4.2 Main Results: DSpark improves macro-average accepted length over Eagle3 by 30.9%, 26.7%, and 30.0% on Qwen3-4B, 8B, and 14B, respectively, and over DFlash by 16.3%, 18.4%, and 18.3%.The advantage holds across evaluated target models and benchmark domains, including different model families.
- 4.2 Main Results: Accepted length varies by domain, reaching 5.57 on math and 5.12 on code versus 3.49 on open-ended chat for Qwen3-4B, making static verification prone to suffix waste.This predictability gap motivates dynamically pruning draft blocks according to expected acceptance.
- 4.2 Main Results: DSpark combines the high initial acceptance of deep parallel drafting with a lightweight sequential head that maintains stable conditional acceptance throughout the draft block.It starts at 0.93 on Math and mitigates the rapid acceptance decay characteristic of independent parallel generation.
- 4.3 Additional Analyses: Increasing DSpark depth improves accepted length monotonically, while longer proposal lengths widen its advantage over DFlash; static confidence thresholds raise acceptance by pruning likely-rejected suffix tokens.These findings support lightweight sequential modeling and confidence scheduling, while high concurrency makes static thresholds sub-optimal because low-confidence verification consumes batch capacity.
5. Real-World Deployment of DSpark
DSpark is deployed with DeepSeek-V4 using semi-autoregressive draft models, optimized training and asynchronous confidence scheduling to balance verification efficiency with production constraints. Under live serving SLAs, it improves throughput over MTP-1 while adapting verification budgets to traffic load, though fixed draft-side costs remain for low-acceptance queries.
- 5.1 Deployment Configuration: The deployed draft models use three MoE backbone layers, mHC, 128-token sliding-window attention, γ=5 blocks, a Markov head, and an end-to-end calibrated confidence head.They are co-deployed with preview versions of DeepSeek-V4-Flash and DeepSeek-V4-Pro.
- 5.1 Training System: Training efficiency is improved by communicating cached hidden states instead of full-vocabulary logits and packing fixed draft anchors with token-level causal attention indices.These optimizations reduce communication and decouple draft computation from the target context length while preserving exact causal masking.
- 5.2 Production Scheduler: The production scheduler operates asynchronously, using historical predictions for capacity limits while ranking current candidate tokens by up-to-date cumulative confidence.Dynamic top-K selection hides scheduling latency, preserves confidence ordering, and supports global search despite jagged hardware-capacity curves.
- 5.3 Production Results: DSpark improves aggregate throughput by 51% on V4-Flash at 80 tok/s/user and 52% on V4-Pro at 35 tok/s/user over MTP-1.At stricter SLAs, the reported relative advantages reach 406% for V4-Pro, where MTP-1 enters a low-concurrency regime.
- 5.3 Production Results: Under moderate production loads, DSpark expands verification from MTP-1’s static 2 tokens to roughly 4–6 tokens per request, while preserving batch capacity under heavy traffic.The scheduler uses idle compute to verify longer prefixes under light traffic and avoids wasting critical capacity when traffic is heavy.
- Limitations: DSpark retains a fixed draft-side cost for generating the initial γ-token block, which is unrecoverable for complex queries with low acceptance rates.The paper suggests difficulty-aware early exiting as a future optimization.
6. Related Work
Related work spans speculative decoding algorithms, system-aware verification scheduling, and parallel generation architectures. DSpark is distinguished by the requirement that its drafter provide exact per-token probabilities for rejection sampling, which limits several parallel-generation approaches.
- Speculative Decoding Algorithms: Speculative decoding accelerates autoregressive generation by decoupling token proposal from verification, with modern rejection-sampling methods exactly preserving the target distribution.Speedup depends directly on drafter efficiency and accuracy.
- System-Aware Scheduling for Speculative Decoding: System-aware methods adapt speculative lengths using confidence heuristics, learned acceptance predictors, or bandit-style policies.These approaches determine how many tokens to generate or verify in each round.
- Parallel Generation: Parallel-generation models offer decoding latency nearly independent of output length, but independently predicting positions can mix fragments from different valid sequences.Non-Autoregressive Transformers pioneered single-pass prediction of all positions.
- Parallel Generation: Speculative decoding requires exact per-token probabilities for rejection sampling, which iterative refinement, latent marginalization, and global normalization generally cannot readily provide.This probability requirement constrains the applicability of many parallel-generation techniques.
- Parallel Generation: CRF-NAT places a sequential module over parallel hidden states but cannot provide exact per-token probabilities because of its globally normalized partition function.This limitation makes the design closely related to, but unsuitable for, speculative decoding’s rejection-sampling requirement.
7. Conclusion
DSpark addresses structural and system-level bottlenecks in high-concurrency LLM inference through a semi-autoregressive speculative decoding framework. Its generation design combines a parallel backbone with a lightweight sequential head to mitigate suffix decay from independent parallel drafters.
- DSpark targets structural and system-level bottlenecks in large language model inference under high-concurrency production workloads.
- Its semi-autoregressive generation couples a computationally heavy parallel backbone with a lightweight sequential head.
- This architecture is designed to mitigate rapid suffix decay caused by independent parallel drafters.
Appendices · A. Counterexample: Selection Bias Without Early-Stopping
The counterexample shows that offline global search without early-stopping makes draft-token admission depend retrospectively on the sampled token, violating the non-anticipating property required for lossless speculative decoding. This selection bias changes the output distribution, whereas early-stopping preserves causal admission decisions.
- A. Counterexample: Selection Bias Without Early-Stopping: The mechanism arises because the Markov confidence head uses the previously sampled token, making c_2 and the second-prefix survival probability depend on x_1.Evaluating continuation-dependent quantities before committing admissions therefore leaks sampled-token information into the scheduling decision.
- A. Counterexample: Selection Bias Without Early-Stopping: For the two possible x_1 realizations, a high-confidence case with c_2 = 0.9 selects ℓ = 2, whereas a low-confidence case with c_2 = 0 selects ℓ = 0.The length-2 choice is the global maximum in the high-confidence case, while Θ_0 = 1.0 remains maximal in the low-confidence case.
- A. Counterexample: Selection Bias Without Early-Stopping: Without early-stopping, the scheduler’s admission of the first draft token depends on that token’s realized value through continuation confidence.A high-confidence continuation for x_1 yields ℓ = 2 and admission, while a low-confidence continuation yields ℓ = 0 and rejection.
- A. Counterexample: Selection Bias Without Early-Stopping: This retrospective dependence introduces selection bias by favoring tokens that lead to highly confident continuations, even though admission must be decided before observing the token.The admission event for the k-th draft token must use scheduler-visible information available before x_{r,k} is sampled.
- A. Counterexample: Selection Bias Without Early-Stopping: The standard first-position speculative acceptance probability is assumed to be a_1 = 0.8, but retrospective admission routes A through draft acceptance and B through fresh target generation.When x_1 = A, the draft token is admitted and accepted with the stated probability; when x_1 = B, the target model generates a fresh token from p_t.
- A. Counterexample: Selection Bias Without Early-Stopping: Early-stopping prevents the bias by halting immediately when Θ_1 < Θ_0, returning ℓ = 0 before evaluating continuation-dependent quantities such as c_2.The first-position admission decision consequently depends only on pre-token information and cannot be biased by x_1.