PaperlayerView exact revision on arXiv

Source-linked AI summary

DART: Diffusion-Inspired Speculative Decoding for Fast LLM Inference

Fuliang Liu, Xue Li, Ketai Zhao, Yinxi Gao, Ziyan Zhou, Zhonghui Zhang, Zhibin Wang, Wanchun Dou, Sheng Zhong, Chen Tian

arXiv:2601.19278v1cs.CL

TL;DR

Existing autoregressive drafters improve speculative-decoding accuracy but leave drafting latency as a bottleneck. DART uses a lightweight, target-coupled parallel drafter with N-gram-guided tree pruning, achieving broad end-to-end speedups while preserving lossless decoding. Its scope includes the limited-horizon assumptions and evaluation choices stated by the authors.

  • Problem

    Autoregressive draft designs remain a speculative-decoding bottleneck because sequential drafting can consume substantial inference time despite improving predictive accuracy.

  • Method

    DART predicts logits for multiple future positions in parallel from target-model hidden states and uses N-gram-guided tree pruning to construct coherent draft candidates.

  • Results

    2.03×–3.44× throughput improvement is reported across tasks, with DART outperforming EAGLE3 by 30% on average.

  • Takeaways & Limitations

    DART demonstrates a practical speculative-decoding framework that combines low drafting latency with high average acceptance length.

  • Takeaways & Limitations

    DART is designed for a limited drafting horizon, with speculative decoding typically benefiting from predicting only a small number of future tokens because of draft–target distribution mismatch.

Abstract

from arXiv · show

Speculative decoding is an effective and lossless approach for accelerating LLM inference. However, existing widely adopted model-based draft designs, such as EAGLE3, improve accuracy at the cost of multi-step autoregressive inference, resulting in high drafting latency and ultimately rendering the drafting stage itself a performance bottleneck. Inspired by diffusion-based large language models (dLLMs), we propose DART, which leverages parallel generation to reduce drafting latency. DART predicts logits for multiple future masked positions in parallel within a single forward pass based on hidden states of the target model, thereby eliminating autoregressive rollouts in the draft model while preserving a lightweight design. Based on these parallel logit predictions, we further introduce an efficient tree pruning algorithm that constructs high-quality draft token trees with N-gram-enforced semantic continuity. DART substantially reduces draft-stage overhead while preserving high draft accuracy, leading to significantly improved end-to-end decoding speed. Experimental results demonstrate that DART achieves a 2.03x--3.44x wall-clock time speedup across multiple datasets, surpassing EAGLE3 by 30% on average and offering a practical speculative decoding framework. Code is released at https://github.com/fvliang/DART.

1. Introduction

DART addresses the drafting-latency bottleneck in speculative decoding by combining lightweight parallel prediction with N-gram-guided tree pruning. Across benchmarks, it reports substantially faster drafting and end-to-end decoding than prior approaches.

  • Motivation: Existing autoregressive drafters can consume nearly 20%–40% of total inference time, making drafting a bottleneck despite lower per-step cost.Vanilla drafting can exceed 75% and 60% of inference time for Qwen3-14B and Qwen3-32B, respectively, under the stated setup.
  • DART: DART predicts multiple future logits in parallel from target-model hidden states using a single customized layer, eliminating autoregressive draft rollouts.The design also avoids complex KV-cache management for an autoregressive draft model.
  • Results: 6.8× faster drafting forward is reported for DART than autoregressive drafters such as EAGLE3, while retaining relatively high average acceptance length τ.Figure 1 compares DART with EAGLE3 and SPS under different draft-length settings.
  • DART: N-gram-guided tree pruning constrains the combinatorial continuation space while preserving high-quality candidates and improving average acceptance length τ.N-gram functions as a continuity-aware pruning mechanism rather than a standalone draft predictor.
  • Results: 2.03×–3.44× speedup over standard autoregressive decoding is reported across multiple benchmarks, with about 30% higher speedup than EAGLE3 on average.The reported improvement reaches 65% on certain code-centric workloads under the same target-model setting.

2. Preliminaries

Speculative decoding accelerates autoregressive generation by drafting future tokens with a lightweight model and verifying them in parallel with the target model. Its effectiveness depends on accepted-token length and low drafting overhead, while diffusion models motivate parallel prediction.

  • Speculative Decoding: Speculative decoding drafts a block of K future tokens and verifies them in parallel with the target model while preserving the target distribution.The target accepts a contiguous prefix of the draft tokens through a rejection-based procedure.
  • Speculative Decoding: The expected number of accepted draft tokens τ measures speculative-decoding effectiveness and determines the amortized reduction in target-model invocations.Practical drafters therefore seek high τ with minimal drafting overhead.
  • Diffusion Motivation: Diffusion-based language models predict multiple masked positions in parallel through iterative denoising conditioned on surrounding context.This masked, non-autoregressive formulation motivates DART’s parallel prediction design.

3. DART

DART adapts diffusion-style parallel prediction to the causal, short-horizon requirements of speculative decoding. It produces future-position logits in one lightweight pass, trains them with prefix-shared objectives, and prunes candidates into a coherent tree.

  • Requirements: DART retains strictly causal attention over both the prefix and the masked block instead of using bidirectional refinement.This matches speculative decoding’s causal conditioning requirements.
  • Requirements: DART limits its drafting horizon because draft–target distribution mismatch makes predicting a small number of future tokens more accurate.The paper gives 8 as an example of a typical limited drafting horizon.
  • Requirements: Earlier draft positions receive greater practical importance because verification proceeds sequentially and an early error terminates acceptance.The architecture is therefore motivated to prioritize early-token accuracy.
  • Architecture: A single customized Transformer decoder layer combines intermediate target-model representations to produce logits for all future positions simultaneously.The draft model operates on hidden states from multiple target-model layers, and subsequent logits are read from masked positions.
  • Parallel Prediction: Shifted logits prediction makes each position predict the next token, improving first-position accuracy and enabling more efficient training.DART directly predicts multiple future conditional distributions in one step, so drafting cost is independent of draft length d.
  • Training: Prefix-shared masked training jointly predicts future tokens for multiple prefixes using sparse, prefix-isolated attention patterns.Masked positions attend only to their respective prefixes, while within-block attention remains causal and cross-block attention is disabled.
  • Training: The annealed training objective weights the t-th future position by λ_t = γ^(t−1), with γ = 0.6, emphasizing earlier predictions and regularizing noisier distant targets.The weighting is reported to improve draft quality at critical positions for downstream tree construction.
  • Tree Construction: DART outputs parallel logits that define candidate sets for future positions, which are then converted into draft token trees through structured pruning.The logits implicitly contain multiple plausible continuations rather than directly forming a tree.

4. Efficient N-gram-based Tree Pruning

DART converts parallel future-position logits into a compact, semantically coherent draft tree. Its N-gram-guided pruning scores candidate continuations for both model likelihood and contextual continuity while controlling tree size.

  • Implicit Huge Search Space in Parallel Logits: DART prunes the exponentially large token-tree space induced by parallel logits into a compact, high-quality draft tree.Direct verification is infeasible, and independently predicted tokens can form locally implausible sequences.
  • Continuity-Aware Tree Pruning: The algorithm maintains a global token tree while retaining only the top-w scoring candidate sequences at each future position.The inputs are future-position logits, prefix context, tree size, and beam width; the output is the final draft token tree.
  • Continuity-Aware Tree Pruning: The pruning algorithm starts from the prefix context and expands candidate tokens position by position using draft logits.At each position, it ranks top-k candidates and retains the highest-scoring partial sequences within the beam.
  • Continuity-Aware Tree Pruning: An N-gram continuity score evaluates each candidate token against the recent n-token suffix of its partial sequence.This score is combined with the logit-based score during tree expansion.

5. Experiments

Experiments evaluate DART across model scales, datasets, baselines, and ablations using throughput, acceptance length, and drafting latency. DART improves end-to-end speed while substantially reducing drafting overhead, and N-gram pruning improves acceptance length across benchmarks.

  • Main Results: DART achieves 2.03×–3.44× throughput improvement across tasks and outperforms EAGLE3 by 30% on average.The evaluation covers instruction following, conversation, mathematics, and code-generation benchmarks.
  • Main Results: DART surpasses EAGLE3 by 65% on CodeAlpaca when accelerating Qwen3-14B.The paper reports larger gains on code-related benchmarks.
  • Main Results: DART maintains relatively high average acceptance length τ across model scales, with differences from EAGLE3 within 0.2 when DART is lower.The results emphasize drafting overhead as a key determinant of throughput beyond acceptance length alone.
  • Main Results: DART consumes 1.5ms for drafting forward and reduces drafting-forward latency by 6.8× versus EAGLE3 and 53.3× versus SPS on Qwen3-14B.Tree pruning adds 2ms and is reported as more efficient than EAGLE3 tree search.
  • Ablation Study: N-gram pruning consistently improves average acceptance length τ across all evaluated benchmarks.It removes low-quality branches while preserving high-probability continuations.
  • Ablation Study: Shifted logits prediction yields modest but consistent accuracy gains at subsequent positions in both hit@1 and hit@10.The comparison uses Qwen3-4B models trained under otherwise identical configurations.
  • Ablation Study: γ = 0.6 provides the best balance between early-position accuracy and average acceptance length τ.Smaller γ favors earlier positions, whereas larger γ favors longer-horizon predictions.
  • Ablation Study: DART retains larger throughput gains than EAGLE3 as batch size increases, although both methods’ improvements decay.The decay is attributed to greater compute-bound behavior at larger batch sizes.

6. Conclusion

DART combines parallel distribution-level drafting with N-gram-guided tree pruning to reduce draft-stage overhead while preserving coherent draft sequences. The framework achieves end-to-end decoding speedups over prior speculative decoding methods.

  • Conclusion: DART predicts multiple future token distributions in one draft-model forward pass, eliminating autoregressive drafting latency.N-gram-guided tree pruning converts the parallel logits into coherent draft sequences without sacrificing parallelism.
  • Conclusion: DART substantially reduces draft-stage overhead and achieves significant end-to-end decoding speedups over prior speculative decoding methods.The conclusion presents parallel, distribution-level drafting with structured pruning as an alternative to conventional autoregressive draft designs.

Impact Statement

The paper presents work advancing speculative decoding and LLM acceleration.

  • The paper advances speculative decoding and acceleration of large language models.

A. EAGLE3 Latency Analysis

EAGLE3 retains an autoregressive drafting bottleneck: sequential drafting consumes nearly 20%–40% of total inference time. Across Qwen3 variants, Qwen3-32B has lower draft ratio because its verification latency is much higher while drafting latency remains similar.

  • 20%–40% of total inference time is spent on EAGLE3 drafting because its sequential dependency remains autoregressive.This drafting cost fundamentally limits achievable acceleration.
  • Qwen3-32B has a lower draft ratio than other Qwen3 models despite nearly identical EAGLE3 draft-model parameters and latency to Qwen3-14B.Both models have hidden dimension 5120, yielding similar single-layer draft-model costs.
  • Qwen3-32B incurs nearly double the verification latency of Qwen3-14B because it has 64 transformer layers versus 40.With drafting time remaining constant, the larger verification cost reduces the draft ratio.

B. Training Details

DART training uses the open-source SpecForge repository as its implementation base.

  • DART training code is based on SGLang’s open-source SpecForge repository.

B.2. Training Setup

DART training combines a fixed optimization setup, target-model hidden-state inputs, sparse attention implementation, and a large N-gram trie for retrieval. The setup uses shared target-model embeddings and parallelized implementation components.

  • Training hyperparameters: DART training uses context length 6400, draft length 8, AdamW, learning rate 2e-5, gradient clipping 0.5, and three epochs.The learning rate follows cosine annealing with linear warmup.
  • Trainable parameters: The trainable components are an FC layer, decoder layer, LM head, and mask representation, while the target model’s embedding layer is shared and frozen.
  • Training data: DART is trained on approximately 280K filtered ShareGPT and UltraChat examples, using target-model hidden states from selected transformer layers.The data sources match those used by EAGLE3.
  • Attention implementation: DART’s sparse attention implementation uses target hidden states, attention masks, position ids, and draft length to construct the draft computation.The provided Flex Attention code defines the corresponding projections and positional encoding operations.
  • N-gram retrieval: The N-gram trie is built from Dolma 3 Mix with family-specific tokenizers and contains approximately 1.3 billion nodes occupying about 43.5 GB on disk.During inference, the trie is loaded into CPU RAM for shared-prefix child-node retrieval.
  • N-gram retrieval: After warmup, N-gram retrieval latency stabilizes around 6 µs per query.

D. Tree Pruning Algorithm Details

DART’s pruning algorithm builds a compact draft tree by combining draft-logit likelihoods with N-gram continuity, depth weighting, and structural limits. The implementation is parallelized for efficient retrieval and expansion, while larger-scale evaluations report throughput and speedup comparisons.

  • Implementation: The pruning procedure is implemented in C++ and parallelized with OpenMP across batches and candidate expansion.N-gram and worker threads are bound to the same NUMA node to reduce cross-node memory-access overhead.
  • Candidate selection: DART selects the top 25 candidates per future position and retains at most 20 partial sequences at each depth through beam pruning.
  • Tree constraints: The global draft tree is capped at 59 nodes to balance drafting diversity with verification efficiency.
  • Scoring: Each candidate extension receives a combined score from logit likelihood and N-gram continuity.The N-gram score uses the pretrained model’s conditional token probability given the preceding context.
  • Scoring: The combined score weights N-gram continuity by 0.5 and applies a depth-dependent logit weight through the COMBINE function.The logit weight decays with tree depth, while level weighting prioritizes N-gram continuity deeper in the tree.
  • Algorithm outcome: The resulting strategy jointly considers model confidence, token continuity, and structural constraints to construct a compact, high-quality draft tree.
  • Evaluation: DART’s larger-batch evaluation reports throughput and relative speedup on Qwen3-4B and Qwen3-8B, while Table 8 reports speedup ratios and mean average acceptance lengths across seven datasets.
  • Cross-platform evaluation: 30% speedup over EAGLE3 is reported for DART across Qwen3 models on a single NVIDIA A100-40G GPU.The evaluation covers Qwen3-1.7B, 4B, 8B, and 14B.
Loading 2601.19278v1…