PaperlayerView exact revision on arXiv

Source-linked AI summary

Domino: Decoupling Causal Modeling from Autoregressive Drafting in Speculative Decoding

Jianuo Huang, Yaojie Zhang, Qituan Zhang, Hao Lin, Hanlin Xu, Linfeng Zhang

arXiv:2605.29707v1cs.CL

TL;DR

Speculative decoding must balance draft quality from causal modeling against the sequential cost of autoregressive drafting. Domino uses a parallel backbone with lightweight causal correction, consistently improving speed over representative baselines on Qwen3 models, reaching 7.92× speedup on GSM8K.

  • Problem

    Speculative decoding lacks a way to retain causal dependency benefits while preserving the low drafting cost of block-parallel generation.

  • Method

    Domino combines a parallel draft backbone with a lightweight head that propagates causal information across draft positions to refine preliminary distributions.

  • Results

    Domino consistently outperforms representative baselines across math, code, and chat benchmarks, achieving up to 7.92× speedup on GSM8K.

  • Takeaways & Limitations

    The results suggest that causal information can be reintroduced into parallel drafting for faster LLM inference.

  • Takeaways & Limitations

    The implementation is mainly adapted to SGLang, and practical speedup may vary across hardware platforms and serving frameworks.

Abstract

from arXiv · show

Speculative decoding accelerates LLM inference by drafting multiple tokens and verifying them in parallel with the target model. However, its practical speedup is constrained by the trade-off between draft quality and drafting cost: autoregressive drafters model causal dependencies among draft tokens but incur sequential overhead, while parallel drafters reduce drafting cost but weaken intra-block dependency modeling. In this paper, we propose Domino, a speculative decoding framework that decouples causal dependency modeling from expensive autoregressive draft execution. Domino first uses a parallel draft backbone to produce preliminary draft distributions for the entire block, and then applies a lightweight Domino head to refine them with prefix-dependent causal information. To stabilize teacher-forced causal encoding, we further introduce a base-anchored training curriculum that first strengthens the parallel backbone and then gradually shifts optimization toward the causally corrected final distribution. Experiments on Qwen3 models show that Domino achieves up to \(5.49\times\) end-to-end speedup under the Transformers backend and up to \(5.8\times\) throughput speedup under SGLang serving.

1 Introduction

Speculative decoding accelerates inference by drafting multiple tokens for parallel verification, but autoregressive drafters trade higher draft quality for sequential cost while parallel drafters weaken causal dependency modeling. Domino decouples these concerns through lightweight causal correction over parallel preliminary draft distributions, improving acceptance and speedup with limited overhead.

  • Motivation: Speculative decoding reduces expensive target-model invocations by proposing multiple tokens for parallel verification, improving throughput while preserving the target model’s output distribution.The approach addresses the sequential, memory-bound nature of standard autoregressive decoding.
  • Quality–cost trade-off: Autoregressive drafting models intra-block causal dependencies and yields longer acceptance, but generating k tokens requires k sequential draft-model forward passes and full-vocabulary projections.This cost grows linearly with draft length and can offset acceptance-length gains.
  • Quality–cost trade-off: EAGLE-3 reaches 4.86 acceptance length and 3.28× speedup, whereas DFlash reaches 4.03 acceptance length and 3.42× speedup through block-parallel drafting.The comparison illustrates the tension between causal draft quality and drafting efficiency.
  • Domino: Domino uses a parallel draft backbone for preliminary block distributions, then applies a causal encoder and low-rank residual correction head to inject prefix-dependent information.This avoids repeated autoregressive draft-model execution while recovering useful intra-block causal dependency.
  • Results: Compared with DFlash, Domino adds 56M parameters (+5.3%) and increases total draft-then-verify latency by 2.8%, while improving average acceptance length by 16.6% and end-to-end speedup by 12.3%.On representative benchmarks, it achieves up to 7.92× speedup on GSM8K, improving over DFlash from 5.21× to 7.92×.

2 Related Work

Related work establishes speculative decoding as a draft-then-verify framework and explores improvements through autoregressive, parallel, and system-oriented drafting methods. These approaches balance draft quality, drafting overhead, and verification efficiency.

  • Speculative Decoding: Speculative decoding uses a draft model to propose candidate tokens and a target model to verify them in parallel.This draft-then-verify design accelerates autoregressive LLM inference.
  • Speculative Decoding: Subsequent work improves speculative decoding through tree-based verification, better serving systems, and more efficient draft-model designs.The resulting speedup depends on both acceptance length and the drafting process.
  • Autoregressive and Efficient Drafting: The EAGLE series drafts tokens sequentially, enabling each token to depend on previous draft tokens and better match the target model’s autoregressive distribution.Other methods reduce drafting overhead using lightweight parallel decoding heads, as in Medusa.
  • Parallel and Non-Autoregressive Drafting: Parallel-drafting methods include diffusion-based drafters and PARD, which predicts multiple future tokens in one draft forward pass.Speculative Diffusion Decoding and DiffuSpec explore diffusion models, while PARD adapts autoregressive models into parallel draft models.

3 Preliminaries

Speculative decoding advances multiple tokens per target-model verification cycle, with speedup governed by acceptance length and drafting cost. Autoregressive drafters model causal dependencies but incur sequential overhead, whereas parallel drafters reduce this overhead through block-level computation.

  • Speculative decoding: Speculative decoding uses a draft model to propose multiple future tokens, which the target model verifies in one forward pass and accepts as a longest valid prefix.Each cycle proposes γ candidate tokens and advances an expected τ tokens, including the target token.
  • Speculative decoding: Speedup depends on increasing acceptance length τ while reducing drafting cost Tdraft, requiring drafters to be both accurate and efficient.Longer accepted prefixes improve target-model progress, while lower drafting overhead preserves the benefit of parallel verification.
  • Autoregressive drafters: Autoregressive drafters condition later draft positions on previously drafted tokens, improving draft quality and acceptance length through explicit causal modeling.Generating γ tokens requires γ sequential draft steps, each combining a draft-model computation and LM-head projection.
  • Autoregressive drafters: Autoregressive drafting cost grows approximately linearly with speculation budget γ, so repeated executions can offset gains from longer accepted prefixes.The overhead can increase further with deeper draft models or larger vocabularies.
  • Parallel drafters: Parallel drafters predict the block-level conditional distribution and generate multiple draft tokens concurrently, incurring draft-model and LM-head costs once per block.This design improves GPU utilization by avoiding γ repeated draft executions.
  • Drafter comparison: Both drafter types use parallel target-model verification, but autoregressive drafters emphasize causal dependency modeling while parallel drafters reduce sequential draft-generation overhead.Their primary difference is how the draft block is generated, not how the target model verifies it.

4 Methodology

Domino combines a parallel draft backbone that produces block-wide preliminary distributions with a lightweight Domino head that injects prefix-dependent causal information through sequential correction. Teacher forcing and a base-anchored curriculum stabilize training while fused implementation reduces rollout overhead.

  • Architecture: Domino has two components: a parallel draft backbone and a Domino head that propagates causal information across draft positions.The backbone computes preliminary distributions for all draft positions in one parallel computation, while the head refines them.
  • Parallel draft backbone: The parallel backbone uses target context features and masked block inputs to produce hidden states for the entire draft block in one non-autoregressive forward pass.Base logits are computed from these representations with the frozen target LM head.
  • Domino head: The Domino head combines a lightweight GRU causal encoder with a low-rank correction head, allowing later positions to use earlier draft tokens without a full autoregressive draft model.The GRU uses hidden dimension 1024, and the correction bottleneck uses rank r = 256.
  • Domino head: Logit-space correction preserves the parallel base LM-head computation while restricting the sequential causal branch to a low-rank residual update.Hidden-space correction would require another full LM-head application after each causal update.
  • Training: Teacher forcing feeds the causal encoder ground-truth token embeddings, aligning training with the accepted-prefix regime of speculative decoding.Self-generated prefixes can be noisy and create input–output mappings absent from the underlying data distribution.
  • Training: The base-anchored curriculum jointly supervises base and final logits while linearly annealing λt from 1 to 0, shifting optimization from backbone learning toward residual correction.This prevents the correction branch from shortcutting the parallel backbone and collapsing its loss.
  • Implementation: 2.64ms to 1.20ms: fused Triton kernels and CUDA Graphs reduce Domino-head latency under the latency setting in Figure 1.The implementation targets kernel-launch and Python-level overhead during rollout.

5 Experiments

Experiments on Qwen3-4B and Qwen3-8B across math, code, and dialogue tasks show that Domino improves speculative-decoding speed and serving throughput by combining causal correction with low-overhead parallel drafting. Ablations attribute these gains to the base-anchored curriculum and lightweight Domino head.

  • Experimental Setup: Domino is evaluated on Qwen3-4B and Qwen3-8B across math reasoning, code generation, and open-ended dialogue benchmarks.The benchmark suite includes GSM8K, MATH, AIME25, HumanEval, MBPP, LiveCodeBench, MT-Bench, and Alpaca.
  • Low-Concurrency Results: 5.49× average speedup is achieved on Qwen3-8B under greedy decoding, improving over DFlash’s 4.66×.On Qwen3-4B, Domino improves the average speedup from 4.70× to 5.47×; under sampling, gains reach 4.61× and 4.46× on Qwen3-4B and Qwen3-8B.
  • High-Concurrency Results: Domino achieves higher SGLang serving throughput than EAGLE-3 and DFlash on both Qwen3-4B and Qwen3-8B.The result indicates that improved draft quality translates into practical throughput while retaining block-parallel drafting’s low overhead.
  • Training Ablation: 3.96 average acceptance length under teacher forcing exceeds 3.80 with training-time testing, while the base-anchored curriculum prevents the parallel backbone from collapsing.The curriculum first strengthens base logits and then shifts optimization toward final logits.
  • Architecture Ablation: 4.19 average acceptance length and 3.31× average speedup with the Domino head exceed 3.49 and 2.84× without causal correction.This ablation identifies lightweight prefix-dependent correction as the key improvement over the parallel backbone alone.

6 Conclusion

Domino improves block-parallel speculative drafting by adding lightweight causal correction without expensive autoregressive execution. Experiments on Qwen3 models show consistent gains in acceptance length and end-to-end speedup over representative baselines.

  • 6 Conclusion: Domino is a speculative decoding framework that improves block-parallel drafting with lightweight causal correction.It decouples causal dependency modeling from expensive autoregressive execution.
  • 6 Conclusion: Domino improves draft quality while maintaining low drafting overhead by decoupling causal dependency from autoregressive execution.
  • 6 Conclusion: Consistent gains in acceptance length and end-to-end speedup are demonstrated on Qwen3 models against representative baselines.

7 Limitations

The work targets inference acceleration, not training or finetuning cost reduction. Its current SGLang-focused implementation, framework compatibility, and deployment speed may require broader evaluation and platform-specific optimization.

  • The work focuses on inference acceleration rather than reducing model training or finetuning costs.
  • The current implementation is mainly adapted to SGLang, while compatibility with other serving frameworks remains unevaluated systematically.
  • Practical speedup can vary across hardware because of differences in memory bandwidth, compute capability, and kernel efficiency.Further platform-specific optimization may be needed for deployment in different environments.

A Appendix · A.1 Training Details

Domino draft modules are trained with frozen Qwen3 target models on regenerated PerfectBlend data, using fixed sequence and block-length limits. The appendix also specifies the training configuration and benchmark resources used for the experiments.

  • A.1 Training Details: DOMINOdraft is trained for both Qwen3-4B and Qwen3-8B while the target model remains frozen.Training uses regenerated PerfectBlend data described in Section 5.
  • A.1 Training Details: Input sequences are truncated to 3072 tokens, and the draft block size is 16.
  • A.1 Training Details: Draft modules are trained for 3 epochs on 8 NVIDIA A100-SXM4-80GB GPUs with a per-GPU batch size of 2.
  • A.1 Training Details: The resulting global batch size is 16 without gradient accumulation.
  • A.1 Training Details: AdamW training uses a learning rate of 6 × 10^-4, zero weight decay, gradient clipping at 1.0, and a cosine schedule with 0.04 warmup ratio.
  • A.1 Training Details: Training uses bfloat16 precision with FSDP and gradient sharding.
  • A.1 Training Details: The appendix lists baseline draft model checkpoints used in the experiments.
  • A.1 Training Details: It also includes full benchmark-level results for a Domino head ablation comparing the causal correction branch disabled versus enabled.
Loading 2605.29707v1…