PaperlayerView exact revision on arXiv

Source-linked AI summary

Sequential Diffusion Language Models

Yangzhou Liu, Yue Cao, Hao Li, Gen Luo, Zhe Chen, Weiyun Wang, Xiaobo Liang, Biqing Qi, Lijun Wu, Changyao Tian, Yanting Zhang, Yuqiang Li, Tong Lu, Yu Qiao, Jifeng Dai, Wenhai Wang

arXiv:2509.24007v1cs.CLcs.LG

TL;DR

Diffusion language models offer parallel-generation efficiency but face fixed decoding lengths, limited KV-cache compatibility, and costly training, while block diffusion retains a fixed block size. The paper introduces NSP and SDLM to adaptively decode variable-length subsequences from pretrained autoregressive models, reporting comparable or better performance with faster inference using only 3.5M training samples.

  • Problem

    Diffusion language models and block diffusion are limited by fixed decoding structure, KV-cache incompatibility, fixed block sizes, and expensive training requirements.

  • Method

    NSP unifies next-token and next-block prediction, while SDLM retrofits pretrained autoregressive models through parallel block training and confidence-based dynamic decoding.

  • Results

    Nearly 2× speedup against Qwen-2.5-32B-SFT is achieved with only 3.5M training data while maintaining comparable performance.

  • Takeaways & Limitations

    SDLM matches SFT-tuned autoregressive models while decoding faster and offers a stronger speed–performance trade-off.

  • Takeaways & Limitations

    Block diffusion and diffusion language models require training from scratch and cannot be easily developed from pretrained autoregressive models, increasing training costs and hindering larger-model development.

Abstract

from arXiv · show

Diffusion language models (DLMs) have strong theoretical efficiency but are limited by fixed-length decoding and incompatibility with key-value (KV) caches. Block diffusion mitigates these issues, yet still enforces a fixed block size and requires expensive training. We introduce Next Sequence Prediction (NSP), which unifies next-token and next-block prediction, enabling the model to adaptively determine the generation length at each step. When the length is fixed to 1, NSP reduces to standard next-token prediction. Building on NSP, we propose Sequential Diffusion Language Model (SDLM), which can retrofit pre-trained autoregressive language models (ALMs) at minimal cost. Specifically, SDLM performs diffusion inference within fixed-size mask blocks, but dynamically decodes consecutive subsequences based on model confidence, thereby preserving KV-cache compatibility and improving robustness to varying uncertainty and semantics across the sequence. Experiments show that SDLM matches or surpasses strong autoregressive baselines using only 3.5M training samples, while achieving 2.1 higher throughput than Qwen-2.5. Notably, the SDLM-32B model delivers even more pronounced efficiency gains, demonstrating the strong scalability potential of our modeling paradigm. Project page and codes: https://github.com/OpenGVLab/SDLM

1 INTRODUCTION

The paper introduces NSP and SDLM to combine diffusion efficiency with autoregressive flexibility while addressing fixed decoding lengths and costly from-scratch training. Experiments report competitive performance against autoregressive models, faster inference, and large-model scalability with limited training data.

  • Motivation: Block Diffusion improves efficiency while preserving autoregressive prediction, but fixed block sizes cannot adapt to varying sequence certainty and semantics.
  • Method: NSP unifies next-token and next-block prediction by dynamically selecting variable-length future sequences according to difficulty and semantics.When the predicted sequence length is always 1, NSP becomes standard next-token prediction.
  • Method: SDLM extends pre-trained autoregressive models using parallel block training and dynamically decodes subsequences within fixed-length blocks.A custom attention mask enables parallel training over multiple future blocks.
  • Results: SDLM achieves on-par performance with existing autoregressive models at 2.1× the speed of Qwen-2.5-3B and outperforms larger diffusion models across multiple benchmarks.
  • Results: Nearly 2× speedup against Qwen-2.5-32B-SFT is achieved with only 3.5M training data while maintaining comparable performance.

2 RELATED WORK

Related work spans autoregressive acceleration through multi-token prediction and speculative decoding, alongside diffusion language models that seek parallel generation. Existing approximate KV-cache methods still incur substantial overhead from padding sequences to maximum length.

  • Autoregressive Acceleration: Multi-token prediction enables parallel future-token prediction and can accelerate autoregressive decoding through speculative validation.DeepSeek-V3 is reported to achieve up to 3× faster inference with this combination.
  • Diffusion Language Models: Diffusion language models achieve competitive language-modeling performance through parallel denoising, with LLaDA scaling to 8B parameters and Dream reducing training costs.
  • Diffusion Language Models: Approximate KV-caching methods for diffusion models still incur substantial computation because each forward pass pads sequences to the maximum sequence length.

3 METHODS

SDLM extends autoregressive next-token prediction to diffusion-based variable-length sequence prediction, using fixed-length masked blocks while dynamically selecting how many tokens to emit. Its training and inference retain causal history, exploit bidirectional within-block attention, and use confidence or verification to balance quality and efficiency.

  • Sequential Diffusion Language Models: NSP generalizes next-token and next-block prediction by allowing each diffusion step to generate a variable-length future sequence.The formulation accommodates dynamic decoding starts and uses shifted prediction to unify the two paradigms.
  • Sequential Diffusion Language Models: SDLM predicts a fixed-length masked block but decodes only an adaptive longest prefix, selecting between 1 and D tokens according to confidence and sequence uncertainty.The adaptive length function extracts contiguous next tokens and repeats prediction from the previous endpoint.
  • Sequential Diffusion Language Models: When the block size is 1, SDLM reduces to the autoregressive paradigm, enabling reuse of pretrained ALM weights and lower training cost.The model’s objective masks the prediction block and minimizes cross-entropy over its masked tokens.
  • Training: Training preserves causal attention over history while applying bidirectional attention within each prediction block, providing the local receptive field needed for dynamic-length decoding.Randomized starting positions train prediction at different locations, and block construction enforces causality between blocks.
  • Training: Parallel block training rearranges target and noise blocks into a structured attention mask with causal prefixes, visible cross-block prefixes, and bidirectional intrablock attention.Multiple noise blocks can be concatenated within the maximum sequence length, and the sparse structure supports efficient attention implementation.
  • Inference: Inference selects the decoding length using either confidence-based stopping or self-speculative verification of parallel predictions.Verification progressively extends sampled-token prefixes, appends a mask at the first unverified position, and uses a second forward pass; confidence can be based on logits or normalized entropy.

4 EXPERIMENTS

Across long-form, multiple-choice, and coding evaluations, SDLM preserves competitive language-model performance while accelerating inference through adaptive multi-token decoding. Experiments also show controllable speed–accuracy trade-offs, scalability to larger blocks and models, and ablation evidence supporting shift prediction and bidirectional block attention.

  • 4.2 MAIN RESULTS: SDLM-32B attains 92.4 on GSM8K, 74.2 on MATH-500, and 78.6 on IFEval, while SDLM-3B matches or surpasses Qwen-2.5-3B-SFT and outperforms larger DLMs.These results come from a single-epoch training run using only 3.5M samples; SDLM-32B remains competitive on coding tasks.
  • 4.2 MAIN RESULTS: SDLM-32B stays within 1 point of its autoregressive counterpart across MMLU, Winogrande, and Hellaswag, while SDLM-3B matches Qwen-2.5-3B-SFT.The reported results indicate that diffusion training preserves the base autoregressive models’ semantic and reasoning abilities during parallel decoding.
  • 4.3 TRADE-OFF BETWEEN SPEEDUP AND PERFORMANCE: As τ decreases, SDLM reaches up to 3.5× speed-up, while MATH-500 accuracy changes from 61.4 to 59.2 and HumanEval+ from 60.4 to 59.8.Code tasks are more sensitive, with HumanEval+ performance remaining high around 1.7 tokens per step; the trade-off applies across GSM8K, MATH-500, and HumanEval+.
  • 4.3 TRADE-OFF BETWEEN SPEEDUP AND PERFORMANCE: D = 8 increases average output from 1.9 to 2.2 tokens per step at τ = 0.98, with only a 1.2-point overall-accuracy drop versus D = 4.Self-speculative decoding accepts 3.4 tokens per step for D = 4 and 5.4 for D = 8, though it incurs additional validation overhead.
  • 4.4 ABLATION STUDY: Removing shift prediction drops HumanEval+ by approximately 14 points, while causal attention reduces average output from 1.88 to 1.82 tokens per step.The reported analysis associates shift prediction with fewer repeated outputs and a stable diffusion starting point; bidirectional attention enlarges the local receptive field and improves parallel-generation efficiency.

5 CONCLUSION

The paper introduces NSP as a unified framework bridging autoregressive and diffusion decoding, then develops SDLMs that adapt pretrained ALMs through parallel block training and dynamic decoding.

  • NSP unifies autoregressive and diffusion decoding.
  • SDLM adapts pretrained ALMs using parallel block training and dynamic decoding.
  • SDLM matches SFT-tuned ALMs in performance while decoding faster, improving the speed–performance trade-off.

A DETAILS OF TRAINING

The training setup uses a 3.5-million-sample corpus and evaluates SDLM across diverse general, mathematics, science, and coding benchmarks.

  • Training hyperparameters for SDLM are reported in Table 4.
  • The training corpus combines six datasets totaling 3.5 million samples and approximately 2.3 billion tokens.
  • General Tasks: Evaluation covers general-language benchmarks including MMLU, Winogrande, Hellaswag, ARC-C/E, and IFEval.
  • Mathematics & Science Tasks: Mathematics and science evaluation uses GSM8K, MATH-500, and GPQA.
  • Coding Tasks: Coding evaluation uses HumanEval, Humaneval+, MBPP, and MBPP+.

B COMPARE WITH MULTI-TOKEN PREDICTION

SDLM parallels MTP by predicting multiple future tokens in one forward pass, but uses local bidirectional attention and extends its horizon without adding output heads.

  • For a prediction horizon of D tokens, SDLM maps the last token and D−1 mask positions to successive future tokens, paralleling MTP’s separate output heads.
  • SDLM jointly uses context across predicted tokens through a local bidirectional attention window, unlike MTP’s isolated heads or left-to-right attention.
  • Extending SDLM’s prediction horizon requires only appending mask tokens, whereas MTP requires additional output heads.
Loading 2509.24007v1…