PaperlayerView exact revision on arXiv

Source-linked AI summary

From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs

Yuchuan Tian, Yuchen Liang, Shuo Zhang, Yingte Shu, Guangwen Yang, Wei He, Sibo Fang, Tianyu Guo, Kai Han, Chao Xu, Hanting Chen, Xinghao Chen, Yunhe Wang

arXiv:2512.06776v2cs.CLcs.AI

TL;DR

DLMs offer faster parallel generation than AR models, but adapting AR weights leaves the destination and transition strategy unresolved. The paper makes Block-Diffusion the adaptation path, preserving causal context while adding AR guidance and gradual block growth; the resulting NBDIFF-7B reaches state-of-the-art performance among 7B diffusion models.

  • Problem

    Existing AR-to-DLM adaptations leave unresolved where adaptation should end and how to transition effectively, despite the high cost of training large DLMs from scratch.

  • Method

    The method reframes AR as Block-Diffusion with block size 1 and uses context-causal attention, auxiliary AR supervision, and gradual block-size growth.

  • Results

    NBDIFF-7B achieves state-of-the-art performance among 7B-parameter diffusion models, outperforming strong baselines on math, code, and general reasoning benchmarks.

  • Takeaways & Limitations

    Adapting existing open-source AR checkpoints can provide a compute-efficient path to high-quality diffusion LLMs with parallel generation capabilities.

  • Takeaways & Limitations

    Random attention-mask growth can expose early tokens to incomplete and potentially misleading future context, limiting adaptation potential.

Abstract

from arXiv · show

Diffusion Language Models (DLMs) enable fast generation, yet training large DLMs from scratch is costly. As a practical shortcut, adapting off-the-shelf Auto-Regressive (AR) model weights into a DLM could quickly equip the DLM with strong long-context generation capabilies. Prior "adaptation" attempts either modify logits or randomly grow attention masks to Full-Sequence diffusion, or simply transplant AR weights into a Block-Diffusion recipe, leaving two key questions unaddressed: where is the final destination of adaptation, and how to adapt better? For manifold benefits, we reframe the whole AR-to-DLM adaptation under the Block-Diffusion paradigm, transitioning from block size 1 to the final Block-Diffusion state. Concretely, the principled pathway of adaptation is designed as follows: we keep a context-causal path where causal attention is kept in the prefix, an efficient parallel adaptation procedure where an AR guidance is maintained, and gradual increment of the generation block size for a smoother transition. Built on these components, the adaptation is proved competitive on various models at different scales. With better adaptation, we propose NBDiff-7B that could inherit the long-context modeling and reasoning capabilities, and achieve state-of-the-art performance among the 7B-class DLMs. Codes: https://github.com/YuchuanTian/NBDiff.

1 Introduction

The paper motivates DLMs as a faster alternative to sequential AR decoding, then frames AR-to-DLM adaptation as a principled transition toward Block-Diffusion. Its method preserves causal context, uses auxiliary AR guidance, and grows block size gradually, producing competitive NBDIFF-7B results.

  • DLMs reduce sequential decoding dependencies, offering higher throughput and lower latency than token-by-token AR generation.
  • Prior adaptations either modify logits or randomly grow attention masks toward Full-Sequence Diffusion, or transplant AR weights into Block-Diffusion without resolving the adaptation mismatch.
  • The proposed pathway treats AR as Block-Diffusion with block size 1 and transitions toward the target state through causal context, auxiliary AR guidance, and gradual block-size growth.
  • NBDIFF-7B models contexts up to 32K and its Base and Instruct variants outperform strong baselines across general, math, and code benchmarks.

2 Related Work

Related work contrasts Full-Sequence Diffusion, Block-Diffusion, and AR-to-diffusion adaptation approaches. The paper positions Block-Diffusion as an efficient intermediate paradigm combining blockwise causal order with bidirectional within-block generation.

  • Block-Diffusion addresses Full-Sequence Diffusion efficiency by fixing prior context while updating the current block in parallel and reusing cached context.
  • Full-Sequence Diffusion denoises a fully masked sequence with bidirectional attention, whereas Block-Diffusion decodes causal blocks with bidirectional attention inside each block.

3 Rethinking DLM Adaptation from AR: to Where, and How?

The paper argues that Block-Diffusion is a better adaptation destination than Full-Sequence Diffusion because it preserves causal structure while enabling parallel blocks. It examines training stability, inference efficiency, and alignment with AR models.

  • 3.1 Revisiting Previous Adaptation: Randomly growing attention masks can expose early tokens to arbitrary future subsets, creating incomplete or misleading context during adaptation.
  • 3.2 Block-Diffusion and its Advantages: Block-Diffusion is presented as the destination because it lies between AR and Full-Sequence Diffusion, retaining causal order across blocks while allowing bidirectional within-block generation.
  • 3.2 Block-Diffusion and its Advantages: As sequence length increases, Full-Sequence Diffusion shows loss fluctuations whereas Block-Diffusion remains stable, supporting Block-Diffusion for long-sequence training.
  • 3.2 Block-Diffusion and its Advantages: Block-Diffusion reduces inference cost by reusing KV-Cache from fixed previous blocks and processing only the final block being generated.

4 Designing Transition Paradigms

The adaptation pathway moves from AR-style blocksize 1 to Block-Diffusion through Context-Causal attention, parallel AR-guided training, and gradual block growth.

  • 4.1 How Should the Unmasked Context Attend?: Context-Causal attention preserves strict causality in committed context while enabling bidirectional refinement only within the active block.Future blocks remain hidden, preserving left-to-right generation across blocks.
  • 4.1 How Should the Unmasked Context Attend?: Context-Causal adaptation consistently outperforms Block-Causal adaptation in preliminary math and coding experiments.The comparison is reported in Table 1.
  • 4.2 Parallel Training: A concatenated clean sequence and structured attention mask enable all blocks to provide learning signal in one forward pass.The clean branch supplies stable context while the noised branch receives blockwise diffusion supervision.
  • 4.3 AR Loss Guidance: An auxiliary AR loss supervises otherwise unused clean-context predictions and anchors adaptation to AR behavior.The AR objective is combined with the masked diffusion loss, with λ set to 0.5 in practice.
  • 4.4 Gradual Block Growth: Gradual block growth increases the generation blocksize from 1 toward the target, smoothing the transition from next-token prediction to next-block refinement.The curriculum progressively unlocks intra-block bidirectionality and parallel supervision.
  • 4.4 Gradual Block Growth: The full adaptation method and its components are compared across models and settings in Table 2.The table includes component ablations for AR loss and gradual block-size growth.

5 Experiments

Experiments evaluate the adaptation method across multiple pretrained models and show consistent gains over annealed attention masks and plain fine-tuning.

  • Experiment setup: The study tests Qwen3-4B-Base, Qwen3-8B-Base, and openPangu-Embedded-7B using long-sequence adaptation experiments.The common setup uses sequence length ℓ=8k and approximately 30B training tokens.
  • Comparison with existing baselines: Our method consistently outperforms annealed attention masks and plain fine-tuning across model scales and evaluation benchmarks.It achieves the highest average performance on Qwen3-4B, Qwen3-8B, and openPangu-7B, with notable gains on GSM8K, MATH, and MBPP.
  • Contribution of adaptation: Adding AR loss raises Avg from 48.95 to 52.97 (+4%) over plain fine-tuning.The largest gains occur on math and MBPP.
  • Contribution of adaptation: Adding gradual block-size growth further raises Avg to 54.94.The additional gains are especially evident for coding and multi-step reasoning.

6 NBDiff-7B

NBDIFF-7B adapts openPangu-Embedded-7B into a diffusion language model and is evaluated across general knowledge, mathematics, and coding.

  • 6.1 Setup: The model is trained through an 84,000-iteration adaptation stage processing approximately 700B tokens.Training uses sequence length ℓ=8k and global batch size B=1024.
  • 6.1 Setup: Inference generates left-to-right blocks of size 32 while refining tokens bidirectionally within each block.Refinement can use a small-block schedule or a single full-block pass when latency matters.
  • 6.2 Evaluation: NBDIFF-7B-INSTRUCT achieves the highest SFT macro average of 79.9 among diffusion baselines.It leads on several general-knowledge, math, and coding benchmarks.
  • 6.2 Evaluation: NBDIFF-7B-INSTRUCT records MMLU 82.9, MMLU-Pro 71.9, and CMMLU 79.8 on general knowledge.It ranks second on CEval at 72.5 and remains competitive on IFEval at 60.8.
  • 6.2 Evaluation: NBDIFF-7B-INSTRUCT achieves GSM8K 91.0 and state-of-the-art MATH performance of 84.0.These results are reported for instruction-following mathematical reasoning.
  • 6.2 Evaluation: NBDIFF-7B-INSTRUCT tops MBPP at 87.6 and HumanEval at 89.0 among the compared models.The results narrow or reverse gaps favoring autoregressive models in some base-model evaluations.

7 Conclusion

The paper presents a principled AR-to-Block-Diffusion adaptation framework that combines context-causal attention, parallel training with auxiliary AR supervision, and gradual block-size growth. NBDIFF-7B achieves state-of-the-art performance among 7B-parameter diffusion models and outperforms strong baselines across math, code, and general reasoning benchmarks.

  • The framework bridges AR and Block-Diffusion by viewing AR as block size one and smoothly transitioning toward parallel generation.
  • Context-causal attention, auxiliary AR supervision, and efficient parallel training preserve pretrained knowledge during adaptation.
  • NBDIFF-7B achieves state-of-the-art performance among 7B-parameter diffusion models, outperforming strong baselines on math, code, and general reasoning benchmarks.

A Methodology Details

The methodology restructures Block-Diffusion training to model all blockwise conditionals in one transformer call while preserving causal context semantics. Its structured mask separates noised-block attention, earlier clean-block conditioning, and causal clean-context attention, improving token utilization and amortizing computation.

  • One forward pass supplies gradients for masked tokens across all blocks, avoiding the token waste of random cropping and last-block-only supervision.The design targets the data inefficiency of naive Block-Diffusion training, where many sequence tokens do not contribute to the loss.
  • The method concatenates a noised view with the clean sequence and uses a structured 2L × 2L mask to update all representations jointly.This avoids invoking the denoiser separately for each block.
  • Within the noised view, tokens attend only within their own blocks, producing block-diagonal attention.
  • Noised tokens attend only to earlier clean blocks, while clean-context positions retain strict left-to-right causality.These two pathways encode offset block-causal conditioning and context-causal attention.
  • Joint processing amortizes KV-cache construction, maximizes per-step token utilization, and empirically stabilizes training compared with randomly growing global masks.The paper illustrates the setup with sequence length 16 and block size 4, while using block size 32 in practice.

B Other Experiments

NBDIFF-7B-BASE attains the highest macro average among the compared 7B diffusion baselines, with particularly strong general-knowledge and math results. Base initialization provides a stronger overall balance than SFT initialization, which is influenced by HumanEval volatility.

  • NBDIFF-7B-BASE attains the highest macro average, leads several general-knowledge benchmarks, and ranks first on GSM8K and MATH.It leads MMLU-Pro at 52.7, CMMLU at 76.9, CEval at 75.9, BBH at 69.4, GSM8K at 79.6, and MATH at 46.0.
  • NBDIFF-7B-BASE remains competitive on MMLU and is consistently runner-up in coding, slightly behind Dream-v0 but ahead of LLaDA baselines.The reported MMLU score is 69.1, second only to Dream's 69.5.
  • Base-initialized adaptation achieves a stronger overall balance than SFT-initialized adaptation, despite SFT's higher overall average.The reported overall averages are 53.26 for Base and 54.39 for SFT, with SFT's advantage driven largely by HumanEval's small-set volatility.
  • The comparison table evaluates DLMs adapted from Base and SFT weight versions and reports a clear advantage for Context-Causal adaptation from AR.
Loading 2512.06776v2…