Source-linked AI summary
LLaDA2.0: Scaling Up Diffusion Language Models to 100B
Tiwei Bie, Maosong Cao, Kun Chen, Lun Du, Mingliang Gong, Zhuochen Gong, Yanmei Gu, Jiaqi Hu, Zenan Huang, Zhenzhong Lan, Chengxi Li, Chongxuan Li, Jianguo Li, Zehuan Li, Huabin Liu, Lin Liu, Guoshan Lu, Xiaocheng Lu, Yuxin Ma, Jianfeng Tan, Lanning Wei, Ji-Rong Wen, Yipeng Xing, Xiaolu Zhang, Junbo Zhao, Da Zheng, Jun Zhou, Junlin Zhou, Zhanchao Zhou, Liwang Zhu, Yihong Zhuang
TL;DR
Existing diffusion language models have not established whether AR-initialized approaches can scale beyond 30B parameters, while post-training at hundreds-of-billions scale remains underexplored. LLaDA2.0 converts pretrained AR models into diffusion models using progressive training and alignment recipes, producing 16B and 100B instruction-tuned variants. Extensive evaluations report competitive performance with AR counterparts, with LLaDA2.0-flash showing advantages in code generation, mathematical reasoning, and agentic tool use.
Problem
AR-initialized diffusion models have been explored only at 7B–30B parameters, and systematic dLLM post-training at hundreds-of-billions scale remains underexplored.
Method
LLaDA2.0 systematically converts pretrained AR checkpoints through WSD-based diffusion training, checkpoint merging, complementary masking, and SFT/DPO alignment.
Results
LLaDA2.0-mini and LLaDA2.0-flash achieve performance competitive with AR counterparts, while flash shows advantages in code generation, mathematical reasoning, and agentic tool use.
Takeaways & Limitations
The paper validates AR-to-diffusion conversion as a practical route to large-scale dLLMs retaining parallel decoding advantages.
Takeaways & Limitations
Systematic interactions among dLLM SFT, reinforcement learning, and acceleration, and their scaling to hundreds of billions of parameters, remain open.
Abstract
from arXiv · showhide
This paper presents LLaDA2.0 -- a tuple of discrete diffusion large language models (dLLM) scaling up to 100B total parameters through systematic conversion from auto-regressive (AR) models -- establishing a new paradigm for frontier-scale deployment. Instead of costly training from scratch, LLaDA2.0 upholds knowledge inheritance, progressive adaption and efficiency-aware design principle, and seamless converts a pre-trained AR model into dLLM with a novel 3-phase block-level WSD based training scheme: progressive increasing block-size in block diffusion (warm-up), large-scale full-sequence diffusion (stable) and reverting back to compact-size block diffusion (decay). Along with post-training alignment with SFT and DPO, we obtain LLaDA2.0-mini (16B) and LLaDA2.0-flash (100B), two instruction-tuned Mixture-of-Experts (MoE) variants optimized for practical deployment. By preserving the advantages of parallel decoding, these models deliver superior performance and efficiency at the frontier scale. Both models were open-sourced.
1 Introduction
LLaDA2.0 addresses diffusion language models’ scale and training challenges by converting pretrained AR models through progressive continual pre-training and post-training alignment. The resulting 16B and 100B variants retain parallel decoding while targeting practical deployment.
- Motivation: MDLMs support parallel generation and bidirectional context, but existing diffusion models generally remain at 8B parameters or below.The paper identifies scaling diffusion models toward hundreds of billions of parameters as a frontier for practical deployment.
- Approach: LLaDA2.0 systematically converts existing AR checkpoints into diffusion language models, preserving linguistic knowledge while introducing diffusion capabilities.This avoids training the diffusion models entirely from scratch.
- Approach: Warmup–Stable–Decay training progressively bridges the AR-to-diffusion distribution gap while improving continual pre-training efficiency.The strategy addresses the mismatch between left-to-right generation and bidirectional denoising, including blockwise reconstruction’s low data utilization.
- Alignment: Complementary masking improves post-training data utilization by ensuring every token contributes to learning, accelerating convergence.It addresses the partial learning signal caused by random masking during diffusion fine-tuning.
- Results: LLaDA2.0-mini has 16B parameters and LLaDA2.0-flash has 100B parameters, with both optimized for instruction following, safety, and practical deployment.Both instruction-tuned variants retain the parallel decoding advantages of diffusion training.
- Takeaway: The training recipe combines AR stability with diffusion parallelism as a practical approach to efficient large-scale language modeling.The paper presents this combination as a contribution for the broader research community.
2 Related Work
Related work has established diffusion language modeling, AR initialization, block diffusion, post-training, and inference acceleration, but important scaling gaps remain. Existing AR-initialized methods reach only 7B–30B parameters, while systematic post-training at hundreds-of-billions scale remains open.
- Diffusion language models: MDLMs offer an alternative to AR generation, while scratch-trained examples demonstrate competitiveness but remain smaller than leading AR models.The related work contrasts MDLMs’ emerging capabilities with the scale and maturity advantages of AR systems.
- Diffusion language models: Scratch-trained MDLMs typically remain at 8B parameters or below because of data, infrastructure, computational-cost, and training-cycle constraints.These models also lag behind state-of-the-art AR models in overall performance.
- AR initialization: AR-initialized methods use pretrained checkpoints to reduce training costs and narrow the performance gap, including mask annealing and loss reweighting approaches.DiffusionLLaMA and Dream-7B gradually transition attention and rebalance training losses.
- AR initialization: Block diffusion generates tokens within blocks through diffusion while producing blocks autoregressively, enabling variable-length generation and KV-cache reuse.This hybrid design balances diffusion modeling with inference efficiency and can be initialized from AR models.
- Open scaling problem: Existing AR-initialized diffusion methods range from 7B to 30B parameters, leaving larger-scale feasibility and scalability unexplored.The passage also identifies low block-diffusion training efficiency as a barrier to large-scale corpus use.
- Post-training: Post-training research covers SFT, reinforcement learning, and inference acceleration, but their interactions and scaling to hundreds of billions of parameters remain open.The field’s post-training development is described as nascent.
3 LLaDA2.0 Training Paradigm
LLaDA2.0 uses a staged pipeline to transform an AR base model into a diffusion model, then into an aligned assistant. The framework combines token- and block-level denoising, progressive adaptation, and efficiency-aware attention mechanisms.
- Training pipeline: The pipeline has three stages: continual pre-training from AR to MDLM, block diffusion pre-training, and post-training for alignment and specialization.This progression moves from model conversion through block-level modeling to deployment-oriented alignment.
- Continual pre-training: Continual pre-training teaches the AR base model to reconstruct randomly masked tokens bidirectionally while preserving its representational geometry.This phase bridges autoregressive and diffusion-based generation.
- Block diffusion pre-training: Block diffusion pre-training shifts denoising from individual tokens to contiguous text spans, improving computational efficiency and long-range coherence.These spans are referred to as blocks.
- Post-training: Post-training uses SFT and DPO to align outputs with human intent, instruction following, and downstream application requirements.The alignment stage follows token- and block-level non-autoregressive generation training.
- Design principles: The overall design emphasizes knowledge inheritance, progressive adaptation, and efficiency-aware evolution from AR models to fast, flexible diffusion models.The framework is presented as a seamless transformation rather than training a diffusion model from scratch.
- Attention design: The document-level block diffusion mask packs noisy and clean examples into one sequence and combines block-diagonal, offset block-causal, and block-causal masks.This construction enables an efficient vectorized forward pass during the training framework.
4 Continual Pre-training via Warmup-Stable-Decay (WSD)
WSD converts pretrained AR models into diffusion models through progressively changing block size, full-sequence diffusion training, and compact blockwise decay. The section also specifies document-isolated attention and checkpoint merging for efficient, stable training.
- WSD overview: WSD addresses the mismatch between AR objectives and bidirectional diffusion modeling through a smooth three-phase conversion schedule.The strategy is designed to preserve AR priors while adapting the model to diffusion’s structural requirements.
- Warmup: Warmup increases block size from 1 to 4096, transforming block diffusion into full-sequence masked diffusion.The schedule progresses through block sizes 1, 4, 32, 64, and 4096, with sequence lengths divisible by the current block size.
- Stable: Stable training uses the 4096-block MDLM regime to deepen diffusion understanding while reducing attention computation.At this stage, the clean attention component is no longer maintained, improving processing efficiency.
- Decay: Decay reduces block size from 4096 to a compact value such as 32, distilling global contextual knowledge into efficient blockwise inference.Stepwise reduction preserves semantic understanding while recovering KV-cache reuse and fast variable-length generation.
- Training objective: The BDLM objective reconstructs original tokens in masked blocks using cross-entropy over diffusion timesteps and corrupted sequences.Predictions are restricted to masked tokens, with blocks defined by K = Ltotal/LB and block size LB.
- Training stability and efficiency: Document-level attention prevents cross-document contamination, while top-k checkpoint merging averages high-performing states to improve robustness and generalization.The mask supports bidirectional reconstruction within document boundaries; merging is an offline, optimizer-agnostic procedure distinct from EMA.
5 Post-training
The post-training pipeline improves diffusion-model data efficiency, predictive confidence, preference alignment, and practical decoding through complementary masking, CAP training, and adapted DPO.
- Post-training methods: SFT adapts diffusion training to condition response generation on the prompt, prior clean blocks, and the current noisy block being denoised.The loss is computed only on masked tokens within the current noisy block.
- Post-training methods: Complementary masking creates paired inverse-mask instances, exposing every token in its uncorrupted state exactly once and doubling effective data utilization.The pairing also eliminates token-level sampling bias and provides a more uniform learning signal per optimization step.
- Post-training methods: CAP Training adds a confidence loss that selectively sharpens correctly predicted tokens to improve confidence for parallel decoding.Its objective combines the standard SFT loss with an entropy-minimizing auxiliary loss weighted by λ.
- Results: CAP training improves LLaDA2.0-flash decoding efficiency while maintaining competitive compression performance.Figure 3 evaluates average score and tokens-per-forward across 12 benchmarks, alongside inference speed against similarly sized autoregressive models on four code and math benchmarks.
- Post-training methods: DPO replaces intractable exact conditional log-likelihoods with a conditional Block Diffusion ELBO and maximizes the policy’s margin over a frozen reference model.The reference model is initialized from the post-SFT model, and β controls deviation from it.
6 Evaluation
LLaDA2.0 is evaluated across 47 benchmarks, inference settings, and context lengths, showing competitive or superior results in structured tasks and clear quality–speed trade-offs. The models are robust within 32k contexts, while 64k extension incurs a performance cost.
- Benchmark evaluation: The 47-benchmark evaluation compares LLaDA2.0 against strong open-source auto-regressive models across knowledge, reasoning, coding, math, and other dimensions.Inference uses temperature 0.0, block size 32, and decoding threshold 0.95.
- Benchmark evaluation: LLaDA2.0-mini reaches an average score of 64.34 versus Ling-mini-2.0 at 65.77, while scoring 86.50 on SQuAD 2.0, 80.78 on IFEval, and 86.59 on HumanEval.The reported results indicate competitive overall performance alongside strengths in reasoning, instruction following, and coding.
- Benchmark evaluation: LLaDA2.0-flash scores 73.18 versus Qwen3-30B-A3B-Instruct-2507 at 73.60 and exceeds AR peers on HumanEval (94.51), MBPP (88.29), and MultiPL-E (74.87).It also reaches BFCL v3 at 75.43 and AIME 2025 at 60.00.
- Inference hyper-parameters: A denoising threshold of 0.95 gives the highest quality score of 70.15 at 2.55 TPF, whereas 0.85 reaches 3.31 TPF but lowers score to 67.90.The threshold analysis therefore exposes a direct quality–inference-speed trade-off.
- Inference hyper-parameters: Block size 16 scores 70.26 at 2.44 TPF, while block size 32 reaches 2.55 TPF at 70.15; block size 64 degrades both metrics relative to size 32.The authors select block size 32 as the balance for main evaluation because its quality cost is marginal.
- Context-length analysis: On RULER, LLaDA2.0-flash stays above 93 from 4k through 32k contexts, while LLaDA2.0-mini declines from 93.29 at 4k to 83.94 at 32k.Extending to 64k with YaRN scaling factor 2.0 succeeds but degrades performance for both models.
7 Training & Inference Infrastructure
The infrastructure combines distributed parallelism, optimized block-diffusion attention, and a specialized inference engine to scale training and deployment. Reported optimizations improve training efficiency and inference throughput while enabling fair comparison with AR baselines.
- Pretraining infrastructure: Megatron-LM scales the 100B model using data, pipeline, tensor, context, and expert parallelism, broadcasting masked tokens across model-parallel ranks.Masked-token generation occurs on one model-parallel rank to ensure consistency.
- Training efficiency: cuDNN-based block-diffusion attention provides more than 1.3x end-to-end speedup and over 90% attention-layer memory savings versus unfused TransformerEngine attention.The optimization is reported for LLaDA2.0-mini training.
- Training stability: AR-to-diffusion training can suffer gradient explosion at high document mask ratios because masked-token embeddings were never observed during AR training and decay toward zero.The passage identifies numerical stability as a specific conversion challenge.
- Post-training infrastructure: Post-training combines data and expert parallelism through dFactory2 and VeOmni to support scalable, stable fine-tuning with improved data throughput and hardware utilization.The cited infrastructure is designed for complex parallelization schemes during fine-tuning.
- Inference engine: dInfer adapts AR-oriented optimizations, including KV-cache reuse, to block-diffusion inference and reduce prefill computation.The execution pattern of block diffusion closely resembles auto-regressive generation.
- Inference throughput: LLaDA2.0-flash-CAP reaches 535 TPS versus 383 TPS for standard flash and up to 2.1x the 256 TPS and 237 TPS AR baselines.The comparison covers HumanEval, MBPP, GSM8K, and CRUXEval under a consistent generation setup, with diffusion models served by dInfer and AR models by SGLang.
8 Conclusion
LLaDA2.0 scales discrete diffusion language models to 100B parameters through systematic conversion from auto-regressive models. Evaluations support the feasibility of this paradigm, while future work targets larger models, RL/thinking, and faster decoding.
- LLaDA2.0 introduces discrete diffusion language models scaling to 100B total parameters through systematic conversion from auto-regressive models.
- LLaDA2.0-mini and LLaDA2.0-flash achieve performance competitive with their auto-regressive counterparts.
- LLaDA2.0-flash shows apparent advantages in code generation, mathematical reasoning, and agentic tool use.
- Future work may further scale parameter volume, develop RL/thinking approaches, and extend decoding speed.