PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Diffusion Models

Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, Chongxuan Li

arXiv:2502.09992v3cs.CLcs.LG

TL;DR

The paper examines whether core LLM capabilities depend on autoregressive modeling. It introduces LLaDA, a masked diffusion language model trained with pre-training and SFT, and reports competitive scaling and task performance while addressing reversal limitations. The authors conclude that diffusion models are promising alternatives for language modeling at scale.

  • Problem

    Whether scalability, in-context learning, and instruction following require the autoregressive paradigm remains an open question.

  • Method

    LLaDA uses forward random masking and reverse diffusion generation with a Transformer mask predictor, trained through pre-training and supervised fine-tuning.

  • Results

    Across diverse tasks, LLaDA achieves performance comparable to strong LLMs, including LLaMA3, while showing instruction following after SFT and improved reversal-task performance.

  • Takeaways & Limitations

    The results challenge the assumption that essential LLM capabilities are inherently tied to autoregressive models and support diffusion models as a language-modeling paradigm.

  • Takeaways & Limitations

    LLaDA remains smaller in model scale and training data than leading autoregressive counterparts, so further scaling is needed to fully evaluate its capabilities.

Abstract

from arXiv · show

The capabilities of large language models (LLMs) are widely regarded as relying on autoregressive models (ARMs). We challenge this notion by introducing LLaDA, a diffusion model trained from scratch under the pre-training and supervised fine-tuning (SFT) paradigm. LLaDA employs a forward data masking process and a reverse generation process, parameterized by a Transformer to predict masked tokens. It provides a principled generative approach for probabilistic inference by optimizing a likelihood lower bound. Across extensive benchmarks on general tasks, math, code, and so on, LLaDA demonstrates strong scalability and performs comparably to our self-constructed ARM baselines. Remarkably, LLaDA 8B is competitive with strong LLMs like LLaMA3 8B in in-context learning and, after SFT, exhibits impressive instruction-following abilities in case studies such as multi-turn dialogue. Moreover, LLaDA addresses the reversal curse, surpassing GPT-4o in a reversal poem completion task. Our findings show the promise of diffusion models for language modeling at scale and challenge the common assumption that core LLM capabilities discussed above inherently depend on ARMs. Project page and codes: https://ml-gsai.github.io/LLaDA-demo/.

1 Introduction

The paper asks whether core LLM capabilities require autoregressive modeling and proposes LLaDA to test diffusion-based generative modeling as an alternative. At 8B scale, LLaDA is competitive with strong autoregressive models across scalability, in-context learning, instruction following, and reversal tasks.

  • LLMs commonly use autoregressive next-token prediction, but whether this paradigm is necessary for scalability, in-context learning, and instruction following remains unanswered.
  • LLaDA argues that generative modeling principles, rather than autoregressive factorization itself, underpin essential LLM capabilities.
  • LLaDA uses masked diffusion with forward masking and reverse prediction, constructing bidirectional dependencies while optimizing a variational likelihood lower bound.
  • LLaDA scales to 8B parameters and achieves comparable results to same-data autoregressive baselines across six tasks, including MMLU and GSM8K, at a compute budget of 1023 FLOPs.The model was pretrained from scratch on 2.3 trillion tokens using 0.13 million H800 GPU hours, followed by SFT on 4.5 million pairs.
  • LLaDA 8B Base surpasses LLaMA2 7B Base on nearly all 15 zero/few-shot tasks and performs on par with LLaMA3 8B Base.
  • After SFT, LLaDA improves instruction following in multi-turn dialogue and addresses the reversal curse, outperforming GPT-4o in a reversal poem completion task.

2 Approach

LLaDA replaces left-to-right autoregressive modeling with a masked diffusion process for pre-training, supervised fine-tuning, and inference. It masks tokens at variable ratios, predicts masked tokens simultaneously with a bidirectional Transformer, and reverses masking iteratively for generation.

  • Probabilistic formulation: LLaDA defines its distribution through forward masking and a reverse process that gradually recovers masked data as t moves from 1 to 0.Tokens are independently masked with probability t, producing partially masked sequences for intermediate t.
  • Probabilistic formulation: The mask predictor takes a partially masked sequence and predicts all masked tokens simultaneously using cross-entropy computed only on those tokens.
  • Probabilistic formulation: Randomly varying the masking ratio from 0 to 1 gives LLaDA a principled generative objective linked to maximum likelihood, unlike BERT’s fixed ratio and MaskGIT’s heuristic objective.
  • Model architecture: LLaDA uses a non-causal Transformer that can see the entire input when predicting masked tokens.
  • Supervised fine-tuning: During SFT, the prompt remains unchanged while response tokens are independently masked to model the conditional distribution pθ(r0|p0).The LLaDA 8B model uses 4.5 million prompt-response pairs for SFT.
  • Inference: For inference, generation starts from a fully masked response and iteratively predicts all masked tokens, remasking low-confidence predictions as the process moves toward an unmasked sequence.The number of sampling steps and generation length are user-specified hyperparameters that trade efficiency against sample quality.

3 Experiments

Experiments evaluate LLaDA’s scalability, benchmark performance, instruction following, sampling, and reversal reasoning against autoregressive baselines and existing LLMs. LLaDA scales competitively, performs near strong models across diverse tasks, supports instruction-following dialogue, and substantially improves reversal-task performance.

  • 3.1 Scalability of LLaDA on Language Tasks: LLaDA’s performance trend remains highly competitive with ARM baselines across six tasks as pre-training compute increases.The comparison uses pre-training computational cost as a unified scaling metric; LLaDA is especially strong on MMLU and GSM8K, while its PIQA gap narrows with scale.
  • 3.2 Benchmark Results: After 2.3T-token pretraining, LLaDA 8B Base surpasses LLaMA2 7B Base on nearly all tasks and is overall competitive with LLaMA3 8B Base.The evaluation covers general tasks, mathematics, code, and Chinese under protocols aligned with existing studies.
  • 3.2 Benchmark Results: SFT improves LLaDA on most downstream tasks, while some metrics, including MMLU, decline and remain slightly behind LLaMA3 8B Instruct without reinforcement-learning alignment.LLaDA nevertheless demonstrates instruction-following ability in multi-turn dialogue and multilingual responses.
  • 3.3 Reversal Reasoning and Analyses: LLaDA outperforms Qwen 2.5 and GPT-4o by a large margin on reversal tasks while maintaining consistent zero-shot performance across forward and reversal settings.The authors report that no special reversal-task design was used; tokens are treated uniformly without inductive bias.
  • 3 Experiments: Pure diffusion sampling achieves the best overall performance, while LLaDA’s sampling methods offer a quality–speed trade-off.The experiments also examine autoregressive, block-diffusion, and pure-diffusion sampling together with sampling speed and memory consumption.

4 Related Work

Prior language diffusion work established discrete masked diffusion and related acceleration techniques, but large-scale language modeling remained insufficiently verified. This study scales masked diffusion to 8B parameters from scratch and reports performance comparable to leading LLMs.

  • Large-scale language diffusion remained unverified despite diffusion models’ success in vision and growing research interest.
  • Continuous diffusion approaches face scalability challenges; a 1B model may require 64 times an ARM’s compute for comparable performance.
  • Discrete diffusion introduced alternative forward and reverse dynamics, with masked diffusion previously reaching GPT-2-scale perplexity comparable to or surpassing ARMs.
  • LLaDA scales masked diffusion to 8B parameters from scratch and achieves performance comparable to leading LLMs such as LLaMA 3.
  • Prior work also explored architectural optimization, distillation, and sampling design to accelerate masked diffusion sampling.

5 Conclusion and Discussion

The paper presents LLaDA as an 8B diffusion language model with capabilities comparable to strong LLMs while offering bidirectional modeling and addressing limitations associated with autoregressive generation. It also identifies substantial areas for further evaluation and development.

  • LLaDA reaches 8B parameters and demonstrates scalability, in-context learning, and instruction-following performance comparable to strong LLMs such as LLaMA3.
  • The model’s bidirectional modeling and enhanced robustness are presented as advantages over limitations of existing LLMs.
  • LLaDA’s full potential remains unexplored, including adaptive generation length, more efficient sampling, reinforcement-learning alignment, and specialized architectural or system optimizations.
  • LLaDA is smaller and trained on less data than leading ARM counterparts, so further scaling is needed to fully evaluate its capabilities.
  • Multimodal processing, prompt tuning, agent-based integration, and systematic post-training remain insufficiently understood.

A Formulation of Masked Diffusion Models

Masked diffusion trains a Transformer to recover independently masked tokens through a reverse process, defining a bidirectional generative model rather than a left-to-right factorization. Its masked-token cross-entropy objective supports approximate maximum-likelihood training and several inference strategies.

  • A.1 Training: Masked diffusion defines a model distribution differently from autoregressive models by using forward masking and reverse generation.
  • A.1 Training: The forward process independently masks tokens with probability t, progressing from fully observed data at t = 0 to a fully masked sequence at t = 1.
  • A.1 Training: The reverse process starts from fully masked data and factorizes token recovery while predicting original values for masked positions.
  • A.1 Training: The time-free parameterization uses unmasked tokens that remain identical to the original data, so the parametric model need not receive t.
  • A.1 Training: A Transformer mask predictor estimates all masked tokens simultaneously, and the induced t = 0 marginal defines the model distribution.
  • A.1 Training: The masked-token cross-entropy objective is an upper bound on the model distribution’s negative log-likelihood, yielding an approximate maximum-likelihood training framework.
  • A.2 Inference: Uniformly masking exactly l tokens gives the same expectation as independent masking but lower variance, requiring 128 rather than over 1000 Monte Carlo estimates for stable results.
  • A.2 Inference: Inference includes random and low-confidence remasking, while classifier-free guidance is compatible with LLaDA but omitted from main comparisons for fairness.

B.3 Ablation on Classifier-free Guidance

The appendix evaluates classifier-free guidance and sampling strategies for LLaDA 8B. Guidance consistently improves selected benchmark results, while pure diffusion sampling achieves the best overall performance in both reported ablations.

  • Classifier-free guidance consistently improves LLaDA 8B Base performance across six representative benchmarks.The benchmarks are ARC-C, HellaSwag, TruthfulQA, WinoGrande, PIQA, and GPQA.
  • The main results omit classifier-free guidance to ensure fair comparison with autoregressive models.
  • Figure 4 illustrates flexible sampling strategies using colored squares for unmasked tokens and × marks for masked tokens, with block length 4 in the example.
  • Pure diffusion sampling achieves the best overall performance for LLaDA 8B Base and LLaDA 8B Instruct.The Instruct ablation sets the block length to 32 for efficiency.

B.4 Details and Ablation on Sampling Strategies

LLaDA supports multiple sampling strategies after pre-training or SFT, enabling trade-offs between quality and speed. Ablations find pure diffusion and low-confidence remasking strongest overall, while performance is relatively insensitive to generation length.

  • Sampling strategies: LLaDA supports reverse diffusion, autoregressive, and block diffusion sampling without modification or retraining.Block diffusion applies reverse diffusion within blocks and autoregressive sampling across blocks.
  • Sampling results: Pure diffusion sampling achieves the best overall performance, followed by block diffusion LLaDA and standard block diffusion sampling.Block diffusion also consistently outperforms autoregressive sampling, with larger block lengths improving performance.
  • Remasking ablation: Low-confidence remasking consistently outperforms random remasking in pure diffusion sampling.The authors hypothesize that it improves accuracy by reducing generated-sentence diversity, similarly to annealed sampling in autoregressive models.
  • Model-dependent behavior: Autoregressive sampling performs very poorly for LLaDA 8B Instruct because SFT examples are complete sentences.LLaDA 8B Base avoids this issue because pre-training uses truncated documents and random sequence lengths.
  • Block-length ablation: 77.5 to 78.6: setting block length to 8 improves the GSM8K score.
  • Sampling efficiency: Generation quality and speed can be traded off by varying sampling steps, which correspond to decoding 8, 4, 2, or 1 tokens per forward pass.The generation length is fixed at 256 in the efficiency analysis.

B.6 Standard Benchmarks and Evaluation Details

The evaluation covers general, mathematical and scientific, code-generation, and Chinese-understanding abilities using conditional likelihood estimation or conditional generation. The setup differs between base and instruct models and includes memory-consumption comparisons with LLaMA3.

  • Benchmarks: LLaDA is evaluated across general ability, math and science, code generation, and Chinese understanding.The benchmark groups include MMLU, GSM8K, HumanEval, CMMLU, and related tasks.
  • Evaluation process: Evaluation uses conditional likelihood estimation for candidate-answer selection or conditional generation, with accuracy as the evaluation metric where applicable.
  • Model-specific evaluation: The base model uses likelihood estimation for several multiple-choice benchmarks and conditional generation for the remaining tasks, whereas the instruct model uses conditional generation throughout.
  • Memory comparison: LLaDA memory usage is comparable to LLaMA3 without KV Cache and slightly higher than LLaMA3 with KV Cache.Memory is measured in GB under fixed input and output lengths.
  • Likelihood estimation: LLaDA likelihoods are estimated with Monte Carlo sampling: one sample suffices for single-token tasks, while 128 samples stabilize other benchmarks.
  • Generation settings: Conditional-generation settings use pure diffusion with low-confidence remasking, with generation lengths or sampling steps tuned by model and task.For the base model, generation length and sampling steps are 1024; instruct-model settings vary by benchmark.

B.7 Analysis of Sampling Efficiency

The efficiency analysis measures throughput and memory for LLaDA 8B Base against LLaMA3 8B Base. LLaDA offers adjustable quality-speed trade-offs and can match LLaMA3 performance with higher throughput on some tasks, while its study does not target beating autoregressive models in speed.

  • Sampling speed: 1.5 and 1.8 times higher throughput: LLaDA matches LLaMA3 8B Base performance on GSM8K and Math, respectively.LLaMA3 uses KV Cache, whereas LLaDA uses no inference optimization techniques in this comparison.
  • Sampling speed: LLaDA’s sampling steps provide a flexible trade-off between generation quality and speed.The analysis uses fixed 256-token outputs and varies the number of sampling steps.
  • Prompt-length effect: KV-cache acceleration is weaker on HumanEval because its average prompt length is 132 tokens, versus 894 for GSM8K, 680 for Math, and 628 for MBPP.
  • Memory consumption: LLaDA memory remains constant across sampling-step counts and is comparable to LLaMA3 without KV Cache, but slightly higher with KV Cache.
  • Scope: The study’s goal is to demonstrate scalable language diffusion rather than propose a model faster than autoregressive models.Efficiency-oriented algorithmic and architectural improvements are left for future work.

B.8 Evaluation on iGSM Dataset

The iGSM evaluation tests LLaDA on synthetic, rule-generated GSM8K-like problems with varying solution-step difficulty. LLaDA 8B Base shows consistent advantages over LLaMA3 8B Base on unseen mathematical problems, matching the paper’s broader mathematical results.

  • Dataset: iGSM is an infinite synthetic GSM8K-like dataset whose difficulty is controlled by the number of solution steps.The evaluation appends “#### $answer” to match the GSM8K format.
  • Evaluation setup: The evaluation uses four-shot question-answer examples and a system prompt because iGSM differs slightly from GSM8K, including its mod 5 algorithmic system.
  • Results: For solution steps ranging from 4 to 6, LLaDA 8B Base demonstrates significant and consistent advantages over LLaMA3 8B Base on unseen mathematical problems.The results align with the comparison reported on GSM8K-like benchmarks in Table 1.

C Impact Statement

The paper highlights diffusion language models’ potential applications while noting that they raise societal concerns similar to traditional LLMs. These concerns include environmental impact, harmful-content misuse, and bias amplification.

  • Diffusion models may support applications in conversational AI, code generation, and complex reasoning tasks.
  • Large-scale training creates environmental concerns that require attention for responsible deployment.
  • Diffusion language models could be misused to generate harmful content.
  • Training data biases may be amplified by diffusion language models.
  • Responsible development and deployment requires addressing these societal challenges.
Loading 2502.09992v3…