PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Diffusion Language Models

Tianyi Li, Mingda Chen, Bowei Guo, Zhiqiang Shen

arXiv:2508.10875v3cs.CLcs.AIcs.LG

TL;DR

Diffusion language models seek alternatives to the sequential inference of autoregressive models. This survey synthesizes their paradigms, training, inference, multimodal extensions, applications, and challenges, emphasizing parallel generation and iterative refinement as central capabilities. It also identifies quality, reasoning, and infrastructure limitations that shape future work.

  • Problem

    Autoregressive generation is sequential and limits inference parallelism, while the DLM ecosystem lacks a comprehensive survey covering its models, techniques, and challenges.

  • Method

    The survey organizes DLM foundations, taxonomy, training and post-training, inference optimization, multimodal extensions, applications, and performance comparisons.

  • Results

    DLMs generate multiple tokens in parallel through iterative denoising, while iterative refinement progressively improves uncertain regions and generation quality.

  • Takeaways & Limitations

    DLMs provide a unified research direction for controllable, parallel, and multimodal language generation, with the survey serving as a reference for the field.

  • Takeaways & Limitations

    Parallel decoding can trade generation quality and consistency for speed because simultaneously unmasking tokens increases denoising burden and error accumulation.

Abstract

from arXiv · show

Diffusion Language Models (DLMs) are rapidly emerging as a powerful and promising alternative to the dominant autoregressive (AR) paradigm. By generating tokens in parallel through an iterative denoising process, DLMs possess inherent advantages in reducing inference latency and capturing bidirectional context, thereby enabling fine-grained control over the generation process. While achieving a several-fold speed-up, recent advancements have allowed DLMs to show performance comparable to their autoregressive counterparts, making them a compelling choice for various natural language processing tasks. In this survey, we provide a holistic overview of the current DLM landscape. We trace its evolution and relationship with other paradigms, such as autoregressive and masked language models, and cover both foundational principles and state-of-the-art models. Our work offers an up-to-date, comprehensive taxonomy and an in-depth analysis of current techniques, from pre-training strategies to advanced post-training methods. Another contribution of this survey is a thorough review of DLM inference strategies and optimizations, including improvements in decoding parallelism, caching mechanisms, and generation quality. We also highlight the latest approaches to multimodal extensions of DLMs and delineate their applications across various practical scenarios. Furthermore, our discussion addresses the limitations and challenges of DLMs, including efficiency, long-sequence handling, and infrastructure requirements, while outlining future research directions to sustain progress in this rapidly evolving field. Project GitHub is available at https://github.com/VILA-Lab/Awesome-DLMs.

1 INTRODUCTION

Diffusion language models address the sequential inference bottleneck of autoregressive models through parallel, iterative denoising with bidirectional context and controllable generation. This survey organizes their evolution, training, inference, multimodal extensions, applications, and remaining challenges.

  • Motivation: Autoregressive models generate tokens sequentially, creating a major inference-speed and computational-efficiency bottleneck.
  • Evolution: DLM development spans continuous embedding-space models and discrete token-space models, with early work including Diffusion-LM, SED, D3PM, and DiffusionBERT.
  • DLM Principles: DLMs generate multiple tokens in parallel through iterative denoising, improving inference speed and throughput over autoregressive models.
  • DLM Principles: Bidirectional context gives DLMs richer contextual representations and supports fine-grained control over generation.
  • DLM Principles: Iterative denoising progressively refines uncertain regions while retaining high-confidence tokens, often improving coherence and generation quality.
  • Survey Scope: The survey covers language-modeling paradigms, pre-training and fine-tuning, inference optimization, multimodal models, applications, performance comparisons, and challenges.

2 PARADIGMS OF DIFFUSION LANGUAGE MODELS

This section situates diffusion language models among masked, autoregressive, and hybrid paradigms, emphasizing their bidirectional context, iterative refinement, and parallel generation. It also reviews representative models that combine these principles with improved scalability, controllability, or efficiency.

  • Diffusion Language Models: Diffusion language models achieve bidirectionality through parallel iterative refinement, distinguishing them from models that generate tokens strictly sequentially.The survey categorizes DLMs by diffusion space and contrasts their refinement process with autoregressive generation.
  • Masked Language Models: Masked language models predict randomly masked tokens using bidirectional context but require specialized fine-tuning or decoding for open-ended generation.Their encoder-only denoising objective supports language understanding but does not directly provide general-purpose generation.
  • Autoregressive Language Models: Autoregressive models use causal attention and teacher forcing to generate tokens left to right, making sequential inference a fundamental speed bottleneck.The same sequential formulation supports straightforward text generation while preventing token-level parallelization during inference.
  • Discrete Diffusion Language Models: LLaDA-8B combines masked-language-model bidirectional context with controllable parallel generation and reports performance on par with LLaMA3-8B.This result is presented as evidence that diffusion-based language models can approach strong autoregressive baselines.
  • Discrete Diffusion Language Models: Discrete diffusion research includes denoising-based models, score-based objectives, and adaptations of pretrained masked or autoregressive models.Examples include Diffusion-LLM, Diffusion-NAT, Plaid, SEDD, RADD, and scaling-oriented models such as DFM.
  • Hybrid AR-Diffusion Language Models: Hybrid AR-diffusion models combine autoregressive long-range modeling with parallel diffusion within blocks or across architectural components.These designs support flexible output lengths, KV-cache reuse, and efficient generation while targeting autoregressive-level quality.

3 DLMS: PRE-TRAINING AND POST-TRAINING

DLM pre-training largely adapts procedures from autoregressive and image diffusion models, while post-training develops diffusion-specific methods for reasoning and policy optimization. Key challenges include inefficient loss coverage, training–inference mismatch, and intractable sequence log-likelihoods.

  • 3.1 Pre-training and Supervised Fine-tuning: DLM pre-training generally follows autoregressive procedures for discrete models and image-diffusion procedures for continuous models, while supervised fine-tuning masks or corrupts response segments.Initialization from pretrained AR models can reduce training time and cost while achieving comparable or superior performance.
  • 3.1 Pre-training and Supervised Fine-tuning: DLM scaling laws show greater data hunger under compute constraints but stronger data reuse potential during multi-epoch training than AR models.These tradeoffs inform the design of DLM training regimes.
  • 3.1 Pre-training and Supervised Fine-tuning: Masked DLM training can omit roughly 50% of tokens from loss computation, while training–inference discrepancy can make training performance substantially exceed inference performance.LaViDa duplicates samples with disjoint masks, and other work uses two-step diffusion with improved scheduling to address these issues.
  • 3.2 Post-training for Reasoning: Because DLM generation is iterative and nonsequential, sequence log-likelihood is intractable, complicating reinforcement-learning methods based on policy gradients.Diffusion-of-Thought instead refines intermediate thoughts in parallel, while DCoLT optimizes the full denoising trajectory with outcome-based reinforcement learning.
  • 3.2.2 Adapting Policy Gradient Methods to DLMs: SEPO adapts PPO- and GRPO-style policy gradients to discrete DLMs using importance sampling for stable, low-variance updates with non-differentiable rewards.Its formulation supports both conditional and unconditional generation.
  • 3.2.2 Adapting Policy Gradient Methods to DLMs: DLM-specific policy-gradient methods estimate likelihoods through mean-field factorization, masked-token averaging, complementary masks, or diffusion-native coupled sampling.Coupled-GRPO provides full token coverage and more stable gradients, while improving code generation and encouraging parallel generation.
  • 3.2.2 Adapting Policy Gradient Methods to DLMs: JustGRPO finds that forgoing arbitrary-order generation during reinforcement-learning training can better elicit reasoning in diffusion language models.The result makes generation order a training design choice rather than an assumed advantage.

4 INFERENCE STRATEGIES

DLM inference strategies target generation quality, controllability, and efficiency through parallel decoding, remasking, guidance, caching, and distillation. Adaptive decoding and caching can substantially reduce latency while preserving output quality.

  • 4 INFERENCE STRATEGIES: DLM inference techniques pursue three goals: improving generation quality, enabling finer content control, and increasing efficiency.The surveyed strategies include unmasking and remasking schedules, guidance, KV or feature caching, and step distillation.
  • 4.1 Parallel Decoding: Parallel decoding generates multiple tokens simultaneously, but adaptive confidence-based strategies are needed to balance efficiency against coherence.Fast-dLLM reports up to 27.6× speedups without compromising quality.
  • 4.1 Parallel Decoding: Parallel decoding approaches narrow the latency gap between diffusion and autoregressive models while preserving, and sometimes improving, generation quality.
  • 4.2 Unmasking and Remasking: Mask-predict models iteratively unmask high-confidence tokens and remask uncertain positions, making the unmasking/remasking policy a critical determinant of quality and convergence speed.Policies include low-confidence sampling, random selection, and adaptive temperature.
  • 4.3 Guidance: Guidance modifies the denoising trajectory toward desired conditions, offering a tunable mechanism for prompt-controlled generation and attributes such as topic or sentiment.Its guidance scale balances fidelity to the condition against sample diversity.
  • 4.4 Efficient Inference: Feature caching exploits stable prompt tokens and sparsely changing response tokens, achieving up to 9× speedups with adaptive refresh and up to 34× while preserving fidelity.FreeCache refreshes only dynamic positions, while dLLM-Cache uses prompt and response caches with a value-similarity test.

5 MULTIMODAL AND UNIFIED APPROACHES

DLMs are being extended into unified multimodal systems that jointly handle text and images through discrete diffusion, with architectures targeting bidirectional interaction, controllability, and efficient generation. These approaches span vision-encoder adaptations, modality-agnostic tokenization, parallel reasoning and generation, and fully diffusion-based multimodal modeling.

  • Vision-encoder adaptations: LLaDA-derived models extend diffusion language modeling to multimodal inputs through vision encoders, projector training, and staged visual instruction tuning.LLaDA-V aligns visual features with language-token embeddings using an MLP projector and follows a three-stage tuning strategy.
  • Vision-encoder adaptations: LaViDa addresses multimodal DLM training and inference challenges through a pretrained vision encoder and a two-stage projector-and-model fine-tuning strategy.Its motivation includes the reduced efficiency of masking only about half the tokens for loss computation in masked DLMs.
  • Unified architectures: LLaDA-based systems increasingly support unified generation and understanding, including localized object understanding, image editing, 1024px text-to-image synthesis, and interleaved reasoning.LaViDa-O uses Elastic-MoT to separate lightweight generation from more powerful understanding branches.
  • Unified architectures: MMaDA jointly models text and images with a modality-agnostic diffusion transformer by tokenizing images into discrete VQ-VAE codes without an explicit vision encoder.Mixed long-CoT fine-tuning aligns reasoning formats across modalities, while UniGRPO supports cross-modal reasoning.
  • Parallel multimodal diffusion: MMaDA-Parallel replaces sequential reasoning-then-generation with fully parallel multimodal denoising, jointly generating reasoning traces and visual outputs.Trajectory-level ParaRL further optimizes cross-modal consistency, semantic alignment, and thinking-aware image synthesis.
  • Inference optimization: Dimple improves diffusion inference through confidence-based iteration control, prompt-token caching, and structure priors for response-format and length control.Prompt prefilling achieves up to 7× speedup with minimal performance loss.
  • Fully diffusion-based systems: D-DiT jointly trains continuous image diffusion and discrete masked-token text diffusion, while UniDisc uses masked discrete diffusion for unified text-image modeling.UniDisc supports zero-shot joint image-text inpainting and reports stronger conditional-generation performance, although it is less training-efficient than a comparable AR model.
  • Fully diffusion-based systems: Fudoki is initialized from a pretrained AR-based multimodal model and adapted to discrete flow matching, reducing the cost of training a unified diffusion system from scratch.Its full attention mask is intended to capture global context, while timestep information is inferred from the corrupted input.

6 PERFORMANCE STUDY

Across benchmarks, DLMs generally perform competitively with similarly sized AR models, with particularly strong results in mathematical, scientific, multimodal, and some code-generation settings. The survey also reports evidence that DLMs can be advantageous in data-constrained, multi-epoch training regimes.

  • Evaluation setup: The study compares DLMs and similarly sized AR models across language understanding, mathematical reasoning, code generation, and multimodal benchmarks.Evaluations include PIQA, HellaSwag, HumanEval, GSM8K, GPQA, MATH, GenEval, MME, MMMU, and GQA.
  • Overall comparison: DLMs generally perform competitively with AR models of comparable size across the surveyed benchmarks.The comparison uses performance data primarily from original publications, supplemented by later comparable evaluations when necessary.
  • Language understanding: On PIQA and HellaSwag, LLaDA performs slightly below or on par with AR models such as LLaMA2 and Qwen2.5.This indicates broadly competitive general language understanding rather than consistent superiority in these benchmarks.
  • Reasoning and science: LLaDA and Dream consistently outperform similarly sized AR counterparts on GSM8K, GPQA, and MATH.The reported advantage is concentrated in mathematical and science-related benchmarks.
  • Multimodal and code tasks: MMaDA and LLaDA-V often surpass AR-based multimodal models, while DiffuCoder remains competitive on HumanEval among open-source models.Closed-source DLMs such as Gemini Diffusion and Mercury reportedly rival top-tier AR models including GPT-4o.
  • Implications: Given the limited training data and computational resources used by most current DLMs, the results suggest strong potential as alternatives to AR models in many applications.This conclusion is framed as potential rather than an established replacement across all settings.
  • Scaling behavior: Recent scaling studies report that DLMs tend to outperform AR models in data-constrained, multi-epoch regimes.The passage attributes this pattern to more effective reuse of limited data under an any-order denoising objective.

7 APPLICATIONS ON DOWNSTREAM TASKS

DLMs have been applied across conventional NLP, code generation, molecular and protein design, and multimodal control tasks. These applications exploit diffusion's iterative refinement, parallelism, controllability, or multimodal generation capabilities.

  • Conventional NLP Tasks: DLMs support classification, entity recognition, sentiment analysis, summarization, style transfer, constrained generation, and machine translation.Examples include diffusion over class labels, entity boundaries, summary representations, and controllable text edits.
  • Code Generation: Diffusion-based code models use global planning and iterative refinement for nonsequential code generation.DCoLT treats reverse diffusion as lateral thinking, while DUS improves the speed-quality trade-off through non-adjacent unmasking.
  • Code Generation: Mercury Coder achieves up to 10× higher throughput than speed-optimized autoregressive models while maintaining comparable quality on major code benchmarks.The passage presents this as evidence of the practical potential of DLM inference speed.
  • Biological and Scientific Applications: DLMs are used for molecular optimization, molecular generation, DNA and protein design, and joint generation of compatible protein sequences and structures.DPLM-2 models sequence and 3D structural coordinates jointly, while CFP-GEN integrates functional, sequence, and structural constraints.
  • Multimodal Applications: DLM-based vision-language-action models aim to unify perception, reasoning, and control through visual subgoal and action generation.LLaDA-VLA and dVLA extend diffusion backbones toward simulation and real-world embodied tasks.

8 CHALLENGES AND FUTURE DIRECTIONS

DLMs face a quality–parallelism trade-off, infrastructure gaps, and limited evidence at large scale. Future work targets more efficient training and deployment, low-bit models, compression, unified multimodal reasoning, and agents.

  • Major Challenges: Generating multiple tokens in parallel can reduce coherence because independently sampled positions fail to capture inter-token dependencies.The parallel decoding curse becomes more severe with fewer denoising steps and can produce invalid patterns such as “AAABBA”.
  • Major Challenges: DLMs lack mature ecosystem and serving support comparable to autoregressive models and vLLM.The survey identifies limited native framework support and insufficient open-source deployment infrastructure as practical barriers.
  • Major Challenges: Large-scale DLMs remain relatively underexplored, while some existing models rely on pretrained autoregressive models, baseline DLMs, or limited datasets.The survey states that the scalability and performance of DLMs therefore still require validation.
  • Future Directions: Future research could improve training efficiency through hybrid architectures or training schemes that address DLMs' lower token-use efficiency.The survey also identifies quantization, binarization, pruning, and distillation as largely unexplored opportunities for reducing inference cost and memory use.
  • Future Directions: Unified multimodal reasoning and DLM-based agents are proposed as underexplored directions.These directions build on diffusion models' multimodal generation, bidirectional context modeling, parallel decoding, and iterative refinement capabilities.

9 CONCLUSION

This survey organizes the diffusion language model landscape, compares DLMs with autoregressive models, and reviews training, inference, multimodal extensions, applications, limitations, and future directions. It is intended as a comprehensive reference for understanding the field's current state and prospects.

  • Conclusion: The survey covers DLM principles, taxonomy, modeling paradigms, and comparisons with mainstream autoregressive models.It highlights DLM characteristics and advantages while organizing the field's foundational concepts.
  • Conclusion: It reviews training and inference design spaces, multimodal DLM advances, applications, limitations, and future research directions.The stated goal is to provide researchers with a comprehensive reference on diffusion-based language modeling.
Loading 2508.10875v3…