PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffusion Models: A Comprehensive Survey of Methods and Applications

Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, Ming-Hsuan Yang

arXiv:2209.00796v15cs.LGcs.AIcs.CV

TL;DR

The rapid growth of diffusion-model research makes the field increasingly challenging to navigate. This survey organizes recent work around efficient sampling, likelihood maximization, and structured data, while identifying model combinations as a route toward stronger performance.

  • Problem

    The rapid growth of diffusion-model research makes the expanding body of work increasingly challenging to navigate.

  • Method

    The survey presents a taxonomy organized around efficient sampling, improved likelihood estimation, and methods for handling data with special structures.

  • Results

    The survey highlights sampling efficiency, likelihood maximization, and handling structured data as three major directions for improving diffusion models.

  • Takeaways & Limitations

    Combining diffusion models with other generative models has the potential to achieve stronger performance.

  • Takeaways & Limitations

    ODE-based stochastic counterparts can incur slightly inferior sample quality.

Abstract

from arXiv · show

Diffusion models have emerged as a powerful new family of deep generative models with record-breaking performance in many applications, including image synthesis, video generation, and molecule design. In this survey, we provide an overview of the rapidly expanding body of work on diffusion models, categorizing the research into three key areas: efficient sampling, improved likelihood estimation, and handling data with special structures. We also discuss the potential for combining diffusion models with other generative models for enhanced results. We further review the wide-ranging applications of diffusion models in fields spanning from computer vision, natural language generation, temporal data modeling, to interdisciplinary applications in other scientific disciplines. This survey aims to provide a contextualized, in-depth look at the state of diffusion models, identifying the key areas of focus and pointing to potential areas for further exploration. Github: https://github.com/YangLing0818/Diffusion-Models-Papers-Survey-Taxonomy.

1 INTRODUCTION

This survey organizes rapidly expanding diffusion-model research into core methodological directions, connections with other generative models, and broad applications. It also provides foundations, a taxonomy, and future research directions.

  • Diffusion models have emerged as a new state-of-the-art family of deep generative models, challenging the previous dominance of GANs in image synthesis.
  • The survey provides a self-contained foundation covering DDPMs, SGMs, and Score SDEs under a shared diffusion principle.These approaches progressively add random noise to data and then remove noise to generate samples.
  • Its taxonomy centers on efficient sampling, improved likelihood estimation, and methods for data with special structures such as relational, invariant, and manifold data.The survey further divides each category into more detailed subcategories, illustrated in Figure 1.
  • The survey examines connections between diffusion models and VAEs, GANs, normalizing flows, autoregressive models, and EBMs, noting potential for stronger performance through combinations.
  • It reviews applications across computer vision, natural language processing, temporal data modeling, multimodal learning, robust learning, and interdisciplinary fields.For each task, it defines the problem, describes diffusion-model use, and summarizes relevant prior work.
  • The paper concludes with an outlook on future directions for diffusion-model research.

2 FOUNDATIONS OF DIFFUSION MODELS

Diffusion models generate data by learning to reverse a progressive noise-injection process. The survey presents DDPMs, SGMs, and Score SDEs as connected formulations with shared denoising principles.

  • Core intuition: Diffusion models progressively inject noise into data, then learn a reverse process that generates samples from noise.The reverse process starts from a prior noise vector and iteratively removes noise through learned transitions.
  • Three formulations: Research on diffusion models is organized around DDPMs, score-based generative models, and stochastic differential equations.The survey introduces these three formulations and discusses their connections.
  • DDPMs: DDPMs use a hand-designed forward Markov chain to transform data toward a tractable prior and a learnable reverse chain to reconstruct data.Gaussian perturbation is a common forward transition, while deep networks parameterize the reverse transition mean and variance.
  • DDPMs: DDPM sampling begins with a prior noise vector and repeatedly samples reverse transitions until a data sample is produced.Training adjusts the reverse chain to approximate the time reversal of the forward process by minimizing a KL divergence.
  • SGMs: Score-based models estimate score functions across noise levels and generate samples by chaining score-based samplers as noise decreases.The score is the gradient of log density with respect to data, pointing toward increasing probability density.
  • Connections: DDPM and SGM training objectives are equivalent under a parameter mapping, while reverse-time SDEs and probability-flow ODEs can share data-distribution marginals.The equivalence follows by setting 𝝐𝜃(x,𝑡) = −𝜎𝑡s𝜃(x,𝑡).

3 DIFFUSION MODELS WITH EFFICIENT SAMPLING

Efficient-sampling research reduces diffusion models’ many iterative evaluation steps through learning-free and learning-based methods. The main trade-offs involve sampling speed, discretization error, and sample quality.

  • Overview: Efficient sampling methods are classified as learning-free or learning-based approaches that reduce evaluation steps while targeting sample quality.Sampling cost increases with the number of discretized time steps.
  • SDE solvers: Discretization schemes seek fewer time steps while minimizing errors when solving reverse-time SDEs or probability-flow ODEs.DDPM generation can be viewed as a particular discretization of the reverse-time SDE.
  • SDE solvers: Reverse diffusion is a numerical solver for reverse-time SDEs and can apply to any forward SDE, with slightly better empirical performance than DDPM for VP-SDEs.The method uses a trained noise-conditional score model in its reverse discretization.
  • SDE solvers: Predictor-corrector sampling combines an SDE solver with score-based MCMC, and Langevin correctors are reported as more efficient than adding another predictor.The corrector adjusts the sample’s marginal distribution after the predictor produces a coarse sample.
  • ODE solvers: Heun’s second-order method offers a quality–speed trade-off by reducing discretization error at the cost of one additional score evaluation per step.It produces quality comparable to or better than Euler’s method with fewer sampling steps.
  • Knowledge distillation: Progressive Distillation compresses the full sampling process into a sampler requiring half as many steps, though repeated reduction can lower sample quality.The distilled sampler is trained to match the input and output of DDIM sampling.

4 DIFFUSION MODELS WITH IMPROVED LIKELIHOOD

The survey reviews methods for improving diffusion-model likelihoods by optimizing noise schedules, learning reverse variances, and computing likelihoods through continuous-time formulations. These approaches seek tighter variational bounds or higher likelihood values, while exact probability-flow likelihood optimization can be computationally expensive.

  • The survey organizes likelihood-improvement methods around noise schedule optimization, reverse variance learning, and exact log-likelihood computation.
  • Noise Schedule Optimization: iDDPM reports that a cosine noise schedule can improve log-likelihoods, alongside a reverse-variance interpolation between β_t and 1 − ᾱ_t.The interpolation is performed in the log domain.
  • Noise Schedule Optimization: Noise schedules with identical endpoint signal-to-noise ratios leave the VLB unchanged and affect only Monte Carlo estimator variance.The schedule endpoints are Rmin and Rmax.
  • Reverse Variance Learning: Analytic-DPM derives optimal reverse variances from first- and second-order moments of a pretrained score model, producing tighter VLBs and higher likelihood values.
  • Exact Likelihood Computation: Score SDEs provide efficiently computable variational bounds for likelihood training, while ScoreFlows maximize a variational lower bound instead of directly optimizing probability-flow likelihoods.Higher-order score matching has also been reported to improve probability-flow likelihoods.
  • Exact Likelihood Computation: Probability-flow likelihoods can be evaluated with numerical ODE solvers and trace estimation, but direct optimization requires expensive ODE solves for each data point.

5 DIFFUSION MODELS FOR DATA WITH SPECIAL STRUCTURES

The survey describes adaptations that make diffusion models suitable for discrete, invariant, graph, molecular, and manifold-structured data. These methods replace continuous Gaussian assumptions or impose symmetry and geometric constraints through specialized processes, architectures, and objectives.

  • Continuous-domain diffusion models require adaptation for discrete data, low-dimensional manifolds, and other domains with special structures.
  • Discrete Data: Discrete diffusion methods replace Gaussian perturbations with masking, random walks, absorbing-state kernels, or discretized Gaussian kernels.D3PM uses absorbing-state or discretized Gaussian transition kernels, while VQ-Diffusion uses random walks or masking.
  • Discrete Data: Cross-modal information and masked autoencoder architectures extend discrete diffusion to complex discrete sequences and multimodal discrete data generation.
  • Discrete Data: Continuous-time discrete diffusion models use continuous-time Markov chains to derive efficient samplers that outperform discrete counterparts and analyze sample-distribution error.
  • Invariant Data: Permutation-equivariant networks and message passing enable diffusion models to generate permutation-invariant graphs, including joint node-and-edge distributions.
  • Invariant Data: If the prior and transition kernels share rotation and translation invariance, the generated molecular-conformation distribution is guaranteed to be invariant to those transformations.
  • Manifold Data: Riemannian score-based and variational diffusion models extend diffusion to manifolds, while latent-space models use intrinsic structure for faster sampling and discrete-data handling.

6 CONNECTIONS WITH OTHER GENERATIVE MODELS

This section connects diffusion models with other generative-model families, showing how hybrid designs combine their structures, objectives, or guidance mechanisms. It also highlights integrations with language models for more controllable and semantically coherent generation.

  • Several hybrid approaches report stronger generation outcomes, including state-of-the-art compositional synthesis and autoregressive models retaining reasonable likelihoods.The survey also describes language-model and diffusion-model collaboration as producing more controllable and semantically coherent outputs.
  • Large Language Models and Connections with Diffusion Models: LLM-guided diffusion uses language-model planning or reasoning to guide detailed generation and improve compositional synthesis.Examples include temporal prompt decomposition, multimodal discrete-sequence generation, action-sequence sampling, and spatial layout planning.
  • Variational Autoencoders and Connections with Diffusion Models: Diffusion models connect with VAEs through latent-space diffusion and hierarchical-VAE interpretations of their forward and reverse processes.The DDPM forward process acts as an encoder, while its reverse process acts as a shared decoder across multiple steps.
  • Other Generative Models: Diffusion-enhanced GANs, normalizing flows, autoregressive models, and EBMs modify noise injection, flow processes, training objectives, or likelihood learning.Reported outcomes include sharper DiffFlow boundaries, fewer discretization steps, improved likelihood and sampling speed, parallel ARDM generation, and stronger autoregressive sample quality.

7 APPLICATIONS OF DIFFUSION MODELS

The survey organizes diffusion-model applications across six categories and describes conditioning mechanisms that support controllable generation. Reported applications span vision, language, multimodal generation, 3D synthesis, motion, and video.

  • Applications are grouped into computer vision, natural language processing, temporal data modeling, multi-modal learning, robust learning, and interdisciplinary applications.The survey introduces each category and summarizes representative diffusion-model applications in Table 3.
  • Conditioning Mechanisms: Conditional diffusion models use labels, classifiers, texts, images, semantic maps, and graphs to control generation.The survey discusses concatenation, gradient-based guidance, cross-attention, and adaptive layer normalization as conditioning mechanisms.
  • Computer Vision and Multimodal Learning: Diffusion applications include image restoration and editing, semantic segmentation, multimodal translation, scene-graph generation, and subject-specific image synthesis.Examples include super-resolution, inpainting, few-shot segmentation, unified image-text generation, and DreamBooth subject recontextualization.
  • Text and Multimodal Generation: RPG outperforms all SOTA methods, such as SDXL and DALL-E 3, in the demonstrated compositional text-to-image setting.The framework uses multimodal-LLM reasoning to enhance compositionality and generalizes to different MLLM architectures.
  • 3D, Motion, and Video: Diffusion models support controllable 3D-object synthesis, text-conditioned motion generation and editing, and temporal-consistent video generation.Reported systems include controllable high-quality 3D objects, variable-length motion handling, and video generation or editing without additional fine-tuning.

7.5 Temporal Data Modeling

Diffusion models are applied to temporal data through imputation, forecasting, waveform synthesis, robustification, and molecular or protein generation. Methods adapt diffusion to temporal structure, retrieval guidance, and geometric constraints.

  • Time-Series Imputation and Forecasting: CSDI uses self-supervised training to exploit temporal correlations and reports superiority over previous methods on some real-world datasets.SSSD combines conditional diffusion with structured state-space models to capture long-term dependencies.
  • Temporal-data applications include time-series imputation, forecasting, waveform generation, adversarial purification, and molecular or protein design.The section covers both temporal modeling and interdisciplinary structured-generation tasks.
  • Time-Series Imputation and Forecasting: RATD retrieves corresponding historical references and uses them as guidance during the denoising process for complex time-series forecasting.The historical series is concatenated with noise as the main model input, while retrieved references guide denoising.
  • Waveform Generation: Waveform diffusion models iteratively refine noisy signals, trading inference speed for sample quality through the number of refinement steps.DiffWave is non-autoregressive and produces high-fidelity audio across waveform-generation tasks.
  • Molecule and Protein Generation: Geometric diffusion methods incorporate equivariance or intrinsic coordinates for molecule and protein generation, including torsion-angle diffusion and equivariant Markov kernels.IPDiff reports better binding-related metrics and molecular properties than previous diffusion-based and autoregressive models.

8 FUTURE DIRECTIONS

The survey identifies diffusion-model research as still early, with opportunities to improve theory and empirical performance. Future work centers on sampling, likelihoods, structured data, hybrid generative models, and applications.

  • Diffusion-model research remains in its early stages, leaving substantial room for theoretical and empirical improvement.
  • Key directions include efficient sampling, improved likelihood estimation, and methods for handling special data structures.
  • The survey also points to combining diffusion models with other generative models and tailoring them to a broad range of applications.

Revisiting Assumptions.

The survey revisits the assumption that diffusion’s forward process fully erases data information, noting that complete removal is unattainable in finite time. It highlights finite-time formulations that converge to a specified prior while balancing sampling efficiency and sample quality.

  • Revisiting Assumptions.: Complete information removal during forward diffusion is unachievable in finite time.
  • Revisiting Assumptions.: Determining when to halt forward noising requires balancing sampling efficiency against sample quality.
  • Revisiting Assumptions.: Schrödinger bridges and optimal transport suggest alternative diffusion formulations converging to a specified prior in finite time.

Theoretical Understanding.

The survey examines why diffusion models can rival GANs without adversarial training and calls for theoretical explanations of when they outperform alternative generative models. It also emphasizes systematic guidance for understanding their quality, likelihood, and hyperparameter choices.

  • Theoretical Understanding.: Diffusion models can rival GANs in most applications without adversarial training.
  • Theoretical Understanding.: Theoretical analysis should identify when diffusion models are effective over VAEs, energy-based models, and autoregressive models.
  • Theoretical Understanding.: Understanding model distinctions may explain how diffusion models achieve excellent sample quality while attaining top likelihood.
  • Theoretical Understanding.: The survey highlights the need for systematic theoretical guidance when selecting diffusion-model hyperparameters.

Latent Representations.

Compared with VAEs and GANs, diffusion models provide less effective latent-space representations. This limits semantic manipulation and can reduce sampling efficiency when latent and data spaces share dimensionality.

  • Latent Representations.: Diffusion models are less effective than VAEs or GANs at providing useful latent-space representations.
  • Latent Representations.: Weak latent representations make semantic data manipulation difficult.
  • Latent Representations.: Latent spaces often match data dimensionality, negatively affecting sampling efficiency.
  • Latent Representations.: Diffusion models may not learn representation schemes well.

AIGC and Diffusion Foundation Models.

The survey discusses extending diffusion models into foundation-model and AIGC settings, including transferring generative pre-training from GPT to diffusion models and developing specialized models for different domains. It also highlights hybrid systems combining diffusion models with LLMs and reinforcement learning.

  • AIGC and Diffusion Foundation Models.: AIGC research increasingly combines generative pre-training, LLMs, and visual foundation models for strong generation performance and emergent abilities.
  • AIGC and Diffusion Foundation Models.: Transferring decoder-only generative pre-training from GPT series to diffusion models enables scaled evaluation of generation and emergent abilities.
  • AIGC and Diffusion Foundation Models.: Specialized diffusion foundation models are being developed for different domains and applications.
  • AIGC and Diffusion Foundation Models.: TraceRL adapts diffusion foundation models for reinforcement learning by generating coherent action sequences while retaining scalable foundation-model properties.
  • AIGC and Diffusion Foundation Models.: Combining LLMs with diffusion models offers a promising direction for sophisticated AIGC systems using complementary generative paradigms.
  • AIGC and Diffusion Foundation Models.: Future AIGC systems may use hybrid architectures that combine different generative principles for enhanced performance and broader capabilities.

9 CONCLUSION

The survey organizes diffusion-model research around foundational formulations, three major improvement directions, connections with other generative models, broad applications, and future research avenues.

  • The survey introduces three fundamental diffusion-model formulations: DDPMs, SGMs, and Score SDEs.
  • It identifies sampling efficiency, likelihood maximization, and techniques for data with special structures as three major improvement directions.
  • The survey examines connections between diffusion models and other generative models, including potential benefits of combining them.
  • A survey of applications across six domains illustrates the wide-ranging potential of diffusion models.
  • The paper concludes by outlining possible avenues for future research.
Loading 2209.00796v15…