PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffusion Models in Vision: A Survey

Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, Mubarak Shah

arXiv:2209.04747v6cs.CVcs.AIcs.LG

TL;DR

Diffusion models have achieved strong generative results in vision but remain computationally slow because sampling uses many steps. This survey reviews their core frameworks and relationships, categorizes applications, and identifies limitations and future research directions.

  • Problem

    Diffusion models are rapidly developing across computer vision, creating a need for a comprehensive and timely understanding of their frameworks and applications.

  • Method

    The survey synthesizes three diffusion modeling frameworks, relates them to other deep generative models, and categorizes vision applications across multiple perspectives.

  • Results

    The survey presents a comprehensive review of denoising diffusion models in computer vision, covering theoretical and practical contributions.

  • Takeaways & Limitations

    The review provides a framework for understanding diffusion models and helps researchers locate relevant work for specific computer vision domains.

  • Takeaways & Limitations

    Diffusion inference can be inefficient because generating a sample may require thousands of evaluation steps, while long-term temporal relations remain challenging for video generation.

Abstract

from arXiv · show

Denoising diffusion models represent a recent emerging topic in computer vision, demonstrating remarkable results in the area of generative modeling. A diffusion model is a deep generative model that is based on two stages, a forward diffusion stage and a reverse diffusion stage. In the forward diffusion stage, the input data is gradually perturbed over several steps by adding Gaussian noise. In the reverse stage, a model is tasked at recovering the original input data by learning to gradually reverse the diffusion process, step by step. Diffusion models are widely appreciated for the quality and diversity of the generated samples, despite their known computational burdens, i.e. low speeds due to the high number of steps involved during sampling. In this survey, we provide a comprehensive review of articles on denoising diffusion models applied in vision, comprising both theoretical and practical contributions in the field. First, we identify and present three generic diffusion modeling frameworks, which are based on denoising diffusion probabilistic models, noise conditioned score networks, and stochastic differential equations. We further discuss the relations between diffusion models and other deep generative models, including variational auto-encoders, generative adversarial networks, energy-based models, autoregressive models and normalizing flows. Then, we introduce a multi-perspective categorization of diffusion models applied in computer vision. Finally, we illustrate the current limitations of diffusion models and envision some interesting directions for future research.

1 INTRODUCTION

Diffusion models have rapidly expanded in computer vision because they generate detailed, diverse, and text-aligned samples across many tasks. This survey reviews their frameworks, relationships, applications, limitations, and future directions.

  • Diffusion models generate detailed and diverse samples, with Stable Diffusion producing images that show few artifacts and strong alignment with text prompts.
  • Their applications span image generation, super-resolution, inpainting, editing, image-to-image translation, segmentation, classification, and anomaly detection.
  • It synthesizes three generic diffusion modeling frameworks and relates diffusion models to VAEs, GANs, EBMs, autoregressive models, and normalizing flows.
  • Inference remains inefficient because generating a sample can require thousands of evaluation steps, motivating faster sampling without sacrificing quality.
  • The survey provides a comprehensive review of denoising diffusion models in computer vision to help readers understand the generic modeling framework.
  • The survey introduces a multi-perspective categorization based on framework, target task, and denoising condition to help researchers find relevant work.

2 GENERIC FRAMEWORK

Diffusion models use a forward process that progressively converts data into noise and a reverse process that reconstructs data from noise. The survey presents DDPMs, NCSNs, and SDEs as three formulations, while relating them to other generative models and noting practical limitations.

  • General diffusion process: Diffusion models train through forward corruption and backward denoising, generating images by reconstructing data from random Gaussian noise.The forward phase adds noise over multiple steps; inference reverses those steps with noise estimates from a neural network.
  • Three generic formulations: The survey organizes diffusion models into DDPM, NCSN, and SDE formulations, with SDEs generalizing the first two.Each formulation specifies how noise is added, how reversal is learned, and how samples are generated.
  • DDPMs: DDPMs use a Gaussian Markov forward process whose variance schedule enables direct sampling of any noisy state from the original image.The schedule uses β1, …, βT as stepwise variance hyperparameters, and the recursive formulation permits single-step sampling of xt.
  • DDPMs: DDPM reverse models start from xT ∼ N(0, I) and learn Gaussian transitions that recover progressively less noisy images.Small forward noise steps make reverse transitions amenable to Gaussian modeling; a neural network predicts the reverse-process mean.
  • NCSNs: Score-based formulations estimate gradients of log data density to guide iterative sampling toward high-density regions, but manifold structure can make score estimation inconsistent.The survey notes that low-dimensional data manifolds can prevent Langevin dynamics from converging to high-density regions.
  • SDEs: SDE-based sampling includes Predictor-Corrector methods and reverse-time ODE solvers, with the ODE strategy offering an efficiency advantage.Predictor-Corrector sampling combines numerical reverse-SDE integration with a score-based correction step.
  • Relations to other generative models: Compared with GANs, diffusion models offer stable training and greater diversity but require multiple network evaluations during inference.The survey also contrasts their image-sized Gaussian latent representation with GANs’ low-dimensional latent space.

3 A CATEGORIZATION OF DIFFUSION MODELS

The survey categorizes diffusion models by task, denoising condition, framework, and datasets, while reviewing major unconditional and framework-specific developments.

  • The taxonomy uses task, denoising condition, underlying framework or architecture, and datasets as classification criteria.
  • Unconditional Image Generation: Unconditional image generation is treated as the basic unsupervised setting for diffusion models.
  • Denoising Diffusion Probabilistic Models: DDPM research includes discrete transition matrices, noise prediction, improved noise schedules and variance learning, and non-Markovian DDIM sampling.
  • Denoising Diffusion Probabilistic Models: The survey also reports diffusion models achieving state-of-the-art likelihoods for image density estimation.
  • Score-Based Generative Models: Score-based models improve training and sampling through noise-scale strategies, noise conditioning, parameter averaging, and stabilized annealed sampling.
  • Stochastic Differential Equations: Other frameworks include Schrödinger-bridge methods, latent-space diffusion with up to 56 times faster sampling, critically damped Langevin diffusion, and reduced-space sampling.

3.2 Conditional Image Generation

Conditional diffusion models incorporate class labels, representations, noise signals, or combinations of models to control image synthesis. The section also covers methods targeting quality, sampling efficiency, and high-resolution generation.

  • Conditional diffusion commonly uses class labels, while some methods support both conditional and unconditional generation.
  • Denoising Diffusion Probabilistic Models: Classifier guidance uses classifier gradients during sampling, alongside architectural changes intended to improve FID.
  • Conditional Guidance: Other conditioning strategies use self-supervised representations, noise signals, compositional diffusion models, or classifier-free guidance.
  • Efficiency and Sampling: Several methods target the data manifold or reduce sampling cost through fewer diffusion steps, altered discretization, higher-order solvers, or distillation.
  • High-Resolution Generation: Cascaded Diffusion Models generate increasingly higher-resolution images using successive models conditioned on class labels and lower-resolution outputs.

3.3 Image-to-Image Translation

Diffusion models support image-to-image translation through a shared multi-task framework, jointly trained models, energy-based guidance, pretrained semantic spaces, and latent Brownian bridges.

  • A single diffusion framework addresses colorization, inpainting, uncropping, and JPEG restoration without custom changes for each task.
  • Unpaired translation can use two jointly trained diffusion models that condition each reverse process on the other model’s intermediate sample.
  • Energy-based guidance can preserve domain-agnostic features while translating source-domain-specific characteristics.
  • Pretrained GLIDE models can be fine-tuned for conditional downstream image-generation tasks through staged encoder and decoder training.
  • Brownian-bridge diffusion in a VQ-GAN latent space maps source-domain representations to target-domain representations for translation.
  • Task-specific regressors or segmentation models can replace classifiers to guide the diffusion process at every sampling step.

3.4 Text-to-Image Synthesis

Text-to-image diffusion systems combine text or multimodal embeddings with latent, discrete, cascaded, and transformer-based generation pipelines. The surveyed methods also address sampling efficiency and image-guided synthesis.

  • Text-to-image synthesis highlights diffusion models’ ability to combine unrelated concepts and generate unusual examples from text prompts.
  • Imagen uses a text encoder and cascaded diffusion models conditioned on text embeddings to generate high-resolution images.
  • VQ-Diffusion uses a VQ-VAE’s discrete latent space and masking mechanism to avoid the unidirectional bias and error accumulation of previous approaches.
  • CLIP-conditioned diffusion first generates an image embedding and then decodes the final image from that embedding and the text caption.
  • Sampling efficiency is improved through discretization schemes, polynomial score extrapolation, and exponential integration of the reverse SDE.
  • Other systems combine VQ-VAE representations with diffusion or guide reverse denoising using nearest-neighbor CLIP embeddings and text prompts.

3.5 Image Super-Resolution

Diffusion models have been applied to image super-resolution using both score-based sampling and conditional reverse processes. The surveyed approaches target high-quality reconstruction from low-resolution inputs or distribution couplings.

  • Conditional super-resolution: Saharia et al. condition the reverse process on low-resolution images to generate high-quality super-resolved images.The method uses CelebA-HQ, FFHQ, and ImageNet with architectures from earlier diffusion models.
  • Score-based sampling: Daniels et al. use score-based models and Langevin dynamics to sample from the Sinkhorn coupling of two distributions.Their experiments include image super-resolution with a U-Net architecture.

3.6 Image Editing

Diffusion models support guided image editing by progressively noising an input and denoising it toward a specified modification. Methods address stroke-based guidance, text-based regional editing, and latent-space processing.

  • Guided editing: Guided editing preserves properties such as shapes and colors while progressively smoothing deformations through forward noising and reverse denoising.The approach covers stroke painting, stroke-based editing, and image composition.
  • Text-guided editing: Mask-based natural-language editing modifies selected regions while denoising to improve global image coherence.The method uses CLIP guidance and updates the denoising process after each step.
  • Latent-space editing: Latent diffusion edits local image regions by encoding the image and adaptive mask into latent space and applying text-guided iterative denoising.The decoded output demonstrates superior performance while being comparably faster.

3.7 Image Inpainting

Diffusion-based inpainting modifies reverse sampling to preserve known image regions while generating unknown content. The surveyed methods also use text conditioning and address structural or semantic errors in the missing regions.

  • Text-conditioned inpainting: Nichol et al. condition diffusion models on text descriptions and fine-tune them for image inpainting, enabling text-based image modifications.They report better results with classifier-free guidance than CLIP-based guidance.
  • Mask-agnostic inpainting: Lugmay et al. sample known regions from the masked image and denoise unknown regions during each reverse step.The unknown region obtains the right structure but can remain semantically incorrect, motivating repeated processing.

3.8 Image Segmentation

The surveyed passages show diffusion models extending beyond generation into segmentation, score-based modeling, alternative diffusion processes, and energy-based modeling. These approaches combine diffusion representations, stochastic sampling, and modified forward or reverse processes across multiple vision tasks.

  • Image Segmentation: Diffusion U-Net feature maps can support semantic segmentation by combining decoder representations across scales and classifying each pixel with multilayer perceptrons.Later denoising steps provide the feature maps used for segmentation.
  • Image Segmentation: Diffusion probabilistic models can generate multiple segmentation samples for one input, enabling computation of a mean segmentation map.The architecture uses separate encoders for the input image and current estimated image before combining them in the U-Net.
  • Score-Based Models: The NCSN estimates score functions at different noise scales and samples images using annealed Langevin dynamics.Experiments include image generation and inpainting.
  • SDE Frameworks: The SDE formulation represents forward and reverse diffusion as stochastic differential equations and enables Predictor-Corrector and ODE-based deterministic samplers.Experiments cover image generation, inpainting, and colorization.
  • Alternative Diffusion Processes: Non-uniform diffusion assigns each pixel a different SDE and uses multiple networks corresponding to different diffusion scales.The method includes a conditional sampler and is evaluated on synthesis, super-resolution, inpainting, and edge-to-image translation.
  • Discrete Diffusion: ImageBART reverses a multinomial diffusion process on compact image representations with an autoregressive transformer.It is evaluated on unconditional, class-conditional, and text-conditional generation, plus local editing.

3.10 Medical Image Generation and Translation

Diffusion models are applied across medical imaging tasks, including segmentation, inverse reconstruction, denoising, and anomaly detection. These methods use conditioning, stochastic sampling, and generated healthy images to support reconstruction or identify abnormalities.

  • Medical image generation and translation: Diffusion models support medical image segmentation, reconstruction from measurements, and denoising tasks.Applications include brain tumor segmentation, accelerated MRI reconstruction, and speckle-noise removal in OCT scans.
  • Medical image generation and translation: Multiple stochastic segmentation samples can provide both a mean segmentation map and its associated variance.The brain MR image conditions intermediate denoising steps, and repeated samples differ because of stochasticity.
  • Medical image generation and translation: Medical-image inverse problems can incorporate measurements through proximal optimization, closed-form sampling, or data-consistency mappings.Score models are pretrained unconditionally before conditional information is introduced during sampling.
  • Medical image generation and translation: Diffusion-based anomaly detection commonly generates healthy images and subtracts them from lesion-containing or original images to produce anomaly maps.Methods use healthy-image training, classifier guidance, latent-space diffusion, or attention-based conditioning.

3.12 Video Generation

Diffusion models have been extended to video generation, infilling, restoration, shape modeling, classification, registration, and other emerging vision tasks. Video methods vary in conditioning and sampling design, with sampling effectiveness depending on the dataset.

  • Video generation: Diffusion models generate videos using 3D U-Nets, recurrent context conditioning, and autoregressive continuation for longer sequences.One approach generates frame by frame, while another conditions later video chunks on earlier ones.
  • Video generation: RaMViD enables video generation and infilling by randomly splitting frames into masked and unmasked groups during training.Unmasked frames condition the diffusion process, while masked frames undergo forward diffusion.
  • Video generation: Flexible diffusion models randomly choose diffused and conditioning frames and support multiple sampling schemes for long video generation.The reported conclusion is that the preferred sampling choice depends on the dataset.
  • Other applications: Diffusion models have also been applied to semantic image synthesis, weather-degraded image restoration, linear inverse problems, point clouds, classification, registration, counterfactuals, adversarial purification, and few-shot learning.These applications adapt conditioning, latent-space, spectral, deformation, classifier-guided, or transformer-based mechanisms.

4 CLOSING REMARKS AND FUTURE DIRECTIONS

The survey organizes diffusion modeling into three primary formulations and reports strong image-generation results, including greater diversity than GANs. It also identifies efficiency, task coverage, text rendering, and long-term video modeling as open issues and directions.

  • Closing remarks: The survey identifies DDPMs, NCSNs, and SDEs as three primary diffusion-model formulations.These formulations are presented as the main frameworks for diffusion modeling.
  • Closing remarks: Diffusion models obtain remarkable image-generation results, surpass GANs, and increase generated-sample diversity.The survey characterizes the research area as still being in an early phase.
  • Limitations: The main efficiency limitation is that generating one sample requires multiple inference steps, leaving GANs faster at image production.The survey notes continued research toward more efficient diffusion models.
  • Future directions: Future research includes diffusion models for additional vision tasks and evaluation or use of their learned representations in discriminative settings.Suggested approaches include training discriminative models on latent representations and augmenting datasets with generated samples.
  • Future directions: Other proposed directions include simulating possible video futures for reinforcement learning and developing multipurpose models with varied conditioning and outputs.The latter direction is discussed in relation to understanding steps toward artificial general intelligence.
  • Future directions: Text-to-video diffusion has reduced artifacts and achieved strong generative performance, but generated videos remain rather short.Modeling long-term temporal relations and interactions between objects remains an open challenge.

APPENDIX A VARIATIONAL BOUND.

The appendix derives the diffusion variational bound by expressing the data likelihood over noisy latent variables and exploiting Markov structure. Rearrangement and cancellation yield a formulation based on reconstruction and KL-divergence terms.

  • Variational bound: The variational-bound derivation treats noisy images x_1:T as latent variables and the original image x_0 as observed.The data log likelihood is written as the log marginal of the joint probability over x_0:T.
  • Variational bound: Jensen’s inequality converts expected negative log likelihood into an upper-bound minimization objective.The appendix applies the inequality and reverses its direction because the logarithm is concave.
  • Variational bound: Markov assumptions factorize the forward and reverse processes into timestep-wise probabilities.The resulting products are transformed into sums using logarithm properties.
  • Variational bound: Conditioning the forward-process posterior on x_0 makes it tractable when the unconditioned posterior is intractable.The appendix uses Bayes’ rule and the Markov property to obtain the additional conditioning.
  • Variational bound: After cancellation and rearrangement, the variational bound is expressed using KL divergences.The appendix states that the resulting expression is the formulation given in Eq. (4).

APPENDIX B NOISE ESTIMATION.

The appendix explains how fixing the reverse-process covariance and reparameterizing its mean converts the variational objective into a simple noise-estimation objective. The neural network therefore predicts the noise in a noisy image rather than directly outputting the reverse-process mean.

  • Covariance simplification: The reverse-process covariance is fixed to σ_t^2 · I, with σ_t^2 = β_t, so the corresponding divergence reduces to a mean-distance term plus a θ-independent constant.This removes covariance training from the model objective.
  • Mean reparameterization: The true reverse-process mean can be expressed using the noisy image x_t and its noise z_t, while x_t is already the model input.This motivates reparameterizing the model mean in the same form.
  • Noise prediction: The reparameterized network output z_θ(x_t, t) estimates the noise z_t from the noisy image x_t.The network no longer directly outputs μ_θ(x_t, t) at this stage.
  • Objective derivation: Substituting the mean parameterizations into the KL term gives the final simplified training formulation.The simplification follows the covariance fixing, mean reparameterization, and weight removal steps.
  • Simplified objective: Discarding the time-dependent weights yields a time-weighted noise-distance term and produces the simplified objective L_simple.The resulting objective is the expected squared distance between sampled noise z_t and predicted noise z_θ(x_t, t).
Loading 2209.04747v6…