PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Diffusion Models for Vision: A Survey

Anwaar Ulhaq, Naveed Akhtar

arXiv:2210.09292v3cs.CVcs.AI

TL;DR

Diffusion models’ computational demands can restrict access to their training and sampling. This survey reviews vision-focused efficiency design strategies and compares quality and efficiency, concluding that diffusion methods lead image-synthesis results while efficiency remains a design concern.

  • Problem

    Diffusion models require substantial computation for training and repeated sampling, while access to high-end resources remains a bottleneck to broader research participation.

    Source: §I · p. 1, C. Comparative performance · p. 12

  • Method

    The survey reviews recent vision diffusion-model design choices that affect computational efficiency and examines representative methods’ quality–efficiency trade-offs.

    Source: §I · pp. 2–3, §VI · p. 14

  • Results

    Diffusion-based techniques have established state-of-the-art image-synthesis results across the surveyed benchmarks, and GAN-based methods increasingly rarely outperform them on quantitative metrics.

    Source: C. Comparative performance · pp. 11–12

  • Takeaways & Limitations

    The survey identifies computational efficiency as a key design perspective for making diffusion models more practicable while retaining high sample quality.

    Source: §I · pp. 2–3

Abstract

from arXiv · show

Diffusion Models (DMs) have demonstrated state-of-the-art performance in content generation without requiring adversarial training. These models are trained using a two-step process. First, a forward - diffusion - process gradually adds noise to a datum (usually an image). Then, a backward - reverse diffusion - process gradually removes the noise to turn it into a sample of the target distribution being modelled. DMs are inspired by non-equilibrium thermodynamics and have inherent high computational complexity. Due to the frequent function evaluations and gradient calculations in high-dimensional spaces, these models incur considerable computational overhead during both training and inference stages. This can not only preclude the democratization of diffusion-based modelling, but also hinder the adaption of diffusion models in real-life applications. Not to mention, the efficiency of computational models is fast becoming a significant concern due to excessive energy consumption and environmental scares. These factors have led to multiple contributions in the literature that focus on devising computationally efficient DMs. In this review, we present the most recent advances in diffusion models for vision, specifically focusing on the important design aspects that affect the computational efficiency of DMs. In particular, we emphasize the recently proposed design choices that have led to more efficient DMs. Unlike the other recent reviews, which discuss diffusion models from a broad perspective, this survey is aimed at pushing this research direction forward by highlighting the design strategies in the literature that are resulting in practicable models for the broader research community. We also provide a future outlook of diffusion models in vision from their computational efficiency viewpoint.

I. INTRODUCTION

Diffusion models have broad vision applications and strong generative capabilities, but their iterative training and sampling make them computationally demanding. This survey focuses on efficiency-oriented design strategies, an emphasis missing from earlier broad reviews.

I. INTRODUCTION

  • Diffusion models are being applied across low- and high-level vision tasks, while text-to-image and text-to-video models have further fueled interest.Applications cited include denoising, inpainting, super-resolution, segmentation, and image translation.Source: §I · p. 1
  • Iterative evaluations and gradient computations make diffusion models costly, especially for high-dimensional images and videos; training may require 150–1,000 V100 GPU-days.Sampling also requires repeated model evaluations, creating a practical burden in resource-constrained settings.Source: §I · pp. 1–2
  • The survey reviews efficiency-oriented designs while considering the trade-off between sampling speed and sample quality; diffusion models remain slower than GANs at generation.The review describes early work as prioritizing quality over computational cost, with more recent work also addressing efficiency.Source: §I · p. 2
  • Earlier reviews covered diffusion models broadly, but none focused on computational efficiency, the design perspective this survey aims to address.The survey organizes existing methods around schemes that improve efficiency.Source: §I · pp. 2–3

II. AN OVERVIEW OF DIFFUSION MODELS

  • Diffusion models progressively corrupt data with noise, then learn a reverse process that transforms noise into samples from the target distribution.The survey introduces DDPM, latent diffusion, and feature pyramid latent diffusion as influential models for efficient diffusion modelling.Source: §II · p. 3

A. The Baseline: Denoising Diffusion Probabilistic Model (DDPM):

DDPM progressively adds Gaussian noise to training data and trains a neural network to reverse that corruption. Its Markov-chain sampling can be slow, particularly when many reverse steps are required.

A. The Baseline: Denoising Diffusion Probabilistic Model (DDPM):

  • The forward process gradually adds Gaussian noise according to a variance schedule, producing progressively corrupted samples.The process begins with data sampled from the real-data distribution and proceeds through noisy samples x1 to xT.Source: A. The Baseline: Denoising Diffusion… · p. 3
  • A neural network learns reverse conditional probabilities to denoise corrupted samples, with Gaussian reverse transitions assumed when the variance schedule is sufficiently small.The reverse transition is modelled as a Gaussian whose mean and variance parameters the network can predict.Source: A. The Baseline: Denoising Diffusion… · pp. 3–4
  • The variational lower-bound objective is decomposed into KL-divergence terms, and training uses a simplified objective that omits the weighting term.The simple objective is given as L = Lt + C, where C does not depend on θ.Source: A. The Baseline: Denoising Diffusion… · p. 4
  • Following the reverse Markov chain can take thousands of steps and make DDPM sampling far slower than GAN sampling on the same hardware.The passage gives an example of about 20 hours for 50,000 32 × 32 images from DDPM on an NVIDIA 2080 Ti, versus under a minute for a GAN.Source: A. The Baseline: Denoising Diffusion… · p. 4

B. Latent Diffusion Model (LDM):

LDM performs diffusion in a compressed latent space rather than pixel space, lowering training costs and improving inference speed. An encoder-decoder and conditioned UNet form its main components.

B. Latent Diffusion Model (LDM):

  • LDM moves diffusion from pixel space to latent space, reducing training costs and improving inference speed.The approach separates pixel-level perceptual compression from semantic generation in the learned latent.Source: B. Latent Diffusion Model (LDM) · p. 4
  • An encoder E maps an image to a smaller 2D latent z, which decoder D uses to reconstruct the image.The downsampling rate is f = H/h = W/w = 2^m.Source: B. Latent Diffusion Model (LDM) · p. 4
  • A time-conditional UNet denoises the latent, while cross-attention incorporates flexible conditioning such as class labels, semantic maps, or blurred images.Domain-specific encoders project conditioning inputs into representations mapped to intermediate UNet layers through cross-attention.Source: B. Latent Diffusion Model (LDM) · p. 5

C. Feature Pyramid Latent Diffusion Model (Frido):

Frido combines multiscale latent features with a feature-pyramid UNet for coarse-to-fine denoising, retaining both high-level structure and lower-level visual detail.

C. Feature Pyramid Latent Diffusion Model (Frido):

  • Frido encodes images as multiscale latent variables, with lower levels retaining visual details and higher levels preserving shapes and structures.The latent pyramid’s layer sizes decrease by half at each level, and a decoder combines the hidden variables into the output image.Source: C. Feature Pyramid Latent Diffusion… · p. 5
  • Noise is added sequentially across scales, first destroying image details, then high-level shape, and finally the image’s overall structure.Source: C. Feature Pyramid Latent Diffusion… · p. 6
  • PyU-Net performs multiscale denoising with coarse-to-fine information flow, while gating lets low-level denoising use high-level information.Teacher forcing is used to maintain training efficiency while preventing overfitting; a level-specific projection predicts the added noise.Source: C. Feature Pyramid Latent Diffusion… · p. 6

III. STRATEGIES FOR EFFICIENT DIFFUSION MODELS

The survey groups efficiency approaches into design changes and process improvements, addressing the slow, resource-intensive sampling of diffusion models. Classifier guidance illustrates a quality and computational-cost trade-off.

III. STRATEGIES FOR EFFICIENT DIFFUSION MODELS

  • Long Markov chains make diffusion sampling slow and costly, motivating Efficient Design Strategies (EDS) and Efficient Process Strategies (EPS).EDS modify baseline model design, whereas EPS improve the diffusion process or speed sampling.Source: §III · p. 6

A. Efficient Design Strategies (EDS)

1) Classifier Guidance:

  • Classifier guidance steers conditional samples using gradients from a noise-trained classifier, improving conditioning and sample quality but potentially adding substantial computational cost.Classifier-free guidance avoids classifier gradients but reduces sample diversity.Source: 1) Classifier Guidance · pp. 6–7

2) Discrete vs Continuous Design:

The reviewed literature spans continuous and discrete diffusion, although much current work focuses on continuous Gaussian processes. The section notes potential modeling concerns with discrete approaches and a sample-count advantage attributed to discrete design.

2) Discrete vs Continuous Design:

  • Discrete modeling can raise de-quantization, gradient, and log-likelihood interpretation concerns; ordinal data may instead use a discretized, truncated Gaussian transition.Source: 2) Discrete vs Continuous Design · p. 8
  • Discrete diffusion includes binary-state processes and later categorical transitions, while the survey says discrete design helps reduce the number of samples.The cited early work considered binary random variables; subsequent work extended the approach to categorical variables.Source: 2) Discrete vs Continuous Design · pp. 7–8
  • Continuous diffusion can be described by a stochastic differential equation, and much current work focuses on Gaussian processes in continuous state spaces.Continuous-space denoising models have reported strong log-likelihood scores and high-quality image production.Source: 2) Discrete vs Continuous Design · pp. 7–8

3) Score Matching Networks or SDEs Design:

Diffusion efficiency can be improved through differential-equation solvers, multiscale conditioning, and latent-space generation. The reviewed approaches reduce sampling effort or computation across image scales and representations.

3) Score Matching Networks or SDEs Design:

  • 20X speedup without degrading image quality was reported for accelerated DDPM in low-dose CT, while DPM-solver reached SOTA on CIFAR-10 within 50 steps and generated high-quality images in ten steps.The score-network approach also constructs a scoring-based diffusion ODE for evaluating precise probability.Source: 3) Score Matching Networks or SDEs… · p. 8

4) Pyramidal or non-Pyramidal Design:

  • A pyramidal approach conditions on pixel coordinates and randomizes image resolution during training, enabling one UNet-like network to operate across resolutions and faster reverse sampling than a full DDPM.Coordinates are sinusoidally encoded and expanded into a high-dimensional conditioning representation.Source: 4) Pyramidal or non-Pyramidal Design · p. 9

5) Pixel or Latent Representation-based Design:

  • Latent Diffusion Models generate images from a learned latent space in one network pass, whose spatial dimensionality has better scaling properties than pixel-space representations.The survey characterizes latent models as more efficient than pixel-based designs.Source: 5) Pixel or Latent… · p. 9

B. Efficient Process Strategies (EPS)

Efficient process strategies modify sampling, noise, or model combinations to reduce sampling time and computational demands. Training-free methods can augment a pretrained model, while other approaches alter noise schedules or combine and distill generative models.

1) Training Strategy:

  • Training-free sampling augments a pretrained model to sample in fewer steps, while avoiding the extra processing and instability risk associated with retraining.The survey groups approaches into analytical methods, implicit samplers, differential-equation solvers, and dynamic-programming adjustments.Source: 1) Training Strategy · p. 9
  • Dynamic programming uses memory to traverse sampling options and find an optimal sample path rather than merely constructing steps that reduce error faster.The survey describes this search as taking a relatively short time.Source: 1) Training Strategy · p. 9

2) Noise Distribution Strategy:

  • Learning the noise schedule can guide noise reduction during diffusion and sampling, while DDGM [56] demonstrates that Gamma-distributed noise improves image and voice generation.Most approaches use Gaussian noise, but distributions with more degrees of freedom are also being investigated.Source: 2) Noise Distribution Strategy · p. 9

3) The Mixing or Unifying Strategy:

  • Mixed-modeling combines diffusion with models offering faster sampling or greater expressiveness, including adversarial networks, autoregressive encoders, and normalizing flows [80],, [64].The survey also describes truncated diffusion and sampling patterns using models such as GANs and VAEs, and student distillation that can halve sampling steps per operation [83].Source: 3) The Mixing or Unifying Strategy · p. 9

4) Scheduling Strategy:

Scheduling strategies seek fewer sampling steps while maintaining sample quality, and retrieval strategies condition generation on related visual or multimodal examples. The survey introduces quality metrics as measures of generated-image quality, while noting human assessment as a reference point.

4) Scheduling Strategy:

  • Reducing SDE discretization step size can speed sampling but may introduce errors that significantly affect model performance, motivating schemes that minimize steps while maintaining sample quality.Non-Markovian transitions can use information from multiple earlier samples and produce accurate predictions at a higher step size.Source: 4) Scheduling Strategy · p. 10

5) Retrieval Strategy:

  • Retrieval-augmented diffusion conditions generation on retrieved examples, including prompt-relevant image-text pairs from an external multimodal knowledge base in Re-Imagen [88].The survey describes Re-Imagen as producing highly faithful images, including for rare or invisible entities.Source: 5) Retrieval Strategy · p. 10

IV. COMPARATIVE PERFORMANCE AND DISCUSSION

  • The survey compares techniques along computational requirements and sample quality, noting that Stable Diffusion helped shift attention toward efficiency.It introduces commonly used quality and efficiency metrics before discussing comparative performance.Source: §IV · p. 10

A. Quality measures

  • Quality metrics quantify generated-image visual quality, but the survey describes human visual assessment as the gold standard.The cited passage distinguishes metric-based evaluation from qualitative assessment.Source: A. Quality measures · p. 10

1) Inception Score (IS):

The survey describes image-quality metrics and computational-efficiency measures used to evaluate generative models. It notes limitations in Inception Score and NLL, and reports that diffusion methods lead conditional ImageNet synthesis results while requiring substantial model size and computation.

1) Inception Score (IS):

  • Inception Score measures the exponential of the KL divergence between marginal and conditional predicted class distributions, with high class probability and diversity as its rationale.Its effectiveness depends on ImageNet and a pretrained model, and small sample batches can make statistics unreliable [92].Source: 1) Inception Score (IS) · p. 10, 2) Frechet Inception Distance (FID) · p. 11
  • Diffusion models provide state-of-the-art results among five top-performing conditional ImageNet 256×256 synthesis techniques, while their model sizes and computational requirements are overwhelmingly large in comparison.The table caption provides this comparison but no individual metric values.Source: Table III · p. 11

2) Frechet Inception Distance (FID):

  • FID compares Gaussian distributions of pretrained features from real and generated images, and is popular because it correlates well with human perception.The survey presents FID as an alternative given Inception Score's limitations.Source: 2) Frechet Inception Distance (FID) · p. 11

3) Negative Log Likelihood (NLL):

  • NLL can be low even when generated images have poor visual quality, so the survey excludes NLL scores from its results in favor of FID and IS.The passage notes that low NLL may accompany memorization without diverse or realistic generation.Source: 3) Negative Log Likelihood (NLL) · p. 11

B. Efficiency measures:

  • Efficiency measures focus on computational demands rather than generated-image quality.The survey introduces these measures separately from sample-quality metrics.Source: B. Efficiency measures · p. 11

1) Sampling time/steps:

  • Sampling efficiency is commonly reported as samples per second or sampling steps, while training-step counts and V100 or A100 operation-days indicate computational requirements.Fewer steps generally imply higher efficiency; hardware reporting reflects the use of these GPUs in HPC clusters.Source: 1) Sampling time/steps · p. 11, 2) Hardware requirements · p. 11

3) Model size:

The survey distinguishes model size from direct sampling efficiency, compares image-synthesis quality alongside resource demands, and maps research trends around latent diffusion and future applications.

3) Model size:

  • Fewer parameters do not necessarily mean fewer computations, although larger models have a larger memory footprint that can indirectly affect efficiency.The survey notes that model size is often described by parameter count.Source: 3) Model size · p. 11

C. Comparative performance

  • Diffusion-based techniques have established state-of-the-art image-synthesis results across ImageNet 256×256, Celeb-A, and CIFAR-10, while GANs increasingly rarely outperform them quantitatively.The comparison reports top-performing methods based on FID scores.Source: C. Comparative performance · pp. 11–12
  • Diffusion methods often require high-end resources, many training days, and large models, limiting research access mainly to labs with HPC facilities.Public efficiency challenges and resource-limited image synthesis are proposed as directions for addressing this open gap.Source: C. Comparative performance · p. 12

D. Further discussion

  • The bibliometric analysis groups 50 representative papers into four clusters to examine latent diffusion's impact on emerging research.The analysis uses clustering, with the number of clusters controlled by a resolution parameter.Source: D. Further discussion · p. 12

V. EMERGING AND FUTURE DIRECTIONS

  • The survey anticipates research on adversarial manipulation, training-data privacy, DeepFake detection, and diffusion-based adversarial defenses.These are presented as prospective directions rather than established capabilities.Source: §V · pp. 12–13
  • Interpretability and explainability may inform efficient diffusion designs; DAAM is cited as a method producing pixel-level attribution maps from latent denoising subnetworks.The survey connects interpreting underlying model details with the prospect of efficient designs.Source: §V · p. 13
  • Few-shot generation, DreamBooth personalization, and zero-shot image editing are emerging directions, alongside research changes following Stable Diffusion's open-source release.CycleDiffusion is described as a plug-and-play approach for guiding pretrained diffusion models and GANs through latent-code control.Source: §V · p. 13
  • Future applications include motion generation, with Motion Diffusion Model supporting different conditioning modes and tasks and MotionDiffuse generating text-driven motion.The survey also identifies video diffusion architectures, including joint image-video training and video generation without text-video data, as directions for exploration.Source: §V · pp. 13–14

VI. CONCLUSION

The conclusion reviews efficiency-relevant design choices in diffusion models and compares image-synthesis approaches. It reports that diffusion models surpass GANs in performance, while their success remains dependent on high computational requirements.

VI. CONCLUSION

  • The review focuses on recent diffusion-model advances and design choices that directly affect computational efficiency.It contrasts this focus with earlier reviews that generally categorized diffusion models.Source: §VI · p. 14
  • The comparative image-synthesis analysis finds diffusion models have surpassed GANs as top performers.Source: §VI · p. 14
  • Diffusion models' success currently depends heavily on high computational requirements, while future efficiency improvements are expected to support wider use.Source: §VI · p. 14

Related papers

Loading 2210.09292v3…