Efficient Diffusion Models: A Survey

1 Introduction

• Multi-step denoising makes diffusion models time-consuming and computationally intensive, while high-resolution complex-scene generation remains demanding.These costs motivate efficiency research for practical deployment.Source: §1 · p. 2
• Efficiency-focused surveys are scarce; this work offers a broader, more recent taxonomy than the earlier initial exploration.The survey notes many broader diffusion surveys and task-specific surveys, but few focused on efficiency.Source: §1 · p. 2
• The survey reviews efficient diffusion techniques through algorithm-level, system-level, and framework perspectives.The perspectives cover distinct but interconnected research topics.Source: §1 · p. 2, Figure 1 · p. 3
• Algorithm-level coverage spans efficient training, fine-tuning, sampling, and model compression.These methods address resource-intensive training and inference.Source: §1 · p. 2
• System-level coverage includes hardware-software co-design, parallel computing, and caching to optimize infrastructure and computational resources.The survey frames these techniques around training and deployment.Source: §1 · p. 2
• The framework discussion covers specialized frameworks’ features, libraries, and specializations for diffusion training, fine-tuning, inference, and serving.The survey notes that mainstream frameworks lack some diffusion-specific optimizations and features.Source: §1 · p. 2

2.1.1 DDPM

• DDPMs progressively add Gaussian noise in a fixed forward Markov chain, then learn reverse transitions that denoise samples toward the data distribution.The reverse transition is parameterized as a conditional Gaussian, and training minimizes a variational-bound objective.Source: §2.1.1 · pp. 3–4

2.1.2 Score Matching

• Score matching estimates a model’s score by minimizing Fisher divergence, avoiding the intractable partition function required by maximum-likelihood estimation.Integration by parts yields an alternative objective when the data score is unavailable; its estimator involves the model-score Hessian trace.Source: §2.1.2 · pp. 4–5

2.1.3 Solvers

• Because sampling cost grows with the number of discretized timesteps, continuous-time SDE or ODE solvers offer learning-free ways to reduce steps.The discrete process approaches a continuous transformation as the step size decreases.Source: §2.1.3 · p. 5
• Reverse-time SDEs gradually convert noise into data; their drift includes the score function, and the forward SDE injects noise.The overview figure depicts the forward and reverse SDE processes.Source: §2.1.3 · p. 5, Figure 2 · p. 5
• ODE solvers produce deterministic trajectories and can converge faster than stochastic solvers, often with a marginal reduction in sample quality.The probability-flow ODE removes the stochastic term while sharing the reverse SDE’s marginal probability densities.Source: §2.1.3 · p. 6

2.1.4 Flow Matching

• Flow matching learns a time-dependent vector field that transforms a simple prior into a target data distribution through a continuous normalizing flow.The section presents flow matching as an efficient generative-modeling paradigm.Source: §2.1.4 · p. 6
• Direct flow-matching optimization is intractable, so conditional paths and vector fields are constructed per data sample and marginalized into global counterparts.The conditional endpoint at t = 1 is a narrow Gaussian centered on the sample; the resulting marginal at t = 1 approximates the data distribution and supports scalable training.Source: §2.1.4 · pp. 6–7
• The conditional flow-matching objective has the same gradients as the flow-matching objective but is computationally simpler, avoiding score matching’s Hessian calculations.It uses an L2 distance between vector fields.Source: §2.1.4 · p. 7

2.2.1 Image Generation

• Stable Diffusion uses compressed latent space, while LCM enables high-quality image generation in 4 steps and progressive distillation matches 50-step quality with 2-8 steps.The section also describes ControlNet’s spatial conditioning and EDM’s training and sampling improvements.Source: §2.2.1 · p. 7

2.2.2 Video Generation

• Video methods trade inference time against quality or use frozen pretrained models with added modules, consistency distillation, grid interpolation, or parallelizable coarse-to-fine generation.AdaDiff targets prompt-specific policy gradients; ED-T2V and spatial-temporal adapters freeze the original model during training, while VideoLCM distills in latent space.Source: §2.2.2 · p. 8

2.2.3 Text Generation

• Text diffusion faces challenges with discrete corruption and unstable continuous-space objectives; proposed approaches use cross-entropy, adaptive noise schedules, self-conditioning, or continuous latent spaces.Masked-Diffuse LM applies cross-entropy at each diffusion step, and SeqDiffuSeq uses adaptive noise scheduling and self-conditioning.Source: §2.2.3 · p. 8

2.2.4 Audio Generation

• Audio methods reduce diffusion steps or generalize discrete steps to continuous ones, addressing demands for high fidelity, temporal consistency, and low-latency generation.FastDPM uses a bijective mapping between noise levels and is reported to perform well in high-stochasticity speech generation tasks.Source: §2.2.4 · pp. 8–9

2.2.5 3D Generation

• For 3D generation, larger sampling steps, 2D-to-3D sampling changes, and multi-view parallelism improve sampling efficiency, while state-space models and lightweight feature extractors reduce processing burden.The approaches address computational demands associated with high-resolution volumetric data and point clouds.Source: §2.2.5 · p. 9

3.1.1 Latent Space

• Latent diffusion moves computation into lower-dimensional representations, reducing computational complexity and memory use while supporting generation across image, video, 3D, audio, tabular, and other domains.For example, LVDM supports videos exceeding 1,000 frames, while latent diffusion for policy archives achieves a 13x compression ratio.Source: §3.1.1 · pp. 10–11

3.1.2 Loss Formulation

• Random-direction projections in sliced score matching reduce complexity by avoiding full Hessian computation.This addresses the high-dimensional cost of score matching, whose Hessian trace requires many more backward passes than a gradient.Source: §3.1.2 · pp. 11–12
• FID 23.3 on MS COCO 2017-5k: InstaFlow applies Rectified Flow to text-conditioned one-step image generation.Rectified Flow learns straight transport paths between distributions; Flow Matching’s linear parameterization likewise enables straight-line trajectories.Source: §3.1.2 · p. 12

3.1.3 Training Tricks

• Data-dependent adaptive priors align initialization with modality-specific data distributions rather than relying solely on a standard Gaussian prior.The survey describes applications across speech, graphs, and trajectories, with faster convergence and outputs that better reflect underlying data structure.Source: §3.1.3 · pp. 13–14
• The Leapfrog Diffusion Model uses an adaptive-prior initializer to skip multiple denoising steps while maintaining accuracy for real-time trajectory prediction.MoFlow is described as achieving a 100x faster one-step student model with comparable performance through IMLE-based distillation.Source: §3.1.3 · p. 14, Figure 6 · p. 14
• Over 17.5× speedup in some cases and FID 1.42 on ImageNet 256x256 with classifier-free guidance: REPA aligns denoiser hidden states with a pretrained visual encoder.This adds a loss term to align the model’s representations with external visual representations such as DINOv2.Source: §3.1.3 · p. 14
• FID 1.35 on ImageNet 256x256 and over 21× faster convergence than baseline DiT models: VA-VAE aligns its latent space with a pretrained vision foundation model.Combined with an optimized LightningDiT architecture, the approach is reported to mitigate the reconstruction-fidelity versus generation-quality trade-off.Source: §3.1.3 · p. 14
• Noise schedules include systematic noise addition and dynamic adjustment; the Laplace schedule samples more often around medium noise intensities.The survey describes this focus as improving training efficiency, convergence, and sample quality.Source: §3.1.3 · p. 15

3.2.1 LoRA

• LoRA keeps pretrained weights frozen and injects trainable low-rank matrices, enabling parameter-efficient adaptation with standard inference and no added latency.The method represents the weight update as BA and computes with W = W0 + BA.Source: §3.2.1 · p. 17
• LCM-LoRA combines task-specific and acceleration LoRAs to reduce sampling from dozens of steps to around 4 while maintaining generation quality.It adds the acceleration vector to a base latent diffusion model without architectural modifications.Source: §3.2.1 · p. 17
• Concept Sliders learn low-rank directions for smooth control of image attributes, while LoRA-Composer modifies U-Net attention blocks to fuse multiple customized concepts without retraining a fusion matrix.Concept strength is adjusted through a guidance scale; LoRA-Composer’s comparison with Mix-of-Show is described in the passage.Source: §3.2.1 · pp. 17–18

3.2.2 Adapter

• Adapters add small task-specific modules while keeping pretrained weights frozen, reducing memory and computation for tasks such as text-to-image generation.The described module uses down-projection, nonlinearity, and up-projection layers within a transformer block.Source: §3.2.2 · p. 18
• T2I-Adapter adds sketch, depth, or segmentation conditions for structural guidance, aligning inputs with intermediate U-Net features without changing the pretrained model.Its convolutional and residual blocks align conditions with spatial feature dimensions at multiple scales.Source: §3.2.2 · p. 19
• IP-Adapter uses image prompts for visual-detail consistency, whereas CTRL-Adapter guides attributes such as emotion or object type; IP-Adapter trains added modules while freezing the pretrained model.The IP-Adapter architecture uses decoupled cross-attention.Source: §3.2.2 · p. 19
• SimDA uses simulated data in an adapter for cross-domain generation, aiming to generalize to previously unseen data distributions.The passage describes transfer between image datasets as an example.Source: §3.2.2 · pp. 19–20

3.2.3 ControlNet

• ControlNet integrates spatial conditions through zero-convolution layers while preserving pretrained features; its derivatives pursue more accurate, unified, or efficient control.ControlNet++ reports gains in mIoU (11.1%), SSIM (13.4%), and RMSE (7.6%); ControlNet-XS is about twice as fast in inference and training, while ControlNeXt reduces learnable parameters by up to 90%.Source: §3.2.3 · p. 20

3.3.1 Efficient Solver

• SA-Solver (Xue et al., 2024b) combines variance-controlled SDEs with a stochastic Adams predictor-corrector, while Restart (Xu et al., 2023b) alternates noise injection and ODE steps to contract errors.SA-Solver uses prior evaluations and noise control to reduce function evaluations; Restart leverages SDE error contraction and ODE efficiency in low-NFE regimes.Source: §3.3.1 · p. 22
• PNDM (Liu et al., 2022a) achieves second-order convergence, a 20x speedup, and a 0.4-point FID reduction at the same step count.The method decomposes the numerical solver into gradient and transfer components to address limitations of DDIM’s ODE formulation.Source: §3.3.1 · p. 23
• TrigFlow (Lu & Song, 2024) enables consistency models up to 1.5B parameters to achieve state-of-the-art performance in two sampling steps.Its simplified ODE parameterization bridges EDM and Flow Matching; Du et al. (2024) instead uses deterministic ODE trajectories, fixed noise, and consistency distillation to reduce redundancy while maintaining fidelity.Source: §3.3.1 · p. 24

3.3.2 Sampling Scheduling

• Deep-equilibrium DDIM sampling (Pokle et al., 2022) treats sampling steps as a fixed-point system for simultaneous updates, while ParaDiGMS and ParaTAA use iterative parallel updates.StreamDiffusion combines batched denoising, residual classifier-free guidance, asynchronous queues, and similarity filtering for real-time interactive generation.Source: §3.3.2 · pp. 24–25
• FastDPM maps continuous diffusion steps to noise levels, while Watson et al. (2021) finds a 32-timestep DDPM schedule with performance comparable to thousands of steps.Watson et al. use dynamic programming to optimize timestep selection over the decomposable ELBO.Source: §3.3.2 · p. 25
• AYS (Sabour et al., 2024) reduces sampling steps by 40% while maintaining quality, with equivalent FID at 20 steps versus 30 baseline steps.It co-optimizes schedules and solvers through KLUB minimization and supports plug-and-play acceleration across modalities and solver types without model retraining.Source: §3.3.2 · p. 26

3.3.3 Truncated Sampling

• ASE progressively skips more blocks near the noise regime while retaining more near the data regime, accelerating sampling while preserving generation quality.U-ViT skips decoder blocks only, retaining encoder information; DeeDiff instead exits adaptively when layer uncertainty falls below a threshold.Source: §3.3.3 · p. 27
• Retrieval-guided methods condition generation on retrieved examples: kNN-Diffusion uses CLIP embeddings without paired text-image training data, while Re-Imagen retrieves multimodal pairs for rare or unseen entities.Re-Imagen’s cascaded diffusion architecture progressively refines images at different resolutions to reduce computational cost.Source: §3.3.3 · pp. 27–28
• ReDi achieves a two times speedup with comparable quality and enables zero-shot image stylization without domain-specific knowledge bases.It retrieves a similar state from a knowledge base and skips intermediate sampling steps, reducing the number of function estimations.Source: §3.3.3 · p. 28

3.3.4 Knowledge Distillation

• 250 times greater efficiency with only a 5% drop in performance is achieved by progressive distillation, which iteratively trains students to halve teacher NFEs.Luhman & Luhman (2021) distill DDIM into a Gaussian model requiring one NFE, but training requires generating teacher outputs that can entail hundreds of NFEs per batch.Source: §3.3.4 · p. 29
• Meng et al. (2023) use two-stage distillation to transfer classifier-free guidance and then progressively reduce steps for conditional diffusion models.The approach is applied to pixel- and latent-space models for text-guided generation and image inpainting.Source: §3.3.4 · p. 29
• Consistency distillation uses generative ODE self-consistency to support few-step or one-step diffusion generation.Song et al. (2023) minimize differences in a self-consistency function; Chen et al. (2025) enable 1-4 step generation, while Wu et al. (2024) use consistency distillation for one-step generation with multi-scale latent space and 3D attention.Source: §3.3.4 · pp. 29–30
• Score Distillation Sampling (SDS) transfers a 2D text-to-image diffusion model into a 3D NeRF by optimizing rendered images from a fixed view.Wang et al. (2024b) extend SDS with Variational Score Distillation (VSD), treating the scene as a distribution and enabling more varied, realistic scenes, even with a single particle.Source: §3.3.4 · p. 30
• Diff-Instruct transfers pre-trained diffusion-model knowledge to diverse generators without extra data, while Cascaded Score Distillation supervises across multiple resolutions.Luo et al. (2024) introduce an Integral Kullback-Leibler divergence for Diff-Instruct; Decatur et al. (2024) integrate independently trained cascaded stages to guide both fine details and global structure in 3D representations.Source: §3.3.4 · p. 30

3.4 Compression

• Compression reduces diffusion-model size and arithmetic operations through two orthogonal approaches: quantization and pruning.Source: §3.4 · p. 30, Figure 22 · p. 31

3.4.1 Quantization

• PTQ methods address diffusion-specific calibration and accumulated quantization error, with later approaches using time-step-aware sampling, specialized quantizers, mixed precision, or unified noise formulations.PTQ4DM is limited to lower resolutions and 8-bit precision; Q-Diffusion adds time-step-aware calibration and a specialized noise-estimation quantizer, while PTQD unifies quantization and diffusion-perturbed noise.Source: §3.4.1 · p. 31, Figure 23 · p. 31
• QAT trains diffusion models with quantization operators to learn quantization-friendly representations, at greater training cost than PTQ.Methods include time-dependent quantization intervals and a jointly quantized low-rank adapter that distills full-precision denoising without training data.Source: §3.4.1 · p. 32

3.4.2 Pruning

• Diff-Pruning estimates weight importance using Taylor expansion over pruned timesteps, while LD-Pruner uses latent-vector shifts in central tendency and variability to assess operator importance.Source: §3.4.2 · p. 32, Figure 24 · p. 32
• LayerMerge jointly prunes convolution layers and activation functions, using dynamic programming to solve its surrogate optimization problem exactly and efficiently.Source: §3.4.2 · p. 32
• LAPTOPDiff uses an additive one-shot criterion and combinatorial optimization to prune diffusion-model U-Nets, with pruning performance reported as superior to traditional layer-pruning and manual-removal methods.Source: §3.4.2 · pp. 32–33

4.1 Hardware-Software Co-Design

• Mobile GPU kernel and operator optimizations target near-real-time diffusion inference, while SDA combines quantization-aware training and a hybrid systolic array for edge FPGAs.SDA uses two-level pipelining for nonlinear operators and reports a 97.3x speedup over an ARM Cortex-A53 CPU.Source: §4.1 · p. 33, Figure 26 · p. 33

4.2 Parallel Computing

• Distributed inference and training methods partition diffusion workloads across devices, with approaches including image patches, network-layer distribution, and pipeline parallelism.DistriFusion reports 6.1x speedup on 8xA100 GPUs versus one card while maintaining output quality; SwiftDiffusion reports 5x lower inference latency and 2x higher throughput, while DiffusionPipe reports 1.41x training speedup from pipeline parallelism and an additional 1.28x from data parallelism.Source: §4.2 · pp. 34–35, Figure 27 · p. 34, Figure 28 · p. 34

4.3 Caching Technique

• Caching exploits temporal locality in diffusion computation by reusing noise states or intermediate features to avoid redundant denoising work.NIRVANA reuses and reconditions cached noise states to skip initial denoising steps; DeepCache reuses high-level U-Net features while updating low-level features, and Block Caching schedules reuse of layer outputs with adjustments for alignment.Source: §4.3 · pp. 35–36, Figure 29 · p. 35, Figure 30 · p. 36
• Diffusion Transformer caching methods reuse computations across layers or consecutive denoising steps, including dynamic layer selection in L2C and regular-interval output reuse in FORA.MD-DiT combines block skipping and caching for dynamic depth adjustment; ∆-DiT combines step-sensitive block caching with resource allocation.Source: §4.3 · p. 36

5 Frameworks

• Flash Attention, xFormers, DeepSpeed, and OneFlow support both training and inference optimization, with attention, memory, distributed-computing, or compiler-oriented capabilities.Source: §5 · p. 37, Table 2 · p. 37
• Inference-focused tools such as Stable-Fast, Onediff, DeepCache, TGATE, and xDiT target faster generation through low-latency optimization, caching, quantization, or parallelization.Source: §5 · p. 37, Table 2 · p. 37

Hybridizing Diffusion and Autoregressive Models

• Hybridizing autoregressive and diffusion models could bring key-value caching and streamable generation to diffusion, with Block Diffusion applying autoregressive-style caching across denoising blocks.Source: §6 · p. 38

Without Classifier-Free Guidance

• Model-guidance removes classifier-free guidance's second network forward pass, reporting doubled inference speed, 6.5× faster convergence, and an ImageNet 256 FID of 1.34 at about 12% of comparable approaches' computational resources.Source: §6 · p. 38

Efficient Attention Mechanisms for Video Diffusion

• Efficient video-diffusion attention methods exploit structured sparsity, with sparse 3D attention reported to have linear complexity in frame count and enable up to 7.8× faster high-resolution video generation.Source: §6 · p. 38

• The survey organizes efficient diffusion research across algorithm-level and system-level perspectives, alongside frameworks with optimizations and features for efficiency.Its review is guided by a taxonomy.Source: §7 · p. 38
• The authors frame efficient diffusion models as an area aimed at democratizing diffusion models.Source: §7 · p. 38
• The survey aims to help researchers and practitioners get started in the field and inspire new research.The authors expect efficiency to play an increasingly important role in diffusion models and related systems.Source: §7 · p. 38