PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffusion Models in Low-Level Vision: A Survey

Chunming He, Yuqi Shen, Chengyu Fang, Fengyang Xiao, Longxiang Tang, Yulun Zhang, Wangmeng Zuo, Zhenhua Guo, Xiu Li

arXiv:2406.11138v2cs.CVcs.AI

TL;DR

Low-level vision research lacks a comprehensive survey organizing diffusion-model techniques despite their use in restoring degraded data and the diversity of methods across tasks. This survey reviews frameworks, applications, benchmarks, and evaluations; its experiments report that real-world-trained models outperform synthetic-trained models on a real-world test, while computational overhead remains high, especially for high-resolution images.

  • Problem

    Diffusion methods span diverse, complex low-level vision tasks, but existing surveys tend to focus on foundational theory, generation, or a limited set of tasks.

    Source: §1 · p. 2

  • Method

    The survey reviews generic diffusion frameworks and their connections to other generative models, categorizes methods by framework and task, and covers applications, benchmarks, and evaluation.

    Source: §1 · p. 2, §7 · p. 16

  • Results

    Models trained on real-world data consistently outperform synthetic-trained models on the real-world LOLv1 test, while computational costs and parameter counts are not necessarily correlated with performance.

    Source: §5.3 · p. 14

  • Takeaways & Limitations

    Diffusion methods are being applied in increasingly complex and higher-dimensional areas, including 3D objects, locomotion, and 4D scenes, calling for continued research.

    Source: §7 · p. 16

    Despite progress in reducing sampling steps, computational cost remains high, particularly for high-resolution images, leaving a substantial gap from real-time applications.

    Source: §6.1 · pp. 14–15

Abstract

from arXiv · show

Deep generative models have garnered significant attention in low-level vision tasks due to their generative capabilities. Among them, diffusion model-based solutions, characterized by a forward diffusion process and a reverse denoising process, have emerged as widely acclaimed for their ability to produce samples of superior quality and diversity. This ensures the generation of visually compelling results with intricate texture information. Despite their remarkable success, a noticeable gap exists in a comprehensive survey that amalgamates these pioneering diffusion model-based works and organizes the corresponding threads. This paper proposes the comprehensive review of diffusion model-based techniques. We present three generic diffusion modeling frameworks and explore their correlations with other deep generative models, establishing the theoretical foundation. Following this, we introduce a multi-perspective categorization of diffusion models, considering both the underlying framework and the target task. Additionally, we summarize extended diffusion models applied in other tasks, including medical, remote sensing, and video scenarios. Moreover, we provide an overview of commonly used benchmarks and evaluation metrics. We conduct a thorough evaluation, encompassing both performance and efficiency, of diffusion model-based techniques in three prominent tasks. Finally, we elucidate the limitations of current diffusion models and propose seven intriguing directions for future research. This comprehensive examination aims to facilitate a profound understanding of the landscape surrounding denoising diffusion models in the context of low-level vision tasks. A curated list of diffusion model-based techniques in over 20 low-level vision tasks can be found at https://github.com/ChunmingHe/awesome-diffusion-models-in-low-level-vision.

1 INTRODUCTION

Low-level vision methods seek to recover high-quality images from degraded inputs, but conventional and deep-learning approaches face limits in complex degradations and realistic texture generation. The survey organizes diffusion-model research across frameworks and tasks, reviews applications and evaluations, and discusses limitations and future directions.

1 INTRODUCTION

  • CNNs and transformers show promising distortion metrics but still produce poor texture, while GANs targeting realistic details face unstable training and artifacts that can undermine global coherence.Earlier handcrafted approaches also struggle with complex degradations because of limited generalizability.Source: §1 · p. 1
  • Diffusion models add noise in a forward process and learn to remove it in reverse, offering extensive distribution coverage, a stable objective, and straightforward scalability.Source: §1 · p. 1
  • The survey illustrates low-quality inputs beside diffusion-enhanced outputs and depicts continuous growth of diffusion methods across four low-level vision scenarios.Source: Figure 1 · p. 2, Figure 3 · p. 3
  • The diversity and complexity of diffusion techniques make them difficult to understand and develop into general-purpose reconstruction models, while earlier surveys often cover theory, generation, or limited task ranges.Source: §1 · p. 2
  • The survey presents diffusion frameworks and their connections to other generative models, categorizes methods by framework and task, reviews related applications, benchmarks, metrics, experiments, and future directions.Its experimental evaluation covers six representative tasks quantitatively and qualitatively.Source: §1 · p. 2
  • The authors selected studies through searches of DBLP, Google Scholar, and ArXiv, prioritizing reputable venues, available code, and higher citation counts.They also assessed each paper's contribution and whether it was a seminal work.Source: §1 · p. 2

2 A WALK-THROUGH OF DIFFUSION MODELS

This section explains DDPM, NCSN, and SDE formulations of diffusion, then compares diffusion models with other generative models. It describes both shared principles and differences in training, sampling, and latent-space representations.

2.1 Denoising Diffusion Probabilistic Models

  • A DDPM progressively perturbs data toward a simple prior and learns reverse Gaussian transitions to generate samples from noise.The reverse chain is trained to match the time reversal of the forward chain by minimizing a KL divergence.Source: §2.1 · p. 3

2.2 Noise Conditioned Score Networks

  • NCSNs estimate the score ∇x log p(x) and use Langevin updates to sample, while decreasing noise scales address inaccurate estimates in low-density regions.Conventional score estimation with Langevin dynamics can collapse to a low-dimensional manifold.Source: §2.2 · p. 4

2.3 Stochastic Differential Equations

  • SDEs model forward and reverse diffusion continuously, with reverse-time dynamics using the score function to convert noise into data.Euler-Maruyama discretization and ODEs are among the practical numerical sampling techniques described.Source: §2.3 · p. 4
  • DDPM and NCSN training objectives differ only by a fixed scaling factor, and their discrete processes correspond to VP and VE SDE formulations.The survey identifies VP SDEs as continuous versions of DDPMs and annealed Langevin dynamics as the discrete version of VE SDEs.Source: §2.3 · p. 4

2.4 Comparisons With Other Deep Generative Models

  • VAE latent-space diffusion can build more efficient models, whereas normalizing flows require invertible, differentiable architectures; FlowIE straightens probability-transfer trajectories and significantly speeds inference, achieving task enhancement in fewer than five steps.Source: §2.4 · p. 5
  • Compared with GANs, diffusion models offer stable training and greater diversity but require multiple iterative inference steps; flow models use deterministic invertible mappings for faster computation.The survey also notes that diffusion sampling is inefficient relative to GAN sampling.Source: §2.4 · pp. 4–5
  • Diffusion latent-space semantic manipulation is more implicit and less controllable than GAN manipulation, and its latent space has been less extensively explored.The survey notes that diffusion latent space nevertheless exhibits a well-defined structure [31].Source: §2.4 · p. 5

3 DIFFUSION MODELS FOR NATURAL IMAGE PROCESSING IN LOW-LEVEL VISION

The section frames low-level vision as restoration of degraded images and surveys diffusion methods by training manner and application goal. It contrasts task-specific supervised approaches with adaptable zero-shot priors, then reviews methods for inverse problems and individual restoration tasks.

3 DIFFUSION MODELS FOR NATURAL IMAGE PROCESSING IN LOW-LEVEL VISION

  • Low-level vision restoration estimates clean images x0 from degraded observations y, and diffusion models condition stochastic refinement on y to approximate p(x|y).The forward model is y = H(x0) + n.Source: §3 · p. 6

3.1 DM-based methods with different training manners

  • After reviewing over 300 works, the survey classifies methods by training manner and application goal, contrasting supervised task-specific fit with zero-shot adaptability across tasks.Supervised methods need paired data and training from scratch; zero-shot methods use pretrained generative priors without task-specific retraining.Source: §3 · p. 6, §3.1 · p. 6

3.2 DM-based methods with different application goals

  • Posterior-sampling methods condition score estimation on the degraded observation or jointly diffuse image and observation, while ILVR [72] guides pretrained-DDPM generation with low-frequency information.ILVR is learning-free and limited to noiseless inverse problems.Source: §3.2 · p. 6, Figure 7 · p. 7
  • For general-purpose restoration, DDRM uses SVD to reconstruct missing information in the measurement operator’s spectral space, while DDNM [77] refines null-space contents for linear operators.These approaches presuppose a known forward operator and often simplify problems to linear inverse problems, limiting their effectiveness on real-world blind tasks.Source: §3.2 · pp. 6–7
  • For super-resolution, cascaded diffusion progressively increases resolution, while SinSR distills sampling to one step and reports up to tenfold faster inference.SRDiff was the pioneering diffusion-based single-image SR model; implicit DMs address continuous-resolution SR.Source: §3.2 · p. 7
  • RePaint uses resampling iterations to condition images, while Spatial DM [95] uses surrounding context to inpaint large missing regions; deblurring includes residual modeling and multiscale structure guidance.Source: §3.2 · p. 8
  • WeatherDiffusion restores multiple weather degradations across image sizes, while LLIE and fusion methods incorporate features such as wavelets, semantic guidance, and multimodal distributions.PyDiff [54] increases resolution during reverse diffusion; Dif-Fusion creates a multichannel distribution to improve color fidelity in infrared-visible fusion.Source: §3.2 · pp. 8–9

4 EXTENDED DIFFUSION MODELS

The survey extends diffusion-model applications beyond natural images to medical imaging, remote sensing, and video. Across these areas, methods address reconstruction, missing data, modality-specific degradations, and temporal coherence through task-specific conditioning and model designs.

4.1 Diffusion models for medical image processing

  • Medical diffusion methods address image degradation from acquisition constraints and generate missing modalities that can complement multimodal diagnosis.The survey covers MRI, CT, multimodal imaging, and other modalities including PET and OCT.Source: §4.1 · p. 9
  • For MRI, score-based reconstruction uses data consistency to handle multi-coil images and subsampling patterns, while DOLCE conditions limited-angle CT diffusion on filtered back-projection priors and enforces sinogram consistency.DOLCE was evaluated on C4KC-KiTS; other MRI approaches include adaptive priors and self-supervised training.Source: §4.1 · pp. 9–10
  • PET-DDPM uses MR images as denoising-network input and PET data consistency during inference, while DenoOCT-DDPM uses self-fusion preprocessing to remove speckle while preserving small vessels.The survey reports the MR-prior and PET-consistency configuration as PET-DDPM’s best-performing setup.Source: §4.1 · p. 10

4.2 Diffusion models for remote sensing data

  • Remote-sensing methods span visible, hyperspectral, and radar imagery; examples include arbitrary-size dehazing with an adaptive-region diffusion model and HSI generation using spectral folding.The dehazing method uses cyclic shifts to address inconsistent color and artifacts.Source: §4.2 · pp. 10–11
  • HSR-Diff fuses high-resolution multispectral and low-resolution hyperspectral images, while SAR methods address despeckling, cloud removal, and cross-modal translation.DDS2M is a self-supervised HSI restoration method that learns from degraded HSI alone; DDPM-CR uses auxiliary SAR data for cloud removal.Source: §4.2 · p. 11

4.3 Diffusion models for video processing

  • Video diffusion research remains limited to several fundamental tasks, including prediction, interpolation, super-resolution, and restoration, where temporal consistency is a central requirement.Approaches include masked conditional generation, pretrained image-model adaptation with temporal layers, and test-time adaptation for unseen weather conditions.Source: §4.3 · pp. 11–12

5 EXPERIMENTS

The experiments review datasets and evaluation criteria, then compare diffusion methods across six low-level vision tasks and discuss runtime and scalability. Results vary by task and metric, with different methods leading perceptual, distortion, and structural measures.

5.1 Datasets

  • The review describes large-scale pretraining datasets and task-specific degradation datasets, with models typically pretrained before task-specific fine-tuning.ImageNet and CelebA are cited as generative-model pretraining datasets.Source: §5.1 · p. 12

5.2 Evaluation metrics

  • Evaluation spans distortion metrics, perceptual image metrics, human ratings, and downstream-task effects.PSNR uses mean squared error; SSIM assesses contrast, brightness, and structure; MOS summarizes participant judgments.Source: §5.2 · p. 13

5.3 Experimental results

  • Runtime was measured at 256 × 256 resolution on an RTX 4090 GPU, and computational cost and parameter count are not necessarily correlated with performance.IR-SDE is reported to perform strongly in both super-resolution and motion deblurring, while zero-shot methods remain confined to linear degradation scenarios.Source: §5.3 · pp. 13–14
  • Tiramisu leads most zero-shot inpainting scenarios, especially large masks, while Repaint stands out in narrow regions; MRI reconstruction and fusion leaders also vary by metric and setting.For MRI, AdaDiff has the best overall performance, particularly at R=8x, while SSDiffRecon approaches it on PSNR and SSIM at R=4x.Source: §5.3 · pp. 13–14, Table 4 · p. 13, Table 6 · p. 13
  • Across 4× super-resolution and deblurring, methods divide by metric: IDM and DiffIR perform well on LPIPS, Resdiff on PSNR and SSIM, and MSGD on perceptual deblurring metrics.DiffEvent and HI-Diff are competitive on deblurring PSNR and SSIM; Fig. 18 presents qualitative comparisons across six tasks.Source: §5.3 · p. 13, Table 2 · p. 13, Table 3 · p. 13
  • Low-light enhancement results likewise differ by measure: GSAD leads PSNR, Reti-Diff is competitive on LPIPS, and CFWD reaches the best real-world performance.Cross-dataset tests find models trained on real-world data consistently outperform those trained on synthetic data on real-world LOLv1; GSAD and PyDiff use ground-truth mean brightness tuning.Source: Table 5 · p. 13, §5.3 · p. 14, Figure 18 · p. 14

6 FUTURE DIRECTIONS

The future directions address diffusion models’ computational burden and the need to balance perceptual quality with content fidelity and downstream utility. They also identify challenges in real-world generalization, data availability, controllability, and the use of multimodal inputs.

6.1 Mitigating the limitations of DMs

  • Faster sampling methods reduce diffusion inference to 10–20 steps, with SinSR reported to work in one step, but computational cost remains high for high-resolution images and real-time use.The survey also identifies model quantization, pruning, and distillation as compression avenues, while noting current compression work primarily targets generation tasks.Source: §6.1 · pp. 14–15

6.2 Amalgamating the strengths of DMs with the traits of low-level vision

  • Diffusion methods can excel on perceptual metrics while underperforming on distortion metrics when diversity compromises content consistency; hybrid models and Pareto-frontiers are proposed as responses.The survey notes that breakthrough progress remains absent and calls for novel mixed structures and metrics.Source: §6.2 · p. 15
  • Downstream-task designs include joint bi-level optimization and feature-level aggregation, but existing strategies are often tailored to specific tasks.The survey calls for a unified diffusion-based strategy that serves a wider range of downstream tasks.Source: §6.2 · p. 15

6.3 Tackling the inherent challenges of low-level vision

  • Distortion-invariant learning can support generalization to diverse or unknown degradations, while distortion estimation is needed to extend zero-shot diffusion models to real-world applications.The survey also identifies multi-degradation real-world scenarios as a future direction.Source: §6.3 · pp. 15–16
  • Data generation can create pseudo image pairs for training, while user-controllable generation is proposed as a way to filter negative data and improve controllability.The survey notes that some extreme tasks have severely limited data availability because acquisition is difficult or costly.Source: §6.3 · pp. 15–16

6.4 Empowering low-level vision through multi-modal advances

  • Text prompts and pretrained diffusion models support customizable and zero-shot restoration, while audio and embodied sensor inputs are proposed as further multimodal directions.The survey gives examples including environmental sensors for dehazing and motion sensors for deblurring.Source: §6.4 · p. 16

7 CONCLUSIONS

The conclusion presents the survey as addressing an overlooked review gap in diffusion models for low-level vision. It summarizes frameworks, applications, and limitations, and points to emerging higher-dimensional areas for continued research.

7 CONCLUSIONS

  • The survey examines diffusion models applied to low-level vision, addressing a gap the authors identify in previous surveys.Source: §7 · p. 16
  • Its review identifies generic diffusion frameworks, categorizes low-level vision methods from multiple angles, and discusses existing limitations and future directions.Source: §7 · p. 16
  • The conclusion points to emerging applications involving 3D objects, locomotion, and 4D scenes as areas warranting continued research.Source: §7 · p. 16

Related papers

Loading 2406.11138v2…