PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffusion Models for Image Restoration and Enhancement: A Comprehensive Survey

Xin Li, Yulin Ren, Xin Jin, Cuiling Lan, Xingrui Wang, Wenjun Zeng, Xinchao Wang, Zhibo Chen

arXiv:2308.09388v3cs.CV

TL;DR

Image restoration needs better texture recovery, while comprehensive guidance on rapidly diversifying diffusion-based methods remains limited. This paper surveys diffusion-model approaches across learning paradigms, degradation scenarios, techniques, benchmarks, and evaluations, finding broad application across restoration tasks while identifying practical limitations and five future directions.

  • Problem

    Image restoration remains challenging for degraded images, and a comprehensive review of diffusion-model-based restoration methods is scarce.

  • Method

    The paper systematically categorizes diffusion-based IR into supervised, zero-shot, and blind or real-world flows, then reviews techniques, datasets, metrics, comparisons, and future directions.

  • Results

    Diffusion-based methods are reviewed across image super-resolution, deblurring, inpainting, and real-world restoration, with DiffLL and LightenDiffusion achieving optimal subjective generalization on the compared low-light datasets.

  • Takeaways & Limitations

    The survey identifies sampling efficiency, model compression, distortion simulation and estimation, distortion invariant learning, and framework design as future research directions.

  • Takeaways & Limitations

    Current methods face limited task relevance and comparison reasonability, while large models such as DDPM and SR3 constrain computationally efficient real-time restoration.

Abstract

from arXiv · show

Image restoration (IR) has been an indispensable and challenging task in the low-level vision field, which strives to improve the subjective quality of images distorted by various forms of degradation. Recently, the diffusion model has achieved significant advancements in the visual generation of AIGC, thereby raising an intuitive question, "whether diffusion model can boost image restoration". To answer this, some pioneering studies attempt to integrate diffusion models into the image restoration task, resulting in superior performances than previous GAN-based methods. Despite that, a comprehensive and enlightening survey on diffusion model-based image restoration remains scarce. In this paper, we are the first to present a comprehensive review of recent diffusion model-based methods on image restoration, encompassing the learning paradigm, conditional strategy, framework design, modeling strategy, and evaluation. Concretely, we first introduce the background of the diffusion model briefly and then present two prevalent workflows that exploit diffusion models in image restoration. Subsequently, we classify and emphasize the innovative designs using diffusion models for both IR and blind/real-world IR, intending to inspire future development. To evaluate existing methods thoroughly, we summarize the commonly-used dataset, implementation details, and evaluation metrics. Additionally, we present the objective comparison for open-sourced methods across three tasks, including image super-resolution, deblurring, and inpainting. Ultimately, informed by the limitations in existing works, we propose five potential and challenging directions for the future research of diffusion model-based IR, including sampling efficiency, model compression, distortion simulation and estimation, distortion invariant learning, and framework design.

1 Introduction

Image restoration improves degraded images but conventional networks often produce weak textures. This survey organizes diffusion-based restoration research and evaluates its methods, scenarios, benchmarks, and challenges.

  • Motivation: Conventional CNN- and Transformer-based restoration networks improve PSNR and SSIM but still generate unsatisfactory textures.These texture limitations hinder application in real-world scenarios.
  • Motivation: GAN-based restoration methods introduce adversarial loss because pixel-wise MSE and L1 losses are susceptible to blurred textures.SRGAN and DeblurGAN combine pixel-wise and adversarial objectives for texture generation.
  • Diffusion models for IR: Diffusion models progressively add pixel-wise noise and reverse that process through denoising to reconstruct images.Their generative capability motivates applying them to image restoration and texture recovery.
  • Survey scope: Diffusion-based restoration studies divide into supervised models optimized from scratch and zero-shot methods.The categorization is based on training strategy.
  • Survey scope: The survey addresses the lack of a comprehensive review by organizing methods, practical blind or real-world settings, comparisons, and future challenges.It covers diffusion-model background, supervised and zero-shot IR, blind or real-world degradation, benchmarks, and potential directions.

2 Background on Diffusion Model (DM)

Diffusion models formulate generation as forward corruption followed by reverse recovery, using discrete probabilistic or continuous SDE-based formulations. DDPM, NCSN, and SDE provide foundational modeling perspectives for restoration.

  • Overview: Diffusion models transform generation into several independent and stable reverse processes through Markov-chain modeling.The survey identifies DDPM, NCSN, and SDE as widely used foundational diffusion models.
  • DDPM: DDPM progressively corrupts training data with scheduled noise until the latent distribution approaches standard Gaussian noise.The forward process is a parameterized Markov chain over T steps.
  • DDPM: DDPM reverses the corruption by approximating the posterior and learning a denoising network while keeping the variance schedule predefined.The network estimates the mean-related denoising quantity in the reverse process.
  • SDE: SDEs unify score-based and diffusion probabilistic modeling through continuous diffusion processes with drift and diffusion coefficients.The diffusion coefficient controls random-noise perturbation, while the drift coefficient can support Gaussian distributions.
  • SDE: Reverse-time SDE sampling estimates the score function with a neural network and solves the process backward from T to 0.DDPM and NCSN correspond to discretizations of variance-preserving and variance-exploding SDEs.

3 Diffusion model-based Image Restoration Methods

Diffusion-based image restoration methods are organized around supervised learning from scratch and zero-shot exploitation of pretrained generative priors. The survey further distinguishes conditioning, diffusion-space, and inverse-problem strategies, while noting substantial training and data costs for supervised approaches.

  • Learning paradigms: Supervised methods train diffusion models for restoration, whereas zero-shot methods reuse pretrained diffusion priors without retraining.Supervised approaches require paired distorted/clean data; zero-shot methods use distorted images and exploit learned structure and texture priors.
  • Supervised conditioning: SR3 conditions a DDPM-style U-Net on an upsampled low-resolution image and progressively denoised output to predict restoration noise.At t = 0, the model delivers an upsampled high-quality image.
  • Survey organization: The survey addresses a fragmented research landscape by organizing diffusion-based restoration methods across training strategy, conditioning, architecture, modeling space, and inverse-problem design.Its taxonomy covers supervised and zero-shot methods, including real-world restoration models.
  • Zero-shot restoration: Zero-shot methods guide pretrained diffusion sampling through reference refinement, including ILVR’s low-frequency substitution for structural and semantic consistency.Most subsequent zero-shot methods focus on improving refinement strategies during sampling while remaining training-free.
  • Framework and conditioning designs: Supervised conditioning strategies use low-quality references, pre-processed references, or revised diffusion processes, while framework designs include image-, residual-, and latent-space generation.Preprocessing can produce feature or clean-image references, residual diffusion simplifies whole-image generation, and latent diffusion targets efficiency.
  • Inverse-problem strategies: Zero-shot inverse-problem methods improve restoration through spectral or range-null-space decompositions and posterior estimates that balance data consistency with perceptual quality.DDNM assigns data consistency to the range space and perceptual quality to the null space; DPS estimates the posterior using the clean-image expectation.

4 Diffusion Models for Blind/Real-world Image Restoration

Diffusion-based blind and real-world image restoration addresses unknown, complex degradations through distortion simulation, kernel estimation, domain translation, and enhancement modules. Applications include low-light enhancement and adverse-weather removal, but comparisons and efficiency remain constrained.

  • Challenges: Unknown real-world degradation modes and scarce distorted/clean pairs make blind and real-world restoration difficult.Synthetic distortions often differ from real-world distributions, limiting out-of-distribution performance.
  • Distortion Simulation: Distortion simulation improves robustness by generating training degradations that approximate real-world corruption.SR3+ introduces RealESRGAN’s second-order degradation simulation during training.
  • Kernel Estimation: Kernel estimation guides blind restoration by modeling degradation as y = (x∗k) ↓s +n and estimating the unknown kernel during sampling.BlindDPS uses a parallel diffusion model, whereas GibbsDDRM samples kernel parameters and images jointly from a posterior.
  • Domain Translation: Domain translation creates more realistic training pairs or enables unpaired restoration through cycle-consistent generators.Yang et al. translate synthetic low-quality images toward the real-world domain, while RainDiffusion uses cooperative unpaired branches.
  • Applications and Limitations: Real-world applications include low-light enhancement and adverse-weather removal, with LightenDiffusion showing strong subjective generalization but lower objective quality.The comparison uses LOL, DICM, NPE, and VV datasets and NIQE and PI subjective metrics.
  • Applications and Limitations: Current real-world diffusion methods face limited task relevance, comparison reasonability, and restoration efficiency.Real-world datasets are often insufficient, and perceptual quality commonly depends on multiple sampling steps.

5 Experiments

The survey standardizes evaluation by summarizing datasets, implementation settings, and metrics, then compares diffusion-based restoration methods across several tasks. Its detailed objective comparison focuses on super-resolution, deblurring, and inpainting.

  • Evaluation Setup: The survey summarizes datasets, experimental configurations, and evaluation metrics to support efficient and thorough comparisons.Dataset summaries are organized by restoration task, while implementation details distinguish supervised and zero-shot methods.
  • Supplementary Results: Additional comparisons for deblurring, inpainting, JPEG artifact removal, and real-image restoration are reported in the supplementary material.The supplementary section also contains comparisons involving severe composite degradation.
  • Evaluation Setup: Objective comparisons cover image super-resolution, image deblurring, and image inpainting using PSNR, SSIM, FID, and LPIPS.Running time, parameter count, and FLOPs are also measured to compare computation cost and network complexity.
  • Super-resolution Results: Super-resolution experiments test full DIV2K and Urban100 images, retrain SR3 and SRDiff on DF2K, and assess DiffBIR and SUPIR on bicubic degradations.These settings are intended to maintain fair comparisons and examine real-world SR generalization on synthetic data.

6 Challenges and Future Directions

The survey identifies practical barriers to diffusion-based restoration, including sampling cost, model size, real-world distortion mismatch, distortion estimation, limited invariance, and framework design. It proposes efficiency, robustness, and architecture directions to address them.

  • 6.1 Sampling Efficiency: Sampling efficiency limits practical restoration because diffusion quality depends on many steps; SR3 takes about 50 seconds for one 224 × 224 image.DDIM reduces this example to 8 seconds, while several strategies reduce sampling to roughly 10–20 steps.
  • 6.1 Sampling Efficiency: Starting diffusion from low-quality image information can remove extra sampling steps required by starting from pure noise.One-step methods such as OSEDiff use latent information from the low-quality image and knowledge distillation.
  • 6.2 Model Compression: Large diffusion models limit real-time deployment; DDPM and SR3 contain 113.7M and 155.3M parameters, respectively.Pruning, quantization, distillation, and low-rank decomposition are identified as compression directions, but diffusion-based IR remains under-explored.
  • 6.3 Distortion Simulation and Estimation: Synthetic degradation is difficult to align with real-world corruption, while unpaired learning can yield unsatisfactory texture consistency.Domain translation and diffusion models are proposed to generate or translate more realistic training pairs.
  • 6.3 Distortion Simulation and Estimation: Distortion estimation is needed for real-world extension because many zero-shot methods model linear reverse problems and rely on identifiable degradation modes.The survey distinguishes distortion-adaptive learning from inverse-problem approaches.
  • 6.4 Distortion Invariant Learning: Distortion invariant learning seeks representations that remain invariant across degradation modes while preserving structure and textual information.Suggested domain-generalization routes include domain alignment, data augmentation, and meta-learning.
  • 6.5 Framework Design: Framework design can vary condition strategies, generation spaces, and noise predictors, with most methods still based on DDPM’s U-Net.Generation spaces include image, residual, latent, and frequency spaces; transformer-based noise predictors remain less explored.
  • 6.5 Framework Design: All-in-one restoration requires adaptation across diverse degradations while sharing restoration knowledge, exemplified by degradation-aware CLIP conditioning.The approach is represented by DACLIP.

7 Conclusions

The survey organizes diffusion-based image restoration by training strategy, degradation scenario, technical design, and evaluation. It concludes that diffusion models offer strong generative capacity for structure and texture restoration while leaving efficiency and broader applicability as continuing challenges.

  • Conclusions: The survey groups diffusion-based restoration into supervised, zero-shot, and blind/real-world flows, with fine-grained taxonomies of their techniques and trade-offs.It also summarizes datasets and metrics and compares open-sourced methods on super-resolution, deblurring, and inpainting.
  • Diffusion Model Development: Diffusion-model development spans text-to-image synthesis, image restoration, and general image generation applications.The chronological overview is presented in Fig. 13.
  • Diffusion Model Foundations: Diffusion models estimate data distributions through score functions and progressively denoise samples toward the data distribution.The survey reviews NCSN, DDPM, SDE, forward perturbation, reverse reconstruction, and noise-score matching foundations.
  • Model Development: Optimization, sampling, architecture, and conditioning strategies are major axes for improving diffusion models.The survey discusses noise schedules, ODE-based acceleration, U-Net and transformer architectures, and conditional mechanisms.

C Methods

The survey organizes diffusion-based image restoration methods by task, workflow, and design strategy, covering supervised, zero-shot, unified, and task-specific approaches.

  • Taxonomy: Diffusion-based image restoration methods include supervised and zero-shot models, organized in the survey by degradation type and workflow.
  • Image super-resolution: SR3 restores directly in image space, whereas SRDiff operates in residual space to accelerate diffusion-based super-resolution.
  • Image super-resolution: Transfer-learning methods such as adaptors and ControlNet fix pretrained diffusion models while preserving their texture-generation capability.
  • Unified models: Unified diffusion frameworks address multiple independent restoration tasks, including super-resolution, deblurring, inpainting, deraining, dehazing, and desnowing.
  • Task-specific restoration: Task-specific methods introduce specialized mechanisms, including degradation-aware conditioning, patch-based weather removal, blind blur-kernel estimation, and degradation-prior-guided shadow removal.

D Experiments

The survey catalogs datasets by restoration task and highlights substantial differences in dataset content and degradation modes across tasks.

  • Dataset coverage: Table 7 summarizes training and testing datasets for super-resolution, inpainting, deblurring, denoising, shadow removal, desnowing, deraining, and dehazing.
  • Super-resolution: Super-resolution evaluations use datasets including Set5, Set14, BSD100, Manga109, Urban100, ImageNet 1K, and CelebA-HQ.
  • Deblurring and shadow removal: Deblurring commonly uses GoPro, RealBlur-J, REDS, and HIDE, while shadow removal uses ISTD and SRD.
  • Dehazing and desnowing: Dehazing benchmarks include Haze-4K, Dense-Haze, and RESIDE, while desnowing benchmarks include CSD, Snow100k, and SRRS.
  • Dataset coverage: The survey provides consolidated training and testing dataset lists spanning natural images, faces, medical images, rain, snow, and other restoration settings.

D.2 Evaluation Metrics

The survey reviews objective and subjective image-quality metrics, emphasizing pixel fidelity, perceptual similarity, distributional quality, and no-reference assessment.

  • Metric overview: The reviewed metrics are PSNR, SSIM, LPIPS, DISTS, FID, KID, NIQE, and PI.
  • Reference-based metrics: PSNR measures pixel-wise distance through mean square error, while SSIM compares contrast, brightness, and structure between distorted and clean images.
  • Perceptual metrics: LPIPS uses pretrained AlexNet features for perception-oriented comparison, with lower values indicating greater similarity in perceptual space.
  • Distributional metrics: FID compares generated and reference image distributions using Inception features, whereas KID uses polynomial-kernel maximum mean discrepancy and is more stable with few samples.
  • Perceptual metrics: PI is defined as PI = 0.5((10 −Ma) + NIQE) for perceptual-quality evaluation of super-resolved images.
  • Perceptual metrics: DISTS evaluates texture and structure similarity using VGG features and an SSIM-like feature-space distance.

D.3 Implementation details

Implementation practices vary across supervised and zero-shot methods, with dataset, optimization, sampling, pretrained-model, and image-size choices shaping evaluation settings.

  • Implementation reporting: The survey reports training datasets, testing datasets, batch sizes, iterations, learning rates, and sampling steps for supervised methods, plus pretrained models for zero-shot methods.
  • Arbitrary-size restoration: Diffusion restoration models generally require generated-image resolution to match the optimization process, limiting arbitrary-size processing for high-resolution inputs.
  • Arbitrary-size restoration: Fully convolutional arbitrary-size processing can be computationally expensive, while patch-based methods denoise overlapping patches and average overlap regions for consistency.
  • Dataset choices: Training data varies by task: traditional super-resolution commonly uses DIV2K and Flickr2K, while SR3 uses ImageNet for natural images and FFHQ for faces.
  • Task-specific settings: Deblurring commonly uses GoPro, shadow removal uses ISTD and SRD, and dehazing uses Haze-4K, Dense-Haze, and RESIDE.

D.4 Results on Image Inpainting.

The inpainting comparison evaluates five zero-shot diffusion models, revealing a trade-off between distortion fidelity, perceptual quality, and sampling efficiency.

  • Five zero-shot diffusion models are evaluated on narrow-mask image inpainting, including DPS, DDRM, DDNM, Repaint, and CoPaint.The comparison includes three multitask models and two models designed specifically for inpainting.
  • DDRM and DDNM achieve better distortion-metric performance, while Repaint and CoPaint outperform other models in perceptual quality.
  • CoPaint reduces FID by 0.08dB on CelebA-HQ and 1.33dB on ImageNet compared with RePaint.
  • DDRM generates a single image fastest because its low sampling-step count keeps runtime almost linearly correlated with NFE.
  • CoPaint’s time-travel strategy improves restoration quality but raises computation complexity, requiring approximately 298 seconds per generated image.

D.5 Results on Composite Degradation.

The severe DIV2K dataset tests diffusion-based restoration under extra noise and composite degradations, which cause substantial performance deterioration.

  • The DIV2K severe dataset combines severe noise, blur, JPEG compression artifacts, and downsampling to evaluate diffusion methods under composite degradation.
  • Extra noise and composite degradations cause severe performance drops; SUPIR achieves 27.19dB PSNR with simple bicubic degradation.

D.6 More visual results

Additional visual comparisons examine zero-shot diffusion models across deblurring and inpainting, with differences in structural recovery and perceptual fidelity.

  • The visual-results section presents diffusion-based restoration outputs across multiple tasks, covering Figures 15 through 19.
  • Figure 15 compares zero-shot models on deblurring, while Figure 16 presents their image-inpainting results.
  • DiffPIR excels among the compared models, especially in recovering structural information, whereas DPS provides perceptually strong but lower-fidelity restorations.

E Comparisons with other surveys on diffusion models

Compared with existing diffusion and restoration surveys, this paper positions itself as an early, comprehensive, restoration-centric review with cross-task quantitative analysis.

  • Contributions: The paper presents itself as the earliest diffusion-based survey dedicated specifically to image restoration and enhancement.It was released in August 2023 and includes an actively maintained repository.
  • Evaluation: The survey includes quantitative comparison tables for inpainting, severe mixed degradation, and deblurring.
  • Comparisons with other surveys: The survey balances general-purpose diffusion reviews with single-task restoration surveys by providing a unified, restoration-focused perspective.
  • Scope: It covers super-resolution, motion deblurring, JPEG artifact removal, deraining/dehazing, low-light enhancement, blind real-world restoration, and inpainting.
  • Cross-task quantization analysis: The paper provides cross-task quantitative analysis rather than restricting evaluation to a single restoration task.
Loading 2308.09388v3…