PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffusion Models for Image Restoration and Enhancement: A Comprehensive Survey

Xin Li, Yulin Ren, Xin Jin, Cuiling Lan, Xingrui Wang, Wenjun Zeng, Xinchao Wang, Zhibo Chen

arXiv:2308.09388v3cs.CV

TL;DR

Image restoration methods can achieve strong objective quality yet still produce unsatisfactory textures, while diffusion-based IR lacks a comprehensive synthesis. This survey organizes diffusion-based IR workflows, techniques, evaluations, and real-world applications, finding differing strengths across subjective and objective metrics and identifying key research directions.

  • Problem

    Conventional image restoration can achieve strong PSNR and SSIM yet still generate unsatisfactory textures, limiting real-world application.

    Source: §1 · p. 2

  • Method

    The survey categorizes supervised, zero-shot, and blind/real-world diffusion-based IR methods, and summarizes their techniques, datasets, metrics, and benchmarks.

    Source: §1 · p. 2, §5 · p. 14, §7 · p. 21

  • Results

    DiffLL and LightenDiffusion achieve optimal subjective generalization on real-world low-light restoration, while LightenDiffusion has lower objective quality than DiffLL.

    Source: §4.5 · p. 13

  • Takeaways & Limitations

    The survey identifies sampling efficiency, compression, distortion simulation and estimation, distortion-invariant learning, and framework design as future directions for diffusion-based IR.

    Source: §6.2 · p. 18, §6.3 · p. 19, §6.4 · p. 19

    Comparisons remain difficult because diffusion-based methods sometimes use different datasets for the same restoration task.

    Source: §4.5 · p. 14

Abstract

from arXiv · show

Image restoration (IR) has been an indispensable and challenging task in the low-level vision field, which strives to improve the subjective quality of images distorted by various forms of degradation. Recently, the diffusion model has achieved significant advancements in the visual generation of AIGC, thereby raising an intuitive question, "whether diffusion model can boost image restoration". To answer this, some pioneering studies attempt to integrate diffusion models into the image restoration task, resulting in superior performances than previous GAN-based methods. Despite that, a comprehensive and enlightening survey on diffusion model-based image restoration remains scarce. In this paper, we are the first to present a comprehensive review of recent diffusion model-based methods on image restoration, encompassing the learning paradigm, conditional strategy, framework design, modeling strategy, and evaluation. Concretely, we first introduce the background of the diffusion model briefly and then present two prevalent workflows that exploit diffusion models in image restoration. Subsequently, we classify and emphasize the innovative designs using diffusion models for both IR and blind/real-world IR, intending to inspire future development. To evaluate existing methods thoroughly, we summarize the commonly-used dataset, implementation details, and evaluation metrics. Additionally, we present the objective comparison for open-sourced methods across three tasks, including image super-resolution, deblurring, and inpainting. Ultimately, informed by the limitations in existing works, we propose five potential and challenging directions for the future research of diffusion model-based IR, including sampling efficiency, model compression, distortion simulation and estimation, distortion invariant learning, and framework design.

1 Introduction

Diffusion models offer high-fidelity, diverse generation that may address texture limitations in image restoration, but existing approaches face practical constraints and lack a comprehensive review. This survey organizes diffusion-based image restoration methods, benchmarks, and future challenges across supervised, zero-shot, and blind/real-world settings.

Motivation

  • Diffusion-based image restoration is motivated by conventional and GAN-based methods’ limited texture realism, including blurry, fake, or counterfactual textures despite improved objective quality.CNN- and Transformer-based methods improve metrics such as PSNR and SSIM, while GAN-based methods remain vulnerable to mode corruption and unstable optimization.Source: §1 · p. 2

Diffusion-model background

  • Diffusion models progressively add noise and reverse it through denoising, producing high-fidelity and diverse generations that motivate their application to image restoration.The survey describes score estimation [71] and noise prediction [70] as alternative reverse-process mechanisms, and contrasts diffusion models with GANs.Source: §1 · p. 2

Diffusion-based IR taxonomy

Blind/real-world IR

  • Blind and real-world restoration extends diffusion models through distortion simulation, kernel estimation, domain translation, and distortion-invariant learning to address diverse, unknown degradations [100] [101] [102] [103] [104] [105].These techniques target limitations that impede supervised and zero-shot methods in real-world scenarios.Source: §1 · p. 2

Survey scope

  • The survey provides the first comprehensive review of diffusion-based image restoration, organizing methods, practical benchmarks, cross-task comparisons, and future challenges to support further development.It covers diffusion-model foundations, supervised and zero-shot IR, blind/real-world degradation, comparisons across image SR, inpainting, deblurring, and real-world restoration, and future directions [70] [72].Source: §1 · pp. 2–3

2 Background on Diffusion Model (DM)

Diffusion models convert unstable generation into stable reverse processes through Markov-chain modeling, with DDPM and SDE providing two commonly used formulations for image restoration. DDPM progressively adds Gaussian noise and learns denoising reversals, while SDE unifies continuous diffusion and reverse-time score-based sampling.

2 Background on Diffusion Model (DM)

  • Diffusion models include DDPM [70], NCSNs [71], and SDEs [72], with SDEs showing that DDPM and NCSNs are discretizations of a unified continuous formulation.The survey focuses on DDPM and SDE strategies for image restoration and discusses NCSN [71] in the supplementary material.Source: §2 · p. 3

2.1 Denoising Diffusion Probabilistic Model

  • DDPM [70] progressively corrupts training data with scheduled Gaussian noise until it approaches a standard Gaussian distribution, then learns a reverse denoising process to recover the data distribution.Its fixed variance schedule means the denoising network primarily approximates the reverse-process mean.Source: §2.1 · p. 3

2.2 Stochastic Differential Equations (SDEs)

  • SDEs [72] unify score-based and diffusion probabilistic modeling through continuous stochastic diffusion, parameterized by drift and diffusion coefficients.The diffusion coefficient controls random perturbation, while the drift coefficient can enforce Gaussian distributions such as those associated with DDPM [70] and NCSN [71].Source: §2.2 · pp. 3–4
  • The reverse-time SDE samples data from noise by estimating the score function with a neural network and solving the resulting equation through score matching.DDPM and NCSN correspond to discretizations of different SDEs, specifically variance-preserving and variance-exploding forms.Source: §2.2 · p. 4

3 Diffusion model-based Image Restoration Methods

This section surveys diffusion-based image restoration methods, covering both supervised approaches trained on paired data and zero-shot approaches that reuse pretrained diffusion priors. It organizes representative designs by conditioning, diffusion space or process, inverse-problem strategy, and posterior estimation techniques for guiding restoration.

3 Diffusion model-based Image Restoration Methods

  • Supervised methods train diffusion models with paired distorted/clean images, while zero-shot methods use pretrained structure and texture priors without retraining, trading data requirements against prior-extraction challenges.Supervised methods condition reverse diffusion on distorted images but require substantial paired data; zero-shot methods require only distorted images and must preserve their structure while extracting perceptual priors.Source: §3 · p. 4

3.1 SR3 – Representative Supervised DM for IR

  • SR3 concatenates an upsampled low-resolution image with the denoised state in a DDPM U-Net, predicting noise iteratively until an upsampled high-quality image is produced.Its objective is to learn pθ(x_t−1|y, x_t), with x_0 approximating the corresponding high-quality image.Source: §3.1 · pp. 4–5

3.2 ILVR – Representative Zero-shot DM for IR

  • ILVR enables training-free restoration by replacing denoised low-frequency components with those from a diffused reference image, preserving structure and semantic consistency during sampling.Most subsequent zero-shot methods refine this sampling process rather than retraining the diffusion model.Source: §3.2 · p. 6

3.3 Supervised DM-based IR.

  • Supervised restoration methods improve conditioning through low-quality references, preprocessing, or revised diffusion processes, and reduce generation burden through image-, residual-, or latent-space modeling.Direct references include patch-wise conditioning for arbitrary-size restoration and cascaded diffusion for higher-resolution generation; preprocessing supplies enhanced images, degradation features, or physics-aware conditions.Source: §3.3 · p. 6, §3.3.1 · p. 6, §3.3.2 · p. 7

3.3.3 Condition by Revising Diffusion Process

  • Revised diffusion processes start reverse sampling from noisy degraded images, while residual diffusion generates only differences between low- and high-quality images, reducing unnecessary structure and texture regeneration.Examples include mean-reverting SDE degradation modeling, interpolation-based multi-step restoration, frequency-space diffusion, predict-and-refine deblurring, and residual shifting with as few as 15 sampling steps.Source: §3.3.3 · pp. 7–8, §3.3.4 · p. 8

3.3.5 Diffusion on Latent Space

  • Latent-space diffusion uses autoencoder representations to reduce computational complexity, while transfer-learning methods adapt pretrained diffusion models by injecting distortion information through encoders, zero convolutions, or ControlNet-like modules.Refusion accelerates restoration through latent diffusion, whereas StableSR, DiffBIR, PASD, and SUPIR efficiently fine-tune pretrained generative models.Source: §3.3.5 · pp. 8–9, §3.3.6 · p. 9

3.4 Zero-shot DM-based IR.

  • Zero-shot methods are categorized as projection, decomposition, or posterior-estimation approaches that seek data consistency and perceptual quality using unconditional pretrained diffusion models.Projection methods replace known components during sampling, while decomposition methods separate measurement-consistent and perceptually realistic subspaces; DDNM uses range-null decomposition, and DDRM uses SVD for linear inverse problems.Source: §3.4 · p. 9, §3.4.1 · p. 9, §3.4.2 · p. 10

3.4.3 Posterior estimation.

  • Posterior-estimation methods address noisy linear inverse problems by estimating p(x_t|y) during reverse diffusion, making accurate likelihood estimation p(y|x_t) central to improved solvers.They respond to projection methods’ limitations, including unsatisfactory consistency when projections move sample paths off the data manifold.Source: §3.4.3 · p. 10
  • MCG enforces data consistency through a manifold-constrained gradient, whereas DPS [96] removes MCG’s projection because it can move sampling off the data manifold.Both approximate p(y|x_t) using p(y|x̂_0), with x̂_0 estimated via Tweedie’s formula.Source: §3.4.3 · p. 11
  • ΠGDM generalizes posterior guidance to linear, nonlinear, and differentiable inverse problems using a Moore-Penrose pseudoinverse.Its denoising pipeline applies pseudo-inverse guidance at every step to encourage consistency with degraded image y.Source: §3.4.3 · p. 11, Figure 10 · p. 11
  • GDP [120] heuristically models p(y|x_t) through distance and quality losses, replacing noisy x_t with clean estimate x̂_0 because their noise magnitudes differ.The scaling factor s controls guidance weight, while L denotes the distance metric and Q the quality loss.Source: §3.4.3 · p. 11
  • Copaint [134] predicts x̂_0 with a one-step neural network, while Feng et al. estimate p(x_t|y) variationally using Real-NVP and KL-divergence minimization.These approaches avoid directly modeling the intractable p(y|x_t).Source: §3.4.3 · p. 11

4 Diffusion Models for Blind/Real-world Image Restoration

Diffusion-based blind and real-world restoration addresses unknown, diverse degradations through distortion simulation, kernel estimation, domain translation, and distortion-invariant modeling. Applications demonstrate specialized conditioning and efficient designs, while evaluation highlights generalization gains alongside limitations in comparability, efficiency, and pixel-wise consistency.

4 Diffusion Models for Blind/Real-world Image Restoration

  • Blind/real-world restoration methods are organized into distortion simulation, kernel estimation, domain translation, and distortion-invariant diffusion models [100] [202] [203] [103] [204] [205] [87] [105] [101] [102] [88].These methods target unknown degradation modes and the scarcity of distorted/clean image pairs that hinder synthetic-distortion approaches.Source: §4 · p. 11

4.1 Distortion Simulation

  • Distortion simulation improves robustness by modeling realistic degradations, as SR3+ [100] uses RealESRGAN’s second-order simulation and Yang et al. [205] synthesize real-world training pairs with diffusion models.Real-world blind distortions differ from simple synthetic distributions, so supervised restoration requires stronger generalization or broader simulated coverage.Source: §4.1 · pp. 11–12

4.2 Kernel Estimation

  • BlindDPS [102] estimates degradation kernels with a parallel diffusion model, whereas GibbsDDRM [101] jointly samples kernel parameters and images from p(x_t|k, y) using a partially collapsed Gibbs sampler.The degradation is modeled as y = (x∗k) ↓s +n, with the estimated kernel guiding restoration adaptability.Source: §4.2 · p. 12

4.3 Domain Translation

  • Domain translation bridges synthetic, real-world distorted, and high-quality domains: Yang et al. [205] generate translated training pairs, while RainDiffusion [104] uses cycle-consistent unpaired translation and multi-scale diffusion refinement.The first line translates synthetic low-quality images into the real-world domain; the second uses two generators to connect unpaired distorted and clean images.Source: §4.3 · p. 12

4.4 Distortion-invariant Diffusion Model

  • Distortion-invariant methods derive robust sampling guidance from restoration networks, degradation-augmented representations, or low-pass structure, then optionally enhance the degradation-free output [105] [203].DifFace et al. starts sampling from an initial clean estimate, Ren et al. [203] uses multi-scale invariant guidance, and DR2 [105] performs iterative degradation removal before enhancement.Source: §4.4 · pp. 12–13, Figure 11 · p. 13

4.5 Discussion on Some Real-world Application and Limitation.

  • DiffLL and LightenDiffusion achieve optimal NIQE and PI generalization on low-light benchmarks, although LightenDiffusion has lower objective quality than DiffLL.Other applications use degradation-aware, wavelet, multi-scale, Retinex, physical, frequency, or attention-based conditioning; current limitations include inconsistent comparisons, slow sampling, task irrelevance, and hallucinated textures.Source: §4.5 · pp. 13–14, Table 1 · p. 14

5 Experiments

The experiments organize datasets, implementation settings, metrics, and benchmark comparisons for diffusion-based image restoration. Evaluations cover super-resolution, deblurring, inpainting, real-world restoration, and related degradation-removal tasks using objective quality and complexity measures.

5 Experiments

  • The study summarizes datasets, experimental configurations, and evaluation metrics before comparing diffusion-based image restoration benchmarks across several tasks.The benchmark scope includes super-resolution, inpainting, deblurring, and JPEG artifact removal.Source: §5 · p. 14

5.1 Datasets and Implementation Details

  • Datasets are organized by restoration task, covering super-resolution, deblurring, inpainting, shadow removal, desnowing, draining, and dehazing.These commonly used datasets are summarized in Supplementary Table 3.Source: §5.1 · p. 14
  • Implementation summaries distinguish supervised and zero-shot methods, reporting training and inference configurations, pretrained models, sampling steps, and rotation or flip augmentation.Supplementary Tables 4 and 5 collect the supervised and zero-shot settings, respectively.Source: §5.1 · p. 14

5.2 Experimental Results

  • Objective comparisons evaluate super-resolution, deblurring, and inpainting with PSNR, SSIM, FID, and LPIPS, alongside runtime, parameters, and FLOPs.Subjective comparisons additionally cover deblurring, inpainting, real-world restoration, and severe composite degradation; super-resolution results include supervised and zero-shot tables.Source: §5.2 · pp. 14–15, Table 2 · p. 15, Table 3 · p. 15

6 Challenges and Future Directions

The survey identifies practical barriers to diffusion-based image restoration, including efficiency, complexity, robustness, real-world degradation handling, and controllability. It proposes directions spanning faster and smaller models, distortion-aware learning, new frameworks, all-in-one restoration, agentic systems, semantic prompting, and human preference alignment.

6.1 Sampling Efficiency

  • Sampling efficiency remains a central bottleneck: SR3 [80] needs about 50 seconds for one 224 × 224 image, while DDRM [94] reduces this to 8 seconds using DDIM [73].Approaches include non-Markov diffusion, efficient ODE solvers, knowledge distillation, cross-modality priors, low-quality-image initialization, and one-step methods such as SinSR, which achieves up to 10× speedup while preserving comparable perceptual quality.Source: §6.1 · pp. 17–18

6.2 Model Compression

  • Diffusion models remain too large for real-time restoration: DDPM [70] and SR3 [80] contain 113.7M and 155.3M parameters, motivating pruning, quantization, distillation, and low-rank compression.Existing compression studies include block removal, partial-step pruning, and quantization, but diffusion-based image restoration remains under-explored for efficient deployment.Source: §6.2 · pp. 18–19

6.3 Distortion Simulation and Estimation

  • Real-world and blind restoration requires better degradation simulation and estimation because paired data are scarce, handcrafted distortions are incomplete, and zero-shot diffusion methods usually assume identifiable synthetic degradation modes.Domain translation can create real-world training pairs or transfer real degraded images into synthetic domains, while distortion-adaptive learning and inverse-problem modeling aim to identify degradation types or levels.Source: §6.3 · p. 19

6.4 Distortion Invariant Learning

  • Distortion invariant learning seeks representations that generalize across unknown degradation types and levels while preserving structural and textual information for reconstruction.Proposed strategies treat distortion modes as domains and use domain alignment, distortion augmentation, or meta-learning; diffusion models can also redesign noise predictors or condition modules, as in DifFace and DR2 [105].Source: §6.4 · pp. 19–20

6.5 Framework Design

  • Framework design remains open across conditioning, generation space, and noise prediction, balancing fidelity, spatial and textual information, computational cost, and architectural flexibility.Current methods largely use DDPM [70]-style U-Nets; image, residual, latent, and frequency spaces offer different trade-offs, while transformer-based predictors and unified architectures such as the painter require further study.Source: §6.5 · p. 20

6.6 All-in-one Restoration

  • All-in-one restoration must adapt to diverse and hybrid degradations while sharing restoration knowledge, using degradation-aware or semantic conditioning, quality assessment, LoRA task embeddings, and generated distortions.DACLIP, MPerceiver, Diff-Restore [278], AutoDIR, UIR-LoRA, and GenDeg [280] motivate efficient architectures and multi-task paradigms such as Mixtures-of-Experts.Source: §6.6 · p. 20

6.7 Restoration Systems with LLM/MLLM-based Agents.

  • LLM/MLLM-based restoration agents use language-driven analysis and reasoning to dynamically select restoration tools, while MAIR and 4K-Agent integrate diffusion models and 4K-Agent uses sequential tool invocation under compound, real-world degradations.Source: §6.7 · p. 21

6.9 Human-in-the-Loop Optimization and Preference Alignment.

  • Human-in-the-loop optimization is an emerging direction for aligning diffusion-based restoration with perceptual preferences through reward learning, instruction control, and patch-level preference optimization.PrefPaint, InstructRestore [292], and DSPO [293] motivate RLHF, synthetic preference data, and region-aware restoration under natural-language control.Source: §6.9 · p. 21

7 Conclusions … B.2 Improvements with Diffusion Model

The paper surveys diffusion models for image restoration, organizing methods by training strategy and degradation scenario while comparing open-source methods across super-resolution, deblurring, and inpainting. It also reviews diffusion-model foundations and improvements in optimization, sampling, architecture, and conditioning, identifying future research directions.

7 Conclusions

  • The survey categorizes diffusion-based image restoration into supervised, zero-shot, and blind/real-world workflows, evaluating open-source methods on super-resolution, deblurring, and inpainting.It summarizes datasets and metrics and highlights nine potential future directions.Source: §7 · p. 21

B.1 Noise Conditioned Score Networks (NCSNs)

  • NCSNs estimate the data-distribution score function with a neural network and use annealed Langevin dynamics to improve score estimation across noise levels.Noise perturbation avoids collapse toward a low-dimensional manifold and inaccurate estimation in low-density regions.Source: §B.1 · pp. 40–41

A Timeline

  • Figure 13 chronologically tracks representative diffusion-model works in text-to-image synthesis, image restoration, and general image generation.The figure distinguishes text-to-image, restoration, and general-generation applications by color.Source: A Timeline · p. 40, Figure 13 · p. 41

B Background

  • Diffusion-model improvements address optimization strategy, sampling efficiency, model architecture, and conditional strategy.These developments build on the foundational diffusion models reviewed in the background.Source: §B.2 · p. 41

B.2 Improvements with Diffusion Model

  • Optimization studies improve diffusion stability and performance by optimizing variance and noise schedules in forward and backward processes.The surveyed works explore schedule design among other diffusion-model optimization choices.Source: §B.2 · p. 41
  • Diffusion sampling is accelerated through ODE-based solvers, revised diffusion processes, knowledge distillation, and condition strategies.An ODE-based super-resolution method reduces high-quality generation to 10 to 20 sampling steps.Source: §B.2 · pp. 41–42
  • Diffusion architectures have evolved from CNN-based U-Nets toward Transformer-based models that better capture long-range dependencies and unify modalities.U-Net refinements include cross attention, normalization, multi-head attention, and position encoding; Transformers receive time-step and condition inputs through adaptive normalization or cross attention.Source: §B.2 · p. 42
  • Conditional diffusion uses classifier guidance, classifier-free conditioning, and multimodal inputs such as images, segmentation maps, and latent features.Classifier-free conditioning uses a null token to preserve unconditional generation while supporting text and other conditions.Source: §B.2 · p. 42

C Methods

This section categorizes diffusion-based image restoration methods into supervised and zero-shot workflows, unified multi-task models, all-in-one restoration, and task-specific methods for weather removal, deblurring, and shadow removal.

Diffusion-based Image SR

  • Diffusion-based super-resolution methods progress from image- or residual-space diffusion to deterministic sampling, continuous-resolution representations, semantic conditioning, and pretrained-model adaptation.SR3 [80] and SRDiff [81] initiated diffusion-based SR; later methods use CNN preprocessing, implicit neural representations, SAM conditions, adaptors, ControlNet-like encoders, SDXL, LLaVA, and multimodal semantics.Source: C Methods · pp. 42–43

Diffusion-based Unified Model for Multiple Single IR Tasks

  • Unified diffusion models address independent restoration tasks—including super-resolution, deblurring, inpainting, deraining, dehazing, and desnowing—through supervised or zero-shot frameworks.Palette formulates restoration as image-to-image translation, while IR-SDE, Refusion [85], InDI [164], Resshift, DiffIR, and C2F-DFT [136] modify diffusion processes, priors, architectures, or sampling for unified restoration.Source: C Methods · pp. 43–44

Diffusion-based All-in-one Image Restoration

  • All-in-one restoration models condition diffusion on degradation and semantic representations to handle multiple degradations within one model.DACLIP[158], MPerceiver, Diff-Restore [278], AutoDIR, UIR-LoRA, and GenDeg [280] use CLIP features, quality assessment, low-rank adaptation, dynamic weighting, or generated distortions.Source: C Methods · p. 44

Diffusion-based Image Weather Removal

  • Diffusion-based weather-removal methods cover deraining, dehazing, and desnowing using patch-noise estimation, degradation-specific prompts, multibranch perception, unpaired consistency, physical modeling, and frequency filtering.T3-Diffusion is reported to outperform DA-CLIP and WeatherDiff; RainDiffusion [226] removes the need for paired training data through an unpaired cycle-consistency framework.Source: C Methods · p. 44

Diffusion-based Image Deblurring

  • Diffusion-based deblurring methods estimate hierarchical restoration priors or blind blur kernels, with GibbsDDRM [101] jointly sampling kernel parameters and images through a partially collapsed Gibbs sampler.HI-Diff [112] guides a transformer with hierarchical compact priors, while BlindDPS [102] uses a parallel diffusion model pretrained on synthetic kernels for kernel estimation.Source: C Methods · p. 44

Diffusion-based Image Shadow Removal

  • Diffusion-based shadow removal extracts degradation priors or uses adaptive attention to separate objects from shadows and condition shadow-free image generation.ShadowDiffusion refines a generated shadow mask using transformer-derived degradation features, whereas DeS3 is mask-free and is described as achieving better performance than ShadowDiffusion.Source: C Methods · p. 44

D Experiments · D.1 Datasets

The survey organizes diffusion-based image-restoration datasets by task and training paradigm, emphasizing that degradation diversity and dataset size affect evaluation and diffusion-model performance. It covers supervised and zero-shot restoration across super-resolution, deblurring, inpainting, and several adverse-weather and real-world tasks.

D.1 Datasets

  • Table 7 organizes datasets by restoration task, reporting release year, training and testing sample counts, descriptions, and differing degradation modes.The surveyed tasks include super-resolution, inpainting, deblurring, denoising, shadow removal, desnowing, deraining, and dehazing.Source: §D.1 · p. 46
  • For traditional super-resolution, DIV2K and Flick2K are standard training datasets, while diffusion performance is bounded by dataset size.SR3 [80] therefore trains on ImageNet for natural-image super-resolution.Source: §D.1 · p. 46
  • Super-resolution evaluation uses ImageNet 1K and CelebA-HQ for natural-image and face restoration, alongside Set5, Set14, BSD100, Manga109, and Urban100.SR3+ [80] uses DF2K+OST variants for training and RealSR and DRealSR for real-world evaluation.Source: §D.1 · p. 46
  • Deblurring methods commonly train on GoPro [45] and evaluate on GoPro, RealBlur-J, REDS, and HIDE.Shadow removal uses ISTD and SRD, with ISTD containing 135 scenes and shadow masks.Source: §D.1 · p. 46
  • Dehazing, desnowing, and deraining rely on task-specific datasets spanning synthetic and real conditions, including Haze-4K, Dense-Haze, RESIDE, CSD, Snow100k, and SRRS.Deraining datasets represent varied rain phenomena, including streaks, raindrops, accumulation, and real-world rain captured across continuous rainy images.Source: §D.1 · p. 47
  • The survey divides diffusion-restoration datasets into supervised-learning and zero-shot-restoration categories, with Tables 8 and 9 detailing their training and testing sets.The listed datasets cover super-resolution, deblurring, deraining, denoising, inpainting, JPEG artifact removal, desnowing, and real-world restoration.Source: §D.1 · p. 47

D.2 Evaluation Metrics

The section reviews objective and subjective metrics for evaluating diffusion-model image restoration, emphasizing objective measures because subjective assessment is time-consuming and resource-intensive. It covers pixel-level fidelity, perceptual similarity, generated-image quality, blind assessment, and super-resolution perception.

Reference-based metrics

  • PSNR measures pixel-wise distortion through MSE, while SSIM evaluates contrast, brightness, and structure similarity.PSNR and SSIM assess data consistency against clean or high-quality reference images, but do not fully capture perceptual quality.Source: §D.2 · pp. 47, 49

Perceptual metrics

  • LPIPS and DISTS use pretrained neural features to measure perceptual, texture, and structural similarity, with lower LPIPS indicating greater similarity.LPIPS is designed to align more closely with human perception, while DISTS compares feature-based texture and structure.Source: §D.2 · pp. 47–49

Generation and blind metrics

  • FID and KID [416] evaluate generated-image fidelity and diversity, while NIQE provides blind quality assessment and PI targets super-resolution perceptual quality.KID [416] uses maximum mean discrepancy with a polynomial kernel and is more stable than FID with few samples; PI = 0.5((10 −Ma) + NIQE).Source: §D.2 · pp. 47–48

Overview

  • Objective metrics are widely used because subjective image evaluation requires substantial time and human resources.The section distinguishes objective and subjective image-quality assessment methods.Source: §D.2 · pp. 47–48

D.3 Implementation details

This section summarizes implementation settings for supervised and zero-shot diffusion-model image restoration, including datasets, optimization parameters, pretrained models, and sampling steps. It also describes datasets and patch-based strategies used across restoration tasks and for arbitrary image sizes.

Implementation summaries

  • Tables 8 and 9 summarize supervised and zero-shot implementation details, respectively, covering datasets, batch size, iterations, learning rate, pretrained models, and sampling steps.For supervised methods, the summaries distinguish training and testing data and report optimization and inference settings; zero-shot methods are organized by testing datasets, pretrained models, and sampling steps.Source: §D.3 · p. 49, Table 8 · p. 50, Table 9 · p. 50

Image Restoration with Arbitrary Size

  • Patch-based restoration enables arbitrary-size diffusion inference by denoising overlapping patches and averaging or smoothing their overlap noise to reduce edge mismatches.DG-DPM [203] handles arbitrary sizes with fully convolutional layers but is computationally expensive, whereas Weather-diff [83], GDP [120], and StableSR [103] use patch-based strategies whose overlaps are nearly indistinguishable in Fig. 14.Source: §D.3 · p. 49

Super-resolution

  • Super-resolution implementations use DIV2K and Flick2K for conventional training, while SR3 [80] scales training to ImageNet or FFHQ and evaluates on corresponding natural-image or face datasets.Real-world SR3+ [100] uses DF2K and OST, with an expanded version adding 61M in-house images, and evaluates on RealSR and DRealSR.Source: §D.3 · p. 49

Other restoration tasks

  • Deblurring, shadow removal, dehazing, desnowing, and deraining rely on task-specific datasets including GoPro [45], ISTD, Haze-4K, CSD, and Rain100H [41].Additional evaluations use RealBlur-J, REDS, HIDE, SRD, Dense-Haze, RESIDE, Snow100k, SRRS, Rain100L [41], Rain800, DDN-Data, RainDrop [42], Outdoor-Rain, and SPA-data.Source: §D.3 · pp. 49–50

D.4 Results on Image Inpainting.

Five zero-shot diffusion models are evaluated for narrow-mask image inpainting, revealing trade-offs among distortion metrics, perceptual quality, and sampling efficiency.

Results on Image Inpainting

  • The comparison evaluates DPS, DDRM, DDNM, RePaint, and CoPaint on the narrow-mask image-inpainting task.The five zero-shot models include three multitask methods and two methods specifically designed for inpainting.Source: §D.4 · p. 50
  • CoPaint outperforms RePaint perceptually, reducing FID by 0.08dB on CelebA-HQ and 1.33dB on ImageNet through Bayesian posterior-based coherence modeling.CoPaint also uses DDNM’s time-travel method, improving restoration quality but increasing computational complexity.Source: §D.4 · pp. 50–51
  • DDRM and DDNM achieve better distortion metrics, while DPS performs better perceptually on CelebA-HQ than on ImageNet.RePaint and CoPaint outperform the other models in perceptual quality.Source: §D.4 · p. 50
  • DDRM generates images fastest because its low sampling-step count keeps runtime nearly linearly correlated with NFE.CoPaint takes approximately 298 seconds per image despite using 250 steps, while RePaint requires a similar duration.Source: §D.4 · p. 51

D.5 Results on Composite Degradation.

On the DIV2K severe dataset, combining severe noise, blur, JPEG artifacts, and downsampling causes substantial performance degradation. Diffusion-based RealSR methods nevertheless generalize better to these unseen composite degradations than other diffusion-based image-restoration methods.

D.5 Results on Composite Degradation

  • SUPIR drops from 27.19dB PSNR under bicubic degradation to 19.06dB under severe composite degradation.The DIV2K severe dataset follows and combines severe noise, blur, JPEG compression artifacts, and downsampling.Source: §D.5 · p. 51
  • Diffusion-based RealSR methods show greater generalization capability than other diffusion-based image-restoration methods under extra noise and composite degradations.Their robustness is attributed to distortion simulation or training on more collected datasets, which reduces bias toward unseen degradations.Source: §D.5 · p. 51

D.6 More visual results

The section presents qualitative results of diffusion-based image restoration methods across deblurring, inpainting, and efficient transfer learning. DiffPIR excels in structural recovery, SUPIR provides superior texture details, while StableSR and DiffBIR favor fidelity.

Overview

  • Figures 15–19 provide qualitative comparisons of diffusion-based image restoration methods across multiple tasks.The visual results cover image deblurring, image inpainting, and diffusion-model-based efficient transfer learning.Source: §D.6 · p. 51

Zero-shot restoration

  • DiffPIR excels among zero-shot models in deblurring and inpainting, especially for recovering structural information.DPS [96] produces perceptually pleasing restorations but has lower fidelity.Source: §D.6 · p. 51

Efficient transfer learning

  • Among efficient transfer-learning models, SUPIR achieves superior texture restoration details, whereas StableSR [103] and DiffBIR have advantages in fidelity.The compared models include StableSR [103], DiffBIR, SUPIR, and PASD.Source: §D.6 · p. 51

E Comparisons with other surveys on diffusion models

This survey positions itself as the earliest systematic, task-specific review of diffusion models for image restoration and enhancement. It combines broad restoration coverage with cross-task analysis, addressing gaps in general-purpose and single-task surveys.

Contribution and scope

  • The paper presents the earliest diffusion-based survey dedicated specifically to image restoration and enhancement, released in August 2023, alongside an actively maintained GitHub repository.The repository had exceeded 790 stars at the time described.Source: E Comparisons with other surveys on… · p. 51

Comprehensive but restoration-centric survey

  • Unlike general-purpose surveys and single-application surveys, this work provides a unified, restoration-centric review of diffusion models.It covers super-resolution, motion deblurring, JPEG artifact removal, deraining/dehazing, low-light enhancement, blind real-world restoration, and image inpainting, while summarizing common techniques and theoretical principles.Source: E Comparisons with other surveys on… · p. 52

Cross-task quantization analysis

  • The survey adds cross-task quantization analysis, whereas existing image-restoration and super-resolution surveys [432] often lack experiments or focus on a single task.This comparison distinguishes the survey’s evaluation-oriented scope from prior narrower reviews.Source: E Comparisons with other surveys on… · p. 52

Related papers

Loading 2308.09388v3…