PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Diffusion Prior for Unified Image Restoration and Enhancement

Ben Fei, Zhaoyang Lyu, Liang Pan, Junzhe Zhang, Weidong Yang, Tianyue Luo, Bo Zhang, Bo Dai

arXiv:2304.01247v1cs.CV

TL;DR

Image restoration spans known linear degradations and nonlinear or blind cases with unknown degradation models. GDP uses a pretrained diffusion prior for unified restoration, and on low-light enhancement and HDR recovery it outperforms zero-shot baselines qualitatively and quantitatively.

  • Problem

    Image restoration spans known linear degradations and nonlinear or blind cases with unknown degradation models, where blind recovery requires estimating both degradation and the clean image.

    Source: §1 · pp. 1–2

  • Method

    GDP uses a single ImageNet-pretrained unconditional DDPM as an unsupervised prior, with degraded-image guidance and trainable degradation parameters for unified restoration.

    Source: §1 · p. 2

  • Results

    On low-light enhancement and HDR recovery, GDP is qualitatively and quantitatively superior to other zero-shot baselines, including on images outside ImageNet's training distribution.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The study concludes that GDP can restore arbitrary-size images across linear, nonlinear, and blind problems.

    Source: §7 · p. 8

    GDP's extended sampling time limits its applicability to real-time applications and weak end-user devices such as mobile devices.

    Source: A. Limitations and Future works · p. 14

Abstract

from arXiv · show

Existing image restoration methods mostly leverage the posterior distribution of natural images. However, they often assume known degradation and also require supervised training, which restricts their adaptation to complex real applications. In this work, we propose the Generative Diffusion Prior (GDP) to effectively model the posterior distributions in an unsupervised sampling manner. GDP utilizes a pre-train denoising diffusion generative model (DDPM) for solving linear inverse, non-linear, or blind problems. Specifically, GDP systematically explores a protocol of conditional guidance, which is verified more practical than the commonly used guidance way. Furthermore, GDP is strength at optimizing the parameters of degradation model during the denoising process, achieving blind image restoration. Besides, we devise hierarchical guidance and patch-based methods, enabling the GDP to generate images of arbitrary resolutions. Experimentally, we demonstrate GDP's versatility on several image datasets for linear problems, such as super-resolution, deblurring, inpainting, and colorization, as well as non-linear and blind issues, such as low-light enhancement and HDR image recovery. GDP outperforms the current leading unsupervised methods on the diverse benchmarks in reconstruction quality and perceptual quality. Moreover, GDP also generalizes well for natural images or synthesized images with arbitrary sizes from various tasks out of the distribution of the ImageNet training set.

1. Introduction

GDP uses a pretrained DDPM as an unsupervised prior for unified image restoration and enhancement across linear, nonlinear, and blind problems. It optimizes unknown degradation parameters during denoising and combines conditional guidance with hierarchical, patch-based generation for arbitrary-resolution images.

1. Introduction

  • Real-world images can have multiple complex degradations, while supervised restoration typically assumes a known linear degradation; DDRM [32] remains limited to known linear problems through SVD.The introduction motivates unsupervised generative priors as a way to address different degradation models at inference without retraining.Source: §1 · pp. 1–2
  • GDP uses a single unconditional ImageNet-pretrained DDPM for unsupervised restoration and enhancement across linear, nonlinear, and blind problems.The paper presents GDP as a unified solver that produces diverse, high-fidelity outputs.Source: §1 · p. 2
  • GDP optimizes randomly initialized degradation parameters during denoising and guides generation through an intermediate clean-image prediction from the noisy input.The pretrained DDPM prior is described as regularizing recovered-image realness and fidelity.Source: §1 · p. 2
  • Hierarchical guidance and patch-based generation use predicted low-resolution images and degradation models to guide high-resolution outputs of arbitrary size.The paper identifies these methods as enabling arbitrary-resolution generation.Source: §1 · p. 2
  • GDP is evaluated on ImageNet, LSUN, and CelebA for linear or multi-linear problems using consistency and FID, and outperforms zero-shot baselines on low-light enhancement and HDR recovery.The low-light and HDR results are reported as qualitative and quantitative, and the paper states that ImageNet-trained GDP works beyond its training-set distribution.Source: §1 · p. 2

2. Related works

Prior work has addressed linear, nonlinear, and blind restoration with different models and strategies, but the supplied passages identify gaps in multi-task linear restoration, diffusion-based nonlinear restoration, and unified blind restoration. GDP is presented as a single-model approach spanning these problem types.

Linear Inverse Image Restoration

  • GDP is proposed as a single model for single or multiple linear inverse tasks, addressing a relatively under-explored setting beyond prior unconditional diffusion and task-specific multi-task approaches.Prior diffusion methods commonly use unconditional models [53] [79], while multi-task work focuses on enhancement [65] [95] or multi-scale super-resolution.Source: §2 · p. 2

Non-linear Image Restoration

  • Diffusion models have rarely been studied for nonlinear restoration, despite its relevance to imaging systems such as HDR and the activity of GAN-based approaches.HDR-GAN synthesizes HDR images from multi-exposed LDR images [59], and EnlightenGAN is an unsupervised GAN designed to generalize across real-world test images.Source: §2 · p. 2

Blind Image Restoration

  • Blind restoration remains difficult to unify: prior methods estimate unknown blur, use fixed testing-time down-sampling, or learn mappings with cycle-consistent GANs.Examples include structured denoisers [34], synthetic testing pairs [75], and CinCGAN [96] and MCinC-GAN, which use a pre-trained SR model and cycle consistency [107].Source: §2 · pp. 2–3
  • GDP is proposed for blind restoration by simultaneously estimating the recovered image and a task-specific degradation model.Source: §2 · p. 3

3. Preliminary

DDPMs gradually transform data into Gaussian noise through a forward diffusion process, then recover clean data by iteratively denoising noise. Their reverse process uses a neural network to estimate the denoising quantities needed to sample each step.

3. Preliminary

  • The forward diffusion process gradually corrupts data into Gaussian noise through Gaussian Markov transitions, while allowing any noisy step to be sampled directly from the original data.Diffusion models transform the data distribution into a simple noise distribution [2] [22] [36] [71] [89].Source: §3 · p. 3
  • The reverse process starts from Gaussian noise and iteratively denoises it toward clean data; Ho et al. [27] give a closed-form forward conditional that approaches the noise distribution at large T.The forward conditional is q(x_t | x_0) = N(x_t; √ᾱ_t x_0, (1 − ᾱ_t)I).Source: §3 · p. 3
  • A neural network estimates the reverse-process mean by predicting noise, with variance either fixed over time [27] or learned [58]; in practice, the method predicts x̃_0 and samples x_{t−1} from it and x_t.The passages describe the noise predictor as a function approximator and the predicted clean image as an input to the next reverse step.Source: §3 · p. 3

4. Generative Diffusion Prior

GDP uses a pretrained diffusion model as a prior for image restoration and enhancement, conditioning its reverse denoising process on degraded observations. Its guidance and degradation-model strategies extend the approach to blind, multi-image, and arbitrary-resolution restoration.

4. Generative Diffusion Prior

  • GDP uses a pretrained DDPM as a generic image prior, conditioning reverse denoising on the degraded image to guide restoration toward an image consistent with the observation.The conditioning signal combines a degradation-consistency loss with an optional quality-enhancement loss, such as one controlling brightness.Source: §4 · pp. 3–4

4.1. Single Image Guidance

  • GDP-x0 applies guidance to the DDPM’s predicted clean image rather than noisy xt, and removes the variance term because it can reduce generated-image quality.The paper notes that xt has a noise magnitude that may not match the corrupted observation, making a direct distance loss unreliable.Source: §4.1 · pp. 4–5, §4 · p. 4
  • For blind low-light enhancement and HDR recovery, GDP jointly estimates each image’s degradation parameters and the restored image during reverse denoising.The proposed model uses unknown light factor f and light mask M, randomly initialized and optimized for each corrupted image.Source: §4.1 · p. 5

4.2. Extended version

  • HDR-GDP guides generation with three differently exposed LDR images and optimizes a separate pair of blind degradation parameters for each; patch-based processing enables arbitrary output resolutions.The three inputs are short-, medium-, and long-exposure images.Source: §4.2 · p. 5

5. Loss Function

GDP combines a reconstruction loss to recover information from the conditional signal with quality-enhancement losses tailored to improve final outputs. MSE is the primary reconstruction loss, while the enhancement losses target exposure, color constancy, and illumination smoothness.

Loss Function

  • GDP pairs reconstruction loss, primarily MSE, with quality-enhancement loss to recover conditional-signal information and improve output quality.Other possible reconstruction losses include SSIM and perceptual loss.Source: §5 · p. 6, Figure 5 · p. 6

Exposure Control Loss

  • Exposure control loss lets low-light enhancement adjust brightness, using the RGB gray level as a target that can be varied.It compares average intensities over non-overlapping 8×8 regions.Source: Figure 5 · p. 6, §5 · p. 6, Figure 6 · p. 7

Color Constancy Loss

  • Color constancy loss links average intensities across channel pairs to correct color deviations and produce natural colors in colorization and low-light enhancement.The paired channels are (R, G), (R, B), and (G, B).Source: §5 · p. 7

Illumination Smoothness Loss

  • Illumination smoothness loss maintains monotonicity between neighboring pixels in the optimized light mask, making the estimated mask smoother for low-light enhancement.Its formulation uses horizontal and vertical gradients.Source: §5 · p. 7

6. Experiments

Experiments evaluate GDP across linear, nonlinear, blind, and HDR restoration tasks, reporting strong benchmark performance and testing the guidance, variance, degradation-optimization, and patch-based design choices.

6.1. Linear and Multi-linear Degradation Tasks

  • GDP-x0 outperforms other methods in Consistency and FID, while DDRM achieves better PSNR and SSIM than GDP.The comparison covers 4× super-resolution, deblurring, inpainting, and colorization on ImageNet; GDP also restores 256 × 256 out-of-distribution images and supports multiple outputs per input.Source: §6.1 · p. 7

6.2. Exposure Correction Tasks

  • GDP-x0 performs best among zero-shot methods on FID, LOE, and PI across three low-light datasets.The evaluation uses LOL, VE-LOL-L, and LoLi-phone; brightness can be adjusted through the well-exposedness level E using Exposure Control Loss.Source: §6.2 · p. 8

6.3. HDR Image Recovery

  • HDR-GDP-x0 performs best in PSNR, SSIM, LPIPS, and FID on the NTIRE2021 HDR validation set.The validation uses 100 randomly selected scenes; qualitative results show enhanced low-light areas, adjusted overexposure, and clearer detail.Source: Figure 7 · p. 7, §6.3 · p. 8, Table 4 · p. 8

6.4. Ablation Study

  • GDP-xt and GDP-x0 outperform their respective variants with variance Σ, and GDP-x0 outperforms GDP-xt with or without Σ.These ablations support omitting variance Σ and using the x0 guidance protocol.Source: §6.4 · p. 8, Table 5 · p. 8
  • GDP-x0 ranks first among the tested models, supporting trainable degradation parameters and the hierarchical guidance and patch-based method.Experiments on LOL and NTIRE compare naive patch restoration and fixed degradation parameters across patches; the authors also report best visualization results.Source: §6.4 · p. 8, Table 6 · p. 8

7. Conclusion

The Generative Diffusion Prior unifies image restoration across linear, nonlinear, and blind problems. It handles arbitrary-size images through hierarchical guidance and patch-based methods, and is demonstrated on six restoration and enhancement tasks.

7. Conclusion

  • GDP addresses linear inverse, nonlinear, and blind problems within a unified image-restoration approach.Source: §7 · p. 8
  • Hierarchical guidance and patch-based methods enable GDP to restore images of any size.Source: §7 · p. 8
  • GDP is demonstrated on super-resolution, deblurring, inpainting, colorization, low-light enhancement, and HDR recovery, alongside a systematic study of DDPM guidance.Source: §7 · p. 8

A. Limitations and Future works · B. Implementation Details

GDP’s limitations are longer sampling time and experimentally chosen guidance scales, while the implementation covers six restoration tasks and describes patch-based handling of arbitrary image sizes. The authors propose faster sampling and extensions to 3D restoration, LiDAR recovery, and self-supervised training.

A. Limitations and Future works

  • Adding guidance steps at every diffusion timestep extends inference time, limiting real-time use and weak devices such as mobile devices.The authors identify accelerated diffusion sampling as a direction for addressing this limitation.Source: A. Limitations and Future works · p. 14
  • Guidance scales are selected experimentally, requiring manual tuning across distributions, although approximate degradation models can yield close scales for data from the same distribution.The authors say the reason for this same-distribution pattern may be proved mathematically in future work.Source: A. Limitations and Future works · pp. 14–15
  • GDP-x0 pseudocode initializes degradation-model parameters randomly and updates them by gradient descent; HDR recovery uses three LDR conditioners, while other tasks use one.The algorithm also specifies a learning rate for optimizing the degradation model.Source: A. Limitations and Future works · p. 15
  • Future work includes optimizing timestep and variance schedules and exploring self-supervised training inspired by GDP and supervised techniques [72] to improve unsupervised restoration.Source: A. Limitations and Future works · p. 15
  • The authors propose applying GDP to 3D point-cloud completion and upsampling, potentially integrating it with Shapeinversion [97], and exploring recovery from weather-affected LiDAR’s nonlinear degradations.Source: A. Limitations and Future works · p. 15

B. Implementation Details

  • GDP is applied to colorization, inpainting, super-resolution, deblurring, low-light enhancement, and HDR recovery without task-specific hyperparameter tuning or architecture customization.The first four tasks use 256 × 256 RGB images; low-light enhancement uses 600×400 images, and HDR recovery uses 1900 × 1060 images.Source: B. Implementation Details · p. 15
  • Algorithm 6 handles arbitrary-size images by aggregating overlapping-patch means, variances, and conditioner terms before sampling.The procedure uses a dictionary of overlapping patch locations and a binary mask for each patch.Source: B. Implementation Details · p. 16
  • The implementation targets difficult cases including large inpainting masks and nonlinear low-light and HDR inverse problems, where image statistics are needed to correct over- and under-exposure.The authors frame achieving strong results across tasks without task-specific engineering as the goal; they do not report that claim in this passage as a result.Source: B. Implementation Details · p. 16

B.1. Dataset briefs · B.2. Experimental Setup

The experiments use datasets spanning linear restoration, low-light enhancement, and HDR recovery, with benchmarks and data conditions specified for each task. Experimental setups define synthetic degradations for inverse problems and use naturally exposed low-light and NTIRE images for nonlinear and blind problems.

B.1. Dataset briefs

  • ImageNet, CelebA-HQ, LSUN, and USC-SIPI provide datasets for quantitative evaluation of linear restoration tasks.Evaluation uses 1k ImageNet validation images following [62]; CelebA-HQ is a high-quality subset of CelebA [40] [49], and LSUN has around one million images per category across 10 scene and 20 object categories [94]. USC-SIPI is a collection of digitized images; CelebA, LSUN, and USC-SIPI images are provided by [32].Source: §B.1 · p. 16
  • LOL contains 500 low-light/normal-light pairs, split into 485 training and 15 testing pairs, at 400 × 600 resolution.Its low-light images contain photo-capture noise, and most scenes are indoors [88].Source: §B.1 · p. 16
  • VE-LOL-L pairs well-exposed images with underexposed versions at different levels; its Cap and Syn subsets use different guidance scales.The two subsets have different distributions.Source: §B.1 · p. 16
  • LoLi-Phone provides low-light images and videos captured by different mobile phone cameras under diverse illumination conditions [41].Source: §B.1 · p. 16
  • NTIRE contains 1,494 LDR/HDR training images, 60 validation images, and 201 test images; experiments select 100 training images as the test set.Each of 26 long shots has three LDR images, exposure and alignment information, and HDR ground truth; images are 1060 × 1900. Because validation and test ground truth are unavailable, experiments use the training set [64].Source: §B.1 · p. 16

B.2. Experimental Setup

  • Inverse-problem setups use pixel values in [0,1], with 4× block-average downscaling, a 9×9 unified blur kernel, channel-averaged grayscale, or text overlays and 25% pixel deletion.These operations define super-resolution, deblurring, colorization, and inpainting, respectively.Source: §B.2 · p. 16
  • For nonlinear and blind problems, low-light and NTIRE images are naturally overexposed or underexposed, so no additional operations are applied.Source: §B.2 · p. 16

C. Evaluation Metrics … I. Additional Results on Linear inverse problems

The supplementary sections define evaluation measures, elaborate GDP’s guidance and patch-based methods, and report guidance ablations, DDIM comparisons, theoretical derivations, and additional restoration results.

C. Evaluation Metrics

  • Evaluation supplements PSNR and SSIM with FID, consistency, LPIPS, PI, and LOE to assess synthesis, degradation fidelity, perceptual distance or quality, and lightness naturalness.Consistency measures MSE after reapplying the degradation; lower PI and LOE indicate better perceptual quality and preservation of lightness naturalness, respectively.Source: C. Evaluation Metrics · pp. 16–18

D. Further elaboration of the models

  • GDP-xt guides reverse sampling by shifting the conditional mean, GDP-x0 also optimizes unknown degradation parameters, and hierarchical patching supports low-light and HDR images.For arbitrary-size restoration, corrupted inputs are resized, light masks interpolated to the original size, and patches recombined; HDR-GDP-x0 optimizes three degradation models using their respective losses.Source: D. Further elaboration of the models · p. 18

E. Further Ablation Study on the Guidance

F. The ELBO objective of GDP

  • GDP is a Markov chain conditioned on y, yielding an ELBO objective whose expectation samples data, degraded inputs, and diffusion trajectories.The cited objective is attributed to [77].Source: F. The ELBO objective of GDP · p. 19

H. Image Guidance / H.1. Conditional Reverse Process

  • Image guidance treats corrupted inputs as conditions and approximates conditional reverse sampling by shifting the transition mean using the degradation-loss gradient.The likelihood is modeled through the degradation function and a loss that may include a quality-enhancement term.Source: H. Image Guidance · p. 20, §H.1 · p. 20

H.2. Conditional Diffusion Process / H.3. Scaling Conditioner Gradients

  • The conditional noising derivation recovers the unconditional reverse process when unconditioned and constructs an input conditioner; scaling its gradients sharpens conditioning and may trade diversity for fidelity.The conditioner uses degraded-input likelihood on noisy samples; larger gradient scales concentrate more on its modes.Source: §H.2 · pp. 20–22, §H.3 · p. 22

G. Sampling with DDIM

  • With 20 sampling steps, GDP-x0-DDIM outperforms DDRM on consistency and FID across four ImageNet tasks, while DDRM scores better on PSNR and SSIM; GDP’s qualitative results are better.The paper notes that PSNR and SSIM may not align with human perception for low-resolution inputs and large magnification.Source: G. Sampling with DDIM · pp. 19–20, Table 9 · p. 21

I. Additional Results on Linear inverse problems

  • Additional figures show GDP across datasets and linear inverse problems, include uncurated ImageNet samples, and demonstrate recovery after multi-linear degradations.The passage identifies Figures 16–19 and 21 for datasets and linear problems, Figures 20 and 22–26 for uncurated samples, and Figure 27 for multi-linear degradations.Source: §I · p. 22

J. Additional Results on Low-light Enhancement · K. Additional Results on HDR Recovery

Additional results show GDP-x0 performs well across diverse low-light datasets and compares favorably with several learning approaches, while GDP also supports exposure control and HDR recovery. The results note a tone difference in HDR images due to ImageNet pretraining.

J. Additional Results on Low-light Enhancement

K. Additional Results on HDR Recovery

L. Additional Results on Ablation Study

The ablation study finds that interpolation, naive patch restoration, and fixed parameters each undermine restoration quality in different ways.

L. Additional Results on Ablation Study

Related papers

Loading 2304.01247v1…