PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Diffusion Prior for Unified Image Restoration and Enhancement

Ben Fei, Zhaoyang Lyu, Liang Pan, Junzhe Zhang, Weidong Yang, Tianyue Luo, Bo Zhang, Bo Dai

arXiv:2304.01247v1cs.CV

TL;DR

Existing restoration methods often require known degradation models or supervised training, limiting unsupervised adaptation to complex problems. GDP uses a pretrained DDPM with conditional guidance and optimized degradation parameters to address linear, non-linear, and blind restoration. Across diverse tasks, it outperforms unsupervised and zero-shot baselines and supports arbitrary-size images, though its inference time limits real-time and mobile use.

  • Problem

    Existing restoration methods often assume known degradation or require supervised training, limiting unsupervised adaptation to complex real applications.

  • Method

    GDP uses a pretrained DDPM as a restoration prior, guides denoising with degraded images, and optimizes unknown degradation parameters during inference.

  • Results

    GDP outperforms unsupervised and zero-shot baselines across linear, non-linear, and blind restoration tasks, including low-light enhancement and HDR recovery.

  • Takeaways & Limitations

    GDP provides a unified unsupervised framework for diverse restoration tasks, including arbitrary-size image generation and images outside the ImageNet training distribution.

  • Takeaways & Limitations

    Extended sampling time limits GDP’s applicability to real-time applications and weak end-user devices such as mobile devices.

Abstract

from arXiv · show

Existing image restoration methods mostly leverage the posterior distribution of natural images. However, they often assume known degradation and also require supervised training, which restricts their adaptation to complex real applications. In this work, we propose the Generative Diffusion Prior (GDP) to effectively model the posterior distributions in an unsupervised sampling manner. GDP utilizes a pre-train denoising diffusion generative model (DDPM) for solving linear inverse, non-linear, or blind problems. Specifically, GDP systematically explores a protocol of conditional guidance, which is verified more practical than the commonly used guidance way. Furthermore, GDP is strength at optimizing the parameters of degradation model during the denoising process, achieving blind image restoration. Besides, we devise hierarchical guidance and patch-based methods, enabling the GDP to generate images of arbitrary resolutions. Experimentally, we demonstrate GDP's versatility on several image datasets for linear problems, such as super-resolution, deblurring, inpainting, and colorization, as well as non-linear and blind issues, such as low-light enhancement and HDR image recovery. GDP outperforms the current leading unsupervised methods on the diverse benchmarks in reconstruction quality and perceptual quality. Moreover, GDP also generalizes well for natural images or synthesized images with arbitrary sizes from various tasks out of the distribution of the ImageNet training set.

1 Fudan University, 2Shanghai AI Laboratory, 3S-Lab, Nanyang Technological University

GDP is presented as a unified approach for high-fidelity image restoration across diverse task types.

  • GDP generates high-fidelity image restoration across various tasks.
  • GDP also addresses blind, non-linear, multiple-guidance, and arbitrary-size image restoration, including low-light enhancement and HDR recovery.

1. Introduction

Image restoration spans known linear degradations and unknown non-linear or blind degradations, motivating unsupervised generative priors. GDP uses a pretrained DDPM with degradation guidance and optimized degradation parameters, and is evaluated across diverse tasks and datasets.

  • Linear tasks include super-resolution, deblurring, inpainting, and colorization, while low-light enhancement and HDR recovery involve non-linear or blind degradation.
  • Unsupervised generative priors can address multiple restoration tasks during inference without retraining, but DDRM remains limited to known linear degradation matrices.
  • GDP uses a pretrained DDPM as a general restoration prior, guides denoising with degraded images, and optimizes randomly initialized degradation parameters during denoising.
  • Experiments compare GDP with competitive unsupervised methods on linear or multi-linear problems across ImageNet, LSUN, and CelebA.
  • GDP outperforms zero-shot baselines qualitatively and quantitatively on low-light enhancement and HDR recovery, including images outside ImageNet’s training distribution.
  • The contributions include unified unsupervised restoration with one unconditional ImageNet DDPM, blind degradation optimization, arbitrary-size generation, and an alternative guidance protocol.

2. Related works

Prior restoration work addresses unconditional linear inverse problems and blind degradation with specialized approaches, while GDP’s overview combines diffusion guidance with degradation optimization.

  • Linear Inverse Image Restoration: Unconditional diffusion models address linear inverse problems with one trained model, but multi-linear restoration remains relatively under-explored.
  • GDP guides the reverse diffusion process using corrupted observations and can apply guidance to either the predicted clean image or the intermediate noisy state.
  • GDP-xt adds guidance to x_t at each time step, whereas GDP-x0 guides the predicted clean image ˜x_0.
  • The number of guidance images depends on the task, with three used for HDR recovery and one for other tasks.
  • Existing blind-restoration approaches may fail to obtain degradation parameters or distributions for complicated degradation types.

3. Preliminary

Diffusion models gradually transform clean data into Gaussian noise and learn a reverse process that reconstructs clean images from noise.

  • Diffusion Process: The forward diffusion process converts data x_0 into progressively corrupted states until x_T approaches Gaussian noise.
  • Diffusion Process: The forward process is a Markov chain with a variance schedule controlling Gaussian transitions between diffusion states.
  • Reverse Process: The reverse process iteratively denoises sampled Gaussian noise from x_T to clean data x_0.
  • Reverse Process: A neural network estimates the reverse-process mean, while the variance may be fixed or learned.
  • Reverse Process: In practice, the clean image ˜x_0 is predicted from x_t before sampling the preceding state x_{t-1}.

4. Generative Diffusion Prior

GDP uses a pre-trained DDPM as an unsupervised prior for restoration and enhancement, conditioning diffusion on degraded observations and optimizing unknown degradation parameters when needed. It also introduces guidance and patch-based strategies for diverse tasks, image sizes, and input modalities.

  • Conditional guidance: GDP conditions a pre-trained DDPM on corrupted images to solve unified image restoration and enhancement without retraining.The framework targets linear inverse, non-linear, and blind problems.
  • Blind restoration: GDP-x0 jointly optimizes randomly initialized degradation parameters during denoising, enabling blind restoration for unknown degradation models.The low-light formulation uses unknown light factor f and light mask M as degradation parameters.
  • Conditional guidance: GDP removes the variance Σ from guided denoising because experiments found that variance-based mean shifting could reduce generated-image quality.The commonly used xt-guidance variant is named GDP-xt.
  • Conditional guidance: GDP-x0 guides the predicted clean image ˜x0 rather than the noisy latent xt during diffusion sampling.This addresses the mismatch between the noise level in xt and the corruption level in the observed image.
  • Extended restoration: Hierarchical guidance, patch-based processing, and multi-image guidance extend GDP to arbitrary-size images and HDR recovery from three LDR exposures.HDR-GDP optimizes separate blind degradation parameters for short-, medium-, and long-exposure inputs.
  • Evaluation: On ImageNet linear restoration benchmarks, GDP outperforms other methods in FID and Consistency across super-resolution, deblurring, inpainting, and colorization.The evaluated metrics include PSNR, SSIM, Consistency, and FID.

5. Loss Function

GDP combines reconstruction loss with quality-enhancement losses to recover conditional information while improving restored-image quality. Task-specific losses control color consistency, illumination smoothness, and exposure.

  • Loss design: GDP divides its objective into reconstruction loss and quality-enhancement loss.The reconstruction term can use MSE, SSIM, or perceptual loss.
  • Quality-enhancement losses: Exposure control loss Lexp adjusts the brightness level for low-light enhancement.The loss is introduced to control the exposure level of the restored image.
  • Quality-enhancement losses: Color constancy loss Lcol corrects color deviations and links the three adjusted channels in colorization.It is formulated over pairwise channel-average intensity differences.
  • Quality-enhancement losses: Illumination smoothness loss LtvM maintains monotonicity between neighboring pixels in the optimized light mask M.It uses horizontal and vertical gradient operations over the light mask.
  • Task-specific losses: Color constancy supports natural colors, illumination smoothness makes the low-light light mask smoother, and exposure control manually adjusts brightness.The associated losses are used for colorization and low-light enhancement.

6. Experiments

Experiments evaluate GDP across linear, non-linear, blind, and HDR restoration tasks using an unconditional ImageNet DDPM. GDP generally delivers strong perceptual and reconstruction quality, while ablations support its guidance and patch-based design choices.

  • Linear and multi-linear degradation tasks: GDP is evaluated with PSNR, SSIM, Consistency, and FID across 4× super-resolution, deblurring, inpainting, and colorization.The comparison uses unsupervised methods on ImageNet validation images spanning all 1000 classes.
  • Linear and multi-linear degradation tasks: GDP-x0 outperforms other methods in Consistency and FID, while DDRM achieves better PSNR and SSIM but requires higher Consistency and FID.The reported comparison covers the tested linear tasks on ImageNet.
  • Low-light enhancement: GDP-x0 achieves the best FID, LOE, and PI among zero-shot methods across LOL, VE-LOL-L, and LoLi-phone low-light datasets.The lower LOE indicates better preservation of illumination order.
  • HDR image recovery: HDR-GDP-x0 performs best in PSNR, SSIM, LPIPS, and FID on the NTIRE HDR recovery validation set.It also enhances low-light regions, adjusts over-exposed areas, and recovers clearer details.
  • Ablation study: Removing variance Σ improves image quality, and guidance on x0 outperforms guidance on xt in the ablation study.The comparisons are reported for both GDP-xt and GDP-x0 variants.
  • Ablation study: GDP-x0 ranks first across models when trainable degradation parameters and patch-based methods are evaluated.The results support the proposed hierarchical guidance and patch-based method on LOL and NTIRE datasets.

7. Conclusion

The conclusion presents GDP as a unified diffusion-prior framework for linear, non-linear, and blind image restoration. Hierarchical guidance and patch-based methods extend it to arbitrary-size images and multiple restoration tasks.

  • Unified restoration: GDP tackles linear inverse, non-linear, and blind image-restoration problems within one framework.The conclusion lists super-resolution, deblurring, inpainting, colorization, low-light enhancement, and HDR recovery.
  • Arbitrary-size restoration: Hierarchical guidance and patch-based methods enable GDP to restore images of arbitrary sizes.The framework is applied across the listed restoration and enhancement tasks.

A. Limitations and Future works

The main limitation is prolonged inference caused by additional guidance steps at each diffusion timestep, restricting real-time and weak-device deployment. Future work includes accelerated sampling and broader extensions, while guidance-scale selection remains distribution-dependent.

  • Limitations: Additional guidance steps at every timestep extend sampling time and limit real-time or mobile-device applicability.The authors identify accelerated diffusion sampling as the required direction for addressing this limitation.
  • Limitations: The optimal guidance scale must be manually selected for samples from different data distributions.For the same distribution, approximate degradation models may yield similar guidance scales.
  • Future work: GDP is proposed for future exploration in 3D restoration tasks such as point-cloud completion and upsampling.The authors describe these tasks as linear inverse problems in 3D vision.
  • Future work: Future work also considers recovering LiDAR point clouds affected by weather and non-linear degradations.The proposed direction targets real-world LiDAR recovery.
  • Future work: The authors identify self-supervised training techniques as another direction for improving unsupervised image-restoration models.This direction is inspired by GDP and techniques used in supervised methods.

B.1. Dataset briefs

The experiments use diverse datasets spanning natural images, faces, scenes, low-light enhancement, and HDR recovery, with specified degradation settings for linear inverse problems.

  • Datasets: ImageNet, CelebA-HQ, LSUN, and USC-SIPI provide datasets for evaluating linear image restoration tasks.ImageNet contributes 1k validation images; CelebA-HQ is a high-quality face subset, LSUN contains scene and object categories, and USC-SIPI contains digitized images.
  • Datasets: LOL contains 500 paired low-light and normal-light images, including 485 training pairs and 15 testing pairs at 400 × 600 resolution.The low-light images include capture noise and mostly depict indoor scenes.
  • Datasets: VE-LOL-L pairs well-exposed images with underexposed versions at different underexposure levels.Its VE-LOL-Cap and VE-LOL-Syn subsets use different guidance scales because they have different distributions.
  • Datasets: LoLi-Phone provides low-light images and videos captured by different mobile phone cameras under diverse illumination conditions.
  • Datasets: The NTIRE dataset contains 1,494 training LDR/HDR frames, 60 validation images, and 201 test images, but experiments select 100 training images because validation and test ground truth is unavailable.The images have resolution 1060 × 1900, and the training frames come from 26 long shots.
  • Degradation settings: Linear degradations include 4× block-averaging downscaling, 9×9 unified-kernel blurring, RGB averaging for grayscale conversion, and text-overlay or 25% random-pixel inpainting.
  • Degradation settings: Low-light and NTIRE images are naturally over-exposed or under-exposed, so no additional degradation operations are applied.

C. Evaluation Metrics

The evaluation combines reconstruction, consistency, perceptual, and lightness-preservation measures, while the method conditions diffusion sampling through degradation-based guidance.

  • Metrics: FID, PSNR, SSIM, and Consistency evaluate synthesized-image quality, reconstruction quality, and faithfulness to degraded inputs.Consistency measures the MSE between degraded inputs and outputs after applying the same degradation.
  • Metrics: LPIPS and PI provide perceptual-quality measures, with PI serving as a non-reference perceptual index.
  • Metrics: LOE measures enhancement performance, and lower LOE indicates better preservation of natural lightness.The metric compares lightness-order relationships between enhanced and reference images.
  • Guidance protocols: GDP-xt guides sampling by conditioning on the noisy image x_t, while GDP-x0 predicts an intermediate clean image and degrades it before comparing it with the corrupted input.GDP-x0 can use an unknown degradation model during this process.
  • Guidance protocols: HDR-GDP-x0 optimizes three randomly initialized degradation models during denoising and uses their gradients to sample the next latent state.
  • Arbitrary-size restoration: Hierarchical guidance resizes corrupted images for patch processing, then interpolates and fixes light factors and masks for full-resolution image patches.
  • Ablation studies: GDP-x0 and GDP-xt outperform their v1 variants on four linear tasks in the guidance ablation.
  • Sampling efficiency: With 20 DDIM steps, GDP-x0-DDIM outperforms DDRM on Consistency and FID across four tasks, although DDRM achieves better PSNR and SSIM.The paper notes that PSNR and SSIM may not correlate well with human perception at low input resolution and large magnification.

J. Additional Results on Low-light Enhancement

Additional results show GDP handling low-light, HDR, multi-degradation, and arbitrary-size restoration settings, with qualitative evidence favoring its guidance and patch-based designs.

  • Low-light enhancement: GDP performs well on LOL, VE-LOL-L, and LoLi-Phone despite their different image distributions, producing more satisfactory images than other methods in comparisons.
  • Low-light enhancement: Exposure Control Loss lets users adjust the RGB gray level E to generate images with specific brightness.
  • HDR recovery: HDR-GDP-x0 adjusts over-exposed and under-exposed areas across scenes and recovers more realistic images with more details than compared methods.Because GDP is pretrained on ImageNet, generated tones can differ slightly from ground-truth images.
  • Ablation studies: The ablation shows interpolation, naive patch restoration, and fixed random degradation parameters respectively cause low quality, artifacts, and uncontrollable outputs.
  • Qualitative results: The figures provide qualitative samples for super-resolution, deblurring, inpainting, colorization, low-light enhancement, and HDR recovery across several datasets.
  • Multiple degradations: GDP recovers images undergoing multiple degradations, including gray plus blur, gray plus inpainting, and gray plus down-sampling.Restoration becomes more difficult when two content-damaging degradations occur together, such as down-sampling plus masking.
Loading 2304.01247v1…