PaperlayerView exact revision on arXiv

Source-linked AI summary

Plug-and-Play Image Restoration with Deep Denoiser Prior

Kai Zhang, Yawei Li, Wangmeng Zuo, Lei Zhang, Luc Van Gool, Radu Timofte

arXiv:2008.13751v2eess.IVcs.CV

TL;DR

Plug-and-play restoration lacks a sufficiently flexible and effective denoiser prior despite its potential to combine model-based flexibility with learning-based performance. The paper trains a noise-level-conditioned deep CNN denoiser and inserts it into HQS-based restoration, reporting superior model-based results and competitive or better learning-based performance across three tasks.

  • Problem

    Existing plug-and-play restoration lacks a suitable flexible and effective denoiser prior for exploiting large-capacity deep CNNs.

  • Method

    The paper trains a flexible deep CNN denoiser and uses it as a modular prior in an HQS-based plug-and-play restoration algorithm.

  • Results

    Experiments on deblurring, super-resolution and demosaicing show that the proposed method outperforms state-of-the-art model-based methods and achieves competitive or superior performance against learning-based methods.

  • Takeaways & Limitations

    Deep denoiser priors improve model-based restoration effectiveness while retaining greater flexibility than task-specific learning-based methods without task-specific training.

  • Takeaways & Limitations

    The general-purpose Gaussian denoiser prior and manual hyperparameter selection reduce efficiency and specialization compared with deep unfolding restoration.

Abstract

from arXiv · show

Recent works on plug-and-play image restoration have shown that a denoiser can implicitly serve as the image prior for model-based methods to solve many inverse problems. Such a property induces considerable advantages for plug-and-play image restoration (e.g., integrating the flexibility of model-based method and effectiveness of learning-based methods) when the denoiser is discriminatively learned via deep convolutional neural network (CNN) with large modeling capacity. However, while deeper and larger CNN models are rapidly gaining popularity, existing plug-and-play image restoration hinders its performance due to the lack of suitable denoiser prior. In order to push the limits of plug-and-play image restoration, we set up a benchmark deep denoiser prior by training a highly flexible and effective CNN denoiser. We then plug the deep denoiser prior as a modular part into a half quadratic splitting based iterative algorithm to solve various image restoration problems. We, meanwhile, provide a thorough analysis of parameter setting, intermediate results and empirical convergence to better understand the working mechanism. Experimental results on three representative image restoration tasks, including deblurring, super-resolution and demosaicing, demonstrate that the proposed plug-and-play image restoration with deep denoiser prior not only significantly outperforms other state-of-the-art model-based methods but also achieves competitive or even superior performance against state-of-the-art learning-based methods. The source code is available at https://github.com/cszn/DPIR.

1 INTRODUCTION

Image restoration combines degradation modeling with priors to recover clean images, but model-based and learning-based methods trade flexibility against performance and efficiency. The paper addresses this gap by pairing a flexible deep CNN denoiser with HQS-based plug-and-play restoration.

  • Problem setting: Image restoration recovers a clean image from a degraded observation under tasks defined by the degradation operation T.Examples include denoising, deblurring, super-resolution and color image demosaicing.
  • Motivation: Model-based methods flexibly specify T and optimize degraded images directly, whereas learning-based methods usually require task-specific training but offer fast testing and strong performance.The two categories therefore provide complementary advantages.
  • Plug-and-play formulation: Plug-and-play methods separate data and prior terms with variable splitting, reducing the prior subproblem to denoising solvable by a deep CNN.The paper identifies HQS and ADMM as representative splitting algorithms.
  • Contribution: The proposed denoiser handles a wide range of noise levels through an input noise-level map and combines ResNet and U-Net designs.It is incorporated into HQS-based plug-and-play restoration.
  • Contribution: The work analyzes parameter setting, intermediate results and empirical convergence to clarify the working mechanism of deep plug-and-play restoration.These analyses accompany the proposed denoiser and HQS-based method.
  • Results: Experiments on deblurring, super-resolution and demosaicing demonstrate the superiority of the proposed plug-and-play restoration with a deep denoiser prior.The reported evaluation covers three representative restoration tasks.

2 RELATED WORKS

Plug-and-play restoration decouples degradation handling from prior modeling, enabling off-the-shelf denoisers but imposing requirements on denoiser flexibility and noise conditioning. The paper motivates DRUNet as a stronger single-model prior for this setting.

  • Plug-and-Play IR: Plug-and-play restoration alternates data and prior subproblems after variable splitting, using an off-the-shelf denoiser to define the prior implicitly.This avoids explicitly specifying hand-crafted image priors.
  • Prior methods: Earlier plug-and-play methods used denoisers such as BM3D or CBM3D for deblurring and super-resolution.These methods iteratively combine data updates with denoising steps.
  • Denoiser requirements: Existing CNN denoisers often fail to provide the non-blind, wide-noise-level behavior required by plug-and-play restoration.DnCNN learns separate models for each noise level, while small-noise fixed models may require many iterations.
  • Comparison: Deep unfolding methods can jointly learn task-specific priors and hyperparameters, whereas plug-and-play methods remain easier to adapt across tasks.The comparison highlights a flexibility-versus-specialization trade-off.

3 LEARNING DEEP CNN DENOISER PRIOR

The paper develops DRUNet as a flexible Gaussian denoiser prior for plug-and-play restoration. Its design combines noise-level conditioning with U-Net structure, residual blocks and bias-free convolutions.

  • Denoiser design: Plug-and-play denoisers should target non-blind Gaussian denoising because the prior subproblem has a specified Gaussian noise level.During iterations, the denoiser smooths unknown structural noise whose distribution can vary across tasks and iterations.
  • Network architecture: DRUNet improves FFDNet by combining U-Net’s image-to-image structure with ResNet residual blocks for stronger prior modeling.It is designed as a flexible and powerful pretrained denoiser.
  • Network architecture: DRUNet uses one model for various noise levels by taking a noise-level map as input.This follows the conditioning strategy used by FFDNet.
  • Network architecture: The proposed network is bias-free across convolution, strided-convolution and transposed-convolution layers.The paper links this choice to scaling invariance and observed concerns about bias magnitude harming generalizability.

3.2 Training Details

DRUNet is trained on a large mixed-image dataset with additive Gaussian noise sampled across a broad noise range. Training uses L1 reconstruction loss and Adam optimization with a scheduled learning rate.

  • Training data: The training corpus combines 400 BSD, 4,744 Waterloo, 900 DIV2K and 2,750 Flick2K images.The paper expands beyond a small dataset containing only 400 BSD images.
  • Noise model: Training noise is additive white Gaussian noise with σ randomly sampled from [0, 50].The noise-level map is a uniform map filled with σ and matches the noisy image’s spatial size.
  • Optimization: The network minimizes L1 loss between denoised outputs and ground truth using Adam.The paper motivates L1 as more robust to outliers in sampled Gaussian noise.
  • Optimization: The learning rate starts at 1e-4 and is halved every 100,000 iterations before training ends.This schedule is part of the reported optimization setup.

3.3 Denoising Results

DRUNet delivers strong denoising performance across grayscale and color benchmarks, extends to JPEG deblocking, and improves generalization to unseen noise levels. Its flexibility comes with higher computational cost than lighter CNN denoisers, while remaining substantially cheaper than RNAN.

  • Grayscale Image Denoising: DRUNet achieves the best PSNR at noise levels 15, 25, and 50 on Set12 and BSD68, with an average gain of about 0.9dB over BM3D.The comparison includes model-based methods, single-noise-level CNNs, and wide-range CNN denoisers.
  • Color Image Denoising: DRUNet outperforms competing methods by a large margin on color denoising across CBSD68, Kodak24, and McMaster.The differing DnCNN performance between CBSD68 and McMaster highlights the importance of reducing image-domain gaps.
  • Extended Application to JPEG Image Deblocking: DRUNet is adapted for JPEG deblocking by replacing the AWGN noise-level input with a normalized JPEG quality factor spanning.The adapted model is evaluated on Classic5 and LIVE1 at quality factors 10, 20, 30, and 40.
  • Ablation Study: Bias-free DRUNet generalizes better to unseen noise levels, while matching the biased model at trained noise levels.The visual comparison uses an unseen noise level of 200 and rescales inputs for DnCNN and IRCNN because those models cannot change noise level.
  • Ablation Study: The ablation study evaluates residual connections, training-data size, biases, and the noise-level map as contributors to DRUNet performance.Table 4 compares these variants on Set12 for noise levels 15, 25, and 50.
  • Runtime, FLOPs and Maximum GPU Memory Consumption: DRUNet has better PSNR than DnCNN but doubles runtime, triples FLOPs, and quadruples maximum GPU memory; RNAN is about 60 times slower than DnCNN.FFDNet has the best runtime, FLOPs, and memory results among the compared methods.

4 HQS ALGORITHM FOR PLUG-AND-PLAY IR

The HQS plug-and-play framework separates data fidelity from the image prior, solving alternating subproblems in which a Gaussian denoiser can implicitly specify the prior. Parameter schedules for the denoiser noise level, penalty, regularization, and iteration count are analyzed, alongside a periodical geometric self-ensemble strategy.

  • HQS formulation: HQS introduces an auxiliary variable to decouple the data term and prior term into alternating x and z subproblems.The resulting augmented objective adds a quadratic penalty enforcing z = x.
  • Alternating updates: The x subproblem handles the degradation-dependent data term, while the z subproblem is equivalent to Gaussian denoising of xk with noise level λ/µ.Therefore, Gaussian denoisers can be inserted into the alternating iterations.
  • Denoiser prior: A denoiser can implicitly define the prior R(·), enabling a single CNN denoiser to serve as the plug-and-play prior.This exposes CNN flexibility, GPU efficiency, and deep-network modeling capacity within the iterative method.
  • Parameter setting: The iterations involve penalty parameter µ, regularization parameter λ, and total iteration count K; increasing µ improves fixed-point convergence but may require larger K.Continuation schedules increase µ across iterations, while µ also determines the denoiser noise level σk = λ/µk.
  • Parameter setting: Sampling σk from a large initial value to a small final value in log space determines µk through µk = λ/σk^2, with σ1 fixed to 49 and σK tied to image noise.λ is fixed to 0.23 in the paper after validation-based grid search, with favorable empirical performance reported over [0.19, 0.55].
  • Periodical geometric self-ensemble: Periodical geometric self-ensemble applies transformations before denoising and inverse transformations afterward every successive eight iterations without increasing total inference time.The authors report a general PSNR gain of 0.02dB∼0.2dB.

5 EXPERIMENTS

The experiments evaluate DPIR as an HQS-based restoration algorithm using a deep denoiser prior, degraded observations, task operators, noise schedules, and a trade-off parameter. The study covers deblurring, super-resolution, and demosaicing, with intermediate outputs and convergence also examined.

  • Experimental setup: DPIR is evaluated on image deblurring, single image super-resolution, and color image demosaicing.For each task, the paper specifies the degradation model, data-subproblem solution, K and σK, initialization, and comparisons.
  • DPIR algorithm: Algorithm 1 takes a deep denoiser prior, degraded image y, degradation operation T, image noise level σ, denoiser schedule σk, iteration count K, and λ.The algorithm outputs the restored image zK.
  • DPIR algorithm: At each of K iterations, DPIR computes an xk data-term update followed by the denoiser-prior step involving DRUNet and periodical geometric self-ensemble.The intermediate results and convergence analyses compare the evolving xk and zk states.

5.1 Image Deblurring

For deblurring, DPIR combines a degradation-specific fast data update with a deep denoiser prior and is evaluated on classical images, real blur kernels, and multiple Gaussian noise levels. It outperforms the compared methods quantitatively and recovers sharper, more natural visual results, while intermediate states show denoising and fast convergence.

  • Degradation model and data update: The deblurring data update uses a closed-form FFT solution in which the blur kernel explicitly handles blur distortion.This solution assumes circular boundary conditions.
  • Experimental setup: The evaluation uses six classical images, two real blur kernels of sizes 17×17 and 27×27, and Gaussian noise levels 2.55(1%) and 7.65(3%).The default settings are K = 8, σK = σ, and z0 initialized as y.
  • Quantitative comparison: DPIR outperforms EPLL and FDN by a large margin on Set6 deblurring, alongside comparisons with DMPHN, IRCNN, and IRCNN+.The table reports PSNR(dB) results across the tested settings.
  • Visual comparison: On Leaves with noise level 7.65, DPIR recovers image sharpness and naturalness, whereas competing methods either smooth details, retain artifacts, or fail to restore fine structure.The comparison includes DMPHN, FDN, IRCNN, and IRCNN+.
  • Intermediate results and convergence: The data update can aggravate noise, while the deep denoiser produces noise-free zk states and successive iterations recover more fine details.The PSNR curves show that xk and zk converge quickly to a fixed point.
  • Parameter analysis: Larger initial denoiser noise levels such as 39 and 49 can improve PSNR, whereas smaller σ1 values require larger K for good performance.Increasing K under the smaller-σ1 setting increases computational cost.
  • Blind denoiser: A blind DRUNet denoiser is also tested with the same parameter setting on the noisy and blurry Leaves image.The supplied passage introduces the comparison but does not state its complete outcome.

5.2 Single Image Super-Resolution (SISR)

For SISR, DPIR handles both bicubic and classical blurred, decimated, and noisy degradation models, including diverse Gaussian kernels. Closed-form data updates and the DRUNet prior improve robustness over compared methods, while intermediate iterations remove structured noise and recover sharper details.

  • Degradation models: Existing bicubic SISR methods can deteriorate when the real degradation deviates from the assumed model, motivating a blurred, decimated, and noisy classical model.The paper evaluates both bicubic and classical degradation models.
  • Data-term solutions: For bicubic degradation, DPIR uses iterative back-projection, while the classical model can use a closed-form solution under circular boundary conditions.The closed-form solution uses distinct block processing and downsampling operations.
  • Experimental setup: The SISR evaluation uses bicubic degradation and eight Gaussian kernels: four isotropic kernels with standard deviations 0.7, 1.2, 1.6, and 2.0, plus four anisotropic kernels.The overall settings use K = 24 and σK = max(σ, s).
  • Quantitative comparison: On color BSD68, DPIR improves over IRCNN+ by using a more powerful denoiser, while IRCNN+ improves over IRCNN through the closed-form data solution.RCAN and SRFBN perform well for bicubic degradation with σ = 0 but lose effectiveness when the true degradation differs.
  • Visual comparison: For a classical-degradation example, DPIR produces the best visual result, recovering sharper edges than bicubic interpolation, MZSR, IRCNN, and IRCNN+.The comparison attributes IRCNN+'s improvement to its closed-form data-term solution but notes remaining limitations in clean HR recovery.
  • Intermediate results and convergence: The closed-form x1 update introduces structured noise even without image noise, while DRUNet removes it and later iterations recover sharper edges and more details.The xk and zk states converge quickly to a fixed point.

5.3 Color Image Demosaicing

DPIR is evaluated for Bayer-pattern color demosaicing on Kodak and McMaster datasets, using an HQS-based model with a deep denoiser prior. It achieves results close to the strongest learning-based methods and outperforms the other model-based methods in quantitative and visual comparisons.

  • DPIR demosaicing uses the Bayer CFA with RGGB arrangement, K=40, σK=0.6, and Matlab’s demosaic output to initialize z0.
  • The evaluation uses 24 Kodak images of size 768×512 and 18 McMaster images of size 500×500, mosaiced with the Bayer CFA pattern.
  • Compared methods include conventional, model-based, unfolding, and learning-based demosaicing approaches, including FlexISP, MMNet, RLDD, DeepJoint, RNAN, LSSC, IRI, and MLRI.
  • DPIR achieves an average PSNR improvement up to 1.8dB over IRCNN+ and significantly outperforms the other model-based methods, while RNAN and MMNet achieve the best results.
  • On a Kodak test image, DPIR produces the best visual results among model-based methods, while Matlab’s demosaicing introduces zipper effects and false color artifacts.
  • Intermediate Results and Convergence: Across iterations, DRUNet smooths x, the data update makes x less smooth, and xk and zk PSNR sequences eventually converge despite an initial decrease.

6 DISCUSSION

The discussion examines how denoiser inputs evolve across restoration tasks and iterations. It concludes that denoising and data updates jointly restore images, but general-purpose priors reduce efficiency and specialization compared with task-specific unfolding methods.

  • Denoiser-input noise distributions differ across deblurring, super-resolution, and demosaicing because they correlate with each task’s degradation operation.
  • In deblurring, the later-iteration noise distribution becomes more Gaussian-like as repeated iterations alleviate blur-kernel-induced blurriness.
  • The denoiser prior removes noise together with some fine details, while the subsequent data subproblem alleviates degradation and adds lost details back.
  • Deep plug-and-play restoration loses efficiency and specialization because it uses a general-purpose Gaussian denoiser prior and manually selected hyper-parameters.

7 CONCLUSION

The paper trains flexible deep denoisers and integrates them into HQS-based plug-and-play restoration for three image restoration tasks. Experiments and analysis support improved model-based effectiveness, flexibility without task-specific training, and comparable performance to learning-based methods.

  • The proposed HQS iterations alternate a closed-form data subproblem with a prior subproblem replaced by a trained deep denoiser.
  • Extensive experiments analyze parameter setting, intermediate results, and empirical convergence across deblurring, super-resolution, and color image demosaicing.
  • Deep denoiser priors boost model-based effectiveness through implicit prior modeling and remain more flexible than learning-based methods without task-specific training.
  • The paper identifies integrating other deep image priors, such as deep generative priors, as a direction for further study.
Loading 2008.13751v2…