Source-linked AI summary
DeepRED: Deep Image Prior Powered by RED
Gary Mataev, Michael Elad, Peyman Milanfar
TL;DR
Inverse imaging problems are ill-posed, making regularization central, while DIP still falls short of unsupervised state-of-the-art alternatives. DeepRED combines DIP with RED-based regularization and is reported as effective across inverse problems, including state-of-the-art unsupervised deblurring results.
Problem
Inverse imaging problems are ill-posed, and DIP results still fall short of unsupervised state-of-the-art alternatives.
Method
DeepRED brings RED into DIP and merges the resulting terms while DIP training also serves as inference.
Results
DeepRED shows state-of-the-art unsupervised results for image deblurring and is described as effective for various inverse problems.
Takeaways & Limitations
The reported scheme offers a way to further boost DIP performance for inverse-problem recovery.
Takeaways & Limitations
Further work is required to better understand and improve the scheme.
Abstract
from arXiv · showhide
Inverse problems in imaging are extensively studied, with a variety of strategies, tools, and theory that have been accumulated over the years. Recently, this field has been immensely influenced by the emergence of deep-learning techniques. One such contribution, which is the focus of this paper, is the Deep Image Prior (DIP) work by Ulyanov, Vedaldi, and Lempitsky (2018). DIP offers a new approach towards the regularization of inverse problems, obtained by forcing the recovered image to be synthesized from a given deep architecture. While DIP has been shown to be quite an effective unsupervised approach, its results still fall short when compared to state-of-the-art alternatives. In this work, we aim to boost DIP by adding an explicit prior, which enriches the overall regularization effect in order to lead to better-recovered images. More specifically, we propose to bring-in the concept of Regularization by Denoising (RED), which leverages existing denoisers for regularizing inverse problems. Our work shows how the two (DIP and RED) can be merged into a highly effective unsupervised recovery process while avoiding the need to differentiate the chosen denoiser, and leading to very effective results, demonstrated for several tested problems.
1 Introduction
Inverse imaging problems recover unknown images from corrupted measurements, and deep learning has introduced new regularization strategies such as DIP. DeepRED combines DIP with RED to improve unsupervised recovery across several inverse problems.
- Inverse imaging problems recover an unknown image from a corrupted measurement and commonly formulate recovery as energy minimization.
- DIP uses the deep network itself as an implicit regularizer by assuming the unknown image is generated by a learned network.The network is adapted separately to each corrupted image.
- DIP is effective on denoising, JPEG artifact removal, inpainting, and super-resolution, but falls short of unsupervised state-of-the-art alternatives.
- DeepRED restores explicit regularization by merging DIP with Regularization by Denoising, using denoisers to complement DIP’s architecture-based regularization.RED provides multiple regularization options and can use different denoisers, including NLM and BM3D.
- ADMM trains the combined DIP+RED objective without explicitly differentiating the denoising function and provides stability with respect to the stopping rule.
- DeepRED is tested on denoising, single-image super-resolution, and deblurring, with marked improvements over native RED and DIP and state-of-the-art unsupervised deblurring results.
2 Background
The paper frames image recovery as inverse problems with known linear degradations and emphasizes DIP and RED as complementary regularization approaches. DIP uses a deep network’s architecture and per-image optimization, while RED converts a denoiser into an explicit regularizer.
- Inverse problems: Image recovery is modeled as y = Hx + v, where H is a known linear degradation matrix and v is additive white Gaussian noise.
- Inverse problems: Changing H yields several recovery tasks, including deblurring, inpainting, super-resolution, and tomographic reconstruction.
- Deep Image Prior: DIP constrains the unknown image to x = T_Θ(z), with fixed random input z and learnable network parameters Θ, then presents T_Θ(z) as the recovered image.
- Deep Image Prior: DIP training is also inference: parameters are optimized separately for each input image without an ideal label, by matching the network output to the corrupted measurement.
- Deep Image Prior: DIP avoids poor-quality overfit solutions through the network architecture’s implicit regularization and early stopping, but its reported performance varies across denoising, super-resolution, and inpainting comparisons.
- Regularization by Denoising: RED defines an explicit regularizer from a chosen denoiser f(·), with gradient x − f(x) under mild conditions and a convex functional.
- DeepRED motivation: The paper’s goal is to bring RED into DIP to boost performance by combining DIP’s network-based prior with explicit denoising regularization.
3 The Proposed DeepRED Scheme
DeepRED merges DIP and RED through an ADMM-based optimization that alternates updates of the network parameters, image, and multipliers. The scheme avoids direct denoiser differentiation, adds a proximity-based stabilizing effect, and is evaluated with empirical descent and convergence but without general guarantees.
- Algorithm derivation: DeepRED merges DIP and RED into a single objective and uses ADMM to handle the equality constraint x = T_Θ(z).
- Algorithm derivation: The method avoids directly differentiating the denoiser, which would require back-propagating through repeated denoiser calls.
- Algorithm derivation: ADMM sequentially updates Θ, x, and u; the Θ update resembles DIP back-propagation while adding proximity regularization toward x − u.
- Algorithm derivation: The proximity term provides an additional stabilizing and robustifying effect to DIP minimization.
- Algorithm derivation: The x update can use a RED fixed-point iteration based on ∇ρ(x) = x − f(x), or a steepest-descent step with step size c.
- Limitations and empirical behavior: Because T_Θ(z) is highly nonlinear, convergence guarantees are unavailable; empirical tests nevertheless show a tendency toward consistent descent and convergence.
- Implementation: In experiments, one denoiser application is used per ADMM round, with optional parallel execution and less frequent denoiser calls to reduce runtime.
4 Experimental Results
The experiments evaluate DeepRED for denoising, single-image super-resolution, and deblurring, examining numerical behavior and comparisons with DIP, RED, DIP+TV, and other restoration methods. Across these tests, DeepRED generally improves recovery quality and is reported among the strongest unsupervised methods, while adding denoising computation increases runtime.
- Experimental scope: DeepRED is evaluated on image denoising, single-image super-resolution, and image deblurring, with comparisons against DIP, RED, DIP+TV, and alternative methods.The experiments also assess numerical behavior and position DeepRED among unsupervised restoration algorithms.
- Image Denoising: 29.13dB is the average PSNR of the NLM filter, 29.3dB is obtained when NLM is used in RED, and DeepRED reaches 31.24dB versus DIP’s 30.53dB.The reported DeepRED gain over DIP is 0.71dB.
- Runtime: At 1800 iterations, DIP averages 6.6 minutes on nine test images, whereas DeepRED requires 9.5 minutes because of additional denoising computations.The paper notes that both optimization processes are demanding and that DeepRED is clearly slower under the same iteration count.
- Single Image Super-Resolution (SISR): DeepRED is consistently better than both DIP and RED alone on the reported super-resolution datasets.The paper reports comparisons on Set5 and Set14 and states that DeepRED approaches recent supervised SISR methods.
- Numerical behavior: The DeepRED loss shows a consistent descent, with small fluctuations attributed to noise injected at each iteration.The ADMM equality-constraint gap narrows toward satisfying x = T_Θ(z), while output PSNR is tracked over iterations.
5 Conclusions
DeepRED combines DIP with RED to strengthen unsupervised inverse-problem recovery and is reported as effective across various problems. The authors identify practical speed, denoiser quality, and regularization understanding as priorities for further work.
- DeepRED combines DIP with RED, using an existing denoising algorithm to regularize inverse problems.
- DeepRED is reported as a very effective machine for handling various inverse problems.
- DIP and DeepRED remain less practical and appealing because they should be sped up.
- Better denoisers within RED, potentially deep-learning-based, may further boost performance.
- A more thorough study is needed to understand the regularization effects of DIP and DeepRED.
- DIP and DeepRED estimate maximum a posteriori probabilities, unlike supervised regression methods targeting minimum mean squared error, with perceptual-quality and PSNR trade-offs.