Source-linked AI summary
Image Restoration using Total Variation Regularized Deep Image Prior
Jiaming Liu, Yu Sun, Xiaojian Xu, Ulugbek S. Kamilov
TL;DR
Image restoration is often ill-posed and traditionally relies on analytical priors, while DIP uses an untrained CNN as an image prior. This paper adds an explicit TV penalty to DIP, yielding improved denoising and deblurring performance that is comparable to or sometimes surpasses leading alternatives.
Problem
The paper addresses ill-posed image restoration and the contrast between traditional analytical sparsity priors and CNN-based regularization without data-driven training.
Method
DIP-TV combines an implicit CNN prior with an explicit total variation penalty, restricting synthesized solutions to be piecewise smooth.
Results
DIP-TV improves DIP across denoising and deblurring experiments, with reported gains of around 0.5 dB in grayscale denoising and at least 0.54 dB in PSNR overall.
Takeaways & Limitations
TV regularization further improves DIP and produces high-quality restoration results comparable to, and sometimes surpassing, BM3D and IRCNN.
Takeaways & Limitations
The measurement-noise model assumes additive white Gaussian noise with variance σ^2.
Abstract
from arXiv · showhide
In the past decade, sparsity-driven regularization has led to significant improvements in image reconstruction. Traditional regularizers, such as total variation (TV), rely on analytical models of sparsity. However, increasingly the field is moving towards trainable models, inspired from deep learning. Deep image prior (DIP) is a recent regularization framework that uses a convolutional neural network (CNN) architecture without data-driven training. This paper extends the DIP framework by combining it with the traditional TV regularization. We show that the inclusion of TV leads to considerable performance gains when tested on several traditional restoration tasks such as image denoising and deblurring.
1 Introduction
Image reconstruction is ill-posed and traditionally uses analytical priors, while deep image prior instead exploits CNN structure without data-driven training. DIP-TV combines CNN regularization with an explicit TV penalty to restrict reconstructions to piecewise-smooth solutions and improve restoration performance.
- Ill-posed image reconstruction is traditionally regularized using priors such as nonnegativity, transform-domain sparsity, and self-similarity.
- Deep image prior regularizes reconstruction by optimizing CNN weights to synthesize measurements from a random input vector, without data-driven training.The framework relies on CNNs representing natural images better than random noise and certain degradations.
- DIP-TV combines implicit CNN regularization with an explicit TV penalty, restricting CNN-synthesized solutions to piecewise-smooth images.
- DIP-TV experimentally outperforms traditional DIP and TV formulations and performs on par with BM3D and IRCNN.
2 Background
The paper frames restoration as a noisy linear inverse problem whose ill-posedness motivates regularization. It reviews least-squares, TV, supervised CNN, and DIP formulations, where CNN parameters are optimized to reconstruct images from measurements or random inputs.
- Restoration as an inverse problem: Image restoration reconstructs an unknown image from measurements through a degradation matrix and additive white Gaussian measurement noise.The unknown image is x∈R^N, measurements are y∈R^M, and noise variance is σ^2.
- Experimental setting: The experiments use 14 grayscale images, including the Figure 1 image set, for restoration evaluation.
- Regularization: Regularized least squares combines measurement data fidelity with a prior-based regularizer whose strength is controlled by λ>0.
- Traditional priors: Total variation promotes sparsity in image gradients using an ℓ1-based anisotropic penalty.The finite-difference operators act along the first and second dimensions of a two-dimensional image.
- CNN restoration: A supervised CNN approach optimizes parameters Θ so the network output becomes a restored image under a chosen loss function.The formulation can be optimized with stochastic-gradient methods such as ADAM.
- Deep image prior: DIP optimizes randomly initialized CNN parameters so a network driven by random input reproduces the target measurement without supervised training.The random input vector is denoted z∈R^N.
3 Proposed Method
DIP-TV extends DIP by adding total variation regularization to the CNN-based reconstruction objective, constraining synthesized images toward piecewise smooth solutions. The method uses a standard optimization procedure and supports grayscale and color restoration.
- 3 Proposed Method: DIP-TV replaces the basic DIP objective with one that adds a TV penalty to the CNN reconstruction optimization.The data-fidelity term is retained while TV regularizes the CNN output.
- 3 Proposed Method: The added TV term restricts CNN-synthesized solutions to be piecewise smooth.
- 3 Proposed Method: Optimization of DIP-TV is similar to CNN training and can use standard optimization algorithms.
- 3 Proposed Method: The implemented network adapts a U-net with convolutional layers in its skip connections and scaling-expanding decoder structure.The architecture increases the effective receptive field as inputs move deeper into the network.
- 3 Proposed Method: The denoising and deblurring figures compare DIP-TV with established restoration methods using SNR-labeled reconstructions or visual comparisons.Figure 4 covers EPLL, BM3D, TV-FISTA, DIP, and DIP-TV; Figure 5 covers IRCNN, DIP, and DIP-TV.
- 3 Proposed Method: DIP-TV handles grayscale and color images, with anisotropic TV jointly regularizing all three color channels.The input z can be initialized with uniform noise and further optimized.
4 Experiments
Experiments evaluate DIP-TV on grayscale and color denoising and on image deblurring across standard test images. TV improves DIP across the reported settings and brings performance close to or beyond competing restoration methods.
- 4 Experiments: Experiments use 14 grayscale and 8 color images from set12, set14, and BSD68 for denoising and deblurring evaluation.The images include standard examples such as Monarch, Peppers, Lena, and Jet.
- 4.1 Image Denoising: DIP-TV outperforms original DIP by around 0.5 dB across grayscale denoising input SNR levels from 5 dB to 20 dB.The experiments use AWGN at input SNR levels of 5, 10, 15, 20, and 25 dB.
- 4.1 Image Denoising: DIP-TV exceeds DIP by at least 0.2 dB on the color denoising test images.It outperforms CBM3D as noise increases, including at σ≥35.
- 4.2 Image Deblurring: For Peppers deblurring, DIP-TV raises PSNR by over 0.45 dB against DIP and exceeds IRCNN by 0.15 dB.The test uses a realistic motion blur example with σ=7.65, while average results are also reported for grayscale and color images.
- 4.2 Image Deblurring: Across deblurring results, TV regularization improves DIP by at least 0.54 dB in PSNR and makes it more comparable with IRCNN.For standard Gaussian blur with σ=2, DIP-TV is only 0.01 dB below IRCNN in average color-image PSNR.
5 Conclusion
The paper concludes that DIP-TV is a simple extension of DIP that produces high-quality denoising and deblurring results. Its performance is equivalent to, and sometimes surpasses, leading alternatives such as BM3D and IRCNN.
- 5 Conclusion: DIP-TV improves DIP by combining CNN-based image priors with an ℓ1-norm penalty on image gradients.The reported denoising and deblurring results show that TV regularization further improves DIP.
- 5 Conclusion: DIP-TV achieves performance equivalent to and sometimes surpassing leading alternatives such as BM3D and IRCNN.
- 5 Conclusion: Results on image denoising and deblurring demonstrate that TV regularization can further improve DIP and provide high-quality results.