PaperlayerView exact revision on arXiv

Source-linked AI summary

Denoising Prior Driven Deep Neural Network for Image Restoration

Weisheng Dong, Peiyao Wang, Wotao Yin, Guangming Shi, Fangfang Wu, Xiaotong Lu

arXiv:1801.06756v2cs.CV

TL;DR

Image-restoration networks often ignore the degradation model, while iterative denoising-based methods can be costly and difficult to optimize end to end. This paper unfolds an efficient denoising algorithm into a network combining denoisers with back-projection modules, achieving competitive results across denoising, super-resolution, and deblurring, including gains of up to 0.42 dB and 0.63 dB over a denoising network for deblurring and super-resolution.

  • Problem

    Existing image-restoration networks largely ignore observation models, while iterative denoising-based methods can be computationally expensive and resist end-to-end joint optimization.

  • Method

    The method unfolds an efficient denoising-based restoration algorithm into a feed-forward network with multi-scale DCNN denoisers interleaved with back-projection modules for observation consistency.

  • Results

    The proposed network achieves competitive results across image denoising, super-resolution, and deblurring, with average PSNR gains of up to 0.42 dB and 0.63 dB over a denoising network for deblurring and super-resolution, respectively.

  • Takeaways & Limitations

    The network combines a learned denoising prior with the observation model to support a general image-restoration approach across multiple tasks.

Abstract

from arXiv · show

Deep neural networks (DNNs) have shown very promising results for various image restoration (IR) tasks. However, the design of network architectures remains a major challenging for achieving further improvements. While most existing DNN-based methods solve the IR problems by directly mapping low quality images to desirable high-quality images, the observation models characterizing the image degradation processes have been largely ignored. In this paper, we first propose a denoising-based IR algorithm, whose iterative steps can be computed efficiently. Then, the iterative process is unfolded into a deep neural network, which is composed of multiple denoisers modules interleaved with back-projection (BP) modules that ensure the observation consistencies. A convolutional neural network (CNN) based denoiser that can exploit the multi-scale redundancies of natural images is proposed. As such, the proposed network not only exploits the powerful denoising ability of DNNs, but also leverages the prior of the observation model. Through end-to-end training, both the denoisers and the BP modules can be jointly optimized. Experimental results on several IR tasks, e.g., image denoisig, super-resolution and deblurring show that the proposed method can lead to very competitive and often state-of-the-art results on several IR tasks, including image denoising, deblurring and super-resolution.

I. INTRODUCTION · II. RELATED WORK

The paper frames image restoration as recovering a high-quality image from a degraded observation while combining observation-model priors with the expressive power of deep networks. It proposes unfolding an efficient denoising-based restoration algorithm into a feed-forward network with learned denoisers and back-projection modules, and situates this approach among model-based and learning-based methods.

  • I. INTRODUCTION: Image restoration reconstructs a high-quality image x from a degraded observation y modeled as y = Ax + n, where A represents the degradation system and n additive noise.The formulation covers applications including low-level image processing, medical imaging, remote sensing, and surveillance.
  • I. INTRODUCTION: Existing image restoration methods are broadly divided into model-based methods that solve optimization problems and learning-based methods that learn mappings from observations to desirable images.The paper’s related framing contrasts Bayesian optimization with discriminative learning approaches.
  • I. INTRODUCTION: In Bayesian model-based restoration, maximizing P(x|y) combines a data-likelihood term with a prior term that characterizes knowledge about the original image.For additive Gaussian noise, the likelihood corresponds to an ℓ2-norm data-fidelity term.
  • I. INTRODUCTION: The resulting optimization minimizes both the ℓ2-norm data-fidelity term and a regularizer J(x) weighted by λ, making prior design critical to high-quality reconstruction.The paper notes that proposed regularizers include total variation and sparsity-based regularizers using off-the-shelf transforms or learned dictionaries.
  • I. INTRODUCTION: Learning-based methods infer missing high-frequency details or high-quality images directly from observations, including mappings from low-resolution patches to high-resolution patches in super-resolution.These methods differ from model-based approaches by learning mapping functions rather than relying on carefully designed priors.
  • I. INTRODUCTION: The proposed method combines optimization-based and discriminative learning-based restoration by developing an efficiently computable denoising-based iterative algorithm and unfolding it into a feed-forward neural network.The network layers mimic the proposed algorithm’s process flow.
  • II. RELATED WORK: The unfolded network uses denoiser modules and back-projection modules, while its DCNN denoiser exploits multi-scale redundancies in natural images.The supplied passage introduces these components as part of the proposed architecture; the related-work section identifies denoising-based and discriminative learning-based restoration as relevant method families.

A. Denoising-based IR methods · B. Deep network based IR methods

Denoising-based IR methods decouple data fidelity from image priors and solve the resulting subproblems through alternating optimization, with denoising providing the proximity step. Deep network methods learn nonlinear restoration mappings while increasingly incorporating observation models, domain knowledge, and denoisers as proximal operators.

  • A. Denoising-based IR methods: Denoising-based IR methods decouple the optimization into separate data-likelihood and prior subproblems, enabling more complex image priors than explicitly expressed regularizers.An auxiliary variable v is introduced to separate the two terms.
  • A. Denoising-based IR methods: ADMM converts the resulting equally constrained optimization problem into two alternating subproblems.The cited framework applies ADMM to the denoising-based formulation.
  • A. Denoising-based IR methods: The x-subproblem is a simple quadratic optimization with a closed-form solution, while the augmented Lagrange multiplier is updated using the primal residual.The multiplier update is u(t+1) = u(t) + ρ(x(t+1) −v(t+1)).
  • A. Denoising-based IR methods: The v-subproblem is a proximity operator evaluated at x(t+1) + u(t), and its solution can be obtained with a denoising algorithm.Alternating x and v updates solve the original optimization problem until convergence.
  • B. Deep network based IR methods: Deep convolutional networks have been applied to low-level image processing, including nonlinear mapping from LR patch space to HR patch space.Prior work also introduced residual learning for image restoration.
  • B. Deep network based IR methods: Very deep CNNs achieved state-of-the-art SR results, while deep recursive residual learning was proposed to ease training difficulties in very deep networks.These approaches extend DCNN-based restoration beyond direct shallow mappings.
  • B. Deep network based IR methods: Some IR networks incorporate domain knowledge and observation models by aligning layers with iterative sparse-coding steps, although hand-crafted gradient and PCG components remained.The cited framework was based on LISTA and used PCG to reconstruct images from regularized gradients.
  • B. Deep network based IR methods: DCNN-based pixel-domain denoisers have also been learned as proximal operators within convex energy-minimization algorithms for image restoration.This approach learns denoisers rather than regularizers in the gradient domain.

III. PROPOSED DENOISING-BASED IMAGE RESTORATION ALGORITHM

The proposed denoising-based image restoration algorithm uses half-quadratic splitting to alternate efficient updates for the image and denoised variable. Under sufficient descent and related regularity conditions, the iterates converge to stationary points, while a DCNN denoiser is interpreted as yielding loose local-minimizer convergence.

  • Optimization formulation: Half-quadratic splitting converts the equally constrained denoising-based restoration problem into a nonconstrained optimization problem solved by alternating two sub-problems.The two sub-problems update the image variable x and denoised variable v.
  • Alternating updates: The x-update avoids explicitly inverting the large matrix W by using a precomputed matrix and one efficient iterative update, while the v-update applies a denoiser.Updating x(t+1) once is sufficient for convergence to a local optimal solution, according to the passage.
  • Algorithm 1: Algorithm 1 initializes x(0) = A⊤y, alternates v(t+1) = f(x(t)) with x(t+1) = Āx(t) + δA⊤y + δηv(t+1), and stops when converged.For super-resolution and deblurring, A⊤y is implemented using task-specific upsampling, filtering, or transposed convolution.
  • Convergence analysis: If the energy is lower bounded and coercive and the denoiser satisfies sufficient descent, a subsequence of (x(t), v(t)) converges to a stationary point.The passage states this result as Theorem 1 for Algorithm 1.
  • Convergence analysis: With the Kurdyka-Łojasiewicz property, subsequence convergence upgrades to full-sequence convergence; for a DCNN denoiser viewed as an approximately orthogonal projection, the algorithm loosely converges to a local minimizer.The KL-property examples include real analytic, semi-algebraic, and locally strongly convex functions.

IV. DENOISING PRIOR DRIVEN DEEP NEURAL NETWORK

The proposed method unfolds the iterative denoising-prior algorithm into a deep network that executes T iterations while enabling joint end-to-end optimization. Its architecture combines denoising modules with trainable linear layers whose implementations are adapted to denoising, deblurring, and super-resolution.

  • Network motivation: Unfolding Algorithm 1 into a deep network addresses its high iteration cost and the inability to jointly optimize its parameters and denoiser end to end.The original algorithm requires many iterations to converge and is computationally expensive.
  • Network architecture: The network exactly executes T iterations, beginning with a degradation-matrix parameterized linear layer that produces x(0).Each estimate then enters the denoising module and another linear layer parameterized by ¯A.
  • Network architecture: A shortcut connection combines the denoised signal v(1), weighted by δ1,1, with outputs involving ¯A and A⊤y, weighted by δ1,2, to obtain the next intermediate representation.The denoised signal and linear-layer outputs are combined after the initial estimate passes through the denoising module.
  • Task-specific implementation: For image denoising, A = A⊤= I and ¯A = λI, with λ = 1 −δ(1 + η).For image deblurring, A⊤ can use a convolutional layer, while ¯A = aI−δA⊤A can be computed by convolutional operations.
  • Task-specific implementation: For super-resolution, Gaussian downsampling uses A = DH and A⊤= H⊤D⊤, whereas bicubic downsampling uses bicubic interpolation with s = 2, 3, 4.The Gaussian case upsamples by zero-padding and filtering, and the involved convolutional filters can be discriminatively learned.

A. The DCNN denoiser

The DCNN denoiser uses an encoder–decoder architecture with feature extraction and image reconstruction stages, drawing on U-net- and sharpMask-like designs. Its decoder restores spatial detail through skip-style feature fusion and structured convolutional blocks.

  • A. The DCNN denoiser: The denoising network comprises feature extraction and image reconstruction parts, similarly to U-net and sharpMask architectures.The framework can also accommodate other, more powerful denoising networks.
  • A. The DCNN denoiser: The reconstruction stage uses four feature decoding blocks followed by upsampling layers to increase feature-map spatial resolution.These blocks form the decoder portion of the network.
  • A. The DCNN denoiser: Encoder features at matching spatial resolutions are fused with upsampled decoder features to compensate for lost spatial information.The fusion produces newly upsampled feature maps for reconstruction.
  • A. The DCNN denoiser: Each decoding block contains five convolutional layers: one 1 × 1 layer reducing channels from 128 to 64, followed by four 3 × 3 layers producing 64-channel features.All listed convolutional layers use ReLU nonlinearities.
  • A. The DCNN denoiser: The final decoder features are upsampled with scaling factor 2 after the convolutional layers.This operation increases the spatial resolution of the generated feature maps.

B. Overall network training

The overall network is trained end-to-end without pre-training, using shared denoiser parameters to reduce model size and overfitting. Training uses MSE loss and the ADAM optimizer with specified settings.

  • The DCNN denoisers are trained jointly with the overall deep network through end-to-end training rather than pre-training.
  • Sharing parameters across all DCNN denoisers reduces the number of parameters and helps avoid over-fitting.
  • The network is trained with mean square error loss and ADAM using β1 = 0.9, β2 = 0.999, and ϵ = 10−8.The convolutional kernels were initialized using Xavier initialization.

V. EXPERIMENTAL RESULTS · A. Ablation study

The experiments evaluate the proposed network on image denoising, deblurring, and super-resolution, using six unfolded iterations across tasks. Ablation studies examine degradation-matrix initialization, denoiser initialization, and incorporation of the degradation model, with gains reported for deblurring and super-resolution.

  • V. EXPERIMENTAL RESULTS: The experimental evaluation covers image denoising, deblurring, and super-resolution tasks.Each model was trained separately for the different image-restoration tasks.
  • V. EXPERIMENTAL RESULTS: T = 6 iterations were fixed for all image-restoration tasks because they generally produced satisfactory results.Training used 1000 images of size 256 × 256 from the dataset used in.
  • A. Ablation study: The degradation-matrix initialization ablation compares DPDNN-A, initialized with A, against DPDNN-Random.For super-resolution with bicubic downsampling, A was implemented using the bicubic interpolator function.
  • A. Ablation study: The denoiser-initialization ablation compares a pre-trained denoiser, DPDNN-Pretrain, with a randomly initialized denoiser, DPDNN-Random.This study trained on 450,000 patches of size 40×40 with simulated noise levels from [0, 50].
  • A. Ablation study: The degradation-model ablation compares the denoising network, Den-network, with the proposed DPDNN for image deblurring and super-resolution.The denoising network structure is shown in Fig.1(b), and results are reported in Tables V–VI.
  • A. Ablation study: 0.42 dB is the maximum average PSNR gain over Den-network for image deblurring.The comparison uses deblurred images on the Set10 dataset.
  • A. Ablation study: 0.63 dB is the maximum average PSNR gain over Den-network for super-resolution, demonstrating the benefit of incorporating the degradation model.The comparison evaluates reconstructed HR images.

B. Image denoising

For image denoising, the proposed network uses iterative denoising with back-projection-style noise reinjection, and multiple denoising iterations improve results. It slightly outperforms leading model-based and deep-learning denoisers, by up to 0.2 dB on average over MemNet.

  • Image denoising: Multiple denoising iterations improve the proposed network’s image-denoising results over a single denoising process.With A = I, the weighted noise image is added back to the denoised image for the next iteration.
  • Image denoising: The network is trained using 40 × 40 image patches extracted from training images.
  • Image denoising: Up to 0.2 dB on average, the proposed network outperforms MemNet while competing with BM3D, EPLL, WNNM, TNRD, and DnCNN-S.MemNet is comparable with DnCNN-S at low noise levels and outperforms it at higher noise levels.

C. Image deblurring · D. Image super-resolution

The deblurring experiments train on augmented blurred patches and show that the proposed method outperforms MemNet by up to 0.44 dB on average. For super-resolution, the method is evaluated under bicubic and Gaussian downsampling across four standard datasets, with performance close to or slightly exceeding MemNet depending on the setting.

  • C. Image deblurring: The deblurring model is trained on 120 × 120 blurred patches with additive Gaussian noise and augmentation, producing 450, 000 training patches.The experiments consider 25 × 25 Gaussian and motion blur kernels.
  • C. Image deblurring: The deblurring comparison uses identical training patches for the proposed network and MemNet, with evaluation limited to grayscale deconvolution.The method can reportedly be extended to color image deblurring.
  • C. Image deblurring: The proposed deblurring method outperforms MemNet by up to 0.44 dB on average.The reported comparisons use PSNR results generated from authors’ released codes or implementations based on the papers, except for MemNet.
  • D. Image super-resolution: Super-resolution experiments model bicubic downsampling with scaling factors s = 2, 3, 4 and Gaussian blur followed by subsampling.Each network is trained for both downsampling cases.
  • D. Image super-resolution: Super-resolution evaluation uses Set5, Set14, BSD100, and Urban100, comparing the proposed method with SRCNN, VDSR, MemNet, TNRD, and DnCNN.Urban100 contains 100 high-quality images.
  • D. Image super-resolution: For bicubic downsampling, the proposed method and MemNet outperform other methods on average for Set5, although MemNet is slightly better there.The passage also states that the proposed method slightly outperforms MemNet on average in the comparison group, but does not provide the associated table values.

E. Complexity anlaysis · VI. CONCLUSION

The complexity analysis reports that parameter sharing and progressive feature-map downsampling make the six-stage proposed network smaller and faster than MemNet. The conclusion describes a denoising-based IR network that incorporates observation models by unfolding an efficient algorithm into denoising and back-projection modules.

  • E. Complexity anlaysis: The proposed network was compared with DnCNN and MemNet using the number of network parameters listed in Table XII.The comparison used source code downloaded from the authors’ websites.
  • E. Complexity anlaysis: MemNet has almost three times as many parameters as the proposed network because it is very deep, reaching up to 80 layers.The proposed network reduces total parameters by sharing denoiser parameters across stages.
  • E. Complexity anlaysis: The proposed network has L = 6 stages, yet its running time is smaller than MemNet’s.This result is attributed to gradually downsampled denoiser feature maps.
  • E. Complexity anlaysis: Gradual feature-map downsampling substantially reduces the proposed network’s computational complexity.The reduction is presented as the reason the six-stage network remains faster than MemNet.
  • VI. CONCLUSION: The paper proposes a novel deep neural network for general image restoration tasks based on a denoising-based IR framework.Unlike current deep IR methods, the framework explicitly addresses observation models that are generally ignored.
  • VI. CONCLUSION: An efficient denoising-based IR algorithm was developed and unfolded into a deep network containing multiple denoising modules interleaved with back-projection modules.The supplied conclusion passage ends while describing this network composition.

APPENDIX CONVERGENCE

Under a lower-bounded, coercive energy function, Algorithm 1 has a subsequence converging to a stationary point when the denoiser satisfies sufficient descent. The proof establishes boundedness, summability, and vanishing limiting gradients along the convergent subsequence.

  • Convergence theorem: Algorithm 1 has a subsequence converging to a stationary point of the energy function when ξ is lower bounded and coercive and the denoiser satisfies sufficient descent.The sufficient descent condition involves a continuous limiting subgradient and c2 > 0.
  • Proof: The x-gradient is Lipschitz continuous with constant ∥A⊤A∥+ η, so a gradient step with δ ∈(0, 2 ∥A⊤A∥+η) satisfies descent.This descent property follows from the stated Lipschitz continuity of ∇xξ(x, v).
  • Proof: Because ξ(x(t), v(t)) is coercive and monotonically nonincreasing, the iterates remain bounded and possess a convergent subsequence.Otherwise, coercivity would imply ξ(x(t), v(t)) →∞, contradicting monotonicity.
  • Proof: Telescopic summation of the descent inequalities yields summability properties for the iterative changes.The argument uses monotonicity and boundedness of ξ(x(t), v(t)).
  • Proof: At the subsequential limit, both ∇xξ(x∗, v∗) and the limiting v-subgradient vanish, so (x∗, v∗) is stationary.Continuity and the summability relations establish the two vanishing conditions.
Loading 1801.06756v2…