PaperlayerView exact revision on arXiv

Source-linked AI summary

Non-Local Color Image Denoising with Convolutional Neural Networks

Stamatios Lefkimmiatis

arXiv:1611.06757v2cs.CVcs.AI

TL;DR

Image denoising networks often use local architectures that miss natural images’ non-local self-similarity. This paper introduces discriminatively trained non-local networks for grayscale and color denoising, achieving the best overall performance in the reported comparisons while retaining convolutional-network links for parallel GPU computing.

  • Problem

    Existing image-restoration networks are predominantly local and do not account for natural images’ inherent non-local self-similarity, despite conventional non-local methods’ superior reconstruction results.

  • Method

    The paper builds a discriminatively trained denoising network around grouping similar patches and joint filtering, unrolls proximal gradient iterations, and parameterizes stages with convolutional, de-convolutional, and trainable nonlinear layers.

  • Results

    The proposed non-local models achieve the best overall performance across the reported noise levels, while CNLNet5×5 improves PSNR over CBM3D by approximately 0.2-0.3 dBs.

  • Takeaways & Limitations

    Non-local modeling coupled with discriminative learning improves reported restoration performance, and the models’ direct convolutional-neural-network links support parallel GPU computing.

  • Takeaways & Limitations

    The paper leaves open whether a single model can handle all noise levels and what modifications are needed to apply the non-local models efficiently to other reconstruction problems.

Abstract

from arXiv · show

We propose a novel deep network architecture for grayscale and color image denoising that is based on a non-local image model. Our motivation for the overall design of the proposed network stems from variational methods that exploit the inherent non-local self-similarity property of natural images. We build on this concept and introduce deep networks that perform non-local processing and at the same time they significantly benefit from discriminative learning. Experiments on the Berkeley segmentation dataset, comparing several state-of-the-art methods, show that the proposed non-local models achieve the best reported denoising performance both for grayscale and color images for all the tested noise levels. It is also worth noting that this increase in performance comes at no extra cost on the capacity of the network compared to existing alternative deep network architectures. In addition, we highlight a direct link of the proposed non-local models to convolutional neural networks. This connection is of significant importance since it allows our models to take full advantage of the latest advances on GPU computing in deep learning and makes them amenable to efficient implementations through their inherent parallelism.

1. Introduction

Image-restoration networks often use general or local architectures that underuse problem-specific knowledge and natural-image self-similarity. The paper designs discriminatively trained non-local denoising networks that learn non-local operators and connect directly to CNN implementations.

  • Deep networks improve vision and image-processing performance through deep nonlinear modeling combined with discriminative learning on large datasets.
  • Most image-restoration networks use general architectures, while recent task-specific models remain local and omit natural-image non-local self-similarity.
  • Conventional non-local methods such as BM3D have achieved strong denoising results by exploiting repeated image patterns over distant locations.
  • The proposed network learns a non-local regularization operator and potential function through loss-based discriminative training instead of manually designing the regularization functional.
  • The architecture explicitly groups similar patches, jointly filters them, unrolls proximal-gradient iterations, and supports grayscale and color denoising.
  • The proposed non-local networks have a direct CNN link, enabling GPU-based parallel and efficient implementations.

2. Variational Image Restoration Revisited

Variational image restoration combines noisy observations with image priors through an objective that balances data fidelity and regularization. The paper reviews local and non-local regularization and proximal-gradient minimization, motivating learned operators and potentials for efficient denoising networks.

  • Image denoising restores a latent grayscale or color image from an observation corrupted by i.i.d. Gaussian noise.
  • Because the denoising problem is ill-posed, image evidence must be combined with suitable priors to obtain a meaningful solution.
  • Variational restoration minimizes an objective combining data fidelity D(x, y) and regularization J(x), balanced by λ ≥0.
  • Analysis-based regularizers apply an operator L to the image and a potential function φ to penalize undesired properties.
  • Total Variation is a local regularizer, whereas non-local operators model dependencies beyond restricted image neighborhoods and support competitive restoration.
  • Non-local denoising methods including NLM, BM3D, LSSC, and WNNM exploit image self-similarity and produce excellent results.
  • Proximal-gradient minimization handles non-smooth constraints by splitting the objective and iteratively combining gradient updates with projection onto the constraint set.
  • The reviewed iterative approach requires many iterations and specification of the operator L and potential φ, whose appropriate choices are difficult to determine.

3. Proposed Non-Local Network

The proposed network unrolls proximal-gradient iterations into stages that learn non-local regularization from data. Its operator groups similar patches, transforms and jointly filters them, while convolutional implementations support parallel computation and color processing uses luminance–chrominance channels.

  • Network design: The network unrolls a limited number of proximal-gradient iterations and learns the relevant parameters directly from training data.This replaces hand-designed potential functions and regularization operators with learned quantities.
  • Non-local operator: For each image patch, the non-local operator selects K closest patches, applies a two-dimensional transform, and performs weighted collaborative filtering.The operator is expressed as a composition of linear operations on grouped similar patches.
  • Computational implementation: The operator and its adjoint can be computed with convolutional operations and their transposes, enabling multi-threaded CPU and parallel GPU implementations.Patch extraction and transformation are implemented with convolutional filterbanks, while grouped data uses a 1 × 1 × K filter.
  • Potential parameterization: The potential-gradient functions are represented with mixtures of 63 Gaussian radial basis functions, allowing accurate approximation of arbitrary nonlinear functions.The Gaussian kernels have equidistant centers and share a precision parameter.
  • Stage architecture: Each stage contains convolutional and de-convolutional layers separated by trainable nonlinear functions and follows the proximal-gradient iteration closely.The network absorbs αt into the learned potential gradient ψ.
  • Color extension: For color images, an opponent-color transform produces one luminance and two chrominance channels instead of processing channels independently.The luminance channel has higher signal-to-noise ratio and contains most primitive image-structure information.

4. Discriminative Network Training

The network is trained discriminatively on noisy–ground-truth image pairs using PSNR-based loss minimization. Greedy stagewise initialization is followed by joint optimization of all stages.

  • Training objective: The network is trained for grayscale and color denoising on pairs of i.i.d. Gaussian-noisy inputs and corresponding ground-truth images.Different parameters are learned for each stage, yielding an adaptive architecture while preserving shared convolution and de-convolution filters within each stage.
  • Optimization: Because the objective is non-convex, each stage is first optimized independently with greedy training before all stages are trained jointly.Greedy training uses 100 L-BFGS iterations per stage.
  • Training objective: The loss function is the negative peak signal-to-noise ratio (PSNR), computed using the maximum intensity level for each image type.The maximum intensity is 255 for grayscale images and 1 for color images.
  • Optimization: Gradients are computed efficiently by back-propagation using the chain rule and the Jacobian of each stage’s output.L-BFGS is used to optimize the stage parameters.
  • Grayscale results: At σ = 25, NLNet5 7×7 achieves 29.95 dB PSNR, compared with 29.72 dB for TNRD5 7×7.The figure also reports 29.76 dB for both MLP and WNNM.
  • Color results: At σ = 50, CNLNet5 5×5 achieves 26.06 dB PSNR, compared with 25.65 dB for CBM3D.The noisy input has 14.15 dB PSNR.
  • Optimization: The final joint-training cost depends only on the network’s final output, not intermediate stage results.The joint objective is optimized with respect to all network parameters.

5. Experiments

Experiments evaluate grayscale and color non-local denoising models on Berkeley image sets across multiple noise levels, with comparisons against recent methods. The proposed models generally achieve leading restoration performance, including a 0.2–0.3 dB PSNR advantage over CBM3D for color images.

  • Experimental setup: Training uses 400 BSDS images, while 100 images form the validation/test set; comparisons use a standard 68-image evaluation set.Images are randomly cropped to 180 × 180 pixels for training data generation.
  • Grayscale denoising: The grayscale experiments train five-stage NLNet5×5 and NLNet5×7×7 models using eight nearest patch neighbors searched within a 31 × 31 window.A learned non-redundant patch transform is applied, and the same group indices are used across stages.
  • Grayscale denoising: Both proposed non-local grayscale models achieve the best overall performance across the tested comparisons, except at σ = 50.Table 1 reports average PSNR in dB for three noise levels over 68 Berkeley images.
  • Model comparison: NLNet5×5 achieves better restoration than CSF5×7×7 and TNRD5×7×7 despite using approximately half their parameters.The reported comparison attributes this result to exploited non-local information.
  • Color denoising: Applying single-channel denoising methods independently to color channels loses inter-channel correlations and reduces restoration quality.TNRD and MLP fall more than 1.3 dB behind BM3D, while CBM3D gains exceed 2 dB at low noise levels.
  • Color denoising: CNLNet5×5 outperforms CBM3D at every reported color noise level, with PSNR gains of approximately 0.2–0.3 dB.The color configuration uses 5×5 patches; patch-group and search-window parameters otherwise remain unchanged.

6. Conclusions and Future Work

The paper concludes that non-local self-similarity combined with discriminative learning improves grayscale and color denoising, while retaining links to convolutional neural networks. Future work considers broader inverse imaging applications and a single model for all noise levels.

  • Conclusions: The proposed architecture exploits natural images’ non-local self-similarity for grayscale and color denoising.Its design is inspired by non-local variational methods.
  • Conclusions: The authors identify non-local modeling coupled with discriminative learning as key factors in improved restoration performance.The conclusion compares the models with several recent state-of-the-art methods.
  • Conclusions: Direct links to convolutional neural networks support parallel GPU computing and efficient implementations.The conclusion highlights this connection as an implementation advantage.
  • Future work: Future work will investigate modifications needed to apply the non-local models efficiently to other reconstruction problems.The authors frame image restoration as one of several possible inverse imaging applications.
  • Future work: A further research question is whether one model can be trained to handle all noise levels.

A. Derivative Calculations

The derivative appendix defines how network parameters are learned through greedy or joint optimization across the network’s stages. It establishes the parameter set and the two training strategies used for gradient-based learning.

  • Gradient computation: The appendix derives parameter gradients using the denominator-layout matrix-calculus notation.
  • Parameterization: The network parameters are Θ = {γ_t, π_t, F_t, W_t} for t = 1,…,S across S stages.
  • Training strategies: Greedy training learns each stage’s parameters independently by minimizing that stage’s loss function.
  • Training strategies: Joint training learns the complete network parameter set simultaneously by minimizing the full-network loss.

A.1. Single-Stage Parameter Learning

Single-stage learning derives gradients for the stage loss with respect to its parameters, using chain-rule decompositions and equivalent forms of the non-local operator. The derivations cover weights, expansion coefficients, non-local filtering, and patch-transform coefficients.

  • Setup: The greedy derivation considers one training example and omits stage superscripts when unnecessary to simplify notation.
  • Gradient computation: Gradients for the loss are obtained through the chain rule and applied to the stage parameters.
  • Setup: The stage output is differentiated with respect to stage parameters after expressing the stage as a mapping from input z to output x̂.
  • Projection: The projection operator P_C(u) acts element-wise, clipping values below a, above b, or leaving values within [a,b] unchanged.
  • Parameter gradients: The appendix separately derives gradients for γ, π, W, and F, representing weights, expansion coefficients, weighted sums, and patch-transform coefficients.The non-local operator is rewritten into an equivalent expression to facilitate differentiation with respect to W.
  • Parameter gradients: Common precision is used for all Gaussian RBF mixture components in the expansion-coefficient derivation.The notation drops the component index because ε = ε_j for every component.

A.2. Joint Parameter Learning

The joint-training scheme learns parameters across all network stages simultaneously using a loss that depends only on the final output. Gradients for each stage are computed through chain-rule Jacobian derivations.

  • A.2. Joint Parameter Learning: All network-stage parameters are learned simultaneously by minimizing a loss based only on the final output.The loss is given as Eq. (18) and depends on the network output x̂S.
  • A.2. Joint Parameter Learning: The gradient with respect to each stage’s parameters Θt is computed using the chain rule.The derivation proceeds by decomposing the required gradient into stage-wise Jacobian terms.
  • A.2. Joint Parameter Learning: The remaining derivative calculations reduce to Jacobians involving the network outputs at successive stages.The text identifies Jacobians of x̂S and x̂t+1 as the quantities needed to complete the gradient computation.
  • A.2. Joint Parameter Learning: The derivation concludes by obtaining the required Jacobian expression using Eq. (48).

B. Grayscale and Color Image Denoising Comparisons

The paper compares grayscale denoising methods and a color denoising method against state-of-the-art baselines across different noise levels. Visual comparisons are supplemented by PSNR scores reported in figure captions.

  • Grayscale comparisons: Grayscale experiments compare the non-local models with TNRD, MLP, EPLL, and BM3D.The comparisons cover additional grayscale results at different noise levels.
  • Color comparisons: Color experiments compare the non-local CNN with the state-of-the-art CBM3D method.
  • Evaluation: PSNR scores in decibels accompany the visual comparisons to provide a quantitative evaluation of each method.
Loading 1611.06757v2…