Source-linked AI summary
Non-Local Recurrent Network for Image Restoration
Ding Liu, Bihan Wen, Yuchen Fan, Chen Change Loy, Thomas S. Huang
TL;DR
Deep image-restoration networks had rarely integrated non-local self-similarity through jointly trained operations. The paper proposes NLRN, which combines end-to-end non-local modules, recurrent parameter sharing, correlation propagation, and confined neighborhoods; it reports superior denoising and super-resolution results with fewer parameters.
Problem
Deep networks had limited mechanisms for jointly learning non-local self-similarity, despite its established value as an image-restoration prior.
Method
NLRN integrates an end-to-end non-local module into a recurrent network, propagating neighborhood feature correlations across adjacent states.
Results
NLRN achieves state-of-the-art performance in image denoising and super-resolution benchmarks with fewer parameters than competing network models.
Takeaways & Limitations
Reliable restoration benefits from recurrent non-local operations, correlation propagation, and computing correlations within confined neighborhoods.
Takeaways & Limitations
Conventional non-local methods use fixed or suboptimally obtained parameters because their optimization does not directly target image reconstruction error.
Abstract
from arXiv · showhide
Many classic methods have shown non-local self-similarity in natural images to be an effective prior for image restoration. However, it remains unclear and challenging to make use of this intrinsic property via deep networks. In this paper, we propose a non-local recurrent network (NLRN) as the first attempt to incorporate non-local operations into a recurrent neural network (RNN) for image restoration. The main contributions of this work are: (1) Unlike existing methods that measure self-similarity in an isolated manner, the proposed non-local module can be flexibly integrated into existing deep networks for end-to-end training to capture deep feature correlation between each location and its neighborhood. (2) We fully employ the RNN structure for its parameter efficiency and allow deep feature correlation to be propagated along adjacent recurrent states. This new design boosts robustness against inaccurate correlation estimation due to severely degraded images. (3) We show that it is essential to maintain a confined neighborhood for computing deep feature correlation given degraded images. This is in contrast to existing practice that deploys the whole image. Extensive experiments on both image denoising and super-resolution tasks are conducted. Thanks to the recurrent non-local operations and correlation propagation, the proposed NLRN achieves superior results to state-of-the-art methods with much fewer parameters.
1 Introduction
Non-local self-similarity is an effective image-restoration prior, but deep networks have rarely incorporated it through jointly trained operations. NLRN addresses this gap with recurrent, neighborhood-confined non-local processing and achieves strong denoising and super-resolution performance with few parameters.
- Motivation: Non-local self-similarity captures correlations among image patches and has improved restoration quality in traditional methods.Examples include non-local means, collaborative filtering, joint sparsity, and low-rank modeling.
- Motivation: Deep image-restoration networks perform well but generally do not explicitly exploit image self-similarity.Prior CNN approaches used block matching before CNN processing, but that step was isolated from joint network training.
- Proposed approach: NLRN incorporates a flexible non-local module into deep networks for end-to-end learning of feature correlations between each location and its neighborhood.The method is presented as the first attempt to incorporate non-local operations in CNNs for image restoration.
- Proposed approach: NLRN uses recursively shared operations to improve parameter efficiency while propagating correlation information across adjacent recurrent states.This propagation is designed to improve robustness when degraded images make correlation estimation inaccurate.
- Proposed approach: A confined neighborhood is used instead of the whole image because larger regions can produce inaccurate correlations from degraded measurements.The method reports that useful deep features for non-local priors are more likely to reside in neighboring regions.
- Evaluation: Extensive comparisons report state-of-the-art performance for image denoising and super-resolution across benchmark datasets.The paper attributes the results to non-local operations combined with recurrent architecture and correlation propagation.
2 Related Work
Related work spans traditional restoration methods built on image self-similarity, increasingly deep CNN architectures, and feature-correlation techniques developed in other neural-network applications.
- Non-local restoration: Traditional restoration methods exploit self-similarity through filtering, sparsity, low-rankness, collaborative Wiener filtering, and neighborhood embedding.These approaches cover denoising, super-resolution, and deblurring.
- Deep restoration networks: Deep restoration research includes multilayer perceptrons, CNNs, deconvolution, very deep networks, dense connections, and CNN-based denoising models.The cited work targets both computational efficiency and restoration accuracy.
- Related neural methods: Feature correlations are also used with neural networks for graphical models, relational reasoning, and machine translation.These applications provide broader context for correlation-based neural architectures.
3 Non-Local Operations for Image Restoration
The paper presents a unified framework for non-local image-restoration operations, covering classic matching, correlation, embedding, and normalization choices. It then introduces a learnable neighborhood-based module integrated with neural networks, emphasizing end-to-end training, limited parameters, and proper neighborhood size for degraded images.
- 3.1 A General Framework: The framework represents restoration features with a non-local correlation matrix and a multi-channel transform that produce output features for each image location.Each row of the correlation matrix relates a location to another feature location, while the transform embeds the corresponding features.
- 3.2 Classic Methods: Classic non-local restoration methods fit the framework through different embeddings and correlation structures, including low-rankness, collaborative filtering, joint sparsity, and non-local means.Hard block matching selects a small set of similar patches, whereas soft matching evaluates correlations across a neighborhood.
- 3.2 Classic Methods: Conventional non-local methods use fixed or suboptimally obtained parameters, whereas the proposed neural module supports learnable end-to-end optimization.The paper positions this design as a way to address parameter-selection limitations in conventional restoration methods.
- 3.3 The Proposed Non-Local Module: The proposed module uses learnable linear embeddings and differentiable operations, allowing non-local processing to be jointly trained within a neural network.A skip connection allows insertion into pre-trained models while zero-initializing the transform preserves initial behavior; 1 × 1 convolutions keep parameter growth limited.
- 3.3 The Proposed Non-Local Module: NLRN measures feature correlations within a neighborhood rather than across the whole image because degraded inputs can make large-neighborhood correlations inaccurate.The module is flexible to varying image sizes, and recurrent integration propagates correlation information between adjacent states.
4 Non-Local Recurrent Network
NLRN integrates a non-local module into an RNN so feature correlations can propagate across recurrent states while shared weights preserve parameter efficiency. Its recurrent transition combines non-local processing, convolutional layers, identity connections, and residual correlation paths.
- Recurrent states: The recurrent state contains both feature maps and deep feature correlations, with recurrent transitions driven by the previous state and the initial state.The recurrent input xt is zero for all later time steps, while the output state is calculated only at the final step.
- Architecture: At each recurrent step, the non-local module is followed by two convolutional layers before the initial feature is added through an identity path.The initial state is computed from the input image, and the final output is produced only at time T.
- Parameter efficiency: Non-local-module weights are shared across recurrent states, maintaining parameter efficiency alongside the recurrent computation.The architecture uses shared weights for recursive operations rather than separate parameters at every time step.
- Correlation propagation: Feature correlation from the previous state is added to the current estimate before softmax normalization, making correlation information propagate between adjacent states.The paper presents this design as supporting more robust correlation estimation.
- Architecture: NLRN incorporates non-local operations into an RNN with correlation propagation across adjacent recurrent states.This distinguishes it from earlier restoration RNNs that lack non-local operations or correlation information flow between states.
5 Experiments
Experiments evaluate NLRN through training-protocol comparisons, ablations, model-complexity analyses, and benchmarks for denoising and super-resolution. Results show that non-local recurrent design choices improve restoration while using substantially fewer parameters.
- Experimental setup: NLRN experiments use two BSD-based denoising training settings and evaluate Set12, Set68, Set14, BSD test images, and Urban100 across multiple noise levels.The study also evaluates super-resolution benchmarks with scale factors ×2, ×3, and ×4.
- Model analysis: Embedded Gaussian achieves the best performance among the tested distance metrics and is used in subsequent experiments.The comparison is performed on Set12 at noise level 25 using NLRN with 12 unfolded steps.
- Model analysis: As recurrent states increase, locations with similar features show progressively higher correlations, supporting correlation propagation for exploiting image self-similarity.The correlation maps visualize recurrent states in NLRN and demonstrate increasingly focused feature relationships.
- Model analysis: Denoising performance peaks at q = 45, indicating that a confined neighborhood improves correlation estimation over using the whole image.The authors use q = 45 for the remaining experiments unless stated otherwise.
- Model analysis: Denoising performance rises with unrolling length but saturates after T = 12, leading the authors to adopt T = 12 as an accuracy–inference-time tradeoff.The unrolling length determines NLRN’s maximum effective depth.
- Comparisons with state-of-the-art methods: NLRN achieves state-of-the-art denoising and super-resolution performance across benchmark datasets, with the best results across nearly all reported denoising settings and all listed upscaling factors and datasets.The denoising comparison includes NLRN-MV, which applies multi-view testing, while the super-resolution comparison excludes substantially larger models and datasets.
- Comparisons with state-of-the-art methods: NLRN uses the fewest parameters while outperforming DnCNN, RED, and MemNet, including less than 1/10 of RED’s parameters and half of MemNet’s.These comparisons attribute the efficiency to recurrent weight sharing and the non-local recurrent architecture.
6 Conclusion
NLRN integrates non-local operations into a recurrent network for image restoration, with end-to-end training and correlation propagation. Benchmark evaluations show superiority over existing methods for denoising and super-resolution.
- NLRN incorporates a trainable non-local module into a recurrent network for image restoration.The module is trained end-to-end with the recurrent network.
- Reliable feature correlations are computed within a confined neighborhood rather than across the whole image.The conclusion contrasts confined-neighborhood correlation computation with whole-image computation.
- Feature-correlation messages are passed between adjacent recurrent stages to support restoration.
- Comprehensive benchmark evaluations demonstrate NLRN’s superiority over existing methods for image denoising and super-resolution.
7 Appendix
The appendix extends the paper’s non-local framework to classic patch-based restoration methods and presents qualitative comparisons for denoising and super-resolution. These extensions explain how Wiener filtering and joint sparse coding fit the framework, while NLRN produces sharper visual reconstructions with fewer artifacts.
- Extension to Other Classic Non-Local Methods: The proposed framework extends beyond WNNM and non-local means to BM3D and LSSC.BM3D is collaborative filtering, whereas LSSC is joint-sparsity based.
- Extension to Other Classic Non-Local Methods: BM3D groups similar patches by Euclidean distance before applying Wiener filtering to each matched-patch group.
- Extension to Other Classic Non-Local Methods: The framework represents BM3D’s groupwise Wiener-filtering operation using forward and backward filtering with empirical Wiener coefficients.The appendix identifies τ and τ^-1 as the forward and backward Wiener filters and diag(ω) as the coefficient matrix.
- Extension to Other Classic Non-Local Methods: LSSC exploits matched-patch correlations by enforcing a shared support pattern across sparse-code matrices.
- Extension to Other Classic Non-Local Methods: The LSSC formulation uses a column-counting (0,∞) norm and bounds sparse-modeling error by ϵ; its general solution is NP-hard.
- Extension to Other Classic Non-Local Methods: Under a unitary-dictionary assumption, joint sparse coding becomes a shrinkage-based transform-model operation with a locally linear subspace projection.
- Visual Results: Qualitative comparisons report that NLRN recovers more denoising detail and reconstructs sharper super-resolution edges with fewer repetitive-pattern artifacts.The comparisons include BM3D, WNNM, MemNet, DRCN, LapSRN, and DRRN.
- Visual Results: Figure 7 compares denoising results at noise level 30, while Figure 8 compares super-resolution results with ×4 upscaling.