PaperlayerView exact revision on arXiv

Source-linked AI summary

Residual Non-local Attention Networks for Image Restoration

Yulun Zhang, Kunpeng Li, Kai Li, Bineng Zhong, Yun Fu

arXiv:1903.10082v1cs.CV

TL;DR

Existing restoration networks are limited by local operations and equal treatment of spatial and channel features, especially when corrupted images contain uneven information. RNAN addresses this with local and non-local attention blocks, trunk and mask branches, and residual attention learning; it achieves state-of-the-art restoration results with moderate model size and running time.

  • Problem

    Existing CNN restoration methods have limited receptive fields, miss long-range pixel dependencies, and do not adequately account for uneven information in corrupted images.

  • Method

    RNAN stacks local and non-local attention blocks whose trunk branches extract hierarchical features and whose mask branches adaptively rescale them using mixed attention.

  • Results

    RNAN achieves state-of-the-art results across image denoising, demosaicing, compression artifacts reduction, and super-resolution, with moderate model size and running time.

  • Takeaways & Limitations

    Residual local and non-local attention learning preserves low-level features and supports very deep networks for high-quality image restoration.

Abstract

from arXiv · show

In this paper, we propose a residual non-local attention network for high-quality image restoration. Without considering the uneven distribution of information in the corrupted images, previous methods are restricted by local convolutional operation and equal treatment of spatial- and channel-wise features. To address this issue, we design local and non-local attention blocks to extract features that capture the long-range dependencies between pixels and pay more attention to the challenging parts. Specifically, we design trunk branch and (non-)local mask branch in each (non-)local attention block. The trunk branch is used to extract hierarchical features. Local and non-local mask branches aim to adaptively rescale these hierarchical features with mixed attentions. The local mask branch concentrates on more local structures with convolutional operations, while non-local attention considers more about long-range dependencies in the whole feature map. Furthermore, we propose residual local and non-local attention learning to train the very deep network, which further enhance the representation ability of the network. Our proposed method can be generalized for various image restoration applications, such as image denoising, demosaicing, compression artifacts reduction, and super-resolution. Experiments demonstrate that our method obtains comparable or better results compared with recently leading methods quantitatively and visually.

1 INTRODUCTION

Image restoration methods based on local CNN operations face limited receptive fields and feature discrimination, motivating RNAN’s local/non-local attention and residual learning for diverse restoration tasks.

  • 1 INTRODUCTION: Image restoration recovers high-quality images from corrupted low-quality observations and supports various high-level vision tasks.The paper identifies denoising, demosaicing, and compression artifacts reduction as widely studied restoration tasks.
  • 1 INTRODUCTION: Existing CNN methods mainly use local convolutions, limiting receptive fields and their ability to capture long-range pixel dependencies.Larger receptive fields provide more context for modeling heavily corrupted images.
  • 1 INTRODUCTION: RNAN uses residual local and non-local attention blocks with trunk and mask branches to extract hierarchical features and adaptively rescale them.Non-local mask processing enlarges the receptive field, while the block design combines local and non-local information.
  • 1 INTRODUCTION: Residual non-local attention learning preserves more low-level features while training very deep restoration networks.The method uses non-local low-level and high-level attention to strengthen network representation ability.
  • 1 INTRODUCTION: RNAN is evaluated for image denoising, demosaicing, compression artifacts reduction, and super-resolution, achieving superior results over leading methods.The authors also report moderate model size and fast operation.

2 RELATED WORK

Related work establishes non-local filtering, attention mechanisms, and CNN restoration architectures as foundations for RNAN’s design.

  • 2 RELATED WORK: Non-local means averages all image pixels with weights, allowing distant pixels to contribute to a position’s response.The non-local prior was also used in BM3D denoising and later incorporated into neural networks.
  • 2 RELATED WORK: Attention mechanisms guide processing resources toward informative input components and commonly rescale feature maps with gating functions.Prior work includes trunk-and-mask attention and channel-wise relationship modeling with squeeze-and-excitation blocks.
  • 2 RELATED WORK: CNN restoration architectures progressed from stacked denoising auto-encoders and convolutional models to residual-learning and denoiser-prior methods.Representative systems include SRCNN, ARCNN, DnCNN, and IRCNN.
  • 2 RELATED WORK: RNAN’s framework combines convolutional layers with residual non-local and residual local attention blocks for image denoising.The figure uses Conv, RNAB, and RAB as abbreviations for these components.

3 RESIDUAL NON-LOCAL ATTENTION NETWORK FOR IMAGE RESTORATION

RNAN reconstructs high-quality images with global residual learning and stacked residual local/non-local attention blocks. Its trunk branch extracts hierarchical features, while mask branches adaptively rescale them using local or whole-feature-map attention.

  • Framework: RNAN reconstructs images as IR = HRNAN(IL), using global residual learning so the network focuses on degradation components.The framework treats IL as the low-quality input and uses residual learning for noise, blur, or compression artifacts.
  • Attention blocks: Each attention block combines trunk and mask branches, with residual blocks extracting hierarchical features and masks rescaling them channel- and spatial-wise.Stacked blocks include residual blocks around two middle branches; non-local blocks are inserted in the mask branch when needed.
  • Local attention: Local mask branches enlarge receptive fields through large-stride convolution, residual blocks, deconvolution, and sigmoid normalization while retaining more detail than maxpooling.The resulting mask has a larger receptive field than the trunk branch, but does not cover the whole feature map at once.
  • Non-local attention: Non-local blocks consider all spatial positions through pairwise relationships and embedded Gaussian weighting, producing non-local mixed attention in the mask branch.The operation represents each position using information aggregated from all possible positions, then combines local and non-local attention before sigmoid mapping.
  • Residual attention learning: Residual non-local attention learning computes Htrunk(x)Hmask(x) + x, preserving low-level features and supporting very deep restoration networks.The residual connection addresses the limited trainability of simple trunk-mask multiplication and is intended to strengthen representation ability.

4 EXPERIMENTS

Experiments evaluate RNAN across restoration tasks and ablations, showing benefits from non-local mixed attention, mask branches, and deeper residual attention blocks. RNAN achieves strong denoising, demosaicing, compression-artifact reduction, and super-resolution results, while retaining moderate model size and runtime.

  • Experimental setup: RNAN is evaluated on denoising, demosaicing, compression-artifact reduction, and super-resolution using standard datasets and PSNR and/or SSIM.The main experiments train models on 800 DIV2K images; task-specific settings follow IRCNN or ARCNN protocols.
  • Ablation study: Non-local mixed attention improves performance over a comparable configuration without mask branches and non-local blocks.The ablation compares cases 1 and 4 using the same block number.
  • Ablation study: Mask branches improve performance with or without non-local blocks by providing informative attention and better representational ability.The effect is reported for cases 1 versus 2 and cases 3 versus 4.
  • Ablation study: Non-local blocks improve restoration ability with or without mask branches by supplying non-local information from low- and high-level features.The ablation reports this pattern across cases 1–4.
  • Ablation study: More non-local blocks and more residual attention blocks improve results, but non-local blocks increase computation, leading the authors to use two non-local blocks.The block-number comparison considers cases 2, 4, and 7, and fixed-RNAB comparisons in cases 5–8.
  • Denoising: At noise level σ = 70, RNAN gains 0.48, 0.30, and 1.06 dB PSNR over FFDNet on the reported denoising datasets.RNAN achieves the best results across the evaluated datasets and noise levels.
  • Visual comparisons: RNAN alleviates over-smoothing in denoising, recovers more faithful colors and fewer blocking artifacts in demosaicing, and preserves more details in compression-artifact reduction.These visual comparisons are reported for Figures 4–7, including JPEG quality q = 10.
  • Super-resolution: RNAN+ achieves second-best super-resolution performance, while RNAN achieves third-best results in most cases with 7.5 M parameters versus 43 M for EDSR and 16 M for RCAN.The comparison covers Set5, Set14, B100, Urban100, and Manga109; RNAN is about 120 convolutional layers deep.

5 CONCLUSIONS

RNAN stacks local and non-local attention blocks to extract and adaptively rescale hierarchical features using local and whole-feature-map attention. Residual local and non-local attention learning supports very deep restoration networks, which achieve state-of-the-art results with moderate model size and running time.

  • Conclusion: RNAN stacks local and non-local attention blocks containing trunk and mask branches to extract and adaptively rescale hierarchical features.The mask branches use soft weights, and non-local attention considers the whole feature map.
  • Conclusion: Residual local and non-local attention learning incorporates input features into attention computation to train very deep networks for image restoration.The paper presents this design as more suitable for image restoration.
  • Conclusion: RNAN achieves state-of-the-art image restoration results with moderate model size and running time.The conclusion summarizes the method's reported performance and efficiency.

A.1 TRAIN RNAN WITH SMALL TRAINING DATA

RNAN remains effective when retrained on smaller task-specific datasets. Using the same or smaller training sets than comparison methods, it obtains better results across five restoration tasks.

  • Training with small data: The small-data experiments retrain RNAN on five tasks using BSD400, SR291, or task-specific image collections instead of DIV2K.The main-paper results use DIV2K, while the appendix compares smaller training sets against the second-best main-paper methods.
  • Training with small data: RNAN obtains better results across five tasks with the same or smaller training sets than the comparison methods.The authors interpret these experiments as evidence of effectiveness for general image restoration tasks.

A.2 VISUAL RESULTS

RNAN produces visually sharper and more structurally consistent restorations across denoising, compression artifact reduction, and ×4 super-resolution. The comparisons emphasize preserved details and reduced over-smoothing or blurring artifacts.

  • Color and Gray Image Denoising: At noise level σ = 50, RNAN recovers sharper edges, preserves tiny lines, and alleviates over-smoothing in color and gray-scale denoising.Non-local mixed attention lets RNAN treat different image regions distinctively.
  • Image Compression Artifacts Reduction: At JPEG quality q = 10, RNAN removes blocking artifacts while retaining more details and consistent structures than competing methods.ARCNN, TNRD, and DnCNN are described as over-smoothing some structures under this condition.
  • Image Super-Resolution: With scaling factor s = 4, RNAN produces finer structures and more visually pleasing super-resolution results than the compared methods.Compared methods are described as introducing distortion, incorrect structures, or blurring around tiny lines, edges, and textures.
Loading 1903.10082v1…