Source-linked AI summary
Single Image Super-Resolution via a Holistic Attention Network
Ben Niu, Weilei Wen, Wenqi Ren, Xiangde Zhang, Lianping Yang, Shuzhen Wang, Kaihao Zhang, Xiaochun Cao, Haifeng Shen
TL;DR
Existing channel-attention approaches preserve informative features within layers but miss correlations across layers, contributing to difficulty preserving detailed textures. HAN combines layer attention and channel-spatial attention to model dependencies across layers, channels, and positions, and experiments show favorable performance against state-of-the-art SISR methods.
Problem
Existing channel-attention methods treat layers separately and miss interdependencies among layers, making detailed texture preservation in super-resolution difficult.
Method
HAN combines a layer attention module for hierarchical-layer correlations with a channel-spatial attention module for channel and positional interdependencies.
Results
HAN performs favorably against state-of-the-art SISR methods in accuracy and visual quality across benchmark evaluations.
Takeaways & Limitations
Modeling dependencies across depths, channels, and positions enables HAN to capture more informative multi-level features for super-resolution.
Abstract
from arXiv · showhide
Informative features play a crucial role in the single image super-resolution task. Channel attention has been demonstrated to be effective for preserving information-rich features in each layer. However, channel attention treats each convolution layer as a separate process that misses the correlation among different layers. To address this problem, we propose a new holistic attention network (HAN), which consists of a layer attention module (LAM) and a channel-spatial attention module (CSAM), to model the holistic interdependencies among layers, channels, and positions. Specifically, the proposed LAM adaptively emphasizes hierarchical features by considering correlations among layers. Meanwhile, CSAM learns the confidence at all the positions of each channel to selectively capture more informative features. Extensive experiments demonstrate that the proposed HAN performs favorably against the state-of-the-art single image super-resolution approaches.
1 Introduction
Single-image super-resolution seeks to reconstruct high-resolution images with edge structures and texture details from low-resolution inputs, but existing methods can smooth textures by neglecting intermediate-layer correlations. HAN addresses this gap by modeling dependencies across hierarchical layers, channels, and spatial positions with LAM and CSAM.
- Single-image super-resolution generates high-resolution images with necessary edge structures and texture details from low-resolution inputs.
- Deep CNN methods improved reconstruction through deep architectures and residual learning, which supports learning complicated mappings from low-resolution to high-resolution images.
- Existing CNN-based methods often smooth texture details because they neglect correlations among intermediate layers.
- Channel-attention methods preserve some detailed information but treat feature maps at different layers equally, weakening informative textures and natural details.
- HAN introduces LAM and CSAM to model correlations among hierarchical layers, channels, and positions for more powerful feature expression and correlation learning.
2 Related Work
Deep CNN-based super-resolution methods evolved from pioneering architectures toward deeper, attention-based models, while existing attention methods overlooked dependencies across multi-scale layers. HAN addresses this gap with LAM for hierarchical-layer interactions and CSAM for channel-pixel interdependencies.
- Super-resolution methods comprise traditional algorithms and deep learning models, with this discussion focusing on deep CNN approaches.
- SRCNN pioneered deep learning for single-image super-resolution, motivating progressively deeper and more efficient architectures.
- Perceptual constraints and adversarial training were introduced to improve visual quality beyond MSE-minimizing methods.
- Attention mechanism: Attention-based methods focus on channel and spatial correlations but overlook interdependencies among multi-scale layers.
- Attention mechanism: HAN uses LAM to learn correlations among residual-group outputs and CSAM to model interdependencies between channels and pixels.
3 Holistic Attention Network (HAN) for SR
HAN combines layer attention with channel-spatial attention to model dependencies across hierarchical layers, channels, and spatial positions in single-image super-resolution. It integrates these weighted features with shallow features for final reconstruction.
- Network architecture: HAN extracts shallow and hierarchical features with an RCAN backbone before applying holistic feature weighting.The intermediate features come from residual groups, while the final backbone feature is used by CSAM.
- Layer attention module: LAM learns correlations among residual-group outputs to enhance informative layers and suppress redundant ones.It reshapes feature groups, computes a correlation matrix, and uses the resulting weights to combine hierarchical features.
- Channel-spatial attention module: CSAM is applied only to the final feature layer as a trade-off between reconstruction accuracy and processing speed.The method could modulate all intermediate features, but restricts CSAM to FN.
- Image reconstruction: The LAM and CSAM outputs are summed with the shallow feature and reconstructed using sub-pixel convolution.The long skip connection takes F0 + FL + FCS as input, while the loss uses only L1 distance for comparison with RCAN.
- Channel-spatial attention module: CSAM models channel-wise and spatial interdependencies to adaptively filter the last-layer feature map.A 3D convolution generates joint channel-spatial attention maps, which are multiplied with the input feature and added back residually.
4 Experiments
Experiments evaluate HAN on standard datasets under bicubic and blur-downscale degradation, including ablations of its attention modules. HAN and HAN+ achieve strong quantitative and visual reconstruction results against existing methods.
- Experimental settings: Experiments use DIV2K for training and five standard datasets for testing under bilinear-interpolation and blur-downscale degradation.Evaluation uses luminance-channel PSNR and SSIM after converting reconstructed RGB results to YCbCr.
- Ablation studies: On Manga109 at 4×, CSAM improves PSNR by up to 0.06 dB, LAM by 0.16 dB, and both modules together by 0.2 dB over the baseline.The baseline is identical to HAN except that LAM and CSAM are removed.
- Ablation studies: Using three, six, or ten residual groups, LAM models still achieve higher PSNR values than RCAN despite using fewer residual groups.The comparison is reported across five standard datasets.
- Ablation studies: Increasing the number of CSAMs from one to ten increases PSNR values on the testing datasets.The ablation evaluates one, three, five, and ten CSAMs in residual groups.
- Bicubic degradation results: HAN+ performs best across 2×, 3×, 4×, and 8× bicubic-super-resolution test scales, while HAN also improves over recent methods.At 4× and 8× on additional datasets, HAN is reported as 0.06 dB and 0.35 dB higher than SAN, respectively.
- Blur-downscale degradation results: Under 3× blur-downscale degradation, HAN+ achieves the best quantitative results and HAN the second-best scores across all datasets.HAN is reported as 0.06-0.2 dB better in PSNR than RCAN and SAN, and 0.2-0.8 dB better than SRFBN.
5 Conclusions
HAN adaptively learns global dependencies across depths, channels, and positions through collaborative layer and channel-spatial attention modules. Experiments show favorable accuracy and visual quality compared with state-of-the-art SR algorithms.
- HAN adaptively learns global dependencies among different depths, channels, and positions using self-attention.
- LAM captures long-distance dependencies among hierarchical layers, while CSAM incorporates channel and contextual information within each layer.
- The two attention modules collaboratively process multi-level features to capture more informative representations for single image super-resolution.
- HAN performs favorably against state-of-the-art SR algorithms in accuracy and visual quality on benchmark datasets.