Source-linked AI summary
Image Super-Resolution Using Very Deep Residual Channel Attention Networks
Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, Yun Fu
TL;DR
Deeper SR networks are difficult to train, while treating all feature channels equally limits their ability to distinguish useful information. RCAN combines residual-in-residual skip connections with channel attention, achieving better accuracy and visual results than prior methods across extensive experiments.
Problem
Deeper SR networks are difficult to improve by simply stacking residual blocks, while equal treatment of channel-wise features limits flexible representation of low- and high-frequency information.
Method
RCAN uses residual-in-residual skip connections to enable very deep trainable networks and channel attention to adaptively rescale features using inter-channel dependencies.
Results
RCAN performs best across datasets and scaling factors, with PSNR gains over EDSR of 0.49 dB on Urban100 and 0.55 dB on Manga109.
Takeaways & Limitations
Experiments with bicubic and blur-downscale degradation models demonstrate RCAN’s effectiveness for image super-resolution.
Abstract
from arXiv · showhide
Convolutional neural network (CNN) depth is of crucial importance for image super-resolution (SR). However, we observe that deeper networks for image SR are more difficult to train. The low-resolution inputs and features contain abundant low-frequency information, which is treated equally across channels, hence hindering the representational ability of CNNs. To solve these problems, we propose the very deep residual channel attention networks (RCAN). Specifically, we propose a residual in residual (RIR) structure to form very deep network, which consists of several residual groups with long skip connections. Each residual group contains some residual blocks with short skip connections. Meanwhile, RIR allows abundant low-frequency information to be bypassed through multiple skip connections, making the main network focus on learning high-frequency information. Furthermore, we propose a channel attention mechanism to adaptively rescale channel-wise features by considering interdependencies among channels. Extensive experiments show that our RCAN achieves better accuracy and visual improvements against state-of-the-art methods.
1 Introduction
The introduction frames single-image super-resolution as an ill-posed reconstruction problem where CNN depth improves performance but makes training difficult. RCAN addresses these challenges with residual-in-residual connections and channel attention to bypass low-frequency information and emphasize useful features.
- Problem Setting: Single-image super-resolution reconstructs a high-resolution image from a low-resolution counterpart, but the inverse problem admits multiple solutions.The task supports applications including security, medical imaging, and object recognition.
- Motivation: CNN-based methods substantially improve image super-resolution, while increasing depth has been central to performance gains in architectures such as VDSR, DRCN, EDSR, and MDSR.MDSR reaches about 165 layers, illustrating the field’s move toward deeper networks.
- Motivation: Existing CNN methods treat channel-wise features equally, limiting flexibility because low-resolution inputs contain abundant low-frequency information that requires little computation.The introduction argues that super-resolution should recover as much high-frequency information as possible while forwarding low-frequency content directly to the output.
- RCAN: RCAN combines very deep trainable networks with adaptive channel-wise feature learning through residual-in-residual structure and channel attention.The proposed RIR structure uses residual groups and long skip connections to ease training of networks exceeding 400 layers.
- RCAN: RIR’s long and short skip connections bypass abundant low-frequency information, allowing the main network to learn more effective information.The network’s channel attention further concentrates learning on useful channels and enhances discriminative ability.
2 Related Work
Related work spans increasingly deep CNN architectures for image super-resolution and emerging attention mechanisms, which were less explored in low-level vision. RCAN builds on this context by emphasizing informative high-frequency channel-wise features.
- Attention mechanism: Attention had been popular in high-level vision but remained seldom investigated in low-level vision, motivating its study for image super-resolution.The paper focuses its related-work discussion on CNN-based methods and attention mechanisms.
- Deep CNN for SR: CNN-based SR progressed from SRCNN to residual, recursive, memory, and deeper architectures that improved accuracy but initially interpolated low-resolution inputs.The cited methods include VDSR, DRCN, DRRN, and MemNet.
- Deep CNN for SR: Deep SR architectures later extracted features from original low-resolution inputs and performed upscaling at the network tail, while SRGAN introduced perceptual losses and GANs for photo-realistic reconstruction.FSRCNN accelerated SRCNN training and testing, and SRResNet used ResNet to construct a deeper network.
- Attention mechanism: Attention guides processing toward informative input components and commonly uses gating functions such as sigmoid to rescale feature maps.Prior applications ranged from image localization and understanding to sequence-based networks, including residual attention and squeeze-and-excitation approaches.
- Motivation: For image SR, high-frequency channel-wise features are more informative for high-resolution reconstruction, motivating RCAN’s channel-focused attention in very deep CNNs.The authors propose RCAN and defer its detailed description to the next section.
3 Residual Channel Attention Network (RCAN)
RCAN combines shallow extraction, a very deep residual-in-residual feature extractor, upscaling, and reconstruction. Its RIR structure and channel attention enable very deep, high-accuracy super-resolution by bypassing low-frequency information and rescaling channel-wise features.
- Overall architecture: RCAN comprises shallow feature extraction, RIR deep feature extraction, an upscale module, and one-convolution reconstruction.The network extracts shallow features with one convolution, upscales deep features, and reconstructs the output with one convolution.
- Residual-in-residual structure: RIR contains G residual groups with long skip connections, each containing B residual channel attention blocks with short skip connections, enabling CNNs exceeding 400 layers.The long skip connection stabilizes training and supports residual learning across groups, while each short skip connection supports residual learning within groups.
- Residual learning: Long and short skip connections bypass abundant low-frequency information, allowing the main network to focus on residual and more informative features.The long skip eases information flow across residual groups and learns residual information coarsely; the short skip further promotes residual learning within groups.
- Channel attention: Channel attention exploits interdependencies among feature channels to generate channel statistics and adaptively rescale channel-wise features.Global average pooling summarizes spatial information, followed by channel downscaling, ReLU activation, channel upscaling, and sigmoid gating.
- RCAN construction: Integrating channel attention into residual blocks yields RCABs, which together with RIR form a very deep RCAN for highly accurate image super-resolution and notable improvements over previous leading methods.The implementation uses post-upscaling, which is described as more computationally efficient and higher-performing than pre-upscaling methods.
4 Experiments
Experiments evaluate RCAN across BI and BD degradation models using standard SR benchmarks, ablations, visual comparisons, and object-recognition tests. RCAN and RCAN+ consistently outperform competing methods, while RIR and channel attention improve reconstruction quality and model efficiency.
- Object recognition: RCAN achieves the lowest top-1 and top-5 errors among the evaluated SR methods in the ResNet object-recognition test.The evaluation uses ResNet-50 and the first 1,000 ImageNet CLS-LOC validation images.
- Ablation studies: Removing long and short skip connections from over-400-layer networks produces relatively low Set5 ×2 PSNR, demonstrating the importance of the residual-in-residual structure.The ablation uses 200 residual blocks arranged into 10 residual groups.
- Ablation studies: Networks with channel attention outperform corresponding networks without it, and channel attention improves performance even in very deep models without residual-in-residual connections.The comparison is based on the first and last four columns of the ablation results.
- BI degradation results: RCAN+ performs best across all datasets and scaling factors, while RCAN without self-ensemble also outperforms the compared methods.The BI quantitative comparisons cover ×2, ×3, ×4, and ×8 SR and use PSNR/SSIM.
- BI degradation results: 0.49 dB and 0.55 dB are RCAN’s PSNR gains over EDSR on Urban100 and Manga109, respectively, despite RCAN using 16 M parameters versus EDSR’s 43 M.The gains are reported for the larger scaling-factor setting discussed in the passage.
- BD degradation results: RCAN obtains notable gains over RDN under BD degradation, and self-ensembled RCAN+ performs even better.The comparison evaluates 3× SR using PSNR/SSIM across the benchmark datasets.
5 Conclusions
RCAN is proposed for highly accurate image super-resolution through a very deep residual-in-residual architecture and channel attention. Its skip connections bypass low-frequency information so the main network can focus on high-frequency information.
- RCAN is proposed as a very deep residual channel attention network for highly accurate image super-resolution.
- Its residual-in-residual structure enables very large depth through long and short skip connections.
- Multiple skip connections bypass abundant low-frequency information, allowing the main network to focus on learning high-frequency information.
- RCAN further introduces channel attention to adaptively rescale channel-wise features and improve network ability.