Source-linked AI summary
Lightweight Bimodal Network for Single-Image Super-Resolution via Symmetric CNN and Recursive Transformer
Guangwei Gao, Zhengxue Wang, Juncheng Li, Wenjie Li, Yi Yu, Tieyong Zeng
TL;DR
SISR methods face computational and memory costs from complex operations, while lightweight models can lack sufficient global information. LBNet combines a Symmetric CNN for local extraction and coarse reconstruction with a Recursive Transformer for long-term dependence and texture refinement, achieving a better balance of performance and efficiency.
Problem
SISR methods are difficult to apply in real-world scenarios because complex operations incur computational and memory costs, while lightweight models may lack large receptive fields and global information.
Method
LBNet combines a Symmetric CNN for local feature extraction and coarse reconstruction with a Recursive Transformer that learns long-term dependence and refines texture details using global information.
Results
LBNet achieves a better balance between performance, model size, execution time, and GPU memory consumption.
Takeaways & Limitations
The hybrid CNN–Transformer design provides an efficient SISR model that uses local and global image information for reconstruction.
Takeaways & Limitations
Lightweight CNN-based models may have insufficient receptive fields and global information, limiting their performance.
Abstract
from arXiv · showhide
Single-image super-resolution (SISR) has achieved significant breakthroughs with the development of deep learning. However, these methods are difficult to be applied in real-world scenarios since they are inevitably accompanied by the problems of computational and memory costs caused by the complex operations. To solve this issue, we propose a Lightweight Bimodal Network (LBNet) for SISR. Specifically, an effective Symmetric CNN is designed for local feature extraction and coarse image reconstruction. Meanwhile, we propose a Recursive Transformer to fully learn the long-term dependence of images thus the global information can be fully used to further refine texture details. Studies show that the hybrid of CNN and Transformer can build a more efficient model. Extensive experiments have proved that our LBNet achieves more prominent performance than other state-of-the-art methods with a relatively low computational cost and memory consumption. The code is available at https://github.com/IVIPLab/LBNet.
1 Introduction
CNN-based SISR provides strong local feature extraction, while Transformer modeling supplies long-term image dependence; LBNet combines both in a lightweight architecture. Its Symmetric CNN extracts local features and performs coarse reconstruction, while the Recursive Transformer refines texture details using global information.
- Lightweight strategies reduce parameters but can degrade performance, making high-quality reconstruction with rich details difficult.
- CNN features provide local invariance, whereas Transformer representations model long-term dependence, motivating their fusion for efficient super-resolution reconstruction.
- Symmetric CNN: The Symmetric CNN extracts local features and performs coarse image reconstruction through the LFFM and FRDAB modules.LFFM and FRDAB are designed for feature extraction and feature utilization.
- Recursive Transformer: The Recursive Transformer learns long-term image dependence and uses global information to refine texture details with few parameters and low GPU memory consumption.
- LBNet integrates CNN and Transformer components to balance reconstruction performance, model size, execution time, and GPU memory consumption.
2 Related Works
Related SISR research includes both deep CNN models and lightweight architectures, alongside Transformer-based approaches for modeling long-term image dependence. Lightweight CNN models address efficiency but may lose receptive-field and global-information capacity.
- CNN-based SISR: CNN-based SISR methods achieved strong performance through increasingly deep architectures such as SRCNN, EDSR, and RCAN.
- CNN-based SISR: Lightweight SISR models reduce complexity but may have limited performance because they cannot obtain sufficiently large receptive fields and global information.
- Transformer-based SISR: Transformer-based SISR methods were introduced to model long-term image dependence and advance image restoration.
3 Lightweight Bimodal Network (LBNet)
LBNet combines a Symmetric CNN for local feature extraction with a Recursive Transformer for long-term dependence learning and global texture refinement. Its lightweight design uses parameter sharing, attention, feature reduction, and recursive computation to limit model and memory costs.
- Network Architecture: LBNet comprises a Symmetric CNN, Recursive Transformer, and reconstruction module for extracting local features, learning global dependence, and reconstructing SR images.The shallow features are processed by the Symmetric CNN, enhanced by the Recursive Transformer, then combined with shallow features for reconstruction.
- Network Architecture: The reconstruction module combines refined and shallow features before applying a 3 × 3 convolution and pixel-shuffle layer to produce the SR image.Training optimizes LBNet with an L1 loss function.
- Symmetric CNN: Symmetric CNN uses paired parameter-sharing LFFMs and channel-attention modules to balance feature-extraction performance with parameter efficiency.Outputs from the top branch feed corresponding modules in the down branch, and paired features are fused through channel attention.
- Symmetric CNN: LFFM improves DenseBlock-style extraction by replacing convolutions with FRDABs, adding 1 × 1 group-convolution reduction, and introducing local residual learning.These changes target stronger feature extraction, dimensionality reduction, and improved information transmission.
- Symmetric CNN: FRDAB extracts multi-scale features through branches with different receptive fields, then applies channel and spatial attention before fusing the results.The attention operations strengthen suppression of smooth input regions.
- Recursive Transformer: Recursive Transformer reuses the Transformer module output as its next input, enabling repeated long-term-dependence learning without increasing parameters or GPU memory consumption.The module uses the encoding portion of a Transformer, with layer normalization, multi-head attention, and an MLP; feature reduction further limits memory use.
4 Experiments
LBNet is evaluated against lightweight SISR models and through component ablations, showing strong reconstruction quality alongside low complexity. The experiments also examine the contributions of Symmetric CNN, FRDAB, Recursive Transformer, and different recursion depths.
- 4.3 Comparison with Lightweight SISR Models: LBNet achieves the best reported results among 11 advanced lightweight SISR models, while LBNet-T is also best at the same parameter level.The comparison reports low parameter counts and Mult-Adds for both versions.
- 4.4 Model Complexity Studies: LBNet achieves the best PSNR with comparable execution time and parameter counts, demonstrating an effective performance–complexity trade-off.The paper uses Figure 4 to compare performance, parameter quantity, and execution time.
- 4.5 Ablation Study: FRDAB significantly improves performance over alternative lightweight feature-extraction units despite a small increase in parameters and Mult-Adds.The ablation compares FRDAB with IMDB and RCAB on Manga109 for ×4 super-resolution.
- 4.5 Ablation Study: Recursive Transformer improves model performance with insignificant increases in Mult-Adds and execution time, though it increases the parameter count.The study removes RT to assess its contribution.
- 4.5 Ablation Study: Performance improves as Transformer recursion increases, but the gain at three recursions is not obvious, so the final model uses S = 2.The choice balances model performance, Mult-Adds, and execution time.
- 4.3 Comparison with Lightweight SISR Models: LBNet attains competitive results against SwinIR and ESRT with fewer parameters and Mult-Adds, although its PSNR is 0.14dB below SwinIR.SwinIR additionally uses Flickr2K for training, which the authors identify as a factor improving its performance.
5 Conclusions
LBNet integrates a Symmetric CNN and Recursive Transformer to combine local feature extraction with global information refinement while balancing performance, model size, execution time, and GPU memory consumption.
- 5 Conclusions: LBNet combines a Symmetric CNN for local feature extraction with a Recursive Transformer that learns long-term image dependence for global feature refinement.The CNN uses LFFM and FRDAB for feature extraction and utilization, while recursive Transformer training refines features using global information.
- 5 Conclusions: The integrated design achieves a better balance among performance, model size, execution time, and GPU memory consumption.
- 5 Conclusions: LBNet is proposed as a lightweight SISR network based on Symmetric CNN and Recursive Transformer.