Source-linked AI summary
Lightweight image super-resolution with enhanced CNN
Chunwei Tian, Ruibin Zhuge, Zhihao Wu, Yong Xu, Wangmeng Zuo, Chen Chen, Chia-Wen Lin
TL;DR
Resource-intensive CNNs limit SISR deployment on devices with constrained computation and memory. LESRCNN addresses this with three complementary blocks for feature extraction, reconstruction, and refinement, and the paper reports that it outperforms state-of-the-art SISR methods qualitatively and quantitatively.
Problem
Excessive convolutions, parameters, computation, and memory requirements limit CNN-based SISR on resource-constrained devices.
Method
LESRCNN cascades IEEB, RB, and IRB blocks to extract and enhance LR features, transform them into high-frequency features, and refine SR features.
Results
LESRCNN outperforms state-of-the-art SISR methods in qualitative and quantitative evaluations and supports different scales with a model.
Takeaways & Limitations
Its low computational cost and memory consumption make LESRCNN particularly suitable for resource-constrained mobile-device applications.
Abstract
from arXiv · showhide
Deep convolutional neural networks (CNNs) with strong expressive ability have achieved impressive performances on single image super-resolution (SISR). However, their excessive amounts of convolutions and parameters usually consume high computational cost and more memory storage for training a SR model, which limits their applications to SR with resource-constrained devices in real world. To resolve these problems, we propose a lightweight enhanced SR CNN (LESRCNN) with three successive sub-blocks, an information extraction and enhancement block (IEEB), a reconstruction block (RB) and an information refinement block (IRB). Specifically, the IEEB extracts hierarchical low-resolution (LR) features and aggregates the obtained features step-by-step to increase the memory ability of the shallow layers on deep layers for SISR. To remove redundant information obtained, a heterogeneous architecture is adopted in the IEEB. After that, the RB converts low-frequency features into high-frequency features by fusing global and local features, which is complementary with the IEEB in tackling the long-term dependency problem. Finally, the IRB uses coarse high-frequency features from the RB to learn more accurate SR features and construct a SR image. The proposed LESRCNN can obtain a high-quality image by a model for different scales. Extensive experiments demonstrate that the proposed LESRCNN outperforms state-of-the-arts on SISR in terms of qualitative and quantitative evaluation. The code of LESRCNN is accessible on https://github.com/hellloxiaotian/LESRCNN.
1. Introduction
SISR methods must recover HR images from LR observations while balancing reconstruction quality against computational and memory costs. LESRCNN addresses this tension with three specialized blocks that extract, transform, and refine features for lightweight multi-scale SISR.
- SISR recovers a high-resolution image from a low-resolution observation, making it an ill-posed problem.
- Deep SR networks improve expressive power but often use excessive parameters and memory, limiting deployment on resource-constrained mobile devices.
- LESRCNN cascades an information extraction and enhancement block, reconstruction block, and information refinement block.
- The IEEB aggregates hierarchical LR features to strengthen shallow-layer memory while its heterogeneous architecture reduces parameters, computation, and memory consumption.
- The RB fuses global and local features to convert low-frequency features into high-frequency features, complementing the IEEB in addressing long-term dependency.
- The IRB refines coarse high-frequency features into more accurate SR features, while LESRCNN supports SISR at different scales with a model.
2. Related Work
Related SISR research has explored multistage feature extraction, network compression, plug-and-play blocks, and feature fusion to balance image quality with resource use. The authors position LESRCNN as a deep-CNN block design aimed at that performance–cost tradeoff.
- Deep SISR methods use high-frequency, low-frequency, or combined features as training inputs.
- Cascading multistage networks progressively improve resolution, while coarse-to-fine and cascading dense designs extract hierarchical features before refinement.
- Network compression methods reduce parameters or training cost through cascading connections, 1 × 1 convolutions, grouped channels, distillation, and small convolutions.
- Plug-and-play architectures combine deep CNN blocks across vision tasks, including super-resolution, to facilitate useful information exchange.
- Feature-fusion methods merge feature levels or feature types to capture context and address long-term dependency in SISR.
- The authors seek a tradeoff between SISR performance and computational cost using deep-CNN-based blocks for real applications.
3. Proposed Method
LESRCNN cascades IEEB, RB, and IRB to extract and refine features for lightweight single-image super-resolution. Its heterogeneous IEEB reduces redundancy, while RB combines global and local information before IRB refinement.
- Network architecture: LESRCNN cascades an information extraction and enhancement block, reconstruction block, and information refinement block.The 23-layer architecture allocates 17 layers to IEEB, one to RB, and five to IRB.
- Information extraction and enhancement block (IEEB): IEEB extracts hierarchical low-frequency features, aggregates them through residual learning, and uses heterogeneous convolutions to reduce redundancy and resource consumption.The block uses 3 × 3 Conv+ReLU and 1 × 1 Conv+ReLU layers.
- Reconstruction block (RB): LESRCNN uses sub-pixel convolution in RB instead of bicubic-upsampled inputs, avoiding the additional computation and memory associated with that preprocessing.The sub-pixel convolution supports ×2, ×3, and ×4 scales through scale-specific shuffle components.
- Reconstruction block (RB): RB converts low-frequency features into high-frequency features by upsampling global and local features and fusing them with residual learning.The global and local features come from the outputs of the first and sixteenth IEEB layers, respectively.
- Information refinement block (IRB): IRB refines RB’s coarse high-frequency features into more accurate SR features and produces the SR image.The final convolution maps 64 input channels to 3 output channels.
4. Experiments
Experiments evaluate LESRCNN through ablations, benchmark comparisons, runtime, complexity, and visual quality. The results show improved SISR performance alongside lower computational and memory costs across multiple datasets and scale factors.
- Network design: LESRCNN uses hierarchical low-frequency features, heterogeneous convolutions, reconstruction, and information refinement to balance SISR quality with resource consumption.The IEEB enhances shallow-layer memory while distilling information; the RB converts low-frequency features into high-frequency features using sub-pixel convolution and global-local fusion.
- Network analysis: The HN is more competitive than the SN in SISR performance, running time, and training cost, motivating its use in the IEEB.SN is slightly better on Set5 ×4 PSNR and SSIM, but HN has lower running and training costs.
- Network analysis: The IEEB improves PSNR and SSIM over HN, while IEEB+RB further improves both metrics over IEEB on Set5 at ×4.These ablations support the enhancement operation and the two-step reconstruction block.
- Comparisons with state-of-the-arts: LESRCNN and LESRCNN-S achieve competitive or superior PSNR and SSIM against state-of-the-art methods across Set5 and Set14 at ×2, ×3, and ×4.On Set5 ×3, LESRCNN-S is 0.19dB higher than WaveResNet; on Set14 ×2, LESRCNN exceeds MemNet by 0.04 dB PSNR and 0.0006 SSIM.
- Comparisons with state-of-the-arts: On U100 ×2, LESRCNN gains 0.22dB PSNR and 0.0013 SSIM over CARN-M, while LESRCNN-S gains 0.15dB PSNR and 0.0045 SSIM on U100 ×4.The reported results also describe LESRCNN-S as competitive on B100 and U100.
- Efficiency and qualitative evaluation: LESRCNN runs faster than VDSR, MemNet, and CARN-M, uses fewer parameters and FLOPs than methods such as MemNet, and produces clearer magnified regions in qualitative comparisons.These findings indicate lower execution, computational, and training memory costs together with competitive visual quality.
5. Conclusion
LESRCNN combines three specialized sub-blocks to extract, reconstruct, and refine features for single-image super-resolution. The model is designed to reduce complexity while maintaining strong qualitative and quantitative performance across scales.
- LESRCNN cascades an information extraction and enhancement block, reconstruction block, and information refinement block.
- The IEEB extracts and aggregates hierarchical low-frequency features, while heterogeneous architecture reduces parameters and training complexity.
- LESRCNN is evaluated across ×2, ×3, and ×4 scales on benchmark datasets using PSNR and SSIM, with reported comparisons including B100 and U100.
- The RB fuses global and local features to convert low-frequency features into high-frequency features, complementing the IEEB.
- The IRB uses coarse high-frequency features from the RB to learn more accurate super-resolution features and construct the output image.
(d) (e) (f)
On a Set14 image at ×3 scale, LESRCNN achieves the highest reported PSNR and SSIM among the listed methods.
- LESRCNN achieves 26.82 PSNR and 0.7948 SSIM, exceeding CARN-M at 26.34 PSNR and 0.7856 SSIM.