Source-linked AI summary
Spatially-Adaptive Feature Modulation for Efficient Image Super-Resolution
Long Sun, Jiangxin Dong, Jinhui Tang, Jinshan Pan
TL;DR
Efficient image super-resolution remains challenging because deployment scenarios constrain computation and memory. SAFMN addresses this with spatially-adaptive feature modulation for long-range feature selection and convolutional channel mixing for local context and channel mixing. The method reports favorable efficiency–performance trade-offs, while Urban100 results show weaker PSNR than some methods despite comparable NIQE and LPIPS.
Problem
Resource-constrained deployment motivates image super-resolution methods that recover high-resolution details without the computational burden of large models or expensive self-attention.
Method
SAFMN combines a ViT-like spatially-adaptive feature modulation mechanism for multi-scale long-range feature modeling with a convolutional channel mixer for local contextual information and channel mixing.
Results
The proposed SAFMN achieves a favorable trade-off between reconstruction performance and model complexity, with ablations showing multi-scale features and spatially-adaptive modulation improve PSNR.
Takeaways & Limitations
The paper supports using lightweight feature modulation together with convolutional channel mixing as an efficient alternative for image super-resolution.
Takeaways & Limitations
On Urban100, SAFMN has weaker PSNR than IMDN and LAPAR-A, although it achieves comparable NIQE and LPIPS performance.
Abstract
from arXiv · showhide
Although numerous solutions have been proposed for image super-resolution, they are usually incompatible with low-power devices with many computational and memory constraints. In this paper, we address this problem by proposing a simple yet effective deep network to solve image super-resolution efficiently. In detail, we develop a spatially-adaptive feature modulation (SAFM) mechanism upon a vision transformer (ViT)-like block. Within it, we first apply the SAFM block over input features to dynamically select representative feature representations. As the SAFM block processes the input features from a long-range perspective, we further introduce a convolutional channel mixer (CCM) to simultaneously extract local contextual information and perform channel mixing. Extensive experimental results show that the proposed method is $3\times$ smaller than state-of-the-art efficient SR methods, e.g., IMDN, in terms of the network parameters and requires less computational cost while achieving comparable performance. The code is available at https://github.com/sunny2109/SAFMN.
1. Introduction
Image super-resolution must recover lost details while meeting the computational and memory constraints of deployment platforms. SAFMN combines long-range feature modeling with local feature processing to improve the efficiency–reconstruction trade-off.
- Resource-constrained streaming and high-definition platforms motivate efficient methods for restoring high-resolution images from low-resolution inputs.
- Large deep SR models improve performance but require high computational costs that make deployment challenging.RCAN is cited as an example with 15.59M parameters and more than 400 layers.
- Existing efficient methods reduce complexity or accelerate inference, but faster approaches can have poor reconstruction performance, leaving room for a better trade-off.
- SAFMN uses a ViT-like architecture with multi-scale spatially-adaptive feature modulation for long-range relations and a convolutional channel mixer for local context and channel mixing.
- Quantitative and qualitative evaluations report a favorable trade-off between reconstruction accuracy and model complexity.
2. Related Work
Deep learning replaced interpolation-based upsampling with nonlinear end-to-end mappings, while later SR research pursued improved reconstruction and efficiency. SAFMN is presented against this background as an efficient model using feature mixing and upsampling.
- Interpolation methods construct high-resolution images through zero insertion and low-pass filtering, whereas deep learning methods learn nonlinear input-to-output mappings end to end.
- CNN-based SR progressed from SRCNN through residual-learning and larger-footprint models that improved reconstruction performance.
- SAFMN uses convolutional feature transformation, stacked feature mixing modules, and an upsampler to generate reconstructed high-resolution images.
- SR efficiency can be measured through parameters, FLOPs, activations, memory consumption, and inference time, with trade-offs still remaining.
3. Proposed Method
The proposed network combines spatially-adaptive feature modulation for long-range feature selection with convolutional channel mixing for local context and efficient reconstruction.
- Network Architecture: The network stacks feature mixing modules and an upsampler, transforming the low-resolution image into shallow features before reconstructing the high-resolution output.A 3 × 3 convolution generates F0, while stacked FMMs produce deeper features for reconstruction.
- Spatially-Adaptive Feature Modulation: SAFM uses multi-scale feature representations to dynamically select representative features and model long-range dependencies with reduced memory and computational costs.The method applies channel splitting, multi-scale processing, concatenation, and attention-based element-wise modulation.
- Spatially-Adaptive Feature Modulation: Adaptive max pooling generates multi-scale features, and experiments show that it improves reconstruction performance.The resulting features aggregate local and global relations through concatenation followed by a 1 × 1 convolution.
- Spatially-Adaptive Feature Modulation: The refined representation is passed through GELU to estimate an attention map that modulates the input features through an element-wise product.This modulation uses the multi-scale representation to gather long-range features with small memory and computational costs.
- Convolutional Channel Mixer: CCM complements SAFM by extracting local spatial context and mixing channels using sequential 3 × 3 and 1 × 1 convolutions.The 3 × 3 convolution doubles the channels, GELU provides nonlinear mapping, and the 1 × 1 convolution restores the original channel dimension.
- Feature Mixing Module: The unified feature mixing module formulates SAFM and CCM together to select representative features within a ViT-inspired design.The intermediate features are represented as X, Y, and Z, with LayerNorm included in the formulation.
4. Experimental Results
SAFMN is evaluated on standard super-resolution benchmarks using reconstruction quality, parameter, FLOP, activation, memory, and runtime measures. It achieves comparable reconstruction performance with substantially lower resource use and clearer visual structures.
- Evaluation setup: SAFMN is evaluated quantitatively and qualitatively on five benchmark datasets using PSNR and SSIM for reconstruction quality.The test datasets are Set5, Set14, B100, Urban100, and Manga109.
- Quantitative comparisons: Table 1 compares ×2, ×3, and ×4 SR using PSNR/SSIM, #Params, #FLOPs, and #Acts, with FLOPs and activations measured for 1280 × 720 HR output.#Params and #Acts relate to memory consumption, while #FLOPs relates to energy usage.
- Quantitative comparisons: 85% fewer parameters than CARN, 66% fewer than IMDN, and 42% fewer than ShuffleMixer are reported for ×4 SR on B100.SAFMN also uses 60%, 29%, and 71% fewer activations than these methods, respectively.
- Qualitative comparisons: On ×3 Urban100 SR, SAFMN generates clearer structures and more accurate parallel straight lines and grid patterns than listed methods.The visual comparison is reported for the Urban100 dataset.
- Memory and runtime: 10% of CARN-series GPU consumption and 4% of LAPAR-A GPU consumption are reported, while runtime is nearly twice as fast as most evaluated methods except IMDN.Compared with IMDN, SAFMN has similar running speed while significantly reducing memory usage.
5. Analysis and Discussion
Ablation studies examine feature modulation, multi-scale representation, feature aggregation, CCM, GELU, and normalization. The results support the contributions of these components and identify LayerNorm as important for stable training and stronger performance.
- Feature modulation: Removing feature modulation lowers Manga109 performance by 0.11dB.The ablation treats feature modulation as an adaptive operation in the network.
- Multi-scale representation: Removing multi-scale features causes a 0.13dB performance drop on Manga109.Adaptive max pooling performs better than adaptive average pooling or nearest interpolation for detecting discriminative features.
- Feature aggregation: Combining feature aggregation with modulation improves Manga109 PSNR by 0.14dB.A 1 × 1 convolution aggregates multi-scale features along the channel dimension.
- Combined components: Removing feature modulation, multi-scale representation, and feature aggregation reduces PSNR by 0.12dB on DIV2K-val and 0.2dB on Manga109.The resulting model uses only a 3 × 3 depth-wise convolution for spatial encoding.
- Convolutional channel mixer: Without CCM, accuracy reaches 29.69dB on DIV2K-val and 28.49dB on Manga109, while replacing it with channel MLP causes a 0.63dB Manga109 drop.CCM is designed to encode local contextual information and perform channel mixing simultaneously.
- Normalization: Without LayerNorm, training crashes at a learning rate of 1 × 10^-3; LayerNorm outperforms BatchNorm, Frozen BatchNorm, and L2 normalization.The element-wise product in SAFM can produce abnormal gradients and unstable training, motivating input normalization.
6. Conclution
The paper presents SAFMN as an efficient super-resolution model combining multi-scale spatially-adaptive modulation with a compact convolutional channel mixer. Experiments report competitive performance with improved efficiency.
- Conclusion: SAFMN models long-range adaptability through multi-scale feature representation-based modulation and complements it with local contextual encoding and channel mixing.The proposed CCM is compact and convolutional.
- Conclusion: Experimental evaluations on commonly used benchmarks report that SAFMN is more efficient than state-of-the-art methods while achieving competitive performance.The evaluation includes both qualitative and quantitative comparisons.
- Supplemental Material -
The supplemental material lists the paper’s authors, institutional affiliation, and arXiv version information.
- Bibliographic information: The paper is authored by Long Sun, Jiangxin Dong, Jinhui Tang, and Jinshan Pan.
- Bibliographic information: The authors are affiliated with Nanjing University of Science and Technology.
- Bibliographic information: The paper is identified as arXiv:2302.13800v1, dated 27 Feb 2023.
Overview
The document organizes further analysis around SAFM and LayerNorm effectiveness, challenge-winner comparisons, classical-model comparisons, Urban100 notes, and additional visual comparisons.
- Section 1 examines the effectiveness of spatially-adaptive feature modulation and LayerNorm.
- Section 2 compares the method with challenge winners.
- Sections 3–5 cover classical SR models, Urban100 observations, and visual comparisons.
1. Ablations of the spatially-adaptive feature modulation and the LayerNorm
The ablations examine SAFM scale selection, feature components, and LayerNorm-related efficiency. Removing scale information or SAFM components affects feature representation or reconstruction behavior.
- Effect of scales: Removing any feature scale affects reconstruction performance on the ×4 DIV2K validation set.Learned features at different scales are independent and complementary.
- SAFM components: The SAFM layer combines feature modulation, multi-scale representation, and feature aggregation.
- SAFM ablation: Without feature modulation, the model pays less attention to high-frequency regions, while without the relevant mechanism it fails to capture long-range information.
- Efficiency evaluation: Table 2 evaluates reconstruction quality, GPU memory, FLOPs, activations, and inference time for efficient SR models.
2. Comparison with the challenge winners
Against RFDN and RLFN, SAFMN improves most reported measures but runs more slowly, primarily because LayerNorm and several element-wise operations add inference overhead.
- Comparison with challenge winners: SAFMN obtains a noticeable improvement in all measures except running time compared with challenge winners RFDN and RLFN.
- LayerNorm efficiency: Removing LayerNorm reduces runtime to 8.35ms, close to RLFN’s speed.The authors retain LayerNorm because their ablations indicate its importance.
- Runtime analysis: Element-wise addition, element-wise product, and channel splitting are identified as time-consuming tensor operations.The runtime analysis uses torch.profiler on ×4 SR.
3. Comparison with classic SR models
The large SAFMN is substantially more efficient than evaluated classical SR models while maintaining competitive reconstruction performance, though Urban100 exposes a PSNR weakness.
- Comparison with classic SR models: SAFMN achieves competitive reconstruction performance with significant model-efficiency advantages over EDSR, RCAN, SAN, HAN, and SwinIR.
- Urban100 observations: SAFMN has weak PSNR performance on Urban100 compared with IMDN and LAPAR-A.The authors attribute this to slight local luminance differences affecting pixel-level PSNR.
- Visual results: On ×4 Urban100 visual comparisons, SAFMN produces clearer images with finer detailed structures than the compared state-of-the-art methods.