Source-linked AI summary
Learning A Single Network for Scale-Arbitrary Super-Resolution
Longguang Wang, Yingqian Wang, Zaiping Lin, Jungang Yang, Wei An, Yulan Guo
TL;DR
Existing SR networks are designed for specific integer scales and cannot adequately support non-integer or asymmetric scaling. The paper adds scale-aware feature adaptation and upsampling to existing networks, using dynamically generated filters, and reports promising arbitrary-scale results while preserving integer-scale performance with low overhead. The method is intended as a single-model extension for scale-arbitrary SR.
Problem
Existing SR networks are restricted to specific integer scales, while non-integer and asymmetric SR are needed for applications such as retargeting and image editing.
Method
A plug-in module adds scale-aware feature adaption blocks, scale-aware upsampling, and conditional convolutions that generate filters from scale information.
Results
The module produces promising non-integer and asymmetric SR results while maintaining state-of-the-art integer-scale performance with a single model and small additional computational and memory cost.
Takeaways & Limitations
Existing scale-specific SR networks can be adapted for scale-arbitrary SR without requiring separate models for each scale factor.
Takeaways & Limitations
For memory efficiency, the networks set the dynamic-filter kernel size k to 1 because k=3 would require approximately 31.6G for a 720P HR image.
Abstract
from arXiv · showhide
Recently, the performance of single image super-resolution (SR) has been significantly improved with powerful networks. However, these networks are developed for image SR with a single specific integer scale (e.g., x2;x3,x4), and cannot be used for non-integer and asymmetric SR. In this paper, we propose to learn a scale-arbitrary image SR network from scale-specific networks. Specifically, we propose a plug-in module for existing SR networks to perform scale-arbitrary SR, which consists of multiple scale-aware feature adaption blocks and a scale-aware upsampling layer. Moreover, we introduce a scale-aware knowledge transfer paradigm to transfer knowledge from scale-specific networks to the scale-arbitrary network. Our plug-in module can be easily adapted to existing networks to achieve scale-arbitrary SR. These networks plugged with our module can achieve promising results for non-integer and asymmetric SR while maintaining state-of-the-art performance for SR with integer scale factors. Besides, the additional computational and memory cost of our module is very small.
1. Introduction
Existing SR networks are restricted to specific integer scales, while real-world applications also require non-integer and asymmetric scaling. The paper introduces a scale-aware plug-in module that extends existing networks to scale-arbitrary SR with one model.
- Existing SR networks use fixed upscale filters and therefore cannot handle non-integer or asymmetric scale factors required in applications such as retargeting and image editing.
- Meta-SR predicts filters for different scales but uses scale information only during upsampling and cannot handle asymmetric SR.
- The proposed plug-in module combines scale-aware feature adaption blocks with a scale-aware upsampling layer to adapt backbone features and perform arbitrary upsampling.
- The plugged networks support non-integer and asymmetric SR while maintaining state-of-the-art integer-scale performance with small additional computational and memory cost.
- Conditional convolutions dynamically generate filters from scale information, allowing the module to adapt existing SR networks to different scale factors.
2. Related Work
Related work has advanced CNN-based SR and multi-scale or meta-learning approaches, but existing methods remain limited for asymmetric scaling. The paper addresses this gap with scale-aware processing for both non-integer and asymmetric factors.
- CNN-based SR methods use deep networks to learn mappings from low-resolution to high-resolution images and outperform traditional methods.
- MDSR integrates modules for multiple integer scales but cannot super-resolve images with non-integer scale factors.
- Meta-SR and RSAN support non-integer scales, but neither handles asymmetric SR, which remains underinvestigated despite application demand.
3. Methodology
The method learns scale-aware features and performs scale-arbitrary upsampling by extending scale-specific SR networks with a plug-in module. It adapts features according to scale similarity and uses dynamically generated filters for efficient arbitrary-scale reconstruction.
- Motivation: Scale-specific SR features vary across blocks and regions, motivating pixel-wise adaptation that preserves high-similarity features and adapts low-similarity ones.The similarity analysis compares features from pretrained ×2, ×3, and ×4 EDSR networks and visualizes their differences.
- Our Plug-in Module: The plug-in module inserts scale-aware feature adaptation after backbone blocks and follows them with a scale-aware upsampling layer.A guidance map fuses the original and adapted features, while the upsampling layer handles arbitrary output scales.
- Scale-Aware Feature Adaption: Scale-aware convolution combines expert kernels using routing weights generated from horizontal and vertical scale factors.The resulting dynamically customized filter adapts feature processing to the requested scale.
- Scale-Aware Upsampling: The scale-aware upsampling layer projects HR pixels into LR coordinates, predicts filters and offsets from relative distances and scale factors, and samples neighborhoods by bilinear interpolation.Routing weights combine experts for the bottleneck and expansion layers before the predicted filters process the sampled neighborhood.
- Scale-Aware Upsampling: Generating per-location convolution kernels creates a memory constraint, so the implementation sets k=1, reducing estimated memory from ∼31.6G to ∼3.5G for a 720P HR image.The stated estimates apply to k=3 and k=1, respectively.
4. Experiments
Experiments evaluate the proposed scale-arbitrary networks across symmetric, asymmetric, and continuously varying scales, including ablations of feature adaptation, upsampling, guidance, and expert count. The networks improve non-integer and asymmetric SR while preserving integer-scale performance and reducing efficiency costs relative to Meta-RCAN.
- 4.3. Ablation Study: Scale-aware feature adaptation improves performance over shared backbone features by adapting features according to scale information.Guidance maps place high values where feature similarities are low, supporting pixel-wise fusion of shared and adapted features.
- 4.3. Ablation Study: Increasing experts yields comparable symmetric results but better asymmetric results, especially for highly asymmetric scales; four experts provide the default performance–model-size tradeoff.The reported example improves from 34.23/34.92 to 34.46/34.98 for ×1.3 ×3.5 SR.
- 4.4. Results for SR with Symmetric Scale Factors: ArbEDSR, ArbRDN, and ArbRCAN significantly outperform their baselines on non-integer scales while remaining comparable on integer scales.ArbEDSR matches EDSR at ×2 on Set5 (38.19 vs. 38.19) and improves at ×1.6/1.55 (40.64/40.94 vs. 40.39/40.71).
- 4.5. Results for SR with Asymmetric Scale Factors: ArbRCAN outperforms fine-tuned Meta-RCAN on asymmetric Manga109 SR, reaching PSNR 37.93/33.81/34.41 versus 37.80/33.67/34.28.The comparison covers asymmetric scale factors and is reported in Table 4.
- 4.5. Results for SR with Asymmetric Scale Factors: ArbRCAN uses less memory and time than Meta-RCAN for asymmetric SR, with 0.7G versus 2.8G and 0.29s versus 0.61s.The paper attributes Meta-RCAN’s overhead to enlarging the output before bicubic downscaling, whereas ArbRCAN directly produces the expected size.
- 4.6. Continuous Magnification: Continuous magnification makes text easier to recognize, while ArbRCAN consistently produces better perceptual quality than competing methods.This evaluation concerns continuously varying output resolutions in the visual comparison.
5. Conclusions
The conclusion states that the plug-in module enables existing SR networks to perform scale-arbitrary SR with one model. It reports promising non-integer and asymmetric results, state-of-the-art integer-scale performance, and small added resource costs.
- The plug-in module enables existing SR networks to perform scale-arbitrary super-resolution with a single model.
- Equipped networks achieve promising non-integer and asymmetric SR while maintaining state-of-the-art performance on integer scales.
- The module is easily adapted to scale-specific networks with small additional computational and memory cost.