Source-linked AI summary
Meta-SR: A Magnification-Arbitrary Network for Super-Resolution
Xuecai Hu, Haoyuan Mu, Xiangyu Zhang, Zilei Wang, Tieniu Tan, Jian Sun
TL;DR
Existing SISR methods mainly address selected integer scale factors separately, leaving arbitrary-scale super-resolution underdeveloped. Meta-SR predicts scale- and coordinate-conditioned upscale filters in one model, achieving strong arbitrary-scale benchmark results with low upscale-module runtime.
Problem
Existing SISR methods mainly consider selected integer scale factors as independent tasks, while arbitrary scale factors require an efficient single-model solution.
Method
Meta-SR replaces the traditional upscale module with Meta-Upscale, which dynamically predicts filter weights from scale-factor and coordinate information.
Results
Meta-SR matches retrained baselines for integer factors and outperforms alternative arbitrary-scale strategies across benchmark experiments.
Takeaways & Limitations
A single Meta-SR model can continuously zoom an image with multiple scale factors, while Meta-Upscale takes about 1% of Feature Learning Module runtime.
Abstract
from arXiv · showhide
Recent research on super-resolution has achieved great success due to the development of deep convolutional neural networks (DCNNs). However, super-resolution of arbitrary scale factor has been ignored for a long time. Most previous researchers regard super-resolution of different scale factors as independent tasks. They train a specific model for each scale factor which is inefficient in computing, and prior work only take the super-resolution of several integer scale factors into consideration. In this work, we propose a novel method called Meta-SR to firstly solve super-resolution of arbitrary scale factor (including non-integer scale factors) with a single model. In our Meta-SR, the Meta-Upscale Module is proposed to replace the traditional upscale module. For arbitrary scale factor, the Meta-Upscale Module dynamically predicts the weights of the upscale filters by taking the scale factor as input and use these weights to generate the HR image of arbitrary size. For any low-resolution image, our Meta-SR can continuously zoom in it with arbitrary scale factor by only using a single model. We evaluated the proposed method through extensive experiments on widely used benchmark datasets on single image super-resolution. The experimental results show the superiority of our Meta-Upscale.
1. Introduction
Meta-SR addresses the practical need for single-image super-resolution at arbitrary, including non-integer, scale factors without storing a separate model for each factor. It replaces the fixed upscale module with dynamically predicted filters conditioned on scale and coordinates, and achieves strong benchmark results with low added runtime.
- Motivation: SISR needs user-customized arbitrary positive scale factors, but separate models for every factor are impractical to store and inefficient to compute.Existing methods commonly focus on X2, X3, and X4 as independent tasks.
- Approach: Meta-SR uses Feature Learning and Meta-Upscale Modules to support arbitrary-scale super-resolution with one model.Meta-Upscale replaces the typical upscale module.
- Approach: For each generated HR pixel, Meta-Upscale projects to the LR image and predicts filter weights from coordinate-related and scale-related vectors.The predicted filters convolve with features at the corresponding LR coordinate to generate the pixel value.
- Results: Meta-SR matches separately retrained baselines for integer factors and outperforms input/output zooming or feature-map interpolation for arbitrary factors.These comparisons were evaluated on multiple benchmark datasets for single-image super-resolution.
- Results: Meta-Upscale runs quickly: its inference time is about 1% of the time consumed by the RDN Feature Learning Module.The module consists of several fully connected layers.
2. Related Work
Earlier super-resolution methods evolved from exemplar and dictionary approaches to deep networks, while meta-learning research introduced prediction of network weights. Meta-SR builds on weight prediction to condition its upscale module on both image coordinates and scale factor.
- Single Image Super Resolution: Early exemplar- and dictionary-based methods depended on external image databases and were limited by database size, performance, and high computation time.They transferred relevant patches from database images to generate high-resolution results.
- Single Image Super Resolution: Deep SISR methods introduced convolutional, residual, recursive, adversarial, dense, attention, and iterative architectures, with sub-pixel convolution enabling efficient final upscaling.The literature includes SRCNN, DRCN, DRRN, ESPCNN, EDSR, MDSR, RDN, RCAN, DBPN, and DSRN.
- Meta-Learning: Meta-learning learns from the performance of machine-learning approaches across tasks, and weight prediction uses one network to predict another network’s weights.The paper situates its approach among few-shot, zero-shot, transfer-learning, and other weight-prediction applications.
- Meta-Learning: Parameterized Image Operators dynamically adjust image-operator weights, whereas Meta-SR reformulates the upscale module using coordinate and scale-factor inputs.The comparison identifies Meta-SR’s focus as the upscale module rather than general image filtering or restoration.
3. Our Approach
Meta-SR combines a Feature Learning Module with a Meta-Upscale Module that generates super-resolved images at arbitrary scale factors. The Meta-Upscale Module projects HR pixels to LR features, predicts location- and scale-conditioned filters, and maps those features into output pixels.
- Meta-SR replaces the traditional upscale module with a Meta-Upscale Module while using a Feature Learning Module to extract LR-image features.The implementation selects RDN as the Feature Learning Module, although other SISR networks can also serve this role.
- Feature Mapping: Feature Mapping applies the predicted filter weights to the feature at the corresponding LR coordinate to calculate each SR pixel value.The mapping is formulated as a matrix product between the extracted feature and the predicted weights.
- Weight Prediction: The Meta-Upscale Module predicts filter weights dynamically from coordinate- and scale-related inputs, avoiding separately stored weights for every scale factor.The scale factor is included so that jointly trained models can distinguish filters for different scale factors.
- Location Projection: For each SR pixel, Location Projection identifies a corresponding LR coordinate, including under non-integer scale factors such as r = 1.5.The projection acts like a variable fractional stride: some LR pixels determine two output pixels and others determine one.
- Implementation: The module loops over the output dimensions determined by int(inH × r) and int(inW × r), predicts weights per output pixel, and computes its pixel value.Algorithm 1 takes the scale, input size, weight-prediction function, and LR feature map as inputs.
- Architecture Details: The chosen Meta-Upscale architecture uses fully connected and activation layers, with 256 hidden neurons, two fully connected layers, ReLU, and a 3 × 3 kernel.The paper reports this configuration as balancing speed and performance, while 5 × 5 convolution is more time-consuming on large feature maps.
4. Experiments
Experiments evaluate Meta-SR on benchmark datasets, arbitrary-scale baselines, inference efficiency, and comparisons with retrained state-of-the-art models. Results show that Meta-SR supports strong arbitrary-scale performance with one model and becomes especially efficient for repeated zooming across scales.
- Experimental Setup: Experiments use DIV2K training images and evaluate on Set14, B100, Manga109, and DIV2K benchmark datasets with bicubic degradation.DIV2K contains 800 training, 100 validation, and 100 test images; the DIV2K test ground truth is unavailable publicly.
- Baselines: The arbitrary-scale baselines include bicubic interpolation, CNNs fed pre-upscaled inputs, networks followed by downscaling, and fixed or predicted feature-map upscaling.The baselines are trained jointly across arbitrary scale factors for the BiConv and Meta-Bi comparisons.
- Arbitrary-Scale Results: Meta-SR outperforms the arbitrary-scale baselines on almost all scale factors, while Meta-Bi significantly outperforms BiConv.The results attribute these gains to scale-specific weight prediction and a consistent valid field of view across scale factors.
- Inference Time: The Weight Prediction Module’s running time can be neglected relative to the Feature Learning Module because Meta-Upscale uses only two fully connected layers.Running time is measured on B100 at test scale factor 2, excluding image preprocessing.
- Inference Time: Meta-SR has no efficiency advantage over RDN(x1), RDN(x2), and RDN(x4) at scale r = 2, but is less time-consuming at scale factors 8 or 16.For continuously zooming the same image at different scales, Meta-SR is reported as the fastest because only its Meta-Upscale Module runs for each scale.
- Comparison With SOTA Methods: Meta-RDN achieves comparable or better results than corresponding RDN models retrained separately for X2, X3, and X4.A single Meta-SR model can handle multiple scale factors, whereas typical models require several saved models.
- Visual Results: Visual comparisons report better structural performance for Meta-RDN than RDN(x1), RDN(x2), and RDN(x4), whose textures are worse across scale factors.The reported difference is linked to Meta-SR’s ability to predict independent filter weights for each scale factor.
5. Conclusion
Meta-SR introduces Meta-Upscale, which dynamically predicts scale-specific filter weights to generate arbitrarily sized high-resolution images with one model. It can also continuously zoom the same image across multiple scale factors.
- Meta-Upscale dynamically predicts filter weights for each scale factor and generates a high-resolution image of arbitrary size.It replaces the conventional upscale module and applies the predicted filters to feature maps.
- A single Meta-SR model supports super-resolution with arbitrary scale factors.The method generates scale-specific upscale weights without requiring a separate stored model for each factor.
- Meta-SR can continuously zoom the same image using multiple scale factors.