Source-linked AI summary
MASA-SR: Matching Acceleration and Spatial Adaptation for Reference-Based Image Super-Resolution
Liying Lu, Wenbo Li, Xin Tao, Jiangbo Lu, Jiaya Jia
TL;DR
RefSR must match and transfer useful details from an external reference despite expensive correspondence search and distribution differences between LR and reference images. MASA-SR introduces coarse-to-fine matching and spatial adaptation, and reports state-of-the-art quantitative and qualitative results across datasets.
Problem
RefSR needs efficient, high-quality correspondence matching and effective feature transfer despite costly dense matching and LR–reference distribution disparities.
Method
MASA-SR uses a coarse-to-fine Match & Extraction Module and a Spatial Adaptation Module that remaps reference-feature distributions to LR distributions spatially.
Results
MASA-SR achieves state-of-the-art quantitative and qualitative results across different datasets, with correspondence matching requiring two orders of magnitude fewer FLOPS.
Takeaways & Limitations
The method is designed to reduce matching cost while improving robustness and utilization of reference information across different reference images.
Takeaways & Limitations
The matching design assumes local coherence in natural images when restricting each LR patch’s search to a reference block.
Abstract
from arXiv · showhide
Reference-based image super-resolution (RefSR) has shown promising success in recovering high-frequency details by utilizing an external reference image (Ref). In this task, texture details are transferred from the Ref image to the low-resolution (LR) image according to their point- or patch-wise correspondence. Therefore, high-quality correspondence matching is critical. It is also desired to be computationally efficient. Besides, existing RefSR methods tend to ignore the potential large disparity in distributions between the LR and Ref images, which hurts the effectiveness of the information utilization. In this paper, we propose the MASA network for RefSR, where two novel modules are designed to address these problems. The proposed Match & Extraction Module significantly reduces the computational cost by a coarse-to-fine correspondence matching scheme. The Spatial Adaptation Module learns the difference of distribution between the LR and Ref images, and remaps the distribution of Ref features to that of LR features in a spatially adaptive way. This scheme makes the network robust to handle different reference images. Extensive quantitative and qualitative experiments validate the effectiveness of our proposed model.
1. Introduction
RefSR uses an external reference image to recover high-frequency details, but dense correspondence matching is costly and LR–Ref distribution differences complicate feature transfer. MASA-SR addresses these issues with coarse-to-fine matching and spatially adaptive feature remapping, achieving state-of-the-art results.
- Motivation: RefSR transfers similar content and texture from an external reference image to improve details beyond single-image super-resolution.Reference images may come from web searches or different viewpoints.
- Challenges: Dense feature-space patch matching provides correspondence but incurs very high computational cost and memory usage.Alignment-based alternatives also struggle with long-distance correspondence.
- Challenges: LR and reference images can have substantially different color and luminance distributions, while direct feature concatenation and convolutional fusion is not optimal.This distribution disparity can occur even when the images share similar content.
- Proposed method: MASA-SR’s Match & Extraction Module shrinks each LR patch’s search from the whole reference feature map to a specific block through coarse-to-fine matching.The design leverages local coherence in natural images while maintaining matching quality.
- Proposed method: The Spatial Adaptation Module remaps reference-feature distributions to LR distributions spatially, improving robustness to references with different color and luminance distributions.The module enables more effective utilization of useful reference information.
- Results: A two-orders-of-magnitude reduction in FLOPS is achieved for correspondence matching, and MASA-SR attains state-of-the-art quantitative and qualitative RefSR results.The reported results cover different datasets.
2. Related Work
Related work spans CNN-based single-image super-resolution, perception-driven reconstruction, and reference-based methods using alignment or patch matching. These approaches establish the context for MASA-SR’s focus on correspondence and reference-feature use.
- Single-image super-resolution: CNN-based SISR progressed from early three-layer models to deeper residual and multi-scale architectures with improved accuracy.Examples include SRCNN, VDSR, DRCN, and LapSRN.
- Perception-driven super-resolution: Perception-driven SISR uses perceptual or adversarial losses to improve visual quality and produce more natural details.ESRGAN further improves SRGAN with a relativistic adversarial loss.
- Reference-based super-resolution: RefSR adds a similar-content reference image that can provide high-frequency details for upsampling the LR input.CrossNet estimates multi-scale flow and warps reference features, but relies on a pretrained network with heavy computation and inaccurate flow.
- Reference-based super-resolution: Patch-matching RefSR methods learn correspondence, synthesize or swap reference patches, and increasingly use cross-scale transformer architectures.SRNTT performs multi-level matching, while TTSR stacks transformers cross-scale.
3. MASA-SR Method
MASA-SR combines coarse-to-fine correspondence matching with spatially adaptive feature distribution remapping for reference-based super-resolution. Its framework uses an encoder, MEM for matching and extraction, and SAM/DRAM fusion modules for adapting and enhancing features.
- Framework: The framework comprises an encoder, Match & Extraction Modules, Spatial Adaptation Modules, and Dual Residual Aggregation Modules.The encoder processes LR and reference inputs, while the other modules perform correspondence matching, feature adaptation, and fusion.
- Feature Fusion: The fusion module combines LR and extracted reference features through SAM and DRAM to align statistics and enhance high-frequency details.DRAM aggregates refined LR and reference details into more representative features for the final super-resolution output.
- Match & Extraction Module: MEM reduces correspondence-search cost by matching each LR block to a relevant reference block before performing dense patch matching within that block.The method exploits local coherence to shrink each patch’s search space from the whole reference feature map to a specific reference block.
- Match & Extraction Module: Using coarse-to-fine matching changes the complexity from O(mn) to O(Kn + mn′), substantially reducing computation when K is much smaller than m and n′ than n.Here m and n are the LR and downsampled-reference pixel counts, while n′ is the reference-block size.
- Spatial Adaptation Module: SAM remaps extracted reference-feature distributions to match LR-feature distributions, using spatially predicted adaptation parameters β and γ.The module addresses differing color and luminance distributions by deriving β and γ from concatenated LR and reference features, then adapting normalized reference features.
4. Experiments
MASA-SR is evaluated on three benchmarks against SISR and RefSR methods, with experiments assessing performance, efficiency, visual quality, and module ablations. It achieves strong quantitative and qualitative results while balancing matching accuracy and computational cost.
- Datasets and setup: MASA-SR is trained on CUFED5 with ×4 upscaling and evaluated on CUFED5, Urban100, and Sun80.CUFED5 provides 11,871 training pairs and 126 testing pairs; Urban100 uses self-similarity, while Sun80 uses sampled references.
- Comparison with state-of-the-art methods: The evaluation compares MASA-SR with SISR and RefSR methods using PSNR, SSIM, visual quality, FLOPS, parameters, and runtime.A reconstruction-only version, MASA-rec, is trained for fair comparison with MSE-minimization methods.
- Quantitative evaluation: MASA-SR outperforms state-of-the-art methods on all three testing sets in PSNR and SSIM.The model also achieves the best PSNR/SSIM performance in the efficiency comparison, while CrossNet has lower FLOPS and runtime but lower comparative PSNR/SSIM performance.
- Qualitative evaluation: MASA-SR produces finer hair, wrinkle, and leaf textures with few unpleasing artifacts compared with other SISR and RefSR methods.Visual comparisons cover CUFED5, Sun80, and Urban100 examples.
- Matching ablations: Increasing LR block size lowers both PSNR and FLOPS, whereas increasing Ref↓ block size raises both until PSNR gains slow.A 1 × 1 LR block reaches 27.60 dB PSNR but requires 787.77G FLOPS; larger dilation combinations improve PSNR through broader coarse matching.
- Module ablations: SAM improves PSNR by 0.22 dB and SSIM by 0.007 with 0.15M additional parameters, and remains robust when reference luminance changes.DRAM also outperforms simple concatenation by 0.11 dB.
5. Conclusion
MASA-SR is an end-to-end RefSR network that combines efficient coarse-to-fine matching with spatial adaptation. It achieves state-of-the-art quantitative and qualitative results across different datasets.
- MASA-SR is an end-to-end trainable network for reference-based image super-resolution.
- Its Match & Extraction Module uses coarse-to-fine correspondence matching to significantly reduce computational cost while retaining matching and transfer capability.
- Its Spatial Adaptation Module improves robustness when reference images have different distributions from the low-resolution images.
- MASA-SR achieves state-of-the-art quantitative and qualitative results across different datasets.
A.1. Details of Ablation Study
The ablation study evaluates LR block size, downsampled-reference block size, and dilation under specified input resolutions and fixed settings.
- The ablation study varies LR block size, downsampled-reference block size, and dilation rates.
- Influence of LR block sizes: For LR block-size ablations, FLOPS use 192 × 192 LR and 768 × 768 reference inputs, with reference-block scale factor 1.5 and dilation 1.
- Influence of Ref↓ block sizes: For downsampled-reference block-size ablations, FLOPS use 128 × 128 LR and 512 × 512 reference inputs, with LR block size 8 and dilation 1.
- Influence of dilation rates: For dilation-rate ablations, FLOPS use 120×120 LR and 480×480 reference inputs, with LR block size 12 and reference-block scale factor 1.5.
A.2. More Visual Results
The paper presents additional visual comparisons on CUFED5, Sun80, and Urban100 against several state-of-the-art SR and RefSR methods. MASA is reported to restore more regular structures and photo-realistic details.
- The visual comparisons include RCAN, HAN, ESRGAN, SRNTT, TTSR, and MASA.
- Figures 7 and 8 compare methods on the CUFED5 testing set.
- Figure 9 compares methods on the Sun80 dataset, while Figure 10 compares them on Urban100.
- MASA restores more regular structures and generates photo-realistic details in the visual results.