Source-linked AI summary
Image Super-Resolution by Neural Texture Transfer
Zhifei Zhang, Zhaowen Wang, Zhe Lin, Hairong Qi
TL;DR
SISR loses fine texture, while RefSR depends on sufficiently similar references. The paper introduces SRNTT, which performs adaptive multi-scale neural texture transfer, and reports robust texture recovery across reference-similarity levels while providing the CUFED5 benchmark.
Problem
SISR struggles with lost fine texture, while RefSR quality can degrade severely when reference images are dissimilar or poorly aligned.
Method
SRNTT is an end-to-end model that adaptively transfers reference textures using multi-scale matching in neural feature space.
Results
SRNTT produces finer textures with similar references and remains comparable to SRGAN with fewer artifacts for unrelated references.
Takeaways & Limitations
CUFED5 supports RefSR evaluation with references spanning different similarity levels, while SRNTT extends RefSR to arbitrary references.
Abstract
from arXiv · showhide
Due to the significant information loss in low-resolution (LR) images, it has become extremely challenging to further advance the state-of-the-art of single image super-resolution (SISR). Reference-based super-resolution (RefSR), on the other hand, has proven to be promising in recovering high-resolution (HR) details when a reference (Ref) image with similar content as that of the LR input is given. However, the quality of RefSR can degrade severely when Ref is less similar. This paper aims to unleash the potential of RefSR by leveraging more texture details from Ref images with stronger robustness even when irrelevant Ref images are provided. Inspired by the recent work on image stylization, we formulate the RefSR problem as neural texture transfer. We design an end-to-end deep model which enriches HR details by adaptively transferring the texture from Ref images according to their textural similarity. Instead of matching content in the raw pixel space as done by previous methods, our key contribution is a multi-level matching conducted in the neural space. This matching scheme facilitates multi-scale neural transfer that allows the model to benefit more from those semantically related Ref patches, and gracefully degrade to SISR performance on the least relevant Ref inputs. We build a benchmark dataset for the general research of RefSR, which contains Ref images paired with LR inputs with varying levels of similarity. Both quantitative and qualitative evaluations demonstrate the superiority of our method over state-of-the-art.
1. Introduction
The paper addresses texture loss and similarity assumptions in RefSR by introducing SRNTT, which transfers reference textures adaptively and remains robust to unrelated references. It also contributes CUFED5 for evaluating references with varied similarity levels.
- At 4× upscaling, SISR methods struggle to recover fine texture and perceptual constraints can hallucinate textures or create artifacts.
- RefSR uses HR references to restore lost details, but existing methods degrade when references lack similar content or alignment.
- SRNTT adaptively transfers textures through local matching in feature space, learning to leverage similar textures while suppressing dissimilar ones.
- With similar references, SRNTT synthesizes finer textures; with unrelated references, it remains comparable to SRGAN with similar visual quality and fewer artifacts.
- CUFED5 provides training and testing references with different similarity levels across content, texture, color, illumination, and viewpoint.
- SRNTT is an end-to-end model using multi-scale neural texture transfer and is evaluated for visual improvement, effectiveness, and adaptiveness.
2. Related Works
Related work covers deep-learning SISR and RefSR methods, emphasizing the tension between pixel-based fidelity and perceptual quality and the alignment assumptions of reference usage.
- 2.1. Deep Learning based SISR: Deep-learning SISR methods have improved PSNR and visual quality, but MSE optimization may not match human perceptual evaluation.
- 2.1. Deep Learning based SISR: Perceptual and adversarial losses improve visual quality, but prior work notes that adversarial approaches can hallucinate textures and artifacts.
- 2.2. Reference-based Super-Resolution: RefSR adds reference images to assist super-resolution, typically requiring similar texture or content structure relative to the LR image.
- 2.2. Reference-based Super-Resolution: Existing reference sources include video frames, web retrieval, external databases, different viewpoints, and self-example patches.
- 2.2. Reference-based Super-Resolution: Some RefSR methods use global registration and local matching but assume references are well aligned, while shallow patch fusion inadequately models LR-to-detail dependencies.
3. Approach
SRNTT performs reference-based super-resolution through multi-scale neural texture matching and transfer, using feature swapping and an end-to-end generative network. It adaptively emphasizes textures that match the LR input while suppressing irrelevant reference content.
- Neural texture transfer: SRNTT searches for locally matching textures in neural feature space and transfers them to the SR image across multiple scales.Higher-level features capture semantic similarity, while lower-level features capture textual similarity.
- Feature swapping: Bicubic-upsampled LR features are matched against frequency-aligned, blurred reference features rather than requiring a global transformation or optical flow.Local neural matching reduces sensitivity to reference color, illumination, and global structural differences.
- Feature swapping: The matching scores select reference patches for a swapped feature map, while raw reference patches preserve their original HR information during swapping.Dense patch sampling uses overlapping swapped features to construct the map used for subsequent texture transfer.
- Neural texture transfer: Swapped texture feature maps are merged into a residual network at multiple feature layers, where texture is extracted from each map and combined with target content.The network uses residual blocks, skip connections, channel concatenation, and progressive 2× upscaling to produce the final SR result.
- Training objective: SRNTT adds a texture-consistency loss that weights matched reference textures by their similarity to the LR input.Dissimilar textures receive lower weights and therefore lower texture-transfer penalties.
- Training objective: The training objective combines reconstruction, perceptual, adversarial, and texture losses to preserve structure, improve visual quality, and exploit reference textures.The reconstruction term uses an ℓ1-norm, which the paper states sharpens results relative to MSE and aligns with its WGAN-GP objective.
4. Dataset
CUFED5 is a RefSR benchmark designed to evaluate methods across references with different similarity levels. It pairs HR/LR data with references ranging from highly similar to weakly similar and supports broader generalization testing.
- Dataset motivation: CUFED5 addresses the need for RefSR training and evaluation data containing references with varied similarity to the LR image.Reference similarity can differ in content, texture, color, illumination, and viewpoint.
- Dataset construction: The dataset defines four reference levels, L1 through L4, from high to low similarity according to the number of best SIFT-feature matches.Each training pair uses randomly cropped 160×160 patches from paired images.
- Dataset scale: CUFED5 contains 13,761 paired training patches and 126 testing groups, each pairing one HR image with four references at levels L1, L2, L3, and L4.The testing design evaluates how adaptively RefSR methods handle progressively less similar references.
- Generalization evaluation: For generalization evaluation, the trained model is also tested on Sun80 with web-search references and Urban100, which contains building images without references.These datasets provide settings beyond the CUFED5 reference-pair construction.
5. Experimental Results
Experiments show that SRNTT improves texture recovery with useful references, remains competitive without external references, and adapts as reference similarity changes. Its visual gains are supported by user preference, patch matching, texture loss, and multi-scale transfer.
- Quantitative Evaluation: SRNTT-ℓ2 achieves the highest score on CUFED5 and Sun80 with references, while performing comparably to MDSR on Urban100 without references.The cited comparison covers PSNR/SSIM results across the three datasets.
- Quantitative Evaluation: SRNTT outperforms CrossNet in recovering fine texture when references are misaligned, because CrossNet cannot handle the large disparity.The comparison attributes the difference to reference disparity and misalignment.
- Quantitative Evaluation: SRNTT outperforms other SR methods on Sun80 with references and achieves state-of-the-art SISR performance when the LR input is used as reference.On Urban100, where no HR references are available, the LR input serves as the reference and still yields finer transferable texture.
- Qualitative Evaluation by User Study: Over 90% of users favor SRNTT over each compared algorithm in pair-wise visual-quality evaluations.The study collected 2,400 votes for each reference level on CUFED5 testing results.
- Ablation Study: Texture loss improves fine-texture recovery for highly similar references, while local matching, adaptive neural transfer, and spatial/perceptual regularization preserve consistency.Disabling texture loss produces degraded visual quality compared with the corresponding examples.
- Effect of reference similarity: As reference similarity decreases, SRNTT performance degrades gracefully, while SRNTT-flow shows large degradation even at the highly similar HR level.At the LR level, SRNTT remains comparable to state-of-the-art SISR algorithms.
- Layers for feature swapping: Multi-scale feature swapping improves PSNR, while higher-level relu3 contributes less high-frequency texture information than relu1 and relu2.The relu3 behavior also indicates stronger adaptiveness in suppressing low-similarity textures and emphasizing spatial reconstruction.
6. Conclusion
The paper presents SRNTT as an end-to-end method for adaptive multi-level texture transfer from arbitrary references. Experiments support its effectiveness and adaptiveness, while CUFED5 provides a benchmark for evaluating general RefSR.
- Conclusion: SRNTT performs multi-level adaptive texture transfer from references to recover more plausible texture in super-resolved images.The paper evaluates both effectiveness and adaptiveness quantitatively and qualitatively.
- Conclusion: CUFED5 provides a benchmark for evaluating reference-based super-resolution methods and supporting future RefSR research.The dataset is constructed as part of the paper's evaluation framework.