Source-linked AI summary
CrossNet: An End-to-end Reference-based Super Resolution Network using Cross-scale Warping
Haitian Zheng, Mengqi Ji, Haoqian Wang, Yebin Liu, Lu Fang
TL;DR
RefSR must transfer details from an HR reference to an LR image despite parallax and a large resolution gap, while existing cascaded pipelines face alignment and optimization issues. CrossNet uses end-to-end cross-scale warping with encoders and a fusion decoder to align and synthesize reference-based super-resolution outputs. It reports around 2dB-4dB gains and more than 100x speedups, with generalization challenged by larger cross-dataset parallax.
Problem
RefSR must transfer high-frequency details from an HR reference to an LR image despite parallax and an 8x resolution gap, requiring correspondence and high-resolution synthesis.
Method
CrossNet is an end-to-end, fully convolutional network that extracts multi-scale features, spatially aligns reference features through cross-scale warping, and fuses them to synthesize the HR output.
Results
CrossNet achieves around 2dB-4dB gains over previous SISR and RefSR methods and more than 100x speedup over existing RefSR approaches.
Takeaways & Limitations
CrossNet's end-to-end cross-scale alignment supports precise and efficient reference-based super-resolution, including real-time applications.
Takeaways & Limitations
Cross-dataset generalization is challenged by larger parallax in Stanford and Scene Light Field datasets, for which the model was not specifically trained.
Abstract
from arXiv · showhide
The Reference-based Super-resolution (RefSR) super-resolves a low-resolution (LR) image given an external high-resolution (HR) reference image, where the reference image and LR image share similar viewpoint but with significant resolution gap x8. Existing RefSR methods work in a cascaded way such as patch matching followed by synthesis pipeline with two independently defined objective functions, leading to the inter-patch misalignment, grid effect and inefficient optimization. To resolve these issues, we present CrossNet, an end-to-end and fully-convolutional deep neural network using cross-scale warping. Our network contains image encoders, cross-scale warping layers, and fusion decoder: the encoder serves to extract multi-scale features from both the LR and the reference images; the cross-scale warping layers spatially aligns the reference feature map with the LR feature map; the decoder finally aggregates feature maps from both domains to synthesize the HR output. Using cross-scale warping, our network is able to perform spatial alignment at pixel-level in an end-to-end fashion, which improves the existing schemes both in precision (around 2dB-4dB) and efficiency (more than 100 times faster).
1 Introduction
RefSR uses an HR reference to super-resolve an LR image, but parallax and an 8x resolution gap make correspondence and synthesis difficult. CrossNet replaces cascaded patch matching and synthesis with an end-to-end warping-and-synthesis network that improves precision and efficiency.
- RefSR uses an external HR reference with a similar viewpoint to super-resolve an LR image, often outperforming single-image SR.
- Parallax and the 8x resolution gap create two critical challenges: correspondence between inputs and high-resolution synthesis.
- CrossNet replaces patch matching with warping in an end-to-end Encoder-Warping-Decoder architecture.
- Multi-scale feature extraction and spatial warping align information across scales, while the decoder aggregates features to synthesize the HR output.
- 2dB-4dB gain over state-of-the-art SISR and RefSR methods was reported across datasets, viewpoint disparities, and scales.
2 Related work
Related work spans single-image super-resolution, reference-based super-resolution, and warping-based image or video synthesis. CrossNet builds on these directions by using an encoder-decoder with cross-scale alignment for reference-based reconstruction.
- Single-image super-resolution: SISR methods learn mappings from LR patches to HR patches using techniques including nearest-neighbor search, trees, regression, and deep neural networks.
- Single-image super-resolution: Deep learning substantially improved SISR performance, while some methods accelerated computation by extracting features in low-resolution domains.
- Reference-based super-resolution: CrossNet uses MDSR as a sub-module for LR feature extraction and RefSR synthesis.
- Reference-based super-resolution: RefSR methods use additional images from different viewpoints, including patch-based synthesis, patch registration, dictionary learning, and iterative approaches.
- Image/video synthesis using warping: Warping-and-synthesis methods align additional images to a target using estimated flow before synthesizing an image or video frame.
3 Approach
CrossNet is an end-to-end, fully convolutional RefSR network that aligns reference information with LR features through cross-scale warping before multi-scale fusion and synthesis. Its design replaces patch matching with pixel-wise, multi-scale alignment and directly trains the network to produce the SR output.
- Cross-scale alignment: Cross-scale warping assigns each pixel a shift vector, avoiding the locally constant-depth assumption used by patch matching.The operation is designed for non-rigid image transformation and can avoid blocky and blurry artifacts.
- Cross-scale alignment: The flow estimator predicts cross-scale correspondence at multiple scales from an upsampled LR image and the reference image.It is based on FlowNetS, modified with two ×2 upsampling modules and skip connections for finer flow-field definition.
- Efficiency: The fully convolutional flow estimation is computationally less expensive than sliding-window patch matching.Patch matching calculates pixel-wise flow with a sliding-window scheme, whereas CrossNet estimates the cross-scale flow field using a fully convolutional network.
- Network structure: CrossNet uses separate LR and reference encoders, cross-scale alignment, and a U-Net-like decoder for multi-scale feature fusion and SR synthesis.The LR encoder extracts multi-scale features, while the reference encoder extracts reference features that are spatially aligned before decoding.
- Decoder: Warped reference features and LR features are concatenated with preceding decoder features, then processed through deconvolution layers to generate decoder features.At each scale, the decoder uses 4 × 4 deconvolution filters with stride 2; ⋆ denotes deconvolution.
- Training: CrossNet is trained directly to synthesize the SR output using a loss between its prediction and the ground-truth HR image.The stated loss uses a Charbonnier penalty over training samples and spatial locations.
4 Experiment
Experiments evaluate CrossNet across datasets, parallax settings, upsampling scales, and comparisons with SISR and RefSR methods. CrossNet generally improves reconstruction quality, generalizes with retraining, and substantially reduces inference time.
- Datasets and settings: Experiments use Flower and LFVideo datasets for training and evaluation, with Stanford and Scene Light Field datasets used to assess generalization.Training uses randomly selected LR/reference pairs from the 8 × 8 angular grid; testing uses specified angular positions and cropped 320 × 512 inputs.
- Quantitative evaluation: CrossNet is compared with SS-Net, PatchMatch, SRCNN, VDSR, and MDSR using PSNR, SSIM, and IFC at ×4 and ×8 scales.The comparisons include small and large parallax settings across multiple datasets.
- Quantitative evaluation: 2dB-4dB gain: CrossNet outperforms previous SISR and RefSR approaches under varied parallax settings, upsampling scales, and datasets.The result is reported for quantitative comparisons using PSNR, SSIM, and IFC.
- Parallax analysis: CrossNet consistently outperforms competing approaches across different disparities, datasets, and scales, although RefSR performance deteriorates as parallax increases.SISR methods show a U-shaped PSNR pattern across views, while RefSR methods benefit from reference-image high-frequency details.
- Visual comparison: At ×8 scale, CrossNet produces finer visual details resembling the ground-truth image compared with SRCNN, VDSR, MDSR, PatchMatch, and SS-Net.RefSR methods retain high-frequency details more explicitly than the compared SISR methods.
- Efficiency: CrossNet generates a 320 × 512 ×8 RefSR image within 1 second, compared with 86.3 seconds for PatchMatch and 105.6 seconds for SS-Net.The CrossNet time comprises 0.75 seconds for MDSR preprocessing and 0.12 seconds for final synthesis.
- Ablation study: CrossNet outperforms its single-scale image-warping counterpart by 0.20dB and 0.27dB on the training set and performs especially better at small parallax.The comparison evaluates the effectiveness of multi-scale feature warping.
5 Conclusion
CrossNet is an end-to-end, fully convolutional reference-based super-resolution network for producing 4x or 8x results from LR images using HR references. Experiments report around 2dB-4dB superior performance and over 100x speedup compared with previous methods.
- CrossNet takes an LR image and a similar-viewpoint HR reference image as input and produces a 4x or 8x super-resolved result.
- The fully convolutional pipeline consists of an encoder, cross-scale warping, and decoder.
- Around 2dB-4dB superior performance is demonstrated across several large-scale datasets compared with previous methods.
- More than 100 times faster execution makes CrossNet applicable to real-time applications.