Source-linked AI summary
NAFSSR: Stereo Image Super-Resolution Using NAFNet
Xiaojie Chu, Liangyu Chen, Wenqing Yu
TL;DR
Stereo super-resolution needs to exploit complementary information across binocular views without making systems unnecessarily complex. NAFSSR extends NAFNet with stereo cross-attention and supporting training/testing strategies, achieving state-of-the-art performance across evaluated datasets and winning the NTIRE 2022 challenge.
Problem
Existing stereo super-resolution methods use increasingly specialized architectures and losses to exploit cross-view information, raising system complexity and complicating method analysis and comparison.
Method
NAFSSR extends NAFNet with weight-sharing intra-view feature extraction, stereo cross-attention for cross-view fusion, and training/testing strategies including stochastic depth, channel shuffle, and TLSC.
Results
NAFSSR surpasses existing methods across the KITTI 2012, KITTI 2015, Middlebury, and Flickr1024 datasets, and wins first place in the NTIRE 2022 Stereo Image Super-resolution Challenge.
Takeaways & Limitations
NAFSSR provides a simple, strong baseline with favorable parameter and performance trade-offs for stereo image super-resolution.
Abstract
from arXiv · showhide
Stereo image super-resolution aims at enhancing the quality of super-resolution results by utilizing the complementary information provided by binocular systems. To obtain reasonable performance, most methods focus on finely designing modules, loss functions, and etc. to exploit information from another viewpoint. This has the side effect of increasing system complexity, making it difficult for researchers to evaluate new ideas and compare methods. This paper inherits a strong and simple image restoration model, NAFNet, for single-view feature extraction and extends it by adding cross attention modules to fuse features between views to adapt to binocular scenarios. The proposed baseline for stereo image super-resolution is noted as NAFSSR. Furthermore, training/testing strategies are proposed to fully exploit the performance of NAFSSR. Extensive experiments demonstrate the effectiveness of our method. In particular, NAFSSR outperforms the state-of-the-art methods on the KITTI 2012, KITTI 2015, Middlebury, and Flickr1024 datasets. With NAFSSR, we won 1st place in the NTIRE 2022 Stereo Image Super-resolution Challenge. Codes and models will be released at https://github.com/megvii-research/NAFNet.
1. Introduction
NAFSSR extends the simple NAFNet restoration model with cross-view attention for stereo super-resolution, addressing the complexity of existing methods while using both intra-view and cross-view information. Training and testing strategies further support its performance and efficiency.
- Stereo super-resolution requires both within-view context and information exchanged between left and right images.
- Existing stereo methods often increase system complexity through specialized architectures and losses, hindering convenient analysis and comparison.
- NAFSSR extends NAFNet with simple cross-attention modules that fuse features between views while retaining NAFNet’s intra-view representation.
- Training uses channel shuffle and stochastic depth, while testing uses TLSC to address train-test inconsistency in stereo super-resolution.
- NAFSSR achieves better performance-parameter trade-offs than existing methods and won first place in the NTIRE 2022 Stereo Image Super-resolution Challenge.
2. Related Works
The related work spans increasingly capable single-image restoration and stereo-specific methods. NAFSSR extends a simple NAFNet baseline with stereo cross-attention, using cross-view information without extra pretraining or multiple losses.
- Single-image super-resolution methods improve representation through deeper or wider networks, residual and dense connections, and attention mechanisms.
- SwinIR achieves state-of-the-art single-image super-resolution, while NAFNet provides a simple competitive restoration baseline that this paper extends to stereo super-resolution.
- Stereo super-resolution methods use mechanisms such as continuous parallax mapping, epipolar-line parallax attention, and stereo attention modules to exploit binocular information.
- NAFSSR uses a simple stereo cross-attention module and trains directly on stereo images from scratch with only L1 loss, unlike SAM’s extra pretraining and multiple losses.
- The paper situates stochastic depth within differing evidence about whether regularization benefits image super-resolution.
3. Method
NAFSSR processes stereo pairs with shared NAFNet-style feature extractors, bidirectional cross-view fusion, and reconstruction. Its design combines nonlinear-activation-free NAFBlocks, epipolar-aware attention, regularization, and a local pooling strategy for train-test consistency.
- Architecture: NAFSSR divides stereo processing into intra-view feature extraction, cross-view feature fusion, and reconstruction using two weight-sharing NAFBlock networks.
- Architecture: The network uses convolutional feature mapping, stacked NAFBlocks, global residual learning, and prediction of the residual over a bilinearly upsampled input.
- Cross-view fusion: SCAM is inserted after each NAFBlock to perform bidirectional cross-view interaction and fuse the resulting features with intra-view features.
- NAFBlock: NAFBlocks combine MBConv, channel attention, and FFN modules with layer normalization and residual connections, without nonlinear activation functions.
- NAFBlock: SimpleGate splits channels into two features and multiplies them element-wise, replacing nonlinear activation within the NAFBlock.
- Cross-view fusion: SCAM applies scaled dot-product attention between left and right features along horizontal epipolar lines, then adds cross-view information to each intra-view feature.
- Training: Channel shuffle augments training inputs, stochastic depth regularizes training, and the objective is the pixel-wise L1 distance for both stereo outputs.
- Train-test inconsistency: The train-test inconsistency arises because training patches cover much smaller regions than inference images, motivating local pooling during inference.
4. Experiments
Experiments show that NAFSSR benefits from cross-view attention, augmentation, stochastic depth, and TLSC, achieving strong accuracy, efficiency, and visual quality across stereo SR benchmarks.
- Stereo Cross-Attention Module: 0.29 dB PSNR improvement on Flickr1024 results from adding 32 SCAMs over the single-view baseline, compared with 0.18 dB from one SCAM.SCAMs interact with cross-view information after NAFBlocks, complementing intra-view features.
- Data augmentations: 0.39 dB PSNR improvement raises NAFSSR-S from 23.43 dB to 23.82 dB when combining horizontal flip, vertical flip, and channel shuffle.Channel shuffle alone improves PSNR by 0.19 dB, comparable to horizontal flip (+0.21 dB) and vertical flip (+0.20 dB).
- Stochastic-Depth and TLSC: +0.16 dB on out-distribution test data is obtained for NAFSSR-B with stochastic depth, while NAFSSR-S gains +0.03 dB across datasets.With stochastic depth, NAFSSR-B exceeds NAFSSR-T by 0.20 dB on out-of-distribution data, indicating improved generality for larger models.
- Stochastic-Depth and TLSC: TLSC achieves similar improvements for NAFSSR-T and NAFSSR-B across all datasets during inference.The authors attribute the need for TLSC to train-test inconsistency in stereo image SR.
- Quantitative Evaluations: 0.38–0.66 dB gains over SSRDE-FNet are achieved by NAFSSR-B for 4× SR on KITTI 2012, KITTI 2015, Middlebury, and Flickr1024.NAFSSR-S outperforms prior state-of-the-art results across all datasets and upsampling factors.
- Parameter Efficiency and Scaling Ability: 79% parameter reduction gives NAFSSR-T a state-of-the-art result, while scaling to NAFSSR-B and NAFSSR-L further improves stereo SR performance.All NAFSSR variants exceed SSRDE-FNet by 0.05–0.48 dB on Flickr1024, with up to 5.11× speedup.
- Visual Comparison: NAFSSR-B reconstructs rich details and clear edges in visual comparisons, whereas competing methods may produce unsatisfactory artifacts.The comparisons cover 4× stereo SR on Flickr1024, KITTI 2012, KITTI 2015, and Middlebury.
- NTIRE Stereo Image SR Challenge: 24.239 dB PSNR on the validation set and 23.787 dB PSNR on the test set earned the final submission first place in the NTIRE 2022 challenge.The submission used stronger stochastic depth, self-ensemble, and model ensemble strategies.
5. Conclusion
The paper presents NAFSSR as a simple stereo image super-resolution baseline that combines NAFNet-based intra-view extraction with stereo cross-attention and complementary training/testing strategies. Experiments show state-of-the-art performance.
- Conclusion: NAFSSR combines stacked NAFBlocks for intra-view extraction with stereo cross-attention modules for cross-view interaction.The method also uses data augmentation, stochastic depth, and test-time local conversion to improve performance and generality.