Source-linked AI summary
Learning Parallax Attention for Stereo Image Super-Resolution
Longguang Wang, Yingqian Wang, Zhengfa Liang, Zaiping Lin, Jungang Yang, Wei An, Yulan Guo
TL;DR
Stereo SR can benefit from a second viewpoint, but varying disparities make stereo correspondence difficult to use. PASSRnet applies global, epipolar-line parallax attention and introduces Flickr1024; experiments report improved SR correspondence and state-of-the-art performance on three benchmarks.
Problem
Stereo image pairs provide useful additional information for SR, but varying disparities make their correspondence difficult to incorporate.
Method
PASSRnet extracts multi-scale features and uses parallax attention with a global receptive field along epipolar lines to capture stereo correspondence, supported by the Flickr1024 training dataset.
Results
PASSRnet achieves state-of-the-art performance on the Middlebury, KITTI 2012, and KITTI 2015 datasets.
Takeaways & Limitations
Parallax attention captures stereo correspondence to improve SR with a small computational and memory cost.
Takeaways & Limitations
Video SR methods cannot be directly applied when stereo disparities exceed their receptive fields, and accurate correspondence assumes valid stereo matching relationships.
Abstract
from arXiv · showhide
Stereo image pairs can be used to improve the performance of super-resolution (SR) since additional information is provided from a second viewpoint. However, it is challenging to incorporate this information for SR since disparities between stereo images vary significantly. In this paper, we propose a parallax-attention stereo superresolution network (PASSRnet) to integrate the information from a stereo image pair for SR. Specifically, we introduce a parallax-attention mechanism with a global receptive field along the epipolar line to handle different stereo images with large disparity variations. We also propose a new and the largest dataset for stereo image SR (namely, Flickr1024). Extensive experiments demonstrate that the parallax-attention mechanism can capture correspondence between stereo images to improve SR performance with a small computational and memory cost. Comparative results show that our PASSRnet achieves the state-of-the-art performance on the Middlebury, KITTI 2012 and KITTI 2015 datasets.
1. Introduction
Stereo images offer additional information for super-resolution, but varying disparities make correspondence difficult to incorporate. PASSRnet addresses this with parallax attention and is evaluated against prior SR methods on established stereo benchmarks.
- Stereo image pairs can improve SR through subpixel shifts, but disparity varies with baseline, focal length, depth, and resolution.
- Video SR methods cannot directly handle stereo images because stereo disparities may exceed their receptive fields.
- Existing stereo matching approaches model long-range correspondence with 3D or 4D cost volumes, but 4D volumes impose high computational and memory costs.
- StereoSR uses shifted right-image stacks, but its fixed maximum disparity of 64 limits flexibility across sensors and scenes.
- PASSRnet extracts multi-scale features and computes feature similarities across all possible right-image disparities to form a global parallax-attention map.
- PASSRnet introduces a global-receptive-field parallax-attention mechanism, the Flickr1024 dataset, and reported state-of-the-art performance against single-image and stereo SR methods.
2. Related Work
Prior work models long-range stereo correspondence with cost volumes or attention, but fixed disparity limits and unnecessary search or aggregation create efficiency and flexibility challenges. PASSRnet uses epipolar-constrained parallax attention to address these issues.
- Long-range dependency is necessary for stereo SR because stereo correspondence is non-local, unlike the mainly local correspondence used in video SR.
- Cost Volume: 4D cost volumes model stereo dependency but impose high computational and memory burdens, while 3D volumes are more efficient but fixed in maximum disparity.
- PASSRnet’s architecture overview is presented in Figure 2, while the network takes a stereo pair as input and super-resolves the left image.
- Self-attention Mechanisms: Self-attention captures long-range dependency by weighting positions, but PASSRnet instead develops parallax attention for stereo correspondence.
- Self-attention Mechanisms: Parallax attention restricts search to epipolar lines, focuses on the most similar feature, and is presented as more flexible and efficient than cost volumes.
3. Method
PASSRnet extracts multi-scale stereo features, uses parallax attention to capture correspondence along epipolar lines, and fuses correspondence-aware features with valid masks for super-resolution. Its training combines SR supervision with photometric, smoothness, and cycle-consistency losses.
- 3. Method: PASSRnet takes a stereo image pair and super-resolves the left image using a residual ASPP module, PAM, and feature fusion.The architecture is described in Fig. 2 and Table 1.
- 3.1. Residual Atrous Spatial Pyramid Pooling (ASPP) Module: The residual ASPP module enlarges the receptive field and extracts hierarchical features with dense sampling rates and multiple scales.It alternates residual ASPP blocks with residual blocks; each ASPP block combines dilated convolutions with rates 1, 4, and 8.
- 3.2. Parallax-attention Module (PAM): PAM computes feature similarities between each left-image pixel and all possible disparities in the right image to generate global parallax-attention maps.The attention map weights right-image features, whose weighted sum is integrated with local left-image features.
- 3.2. Parallax-attention Module (PAM): Unlike standard self-attention, parallax attention focuses on the most similar feature along each epipolar line, producing sparse maps that encode correspondence and disparity information.The generated attention patterns are reported to resemble ground-truth maps.
- 3.2. Parallax-attention Module (PAM): Left-right and cycle consistency regularize correspondence learning, while valid masks exclude occluded regions from consistency enforcement and guide feature fusion.Occluded pixels receive low attention weights; the valid-mask threshold τ is empirically set to 0.1.
- 3.3. Losses: PASSRnet is trained with an SR loss plus photometric, smoothness, and cycle losses to exploit stereo correspondence.The photometric loss is applied using valid-mask regions, and the SR loss is based on mean squared error.
4. Experimental Results
Experiments evaluate PASSRnet’s training setup, architectural components, losses, and comparisons with prior stereo SR methods across benchmark datasets. The results show that stereo input, multi-scale features, parallax attention, combined losses, and flexible disparity handling each contribute to performance or efficiency.
- Experimental setup: Experiments use Middlebury, KITTI 2012, and KITTI 2015 data, with Flickr1024 and Middlebury images used for training and benchmark images for testing.Training includes 60 downsampled Middlebury images and 1024 Flickr stereo images; testing uses 5 Middlebury, 20 KITTI 2012, and 20 KITTI 2015 images.
- Single Input vs. Stereo Input: A 0.16 dB PSNR decrease, from 25.43 to 25.27, occurs when PASSRnet is trained with single images instead of stereo pairs.Replicated left-image pairs reach 25.29 dB, comparable to single-image training, indicating that the extra stereo information matters rather than duplicated inputs alone.
- Residual ASPP Module: Removing residual connections lowers PSNR from 25.43 dB to 25.40 dB, while replacing atrous convolutions lowers it from 25.43 dB to 25.38 dB.Residual connections support multi-scale feature extraction, and atrous convolutions provide a larger receptive field for context.
- Parallax-attention Module: Removing PAM lowers PSNR from 25.43 dB to 25.28 dB because plain CNNs struggle to integrate features separated by long spatial distances between stereo images.The ablation links PAM to effective integration of left-right features with long-range dependencies.
- Losses: Training with only SR loss reduces PSNR from 25.43 to 25.35, whereas adding photometric, smoothness, and cycle losses yields the best PSNR=25.43 dB and SSIM=0.776.The additional losses encourage reliable and consistent correspondence and help PAM focus on the most similar feature.
- Comparison to State-of-the-arts: PASSRnet is more efficient than StereoSR on low-resolution images and achieves a large PSNR margin on high-resolution images.StereoSR incurs unnecessary padding below 64 horizontal pixels and its fixed maximum disparity limits longer-range correspondence at high resolution.
5. Conclusion
The paper concludes that PASSRnet uses global parallax attention to incorporate stereo correspondence for SR across large disparity variations. It also introduces Flickr1024, and reports state-of-the-art performance on three benchmark datasets.
- Conclusion: PASSRnet incorporates stereo correspondence through a parallax-attention mechanism with a global receptive field for large disparity variations.The method is presented as effectively capturing stereo correspondence to improve SR performance.
- Conclusion: Flickr1024 is introduced as a new and largest dataset for stereo image super-resolution.The dataset is presented as an additional contribution alongside PASSRnet.
- Conclusion: Comparisons with recent single-image and stereo-image SR methods show state-of-the-art performance on Middlebury, KITTI 2012, and KITTI 2015.This is the paper’s reported benchmark-level conclusion.