PaperlayerView exact revision on arXiv

Source-linked AI summary

Parallax Attention for Unsupervised Stereo Correspondence Learning

Longguang Wang, Yulan Guo, Yingqian Wang, Zhengfa Liang, Zaiping Lin, Jungang Yang, Wei An

arXiv:2009.08250v2cs.CV

TL;DR

Fixed maximum disparities in cost-volume methods limit stereo correspondence across image pairs with varied camera geometry. The paper proposes PAM, applies it in PASMnet and PASSRnet, and reports effective unsupervised correspondence learning under large disparity variations with state-of-the-art performance.

  • Problem

    Fixed maximum disparities in cost-volume techniques hinder handling stereo image pairs with large disparity variations caused by different baselines, focal lengths, and resolutions.

  • Method

    PAM integrates epipolar constraints with attention to calculate feature similarities along epipolar lines, and is used in PASMnet and PASSRnet for stereo matching and stereo image super-resolution.

  • Results

    PASMnet and PASSRnet achieve state-of-the-art performance while PAM learns stereo correspondence under large disparity variations in an unsupervised manner.

  • Takeaways & Limitations

    PAM provides a compact mechanism applicable to stereo matching and stereo image super-resolution without explicit disparity calculation.

  • Takeaways & Limitations

    PASSRnet’s shared features can suffer training conflict across tasks, motivating a transition block to alleviate the problem.

Abstract

from arXiv · show

Stereo image pairs encode 3D scene cues into stereo correspondences between the left and right images. To exploit 3D cues within stereo images, recent CNN based methods commonly use cost volume techniques to capture stereo correspondence over large disparities. However, since disparities can vary significantly for stereo cameras with different baselines, focal lengths and resolutions, the fixed maximum disparity used in cost volume techniques hinders them to handle different stereo image pairs with large disparity variations. In this paper, we propose a generic parallax-attention mechanism (PAM) to capture stereo correspondence regardless of disparity variations. Our PAM integrates epipolar constraints with attention mechanism to calculate feature similarities along the epipolar line to capture stereo correspondence. Based on our PAM, we propose a parallax-attention stereo matching network (PASMnet) and a parallax-attention stereo image super-resolution network (PASSRnet) for stereo matching and stereo image super-resolution tasks. Moreover, we introduce a new and large-scale dataset named Flickr1024 for stereo image super-resolution. Experimental results show that our PAM is generic and can effectively learn stereo correspondence under large disparity variations in an unsupervised manner. Comparative results show that our PASMnet and PASSRnet achieve the state-of-the-art performance.

1 INTRODUCTION

The paper addresses fixed-disparity and computational limitations in cost-volume stereo correspondence by introducing parallax attention, which supports large disparity variations and multiple stereo tasks.

  • Motivation: Fixed maximum disparity limits cost-volume methods when stereo cameras produce large disparity variations.Cost volumes also impose high computational and memory costs, especially in 4D form.
  • Parallax-Attention Mechanism: PAM computes feature similarities along epipolar lines and regularizes attention maps to identify stereo correspondences without a manually fixed maximum disparity.It accesses disparities through matrix multiplication rather than shift operations.
  • Parallax-Attention Mechanism: PAM directly regularizes parallax-attention maps, avoiding ambiguity from disparity regression and enabling accurate, consistent correspondence learning.The paper contrasts this with potentially unreasonable cost distributions produced by regressing disparities from matching costs.
  • Contributions: PAM is applied to stereo matching and stereo image super-resolution, with the resulting networks reported to achieve state-of-the-art performance in both tasks.The paper names these networks PASMnet and PASSRnet.
  • Contributions: Flickr1024 provides 1024 high-quality stereo image pairs covering various scenes for training stereo image super-resolution networks.The dataset is introduced as a new contribution for stereo image SR.

2 RELATED WORK

Prior work uses CNNs, cost volumes, and attention to model stereo correspondence, while unsupervised methods address limited dense depth labels; this paper adapts attention to epipolar stereo structure.

  • Unsupervised Stereo Matching: Unsupervised stereo matching methods use cues such as left-right consistency, semantics, and occlusion interactions because dense ground-truth depth is difficult to collect.The reviewed methods include confidence-map, semantic-loss, and occlusion-aware approaches.
  • Supervised Stereo Matching: CNN-based stereo methods use 3D or 4D cost volumes to capture large disparities, but fixed disparity limits and high 4D costs remain problems.These methods aggregate matching costs with correlation, concatenation, or 3D convolutions.
  • Stereo Image Super-Resolution: Stereo image super-resolution uses a second viewpoint to improve reconstruction, with prior work combining super-resolution and disparity estimation or using parallax priors.This paper focuses on reconstructing a high-resolution left image from a low-resolution stereo pair.
  • Attention Mechanism: Self-attention captures correlations between arbitrary image positions, motivating parallax attention for stereo correspondence.Parallax attention restricts each left-image pixel's attention to disparities along its epipolar line and focuses on the most similar feature.
  • Attention Mechanism: PAM combines epipolar constraints with attention to capture stereo correspondence across disparity variations.The mechanism is presented as a stereo-specific adaptation of self-attention.

3 PARALLAX-ATTENTION MECHANISM (PAM)

PAM uses epipolar-constrained attention to compute stereo feature similarities across all positions on corresponding scanlines, while consistency constraints and valid masks regularize correspondence learning.

  • 3.1 Overview: PAM reshapes features and uses geometry-aware matrix multiplication to correlate each pixel with all positions along the corresponding epipolar line.The resulting parallax-attention map encodes feature correlations across possible disparities without shift-based matching.
  • 3.1 Overview: Unlike cost volumes, PAM considers all disparities without manually fixing a maximum, allowing correspondence learning under large disparity variations.Feature similarities and attention regularization focus the map on the most similar disparity.
  • 3.1.1 Overview: Parallax-attention maps represent scanline correspondence and can encode occlusions as regions that receive no focused position.The toy example uses 30×30 maps for 1×30 corresponding image rows.
  • 3.1.2 Toy Example: PAM handles sub-pixel disparities in practice, but softmax may assign incorrect attention to occluded regions, which valid masks exclude.The toy example itself considers only integer disparities.
  • 3.2 Left-Right Consistency and Cycle Consistency: Left-right and cycle consistency regularize attention maps, while occlusion detection restricts consistency losses to valid regions.Geometry-aware multiplication composes attention maps and feature slices to reconstruct corresponding representations.
  • 3.3 Valid Mask: Valid masks identify occluded pixels using low attention weights and a threshold τ empirically set to 0.1, with morphological operations refining the masks.The implementation uses geometry encoded in the parallax-attention maps to detect discarded regions.

4 PAM FOR UNSUPERVISED STEREO MATCHING

Stereo matching must handle large disparity variation because fixed maximum disparities constrain cost-volume methods. PAM instead correlates arbitrary positions along epipolar lines for unsupervised matching across such variation.

  • 4 PAM FOR UNSUPERVISED STEREO MATCHING: PAM avoids fixed maximum disparity by efficiently computing feature correlations between any two positions along an epipolar line.This design supports unsupervised stereo matching when disparities vary substantially.

4.1 Network Architecture

PASMnet extracts stereo features, applies cascaded parallax-attention blocks for coarse-to-fine matching-cost estimation, and refines the resulting disparity with an hourglass network.

  • 4.1.1 Overview: PASMnet extracts features, processes them through cascaded parallax-attention modules, predicts an initial disparity, and applies refinement to produce the output disparity.The architecture combines feature extraction, matching-cost estimation, output, and disparity-refinement stages.
  • 4.1.2 Cascaded Parallax-Attention Module: The cascaded module uses three stages with four parallax-attention blocks per stage to regress matching costs coarse-to-fine.Features and matching costs are progressively upsampled, concatenated, and passed to later stages.
  • 4.1.2 Cascaded Parallax-Attention Module: Unlike explicit 3D-convolutional cost aggregation, PASMnet implicitly aggregates local matching costs by cascading parallax-attention blocks with residual connections.Each block computes attention-based matching costs from left and right features, while residual paths combine information across depths.
  • 4.1.3 Output Module: The output module converts final matching costs into parallax-attention maps and regresses disparity as a weighted sum of disparity candidates.Because all candidates are weighted, the network does not require a manually fixed maximum disparity.
  • 4.1.3 Output Module: Occluded disparities are excluded because occluded pixels lack correspondences, and partial convolution fills the resulting invalid regions.This handling follows the limitation that correspondence cannot be reliably estimated in occlusions.
  • 4.1.4 Disparity Refinement: A refinement hourglass uses left-image features to add structural information, predicting a residual disparity and confidence map before producing the refined disparity.The initial disparity is concatenated with left-image features as refinement input.

4.2 Losses

The network combines photometric, smoothness, and multi-scale parallax-attention losses to learn accurate and consistent stereo correspondence without groundtruth disparities.

  • Photometric Loss: The photometric loss combines mean absolute error and structural similarity terms over valid, non-occluded pixels after warping the right image with refined disparity.
  • Smoothness Loss: The smoothness loss encourages local disparity smoothness using image-gradient-aware regularization.
  • PAM Loss: Three multi-scale PAM losses regularize photometric consistency, smoothness, and cycle consistency directly on parallax-attention maps.
  • Groundtruth disparities are not used in the overall loss, so the network is trained in an unsupervised manner.

4.3 Experimental Results

Experiments evaluate PASMnet on synthetic and real-world stereo datasets, showing benefits from its architecture, PAM regularization, and cost-volume-free correspondence modeling.

  • Ablation Study: Using four cascaded blocks improves 1-pixel/3-pixel error rates from 19.32/16.09 with one block to 18.99/15.91, while coarse-to-fine regression also improves performance.
  • PAM vs. Cost Volume: 23.5×/2.1× reductions in FLOPs/memory cost are achieved by the parallax-attention block over 3D convolution when C = 32.
  • PAM vs. Cost Volume: Compared with 4D cost volume formation, PAM achieves much better SceneFlow performance with comparable model size and inference time, while using half the memory.
  • Losses: 4.54/18.99/15.91 EPE/1-pixel error rate/3-pixel error rate worsen to 6.07/20.78/17.11 with only photometric loss, while adding smoothness and PAM losses progressively improves performance.
  • Losses: Using PAM loss produces matching-cost and parallax-attention distributions with sharper peaks at the groundtruth disparity.
  • Disparity Variations: The cost-volume technique cannot capture correspondences with disparities larger than its fixed maximum disparity, whereas PAM is evaluated across varying resolutions and depths.

5 PAM FOR STEREO IMAGE SUPER-RESOLUTION

PAM is extended to stereo image super-resolution through PASSRnet, which aggregates stereo features while using contextual feature extraction and a transition block for multi-task learning.

  • PASSRnet applies PAM to aggregate correspondence-guided features from stereo images for stereo image super-resolution.
  • PASSRnet extracts features with residual blocks and residual ASPP, applies parallax attention, and reconstructs super-resolved images with residual blocks and sub-pixel convolution.
  • Parallax-Attention Module: The parallax-attention module adapts left and right features, generates bidirectional attention maps and a valid mask, then fuses the resulting features.
  • Parallax-Attention Module: PASSRnet is a multi-task network for stereo correspondence and super-resolution, and a transition block is used to alleviate training conflict from shared features.

5.2 Losses

PASSRnet combines supervised super-resolution learning with unsupervised stereo-correspondence learning through separate loss terms.

  • Stereo correspondence is learned unsupervised with LPAM, whereas super-resolution is learned supervised with LSR.
  • The overall objective combines the SR loss and PAM loss as L = LSR + λLPAM, with λ empirically set to 0.005.

5.3 The Flickr1024 Dataset

Flickr1024 is a large, diverse stereo image super-resolution dataset assembled from Flickr stereograms, addressing the limited scene coverage of existing stereo datasets.

  • Dataset construction: Flickr1024 contains 1024 RGB stereo-image pairs covering landscapes, urban scenes, people, and man-made objects.The pairs were collected from Flickr using stereography-related tags and originated as cross-eye stereograms captured with dual lenses or cameras.
  • Dataset construction: The dataset was prepared by splitting stereograms into left-right images, removing black margins, correcting cross-eye ordering, and roughly restoring disparity alignment.The source stereograms contain both positive and negative disparities because photographers shifted images toward a common focus plane.
  • Dataset comparison: Flickr1024 is at least five times larger than the compared datasets and has nearly twice the pixels per image of KITTI 2012 and KITTI 2015.Middlebury has higher pixels per image but far fewer image pairs.
  • Dataset comparison: Flickr1024 achieves comparable or better CNNIQA and SR-metric values than the compared datasets, indicating high image quality.The comparison includes Middlebury, KITTI 2012, and KITTI 2015 training sets.

5.4 Experimental Results

Experiments evaluate PASSRnet’s stereo-information use, module choices, efficiency against cost volumes, and robustness to disparity variation. PASSRnet achieves stronger super-resolution results than competing methods while remaining effective across changing baselines and disparities.

  • Single Input vs. Stereo Input: 25.43 dB PSNR with stereo input versus 25.27 dB with single images and 25.29 dB with replicated left-image pairs.The comparison supports the contribution of complementary stereo information.
  • Module Ablations: Removing parallax attention lowers PSNR from 25.43 dB to 25.28 dB, while removing the transition block lowers it to 25.36 dB.The ablations associate performance gains with stereo correspondence aggregation and task-specific feature learning.
  • PAM vs. Cost Volume: 25.43 dB versus 25.23 dB for cost-volume formation, with running time reduced by 1.5 times and memory consumption reported as half that of the cost-volume technique.The comparison uses a 4D cost volume followed by two 3D convolutional layers for 4× SR on KITTI 2015.
  • Comparison with Existing Methods: 1.04 dB higher PSNR than DRRN and 1.00 dB higher than StereoSR on Middlebury for 2× SR.The comparison excludes substantially larger models and reports PASSRnet’s advantage over the best included single-image and stereo baselines.
  • Flexibility to Disparity Variations: 1.30 dB improvement over StereoSR for large-baseline stereo pairs, compared with 1.14 dB for short baselines, while performance drops 0.07 versus 0.23 for StereoSR.These results demonstrate greater flexibility and robustness to large disparity variations.

6 CONCLUSION

The paper presents PAM as a generic mechanism for stereo correspondence under large disparity variations and applies it to stereo matching and stereo image super-resolution. The resulting PASMnet and PASSRnet are reported to achieve state-of-the-art performance, alongside the Flickr1024 dataset introduction.

  • Contributions: PAM calculates feature similarities along epipolar lines and captures stereo correspondence under large disparity variations.The mechanism integrates epipolar constraints with attention and does not require explicit disparity calculation.
  • Applications: PASMnet and PASSRnet apply PAM to stereo matching and stereo image super-resolution tasks.The paper describes PAM as compact and applicable to variant stereo tasks.
  • Dataset and Results: The authors introduce Flickr1024 as a new large-scale dataset for stereo image super-resolution and report state-of-the-art performance for both proposed networks.
Loading 2009.08250v2…