Source-linked AI summary
Revisiting Stereo Depth Estimation From a Sequence-to-Sequence Perspective with Transformers
Zhaoshuo Li, Xingtong Liu, Nathan Drenkow, Andy Ding, Francis X. Creighton, Russell H. Taylor, Mathias Unberath
TL;DR
Stereo depth estimation needs reliable pixel correspondence despite fixed disparity limits, occlusions, and ambiguous matches. STTR reframes it as sequence-to-sequence matching with dense Transformer-based correspondence and optimal transport, reporting competitive benchmark performance and cross-domain generalization without refinement.
Problem
Cost-volume stereo methods impose fixed disparity ranges, while many learning-based approaches omit explicit occlusion handling and uniqueness constraints.
Method
STTR uses Transformer self- and cross-attention with entropy-regularized optimal transport to densely match pixels and support soft uniqueness and occlusion assignment.
Results
STTR performs on par with prior work below 192 disparity, outperforms prior work in unconstrained settings, and generalizes across domains without refinement.
Takeaways & Limitations
The sequence-to-sequence formulation avoids a pre-specified disparity range, explicitly handles occlusion, and imposes match uniqueness.
Takeaways & Limitations
Real-world test sets are small, performance differences are often slight, and KITTI omits metrics for unlimited disparity and occlusion detection.
Abstract
from arXiv · showhide
Stereo depth estimation relies on optimal correspondence matching between pixels on epipolar lines in the left and right images to infer depth. In this work, we revisit the problem from a sequence-to-sequence correspondence perspective to replace cost volume construction with dense pixel matching using position information and attention. This approach, named STereo TRansformer (STTR), has several advantages: It 1) relaxes the limitation of a fixed disparity range, 2) identifies occluded regions and provides confidence estimates, and 3) imposes uniqueness constraints during the matching process. We report promising results on both synthetic and real-world datasets and demonstrate that STTR generalizes across different domains, even without fine-tuning.
1. Introduction
Stereo depth estimation must match corresponding pixels while handling wide disparities, occlusions, and uniqueness constraints. STTR revisits matching as dense sequence-to-sequence correspondence using Transformer attention and position information instead of a fixed-disparity cost volume.
- Motivation: Stereo methods often impose a manually specified maximum disparity, limiting recognition of close objects across camera setups.Typical approaches use a maximum of 192 px, although disparities can theoretically span the image width.
- Motivation: Learning-based approaches often omit explicit occlusion handling and matching uniqueness, despite occluded pixels lacking valid disparities.Confidence estimates would support downstream registration and scene-understanding algorithms.
- Approach: STTR replaces fixed-disparity cost-volume construction with dense pixel-wise correlation along epipolar lines using a Transformer.The model uses self- and cross-attention to capture long-range associations between feature descriptors.
- Results: The proposed network reports competitive synthetic and real-world benchmark performance and generalizes from synthetic training to other domains without refinement.The paper presents STereo TRansformer as an end-to-end architecture for sequence-to-sequence stereo matching.
- Technical advances: STTR combines relative pixel-distance encoding and customized attention to produce discriminative features for ambiguous matching.The implementation includes alternating self- and cross-attention and a memory-feasible design for conventional hardware.
2. Related Work
Prior stereo systems use feature matching and cost aggregation through dynamic programming, correlation, or learned 3D cost volumes. STTR instead exploits epipolar sequence structure, attention, dense matching, and explicit uniqueness constraints.
- Prior approaches: Traditional stereo estimation matches features and aggregates costs using dynamic programming horizontally or multidirectionally.These methods commonly use pixel intensities as matching criteria.
- Prior approaches: Learning-based methods use patch or pixel feature correlations, learned feature extractors, and cross-scale information aggregation.These approaches include Markov Random Field refinement and descriptor-based similarity computation.
- Cost-volume methods: Many contemporary systems build cost volumes with discretized disparities and 3D convolutions, creating memory and maximum-disparity constraints.Disparities beyond the predefined range cannot be inferred correctly by these approaches.
- Research gap: Prior work generally does not exploit stereo matching's sequential and geometric properties, explicitly handle occlusion, or impose uniqueness during matching.These omissions can limit cross-scene generalization and produce matching inconsistencies.
- STTR: STTR uses a CNN feature extractor followed by a Transformer that captures long-range pixel associations and exploits stereo geometry.It alternates self- and cross-attention, incorporates context and position, and matches densely along epipolar lines before cross-line aggregation.
3. The Stereo Transformer Architecture
STTR uses alternating self- and cross-attention with relative positional information to perform dense epipolar-line matching, then applies optimal transport and context-aware refinement. Its design avoids a fixed maximum disparity while enforcing soft uniqueness and producing disparity and occlusion estimates.
- Transformer Architecture: The architecture alternates self-attention within each image and bidirectional cross-attention between corresponding epipolar lines.The final cross-attention layer estimates raw disparity and adds optimal transport plus an attention mask.
- Attention: Multi-head attention projects feature descriptors into separate query, key, and value representations so heads can capture different similarities.The feature extractor produces descriptors at the input resolution, and each head uses a channel subset.
- Relative Positional Encoding: Relative positional encoding makes attention depend on content similarity and relative pixel distance while preserving shift-invariance.The position-position term is removed because disparity fundamentally depends on image content, and an efficient implementation reduces quadratic distance computation to linear cost.
- Optimal Transport: Entropy-regularized optimal transport converts attention-derived matching costs into differentiable soft assignments that impose uniqueness and accommodate occlusions.Uniform marginals make the transport solution suitable for assignment, while learnable dustbins represent unmatched pixels.
- Raw Disparity and Occlusion Regression: STTR estimates disparity from a most-probable match using a renormalized 3 px neighborhood and derives confidence from the same local matching probabilities.A context adjustment layer then uses convolutions for occlusion refinement and residual blocks with long skips for disparity refinement.
- Complexity Analysis: STTR has O(IhIw^2/s^3) memory complexity, providing a trade-off with no maximum disparity and constant memory across disparity ranges for fixed s.Increasing s lowers inference memory while preserving the maximal disparity range, with a slight sacrifice in task performance.
4. Experiments, Results, and Discussion
Experiments evaluate STTR through ablations, disparity-range generalization, attention analysis, cross-domain testing, and benchmark comparisons. Results show benefits from optimal transport and relative positional encoding, strong generalization without fine-tuning, and competitive performance on KITTI.
- Experimental setup: Experiments use Scene Flow for pre-training and ablations, then evaluate generalization on MPI Sintel, KITTI 2015, Middlebury 2014, and SCARED.The protocol includes asymmetric augmentation and evaluates several architectural components.
- Ablation studies: The soft uniqueness constraint improves every metric, with the largest benefit reported for occlusion IOU.It is imposed through the optimal transport layer using interactions among pixels on the same epipolar line.
- Ablation studies: Relative positional encoding improves all three metrics and helps resolve ambiguity in textureless regions.Feature visualizations show edge-parallel structures emerging and propagating when positional encoding is applied across Transformer layers.
- Disparity-range generalization: Training with disparities below 192 px still yields a maximal prediction of 458 px within 1 px error during testing.This result supports generalization beyond the disparity range used for training.
- Attention analysis: Self- and cross-attention shift from global to local context across layers, with late cross-attention reaching a 15 px span.The reported spans decrease from 300 px initially to 114 px for self-attention and 15 px for cross-attention.
- Benchmark results: STTR is comparable to prior work within 192-disparity evaluation, substantially better in unconstrained Scene Flow evaluation, and comparable on KITTI after fine-tuning.Without fine-tuning, STTR generalizes comparably across four datasets and maintains high occlusion IOU, although it does not always achieve the best EPE.
- Shortcomings in challenge design: Small real-world test sets and limited KITTI metrics prevent definitive performance comparisons or a complete assessment of STTR’s central benefits.KITTI reports only disparities below 192 and omits metrics for unlimited disparity and occlusion detection.
5. Conclusion
STTR revisits stereo depth estimation as sequence-to-sequence matching, avoiding fixed disparity ranges while explicitly handling occlusion and uniqueness. Experiments show generalization across domains without fine-tuning, though real-world test sets remain small.
- STTR is an end-to-end architecture that revisits stereo depth estimation from a sequence-to-sequence matching perspective.
- The approach avoids pre-specifying a fixed disparity range, explicitly handles occlusion, and imposes a match uniqueness constraint.
- STTR generalizes to different domains without fine-tuning and achieves promising benchmark results with refinement.
A. Efficient Implementation of Attention with Relative Positional Encoding
The implementation computes relative positional attention efficiently while restricting correspondence searches according to stereo geometry. Transformer features evolve from broad contextual attention toward local, edge-focused matching, with positional encoding helping resolve textureless-region ambiguity.
- Efficient Implementation of Attention with Relative Positional Encoding: Relative positional attention uses a fixed offset pattern whose possible values can be pre-computed and efficiently sliced with increasing offsets.
- Efficient Implementation of Attention with Relative Positional Encoding: The attention mask permits each left-image pixel to attend only to right-image pixels at or left of the same horizontal coordinate after rectification.
- Efficient Implementation of Attention with Relative Positional Encoding: Without positional encoding, textureless regions lack distinct patterns for resolving matching ambiguities, whereas positional encoding progressively propagates edge information into them.
- Efficient Implementation of Attention with Relative Positional Encoding: Attention shifts from global context to local context across layers, with final cross-attention spanning around 15 pixels, or 0.01 of image width.
E. Generalization Mechanism
STTR’s learned feature representations form structured clusters and occupy a shared embedding space across domains. The visualizations and edge-gradient analysis support a mechanism in which texture-related clustering may facilitate matching and generalization.
- E. Generalization Mechanism: STTR feature representations cluster into two regions, with pixel color masks indicating proximity to the cluster centroids.
- E. Generalization Mechanism: Across datasets, the blue cluster generally has larger mean edge-gradients than the red cluster, indicating more texture in the blue cluster except for SCARED.
- E. Generalization Mechanism: Embeddings from different domains remain within the same feature space, and the authors hypothesize that this clustering improves generalization and facilitates matching.
F. Qualitative result of context adjustment layer (CAL)
The context adjustment layer improves disparity predictions by incorporating cross-epipolar-line context. Without CAL, the qualitative result lacks smoothness.
- Without the context adjustment layer, disparity predictions lack smoothness because they lack cross-epipolar-line context.
- The attention module supports changing feature-attention stride at inference, trading performance for faster speed and lower memory without retraining STTR.
H. Lightweight Implementation
A lightweight STTR variant fixes attention stride and reallocates attention dimensions to reduce inference memory and increase speed, with some Scene Flow performance loss.
- Design changes: The lightweight model fixes attention stride at s = 4, removing inference-time stride flexibility and avoiding full-resolution feature maintenance.This reduces memory consumption.
- Design changes: Memory consumption scales with the number of heads Nh, while parameter count scales with NhCh.
- Design changes: Doubling Ch while halving Nh preserves parameter count while reducing memory consumption by half.
- Evaluation: On Scene Flow, lightweight STTR improves inference speed and memory use but loses performance relative to STTR, especially in 3 px Error and EPE.
I. Inference Memory Consumption and Speed
STTR is evaluated for inference memory and speed across disparity ranges and for cross-domain generalization without fine-tuning.
- Inference efficiency: STTR can maintain constant inference memory and speed without manually limiting the disparity range.
- Inference efficiency: The comparison matches STTR's attention stride to PSMNet's feature downsampling rate by setting s = 4.
- Inference efficiency: Prior methods consume more memory and run slower when a larger disparity range is required, whereas disparity ranges exceeding image width are marked N/A.Table 9 reports median memory and speed across 100 runs.
- Cross-domain evaluation: Table 8 evaluates generalization without fine-tuning on MPI Sintel, KITTI 2015, Middlebury 2014, and SCARED after training only on Scene Flow.
J. Dataset Information and Pre-processing
The dataset description covers Scene Flow preprocessing and notes biological stereo vision's ability to infer depth from sparse visual cues under geometric assumptions.
- Dataset preprocessing: Scene Flow's FlyingThings3D Full dataset lacks occlusion information, so it is subsampled using occlusion data from the Disp-Net/FlowNet2.0 dataset.
- Dataset preprocessing: After preprocessing, Scene Flow contains 21,818 training images at 960×540 resolution, with maximum disparity 602 px.
- Dataset preprocessing: The test set contains 4,248 images with maximum disparity 468 px.
- Background: Random-dot stereograms show that humans can perceive stereo depth without meaningful image texture, alongside a piece-wise smoothness prior.
L. Training Time and Number of parameters
STTR training on Scene Flow requires substantial single-GPU time, while its parameter count is reported as low relative to contemporary stereo architectures.
- Training time: Training STTR on Scene Flow takes approximately 120 hours on one Titan RTX GPU with batch size 1 for 15 epochs.Training time varies with batch size and GPU configuration.
- Model size: STTR has the fewest parameters among the compared contemporary architectures except LEAStereo.LEAStereo is optimized using Neural Architecture Search.
- Model size: Table 10 summarizes the number of parameters in contemporary stereo depth estimation architectures.