Source-linked AI summary
Selective-Stereo: Adaptive Frequency Information Selection for Stereo Matching
Xianqi Wang, Gangwei Xu, Hao Jia, Xin Yang
TL;DR
Iterative stereo methods can lose high-frequency details and overlook other frequency information because recurrent units use fixed receptive fields and process noisy cost-volume information. The paper introduces SRU and CSA to adaptively fuse multi-frequency hidden disparity information, and Selective-Stereo achieves leading results across major stereo benchmarks. Its receptive fields remain predefined, while larger branches or kernels increase memory and time costs.
Problem
Iterative stereo methods can lose crucial hidden information and high-frequency details because noisy cost volumes and fixed receptive fields limit multi-frequency processing.
Method
SRU uses multiple GRU branches with distinct kernel sizes, while CSA generates context-based attention maps to adaptively fuse their multi-frequency hidden disparity information.
Results
Selective-IGEV achieves a state-of-the-art EPE of 0.44 on Scene Flow and ranks first on KITTI, ETH3D, and Middlebury leaderboards.
Takeaways & Limitations
Selective-Stereo consistently improves iterative stereo networks without a significant increase in parameters and time.
Takeaways & Limitations
SRU receptive fields remain limited by predefined values, while adding branches or larger kernels increases memory and time costs.
Abstract
from arXiv · showhide
Stereo matching methods based on iterative optimization, like RAFT-Stereo and IGEV-Stereo, have evolved into a cornerstone in the field of stereo matching. However, these methods struggle to simultaneously capture high-frequency information in edges and low-frequency information in smooth regions due to the fixed receptive field. As a result, they tend to lose details, blur edges, and produce false matches in textureless areas. In this paper, we propose Selective Recurrent Unit (SRU), a novel iterative update operator for stereo matching. The SRU module can adaptively fuse hidden disparity information at multiple frequencies for edge and smooth regions. To perform adaptive fusion, we introduce a new Contextual Spatial Attention (CSA) module to generate attention maps as fusion weights. The SRU empowers the network to aggregate hidden disparity information across multiple frequencies, mitigating the risk of vital hidden disparity information loss during iterative processes. To verify SRU's universality, we apply it to representative iterative stereo matching methods, collectively referred to as Selective-Stereo. Our Selective-Stereo ranks $1^{st}$ on KITTI 2012, KITTI 2015, ETH3D, and Middlebury leaderboards among all published methods. Code is available at https://github.com/Windsrain/Selective-Stereo.
1. Introduction
Iterative stereo methods avoid costly full cost-volume aggregation but can lose high-frequency details and overlook information at other frequencies because recurrent updates face noisy inputs and fixed receptive fields. Selective-Stereo addresses these limitations with SRU and CSA, improving benchmark performance across iterative stereo networks.
- Iterative stereo methods: Iterative stereo methods construct all-pairs cost volumes, retrieve local cost volumes, and recurrently update disparity without predefining a disparity range.They also avoid aggregating cost volumes with many redundant convolutions.
- Limitations: Fixed receptive fields cause iterative methods to focus on one frequency while ignoring detailed, edge, textureless, or other frequency information.Noisy all-pairs cost volumes and repeated updates can also cause crucial hidden information loss.
- Proposed method: SRU uses multiple GRU branches with distinct kernel sizes to fuse hidden disparity information across receptive fields and frequencies while reducing local-cost-volume noise.The fused hidden information is passed to the next iteration.
- Proposed method: CSA generates context-based attention maps that weight frequency-specific branch information for adaptive fusion in different image regions.The method gives region-dependent emphasis to information captured by different kernel sizes.
- Results: Selective-Stereo consistently improves its host iterative networks without introducing a significant increase in parameters and time.The approach is presented as applicable across several iterative stereo matching methods.
- Results: Selective-IGEV achieves a state-of-the-art EPE of 0.44 on Scene Flow, while Selective-RAFT reaches 0.47 and Selective-IGEV ranks first on KITTI, ETH3D, and Middlebury leaderboards.Selective-RAFT also surpasses RAFT-Stereo by a large margin on KITTI and remains competitive with state-of-the-art methods.
2. Related Work
Stereo matching research includes aggregation-based methods that regularize cost volumes and iterative methods that update disparity from correlation features. Selective-Stereo builds on this iterative lineage while introducing frequency-aware recurrent processing for stereo updates.
- Aggregation-based methods: Aggregation-based methods construct and regularize cost volumes, commonly using 3D CNNs to filter geometry information before disparity regression.Their cost aggregation can require many convolutions and incur high computational costs.
- Selective-Stereo: Selective-Stereo’s overview uses CSA-derived attention maps to guide SRUs during iterative disparity updates from local cost volumes retrieved from a correlation pyramid.This places the proposed method within the iterative stereo architecture family.
- Iterative-based methods: RAFT-Stereo uses a high-resolution all-pairs cost-volume pyramid, local correlation features, GRU-based iterative disparity updates, and multi-level GRUs to expand the receptive field.IGEV-Stereo adds lightweight cost aggregation before iterations, while CREStereo uses a coarse-to-fine hierarchical design.
- Frequency information in vision: Frequency-aware vision methods separate or process feature information by frequency, including octave convolution and frequency-domain learning approaches.These works motivate treating frequency components explicitly in visual representations.
3. Method
Selective-Stereo combines multi-level feature and cost-volume processing with CSA-guided SRUs that adaptively fuse hidden disparity information across frequencies and resolutions. The design uses smaller and larger receptive fields to preserve fine details while incorporating broader context during iterative disparity updates.
- Overall architecture: Selective-Stereo is designed as a plug-in architecture for iterative stereo networks, illustrated using Selective-RAFT while preserving RAFT-Stereo’s feature extraction network.The feature and context networks produce features and context representations at multiple resolutions for subsequent cost-volume construction and recurrent updating.
- Cost volume construction: The method constructs an all-pairs correlation volume and a four-level correlation pyramid using 1D average pooling along the last dimension.The pyramid supplies multi-scale correlation information for iterative disparity estimation.
- Contextual Spatial Attention: CSA generates channel and spatial attention maps from context information to guide fusion across receptive fields and frequencies.CAE produces channel weights, while SAE produces spatial weights emphasizing regions that need high-frequency information.
- Selective Recurrent Unit: SRU fuses GRU-derived hidden information from smaller and larger kernels using CSA attention, assigning fine-detail processing to smaller kernels and broader context to larger kernels.The smaller-kernel branch targets edges and thin objects, while the larger-kernel branch receives the complementary attention.
- Multi-level update: In the multi-level update structure, adjacent-resolution SRUs exchange hidden information, while 1/4-resolution SRUs additionally receive disparity and local cost volume to generate disparity residuals.The resulting disparity is upsampled to full resolution; multi-level SRUs provide dynamic receptive fields rather than only three fixed fields.
- Training objective: The network is trained with increasing weights on the L1 distance between each predicted disparity and ground-truth disparity, using γ = 0.9 across N iterations.The loss supervises all iterative disparity predictions rather than only the final output.
4. Experiments
Experiments show that Selective-Stereo improves iterative stereo matching across ablations, iteration counts, and multiple benchmarks, with strong results on Scene Flow, KITTI, ETH3D, and Middlebury.
- Effectiveness of proposed modules: The full Selective-RAFT model achieves the best ablation performance with CSA and only a 4% increase in parameters over the baseline.Replacing GRUs with SRUs improves accuracy, while normally weighted CSA performs better than inverted attention weights.
- Qualitative comparisons: Qualitative results show Selective-IGEV outperforming IGEV-Stereo in detailed, textureless, weak-texture, and thin-object regions.On Middlebury, CSA attention maps are reported to separate regions requiring different frequency information.
- Universality of proposed modules: All evaluated iterative methods gain significant Scene Flow EPE improvements with only a slight parameter increase after replacing their recurrent units with SRUs.The CSA module generates network-specific attention patterns, adapting kernel weighting to noisy, aggregated, or over-smoothed cost volumes.
- Number of iterations: Selective-Stereo matches RAFT-Stereo performance with only 8 iterations and slightly improves IGEV-Stereo with fewer iterations.The authors attribute this to secondary filtering that reduces noisy information from the initial cost volume.
- Size of convolution kernels: The default 1 × 1 and 3 × 3 kernel combination provides competitive performance while reducing computational costs.Kernel sizes are evaluated on Scene Flow before selecting this configuration.
- Comparisons with state-of-the-art: Selective-IGEV achieves a new state-of-the-art Scene Flow EPE of 0.44 and ranks first among published methods across KITTI, ETH3D, and Middlebury evaluations.Selective-RAFT also improves over its baseline on KITTI 2012, KITTI 2015, and ETH3D, while Selective-IGEV ranks first on the reported KITTI metrics.
5. Conclusion
Selective-Stereo adaptively fuses information at different frequencies for edge and smooth regions using CSA attention maps. The method ranks first on KITTI, ETH3D, and Middlebury in almost all metrics among published methods, while retaining stated limitations in receptive-field flexibility and computational cost.
- Selective-Stereo uses CSA and SRU to adaptively fuse information at different frequencies for edge and smooth regions.CSA extracts attention maps that guide the fusion.
- The method ranks 1st on KITTI, ETH3D, and Middlebury in almost all metrics among published methods.
- The SRU receptive field remains limited by predefined values, constraining adaptive frequency selection.
- Adding branches or enlarging convolutional kernels increases memory and time costs.
Supplementary Material
Table 9 reports quantitative evaluation on KITTI 2012 reflective regions, which are characterized as ill-posed regions.
- Table 9 evaluates stereo matching quantitatively on KITTI 2012 reflective regions, identified as ill-posed regions.
6. Results on Ill-posed Regions
Selective-Stereo is evaluated on KITTI 2012 reflective regions to assess handling of ill-posed areas. Selective-RAFT outperforms RAFT-Stereo by almost 20%, while Selective-IGEV ranks first among published methods.
- Selective-RAFT outperforms RAFT-Stereo by almost 20% on KITTI 2012 reflective regions.
- Selective-IGEV ranks 1st among all published methods on the reflective-region evaluation.
- Large kernels capture global low-frequency information for local ambiguities, while small kernels preserve detailed structures through local high-frequency information.
- In reflective regions, the model increases large-kernel weights while continuing to fuse small-kernel information.