Source-linked AI summary
Iterative Geometry Encoding Volume for Stereo Matching
Gangwei Xu, Xianqi Wang, Xiaohuan Ding, Xin Yang
TL;DR
Existing stereo methods either lack non-local geometry in efficient iterative updates or incur high computational and memory costs through cost aggregation. IGEV addresses this by combining geometry and context encoding with local matching details, then iteratively refining disparity from an accurate GEV initialization. It ranks 1st on KITTI 2015 and KITTI 2012 Reflective, generalizes across datasets, and extends competitively to MVS on DTU.
Problem
Iterative stereo methods using all-pairs correlations lack non-local geometry and context, making local ambiguities difficult in ill-posed regions, while cost filtering is computationally expensive.
Method
IGEV builds a Geometry Encoding Volume with lightweight 3D regularization, combines it with all-pairs correlations into a combined volume, and iteratively updates disparity with ConvGRUs.
Results
IGEV-Stereo ranks 1st on KITTI 2015 and KITTI 2012 Reflective, achieves 0.47 EPE on Scene Flow, and IGEV-MVS achieves competitive accuracy on DTU.
Takeaways & Limitations
The approach combines non-local geometry and context information with local matching details while retaining efficient iterative refinement and strong cross-dataset generalization.
Takeaways & Limitations
Processing large cost volumes with a 3D CNN can still incur high computational and memory costs for high-resolution images with large disparity ranges.
Abstract
from arXiv · showhide
Recurrent All-Pairs Field Transforms (RAFT) has shown great potentials in matching tasks. However, all-pairs correlations lack non-local geometry knowledge and have difficulties tackling local ambiguities in ill-posed regions. In this paper, we propose Iterative Geometry Encoding Volume (IGEV-Stereo), a new deep network architecture for stereo matching. The proposed IGEV-Stereo builds a combined geometry encoding volume that encodes geometry and context information as well as local matching details, and iteratively indexes it to update the disparity map. To speed up the convergence, we exploit GEV to regress an accurate starting point for ConvGRUs iterations. Our IGEV-Stereo ranks $1^{st}$ on KITTI 2015 and 2012 (Reflective) among all published methods and is the fastest among the top 10 methods. In addition, IGEV-Stereo has strong cross-dataset generalization as well as high inference efficiency. We also extend our IGEV to multi-view stereo (MVS), i.e. IGEV-MVS, which achieves competitive accuracy on DTU benchmark. Code is available at https://github.com/gangweiX/IGEV.
1. Introduction
Stereo methods trade off non-local geometry modeling against computational efficiency: cost filtering is informative but expensive, while iterative correlation methods are efficient yet struggle with ambiguities. IGEV-Stereo combines these advantages through geometry encoding, iterative updates, and an accurate initialization.
- Cost-filtering methods aggregate 4D cost volumes effectively but require substantial 3D convolution, computation, and memory, limiting high-resolution use.
- Iterative methods avoid expensive cost aggregation and repeatedly fetch high-resolution cost information, making them applicable to high-resolution images.
- Without cost aggregation, all-pairs correlations lack non-local geometry and context, hindering disparity estimation in occlusions, textureless regions, and repetitive structures.
- IGEV-Stereo builds a Geometry Encoding Volume with lightweight 3D regularization, combines it with all-pairs correlations, and iteratively updates disparity using ConvGRUs.
- IGEV-Stereo uses GEV to regress an accurate initial disparity, enabling faster convergence than starting from zero.
- 0.47 EPE on Scene Flow and 1st-place rankings on KITTI 2015 and KITTI 2012 Reflective accompany top-10-leading inference speed and strong cross-dataset generalization.The method also achieves competitive accuracy when extended to multi-view stereo on DTU.
2. Related Work
Stereo cost-filtering methods improve cost-volume representation through 3D convolution but incur high computational costs, while iterative methods offer efficient progressive refinement. IGEV-Stereo combines geometry encoding with iterative optimization.
- Cost Filtering-based Methods: Cost-filtering methods use CNN features and 3D convolutional networks to regularize cost volumes and address ambiguity in challenging regions.
- Iterative Optimization-based Methods: IGEV-Stereo combines geometry and context encoding with all-pairs correlations, then iteratively updates disparity using ConvGRUs.
- Cost Filtering-based Methods: Cost-volume filtering can prevent application to high-resolution volumes because of computational demands, motivating coarse-to-fine and iterative alternatives.
3. Method
IGEV-Stereo combines geometry-regularized cost information with local correlations, then iteratively updates disparity using ConvGRUs initialized from a regressed starting point. Its architecture includes multi-scale feature and context extraction, volume indexing, recurrent refinement, spatial upsampling, and supervised losses.
- Architecture: IGEV-Stereo extracts multi-scale feature and context representations for cost-volume construction, cost aggregation guidance, and ConvGRU hidden-state initialization and updating.The feature network produces scales from 1/4 to 1/32 resolution, while the context network produces 1/4, 1/8, and 1/16-resolution features with 128 channels.
- Combined Geometry Encoding Volume: Group-wise correlations form Ccorr, which a lightweight 3D regularization network transforms into geometry encoding volume CG.The regularization network is a lightweight 3D UNet with three down-sampling and three up-sampling blocks; guided cost-volume excitation uses left features.
- Combined Geometry Encoding Volume: The method combines a two-level pooled CG pyramid with an all-pairs correlation pyramid to encode geometry, context, and local matching information.Average pooling with kernel size 2 and stride 2 is applied along the disparity dimension before combining the pyramids.
- ConvGRU-based Update Operator: Soft argmin over CG regresses an initial disparity d0, after which three ConvGRU levels iteratively refine the disparity prediction.The initial disparity uses predetermined disparity indices at 1/4 resolution, and context features initialize the ConvGRU hidden states.
- ConvGRU-based Update Operator: At each iteration, the current disparity indexes the combined volume by linear interpolation, and the resulting geometry features guide ConvGRU hidden-state and disparity updates.Indexed features include local and pooled geometry and all-pairs correlations within radius r; the disparity update is d_{k+1} = d_k + Δd_k.
- Upsampling and Loss: A learned weighted combination upsamples quarter-resolution disparities to full resolution, while training supervises the initial estimate and all predicted disparities.The upsampling weights use higher-resolution context features and left-image features; the initial estimate uses smooth L1 loss and later predictions use exponentially weighted L1 losses with γ = 0.9.
4. Experiment
Experiments show that IGEV-Stereo improves accuracy, efficiency, convergence, ill-posed-region performance, cross-dataset generalization, and multi-view stereo results across several benchmarks.
- Ablation study: Reducing iterations preserves strong performance: the full IGEV-Stereo model surpasses RAFT-Stereo by 69.44% at one iteration.The experiments attribute this to more accurate geometry and context information and an initial disparity regressed from GEV.
- Stereo benchmarks: Scene Flow experiments report a state-of-the-art EPE of 0.47 for IGEV-Stereo.
- Stereo benchmarks: IGEV-Stereo ranks first on the KITTI 2015 leaderboard and outperforms competing methods on several KITTI metrics.It surpasses CREStereo and RAFT-Stereo by 5.92% and 12.64% on KITTI 2015 D1-all, respectively.
- Generalization: Synthetic-to-real evaluations show state-of-the-art cross-domain performance on Middlebury 2014 and ETH3D when models are trained only on Scene Flow.Qualitative comparisons report better details for textureless and fine-structured regions.
- Extension to MVS: The IGEV-MVS extension achieves the best overall score on DTU, with relative overall-quality improvements of 8.0% and 10.7% over PatchmatchNet and IterMVS.
5. Conclusion and Future Work
IGEV provides a unified architecture for stereo matching and MVS, combining geometry, context, and local matching information through iterative volume indexing. The method achieves strong benchmark performance, while high-resolution images with large disparity ranges remain computationally challenging.
- IGEV is a deep network architecture for stereo matching and multi-view stereo that iteratively indexes a combined geometry encoding volume.The volume encodes geometry, context information, and local matching details.
- IGEV-Stereo ranks 1st on the KITTI 2015 leaderboard and achieves state-of-the-art cross-dataset generalization ability.
- IGEV-MVS achieves competitive performance on the DTU benchmark.
- High-resolution images with large disparity ranges can still cause high computational and memory costs because a 3D CNN processes the large cost volume.The authors identify a more lightweight regularization network and cascaded cost volumes as future directions.