Source-linked AI summary
Practical Stereo Matching via Cascaded Recurrent Network with Adaptive Correlation
Jiankun Li, Peisen Wang, Pengfei Xiong, Tao Cai, Ziwei Yan, Lei Yang, Jiangyu Liu, Haoqiang Fan, Shuaicheng Liu
TL;DR
Consumer stereo matching must recover disparities from image pairs affected by thin structures, rectification errors, camera inconsistencies, and hard-case scenes. CREStereo combines recurrent coarse-to-fine cascades, adaptive group correlation, and varied synthetic data, ranking first on Middlebury and ETH3D while performing strongly on real-world scenes. The model remains limited by efficiency.
Problem
Consumer stereo matching remains difficult because thin structures, non-ideal rectification, inconsistent camera modules, and hard-case scenes complicate disparity estimation.
Method
CREStereo uses recurrent coarse-to-fine disparity refinement, stacked cascades, adaptive group local correlation, and a varied synthetic dataset for real-world generalization.
Results
CREStereo ranks 1st on both ETH3D and Middlebury benchmarks and outperforms published state-of-the-art methods by 21.73% on bad 2.0 and 31.00% on A95.
Takeaways & Limitations
The results support jointly designing network architecture and training data to recover delicate details and handle hard-case real-world scenes.
Takeaways & Limitations
The model is not yet efficient enough.
Abstract
from arXiv · showhide
With the advent of convolutional neural networks, stereo matching algorithms have recently gained tremendous progress. However, it remains a great challenge to accurately extract disparities from real-world image pairs taken by consumer-level devices like smartphones, due to practical complicating factors such as thin structures, non-ideal rectification, camera module inconsistencies and various hard-case scenes. In this paper, we propose a set of innovative designs to tackle the problem of practical stereo matching: 1) to better recover fine depth details, we design a hierarchical network with recurrent refinement to update disparities in a coarse-to-fine manner, as well as a stacked cascaded architecture for inference; 2) we propose an adaptive group correlation layer to mitigate the impact of erroneous rectification; 3) we introduce a new synthetic dataset with special attention to difficult cases for better generalizing to real-world scenes. Our results not only rank 1st on both Middlebury and ETH3D benchmarks, outperforming existing state-of-the-art methods by a notable margin, but also exhibit high-quality details for real-life photos, which clearly demonstrates the efficacy of our contributions.
1. Introduction
Practical stereo matching remains difficult for consumer photography because fine structures, non-ideal rectification, camera inconsistencies, and hard-case scenes challenge disparity estimation. CREStereo addresses these issues with recurrent cascaded refinement, adaptive correlation, and varied synthetic training data, achieving leading benchmark and real-world performance.
- Motivation: Consumer stereo matching remains challenging despite CNN progress because everyday image pairs contain practical complications.These complications include thin structures, imperfect rectification, inconsistent camera modules, and difficult scenes.
- Motivation: Fine structures such as nets and wire frames are difficult to recover, and disparity errors can degrade computational-bokeh rendering.Higher consumer-image resolution further worsens the difficulty of recovering fine disparity details.
- Contributions: CREStereo updates disparities recurrently in a hierarchical coarse-to-fine network and uses stacked cascades for high-resolution inference.The method is designed to better recover intricate image details.
- Contributions: An adaptive group local correlation layer mitigates the negative influence of rectification error during feature matching.The design targets stereo pairs whose rectification is non-ideal because of differing camera-module traits.
- Contributions: A synthetic dataset with richer lighting, texture, and shape variations is introduced to improve generalization to real-world scenes.The dataset specifically targets broader variation than standard synthetic training data.
- Results: CREStereo ranks 1st on both ETH3D and Middlebury and achieves competitive performance on KITTI 2012/2015 among published methods.The network also demonstrates superior performance on arbitrary real-world scenes.
2. Related Work
Prior stereo-matching research spans traditional optimization, learning-based cost estimation and end-to-end networks, practical domain adaptation, and synthetic-data generation. The proposed network overview combines shared feature extraction, multiscale correlations, recurrent refinement, and stacked cascades.
- Traditional algorithms: Traditional stereo algorithms use local support windows or global optimization methods such as belief propagation and graph cuts.Semi-global matching later uses mutual information with dynamic programming.
- Learning-based algorithms: Learning-based stereo methods evolved from CNN matching-cost initialization to predominantly end-to-end networks using 2D convolutions.Earlier pipelines refined CNN-derived costs with cross-based aggregation and semi-global optimization.
- Practical stereo matching: Practical stereo matching is less explored, with prior work addressing target-domain adaptation, smartphone bokeh, and synthetic-to-real domain gaps.These approaches include self-adaptation without ground-truth disparity and domain-adaptation pipelines.
- Synthetic datasets: Synthetic datasets provide accurate dense disparity ground truth because obtaining such labels in real-world scenes is difficult.Prior generation approaches use Blender, real-image textures, or randomized polygons for training data.
- Proposed network overview: The proposed architecture uses a three-level feature pyramid to compute multiscale correlations across three cascaded recurrent stages.Shared-weight feature extraction supplies context, while each stage iteratively refines features and predicted disparities.
3. Methods
CREStereo combines adaptive local correlation with recurrent, cascaded disparity refinement and a synthetic data pipeline targeting challenging real-world stereo conditions.
- Adaptive Group Correlation Layer: Adaptive Group Correlation Layer reduces matching ambiguity when imperfect calibration places corresponding points off the same scanline.It uses adaptive search to address rotations and residual lens distortion after rectification.
- Adaptive Group Correlation Layer: Local correlation produces an H × W × D volume, using far fewer pairs than all-pairs or fixed-range cost-volume methods.The reduced volume saves memory and computation, while avoiding a preset displacement range for different stereo baselines.
- Adaptive Group Correlation Layer: The 2D-1D alternate local search combines vertical and horizontal offsets to improve matching under non-ideal rectification.In 1D mode, f(d) ranges from −r to r with r = 4 and g(d) = 0; 2D mode uses a dilated grid.
- Adaptive Group Correlation Layer: Group-wise correlation splits feature maps into G groups, computes local correlations independently, and concatenates the resulting volumes.The output volume has dimensions GD × H × W.
- Cascaded Recurrent Network: Cascaded recurrent refinement computes correlations at separate feature levels and independently refines disparities through multiple iterations.RUMs use GRU blocks and shared weights; later cascade levels initialize from upsampled predictions from earlier levels.
- Stacked Cascades for Inference: Stacked cascades feed image-pyramid levels into the same trained network to enlarge receptive fields while retaining high-resolution detail.The architecture addresses high-resolution inputs where direct downsampling can deteriorate small objects with large displacement.
- Synthetic Dataset: The synthetic dataset uses Blender to generate dense, pixel-accurate disparities with varied shapes, textures, and challenging real-world cases.Each scene contains stereo image pairs and disparity maps captured by dual virtual cameras.
4. Experiments
Experiments evaluate CREStereo through ablations, synthetic-data comparisons, public benchmarks, and practical-scene tests. The results support adaptive local correlation, cascaded refinement, stacked inference, and the proposed synthetic data for accurate and robust stereo matching.
- Evaluation setup: The evaluation covers Middlebury, ETH3D, and KITTI benchmarks using AvgErr, Bad2.0, D1-all, and related metrics.Middlebury includes high-resolution, large-disparity pairs; ETH3D covers monochrome indoor and outdoor scenes; KITTI contains street-view stereo pairs.
- Correlation ablations: Local correlation forms outperform 2D and 1D all-pairs correlation, with accuracy dropping substantially when alternate local correlation is replaced by a single correlation.The degradation becomes more evident with multiple cascade levels because rectification error increases with resolution.
- Correlation ablations: Adaptive group correlation improves accuracy over fixed windows, single-form correlation, and designs without local feature attention.The ablation supports learned offsets, group correlation, and local feature attention as useful AGCL components.
- Cascade ablations: Cascaded recurrent units substantially improve precision, with prediction error decreasing as more cascade levels are used.A single RUM without cascades produces a substantial precision drop.
- Cascade ablations: Multi-level image-pyramid inputs substantially reduce error during inference compared with a single cascade, supporting the stacked cascades scheme.The architecture shares trained parameters across stages and uses shortcuts to preserve high-resolution context.
- Synthetic data: The proposed synthetic data yields lower training loss and better ETH3D and Middlebury validation performance than similarly sized Sceneflow data.Both datasets train the same model with the same augmentation for 50,000 iterations, supporting improved domain generalization.
- Benchmark comparisons: 21.73% on the bad 2.0 metric and 31.00% on the A95 metric surpass the published state-of-the-art on Middlebury.CREStereo ranks first on the majority of metrics among more than 120 methods.
- Benchmark comparisons: 59.84% on the bad 1.0 metric surpasses the published state-of-the-art on ETH3D, while KITTI performance remains competitive and exceeds LEAStereo by 9.47% on KITTI 2012 Out-Noc.Qualitative tests also report advantages on thin objects, textureless regions, and disturbed ETH3D inputs.
5. Conclusion
CREStereo combines cascaded recurrent refinement, adaptive correlation, and synthetic training data to address practical stereo matching. It achieves state-of-the-art results on public benchmarks and real-world scenes, but is not yet efficient enough for current mobile applications.
- Conclusion: CREStereo uses cascaded recurrent refinement, stacked high-resolution inference, adaptive group correlation, and a synthetic dataset for practical stereo matching.The design targets delicate details, non-ideal rectification, and hard-case scenes such as non-textured or repetitive-texture areas.
- Conclusion: The method attains state-of-the-art results on public benchmarks and real-world scenes while recovering delicate depth details better than existing methods.The conclusion connects performance to both network architecture and training-data design.
- Conclusion: The model is not yet efficient enough to run in current mobile applications.Future work is directed toward portable-device adaptation, preferably in real time.