Source-linked AI summary
Hierarchical Deep Stereo Matching on High-resolution Images
Gengshan Yang, Joshua Manela, Michael Happold, Deva Ramanan
TL;DR
Real-time stereo matching on high-resolution imagery is constrained by memory, speed, and limited high-resolution data, despite its importance for depth perception in autonomous driving. The paper proposes an end-to-end coarse-to-fine hierarchical framework, supported by high-resolution datasets and augmentation, and reports SOTA performance on Middlebury-v3 and KITTI-15 while running significantly faster than prior art. Its hierarchy also provides on-demand disparity estimates for nearby objects, although performance increases substantially under lighting changes.
Problem
Real-time high-resolution stereo matching remains difficult because existing methods face memory and speed limitations, while high-resolution training and evaluation datasets are scarce.
Method
The paper uses an end-to-end hierarchical coarse-to-fine stereo framework, high-resolution synthetic and real datasets, and augmentations for calibration and imaging variations.
Results
The model achieves SOTA performance on Middlebury-v3 and KITTI-15 while running significantly faster than prior art.
Takeaways & Limitations
On-demand hierarchical inference supports disparity estimation at different scales, enabling accurate real-time depth prediction for close-by objects.
Takeaways & Limitations
Under lighting changes, the average error of both HSM-H and ELAS increases substantially.
Abstract
from arXiv · showhide
We explore the problem of real-time stereo matching on high-res imagery. Many state-of-the-art (SOTA) methods struggle to process high-res imagery because of memory constraints or speed limitations. To address this issue, we propose an end-to-end framework that searches for correspondences incrementally over a coarse-to-fine hierarchy. Because high-res stereo datasets are relatively rare, we introduce a dataset with high-res stereo pairs for both training and evaluation. Our approach achieved SOTA performance on Middlebury-v3 and KITTI-15 while running significantly faster than its competitors. The hierarchical design also naturally allows for anytime on-demand reports of disparity by capping intermediate coarse results, allowing us to accurately predict disparity for near-range structures with low latency (30ms). We demonstrate that the performance-vs-speed trade-off afforded by on-demand hierarchies may address sensing needs for time-critical applications such as autonomous driving.
1. Introduction
High-resolution stereo is needed for low-latency depth perception but remains difficult because existing methods face accuracy, memory, speed, and data limitations. The paper addresses these constraints with hierarchical matching, high-resolution data, and augmentation, achieving faster SOTA performance.
- Motivation: High-res stereo supports dense long-range sensing, while LiDAR is limited in spatial density and slow scanning introduces latency and rolling-shutter effects.High-resolution global-shutter stereo is presented as a potential way to address these sensing limitations.
- Motivation: Depth error increases quadratically with depth, making far-field stereo estimates unstable and motivating higher-resolution cameras with larger focal lengths.The passage gives ∆Z = Z^2∆d/(b·f) as the triangulation error relation.
- Challenges: Existing stereo methods struggle to scale to high resolution because of running-time and memory overhead, while downscaling blurs results and harms far-field disparity.Deep methods also lack high-resolution training data and have not achieved SOTA results on high-resolution benchmarks.
- Approach: The proposed framework searches correspondences hierarchically from coarse to fine, using coarse estimates to bias or prewarp fine-scale disparity estimates.The architecture constructs multiscale feature volumes and processes them with disparity-strided 3D convolutions to reduce memory.
- Approach: Intermediate coarse volumes produce rough large-disparity estimates before full processing, enabling on-demand reports for nearby objects during real-time navigation.The contribution is an architecture that supports on-demand computation for high-resolution images.
- Contributions: The paper contributes high-resolution synthetic training data, a real testing dataset, and stereo augmentations targeting calibration errors, exposure changes, and occlusions.The authors report SOTA accuracy on Middlebury and KITTI while running significantly faster than prior art.
2. Related Work
Prior stereo systems incur substantial time and memory costs, while deep methods achieve strong low-resolution results but struggle on high-resolution imagery. This paper combines an efficient hierarchical architecture with high-resolution synthetic data and augmentation to address those issues.
- Traditional and efficient stereo: Traditional stereo builds cost volumes and applies global optimization, which imposes considerable time and memory costs and limits scale.The pipeline operates on rectified image pairs and corresponding epipolar scanlines.
- Traditional and efficient stereo: SGM and ELAS reach 3FPS on 1.5-megapixel images but struggle at 6 megapixels; SGM requires 16 GB for a 700px disparity range.Performance can also drop relative to lower-resolution processing.
- Deep stereo matching: Deep stereo methods achieve SOTA performance on several benchmarks but generally require considerable memory and time.Earlier systems combine learned features with traditional cost-volume post-processing, while newer methods use 2D or 3D convolutions.
- Paper positioning: The paper claims to address high-resolution stereo by combining efficient hierarchical matching with high-resolution synthetic data and novel augmentation techniques.This combination targets both efficiency and high-resolution data scarcity.
3. Method
The method combines an efficient hierarchical stereo architecture, asymmetric augmentations, and high-resolution datasets for training and evaluation. It builds increasingly detailed feature and cost volumes while supporting disparity predictions at multiple pyramid levels.
- Hierarchical Stereo Matching: The framework extracts multi-scale features and incrementally builds cost volumes from coarse to fine, increasing spatial and disparity resolution.Potential correspondences from coarse levels guide finer estimates through the volume pyramid.
- Hierarchical Stereo Matching: Strided disparity convolutions, native-resolution pooled features, and volumetric pyramid pooling reduce memory use while preserving global context.The decoder filters each volume, pools contextual features, and upsamples outputs for fusion with the next volume.
- Hierarchical Stereo Matching: Multi-scale losses train predictions at different pyramid levels, enabling on-demand disparity output and regularizing the network.The loss spans fine and coarse levels, with expected disparity performing better than a softmax distribution over candidate disparities.
- Data Augmentation: Asymmetric calibration, chromatic, and masking augmentations address scanline misalignment, imaging differences, occlusions, and missing correspondences.These augmentations expose the model to geometric and appearance asymmetries while encouraging use of contextual cues.
- High-resolution Datasets: The proposed Synthetic HR-VS and HR-RS datasets provide high-resolution driving data for training and real-world benchmarking.They address the scarcity of large-scale high-resolution stereo datasets and outdoor driving benchmarks.
- High-resolution Datasets: Driving-oriented evaluation uses short-, middle-, and long-range metrics based on safe stopping distances at multiple vehicle speeds.The reported stopping distances are 25m, 60m, and 115m for 25, 40, and 55mph, respectively.
4. Experiments
Experiments evaluate HSM across high- and low-resolution benchmarks, anytime inference, ablations, and robustness conditions. The results show strong benchmark accuracy, speed advantages, and improved resilience to calibration, exposure, and occlusion changes, while lighting changes remain difficult.
- Benchmark performance: HSM-F3 reduces Middlebury avgerr by 80.8%, rms by 75.6%, and bad-4.0 by 38.7% versus MC-CNN-acrt while running 294.1 times faster.Against CBMV ROB, it reduces avgerr by 48.3%, rms by 51.6%, and bad-4.0 by 6.0% while running 7737.3 times faster.
- Benchmark performance: HSM ranks 1st among published KITTI-15 methods while running 3.8 times faster than EdgeStereo, which ranks second.
- Anytime on-demand: 10.3% higher bad-4.0 accompanies 1.46x faster inference when HSM-F3 is cut off at HSM-F2, which remains more accurate than ELAS, SGBM, and iResNet.
- Long-range sensing: 14.8% higher long-range bad-4.0 results from halting HSM-F at HSM-F2, which still outperforms HSM-H3 at 17.1% versus 19.7%.The authors suggest the feature pyramid preserves high-resolution detail useful for long-range disparities.
- Diagnostics: Removing high-resolution synthetic data increases bad-1.0 by 8.7% and bad-2.0 by 11.1%, demonstrating its importance for high-resolution stereo training.
- Robustness: HSM-H average pixel error rises 5.9% under calibration error versus 29.6% for ELAS-H, while exposure changes raise errors by 8.2% and 44.2%, respectively.Under 1px translation, HSM-H error rises 1.0%; under 8px translation, it rises 51.5%.
Conclusion
The conclusion attributes HSM’s benchmark performance and speed to hierarchical designs, high-resolution synthetic data, and asymmetric augmentation. It also emphasizes on-demand disparity estimation for real-time close-by depth prediction, while the lighting-change evaluation documents a remaining failure condition.
- Conclusion: HSM achieves SOTA performance on Middlebury-v3 and KITTI-15 while running significantly faster than prior art.
- Conclusion: On-demand disparity estimation at different scales enables accurate depth prediction of close-by objects in real time.
- Robustness: Changed lighting introduces different shading between reference and target images, and both HSM and ELAS cannot handle that Bicycle1 failure case.