Source-linked AI summary
DeepPruner: Learning Efficient Stereo Matching via Differentiable PatchMatch
Shivam Duggal, Shenlong Wang, Wei-Chiu Ma, Rui Hu, Raquel Urtasun
TL;DR
Deep stereo algorithms need lower runtime and resource use without abandoning accurate matching. DeepPruner uses differentiable PatchMatch and learned confidence-range pruning to compute sparse cost volumes, then refines predictions end to end. It achieves competitive stereo results with real-time efficiency, including top performance among real-time methods and substantially faster inference than strong baselines.
Problem
State-of-the-art stereo algorithms remain costly because full disparity search requires substantial computation and memory, hindering real-time inference.
Method
DeepPruner combines differentiable PatchMatch, learned per-pixel confidence-range pruning, sparse cost-volume aggregation, propagation, and image-guided refinement in an end-to-end trainable network.
Results
DeepPruner achieves competitive results on SceneFlow and KITTI, ranks first among real-time methods, and remains several times faster than the best-performing methods.
Takeaways & Limitations
Progressive disparity pruning lets the model focus cost computation on high-likelihood hypotheses while reducing memory consumption and computational cost.
Abstract
from arXiv · showhide
Our goal is to significantly speed up the runtime of current state-of-the-art stereo algorithms to enable real-time inference. Towards this goal, we developed a differentiable PatchMatch module that allows us to discard most disparities without requiring full cost volume evaluation. We then exploit this representation to learn which range to prune for each pixel. By progressively reducing the search space and effectively propagating such information, we are able to efficiently compute the cost volume for high likelihood hypotheses and achieve savings in both memory and computation. Finally, an image guided refinement module is exploited to further improve the performance. Since all our components are differentiable, the full network can be trained end-to-end. Our experiments show that our method achieves competitive results on KITTI and SceneFlow datasets while running in real-time at 62ms.
1. Introduction
Deep stereo methods improve accuracy but remain costly because full disparity search and sophisticated processing demand substantial memory and computation. DeepPruner addresses this by differentiably pruning disparities, propagating likely hypotheses, and refining the result, achieving competitive accuracy with real-time efficiency.
- Stereo estimation remains difficult on repetitive, texture-less, occluded, and thin regions, while real-time inference often requires sacrificing accuracy.
- Full-range cost-volume construction and sophisticated post-processing make modern stereo architectures computationally and memory intensive.
- DeepPruner uses differentiable PatchMatch to obtain sparse disparity hypotheses, learns per-pixel confidence ranges for pruning, and applies image-guided refinement.
- The model progressively reduces and propagates the search space, computing costs only for high-likelihood hypotheses and reducing memory consumption and computational cost.
- DeepPruner ranks second on SceneFlow, is over 8 times faster than the best method, and ranks first among real-time KITTI models.
- The method achieves state-of-the-art results on multiple Robust Vision Stereo Challenge datasets and first place overall.
2. Related work
Prior stereo work combines feature extraction, cost-volume construction, and post-processing, while deep models improve scene understanding but retain efficiency challenges. PatchMatch motivates DeepPruner's differentiable, recurrent pruning and propagation strategy, alongside adaptive rather than fixed search reduction.
- Classical Stereo Matching: Classical stereo pipelines compute pixel-wise representations, construct cost volumes, and apply post-processing before selecting disparities.
- Deep Stereo Matching: Deep stereo methods use CNNs and cost volumes or direct disparity regression to address complex scenes, but time-consuming post-processing remains an issue.
- PatchMatch: PatchMatch uses random samples and neighboring-pixel coherence to find correspondences efficiently and propagate good matches across the image.
- PatchMatch: DeepPruner unrolls particle PatchMatch as recurrent neural-network layers, predicts disparity confidence ranges, and remains differentiable for end-to-end learning.
- Real-time Stereo: Unlike concurrent fixed coarse-to-fine methods, DeepPruner adaptively prunes the search space for each region.
3. Learning to Prune for Stereo Matching
DeepPruner combines differentiable PatchMatch with learned per-pixel confidence ranges to prune stereo disparities before cost aggregation, then refines the resulting estimates. Its end-to-end design progressively narrows the search space while preserving efficient propagation and differentiable training.
- Learning to Prune for Stereo Matching: DeepPruner combines deep learning with PatchMatch to prune the large stereo search space and target real-time inference.The model is designed to produce reliable disparity estimates while reducing runtime.
- Differentiable PatchMatch: Differentiable PatchMatch recurrently samples disparity particles, propagates them to neighboring pixels, and evaluates candidates to retain the best hypotheses.The method unrolls generalized PatchMatch as recurrent neural-network layers and replaces nondifferentiable selection with a soft version.
- Confidence Range Prediction: A confidence-range network uses sparse disparity estimates and image inputs to predict a per-pixel range R_i = [l_i, u_i] that excludes unlikely matches.The predicted range is intended to approximate where each pixel’s true disparity lies.
- Cost Aggregation and Refinement: The pruned ranges restrict cost-volume construction and aggregation to a small number of disparities, with the per-pixel disparity count R more than 10 times smaller than in prior work.This reduction makes the cost-aggregation module more efficient while retaining end-to-end trainability through soft-arg max.
- Cost Aggregation and Refinement: A lightweight fully convolutional refinement network uses low-level image features and the current disparity estimate to reduce noise and improve sharp boundaries.It outputs the final refined disparity prediction after cost aggregation.
- End-to-end Learning: The complete network is trained end-to-end with smooth-ℓ1 losses applied to disparity predictions from the cost-aggregation and refinement stages.The range bounds additionally use an unbalanced smooth-ℓ1 loss with a boomerang shape.
4. Experiments
Experiments evaluate DeepPruner on SceneFlow, KITTI, and the Robust Vision Challenge, showing competitive accuracy with substantially reduced runtime, memory, and disparity-search cost. Component and uncertainty analyses link pruning and refinement to efficiency, detail preservation, and error identification.
- Analysis: PatchMatch and the confidence range predictor reduce the search space, allowing cost aggregation on only a small subset of disparities.The confidence range predictor is described as crucial, while RefineNet adds visual guidance for sharper edges and finer details.
- SceneFlow: DeepPruner ranks second on SceneFlow, with its fast model over 8 times faster than prior art and 40% lower end-point-error than a previous real-time model.The best model is also reported as 2.5 times faster than the state-of-the-art comparison method.
- KITTI 2015: DeepPruner-Best achieves comparable KITTI performance to state-of-the-art approaches, while DeepPruner-Fast reduces the outlier ratio by more than 40% versus real-time stereo models.Qualitative KITTI results are reported as competitive across various scenarios.
- Analysis: Predicted uncertainty correlates highly with disparity error; removing 6% of the most uncertain pixels improves the outlier ratio by 38%.The analysis indicates that high-confidence regions have very low error and most errors occur in low-confidence regions.
- Robustness and generalizability: DeepPruner achieves the highest overall rank in the Robust Vision Stereo Challenge and the highest rank on two evaluated datasets.The model is fine-tuned jointly on KITTI, ETH3D, and MiddleburyV3 and captures fine-grained scene geometry.
- Efficiency: For full-size KITTI stereo images, DeepPruner-Best and DeepPruner-Fast use 1161 MB and 805 MB, respectively, versus 4351 MB for PSM-Net.The reported storage requirements are less than one fourth of the full cost-volume approach.
5. Conclusion
DeepPruner prunes cost-volume regions without fully evaluating matching scores through an end-to-end trainable differentiable PatchMatch network. It achieves the best performance among real-time methods and comparable performance to the best methods while remaining several times faster.
- 5. Conclusion: DeepPruner quickly prunes parts of the cost volume without fully evaluating each pixel's matching score.The network uses a novel differentiable PatchMatch module and is trained end-to-end.
- 5. Conclusion: DeepPruner achieves the best performance among real-time methods and comparable results with the best-performing methods while remaining several times faster.The paper identifies optical flow and scene flow as future application targets.