Source-linked AI summary
Anytime Stereo Image Depth Estimation on Mobile Devices
Yan Wang, Zihang Lai, Gao Huang, Brian H. Wang, Laurens van der Maaten, Mark Campbell, Kilian Q. Weinberger
TL;DR
Stereo depth estimation must be accurate and fast on computationally constrained robotic devices, but existing methods often trade accuracy against latency and require many parameters. AnyNet uses staged, end-to-end residual refinement that can be queried at any time, achieving 10–35 FPS on a Jetson TX2 while retaining competitive accuracy with far fewer parameters.
Problem
Existing stereo depth methods face a trade-off between accurate slow predictions and fast inaccurate predictions, while often requiring too many parameters for constrained devices.
Method
AnyNet performs staged depth refinement, reuses computation, and predicts bounded residual disparities so inference can dynamically trade speed for accuracy.
Results
10–35 FPS is attainable on a Jetson TX2, while AnyNet achieves competitive state-of-the-art accuracy with orders of magnitude fewer parameters than baselines.
Takeaways & Limitations
Anytime stereo depth estimation provides progressively improving predictions that can support rapid reactions or more detailed navigation and mapping as computation becomes available.
Abstract
from arXiv · showhide
Many applications of stereo depth estimation in robotics require the generation of accurate disparity maps in real time under significant computational constraints. Current state-of-the-art algorithms force a choice between either generating accurate mappings at a slow pace, or quickly generating inaccurate ones, and additionally these methods typically require far too many parameters to be usable on power- or memory-constrained devices. Motivated by these shortcomings, we propose a novel approach for disparity prediction in the anytime setting. In contrast to prior work, our end-to-end learned approach can trade off computation and accuracy at inference time. Depth estimation is performed in stages, during which the model can be queried at any time to output its current best estimate. Our final model can process 1242$ \times $375 resolution images within a range of 10-35 FPS on an NVIDIA Jetson TX2 module with only marginal increases in error -- using two orders of magnitude fewer parameters than the most competitive baseline. The source code is available at https://github.com/mileyan/AnyNet .
I. INTRODUCTION
Stereo depth estimation supports 3D perception but remains difficult to deploy in real time on resource-constrained devices. AnyNet addresses this with staged, queryable refinement that dynamically trades speed for accuracy.
- Stereo disparity estimation supports 3D scene reconstruction, robotics, and augmented reality, with deep networks enabling end-to-end matching from image pairs.
- Below 0.3 FPS on an Nvidia Jetson TX2, PSMNet is too slow for timely obstacle avoidance by drones or autonomous robots.
- AnyNet dynamically trades speed for accuracy by refining depth maps in stages and allowing the current estimate to be queried at any time.
- Low-resolution full-range estimation starts the process, while later residual updates restrict computation to 10–20% of the full disparity range.
- 10–35 FPS is attainable on a TX2 module while preserving accurate disparity estimation in high-latency settings, using computation re-use across updates.
- AnyNet achieves competitive state-of-the-art accuracy with orders of magnitude fewer parameters than baseline methods.
II. RELATED WORK
Prior disparity-estimation work spans traditional feature matching and end-to-end convolutional models, while anytime prediction has largely focused on progressively evaluated classifiers. AnyNet builds on these directions for stereo depth estimation.
- Disparity estimation: Traditional disparity estimation matches features through cost computation, cost aggregation, disparity selection, and refinement.
- End-to-end disparity prediction: End-to-end models directly predict disparities from stereo image pairs, with later work adding cascaded refinement and learned matching-cost refinement.
- End-to-end disparity prediction: GCNet combines a 2D Siamese convolutional network for image pairs with a 3D convolutional network operating on the matching-cost tensor.
- Anytime prediction: Most prior anytime-learning approaches use ensembles of decision-tree classifiers evaluated tree by tree to produce progressive prediction updates.
III. ANYNET
AnyNet estimates disparity progressively, beginning with a low-resolution full-range prediction and refining it through residual updates and a final spatial-propagation stage. Its architecture reuses computation across stages while allowing increasingly accurate outputs as more computation becomes available.
- U-Net Feature Extractor: AnyNet’s U-Net extractor computes left- and right-image feature maps at 1/16, 1/8, and 1/4 scales, which feed stages 1–3 when needed.Lower-resolution features capture global context, while higher-resolution features capture local details.
- Disparity Network: The disparity network builds a cost volume whose (i, j, k) entry measures the match between left pixel (i, j) and right pixel (i, j−k).The cost volume has dimensions H × W × M, where M is the maximum disparity.
- Disparity Network: Instead of selecting a noisy hard minimum, AnyNet uses weighted disparity regression, recovering a clear best match or averaging viable candidates under ambiguity.The cost volume is further refined with 3D convolutions before regression.
- Residual Prediction: Stage 1 computes the full disparity map only at very low resolution, while stages 2 and 3 upsample it and predict bounded residual corrections.Restricting residual offsets to −2 through 2 limits the high-resolution maximum disparity to M = 5 instead of the full range.
- Spatial Propagation Network: A fourth SPNet stage sharpens the disparity map using a locally filtered prediction whose weights come from a small CNN applied to the left image.The paper reports significant improvement at relatively little extra cost.
IV. EXPERIMENTAL RESULTS
AnyNet is evaluated on synthetic Scene Flow training data and real-world KITTI benchmarks, with controlled comparisons on the Jetson TX2. Its four-stage predictions progressively refine disparity estimates as computation increases.
- Experimental setup: AnyNet uses four stages and is evaluated on Scene Flow, KITTI-2012, and KITTI-2015 datasets.The KITTI datasets contain 1242 × 375 images, while Scene Flow training uses cropped 512 × 256 patches.
- Experimental setup: 40,000 parameters make AnyNet one order of magnitude smaller than StereoNet and two orders smaller than PSMNet.The model is trained end-to-end with Adam and evaluated using five randomized 80/20 train/validation splits.
- Qualitative evaluation: AnyNet predicts progressively refined disparity maps across four stages as more computation becomes available.The final stage is sharper and more accurate because of SPNet post-processing.
- Baselines: The TX2 evaluation compares AnyNet with PSMNet, StereoNet, DispNet, StereoDNN, Block Matching, and Semi-Global Block Matching.Baseline CNN methods use down-sampled inputs for the controlled comparison.
- Metrics: Three-pixel error is the reported metric for AnyNet on KITTI-2012 and KITTI-2015, with lower values indicating better performance.The evaluation includes numerical results summarized in Table II.
A. Evaluation Results
AnyNet maintains a favorable speed–accuracy trade-off across inference budgets and supports anytime querying during computation. It achieves rapid prediction while retaining lower error than the baselines across the evaluated ranges.
- Non-anytime evaluation: AnyNet and StereoNet are the only evaluated methods capable of rapid real-time prediction at ≥30 FPS, while AnyNet has drastically lower error on both KITTI datasets.The comparison uses three-pixel error and inference time across multiple input resolutions.
- Non-anytime evaluation: AnyNet runs above 10 FPS with full-resolution inputs and dominates all baselines in prediction error across the evaluated inference-time ranges.PSMNet is most accurate overall only at computation rates of 1 FPS or slower.
- Anytime setting: In the anytime setting, AnyNet can be polled at any time to return its most recent disparity prediction.Baselines are approximated by sequentially processing increasingly larger input resolutions.
- Anytime setting: AnyNet produces more accurate results than the baselines in the 10–30 FPS anytime range.Its shorter intervals between completed disparity maps reduce wasted computation when queries arrive during processing.
B. Ablation Study
The ablation study tests feature extraction, residual prediction, and cost-volume construction. Results show that shared multiscale features and residual refinement are central to AnyNet’s efficiency, while distance-based costs improve speed.
- Study design: The ablation compares three variants: separate feature extractors, full-scale disparity prediction, and PSMNet-style cost volumes.All variants are trained from scratch before evaluation.
- Feature extractor: Replacing the U-Net with three separate convolutional networks increases first- and second-stage errors to 20.4% and 7.3%.The authors hypothesize that U-Net context from higher resolutions improves low-resolution cost volumes.
- Residual prediction: Directly predicting disparities instead of residuals increases evaluation time in the final two stages by more than sixfold.The variant reaches similar accuracy, but residual prediction avoids large cost volumes spanning broad disparity ranges.
- Cost-volume construction: The distance-based cost-volume approach is about 10% faster than the method used in PSMNet.The result indicates a better speed–accuracy trade-off when feature distances are explicitly considered.
V. DISCUSSION AND CONCLUSION
AnyNet is presented as the first algorithm for anytime depth estimation from stereo images, targeting accurate and reliable real-time performance across robotic applications.
- AnyNet is presented as the first algorithm for anytime depth estimation from stereo images.