Source-linked AI summary
Practical Deep Stereo (PDS): Toward applications-friendly deep stereo matching
Stepan Tulyakov, Anton Ivanov, Francois Fleuret
TL;DR
End-to-end stereo networks face high memory demands and fixed disparity ranges, while SoftArgmin can produce poor estimates for multimodal distributions. PDS combines compact matching signatures with sub-pixel cross-entropy and MAP inference; it reports leading FlyingThings3D error rankings and strong KITTI rankings.
Problem
End-to-end stereo networks have large inference memory demands, fixed disparity ranges, and SoftArgmin estimates can degrade with multimodal disparity distributions.
Source: §1 · p. 3, §2.2 · p. 4, §2.3 · p. 5Method
PDS compresses left-right descriptors into matching signatures and pairs sub-pixel cross-entropy training with a sub-pixel MAP estimator.
Source: §1 · pp. 2–3Results
PDS has the lowest 3PE and second-lowest MAE on FlyingThings3D, and ranks third on KITTI’15 and fourth on KITTI’12.
Source: §1 · p. 3Takeaways & Limitations
PDS reaches state-of-the-art performance while addressing practical memory and disparity-range constraints.
Source: §4 · p. 9The MAP approximation assumes the disparity distribution is symmetric near a major mode.
Source: §2.3 · p. 5
Abstract
from arXiv · showhide
End-to-end deep-learning networks recently demonstrated extremely good perfor- mance for stereo matching. However, existing networks are difficult to use for practical applications since (1) they are memory-hungry and unable to process even modest-size images, (2) they have to be trained for a given disparity range. The Practical Deep Stereo (PDS) network that we propose addresses both issues: First, its architecture relies on novel bottleneck modules that drastically reduce the memory footprint in inference, and additional design choices allow to handle greater image size during training. This results in a model that leverages large image context to resolve matching ambiguities. Second, a novel sub-pixel cross- entropy loss combined with a MAP estimator make this network less sensitive to ambiguous matches, and applicable to any disparity range without re-training. We compare PDS to state-of-the-art methods published over the recent months, and demonstrate its superior performance on FlyingThings3D and KITTI sets.
1 Introduction
PDS targets practical constraints in end-to-end stereo with a lower-memory matching design and sub-pixel training and inference changes. It reports strong benchmark rankings and the ability to change disparity range without retraining.
1 Introduction
- PDS has the lowest 3PE and second-lowest MAE on FlyingThings3D, and ranks third on KITTI’15 and fourth on KITTI’12.The FlyingThings3D comparison covers 960 × 540 images with 192 disparities.Source: Table 1 · p. 2, §1 · p. 3
- Compact matching signatures compress concatenated descriptors before hourglass regularization, reducing memory use and supporting larger images.The signatures replace the concatenated descriptors as input to the regularizer.Source: §1 · pp. 2–3
- Unlike SoftArgmin, sub-pixel MAP averages near the maximum-posterior disparity, reducing sensitivity to erroneous modes and allowing disparity-range changes without retraining.SoftArgmin can blend modes when the posterior is multimodal.Source: §1 · pp. 2–3
- Earlier end-to-end methods trained on modest patches; PDS trains on full-size images to exploit larger context and uses sub-pixel cross-entropy for faster convergence and accuracy.The loss combines standard cross-entropy with kernel interpolation.Source: §1 · p. 3
2 Method
PDS combines a memory-saving bottleneck matching module with a local sub-pixel MAP estimator and a distance-shaped cross-entropy target. Its network maps stereo images to disparity costs and regularizes compact matching signatures.
2.1 Network structure
- The network embeds left and right images, then feeds concatenated per-disparity matching signatures to a 3D hourglass regularizer.The matching module produces a compact signature for each disparity before the signatures are concatenated.Source: §2.1 · p. 4
2.2 Matching module
- A bottleneck matching module compresses concatenated left-right descriptors into compact signatures, reducing memory versus feeding descriptors directly to the regularizer.The preceding 3D hourglass methods’ descriptor tensor has a large inference memory footprint.Source: §2.2 · p. 4
2.3 Sub-pixel MAP
- SoftArgmin blends modes for multimodal posteriors and may degrade when the disparity range expands; PDS instead estimates locally around the maximum-posterior disparity.The local MAP approximation assumes symmetry near a major mode, uses δ = 4 in the experiments, and is applied at inference; training uses the posterior distribution and sub-pixel cross-entropy.Source: §2.3 · p. 5
2.4 Sub-pixel cross-entropy
- The loss uses a discretized Laplace target centered on ground truth (b = 2), producing a distance-shaped posterior for sub-pixel MAP and claimed faster convergence and better accuracy.The target probability at each discrete location is a smooth function of its distance from the sub-pixel ground truth.Source: §2.4 · p. 6
3 Experiments
Experiments show that full-size training improves use of image context, while sub-pixel MAP and cross-entropy improve aspects of disparity estimation and range flexibility. PDS achieves the lowest 3PE and second-lowest MAE on FlyingThings3D, and ranks third on KITTI’15 and fourth on KITTI’12.
3 Experiments
- Training uses RMSprop with standard settings, and the experiments use available datasets with code planned for open-source release after publication.Training settings are summarized for each dataset in Table 2.Source: §3 · p. 6, Table 2 · p. 6
3.1 Datasets and performance measures
- FlyingThings3D training excludes artifact-containing images and restricts disparities to [0, 255]; benchmarking follows Chang and Chen [2018] protocol.The authors use KITTI’12 and KITTI’15 together as a KITTI set, and report 3PE and MAE.Source: §3.1 · pp. 6–7
3.2 Training on full-size images
- Full-size 960 × 540 training images outperform 512 × 256 patches, with the authors attributing the gain to better use of larger image context.Both networks were trained with SoftArgmin and L1 loss; the smaller-patch setup follows Kendall et al. [2017] and Chang and Chen [2018].Source: §3.2 · p. 7, Table 4 · p. 7
3.3 Sub-pixel MAP and cross-entropy
- Sub-pixel MAP lowers 3PE but slightly increases MAE, and its performance changes little when the disparity range doubles from 255 to 511 pixels.With the larger range, SoftArgmin performance plummets; sub-pixel MAP also enables changing the range without retraining.Source: §3.3 · p. 8, Table 5 · p. 8
- Sub-pixel cross-entropy yields much smaller 3PE and only slightly larger MAE than SoftArgmin with L1 loss, and converges faster than L1.Kendall et al. [2017] also report faster convergence with one-hot cross-entropy, but found L1 gave smaller 3PE.Source: Figure 5 · p. 7, §3.3 · p. 8, Table 5 · p. 8
3.4 Benchmarking
- PDS has the lowest 3PE and second-lowest MAE on FlyingThings3D, ranking third on KITTI’15 and fourth on KITTI’12.The KITTI rankings are from the 15/05/2018 leaderboard snapshots.Source: §3.4 · p. 8
4 Conclusion
PDS addresses the large memory footprint of deep stereo networks and their inability to adjust to a different disparity range without retraining. The authors report state-of-the-art performance, with third place on KITTI’15 and fourth on KITTI’12 in the cited leaderboard snapshots.
4 Conclusion
- The paper addresses deep stereo’s large memory footprint and inability to adjust disparity range without complete retraining.Its conclusion attributes the practical resolution to memory-conscious architecture, analytical range adaptation, and a new loss and estimator for multimodal posteriors and sub-pixel accuracy.Source: §4 · pp. 8–9
- PDS ranks third on KITTI’15 and fourth on KITTI’12 in the 15/05/2018 leaderboard snapshots.The snapshots include recent and not-yet-officially-published methods.Source: Table 6 · p. 9