Source-linked AI summary
Pyramid Stereo Matching Network
Jia-Ren Chang, Yong-Sheng Chen
TL;DR
Stereo CNNs struggle to find correspondences in ill-posed regions because patch-based methods lack sufficient context. PSMNet combines spatial pyramid pooling with stacked hourglass 3D CNN regularization, and it ranked first on the KITTI 2012 and 2015 leaderboards before March 18, 2018.
Problem
Patch-based CNN stereo methods lack sufficient context for accurate correspondence estimation in ill-posed regions such as occlusions, repeated patterns, textureless areas, and reflective surfaces.
Method
PSMNet forms a cost volume from multiscale global context using spatial pyramid pooling, then regularizes it with stacked hourglass 3D CNNs and intermediate supervision.
Results
PSMNet ranked first on both the KITTI 2012 and 2015 leaderboards before March 18, 2018.
Takeaways & Limitations
The end-to-end framework exploits global context and performs stereo matching without post-processing.
Abstract
from arXiv · showhide
Recent work has shown that depth estimation from a stereo pair of images can be formulated as a supervised learning task to be resolved with convolutional neural networks (CNNs). However, current architectures rely on patch-based Siamese networks, lacking the means to exploit context information for finding correspondence in illposed regions. To tackle this problem, we propose PSMNet, a pyramid stereo matching network consisting of two main modules: spatial pyramid pooling and 3D CNN. The spatial pyramid pooling module takes advantage of the capacity of global context information by aggregating context in different scales and locations to form a cost volume. The 3D CNN learns to regularize cost volume using stacked multiple hourglass networks in conjunction with intermediate supervision. The proposed approach was evaluated on several benchmark datasets. Our method ranked first in the KITTI 2012 and 2015 leaderboards before March 18, 2018. The codes of PSMNet are available at: https://github.com/JiaRenChang/PSMNet.
1. Introduction
Stereo depth estimation requires finding pixel correspondences, but patch-based CNN methods struggle in ill-posed regions because local intensity consistency is insufficient. PSMNet addresses this with pyramid context aggregation and stacked hourglass 3D cost-volume regularization, achieving state-of-the-art KITTI accuracy.
- Problem: Stereo depth estimation computes each pixel’s disparity from corresponding points in rectified left and right images.Disparity is the horizontal displacement between corresponding pixels; depth depends on focal length, camera baseline, and disparity.
- Problem: CNN-based correspondence methods struggle in occlusions, repeated patterns, textureless regions, and reflective surfaces.Intensity consistency between viewpoints is insufficient in these inherently ill-posed regions, especially textureless areas.
- Method: PSMNet uses spatial pyramid pooling and dilated convolution to combine global and local features across receptive-field scales when forming the cost volume.The pyramid pooling module incorporates global context into image features and extends pixel-level features to region-level representations.
- Method: A stacked hourglass 3D CNN repeatedly processes the cost volume in top-down and bottom-up directions to improve context utilization.The architecture extends the regional support of context information and regularizes the cost volume with intermediate supervision.
- Method: The framework performs end-to-end stereo matching without post-processing.This is presented as one of PSMNet’s main contributions.
- Results: PSMNet achieves state-of-the-art accuracy on the KITTI dataset.The paper lists state-of-the-art KITTI accuracy as a main contribution.
2. Related Work
Stereo matching research progressed from handcrafted pipelines and patch-based similarity networks toward end-to-end disparity prediction and 3D cost-volume regularization. Drawing on context aggregation ideas from semantic segmentation and optical flow, PSMNet uses multiscale global context for stereo depth estimation.
- Stereo matching pipelines: Traditional stereo matching pipelines compute matching costs, aggregate them, optimize the result, and refine disparity.These four stages summarize the typical stereo matching algorithm described in the related work.
- Disambiguation and refinement: Other approaches resolved matching ambiguities through object-specific 3D models, confidence estimation, label correction, or multiscale residual prediction.Displets modeled vehicles, ResMatchNet learned reflective confidence, and CRL refined disparity through two stages.
- End-to-end methods: End-to-end networks began predicting whole disparity maps without post-processing, including DispNet, FlowNet, CRL, and GC-Net.GC-Net used 3D convolutions for cost-volume regularization, while Scene Flow provided a large synthetic training dataset.
- Context aggregation: Semantic-segmentation methods aggregate global context through encoder-decoder architectures or pyramid pooling.Hourglass architectures integrate top-down and bottom-up information, while pyramid pooling captures information across spatial scales.
- PSMNet: PSMNet applies multiscale whole-image context aggregation to stereo matching for depth estimation.Its pyramid stereo matching network is presented as an adaptation of context-aggregation experience from semantic segmentation.
3. Pyramid Stereo Matching Network
PSMNet combines pyramid pooling for global and local context with 3D CNN cost-volume regularization to improve stereo correspondence in ill-posed regions. Its pipeline forms a 4D cost volume from fused left-right features, processes it with stacked hourglass networks, and regresses continuous disparity.
- PSMNet combines an SPP module for global context with a stacked hourglass module for cost-volume regularization.
- The feature-extraction pipeline uses residual blocks and dilated convolution to enlarge receptive fields before SPP gathers context information.
- SPP concatenates representations from fixed-size pooling regions at multiple scales, then upsamples and combines them into final feature maps.
- Left and right SPP features are concatenated across disparity levels to form a 4D height×width×disparity×feature-size cost volume.
- The stacked hourglass architecture repeatedly performs top-down and bottom-up processing, produces three disparity outputs, and uses intermediate supervision during training.
- Disparity regression converts predicted costs into a continuous disparity map by weighting each disparity with its softmax probability.
- PSMNet uses smooth L1 loss over labeled pixels, with N denoting labeled-pixel count, d ground-truth disparity, and d̂ predicted disparity.
4. Experiments
PSMNet was evaluated on Scene Flow, KITTI 2012, and KITTI 2015 using an end-to-end architecture and ablation studies. It achieved strong benchmark performance, including 2.32% overall three-pixel-error on KITTI 2015 and 1.89% on KITTI 2012.
- Evaluation setup: PSMNet was evaluated on Scene Flow, KITTI 2012, and KITTI 2015, with additional ablations of its architecture and loss weights.The ablations examined dilated convolution, pyramid-pooling levels, stacked hourglass 3D CNNs, and loss-weight combinations.
- Ablation studies: The best architectural setting combined dilated convolution, spatial pyramid pooling, and stacked hourglass 3D CNNs, yielding a 1.83% error rate on KITTI 2015 validation.The stacked hourglass design significantly outperformed the basic 3D CNN when combined with the other components.
- Ablation studies: Loss weights of 0.5 for Loss 1, 0.7 for Loss 2, and 1.0 for Loss 3 produced a 1.98% error rate on the KITTI 2015 validation set.The stacked hourglass 3D CNN provides three outputs for training, enabling comparison of different loss-weight combinations.
- KITTI results: 2.32% overall three-pixel-error was reported for PSMNet on the KITTI 2015 test leaderboard presented on March 18, 2018.The evaluation considered all pixels, non-occluded pixels, and background, foreground, and all regions.
- KITTI results: PSMNet produced more robust disparity estimates than GC-Net and MC-CNN in illustrated KITTI 2015 examples, particularly in ill-posed regions and fence areas.The qualitative comparison used disparity maps and corresponding error maps reported by the KITTI evaluation server.
- Scene Flow and KITTI 2012 results: PSMNet outperformed other state-of-the-art methods on the Scene Flow test set and obtained a 1.89% overall three-pixel-error on the KITTI 2012 leaderboard.The Scene Flow comparison included CRL, DispNetC, and GC-Net; KITTI 2012 qualitative results highlighted car-window and wall regions.
5. Conclusions
PSMNet addresses ill-posed stereo correspondence by combining global-context aggregation with 3D cost-volume regularization. It outperformed other state-of-the-art methods and ranked first on the KITTI 2012 and 2015 leaderboards before March 18, 2018.
- Conclusion: PSMNet combines spatial pyramid pooling and stacked hourglass 3D CNN modules to exploit context and regularize the cost volume.The SPP module forms the cost volume from feature maps at different levels, while repeated top-down and bottom-up processes regularize it.
- Conclusion: PSMNet outperformed other state-of-the-art methods and ranked first on the KITTI 2012 and 2015 leaderboards before March 18, 2018.The conclusion also reports reduced errors in ill-posed regions.
- Conclusion: The estimated disparity maps demonstrated reduced errors in ill-posed regions.The conclusion identifies this behavior as an observed outcome of the proposed architecture.