Source-linked AI summary
Cascade Residual Learning: A Two-stage Convolutional Neural Network for Stereo Matching
Jiahao Pang, Wenxiu Sun, Jimmy SJ. Ren, Chengxi Yang, Qiong Yan
TL;DR
CNN stereo matching improves on traditional methods but still struggles to produce accurate disparities in inherently ill-posed regions. The paper proposes a two-stage cascade CNN whose first stage creates detailed initial disparities and whose second stage performs multiscale residual refinement. The approach achieves state-of-the-art stereo matching performance and ranks first on the KITTI 2015 stereo benchmark.
Problem
CNN-based stereo matching still struggles to generate high-quality disparities for inherently ill-posed regions.
Method
CRL uses a two-stage CNN: the first produces fine-detail initial disparities, while the second explicitly refines them with supervised residual signals across multiple scales.
Results
CRL achieves state-of-the-art stereo matching performance and ranks first on the KITTI 2015 stereo benchmark.
Takeaways & Limitations
Residual learning provides effective disparity refinement and benefits optimization of the overall two-stage network.
Takeaways & Limitations
The paper leaves robust left-right consistency checking for future work because existing CNN disparity estimates require highly reliable added mechanisms for further improvement.
Abstract
from arXiv · showhide
Leveraging on the recent developments in convolutional neural networks (CNNs), matching dense correspondence from a stereo pair has been cast as a learning problem, with performance exceeding traditional approaches. However, it remains challenging to generate high-quality disparities for the inherently ill-posed regions. To tackle this problem, we propose a novel cascade CNN architecture composing of two stages. The first stage advances the recently proposed DispNet by equipping it with extra up-convolution modules, leading to disparity images with more details. The second stage explicitly rectifies the disparity initialized by the first stage; it couples with the first-stage and generates residual signals across multiple scales. The summation of the outputs from the two stages gives the final disparity. As opposed to directly learning the disparity at the second stage, we show that residual learning provides more effective refinement. Moreover, it also benefits the training of the overall cascade network. Experimentation shows that our cascade residual learning scheme provides state-of-the-art performance for matching stereo correspondence. By the time of the submission of this paper, our method ranks first in the KITTI 2015 stereo benchmark, surpassing the prior works by a noteworthy margin.
1. Introduction
Stereo matching offers a cost-effective route to dense depth, but CNN-based methods still struggle in ill-posed regions. The proposed two-stage cascade residual framework refines detailed initial disparities and achieves leading benchmark performance.
- Motivation: Stereo cameras estimate dense scene depth by matching corresponding pixels along scan-lines, with depth computed as f·l/d.Here, d is disparity; because depth is inversely proportional to disparity, accurate dense disparity is the primary target.
- Motivation: CNN-based stereo matching improves accuracy and speed over traditional approaches but remains unreliable in occlusions, repeated patterns, and textureless regions.Occlusions lack a counterpart, while repeated or textureless areas admit multiple plausible correspondences.
- Proposed approach: CRL uses two hour-glass CNN stages: the first produces fine-grained initial disparities, and the second predicts multiscale residuals to rectify them.The final disparity combines the outputs of the two stages, while residuals are explicitly supervised against the ground-truth difference.
- Proposed approach: Residual learning makes refinement easier than directly predicting disparity and can preserve an optimal initialization by producing zero residual.Unlike sequential residual blocks, CRL explicitly supervises residuals across multiple scales.
- Results: The end-to-end CRL network integrates matching-cost computation, cost aggregation, disparity optimization, and refinement, ranking first on KITTI 2015.The paper reports state-of-the-art stereo matching results from the combined two-stage design.
2. Related Works
Prior stereo-matching research spans traditional multistage optimization and CNN-based approaches focused on matching costs, regularity, or end-to-end disparity learning. CRL extends end-to-end learning with a two-stage residual refinement scheme for difficult regions and reports state-of-the-art estimates with acceptable runtime.
- Prior approaches: Traditional stereo matching uses matching-cost computation, cost aggregation, disparity optimization, and disparity refinement.CNN methods estimate disparities corresponding to some or all of these stages.
- Prior approaches: Matching-cost learning replaces hand-crafted metrics such as SAD, NCC, and Birchfield-Tomasi cost with CNN-based patch similarity estimation.Representative approaches use Siamese features or CNNs for pairwise matching classification.
- Prior approaches: Regularity-learning methods impose smoothness constraints or use inconsistent-pixel replacement and higher-level vision tasks to regularize disparity.Examples incorporate texture and edge information, semantic segmentation, or object recognition.
- Prior approaches: End-to-end disparity learning trains CNNs directly from stereo inputs, including DispNet and later methods such as GC-NET.These approaches belong to the category adopted by the present work.
- CRL: CRL addresses inaccurate disparities in ill-posed regions with a high-quality first-stage initialization and multiscale residual refinement, achieving state-of-the-art estimates with acceptable runtime.The authors describe this as a two-stage CNN scheme motivated by iterative refinement methods.
3. Cascade Residual Learning
CRL uses a two-stage CNN: DispFulNet produces full-resolution initial disparities, while DispResNet refines them through multiscale residual learning. The stages are trained and coupled to improve disparity refinement while retaining acceptable execution time.
- Two-stage disparity computation: CRL cascades DispFulNet and DispResNet, using the first stage for initialization and the second for residual-based refinement.The architecture is motivated by the difficulty of producing accurate disparities in ill-posed regions.
- Two-stage disparity computation: DispFulNet adds up-convolution modules to DispNetC, producing disparity estimates at the full input-image resolution.Its outputs provide more details and sharper object-boundary transitions for subsequent refinement.
- Two-stage disparity computation: The second network receives the stereo pair, initial disparity, warped right image, and photometric error, with differentiable warping enabling end-to-end training.The warped image is synthesized from the right image using the initial left-image disparity.
- Multiscale residual learning: Residual learning lets the second stage focus on nonlinear corrections rather than directly relearning disparity, and can output zero when initialization is optimal.The paper reports more gain than direct disparity learning and better overall finetuning behavior than DispNetS.
- Multiscale residual learning: DispResNet predicts residual signals at multiple scales and adds them to downsampled first-stage disparities to form refined disparities.Disparity estimates are supervised across S + 1 scales with ℓ1 losses.
- Design scope and efficiency: Adding more cascade stages would increase computational cost and memory consumption, while the proposed two-stage network produces a KITTI 2015 disparity image in 0.47 sec on an Nvidia GTX 1080 GPU.The authors frame the two stages as complementary roles: full-resolution initialization followed by residual correction.
4. Experiments
The experiments evaluate CRL across synthetic, indoor, and real-world stereo datasets, comparing its architecture and performance with related networks and state-of-the-art methods. CRL achieves the best reported objective qualities across all three datasets and ranks first on KITTI 2015.
- Experimental Settings: CRL is evaluated on FlyingThings3D, Middlebury 2014, and KITTI 2015, with Middlebury used only for testing.FlyingThings3D is synthetic, Middlebury contains high-resolution indoor scenes, and KITTI provides real-world dynamic street views.
- Architecture Comparisons: The experiments compare CRL with plain and two-stage variants using DispNetC, DispFulNet, DispNetS, and DispResNet.The comparison includes separate training and overall finetuning configurations.
- Architecture Comparisons: DispFulNet provides higher accuracy than DispNetC, while DispResNet provides extra gain over direct-learning DispNetS.Appending a second stage improves results, and residual learning yields additional improvement relative to the direct-learning alternative.
- Architecture Comparisons: Overall finetuning improves DispResNet in most cases but worsens DispNetS; the paper attributes the difference to residual learning being less prone to overfitting.The exception is Middlebury, which is not used for training.
- Architecture Comparisons: CRL with overall finetuning achieves the best objective qualities on all three datasets.The model is then used for further comparisons with other state-of-the-art approaches.
- Comparisons with Other Methods: On FlyingThings3D, CRL achieves the best performance on both evaluation metrics and produces sharp disparity estimates with fine details in highlighted regions.The visual comparisons report improvements in object boundaries, occlusions, and textureless regions.
- Comparisons with Other Methods: On KITTI 2015, CRL ranks first with an overall 3PE of 2.67%, versus 2.87% for GC-NET, while using about half GC-NET's runtime.The comparison uses the online leaderboard's three-pixel-error values.
- Discussions: End-to-end CNN stereo methods, including CRL, require substantial ground-truth training data, while real depth collection is costly and synthetic data does not fully reflect real environments.The paper identifies this as a limitation of the training setting rather than a failure specific to CRL.
5. Conclusions
The paper concludes that cascade residual learning combines a detail-preserving first-stage disparity estimate with multi-scale residual refinement, achieving state-of-the-art stereo matching and first place on the KITTI 2015 leaderboard.
- 5. Conclusions: Cascade residual learning uses two stages: an initial disparity-producing network followed by residual-based refinement.The approach is described as a cascade CNN architecture, with residual learning improving refinement and overall optimization.
- 5. Conclusions: The first stage produces disparity images with fine details, while the second stage rectifies them using residual signals across multiple scales.This design targets high-quality disparity estimation in inherently ill-posed regions.
- 5. Conclusions: Visual comparisons report that CRL produces the sharpest and finest disparity images among the compared two-stage architectures.The comparisons include three two-stage network designs.
- 5. Conclusions: Our approach ranks first on the KITTI 2015 stereo online leaderboard.The leaderboard table reports three-pixel-error results for leading submissions as of August 2017.