Source-linked AI summary
Self-Supervised Learning for Stereo Matching with Self-Improving Ability
Yiran Zhong, Yuchao Dai, Hongdong Li
TL;DR
Deep stereo matching often relies on ground-truth disparity maps that are laborious to obtain and may be unavailable or noisy. The paper trains an end-to-end network from stereo pairs using bidirectional image-warping errors, with regularization and consistency mechanisms to resolve ambiguities. It reports superior performance to traditional and supervised deep-learning methods while supporting adaptation to unseen imagery.
Problem
Deep stereo methods commonly require ground-truth depth maps, which are laborious to capture, not always available, and often noisy.
Method
An end-to-end network learns disparity from stereo pairs by minimizing image-warping error, using feature-volume regularization and left-right consistency constraints.
Results
Experiments show superior performance to traditional and supervised deep-learning stereo methods on KITTI and Middlebury datasets.
Takeaways & Limitations
The network can operate in online self-improving mode, adapting to new, never-seen-before images without ground-truth labeling.
Takeaways & Limitations
KITTI-2015 evaluation is biased toward densely labeled vehicles and reflected car-glass regions, affecting ambiguous transparent-surface disparities.
Abstract
from arXiv · showhide
Exiting deep-learning based dense stereo matching methods often rely on ground-truth disparity maps as the training signals, which are however not always available in many situations. In this paper, we design a simple convolutional neural network architecture that is able to learn to compute dense disparity maps directly from the stereo inputs. Training is performed in an end-to-end fashion without the need of ground-truth disparity maps. The idea is to use image warping error (instead of disparity-map residuals) as the loss function to drive the learning process, aiming to find a depth-map that minimizes the warping error. While this is a simple concept well-known in stereo matching, to make it work in a deep-learning framework, many non-trivial challenges must be overcome, and in this work we provide effective solutions. Our network is self-adaptive to different unseen imageries as well as to different camera settings. Experiments on KITTI and Middlebury stereo benchmark datasets show that our method outperforms many state-of-the-art stereo matching methods with a margin, and at the same time significantly faster.
1. Introduction
Stereo matching requires dense disparity maps, but deep methods commonly depend on costly and noisy ground-truth labels. This paper replaces those labels with bidirectional image-warping errors and addresses ambiguities through feature-space regularization and consistency checking.
- Ground-truth depth maps are laborious to capture, sometimes unavailable, and often noisy, limiting their use as training signals for deep stereo matching.
- Traditional stereo methods avoid ground-truth maps but rely on manually designed matching, aggregation, optimization, and refinement modules.
- The proposed end-to-end framework trains without ground-truth disparity maps, using left-to-right and right-to-left stereo warping errors to drive learning.
- 3D feature-volume regularization and a left-right consistency loss address trivial photometric solutions and textureless regions.
- The network adapts to new scenarios, lighting conditions, and camera settings, and outperforms traditional and supervised deep-learning methods on KITTI and Middlebury examples.Figure 1 reports D1-all using a 0.5 pixel threshold; Figure 2 reports D1-all using a 3 pixel threshold.
2. Related work
Related work spans traditional stereo optimization, supervised deep stereo networks, and image-warping approaches for unsupervised monocular depth. The paper positions self-supervised stereo matching as an end-to-end alternative that also supports online self-improvement.
- Traditional Stereo Matching: Traditional stereo methods use local or global matching and optimization, often requiring high computational complexity for global formulations.
- Deep Stereo Matching: Deep stereo methods learn feature correspondences, regularization, or dense disparity prediction with convolutional neural networks.
- Deep Stereo Matching: End-to-end deep stereo methods commonly formulate disparity estimation as supervised regression or multi-class classification.
- Unsupervised monocular depth learning: Unsupervised monocular depth methods use stereo images or video to minimize image reconstruction error through parametric image warping.
3. Our Method
The method learns dense disparity maps end-to-end without ground-truth maps by combining learned feature-volume matching with image-warping supervision. It uses 3D regularization, soft argmin, photometric losses, smoothness, and loop consistency to address ambiguities in self-supervised stereo matching.
- Self-supervised formulation: The network predicts left and right dense disparity maps from rectified stereo images and trains without ground-truth disparity supervision.Warped-image discrepancies between each reconstructed view and its observed counterpart supervise the learned function.
- Network architecture: Its five modules are feature extraction, feature-volume generation, 3D feature matching, soft argmin, and image warping.The architecture follows the traditional stereo pipeline through a network realization.
- Feature volume construction: Learned left and right features are assembled across candidate disparities into cross feature volumes rather than concatenating precomputed matching costs.The volume exhausts disparity levels over a predefined range and contains height, width, disparity, and feature dimensions.
- 3D feature matching and soft argmin: 3D convolutions regularize correlations across spatial and disparity dimensions before soft argmin projects the regularized volume to a 2D disparity map.The soft argmin selects disparities using predicted costs and a softmax over the disparity dimension.
- Loss function: The loss combines photometric reconstruction, disparity smoothness, left-right consistency, and a maximize-depth heuristic.The photometric term uses image and gradient differences together with SSIM, while smoothness uses TGV weighted by image gradients.
4. Self-improving Ability
The network can continue adapting during inference, using new stereo images as self-supervision rather than requiring ground-truth depth maps. Experiments show improvement from random initialization and online tuning across datasets.
- Self-improving mode: The self-improving mode fine-tunes network parameters while testing on new stereo images, enabling adaptation to unseen scenarios.The traditional mode freezes parameters during testing, whereas self-improving mode updates them continuously.
- Learning curves: The self-improving curves show similar trends for network performance and warping error over training iterations.The network achieves reasonable results within 1500 iterations.
- Self-supervision: The network uses input stereo pairs as self-supervision signals and iteratively improves without ground-truth depth maps during training.This supports automatic generalization to unseen images.
- Learning from scratch: After about 1000–1500 iterations, a randomly initialized network can begin predicting accurate depth maps through unsupervised self-learning.The evaluation uses D1 all and image warping error without feeding evaluation signals back into the network.
- Cross-dataset adaptation: After 100 online-tuning iterations on Middlebury, the average error decreased from 21.17% to 13.67% at the 0.5 threshold and from 10.80% to 6.07% at the 1 threshold.The comparison is between a KITTI-pretrained model and the same model after updating with the new dataset.
5. Experiment
The method is evaluated end-to-end without ground-truth depth maps on KITTI and Middlebury, including cross-dataset and qualitative comparisons. It achieves strong benchmark performance while adapting online, though KITTI-2015 evaluation emphasizes densely labeled vehicle regions and reflected surfaces.
- Experimental setup: The network is trained end-to-end on rectified stereo pairs without post-processing or ground-truth depth maps, and evaluated on KITTI 2012, KITTI 2015, and Middlebury.The experiments report both qualitative and quantitative results.
- Implementation details: Inference takes about 0.8 second per 384 × 1280 stereo pair when parameters are not updated.The reported time includes data loading and transfer.
- Implementation details: The network can produce a reasonable result within 1500 iterations when trained from scratch.Training uses a batch size of 1, 256 × 512 inputs, and a disparity range of 160.
- Middlebury: Middlebury evaluation compares the KITTI-trained model, online-tuned results after 100 iterations, SPS-st, MC-CNN, and MeshStereo.MeshStereo is tuned on Middlebury, while MC-CNN uses a KITTI-trained model and KITTI-tuned post-processing.
- KITTI: KITTI-2015 evaluation includes densely labeled vehicle regions and highly reflected car-glass areas, creating a bias toward those regions.For transparent surfaces, the selected value may represent window depth rather than real disparity.
- Middlebury: On Middlebury, the method improves over SPS-st and MC-CNN by 46.60% and 152.16% respectively at the 0.5-pixel threshold on non-occluded pixels.Compared with the Middlebury-tuned conventional method, it performs 31.75% and 14.66% better at 0.5- and 1-pixel thresholds.
- Cross-dataset comparison: The authors conclude that traditional learning-free methods require dataset-specific manual parameter tuning, whereas their method self-adapts to different scenarios.The comparison concerns generalization across different datasets and scenarios.
6. Conclusion
The paper presents an end-to-end deep stereo matching network trained from stereo image pairs alone, without ground-truth depth maps. Its loss exploits image-warping loop constraints and textureless-area handling, enabling online adaptation and superior performance to traditional and supervised deep-learning methods.
- The network is trained end-to-end using input stereo image pairs without ground-truth depth maps.
- A novel training loss exploits the loop constraint in image warping and handles textureless areas.
- The network supports online learning on new, never-seen-before images and self-improves by adapting to new imageries without ground-truth labeling.
- Experiments show superior performance over traditional learning-free methods and recent supervised deep-learning methods.