Self-Supervised Learning for Stereo Matching with Self-Improving Ability

• Ground-truth depth maps are laborious to capture, sometimes unavailable, and often noisy, limiting their use as training signals for deep stereo matching.Source: §1 · p. 1
• Traditional stereo methods avoid ground-truth maps but rely on manually designed matching, aggregation, optimization, and refinement modules.Source: §1 · p. 1
• The proposed end-to-end framework trains without ground-truth disparity maps, using left-to-right and right-to-left stereo warping errors to drive learning.Source: §1 · p. 1
• 3D feature-volume regularization and a left-right consistency loss address trivial photometric solutions and textureless regions.Source: §1 · p. 1
• The network adapts to new scenarios, lighting conditions, and camera settings, and outperforms traditional and supervised deep-learning methods on KITTI and Middlebury examples.Figure 1 reports D1-all using a 0.5 pixel threshold; Figure 2 reports D1-all using a 3 pixel threshold.Source: §1 · p. 1, Figure 1 · p. 2, Figure 2 · p. 2

Traditional Stereo Matching

• Traditional stereo methods use local or global matching and optimization, often requiring high computational complexity for global formulations.Source: §2 · pp. 2–3

Deep Stereo Matching

• Deep stereo methods learn feature correspondences, regularization, or dense disparity prediction with convolutional neural networks.Source: §2 · p. 3
• End-to-end deep stereo methods commonly formulate disparity estimation as supervised regression or multi-class classification.Source: §2 · p. 3

Unsupervised monocular depth learning

• Unsupervised monocular depth methods use stereo images or video to minimize image reconstruction error through parametric image warping.Source: §2 · p. 3

Self-supervised formulation

• The network predicts left and right dense disparity maps from rectified stereo images and trains without ground-truth disparity supervision.Warped-image discrepancies between each reconstructed view and its observed counterpart supervise the learned function.Source: §3.1 · p. 3, §3 · p. 3

Network architecture

• Its five modules are feature extraction, feature-volume generation, 3D feature matching, soft argmin, and image warping.The architecture follows the traditional stereo pipeline through a network realization.Source: §3.1 · pp. 3–4, Figure 3 · p. 4

Feature volume construction

• Learned left and right features are assembled across candidate disparities into cross feature volumes rather than concatenating precomputed matching costs.The volume exhausts disparity levels over a predefined range and contains height, width, disparity, and feature dimensions.Source: §3.1.2 · pp. 4–5, Figure 4 · p. 4

3D feature matching and soft argmin

• 3D convolutions regularize correlations across spatial and disparity dimensions before soft argmin projects the regularized volume to a 2D disparity map.The soft argmin selects disparities using predicted costs and a softmax over the disparity dimension.Source: §3.1.3 · p. 5, §3.1.4 · p. 5

Loss function

• The loss combines photometric reconstruction, disparity smoothness, left-right consistency, and a maximize-depth heuristic.The photometric term uses image and gradient differences together with SSIM, while smoothness uses TGV weighted by image gradients.Source: §3.1.5 · p. 5

Self-improving mode

• The self-improving mode fine-tunes network parameters while testing on new stereo images, enabling adaptation to unseen scenarios.The traditional mode freezes parameters during testing, whereas self-improving mode updates them continuously.Source: §4 · p. 6

Learning curves

• The self-improving curves show similar trends for network performance and warping error over training iterations.The network achieves reasonable results within 1500 iterations.Source: Figure 7 · p. 7

Self-supervision

• The network uses input stereo pairs as self-supervision signals and iteratively improves without ground-truth depth maps during training.This supports automatic generalization to unseen images.Source: §4 · p. 7

Learning from scratch

• After about 1000–1500 iterations, a randomly initialized network can begin predicting accurate depth maps through unsupervised self-learning.The evaluation uses D1 all and image warping error without feeding evaluation signals back into the network.Source: §4 · p. 7

Cross-dataset adaptation

• After 100 online-tuning iterations on Middlebury, the average error decreased from 21.17% to 13.67% at the 0.5 threshold and from 10.80% to 6.07% at the 1 threshold.The comparison is between a KITTI-pretrained model and the same model after updating with the new dataset.Source: §4 · p. 7

Experimental setup

• The network is trained end-to-end on rectified stereo pairs without post-processing or ground-truth depth maps, and evaluated on KITTI 2012, KITTI 2015, and Middlebury.The experiments report both qualitative and quantitative results.Source: §5 · p. 7

Implementation details

• Inference takes about 0.8 second per 384 × 1280 stereo pair when parameters are not updated.The reported time includes data loading and transfer.Source: §5.1 · p. 7
• The network can produce a reasonable result within 1500 iterations when trained from scratch.Training uses a batch size of 1, 256 × 512 inputs, and a disparity range of 160.Source: §5.1 · pp. 7, 9

Middlebury

• Middlebury evaluation compares the KITTI-trained model, online-tuned results after 100 iterations, SPS-st, MC-CNN, and MeshStereo.MeshStereo is tuned on Middlebury, while MC-CNN uses a KITTI-trained model and KITTI-tuned post-processing.Source: Table 1 · p. 9, Table 4 · p. 10

KITTI

• KITTI-2015 evaluation includes densely labeled vehicle regions and highly reflected car-glass areas, creating a bias toward those regions.For transparent surfaces, the selected value may represent window depth rather than real disparity.Source: §5.2 · p. 9

Middlebury

• On Middlebury, the method improves over SPS-st and MC-CNN by 46.60% and 152.16% respectively at the 0.5-pixel threshold on non-occluded pixels.Compared with the Middlebury-tuned conventional method, it performs 31.75% and 14.66% better at 0.5- and 1-pixel thresholds.Source: §5.3 · p. 9

Cross-dataset comparison

• The authors conclude that traditional learning-free methods require dataset-specific manual parameter tuning, whereas their method self-adapts to different scenarios.The comparison concerns generalization across different datasets and scenarios.Source: §5.3 · p. 10

• The network is trained end-to-end using input stereo image pairs without ground-truth depth maps.Source: §6 · p. 10
• A novel training loss exploits the loop constraint in image warping and handles textureless areas.Source: §6 · p. 10
• The network supports online learning on new, never-seen-before images and self-improves by adapting to new imageries without ground-truth labeling.Source: §6 · p. 10
• Experiments show superior performance over traditional learning-free methods and recent supervised deep-learning methods.Source: §6 · p. 10