Source-linked AI summary
Learning monocular depth estimation with unsupervised trinocular assumptions
Matteo Poggi, Fabio Tosi, Stefano Mattoccia
TL;DR
Unsupervised monocular depth estimation commonly uses binocular stereo supervision, but stereo geometry introduces artifacts near occlusions and image borders. This paper trains 3Net with a trinocular assumption and proposes interleaved training so binocular datasets can provide that supervision; experiments report state-of-the-art performance and robust monocular depth estimation.
Problem
Binocular stereo supervision for unsupervised monocular depth estimation inherits artifacts near occlusions and image borders, while real trinocular training data are uncommon.
Method
3Net predicts disparities from a central image toward left and right views, while interleaved role-swapping trains this trinocular formulation from conventional binocular pairs.
Results
3Net outperforms unsupervised techniques known in the literature and establishes itself as state-of-the-art on the reported KITTI evaluations.
Takeaways & Limitations
The method learns a trinocular representation from binocular data and is also suitable for image synthesis purposes.
Takeaways & Limitations
The approach assumes three horizontally aligned views conceptually, but real trinocular rigs would likely train a more accurate model because they provide more images than binocular rigs.
Abstract
from arXiv · showhide
Obtaining accurate depth measurements out of a single image represents a fascinating solution to 3D sensing. CNNs led to considerable improvements in this field, and recent trends replaced the need for ground-truth labels with geometry-guided image reconstruction signals enabling unsupervised training. Currently, for this purpose, state-of-the-art techniques rely on images acquired with a binocular stereo rig to predict inverse depth (i.e., disparity) according to the aforementioned supervision principle. However, these methods suffer from well-known problems near occlusions, left image border, etc inherited from the stereo setup. Therefore, in this paper, we tackle these issues by moving to a trinocular domain for training. Assuming the central image as the reference, we train a CNN to infer disparity representations pairing such image with frames on its left and right side. This strategy allows obtaining depth maps not affected by typical stereo artifacts. Moreover, being trinocular datasets seldom available, we introduce a novel interleaved training procedure enabling to enforce the trinocular assumption outlined from current binocular datasets. Exhaustive experimental results on the KITTI dataset confirm that our proposal outperforms state-of-the-art methods for unsupervised monocular depth estimation trained on binocular stereo pairs as well as any known methods relying on other cues.
1. Introduction
The paper motivates unsupervised monocular depth estimation as a cheaper, more flexible alternative to active sensing and stereo or motion-based approaches. It introduces 3Net, which uses trinocular training assumptions to reduce binocular artifacts while exploiting conventional datasets.
- Active depth sensors face practical limitations: Kinect is unsuitable for sunlit outdoor scenes, while LIDAR is expensive and cumbersome.
- Passive monocular depth estimation is attractive because standard cameras are cheap, lightweight, and usable indoors and outdoors.
- Unsupervised methods replace scarce ground-truth depth labels with image-reconstruction losses derived from multiple viewpoints.
- Stereo-trained monocular methods inherit artifacts near occlusions and image borders, motivating a trinocular training formulation centered on the reference image.
- 3Net learns from left and right viewpoints around a central image, and interleaved training enables this assumption using standard binocular datasets.
- The work positions its approach against supervised depth estimation, stereo-based unsupervised methods, motion-based methods, and earlier multi-baseline stereo systems.
3. Method overview
3Net trains a central-image depth network to predict disparities toward both neighboring views, reducing artifacts associated with binocular supervision. Because real trinocular datasets are uncommon, interleaved role-swapping lets binocular pairs train the full trinocular representation.
- Method overview: The framework replaces ground-truth depth supervision with geometry-based image reconstruction, using predicted depth to project images across viewpoints.
- Trinocular assumption and network design: Binocular training can create disparity artifacts where pixels lack direct matches, especially near occlusions and image borders.
- Trinocular assumption and network design: 3Net processes the central image and predicts disparities toward both left and right views, which are warped back to generate reconstruction losses.
- Trinocular assumption and network design: Two separate decoders estimate the two disparity directions, adding negligible overhead because the encoder dominates computation.
- Interleaved training for binocular images: Interleaved training first treats a binocular pair as left-center and then as center-right, optimizing the corresponding decoder and shared encoder outputs.
- Interleaved training for binocular images: Each stereo iteration trains all framework outputs, whose matching baselines make the learned disparity pairs consistent and combinable.
4. Implementation details
3Net uses an encoder-decoder design with separate decoders for left- and right-view disparity representations, trained with reconstruction, smoothness, and consistency losses. Experiments compare it with Godard et al.'s binocular baseline across KITTI training protocols and post-processing settings.
- Network architecture: 3Net uses an encoder-decoder architecture with two decoders learning disparity representations relative to left and right viewpoints.Each decoder produces outputs at four scales.
- Inference and post-processing: The final disparity map merges left and right decoder outputs using the post-processing procedure applied in Godard et al.The experimental comparison includes raw and post-processed outputs with one or two network forwards.
- Training losses: Training minimizes appearance, disparity-smoothness, and left-right consistency-check terms.The appearance term combines SSIM and L1 reconstruction losses, while smoothness is edge-aware.
- Training protocol: The interleaved training schedule splits the objective into two sub-losses deployed during two different phases.
- Training protocol: Models are trained on KITTI or CityScapes followed by KITTI fine-tuning, using the KITTI and Eigen splits for comparison with unsupervised depth-estimation methods.The reported training sets contain about 23k, 29k, and 22.6k stereo pairs, respectively.
- Evaluation: On KITTI 2015, 3Net is compared with Godard et al. using VGG encoders.The comparison evaluates the two methods on the KITTI 2015 training dataset.
5. Experimental results
Across KITTI evaluations, 3Net consistently improves unsupervised monocular depth estimation by enforcing a trinocular assumption through interleaved training on binocular data.
- Evaluation setup: The evaluation reports four depth-error metrics, three accuracy scores, and D1-all for KITTI stereo and Eigen-split experiments.Depth errors are lower-is-better, while accuracy scores are higher-is-better; D1-all measures disparity errors larger than 3.
- KITTI split: On KITTI-only training, 3Net outperforms Godard et al. on every metric except D1-all with a single forward.With post-processing and two forwards, 3Net again outperforms Godard et al. on all metrics except D1-all.
- KITTI split: With CityScapes pre-training, 3Net surpasses Godard et al. on all metrics without post-processing and ties only the two highest δ scores after post-processing.The comparison uses the same training framework and evaluates both models with and without post-processing.
- Training strategy: The interleaved strategy uses binocular pairs to enforce the trinocular hypothesis, avoiding dependence on a real trinocular training rig.A real trinocular rig could probably train a more accurate model because it would provide more images, but the proposed comparison uses binocular data for fairness.
- Eigen split: With CS+E training, 3Net outperforms Godard et al. under equal conditions, and 3Net ResNet50 + pp outmatches all known unsupervised depth-from-mono frameworks.The paper reports that even 3Net VGG exceeds Godard et al.’s ResNet50 configuration in one comparison.
6. View synthesis
At test time, 3Net uses its learned trinocular representation to synthesize two horizontally aligned views from one image and derive disparity at virtual baselines.
- View generation: Given a central test image, 3Net estimates left and right disparities and warps the image toward two new viewpoints.The resulting views form a horizontally aligned trinocular setup even though only the central image is real.
- Virtual baselines: Traditional stereo applied to the generated views produces disparity maps for narrow and wide virtual baselines.The same reference frame is paired with different targets, yielding distinct disparity ranges.
- Virtual baselines: The narrow baseline is learned from the KITTI acquisition rig, whereas the wide baseline is inherited from the trinocular assumption without existing in the training set.The different baselines are visible through their disparity ranges in the qualitative visualization.
- Potential applications: The synthesized views could support augmented stereo-pair generation for training disparity CNNs or improving single-view stereo methods.These applications are presented as conceivable future developments enabled by the learned representation.
7. Conclusions
The paper introduces unsupervised monocular depth training with a trinocular assumption and an interleaved protocol that works with binocular datasets. 3Net improves reported state-of-the-art results and supports view synthesis.
- Method: The proposed methodology enforces a trinocular assumption to reduce limitations associated with binocular stereo supervision.The network predicts inverse-depth representations from the middle frame using the available viewpoints.
- Training protocol: Interleaved training applies the trinocular protocol to traditional binocular datasets, enabling fair comparison with prior methods.The paper presents this protocol as a solution to the scarcity of trinocular views.
- Results and implications: 3Net outperforms the unsupervised techniques considered and is presented as state-of-the-art for unsupervised monocular depth estimation.The conclusion also identifies image synthesis as a use of the learned trinocular representation.
Supplementary material
The supplementary material adds technical explanations and experiments supporting the main paper.
- Supplementary contents: The supplementary material details the training losses, disparity generation and post-processing procedures, Eigen-split experiments with a 50-meter maximum depth, and additional results.It is organized into Sections 8 through 11.
8. Training losses
Training uses a multi-scale objective combining appearance reconstruction, disparity smoothness, and left-right consistency. These terms respectively measure image agreement, encourage structure-aware smoothness, and align disparities across directions.
- The global loss combines appearance, smoothness, and left-right consistency terms with weights βap = βlcr = 1 and βds = 0.1.
- Appearance loss measures reconstruction error using a weighted combination of an SSIM-based score and pixel-intensity L1 distance.
- Smoothness loss propagates similar disparities in low-textured regions while discouraging smoothness across strong image gradients.
- Left-right consistency loss minimizes the L1 distance between reference-to-target disparity and the corresponding warped target-to-reference disparity.
9. Depth computation and post-processing
The method combines left- and right-view disparity maps through post-processing after multiple network forwards. Normalized pixel coordinates determine the blending weights for the intermediate maps.
- The final output map dc is constructed by combining the left-to-right and right-to-left disparity predictions dcl and dcr.
- The blending weight ω is defined using normalized pixel coordinates i and j.
- The post-processing performs two forwards to generate and combine intermediate disparity maps.
- The right disparity post-processing uses pp = ω · dcr + (1 −ω) · ˆdcr.
- The left disparity post-processing uses pp = ω · ˆdcl + (1 −ω) · dcl.
10. Depth estimation: additional experiments with 50m cap
An additional Eigen-split evaluation capped depth at 50 meters and compared the best 3Net model with prior methods. The results again favored the proposed technique over all competitors.
- 3Net ResNet50 + pp trained on CS+E was superior to all competitors in the additional Eigen-split evaluation capped at 50 meters.
11. View synthesis and multi-baseline stereo
The paper evaluates view synthesis, multi-baseline stereo, and runtime using KITTI examples and comparisons with prior models. Qualitative results show reduced occlusion artifacts, while runtime analysis highlights an accuracy-speed advantage for single-forward 3Net ResNet50 on CPU.
- Evaluation setting: Table 3 evaluates KITTI Eigen-split methods using temporal, stereo, or combined stereo-plus-temporal supervision with a 50m maximum depth.
- View synthesis and multi-baseline stereo: The qualitative evaluation presents generated views, 3Net disparity maps, and SGM disparities from three stereo pairs formed by one real and two synthetic views.
- View synthesis and multi-baseline stereo: Artifacts near depth discontinuities and occlusions in dcl and dcr are greatly reduced in the final output map dc.
- Runtime analysis: On high-end GPUs, Godard et al. and 3Net require 0.09–0.11 seconds for either single or double forward execution.
- Runtime analysis: On CPU, the second 3Net encoder adds about 50% overhead, while two forwards usually double runtime.
- Runtime analysis: 3Net ResNet50 with a single forward is more accurate and faster than Godard et al. ResNet50 with two forwards.