Source-linked AI summary
Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints
Reza Mahjourian, Martin Wicke, Anelia Angelova
TL;DR
The paper addresses unsupervised depth and ego-motion learning from monocular video without depth or multi-view supervisory signals. It combines 2D photometric losses with differentiable 3D point-cloud alignment and validity masks, improving estimates across datasets and enabling training from uncalibrated video. Its main limitation is that largely dynamic scenes can bias depth estimates for moving objects.
Problem
Depth and ego-motion learning typically relies on depth ground truth, multi-view video, or calibrated sensors, limiting supervision from varied monocular video.
Method
The method combines photometric reconstruction losses with a differentiable 3D loss that aligns point clouds from adjacent frames using estimated ego-motion, plus analytical validity masks.
Results
The approach consistently improves depth estimates, outperforms prior methods for depth and ego-motion, and ranks among top-performing methods when trained on uncalibrated video and evaluated on KITTI.
Takeaways & Limitations
Requiring only a single monocular video stream makes training on diverse sources possible, while depth can be produced from a single image at test time.
Takeaways & Limitations
Moving objects can be explained through incorrect depth, and the method does not explicitly handle largely dynamic scenes.
Abstract
from arXiv · showhide
We present a novel approach for unsupervised learning of depth and ego-motion from monocular video. Unsupervised learning removes the need for separate supervisory signals (depth or ego-motion ground truth, or multi-view video). Prior work in unsupervised depth learning uses pixel-wise or gradient-based losses, which only consider pixels in small local neighborhoods. Our main contribution is to explicitly consider the inferred 3D geometry of the scene, enforcing consistency of the estimated 3D point clouds and ego-motion across consecutive frames. This is a challenging task and is solved by a novel (approximate) backpropagation algorithm for aligning 3D structures. We combine this novel 3D-based loss with 2D losses based on photometric quality of frame reconstructions using estimated depth and ego-motion from adjacent frames. We also incorporate validity masks to avoid penalizing areas in which no useful information exists. We test our algorithm on the KITTI dataset and on a video dataset captured on an uncalibrated mobile phone camera. Our proposed approach consistently improves depth estimates on both datasets, and outperforms the state-of-the-art for both depth and ego-motion. Because we only require a simple video, learning depth and ego-motion on large and varied datasets becomes possible. We demonstrate this by training on the low quality uncalibrated video dataset and evaluating on KITTI, ranking among top performing prior methods which are trained on KITTI itself.
1. Introduction
The paper proposes unsupervised monocular-video learning of depth and ego-motion using temporal consistency, explicit 3D geometric constraints, photometric losses, and analytical masking. It targets broader training data by avoiding depth ground truth, multi-view video, and restrictive camera calibration.
- Motivation and setup: Supervised approaches require calibrated setups and often rely on imperfect depth sensors that introduce noise, artifacts, or temporal misalignment.These constraints limit the amount and variety of usable training data.
- Motivation and setup: Depth and ego-motion are learned from monocular videos using consistency and temporal coherence between consecutive frames, with camera intrinsics.The network predicts single-view depth and frame-pair ego-motion; adjacent-frame consistency supplies supervision.
- 3D geometric constraints: The method introduces a 3D loss that directly compares adjacent-frame point clouds after transforming them with estimated ego-motion.This complements 2D photometric losses by enforcing consistency in the inferred scene geometry.
- Validity masking: Analytical validity masks exclude pixels and depths that are not covered in the transformed view, avoiding losses on uninformative regions.Such regions can arise from parallax or objects entering or leaving the field of view.
- Broader training data: Training on uncalibrated handheld-phone bicycle videos and evaluating on KITTI demonstrates that the approach can learn from diverse monocular video sources.The reported setup trains the model only on the newly recorded videos before testing on KITTI.
- 3D geometric constraints: ICP is applied symmetrically in forward and backward directions, and its outputs generate gradients for improving depth and ego-motion estimates.The alignment operates on estimates from two consecutive frames.
2. Related Work
Prior work learns depth from sensors, stereo, or monocular video, but the paper distinguishes its approach by introducing differentiable 3D consistency losses for adjacent-frame geometry.
- Sensor-supervised learning: Early depth-learning methods commonly use depth-sensor supervision and dense-prediction architectures such as FCN or U-Net.These approaches treat depth estimation as a dense prediction problem.
- Stereo-based learning: Stereo-based methods learn depth from calibrated or rectified image pairs, sometimes with additional supervision such as depth or optical flow.Known stereo transformations make image reconstruction a source of supervision.
- Monocular-video learning: Monocular-video methods remove the need for stereo input, with some extensions also modeling object motion or optionally using ground-truth depth and optical flow.The paper positions its setup alongside these unsupervised monocular-video approaches.
- Paper distinction: This work differs by using differentiable 3D losses to establish consistency between the geometry of adjacent frames and improve depth and ego-motion estimation.The training process is explicitly extended from image-level relationships to three-dimensional structure.
3. Method
The method learns depth and ego-motion by combining differentiable image reconstruction with a 3D point-cloud alignment loss. Analytical validity masks exclude pixels that cannot be reliably reconstructed, while approximate ICP gradients support backpropagation through the 3D loss.
- Problem geometry: The model estimates depths for consecutive frames and camera ego-motion, then uses these estimates to project points and reconstruct adjacent images.Depth is projected into structured 3D point clouds, transformed by ego-motion, and mapped between image coordinates for differentiable warping.
- Principled masks: Analytical validity masks exclude pixels outside the target view or otherwise uncovered after camera motion from the reconstruction loss.The masks identify valid coordinates for reconstructing each frame and avoid penalizing areas with no useful correspondence.
- Image reconstruction loss: The photometric reconstruction loss compares reconstructed frames with their inputs, but its approximate differentiable warping can create noisy artifacts and oversmoothed predictions.The reconstruction process cannot fully model lighting, shadows, translucence, or reflections, so strong regularization is needed to reduce artifacts.
- 3D point-cloud alignment loss: The 3D loss directly compares adjacent-frame point clouds after transformation using ICP, rather than relying only on coordinate mappings from image reconstruction.ICP computes correspondences, a best-fit registration transform, and residual distances between corresponding points.
- 3D point-cloud alignment loss: Because ICP is nondifferentiable, the method approximates gradients from its computed transformations and residuals to update ego-motion and depth estimates.The ICP correction approximates the negative gradient for ego-motion, while residual directions approximate the negative gradient for depth; ignored second-order effects work well in practice.
- Additional image-based losses: Additional image-based losses include SSIM, and the overall objective combines reconstruction, 3D, smoothness, and SSIM terms.The SSIM term measures local image-patch similarity, while depth smoothness regularizes the estimated depth maps.
4. Experiments
Experiments on KITTI and an uncalibrated hand-held Bike video dataset evaluate depth, ego-motion, the 3D loss, and cross-dataset transfer. The proposed method improves depth and ego-motion estimates, while the 3D loss reduces artifacts, overfitting, and validation error.
- KITTI: On KITTI, the model is trained using only a single monocular video stream, while LIDAR point clouds and vehicle poses are reserved for evaluation.The split contains about 40k training frames, 4k validation frames, and 697 test frames.
- Depth estimation: 0.163 mean absolute relative depth error is achieved on KITTI when training only on KITTI, improving over Zhou et al. [32] at 0.208.The result is close to Godard et al. [11] at 0.148, which uses rectified stereo images with known camera baseline.
- Depth estimation: Pre-training on Cityscapes followed by KITTI fine-tuning produces significant depth improvements, with qualitative comparisons against Zhou et al. [32] shown in Figure 6.Mean inference time for a 128×416 input is 10.5 ms on a GeForce GTX 1080.
- 3D-loss evaluation: The 3D loss lowers validation depth error throughout training and reduces overfitting, whereas Cityscapes pre-training alone does not reduce overfitting or improve depth quality.Examples also show fewer artifacts in low-texture regions when the 3D loss is used.
- Ego-motion: On KITTI odometry, the proposed method significantly outperforms the unsupervised baseline and matches or outperforms supervised ORB-SLAM using the full video sequence.Depth and ego-motion are learned jointly, and their accuracy is inter-dependent.
- Learning from Bike videos: Training only on hand-held Bike videos transfers to KITTI without fine-tuning despite differences in field of view, distortion correction, scene layout, and camera motion.The transferred model is close in quality to Zhou et al. [32], and the 3D loss again improves learning and reduces overfitting.
- Ablations: Ablations identify the 3D loss and SSIM components as essential, while removing validity masks hurts performance.The relevant comparisons are summarized in Table 3 and Figure 12.
5. Conclusions and Future Work
The proposed unsupervised method learns depth and ego-motion from monocular video by aligning 3D structures across frames, while experiments demonstrate broad applicability and identify dynamic-scene limitations.
- The method learns depth from a single monocular video stream and can produce depth from a single image at test time.
- The approach explicitly incorporates 3D scene structure through a loss that aligns 3D geometries across different frames.
- Training can use diverse low-end-camera video sources because the method requires no rectification and is robust to lens distortions and lack of stabilization.
- When objects move between frames, the losses can explain their motion through biased depth estimates, and largely dynamic scenes are not explicitly handled.
- Principled masks can be extended to account for occlusions and disocclusions caused by viewpoint changes between adjacent frames.