Source-linked AI summary
Unsupervised Learning of Depth and Ego-Motion from Video
Tinghui Zhou, Matthew Brown, Noah Snavely, David G. Lowe
TL;DR
Monocular depth and camera motion estimation from unstructured video typically require supervision or reliable geometric pipelines. This paper jointly trains depth and pose networks with view synthesis on unlabeled videos, then applies them independently, achieving competitive KITTI depth and pose results while retaining assumptions about scene appearance and motion.
Problem
The paper addresses monocular depth and camera motion estimation from video without ground-truth depth or pose supervision.
Method
It jointly trains depth and camera pose networks through differentiable view synthesis, using predicted depth and relative pose to reconstruct target views from source views.
Results
On KITTI, unsupervised monocular depth performs comparably with supervised methods, while pose estimation outperforms same-input baselines but falls short of full-sequence ORB-SLAM.
Takeaways & Limitations
Unlabeled videos can supervise independently usable depth and pose models through a shared view-synthesis objective.
Takeaways & Limitations
The formulation assumes static scenes, no occlusion or disocclusion, and Lambertian surfaces, with violations potentially corrupting training gradients.
Abstract
from arXiv · showhide
We present an unsupervised learning framework for the task of monocular depth and camera motion estimation from unstructured video sequences. We achieve this by simultaneously training depth and camera pose estimation networks using the task of view synthesis as the supervisory signal. The networks are thus coupled via the view synthesis objective during training, but can be applied independently at test time. Empirical evaluation on the KITTI dataset demonstrates the effectiveness of our approach: 1) monocular depth performing comparably with supervised methods that use either ground-truth pose or depth for training, and 2) pose estimation performing favorably with established SLAM systems under comparable input settings.
1. Introduction
The paper introduces an unsupervised, end-to-end framework that learns scene depth and camera motion from unlabeled image sequences through view synthesis. Its coupled training objective encourages physically consistent intermediate predictions while producing independently usable depth and pose models.
- Training setup: Training uses only unlabeled image sequences from different viewpoints, without provided image poses.
- Outputs: The resulting models operate independently at test time: one predicts depth from a single view and the other estimates camera pose from multiple views.
- Framework: The system learns ego-motion and scene structure by explaining observed image sequences through predicted camera motion and per-pixel depth.Camera motion is parameterized as 6-DoF transformation matrices, while scene structure is represented as depth maps under a reference view.
- Learning principle: The framework formulates view synthesis as a convolutional-network inference procedure that jointly learns depth and camera pose.The view-synthesis task supplies the training signal for the intermediate geometric predictions.
- Evaluation: Empirical evaluation on KITTI demonstrates effectiveness for both single-view depth and camera pose estimation.
2. Related work
The work builds on structure-from-motion, warping-based view synthesis, single-view 3D learning, and video representation learning. It distinguishes itself by using view synthesis to distill geometric reasoning while emphasizing unsupervised learning from image observations.
- Structure from motion: Traditional structure-from-motion estimates scene structure and camera motion but can struggle with low texture, complex geometry, thin structures, and occlusions.
- Warping-based view synthesis: Warping-based view-synthesis methods transform input views using intermediate geometry or correspondence, forcing models to learn representations such as depth or flow.
- Learning single-view 3D from registered 2D views: Prior single-view 3D methods learn depth from registered 2D observations, including calibrated stereo supervision, stereoscopic footage, or left-right consistency.
- Learning formulation: Unlike direct structure-and-motion optimization, CNN-based methods update network parameters across batches and can learn an implicit prior from large image corpora.
- Unsupervised learning from video: Video-based self-supervised learning uses pretext tasks including ego-motion estimation, tracking, temporal coherence, temporal order verification, and object-motion prediction.
- Concurrent work: Concurrent work jointly trains depth, camera motion, and scene motion, whereas this paper focuses on the unsupervised aspect of the framework.
3. Approach
The approach jointly trains monocular depth and camera-pose networks from unlabeled video by using differentiable view synthesis and photometric reconstruction as supervision. Predicted depth and relative poses warp source views into a target view, while explainability, smoothness, and multi-scale objectives address modeling and optimization limitations.
- 3. Approach: The framework jointly trains single-view depth and camera-pose CNNs from unlabeled video, while allowing independent test-time use.Training examples are short image sequences captured by a moving camera.
- 3.1. View synthesis as supervision: View synthesis supervises training by reconstructing a target view from predicted per-pixel depth, relative camera pose, and nearby source views.The reconstructed target is compared using a photometric reconstruction loss.
- 3.3. Modeling the model limitation: The method assumes mostly rigid scenes and monocular-video conditions without moving objects, occlusion/disocclusion, or violations of Lambertian appearance.Violating these assumptions can corrupt training gradients.
- 3.2. Differentiable depth image-based rendering: The differentiable renderer projects target pixels into a source view using predicted depth and pose, then samples continuous coordinates with bilinear interpolation.Projective geometry preserves the factorization of scene depth and camera pose.
- 3.3. Modeling the model limitation: An explainability mask discounts pixels where direct view synthesis is unreliable, while regularization prevents the mask from collapsing to zero everywhere.The regularizer encourages nonzero predictions by minimizing cross-entropy against constant label 1.
- 3.4. Overcoming the gradient locality: The objective combines view-synthesis, depth-smoothness, and explainability-regularization losses across image scales and source views.The smoothness term minimizes the L1 norm of second-order gradients for predicted depth maps.
4. Experiments
Experiments evaluate unsupervised single-view depth, cross-dataset generalization, pose estimation, and explainability on KITTI, Cityscapes, and Make3D. The method performs comparably to supervised depth baselines, improves over same-input pose baselines, and reveals domain-gap and scene-condition limitations.
- Experimental setup: Experiments benchmark single-view depth and ego-motion on KITTI, with Make3D used to assess cross-dataset generalization.The evaluation also includes Cityscapes pre-training and explainability ablations.
- Experimental setup: Training uses monocular three-frame sequences, with the central frame as target and adjacent frames as source views, producing 44,540 sequences after filtering.Testing scenes and nearly static sequences with mean optical flow below 1 pixel are excluded from training.
- Single-view depth estimation: Unsupervised KITTI depth performs comparably to supervised baselines and sometimes preserves depth boundaries and thin structures better, but trails Godard et al.’s calibrated-stereo method.Evaluation uses 697 images from the Eigen et al. test split.
- Single-view depth estimation: Cityscapes pre-training can introduce structural mistakes on KITTI because of the domain gap, while fine-tuning on KITTI produces the final model.Observed errors include holes on car bodies and confusing cars or bushes with distant objects.
- Visualizing the explainability prediction: Explainability modeling provides only a modest KITTI performance boost because scenes are mostly static and occlusion effects occupy small regions over three-frame sequences.Predicted masks identify dynamic or disappearing objects, but can also mask thin structures as unexplainable.
- Single-view depth estimation: Direct transfer to Make3D captures global scene layout reasonably well without Make3D training, despite a significant gap behind methods supervised with Make3D ground-truth depth.The model is trained on KITTI and Cityscapes only, then tested directly on Make3D.
- Pose estimation: Pose estimation outperforms mean odometry and ORB-SLAM (short) under the same five-frame setting, but falls short of ORB-SLAM (full), which uses whole-sequence loop closure and re-localization.The metric is Absolute Trajectory Error (ATE), computed on five-frame snippets and averaged over each full sequence.
5. Discussion
The pipeline achieves strong benchmark performance despite relying on unlabeled videos, but remains limited in its treatment of dynamics, occlusions, camera calibration, and scene representation.
- The system uses view synthesis to supervise single-view depth and camera pose estimation from unlabeled videos, performing comparably with methods trained using ground-truth depth or pose.
- The framework does not explicitly estimate scene dynamics or occlusions, instead accounting for them implicitly through explainability masks.The masks identify pixels predicted to be unexplainable because of motion, occlusion or visibility, and other factors.
- The method assumes known camera intrinsics, limiting its use on Internet videos with unknown camera types or calibration.
- Its per-pixel depth maps simplify the underlying 3D scene, motivating future extensions to full 3D volumetric representations.
- The learned representations remain insufficiently understood, including possible image correspondence in pose estimation and structural feature recognition in depth estimation.The authors propose probing whether these representations could support object detection or semantic segmentation.