Source-linked AI summary
Unsupervised Monocular Depth Learning in Dynamic Scenes
Hanhan Li, Ariel Gordon, Hang Zhao, Vincent Casser, Anelia Angelova
TL;DR
The paper addresses monocular depth and object-motion estimation in dynamic scenes, an ill-posed problem requiring learned priors. It jointly learns depth, ego-motion, and dense 3D object motion from monocular video using regularization, achieving state-of-the-art or matched performance on several dynamic-scene benchmarks.
Problem
Estimating depth and object motion in 3D from monocular video is ill-posed and generally relies on prior knowledge.
Method
The method jointly learns depth, ego-motion, and dense 3D object motion from unlabeled monocular videos using photometric consistency, motion smoothness, and sparsity regularization.
Results
The method establishes state-of-the-art unsupervised depth results on Cityscapes and Waymo, matches the state of the art on KITTI, and outperforms prior benchmarks on datasets rich in dynamic scenes.
Takeaways & Limitations
Monocular video alone, without semantic or other auxiliary signals, can support depth prediction and object-motion learning in highly dynamic scenes.
Takeaways & Limitations
Improvements on KITTI are less pronounced because dynamic scenes are rare in that dataset.
Abstract
from arXiv · showhide
We present a method for jointly training the estimation of depth, ego-motion, and a dense 3D translation field of objects relative to the scene, with monocular photometric consistency being the sole source of supervision. We show that this apparently heavily underdetermined problem can be regularized by imposing the following prior knowledge about 3D translation fields: they are sparse, since most of the scene is static, and they tend to be constant for rigid moving objects. We show that this regularization alone is sufficient to train monocular depth prediction models that exceed the accuracy achieved in prior work for dynamic scenes, including methods that require semantic input. Code is at https://github.com/google-research/google-research/tree/master/depth_and_motion_learning .
1 Introduction
The paper addresses ill-posed monocular depth and object-motion estimation in dynamic scenes using only video, without semantic, stereo, or ground-truth supervision. It regularizes sparse, object-constant translation fields and reports strong results across dynamic-scene datasets.
- Monocular estimation of depth and object motion is ill-posed and typically relies on prior knowledge.
- Self-supervised structure-from-motion methods face textureless areas, occlusions, reflections, and moving objects that disrupt view consistency.
- Compared with related approaches, the method avoids auxiliary segmentation, stereo input, and assumptions limited to one specific object-motion pattern.
- The method jointly learns depth, ego-motion, and a dense object-motion map from monocular video without semantic signals, stereo, or ground truth.
- Its residual translation-field regularization exploits sparsity and near-constant motion across rigid moving objects, while supporting arbitrary rigid translation directions.
- The approach establishes state-of-the-art unsupervised depth results on Cityscapes and Waymo, matches the state of the art on KITTI, and is qualitatively evaluated on YouTube videos.
2 Related Work
Prior work estimates depth using classical geometry or learned dense prediction, while dynamic-scene methods increasingly jointly learn depth, camera motion, and object motion from monocular videos. These approaches address the need to model motion beyond static-scene assumptions.
- Structure from Motion and Multiview Stereo: Classical structure-from-motion and multiview stereo identify image correspondences and use epipolar geometry, producing sparse depth maps.Multiview stereo supports dynamic scenes with multiple cameras, whereas monocular structure from motion targets static scenes with a moving camera.
- Depth Estimation: Deep-learning approaches predict dense depth maps and may use direct sensor supervision or combine learned depth and ego-motion with classical vision.These methods extend dense prediction beyond sparse geometric reconstruction.
- Depth and Motion: Dynamic-scene methods must estimate camera and individual object motion to enforce consistency across views.Several approaches jointly learn depth, camera motion, and object motion from monocular videos, while others use multiple views for depth.
3 Method
The method jointly predicts monocular depth, ego-motion, and dense object motion, then uses differentiable warping and self-supervised regularization to train from adjacent frames. Its motion priors favor sparse fields and nearly constant motion within rigid objects, while consistency losses connect forward and backward views.
- Overall Architecture: A depth network predicts one depth map per frame, while a motion network predicts a dense object translation map and a 6D ego-motion vector from adjacent frames.The ego-motion vector parameterizes 3D rotation and translation; the two depth maps and RGB frames are provided to the motion network.
- Overall Architecture: A differentiable view transformer uses predicted depth and motion to connect the two frames during training.The setup applies motion regularization, cycle consistency, and photometric consistency losses, while inference uses one frame for depth and two frames for motion.
- Training Losses: Training combines motion, depth, and consistency regularization, with motion and consistency terms applied in both frame orders.Depth regularization is applied independently to each frame, and disparity uses edge-aware smoothness that weakens near strong color variation.
- Motion Regularization: The motion regularizer combines group smoothness and L1/2 sparsity, encouraging object motion to be nearly constant within moving regions and sparse elsewhere.Group smoothness reflects the prior that moving objects are mostly rigid, while L1/2 encourages more sparsity than L1.
- Motion Regularization: The piecewise-constant motion approximation is expected to suit road traffic because fast object rotation relative to the background is uncommon.Pure translation can be represented exactly, whereas rotation generally produces a non-constant residual translation field.
- Consistency Regularization: Consistency regularization combines cycle consistency, which makes forward and backward motion opposites, with occlusion-aware photometric consistency.Photometric consistency compares original and warped RGB images using L1 and SSIM terms with an occlusion mask.
4 Experiments
Experiments evaluate the method across Cityscapes, KITTI, Waymo, and hand-held YouTube videos. The method achieves strong dynamic-scene performance without semantic masks, while also providing fast inference.
- Datasets and setup: The model was evaluated on Cityscapes, KITTI, Waymo Open Dataset, and hand-held YouTube videos, with learned camera intrinsics for YouTube.All experiments used the same hyperparameter values; dataset camera intrinsics were provided except for YouTube videos.
- Efficiency: Depth inference took 5.3ms per 480x192 frame on an unoptimized NVIDIA V100, equivalent to roughly 190 frames per second.The authors describe this as among the fastest methods they are aware of.
- Cityscapes: On Cityscapes, the method outperformed all prior methods except Ref. [11] on RMSE, whose method uses semantic cues.Cityscapes contains many dynamic scenes and uses standard evaluation protocols.
- KITTI: On KITTI, the model performed on par with the state of the art.KITTI contains relatively few dynamic scenes, making improvements less pronounced.
- Waymo Open Dataset: On Waymo, the model outperformed prior methods that use object masks, and adding masks produced further improvements.The maskless model does not require semantic masks, while the masked version performed better than its maskless setting.
5 Conclusions
The paper presents an unsupervised method that jointly learns depth and 3D motion maps from unlabeled monocular videos without auxiliary semantic information. It performs strongly on dynamic-scene benchmarks, while assuming camera motion and not explicitly modeling object rotation or deformation.
- Conclusion: The method jointly learns depth and 3D motion maps from unlabeled monocular videos without auxiliary semantic information.It uses end-to-end differentiable losses for photometric consistency, motion smoothness, and motion sparsity.
- Conclusion: The approach outperforms prior depth-estimation benchmarks on datasets rich in dynamic scenes, including methods that use semantic cues.The demonstrated datasets include Cityscapes, KITTI, Waymo Open Dataset, and YouTube data.
- Limitations: The method does not explicitly handle object rotation or deformation and requires camera movement to receive learning signals.