Source-linked AI summary
Depth Prediction Without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular Videos
Vincent Casser, Soeren Pirk, Reza Mahjourian, Anelia Angelova
TL;DR
Monocular depth and ego-motion learning for robot navigation must handle scenes with moving objects without depth or ego-motion supervision. This paper models scene and object motion in 3D and adds online refinement, achieving state-of-the-art performance and improved transfer to dynamic-scene settings.
Problem
Unsupervised scene-depth estimation from monocular video remains a challenging vision and robotics problem without depth or ego-motion supervision.
Method
The method jointly predicts dense depth, camera ego-motion, and individual 3D object motions, with online refinement for adaptation to new environments.
Results
The method achieves new state-of-the-art performance, improving absolute relative error from 0.205 to 0.153 when trained on Cityscapes and evaluated on KITTI.
Takeaways & Limitations
Modeling object motion and online refinement improves monocular depth and ego-motion estimation, particularly in dynamic scenes and across datasets or environments.
Takeaways & Limitations
Online refinement works best when inter-frame motion is similar and should not be applied when the camera is static.
Abstract
from arXiv · showhide
Learning to predict scene depth from RGB inputs is a challenging task both for indoor and outdoor robot navigation. In this work we address unsupervised learning of scene depth and robot ego-motion where supervision is provided by monocular videos, as cameras are the cheapest, least restrictive and most ubiquitous sensor for robotics. Previous work in unsupervised image-to-depth learning has established strong baselines in the domain. We propose a novel approach which produces higher quality results, is able to model moving objects and is shown to transfer across data domains, e.g. from outdoors to indoor scenes. The main idea is to introduce geometric structure in the learning process, by modeling the scene and the individual objects; camera ego-motion and object motions are learned from monocular videos as input. Furthermore an online refinement method is introduced to adapt learning on the fly to unknown domains. The proposed approach outperforms all state-of-the-art approaches, including those that handle motion e.g. through learned flow. Our results are comparable in quality to the ones which used stereo as supervision and significantly improve depth prediction on scenes and datasets which contain a lot of object motion. The approach is of practical relevance, as it allows transfer across environments, by transferring models trained on data collected for robot navigation in urban scenes to indoor navigation settings. The code associated with this paper can be found at https://sites.google.com/view/struct2depth.
Previous Work
Previous work established learning-based image-to-depth estimation as a compelling alternative to traditional and stereo or multi-view approaches. However, these methods struggle with object motion in dynamic scenes, motivating optical-flow-based extensions and geometric motion modeling.
- Learning-based image-to-depth estimation emerged from rich feature representations learned from raw data and achieved compelling results compared with traditional methods.
- Existing methods often fail in highly dynamic scenes because they cannot explain object motion.
- Separately trained optical-flow models have addressed object motion with moderate improvements.
Main Method
The method learns dense depth and camera ego-motion from monocular RGB video through differentiable view synthesis, while incorporating explicit 3D structure for moving objects. It also introduces object-scale constraints and optional online refinement to address dynamic-scene and domain-adaptation challenges.
- Online refinement: The proposed dynamic-scene modeling and optional online refinement can be used separately or jointly, with refinement adapting the learning strategy on the fly.The refinement loss can correct wrong depth estimates in already trained models, particularly for moving objects.
- Monocular depth and ego-motion: The model takes at least three RGB frames and camera intrinsics, predicting dense single-frame depth with an encoder-decoder and ego-motion with a sequence network.The ego-motion network predicts transformations from three frames, while the depth network produces one dense map per RGB frame.
- Monocular depth and ego-motion: Differentiable image warping uses predicted depth, ego-motion, and camera intrinsics to reconstruct adjacent frames, with photometric reconstruction error providing the supervisory signal.The baseline further combines minimum bidirectional reconstruction loss, SSIM, depth smoothness, depth normalization, and four-scale training.
- Explicit object motion: An object-motion network predicts per-object 3D transformations from RGB sequences augmented with instance-segmentation masks, while masking moving objects helps estimate camera motion from the static scene.After ego-motion warping, residual misalignment is attributed to moving objects, whose motions are modeled through inverse warping.
- Object-scale constraints: The object-scale loss uses segmentation-based size constraints to prevent moving objects from degenerating into infinite depth and to jointly learn plausible depth and motion.The loss is normalized by the mean estimated depth to reduce trivial shrinkage of priors and the predicted depth range.
Experimental Results
Experiments across KITTI, Cityscapes, and Fetch indoor navigation show that the method improves monocular depth and ego-motion estimation, especially with moving objects and cross-domain transfer. Motion modeling and online refinement each contribute, with refinement particularly beneficial when testing across datasets.
- Depth estimation: 0.1087 absolute relative error outperforms motion-based models with 0.131 (Yang et al. 2018a) and 0.155 (Yin 2018) on KITTI depth estimation.The monocular results also approach methods using stereo or combined stereo and monocular supervision.
- Motion model: The motion model improves depth and ego-motion in dynamic scenes by learning appropriate depth for moving objects and individual object motions.The baseline fails on moving objects, whereas the proposed model produces individual 3D motion estimates, including each object’s direction and speed.
- Refinement model: The refinement model improves recovered scene geometry on both KITTI and Cityscapes, with its strongest benefits appearing during cross-dataset transfer.Visual improvements are especially pronounced for fine structures and zero-shot Cityscapes evaluation after training on KITTI.
- Transfer learning: Training on Cityscapes and evaluating on KITTI without fine-tuning yields significant improvements from the motion and refinement components individually and jointly.The combined gains reflect improved depth for moving objects and active refinement of scene geometry.
- Ego-motion estimation: The algorithm achieves the best ego-motion performance among state-of-the-art methods on KITTI odometry, including methods using more temporal information and ORB-SLAM.The tested driving sequence lengths are 1,702 meters and 918 meters.
Conclusions and Future Work
The method addresses monocular depth and ego-motion by modeling individual object motion in 3D and using online refinement to adapt learning and transfer across datasets or environments. It achieves state-of-the-art benchmark performance with higher-quality results in dynamic scenes, while future work will extend refinement to longer sequences.
- The method models individual objects’ motion in 3D to address monocular depth and ego-motion estimation.
- Online refinement adapts learning on the fly and supports transfer to new datasets or environments.
- The algorithm achieves new state-of-the-art performance on established benchmarks and higher-quality results in dynamic scenes.
- Future work will apply the refinement method over longer sequences to incorporate additional information.