Source-linked AI summary
Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video
Jia-Wang Bian, Zhichao Li, Naiyan Wang, Huangying Zhan, Chunhua Shen, Ming-Ming Cheng, Ian Reid
TL;DR
Unsupervised monocular-video depth and ego-motion learning is limited by moving objects and scale-inconsistent predictions that obstruct long-term camera trajectories. The paper introduces geometry consistency and a self-discovered mask without multi-task networks, achieving state-of-the-art KITTI depth performance and globally scale-consistent long-sequence trajectories with competitive visual odometry.
Problem
Moving objects violate the static-scene assumption, while per-frame scale ambiguity produces inconsistent predictions that prevent full long-video camera trajectories.
Method
The framework uses a geometry consistency loss for scale-consistent predictions and derives a self-discovered mask from depth inconsistency to handle dynamic and occluded regions without additional networks.
Results
The depth estimator achieves state-of-the-art KITTI performance, while ego-motion predicts globally scale-consistent long-sequence trajectories with competitive visual odometry accuracy.
Takeaways & Limitations
Unlabelled monocular-video training can produce globally scale-consistent camera trajectories over long sequences using a simpler framework than multi-task alternatives.
Abstract
from arXiv · showhide
Recent work has shown that CNN-based depth and ego-motion estimators can be learned using unlabelled monocular videos. However, the performance is limited by unidentified moving objects that violate the underlying static scene assumption in geometric image reconstruction. More significantly, due to lack of proper constraints, networks output scale-inconsistent results over different samples, i.e., the ego-motion network cannot provide full camera trajectories over a long video sequence because of the per-frame scale ambiguity. This paper tackles these challenges by proposing a geometry consistency loss for scale-consistent predictions and an induced self-discovered mask for handling moving objects and occlusions. Since we do not leverage multi-task learning like recent works, our framework is much simpler and more efficient. Comprehensive evaluation results demonstrate that our depth estimator achieves the state-of-the-art performance on the KITTI dataset. Moreover, we show that our ego-motion network is able to predict a globally scale-consistent camera trajectory for long video sequences, and the resulting visual odometry accuracy is competitive with the recent model that is trained using stereo videos. To the best of our knowledge, this is the first work to show that deep networks trained using unlabelled monocular videos can predict globally scale-consistent camera trajectories over a long video sequence.
1 Introduction
Unsupervised monocular-video learning faces dynamic-scene errors and scale-inconsistent predictions that prevent reliable long-term camera trajectories. The paper addresses both with geometry consistency, a self-discovered mask, and a simpler framework without additional task networks.
- Per-frame scale ambiguity makes ego-motion predictions inconsistent across samples, preventing full camera trajectories over long videos.
- Geometry consistency converts predicted depth into 3D, projects it using estimated ego-motion, and minimizes disagreement with the other frame’s depth.Iterative consecutive-frame training propagates scale consistency through the video sequence.
- The self-discovered mask identifies dynamic, occluded, and difficult regions from the geometry inconsistency term and downweights them during reconstruction.This avoids adding optical-flow or semantic-segmentation networks.
- The framework reports state-of-the-art KITTI depth performance and competitive visual-odometry accuracy against a model trained with stereo videos.It also produces globally scale-consistent camera trajectories over long video sequences.
- The paper positions its contributions as a scale-consistency constraint, a mask for dynamic scenes and occlusions, and a simpler, more efficient learning framework.
2 Related work
Depth estimation progressed from multi-view disparity and supervised single-image CNNs to stereo- and monocular-video-based learning, while learned monocular systems retain scale-consistency challenges and dynamic-object limitations. The paper positions globally scale-consistent long-video camera trajectories as a novel capability.
- Traditional reconstruction uses disparity between multiple views, requiring at least two images, whereas CNNs can predict depth from a single image with range-sensor supervision.
- Monocular learned models predict scale-inconsistent results across snippets, unlike monocular ORB-SLAM predictions, which are globally scale-consistent despite scale ambiguity.
- Stereo-based unsupervised depth learning replaces depth labels with epipolar geometry and photometric consistency between real and synthesized views.Later work adds left-right consistency as a regularizer.
- Monocular-video learning jointly estimates depth and relative camera pose, using image reconstruction and photometric loss without ground-truth depth.Dynamic objects can violate the static-scene assumption underlying reconstruction.
- Optical-flow and motion-segmentation extensions improve performance but add substantial computational cost and do not resolve scale inconsistency.
- The paper reports that monocular-video-trained ego-motion can predict a globally scale-consistent camera trajectory over a long sequence, with potential applications in Visual SLAM.
3 Unsupervised Learning of Scale-consistent Depth and Ego-motion
The framework combines photometric reconstruction with geometry consistency to enforce scale-consistent depth and ego-motion, while deriving a mask from depth inconsistency to reduce the impact of dynamic and unreliable regions.
- 3.3 Geometry consistency loss: Predicted depth from one frame is warped into the other using the estimated relative pose, then compared with interpolated depth to form the geometry consistency loss.Interpolation is required because the projection flow does not generally lie on the target image’s pixel grid.
- 3.3 Geometry consistency loss: The geometry consistency constraint propagates agreement across consecutive training pairs, yielding scale-consistent predictions over an entire sequence.The pose network is coupled with depth during training, enabling globally scale-consistent camera trajectories.
- 3.3 Geometry consistency loss: The normalized depth inconsistency compares warped and estimated depths symmetrically, treats different absolute depths equally, and produces values from 0 to 1.These properties improve numerical stability during training.
- 3.4 Self-discovered mask: The inconsistency map identifies dynamic objects, occlusions, and difficult regions because each produces increased disagreement between scene structures.The resulting mask assigns lower weights to inconsistent pixels and re-weights the photometric loss.
- 3.4 Self-discovered mask: Applying the self-discovered mask mitigates adverse effects from moving objects and occlusions and reduces the back-propagation weight of inaccurately predicted regions.Visual results show the mask identifying occlusions and moving objects.
4 Experiment
Experiments evaluate depth, visual odometry, ablations, architectural choices, and computational cost on KITTI datasets. The proposed geometry-consistency loss and mask improve robustness, while single-scale supervision and larger models or resolutions produce better results.
- Implementation details: Models are trained for single-view depth on KITTI raw and for visual odometry on KITTI odometry using monocular video inputs.Depth experiments use Eigen’s split and two input resolutions; odometry evaluates sequences 00–08 for training and 09–10 for testing.
- Comparisons with the state-of-the-art: The depth estimator achieves state-of-the-art KITTI raw performance without the multi-task learning used by several competing models.The method also compares favorably with approaches using calibrated stereo pairs or ground-truth depth annotations.
- Comparisons with the state-of-the-art: The ego-motion network produces globally scale-consistent trajectories and achieves competitive visual odometry results against a stereo-trained method.The comparison uses one global scale for the proposed method, unlike per-frame scale alignment for the scale-inconsistent baseline.
- Ablation study: The proposed geometry-consistency loss and self-discovered mask improve depth results and prevent overfitting in ablation studies.The loss effectively prevents overfitting, while the ablation results demonstrate contributions from the proposed terms.
- Ablation study: Single-scale supervision outperforms widely used four-scale supervision in the reported depth experiments.The paper attributes this empirically to inaccurate photometric loss at low resolutions and the small displacement between consecutive views.
- Ablation study: Higher-resolution images and deeper networks result in better depth performance.The comparison covers DispNet and DispResNet across different image resolutions.
5 Conclusion
The framework combines geometry consistency with a self-discovered mask for scale-consistent depth and ego-motion estimation. It achieves state-of-the-art KITTI depth performance, competitive stereo-trained visual odometry, and globally scale-consistent long-sequence trajectories.
- The proposed framework targets scale-consistent depth and ego-motion estimation using geometry consistency loss and a self-discovered mask for dynamic scenes.
- The depth model achieves state-of-the-art performance on the KITTI dataset.
- The ego-motion network produces competitive visual odometry results relative to a model trained using stereo videos.
- The method predicts globally scale-consistent camera trajectories over long sequences from unlabelled monocular videos.
- Future work will improve visual odometry accuracy by incorporating drift-correcting solutions.
6 Supplementary
The supplementary experiments evaluate pose estimation on short KITTI snippets, depth generalization on Make3D without fine-tuning, and qualitative depth and occlusion results. Performance remains close to or slightly better than state-of-the-art methods in the reported comparisons.
- 6.1 Pose estimation results on 5-frame snippets: Pose estimation on 5-frame snippets is slightly below state-of-the-art methods, with a small performance gap.
- 6.2 Generalization ability: Make3D relative depth error is slightly better than other state-of-the-art methods without fine-tuning on Make3D.
- 6.3 More qualitative results: Figure 5 presents qualitative depth-estimation and occlusion-detection results from the proposed approach.