Source-linked AI summary
Unsupervised Scale-consistent Depth Learning from Video
Jia-Wang Bian, Huangying Zhan, Naiyan Wang, Zhichao Li, Le Zhang, Chunhua Shen, Ming-Ming Cheng, Ian Reid
TL;DR
Monocular video-based depth learning lacks reliable scale consistency across frames, limiting its use for video applications such as Visual SLAM. SC-Depth addresses this with geometry consistency and self-discovered masking, producing accurate, scale-consistent depths that integrate into hybrid Pseudo-RGBD SLAM. The system supports robust tracking and dense reconstruction, while absolute metric scale remains unrecoverable from monocular videos alone.
Problem
Video-based monocular depth learning suffers from per-frame scale ambiguity, so predictions can be inconsistent across frames and unusable for camera tracking.
Method
SC-Depth trains on unlabelled monocular videos using a geometry consistency loss and a self-discovered mask for moving objects, then integrates the networks into ORB-SLAM2.
Results
The method produces scale-consistent and accurate depths across indoor and outdoor scenes and enables Visual SLAM for camera tracking and dense reconstruction.
Takeaways & Limitations
Unsupervised monocular-video depth networks can be plugged into Visual SLAM to support camera tracking and dense reconstruction.
Takeaways & Limitations
The method cannot recover absolute scale because it uses only monocular videos, although IMU and road landmarks can provide metric-scale cues.
Abstract
from arXiv · showhide
We propose a monocular depth estimator SC-Depth, which requires only unlabelled videos for training and enables the scale-consistent prediction at inference time. Our contributions include: (i) we propose a geometry consistency loss, which penalizes the inconsistency of predicted depths between adjacent views; (ii) we propose a self-discovered mask to automatically localize moving objects that violate the underlying static scene assumption and cause noisy signals during training; (iii) we demonstrate the efficacy of each component with a detailed ablation study and show high-quality depth estimation results in both KITTI and NYUv2 datasets. Moreover, thanks to the capability of scale-consistent prediction, we show that our monocular-trained deep networks are readily integrated into the ORB-SLAM2 system for more robust and accurate tracking. The proposed hybrid Pseudo-RGBD SLAM shows compelling results in KITTI, and it generalizes well to the KAIST dataset without additional training. Finally, we provide several demos for qualitative evaluation.
1 Introduction
Video-based unsupervised depth learning avoids ground-truth range sensors but suffers from unknown motion and per-frame scale ambiguity, limiting consistency across video. SC-Depth addresses this with geometry consistency, self-discovered masking, and integration into Pseudo-RGBD SLAM.
- Video-based unsupervised learning uses unlabelled videos and avoids the expensive range sensors required for supervised ground-truth depth.
- Per-frame scale ambiguity can produce inconsistent depths across frames, which is critical because inconsistent predictions cannot support camera tracking in Visual SLAM.
- Geometry consistency loss penalizes pixel-wise depth inconsistency between adjacent frames, encouraging scale-consistent predictions during training and inference.
- On self-captured Adelaide video, networks trained on KITTI without finetuning supported an accurate trajectory while ORB-SLAM2 failed to initialize or quickly lost tracking.
- The self-discovered mask localizes moving objects that violate the static-scene assumption and improves performance without additional overhead because it is derived from the geometry consistency loss.
- The hybrid Pseudo-RGBD SLAM system integrates trained depth and pose networks into ORB-SLAM2 for more robust tracking and dense 3D reconstruction.
2 Related work
Prior work addresses monocular depth through geometry, stereo, and video supervision, while scale consistency and moving-object handling remain important challenges. The paper combines deep learning with geometric optimization to improve generalization in Visual SLAM.
- Single-view depth estimation: Traditional geometry-based depth methods require at least two scene views, whereas deep networks enable single-view depth estimation.
- Single-view depth estimation: Stereo-based methods can predict metric depth but require well-calibrated stereo cameras, while video-based methods train depth and pose from unlabelled videos.
- Scale consistency: The paper identifies scale inconsistency as a previously under-discussed issue in unsupervised video-based depth learning and proposes a differentiable geometry consistency loss.
- Moving objects: Moving objects violate the static-world assumption, motivating methods that detect dynamic regions and mask them during photometric-loss computation.
- Depth estimation for Visual SLAM: Traditional Visual SLAM uses feature matching or direct image alignment, while learning-based methods support depth estimation and dense reconstruction but can overfit.
- Depth estimation for Visual SLAM: The proposed hybrid system combines deep learning with geometry-based methods and generalizes to previously unseen datasets and self-captured videos.
3 SC-Depth
SC-Depth trains depth and pose CNNs from adjacent unlabeled video frames using photometric and smoothness losses, while geometry consistency and masking address scale inconsistency, dynamics, occlusions, and invalid stationary points.
- Framework Overview: Adjacent frames provide predicted depth maps and relative pose for differentiable image warping, supervising the network through photometric reconstruction.The synthesized reference image is compared with the real reference image.
- Framework Overview: The objective combines photometric loss weighted by the self-discovered mask, smoothness loss, and geometry consistency loss, averaged over points retained by the auto-mask.The loss terms have weighting coefficients, while the auto-mask determines valid points.
- Geometry Consistency Loss: A differentiable depth inconsistency operation transforms one predicted depth map through the relative pose, interpolates it for alignment, and compares it with the other map.The operation projects depth into 3D and then into the other image plane; bilinear interpolation handles non-grid projections.
- Geometry Consistency Loss: The relative inconsistency function normalizes depth differences, is symmetric, ranges from 0 to 1, and is used to define the geometry consistency loss.Normalization treats points at different absolute depths equally during optimization and improves training stability.
- Geometry Consistency Loss: Consistency across overlapping frame pairs propagates through a sequence, so depths of successive frames eventually agree at a common scale.The method links I1 to I2 in one batch and I2 to I3 in another, producing sequence-level scale consistency.
- Self Discovered Mask: The self-discovered mask assigns low weights to geometrically inconsistent pixels, targeting moving objects and occlusions that violate the static-scene assumption.The auto-mask separately removes stationary points when the camera is static; visual results show dynamics and occlusions in Ms.
- Self Discovered Mask: The proposed masks mitigate noisy training signals, and the ablation study reports significant performance improvement from including them.Ms identifies regions such as cars and riders, while Ma marks invalid stationary points.
4 Pseudo-RGBD SLAM
Pseudo-RGBD SLAM combines SC-Depth predictions with ORB-SLAM2 for globally consistent tracking and mapping, then uses InfiniTAMv3 for dense voxel reconstruction. Its design addresses initialization, tracking robustness, map density, and scale limitations while retaining geometric optimization.
- System pipeline: The system uses predicted depth and relative pose from a monocular RGB video, then feeds them to ORB-SLAM2 for camera trajectory and sparse mapping.InfiniTAMv3 subsequently fuses predicted depths with optimized poses for dense voxel reconstruction used only for qualitative demonstration.
- Depth integration: Predicted depths initialize the 3D map and augment reprojection optimization with a projected-depth consistency term.This extends reprojection error from 2D into 3D; inconsistent depths can cause correct matches to be rejected as outliers and make tracking fail.
- Pose integration: The predicted pose initializes tracking before Bundle Adjustment optimizes the camera pose and supports subsequent map enrichment.This replaces ORB-SLAM2's constant-velocity initialization, which can be violated by abrupt motion in driving scenes.
- Advantages: Compared with classical monocular SLAM, the system can initialize at any frame, enrich sparse maps with dense depth, and enable dense 3D reconstruction.These claimed advantages are attributed to the estimated dense depth supplied to the SLAM pipeline.
- Advantages: Compared with learning-based methods, post-geometric optimization such as loop closing can correct drift and improve performance.The system combines learned predictions with a multi-view geometry-based framework.
- Limitations: The method cannot recover absolute scale from monocular videos alone, although IMU measurements and road landmarks can provide metric scale.Dense reconstruction is also highly sensitive to geometry consistency and is presented for visualization rather than as a primary system output.
5 Experiments
Experiments show that SC-Depth’s components improve unsupervised depth estimation and that scale-consistent predictions support robust visual odometry and reconstruction. The hybrid Pseudo-RGBD SLAM system performs strongly on KITTI, generalizes to KAIST without additional training, and produces temporally consistent qualitative outputs.
- Implementation details: Single-scale supervision reduces training cost while achieving performance on par with MonoDepth2’s multi-scale solution.The method computes losses only at the original image resolution; upsampling depth maps instead requires two times longer training.
- Depth estimation: On KITTI, SC-Depth outperforms earlier monocular methods and achieves on par performance with MonoDepth2.Its reported advantage over MonoDepth2 is depth consistency, which enables integration into visual SLAM systems.
- Depth estimation: On NYUv2, SC-Depth outperforms previous unsupervised methods by a large margin but remains inferior to the state-of-the-art supervised method.The reported NYUv2 result removes relative rotation between training image pairs, which significantly improves performance for handheld videos.
- Visual odometry: Pseudo-RGBD SLAM achieves accuracy on par with or better than ORB-SLAM2 across eleven KITTI sequences and maintains consistent scale over long videos.The system tracks about 800 points versus 100 for ORB-SLAM2 in the cited comparison and avoids the latter’s heavy long-sequence scale drift.
- Visual odometry: Using 2K keypoints is more accurate than using 8K, reflecting a trade-off between geometric accuracy, outliers, and robustness.The authors recommend selecting keypoint counts by considering accuracy and robustness.
- Generalization: Pseudo-RGBD SLAM consistently outperforms ORB-SLAM2 on the KAIST urban dataset when trained only on KITTI.The paper also demonstrates a real-world driving-video result without an accurate ground-truth trajectory.
- Qualitative evaluation: The predicted depths support temporally consistent per-frame point-cloud visualization and dense reconstruction, although outdoor reconstruction quality is degraded by reduced TSDF voxel size.The per-frame visualization avoids flickering, while the dense reconstruction remains inferior to state-of-the-art methods.
6 Conclusion
The paper presents video-based unsupervised depth learning that produces scale-consistent, accurate depths and supports visual SLAM integration for tracking and dense reconstruction.
- The method learns monocular depth from video and predicts scale-consistent, accurate depths across indoor and outdoor scenes.
- Scale consistency enables the trained depth network to integrate with visual SLAM for camera tracking and dense reconstruction.