Source-linked AI summary
Deep Virtual Stereo Odometry: Leveraging Deep Depth Prediction for Monocular Direct Sparse Odometry
Nan Yang, Rui Wang, Jörg Stückler, Daniel Cremers
TL;DR
Monocular geometric odometry suffers from scale drift and motion-parallax requirements. DVSO adds deep monocular depth predictions to DSO as virtual stereo measurements, using a semi-supervised two-stage depth network. On KITTI, it reaches stereo-VO-level performance while relying on one camera.
Problem
Single-camera geometric visual odometry has unknown scale, scale drift, and motion-parallax requirements, while stereo or active-depth sensors increase calibration effort and cost.
Method
DVSO integrates deep depth predictions into DSO as virtual stereo constraints, using a stacked two-stage network trained with stereo photoconsistency and sparse Stereo DSO reconstructions.
Results
DVSO outperforms state-of-the-art monocular methods by a large margin and achieves comparable results to stereo VO methods on KITTI.
Takeaways & Limitations
The method recovers accurate, scale-consistent odometry using only a single camera.
Takeaways & Limitations
The deep network was trained on driving sequences, and its generalization to other camera trajectories and environments remains to be investigated.
Abstract
from arXiv · showhide
Monocular visual odometry approaches that purely rely on geometric cues are prone to scale drift and require sufficient motion parallax in successive frames for motion estimation and 3D reconstruction. In this paper, we propose to leverage deep monocular depth prediction to overcome limitations of geometry-based monocular visual odometry. To this end, we incorporate deep depth predictions into Direct Sparse Odometry (DSO) as direct virtual stereo measurements. For depth prediction, we design a novel deep network that refines predicted depth from a single image in a two-stage process. We train our network in a semi-supervised way on photoconsistency in stereo images and on consistency with accurate sparse depth reconstructions from Stereo DSO. Our deep predictions excel state-of-the-art approaches for monocular depth on the KITTI benchmark. Moreover, our Deep Virtual Stereo Odometry clearly exceeds previous monocular and deep learning based methods in accuracy. It even achieves comparable performance to the state-of-the-art stereo methods, while only relying on a single camera.
1 Introduction
Monocular geometric visual odometry suffers from unknown scale, scale drift, and insufficient motion parallax, while stereo and active-depth sensors add calibration effort and cost. DVSO integrates deep depth predictions into a direct geometric pipeline to recover metric-scale odometry from one camera.
- Motivation: Single-camera geometric VO estimates camera trajectory and maps only up to unknown scale, causing scale drift and requiring sufficient motion parallax.Stereo rigs or active depth cameras avoid these issues but increase calibration effort and system cost.
- Approach: DVSO incorporates deep stereo disparity into windowed direct bundle adjustment as virtual direct image-alignment constraints.The predicted disparities initialize the system and provide virtual stereo constraints in the optimization pipeline.
- Results: DVSO achieves comparable performance to state-of-the-art stereo visual odometry on KITTI while using only a single camera.With virtual baseline tuning, it can outperform state-of-the-art geometric VO methods on scale-dependent parameters.
- Approach: Deep depth prediction is trained semi-supervised using stereo photoconsistency and accurate sparse 3D reconstructions from Stereo DSO.This combines self-supervised image-based training with sparse geometric supervision instead of requiring large amounts of active-sensor ground truth.
2 Semi-Supervised Deep Monocular Depth Estimation
The depth estimator combines stereo photoconsistency, sparse Stereo DSO supervision, and two-stage refinement in a stacked encoder-decoder architecture. Its loss includes reconstruction, supervision, consistency, smoothness, and occlusion terms across multiple scales.
- Method: The method combines stereo photoconsistency, sparse Stereo DSO depth supervision, and two-stage refinement for semi-supervised monocular depth estimation.Stereo DSO supplies accurate sparse depth reconstructions, avoiding costly LiDAR collection.
- Network Architecture: StackNet stacks SimpleNet and ResidualNet, with ResidualNet learning additive disparity residuals to refine SimpleNet predictions.Both subnetworks use fully convolutional encoder-decoder designs, while ResidualNet receives SimpleNet outputs as inputs.
- Network Architecture: SimpleNet uses a ResNet-50 encoder, skip connections, and four output-resolution pairs of left and right disparity maps.Skip connections help the decoder recover high-resolution results with fine-grained details.
- Network Architecture: Only the left image is required at inference, although the network predicts refined disparity maps for both left and right views.The refined outputs add SimpleNet and ResidualNet disparities element-wise at four scales.
- Loss Function: The multiscale loss combines self-supervised reconstruction, sparse supervised disparity, left-right consistency, smoothness, and occlusion regularization.The reconstruction term warps rectified stereo images and uses an ℓ1-plus-SSIM objective; regularizers address ambiguous homogeneous and occluded regions.
3 Deep Virtual Stereo Odometry
DVSO extends monocular DSO with deep disparity predictions used for metric depth initialization and virtual stereo alignment within direct sparse bundle adjustment.
- Virtual stereo constraints: The predicted right disparity supplies virtual direct image-alignment constraints by warping the estimated-depth image under a virtual stereo setup.The actual right camera image is never used, so the odometry remains monocular.
- Depth initialization: DVSO initializes new-keyframe depths from the predicted left disparity, yielding a more stable initialization with consistent metric scale.The disparity is converted to inverse depth using the training stereo intrinsics and baseline.
- Optimization: Each incoming frame is tracked against the reference keyframe by coarse-to-fine direct alignment before DVSO decides whether to create a new keyframe.Point selection additionally uses a left-right consistency check to reject pixels likely belonging to occluded regions.
- Optimization: The objective jointly optimizes temporal photometric errors and virtual stereo errors using Gauss–Newton within a fixed seven-keyframe active window.Old keyframes are marginalized with the Schur complement, while the coupling factor λ balances the two terms.
- Virtual stereo constraints: DVSO adds a virtual stereo energy term for each point to make estimated depths consistent with StackNet disparity predictions.The residual is formulated in photoconsistency rather than direct disparity differences, reflecting prediction uncertainty and matching the temporal alignment residual units.
4 Experiments
Experiments evaluate StackNet depth prediction and DVSO odometry on KITTI, including qualitative comparisons, ablations, and comparisons with monocular and stereo systems. StackNet outperforms major monocular depth baselines on most metrics, while DVSO achieves strong monocular odometry performance and competitive or superior results against stereo and end-to-end methods.
- Monocular Depth Estimation: Qualitatively, StackNet produces more detailed disparity maps and more accurate predictions on thin structures than the compared self-supervised and LiDAR-based methods.The comparisons highlight fewer checkerboard artifacts, no strong contour-shadow effect, and successful traffic-sign prediction relative to the baselines.
- Monocular Depth Estimation: StackNet outperforms Godard et al.’s self-supervised method by a large margin and surpasses Kuznietsov et al.’s LiDAR-based method on all but two less restrictive metrics.The exceptions are δ < 1.252 and δ < 1.253.
- Monocular Depth Estimation: StackNet is evaluated on the KITTI Raw test split against self-supervised and LiDAR-based semi-supervised monocular depth methods.The evaluation uses Eigen et al.’s crop and reports separate depth ranges of 0–80 m and 1–50 m.
- Monocular Visual Odometry: DVSO’s ablation study evaluates depth initialization, virtual stereo constraints, left-right consistency checks, and tuned virtual baselines using translational and rotational RMSE.The full DVSO configuration with StackNet depth clearly outperforms the other tested variants.
- Monocular Visual Odometry: With baseline tuning, monocular DVSO achieves better average rotational and translational performance than the compared stereo systems, while without tuning it remains better than Stereo LSD-VO.On KITTI sequence 00, DVSO also reduces scale drift relative to monocular ORB-SLAM2 and DSO.
- Monocular Visual Odometry: DVSO outperforms the compared end-to-end deep visual odometry and learned scale-recovery methods on the available KITTI sequences.The compared systems include DeepVO, UnDeepVO, SfMLearner, and the scale-recovery method of Yin et al.
5 Conclusion
DVSO combines deep monocular depth prediction with geometric odometry to recover metric, scale-consistent motion from a single camera. On KITTI, it outperforms monocular and deep-learning VO methods, matches stereo systems, and can surpass them with virtual-baseline tuning.
- 5 Conclusion: DVSO uses semi-supervised deep depth prediction to initialize sparse depths and add a virtual stereo term to DSO bundle adjustment.The stacked network refines disparity estimates, while image reconstruction and Stereo DSO sparse depths provide supervision.
- 5 Conclusion: DVSO recovers metric scale and reduces scale drift in geometric monocular visual odometry.The system uses predicted monocular depth maps to initialize sparse depths at a consistent metric scale.
- 5 Conclusion: DVSO outperforms state-of-the-art monocular methods by a large margin and achieves comparable results to stereo VO methods on KITTI.The comparison includes deep-learning-based VO methods, which DVSO clearly outperforms.
- 5 Conclusion: With virtual-baseline tuning, DVSO can outperform state-of-the-art stereo VO methods while using only monocular images.The evaluated stereo baselines include Stereo LSD-VO, ORB-SLAM2 without global optimization and loop closure, and Stereo DSO.
- 5 Conclusion: The method’s practical benefit is accurate, scale-consistent odometry from a single camera, while generalization beyond driving sequences remains future work.The deep network was trained on driving sequences, and adaptation to other trajectories and environments was identified as future work.