PaperlayerView exact revision on arXiv

Source-linked AI summary

D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry

Nan Yang, Lukas von Stumberg, Rui Wang, Daniel Cremers

arXiv:2003.01060v2cs.CVcs.AI

TL;DR

Monocular visual odometry remains less robust and less competitive than stereo or visual-inertial approaches, motivating better use of learned predictions. D3VO combines self-supervised depth, pose, and uncertainty with direct VO, achieving strong depth and tracking performance across KITTI and EuRoC MAV using one camera.

  • Problem

    Monocular VO has low hardware cost but suffers from scale drift and low robustness, while learned depth, pose, and uncertainty have not been fully integrated into geometric VO.

  • Method

    D3VO trains depth and adjacent-frame pose self-supervised on stereo videos, predicts brightness transformations and photometric uncertainty, and integrates these outputs into direct VO tracking and optimization.

  • Results

    D3VO achieves state-of-the-art monocular depth and camera tracking, outperforming traditional monocular VO and matching stereo/LiDAR KITTI and visual-inertial EuRoC MAV results with one camera.

  • Takeaways & Limitations

    Combining deep depth, pose, and uncertainty can substantially strengthen a monocular direct VO pipeline across outdoor and indoor benchmarks.

Abstract

from arXiv · show

We propose D3VO as a novel framework for monocular visual odometry that exploits deep networks on three levels -- deep depth, pose and uncertainty estimation. We first propose a novel self-supervised monocular depth estimation network trained on stereo videos without any external supervision. In particular, it aligns the training image pairs into similar lighting condition with predictive brightness transformation parameters. Besides, we model the photometric uncertainties of pixels on the input images, which improves the depth estimation accuracy and provides a learned weighting function for the photometric residuals in direct (feature-less) visual odometry. Evaluation results show that the proposed network outperforms state-of-the-art self-supervised depth estimation networks. D3VO tightly incorporates the predicted depth, pose and uncertainty into a direct visual odometry method to boost both the front-end tracking as well as the back-end non-linear optimization. We evaluate D3VO in terms of monocular visual odometry on both the KITTI odometry benchmark and the EuRoC MAV dataset.The results show that D3VO outperforms state-of-the-art traditional monocular VO methods by a large margin. It also achieves comparable results to state-of-the-art stereo/LiDAR odometry on KITTI and to the state-of-the-art visual-inertial odometry on EuRoC MAV, while using only a single camera.

1. Introduction

D3VO addresses the limited robustness and performance of monocular visual odometry by combining learned depth, pose, and uncertainty with direct geometric VO. Its self-supervised training models temporal information and illumination variation, while the resulting predictions support tracking and optimization.

  • Monocular VO offers lower hardware and calibration costs but suffers from scale drift and low robustness relative to stereo or visual-inertial methods.
  • D3VO uses deep networks for monocular depth, relative pose, and photometric uncertainty within a sparse direct VO framework.These predictions are integrated into both front-end tracking and back-end non-linear optimization.
  • The self-supervised network learns depth from stereo videos and adjacent-frame pose by minimizing photometric errors from stereo and temporal warping.Predictive brightness parameters align source and target illumination during training.
  • Photometric uncertainty down-weights pixels likely to violate brightness constancy because of reflections, high-frequency regions, non-Lambertian surfaces, or moving objects.
  • D3VO incorporates deep pose predictions into tracking and optimization to improve robustness during low-texture areas or fast motion.The approach targets failure modes that traditionally motivate adding an IMU.
  • D3VO achieves state-of-the-art monocular depth and camera tracking results, comparable to stereo/LiDAR methods on KITTI and visual-inertial methods on EuRoC MAV.The evaluations use only a single camera.

2. Related Work

Prior work established strong learned monocular depth and pose estimation, but left opportunities to model illumination-aware uncertainty and integrate learned predictions more fully into traditional VO. D3VO targets these gaps by combining brightness-aligned uncertainty estimation with geometric optimization.

  • Supervised and self-supervised networks have achieved strong monocular depth estimation from single images, with self-supervised methods also predicting adjacent-frame poses.
  • Prior uncertainty methods estimate photometric and depth uncertainty but do not explicitly model brightness changes across images.
  • D3VO predicts photometric uncertainty from brightness-aligned images and incorporates the learned uncertainties into traditional VO systems.
  • End-to-end learned VO and SLAM methods predict poses or jointly optimize learned geometry, but remain inferior in pose accuracy to classical stereo or visual-inertial approaches.

3. Method

D3VO combines self-supervised depth, pose, brightness, and uncertainty prediction with direct visual odometry. These predictions support photometric training and improve initialization, weighting, and optimization in the VO pipeline.

  • Self-supervised Network: D3VO’s self-supervised network predicts depth, pose, uncertainty, and affine brightness parameters from stereo video without external supervision.DepthNet predicts depth and PoseNet predicts motion; training minimizes photometric reprojection errors from temporal and stereo images.
  • Brightness transformation: Predicted affine brightness parameters align images with different exposure, making the brightness constancy assumption more suitable for self-supervised training.The parameters are learned without supervision and transform target images toward the source image’s illumination.
  • Photometric uncertainty: The network predicts per-pixel photometric uncertainty so residuals from likely brightness-constancy violations receive lower weight.Higher uncertainty is assigned to regions affected by reflectance, moving objects, and depth discontinuities, and the learned uncertainty map is used in D3VO.
  • D3VO integration: The photometric backend minimizes weighted residuals over keyframes and observed points using learned uncertainty alongside brightness parameters and robust norms.The weighting can capture higher-level noise patterns beyond local image gradients, including reflections and moving objects.
  • D3VO integration: D3VO initializes direct VO points with predicted metric-scale depth and uses predicted poses in front-end tracking and backend optimization.Deep poses initialize tracking and bundle adjustment, enter the factor graph, and regularize the photometric bundle-adjustment energy.

4. Experiments

Experiments on KITTI and EuRoC MAV evaluate D3VO's depth estimation and monocular visual odometry against established baselines. Results show improvements over self-supervised depth and monocular VO methods, with competitiveness against semi-supervised, stereo/LiDAR, and visual-inertial systems.

  • Monocular Depth Estimation: On KITTI Eigen, the proposed depth network outperforms Monodepth2 on all metrics and is comparable to semi-supervised DVSO.The evaluation includes brightness-transformation and photometric-uncertainty ablations; the uncertainty component contributes much of the KITTI improvement.
  • Monocular Depth Estimation: On EuRoC MAV, predictive brightness transformation parameters significantly improve monocular depth estimation, while cross-scenario generalization remains challenging.Models trained on one scene and tested on another perform worse than models trained on both MH and V sequences.
  • Monocular Visual Odometry: D3VO achieves the best average performance on the KITTI Odometry test split despite using only monocular input.The evaluation compares D3VO with state-of-the-art monocular and stereo methods and includes ablations of deep depth, pose, and uncertainty.
  • Monocular Visual Odometry: D3VO outperforms all compared end-to-end methods on KITTI sequences 09 and 10 by a notable margin.The comparison is restricted to these sequences because most end-to-end methods report results only there.
  • Monocular Visual Odometry: On challenging EuRoC MAV sequences, D3VO achieves results comparable to state-of-the-art monocular and stereo VIO methods using a single camera.Predicted pose improves performance significantly on V1 03 difficult and V2 03 difficult, where camera motion is violent.

5. Conclusion

D3VO combines deep monocular depth, photometric uncertainty, and relative pose predictions to enhance geometric monocular visual odometry. It achieves state-of-the-art results on KITTI and EuRoC MAV while using only one camera.

  • D3VO integrates monocular depth, photometric uncertainty, and relative camera pose predictions into a geometric monocular VO method.
  • The self-supervised depth network models illumination changes using predictive brightness transformation parameters.
  • Predicted depth, uncertainty, and pose are incorporated into front-end tracking and back-end nonlinear optimization.
  • D3VO achieves state-of-the-art performance on KITTI Odometry and EuRoC MAV, rivaling leading mono-inertial and stereo-inertial methods with one camera.

A. Network Training Details

The networks are trained in PyTorch on resized KITTI and EuRoC MAV images using a ResNet-18 encoder and scheduled learning rates. Brightness parameters share values across channels and exclude over-exposed pixels.

  • DepthNet and PoseNet are implemented in PyTorch and trained on a single Titan X Pascal GPU with images resized to 512 × 256.
  • DepthNet uses ImageNet-pretrained ResNet-18 features, while grayscale EuRoC MAV images are duplicated into three-channel inputs.
  • The learning rate starts at 10^-4 and decreases to 10^-5 for the final five epochs.
  • Brightness transformation parameters are shared across input channels, and over-exposure pixels are masked during affine brightness estimation.
  • Training uses four output scales with λs = 10^-3 × 1 2s−1 and sets β = 10^-2.

B. Network Architectures

D3VO uses UNet-like DepthNet and PoseNet architectures, then applies their predictions within a factor-graph front-end and direct image-alignment pipeline.

  • Network Architectures: DepthNet combines a ResNet-18 encoder, a skip-connected decoder, and three-channel outputs for depth, scale-related output, and uncertainty.
  • Network Architectures: PoseNet takes two channel-wise concatenated images and predicts relative pose together with affine brightness parameters a and b.
  • Network Architectures: DepthNet and PoseNet follow UNet-like convolutional designs, with PoseNet omitting the explainability-mask decoder.
  • C. Factor Graph of Front-end Tracking: The front-end tracks each non-keyframe against the latest keyframe using direct image alignment.
  • C. Factor Graph of Front-end Tracking: PoseNet predictions provide prior factors between consecutive frames, while the oldest non-keyframe is marginalized as new frames arrive.

D. Additional Experiments on Brightness Parameters

Additional experiments examine brightness alignment and trajectory evaluation under different alignment choices. Predicted brightness transformation reduces photometric error, while SE(3) alignment prevents early pose errors from dominating ATE.

  • Brightness Parameters: Predicted brightness transformations substantially decrease average photometric error on V2 03 difficult stereo pairs.
  • Brightness Parameters: Ground-truth depth maps are used to project visible 3D points between corresponding stereo images for photometric-error measurement.
  • Trajectory Evaluation: Without SE(3) alignment, inaccurate initial pose estimates can produce a large overall absolute trajectory error.

E. Absolute Translational Error on KITTI

The section reports KITTI absolute translational error as RMSE, distinguishing results without and with SE(3) alignment. It also notes that inaccurate initial poses can inflate overall ATE and presents cross-dataset uncertainty generalization results.

  • E. Absolute Translational Error on KITTI: ATE is reported as translational-error RMSE, with separate KITTI results without and with SE(3) alignment.The upper table portion uses no alignment, while the lower portion uses SE(3) alignment.
  • E. Absolute Translational Error on KITTI: Large ATE can result when pose estimates are less accurate in the initial frames.The initial-frame errors may accumulate into a large overall absolute translational error.
  • Cross-dataset evaluation: Figure 10 evaluates Cityscapes using a model trained on KITTI.The figure concerns cross-dataset evaluation of the model.
  • Cross-dataset evaluation: The network generalizes depth and uncertainty prediction, assigning high uncertainty to reflectance, object boundaries, high-frequency regions, and moving objects.These uncertainty patterns are reported for Cityscapes evaluation with KITTI-trained models.
Loading 2003.01060v2…