Source-linked AI summary
UnDeepVO: Monocular Visual Odometry through Unsupervised Deep Learning
Ruihao Li, Sen Wang, Zhiqiang Long, Dongbing Gu
TL;DR
UnDeepVO tackles the difficulty and expense of obtaining labeled pose and depth data for visual odometry. It trains pose and depth networks unsupervised with stereo image pairs and spatial-temporal geometric losses, then tests with consecutive monocular images. On KITTI, it reports good monocular pose-estimation performance while recovering scaled poses and depth maps.
Problem
Ground-truth camera poses and depth images are difficult and expensive to obtain, and labeled datasets remain limited for supervised VO training.
Method
UnDeepVO trains pose and depth estimators unsupervised on stereo image pairs using spatial and temporal geometric losses, then estimates scaled pose and depth from monocular images.
Results
KITTI experiments show good pose-estimation performance for monocular cameras, with scaled poses and dense depth maps produced without scale post-processing.
Takeaways & Limitations
The system demonstrates monocular VO with recovered absolute scale while requiring stereo imagery for training rather than labeled datasets.
Takeaways & Limitations
Depth estimation is inferior to, partly because UnDeepVO uses less training data, lower input resolution, a different architecture, and temporal-loss noise from moving objects.
Abstract
from arXiv · showhide
We propose a novel monocular visual odometry (VO) system called UnDeepVO in this paper. UnDeepVO is able to estimate the 6-DoF pose of a monocular camera and the depth of its view by using deep neural networks. There are two salient features of the proposed UnDeepVO: one is the unsupervised deep learning scheme, and the other is the absolute scale recovery. Specifically, we train UnDeepVO by using stereo image pairs to recover the scale but test it by using consecutive monocular images. Thus, UnDeepVO is a monocular system. The loss function defined for training the networks is based on spatial and temporal dense information. A system overview is shown in Fig. 1. The experiments on KITTI dataset show our UnDeepVO achieves good performance in terms of pose accuracy.
I. INTRODUCTION
UnDeepVO addresses the cost and scarcity of labeled pose or depth data with an unsupervised monocular VO system. It uses stereo training to recover absolute scale and reports good monocular pose-estimation performance on KITTI.
- Model-based VO can be fragile under camera-parameter variation, featureless scenes, motion blur, and lighting changes.
- Supervised deep VO methods require ground-truth camera poses or depth images, which are difficult and expensive to obtain, while labeled datasets remain limited.
- UnDeepVO is proposed as a monocular VO system based on an unsupervised deep-learning scheme.
- Stereo image pairs during training enable both estimated poses and dense depth maps to have absolute scale.
- KITTI experiments show good pose-estimation performance for monocular cameras, while training requires stereo imagery but no labeled datasets.
II. SYSTEM OVERVIEW
UnDeepVO combines pose and depth estimators that operate on consecutive monocular images at test time. Its architecture and stereo-based training scheme support scaled pose and depth prediction through geometric constraints.
- The pose estimator uses a VGG-based CNN and predicts the 6-DoF transformation between two consecutive monocular images.
- Separate fully connected branches decouple translation and rotation, enabling separate normalization weights during unsupervised training.
- The depth estimator uses an encoder-decoder architecture that directly predicts dense depth maps rather than disparity images.
- During training, left and right stereo images are processed to estimate sequences’ poses and depths, which are combined with stereo geometry to construct the loss.
- Spatial and temporal geometric consistencies encode projective constraints across stereo pairs and consecutive monocular images.
III. OBJECTIVE LOSSES FOR UNSUPERVISED TRAINING
UnDeepVO trains without labeled data by backpropagating losses built from geometric constraints. Spatial losses support scaled depth recovery from stereo pairs, while temporal losses reduce camera-motion errors from consecutive monocular images.
- The total training loss combines spatial image losses and temporal image losses.
- Spatial image losses use stereo image pairs to drive recovery of scaled depth maps.
- Temporal image losses use two consecutive monocular images to minimize camera-motion errors.
- Spatial losses comprise left-right photometric consistency, disparity consistency, and pose consistency losses, supporting absolute-scale recovery.
1) Photometric Consistency Loss:
The photometric consistency loss uses stereo correspondences derived from predicted depth to synthesize one image from the other and compare them.
- Each pixel in the overlapping stereo views is matched using a horizontal distance Dp.The spatial constraint is ul = ur and vl = vr + Dp.
- Dp is computed from the stereo baseline, focal length, and corresponding-pixel depth.The relation uses B, f, and Ddep.
- The predicted depth-derived Dp map enables spatial-transformer synthesis of one stereo image from the other.This creates synthesized left and right images for the photometric comparison.
- The stereo photometric loss combines an L1 term with an SSIM term.The formulation applies the loss to original and synthesized left-right image pairs.
2) Disparity Consistency Loss:
Disparity consistency constrains the left and right disparity maps using the disparity implied by the stereo geometry.
- The left and right disparity maps are constrained by the same Dp used for stereo correspondence.Disparity is the inverse-depth representation used in this consistency term.
- The disparity map is defined by scaling Dp according to the original image width IW.The equation is given as Ddis = Dp × IW.
- Disparity consistency losses compare the predicted disparity maps with disparity maps synthesized from Dp.Separate left and right disparity consistency terms are introduced.
3) Pose Consistency Loss:
Pose consistency compares transformations predicted from the left and right image sequences, while temporal losses connect consecutive monocular frames.
- Pose Consistency Loss: The system predicts 6-DoF camera transformations separately from the left and right image sequences.The two predictions should ideally be basically identical.
- Pose Consistency Loss: Pose consistency penalizes differences between the left- and right-sequence predictions.Separate position and orientation consistency weights are used.
- Temporal Image Losses: Temporal loss uses geometric constraints between consecutive monocular images and includes photometric consistency and 3D geometric registration losses.This temporal component is described as essential for recovering 6-DoF camera motion.
1) Photometric Consistency Loss:
The temporal photometric loss reconstructs corresponding pixels across consecutive monocular frames using depth and estimated camera motion.
- Corresponding pixels in consecutive frames are related through the camera intrinsics, depth, and inter-frame transformation.The transformation maps coordinates from frame k to frame k+1.
- Photometric losses are computed between the original and synthesized images in the monocular sequence.The loss is based on the projective photometric error.
- The system synthesizes one consecutive frame from the other using estimated poses and a spatial transformer.The synthesized image is then compared with the target frame.
2) 3D Geometric Registration Loss:
The 3D geometric registration loss aligns transformed point clouds across consecutive camera frames, contributing to a final loss that combines spatial and temporal constraints.
- 3D geometric registration estimates camera transformation by aligning two point clouds, similarly to Iterative Closest Point.Points from one camera coordinate frame are transformed using T_k,k+1 before registration.
- The loss computes L_k geo and L_k+1 geo by comparing point clouds with their transformed counterparts using an L1 loss.
- The final UnDeepVO objective combines spatial and temporal losses with 3D geometric registration and pose consistency losses.This combination is reported as enabling recovery of scaled camera poses and depth maps.
- The evaluation used NVIDIA Tesla P100 GPUs for training and achieved less than 400MB GPU memory usage with 40Hz real-time pose estimation.Testing used a laptop with an NVIDIA GeForce GTX 980M and Intel Core i7 2.7GHz CPU.
- Training used Adam for up to 20–30 epochs, with β1 = 0.9 and β2 = 0.99.
- In Fig. 4, UnDeepVO and SfMLearner use 416×128 images, whereas VISO2-M uses 1242×376 images.SfMLearner results receive 7-DoF alignment because it cannot recover scale.
- The networks used 416 × 128 inputs, two-image pose-estimator sequences, higher-resolution loss computation, and several augmentation methods.Augmentations included color, rotational, and left-right pose-estimation transformations.
A. Trajectory Evaluation
The KITTI trajectory evaluation compares UnDeepVO with monocular and stereo-based systems under differing resolutions and scale-processing conditions. UnDeepVO is reported as qualitatively closest to ground truth and comparable to stereo VISO2-S on sequences without pose ground truth.
- Trajectory Evaluation: The evaluation uses KITTI sequences 00–08 for training and compares UnDeepVO with SfMLearner, monocular VISO2-M, and ORB-SLAM-M.All methods use monocular images for testing, while scale post-processing is applied to methods unable to recover scale.
- Trajectory Evaluation: Trajectory evaluation reports average translational RMSE drift (%) and rotational RMSE drift (°/100m) over lengths of 100m–800m.
- Trajectory Evaluation: For sequences 00–10, UnDeepVO trajectories are qualitatively closest to ground truth among the compared methods.KITTI provides ground-truth 6-DoF poses for these sequences.
- Trajectory Evaluation: For sequences 11–21 without ground truth, UnDeepVO trajectories are compared with monocular VISO2-M and stereo VISO2-S.The reported UnDeepVO performance is comparable to VISO2-S.
- Trajectory Evaluation: UnDeepVO achieves good pose estimation performance among monocular methods using low-resolution images without scale post-processing.SfMLearner and ORB-SLAM-M receive 7-DoF alignment, while VISO2-M and ORB-SLAM-M results use 1242×376 inputs.
B. Depth Estimation Evaluation
UnDeepVO produces scaled dense depth maps, with qualitative estimates shown alongside RGB inputs and quantitative evaluation on KITTI. It outperforms supervised and unsupervised no-scale methods but trails under the reported setup.
- UnDeepVO produces scaled depth maps, with RGB images and corresponding estimated depth images shown in Fig. 6.The estimates explicitly recover different depths for cars, trees, and tree trunks.
- UnDeepVO outperforms the supervised method and the unsupervised method without scale on the reported KITTI depth-estimation results.
- UnDeepVO performs worse than on depth estimation under the reported comparison.
- The comparison is based on KITTI depth estimation using the split of Eigen et al. [23].
- Three factors may explain the gap with: less training data, lower input resolution and a different architecture, and temporal-loss noise from moving objects.UnDeepVO used only the KITTI odometry dataset, 416 × 128 inputs, and a non-ResNet architecture.