Source-linked AI summary
DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networks
Sen Wang, Ronald Clark, Hongkai Wen, Niki Trigoni
TL;DR
Monocular VO traditionally requires engineered pipeline modules and scale-related prior knowledge. This paper replaces that pipeline with an end-to-end RCNN that learns image features and sequential dynamics from raw RGB sequences. On KITTI, DeepVO produces accurate, scaled trajectories and works in new scenarios, while the authors position it as a complement rather than a replacement for geometry-based VO.
Problem
Monocular VO systems traditionally require carefully designed pipeline modules and prior knowledge to recover absolute scale.
Method
DeepVO uses an end-to-end deep RCNN to estimate poses directly from raw RGB image sequences, combining CNN representation learning with RNN sequential modelling.
Results
DeepVO produces accurate VO results with precise scales and works well in completely new scenarios on the KITTI VO benchmark.
Takeaways & Limitations
The paper presents end-to-end Deep Learning as a viable complement to traditional geometry-based VO systems.
Takeaways & Limitations
The authors do not expect the proposed method to replace classic geometry-based VO and instead suggest combining learned models with geometry.
Abstract
from arXiv · showhide
This paper studies monocular visual odometry (VO) problem. Most of existing VO algorithms are developed under a standard pipeline including feature extraction, feature matching, motion estimation, local optimisation, etc. Although some of them have demonstrated superior performance, they usually need to be carefully designed and specifically fine-tuned to work well in different environments. Some prior knowledge is also required to recover an absolute scale for monocular VO. This paper presents a novel end-to-end framework for monocular VO by using deep Recurrent Convolutional Neural Networks (RCNNs). Since it is trained and deployed in an end-to-end manner, it infers poses directly from a sequence of raw RGB images (videos) without adopting any module in the conventional VO pipeline. Based on the RCNNs, it not only automatically learns effective feature representation for the VO problem through Convolutional Neural Networks, but also implicitly models sequential dynamics and relations using deep Recurrent Neural Networks. Extensive experiments on the KITTI VO dataset show competitive performance to state-of-the-art methods, verifying that the end-to-end Deep Learning technique can be a viable complement to the traditional VO systems.
I. INTRODUCTION
Monocular VO traditionally relies on a carefully engineered pipeline and prior knowledge for scale, while DeepVO proposes an end-to-end RCNN that estimates poses directly from RGB image sequences.
- Conventional monocular VO combines calibration, feature processing, outlier rejection, motion and scale estimation, and local optimisation.These modules require careful design and fine-tuning for performance.
- DeepVO uses deep RCNNs to learn VO representations with CNNs and model sequence dynamics with RNNs.The method combines learned feature extraction and sequential modelling.
- The proposed system estimates poses directly from raw RGB images without modules from the conventional VO pipeline.The paper specifically states that even camera calibration is unnecessary.
- The approach is intended to generalise to new environments through geometric features learned by the CNN.Sequential dependence and complex motion dynamics are automatically learned by deep RNNs.
II. RELATED WORK
Related monocular VO work is dominated by geometry-based methods, including sparse feature and direct approaches, each with distinct computational and information-use trade-offs.
- Geometry-based VO methods rely on geometric constraints extracted from imagery and include sparse feature-based and direct methods.These methods are grounded in established geometric principles.
- Sparse feature methods extract and match salient points before estimating motion using multi-view geometry.Visual SLAM or SfM can maintain feature maps for drift correction alongside pose estimation.
- Feature extraction and matching are computationally expensive, and sparse methods do not use all information in the image.Direct methods instead exploit all pixels under photometric consistency.
- Direct methods are more accurate in principle than feature-based methods and are increasingly favored in texture-less environments.The passage presents this as a general methodological contrast.
B. Methods based on Learning
Learning-based monocular VO methods infer motion from sensor data, but earlier approaches had limited coverage of RGB-image VO and did not model sequential information; DeepVO addresses this with RCNNs.
- Learning-based VO methods infer motion models from sensor readings without explicitly applying geometric theory.Earlier examples used optical flow with KNN, Gaussian Process, or SVM regression.
- Deep Learning offers an alternative because it can learn feature representations from large-scale RGB-image datasets.Traditional machine-learning methods are described as inefficient for highly nonlinear, high-dimensional RGB data.
- Prior deep-learning VO work predicted discretised direction and velocity changes after estimating depth from stereo images.The cited approach used synchrony detection between image sequences and features.
- Earlier work did not consider image sequences for sequential learning because CNNs cannot model sequential information.DeepVO tackles this limitation by leveraging RNNs.
- The proposed framework realises monocular VO end-to-end through CNN-based feature extraction and RNN-based sequential modelling.This is the framework described in the methods section.
A. Architecture of the Proposed RCNN
The proposed RCNN architecture combines CNN feature extraction with RNN sequential modelling so that monocular VO can estimate poses from image sequences over time.
- Recognition-oriented DNN architectures learn appearance and image context, whereas VO requires representations suited to geometry.The paper therefore motivates a task-specific architecture rather than directly using standard recognition networks.
- The system accepts a monocular image sequence, stacks consecutive frames, extracts features with a CNN, and passes them to an RNN for pose estimation.Images are mean-subtracted and may optionally be resized to multiples of 64.
- Combining CNNs and RNNs enables simultaneous feature extraction and sequential modelling of VO.This is identified as the central advantage of the RCNN architecture.
- The CNN configuration is documented in Table I, while Fig. 2 illustrates example tensor dimensions based on KITTI image sizes.CNN tensor dimensions vary with input-image size.
B. CNN based Feature Extraction
The VO framework combines CNN feature extraction with recurrent sequential modelling. CNN features are passed to stacked LSTMs, which use temporal dependencies to estimate poses over image sequences.
- CNN based Feature Extraction: The CNN extracts features from the concatenation of two consecutive monocular RGB images, aiming for geometric rather than appearance-based representations.The representation is intended to generalise to unknown environments.
- CNN based Feature Extraction: Raw RGB images are compressed into a compact convolutional representation that is passed to the RNN for sequential modelling.The learned representation also supports the successive sequential training procedure.
- Sequential Modelling: The RNN models dynamics and relations among CNN features implicitly, allowing it to learn sequential knowledge beyond explicitly specified physical movement and geometry models.The recurrent model is designed to discover appropriate sequence information automatically.
- Sequential Modelling: RNN hidden states retain information over time, making the current state a function of previous states and connecting current inputs with earlier sequence states.At time k, the RNN updates its hidden state and output from the convolutional feature x_k and prior state information.
- Sequential Modelling: Standard RNNs are limited in practice to short sequences by the vanishing-gradient problem, so the framework uses LSTMs to learn long-term dependencies through memory gates and units.LSTMs determine which previous hidden states to discard or retain when updating the current state.
- Sequential Modelling: The deep recurrent component stacks two LSTM layers, with one layer’s hidden states feeding the other; each layer has 1000 hidden states.The deep structure is intended to learn high-level representations and model complex dynamics.
- Sequential Modelling: The deep RNN outputs a pose estimate at each time step from visual features as the camera moves and captures images.This produces a temporally progressing pose sequence.
D. Cost Function and Optimisation
The RCNN models the conditional probability of pose sequences given monocular RGB image sequences and trains its parameters by minimising position and orientation error.
- Probabilistic Formulation: The RCNN represents the conditional probability of poses Y_t given monocular RGB images X_t up to time t.The pose and image sequences are written as Y_t = (y_1, …, y_t) and X_t = (x_1, …, x_t).
- Probabilistic Formulation: The deep RCNN performs the modelling and probabilistic inference, with parameters learned by maximising the pose-sequence probability.The optimisation targets the conditional model defined for the pose sequence.
- Cost Function: Training minimises Euclidean distance between ground-truth and estimated poses at each time step.The loss includes errors for both position and orientation.
- Cost Function: Orientation is represented with Euler angles rather than quaternions because quaternion unit constraints hinder deep-learning optimisation and degrade orientation estimates in practice.The scale factor κ is 100 in the experiments and balances position and orientation weights.
IV. EXPERIMENTAL RESULTS
The method is evaluated on the KITTI VO/SLAM benchmark, using LIBVISO2 as a comparison because monocular VO methods generally require scale alignment or recovery.
- Experimental Results: The KITTI evaluation compares the proposed monocular VO approach with the open-source LIBVISO2 library, whose fixed camera height recovers scale.Most existing monocular VO algorithms do not estimate absolute scale directly, so their results require alignment with ground truth.
A. Training and Testing
KITTI contains 22 image sequences, but ground truth is available for only 11. The benchmark is challenging because recordings are low-rate urban drives with dynamic objects and speeds up to 90 km/h.
- Training and Testing: KITTI provides ground truth for Sequences 00–10, while Sequences 11–21 contain only raw sensor data.The benchmark has 22 sequences in total.
- Training and Testing: The dataset challenges monocular VO with 10 fps urban imagery, many dynamic objects, and driving speeds reaching 90 km/h.These conditions make the benchmark difficult for monocular VO algorithms.
1) Dataset:
The experiments use KITTI sequences with ground truth for quantitative evaluation, reserving relatively long sequences for training and separate sequences for testing. A second setup evaluates generalisation to previously unseen environments without ground truth.
- Dataset split: Sequences 00–10 provide ground truth for quantitative evaluation, while relatively long Sequences 00, 02, 08 and 09 train the first model.The training set is segmented into 7410 samples, and Sequences 03, 04, 05, 06, 07 and 10 are used for testing.
- Generalisation: A separate generalisation experiment trains on all Sequences 00–10 and tests on Sequences 11–21, which lack ground truth for training.
- Training setup: Training uses Theano, a NVIDIA Tesla K40 GPU, Adagrad, dropout and early stopping, with training running for up to 200 epochs at learning rate 0.001.
- Training analysis: The study examines overfitting by comparing training, validation and VO results for over-fitted and well-fitted models.
3) How overfitting affects the VO:
The experiments show that overfitting can improve training-set trajectories while degrading testing performance, and that DeepVO’s errors and generalisation depend on path length, speed and environment.
- Overfitting effects: The authors report that orientation is more prone to overfitting than position, likely because orientation changes are usually smaller.
- Benchmark results: DeepVO is consistently better than monocular VISO2 except for slightly higher translational errors at high speeds, while remaining worse than stereo VISO2.Average translational and rotational RMSEs are evaluated across path lengths from 100 to 800 meters and different speeds.
- Speed effects: High-speed translational errors are attributed to limited high-speed training data, whereas rotational errors decrease at high velocities in the tested driving data.
- Path-length effects: As trajectory length increases, DeepVO’s translation and rotation errors significantly decrease and approach stereo VISO2 performance.
- Generalisation: On qualitative tests in unknown scenarios, DeepVO trajectories are much better than monocular VISO2 and roughly similar to stereo VISO2, but Sequence 12 has high localisation errors.The Sequence 12 trajectory shape remains close to stereo VISO2’s despite the high localisation errors.
V. CONCLUSIONS
The paper presents an end-to-end monocular VO system based on Deep RCNNs that avoids the conventional VO pipeline and estimates scale directly. KITTI experiments show accurate, scaled results in new scenarios, while the method is intended as a complement rather than a replacement for geometry-based VO.
- Conclusion: Deep RCNNs combine CNN-based representation learning with RNN-based sequential modelling for end-to-end monocular VO.
- Conclusion: The system estimates poses without conventional VO modules, camera calibration or prior knowledge for recovering absolute scale.
- Conclusion: KITTI VO benchmark results show accurate VO with precise scales and good performance in completely new scenarios.
- Conclusion: The method is presented as a viable complement to classic geometry-based VO, not as its replacement.