Source-linked AI summary
VidLoc: A Deep Spatio-Temporal Model for 6-DoF Video-Clip Relocalization
Ronald Clark, Sen Wang, Andrew Markham, Niki Trigoni, Hongkai Wen
TL;DR
Single-image localization methods often ignore temporal smoothness even though image sequences are readily available, producing noisy pose estimates. The paper proposes a CNN-RNN recurrent model using bidirectional temporal context, adds probabilistic pose prediction, and evaluates it on indoor and outdoor datasets. The method improves localization over single-image Posenet across tested scenes, while depth utilization and variance prediction remain limited.
Problem
Learning-based monocular localization methods treat frames as uncorrelated, despite available sequences and perceptual aliasing that makes single-frame global pose prediction ambiguous.
Method
A CNN-RNN model uses bidirectional LSTM temporal context for sequence-based 6-DoF pose prediction and mixture density networks for multi-modal uncertainty.
Results
The proposed method significantly outperforms Posenet in all tested 7-Scenes scenes, with a 23.4%−55% increase in accuracy.
Takeaways & Limitations
Using sequential images improves global relocalization, mitigates perceptual aliasing, and improves localization accuracy relative to single-image estimates.
Takeaways & Limitations
The method cannot exploit depth as fully as SCoRe-Forest, and predicted uncertainty variance is often high.
Abstract
from arXiv · showhide
Machine learning techniques, namely convolutional neural networks (CNN) and regression forests, have recently shown great promise in performing 6-DoF localization of monocular images. However, in most cases image-sequences, rather only single images, are readily available. To this extent, none of the proposed learning-based approaches exploit the valuable constraint of temporal smoothness, often leading to situations where the per-frame error is larger than the camera motion. In this paper we propose a recurrent model for performing 6-DoF localization of video-clips. We find that, even by considering only short sequences (20 frames), the pose estimates are smoothed and the localization error can be drastically reduced. Finally, we consider means of obtaining probabilistic pose estimates from our model. We evaluate our method on openly-available real-world autonomous driving and indoor localization datasets.
1. Introduction
Monocular 6-DoF localization is important but traditional feature-matching methods face correspondence and appearance-change challenges. Learning-based single-image regressors avoid expensive matching yet ignore temporal information, motivating a CNN-RNN model that uses image sequences and estimates pose uncertainty.
- 6-DoF monocular localization supports SLAM, autonomous navigation, first-response robotics, and wearable-device applications.
- Traditional localization builds feature-based maps and uses correspondences with RANSAC, but hand-crafted local features make reliable matching difficult under noise, outliers, and appearance changes.
- Perceptual aliasing can make visually similar locations impossible to distinguish from single frames, as illustrated by views from the bottom and top of a staircase.
- CNNs and random forests directly regress pose without expensive feature extraction or matching, but independent frame estimates are noisy on image sequences.
- The paper proposes a recurrent model for global localization from monocular image sequences, integrates instantaneous pose covariances, and evaluates it against post-processing smoothing.
2. Proposed Model
VidLoc combines CNN image features with a bidirectional LSTM to predict global 7-dimensional poses from video clips, using past and future context. It also models pose uncertainty with mixture density networks to represent multi-modal posterior distributions.
- Temporal modelling: VidLoc processes video frames with a CNN and integrates temporal information through a bidirectional LSTM.The CNN produces feature vectors, which the LSTM maps to global pose outputs.
- Temporal modelling: Temporal regularity provides useful information because adjacent frames often show the same objects and constrain plausible inter-frame motion.
- Pose output: The model outputs each pose as a 7-dimensional vector containing translation and orientation quaternion parameters.
- Temporal modelling: Bidirectional processing combines past and future context for each frame using forward and backward hidden states.The two hidden states are concatenated into a single hidden state before computing the output pose.
- Network loss: Training minimizes a weighted sum of translation and orientation error magnitudes across temporal frames using back-propagation through time and Adam.The loss separates translation and orientation components before combining their error magnitudes.
- Probabilistic pose estimates: The standard squared-error objective models only a unimodal conditional mean, which is inadequate when perceptual aliasing creates multiple plausible locations.
- Probabilistic pose estimates: A mixture density network replaces the single Gaussian with a mixture model, allowing the predicted pose posterior to be multi-modal.The mixture coefficients depend on the input image and sum to one, and the network is trained by maximizing training-data likelihood.
3. Experiments
Experiments evaluate VidLoc on indoor 7-Scenes and outdoor RobotCar sequences against established monocular localization methods. Results examine accuracy, sequence length, computational cost, uncertainty, and global relocalization.
- Datasets: Experiments use the Microsoft 7-Scenes indoor dataset and Oxford RobotCar outdoor autonomous-driving dataset.The datasets represent indoor human motion and outdoor car-driving scenarios.
- 7-Scenes results: VidLoc significantly outperforms Posenet across all 7-Scenes test scenes, increasing accuracy by 23.4%−55%.SCoRe-forest performs better than RGB-only VidLoc, but requires depth and may lack estimates for some frames.
- Sequence length: Increasing sequence length improves localization accuracy, including for sequences longer than 200 frames, although very long sequences yield diminishing returns.The diminishing returns may reflect the limited utility of very long-term dependencies for predicting the current pose.
- Timings: Per-frame inference takes 18ms with GoogleNet and 43ms with VGG16 on a Titan X Pascal GPU.Each frame uses the current image and the RNN hidden state from the previous time step.
- RobotCar results: On RobotCar, longer sequences improve relocalization; lengths 50 and 100 match roads consistently and track motion shape.For sequences of length 100, more than half of VidLoc poses are within 20 meters, compared with less than 15% for Posenet.
4. Conclusion
The proposed 6-DoF video-clip relocalization approach uses temporal smoothness to improve global pose accuracy and examines localization-error distributions for sequences of length 100. Future work includes making better use of depth information.
- The approach exploits temporal smoothness in video streams to improve the accuracy of global 6-DoF pose estimates.
- Figure 11 presents the distribution and histogram of localization errors across all sequences with length 100.
- Future work will investigate using depth information more effectively, potentially by incorporating geometrical information into the model.