Source-linked AI summary
VINet: Visual-Inertial Odometry as a Sequence-to-Sequence Learning Problem
Ronald Clark, Sen Wang, Hongkai Wen, Andrew Markham, Niki Trigoni
TL;DR
Monocular visual-inertial odometry must support accurate navigation while traditional methods require strict calibration and synchronization. VINet addresses this with an on-manifold, end-to-end sequence-to-sequence model that fuses camera and IMU data through learned intermediate representations. It performs competitively with traditional methods under accurate calibration and can outperform them under calibration and synchronization errors.
Problem
Traditional visual-inertial odometry can achieve high accuracy but requires strict camera–IMU calibration and synchronization.
Method
VINet is a fully differentiable CNN-RNN sequence-to-sequence model that maps monocular RGB images and six-dimensional IMU data to a seven-dimensional pose change.
Results
VINet performs competitively with traditional methods and remains robust to calibration and synchronization errors, including by ignoring IMU data when streams are entirely unsynchronized.
Takeaways & Limitations
VINet shows that learned visual-inertial fusion can match traditional approaches while learning robustness to calibration errors.
Abstract
from arXiv · showhide
In this paper we present an on-manifold sequence-to-sequence learning approach to motion estimation using visual and inertial sensors. It is to the best of our knowledge the first end-to-end trainable method for visual-inertial odometry which performs fusion of the data at an intermediate feature-representation level. Our method has numerous advantages over traditional approaches. Specifically, it eliminates the need for tedious manual synchronization of the camera and IMU as well as eliminating the need for manual calibration between the IMU and camera. A further advantage is that our model naturally and elegantly incorporates domain specific information which significantly mitigates drift. We show that our approach is competitive with state-of-the-art traditional methods when accurate calibration data is available and can be trained to outperform them in the presence of calibration and synchronization errors.
Introduction
Monocular visual-inertial odometry offers an inexpensive route to GPS-denied navigation but remains sensitive to calibration and synchronization. VINet addresses this challenge by framing VIO as an end-to-end sequence-to-sequence learning problem.
- Monocular camera–IMU fusion is attractive for GPS-denied navigation because the sensors are cheap, ubiquitous, and complementary.
- State-of-the-art traditional VIO methods achieve impressive accuracy but require strict calibration and synchronization.
- VINet treats visual-inertial odometry as a sequence-to-sequence regression problem for fully end-to-end trainable motion estimation.
- Unlike standard frameworks, VINet hides its parameters from users and learns them from data rather than requiring setup specification.
- VINet introduces a recurrent architecture and differentiable pose concatenation layer designed to conform predictions to the SE(3) manifold.
Related work
Prior work develops monocular visual odometry and visual-inertial methods through hand-designed geometric pipelines, while deep learning had not yet supplied end-to-end monocular visual-inertial odometry.
- Traditional monocular visual odometry extracts and matches image features, computes optical flow, and estimates incremental camera motion.
- SVO improves efficiency by operating directly on image patches with probabilistic depth filters and whole-image alignment.
- SVO requires suitable hardware and bootstrapping, while its probabilistic formulation makes tuning difficult.
- Existing deep-learning work included stereo visual odometry and CNN-based egomotion, but not end-to-end monocular visual-inertial odometry.
- The paper compares against OK-VIS and uses MSCKF for robustness evaluation.
Networks
The network section introduces recurrent architectures for retaining temporal information, including LSTM gating and convolutional processing for high-dimensional inputs, alongside VINet’s multirate design.
- RNNs maintain an internal state by operating on current inputs and delayed hidden layers or outputs.
- LSTMs extend recurrent networks with gates that selectively store and forget memories, supporting longer-term input trends.
- The network’s weights and biases fully parameterize its operations and are learned during training.
- VINet uses a core LSTM at camera rate and an IMU LSTM at IMU rate to process the sensors’ different temporal frequencies.
- Convolutional layers can be integrated into recurrent networks to extract features from high-dimensional inputs such as images.
Our Approach
VINet is an end-to-end CNN-RNN sequence-to-sequence model that fuses monocular images and IMU data to estimate camera motion while respecting SE(3) structure.
- Model overview: VINet maps monocular RGB images and six-dimensional IMU measurements to seven-dimensional pose changes using a fully differentiable CNN-RNN.The output contains three translation components and a four-dimensional orientation quaternion.
- Manifold-aware estimation: The network predicts frame-to-frame motion in se(3), converts it to SE(3) with an exponential map, and composes motions into a trajectory.This keeps the approximated frame-to-frame motion bounded while trajectory composition occurs on SE(3).
- Manifold-aware estimation: The SE(3) composition layer is parameter-free and concatenates transformations between frames on the SE(3) manifold.The model uses two-layer LSTMs with 1000 cells and a CNN containing 55M trainable weights.
- Recurrent estimation: A feedback connection returns the concatenated output pose to the Core-LSTM at the next timestep, supporting accumulated displacement estimation.The recurrent hidden state is also carried across timesteps.
- Sensor fusion: The IMU-LSTM processes faster inertial measurements, passes its hidden activation to the Core-LSTM, and accommodates multi-rate sensing.The IMU stream typically arrives around ten times faster than visual data.
- Sensor fusion: The CNN extracts motion features from sequential images, while the Core-LSTM fuses intermediate visual and inertial representations to produce pose estimates.The CNN takes two sequential images and produces a motion-describing feature vector for the Core-LSTM.
Training
VINet trains recurrently with BPTT and RMSProp-adjusted SGD, using sliding windows and complementary local and accumulated-pose losses to improve optimization.
- Optimization: BPTT unfolds the network for T timesteps, computes activations forward, backpropagates derivatives backward, and sums them across timesteps.SGD with an RMSProp adaptive learning rate updates the network weights.
- Sequence training: Sliding-window training reduces memory use for long sequences while carrying the LSTM hidden state between windows to preserve continuity.The batch structure is illustrated in Figure 4.
- Loss design: Training uses a frame-to-frame se(3) loss, a full concatenated SE(3) pose loss, or joint optimization of both losses.The two losses target local motion predictions and pose relative to the sequence start.
- Loss design: Joint training starts with λ2/λ1 ≈100 for the se(3) loss and reduces it to λ2/λ1 ≈0.1 during later epochs.The schedule first emphasizes frame-to-frame learning and later fine-tunes concatenated pose estimation.
Results
VINet is evaluated on indoor EuRoC and outdoor KITTI data for accuracy, robustness to calibration and synchronization errors, and training behavior. It remains competitive with traditional methods while showing improved robustness to sensor errors, though orientation estimation can lag a traditional method on KITTI.
- EuRoC MAV: VINet was evaluated on EuRoC MAV data against the optimization-based OK-VIS method using reconstructed 6D trajectories.The experiments used tightly synchronized camera and IMU data with Vicon ground truth.
- Calibration robustness: VINet trained without augmentation performs competitively with OKVIS and does not fail under high calibration errors.The robustness test compared estimated MAV trajectories across varying mis-calibration levels.
- Calibration robustness: Calibration augmentation makes VINet’s accuracy decrease more slowly as calibration errors increase.Training on mis-calibrated data makes the network more robust to mis-calibration errors.
- Synchronization robustness: VINet copes with time-synchronization errors even better than extrinsic calibration errors and ignores IMU data when streams are entirely unsynchronized.In that case, the network resorts to vision-only motion estimation.
- Training performance: Joint training converges more quickly to low-error estimates than frame-to-frame-only or full-pose-only training.Frame-to-frame training converges slowly, while full-pose training converges to a high-error estimate.
- KITTI: On KITTI, VINet outperforms image-only VNet and has lower translational error than Viso2, but IMU-Viso2 estimates orientation better.The authors suggest VINet’s translational accuracy may reflect learning scale from both image and IMU data.
Conclusion and Future Work
The paper presents VINet as an end-to-end monocular visual-inertial navigation system that performs on par with traditional approaches while learning robustness to calibration errors. Future work targets loop closures, map-building, and deeper analysis of monocular visual odometry without inertial data.
- Conclusion: VINet performs on par with traditional approaches while requiring less hand-tuning during setup.The paper identifies learned robustness to calibration errors as a key advantage over traditional methods.
- Future Work: Future work will integrate VINet into systems with loop closures and map-building.
- Future Work: Future work will analyze monocular visual odometry and its ability to handle scale without inertial data.