Source-linked AI summary
Selective Sensor Fusion for Neural Visual-Inertial Odometry
Changhao Chen, Stefano Rosa, Yishu Miao, Chris Xiaoxuan Lu, Wei Wu, Andrew Markham, Niki Trigoni
TL;DR
Deep VIO systems need fusion strategies that remain reliable when sensory inputs are missing or corrupted. This paper introduces end-to-end deterministic soft and stochastic hard selective fusion for monocular VIO, and finds improved performance over direct fusion, especially under corruption, while exposing interpretable modality-selection patterns.
Problem
Deep VIO methods rarely model imperfect sensory inputs during fusion, although corruption, missing data, and synchronization problems can undermine estimation.
Method
The framework learns deterministic soft or stochastic hard masks over visual and inertial features, trained end-to-end and compared with direct fusion.
Results
Selective fusion performs better than direct fusion, particularly with corrupted data, while mask visualizations correlate feature selection with sensor conditions and self-motion.
Takeaways & Limitations
Selective fusion provides more robust VIO and interpretable evidence about the relative strengths of visual and inertial modalities under changing conditions.
Takeaways & Limitations
Evaluation uses the KITTI, EuRoC, and PennCOSYVIO datasets because few public datasets provide images, high-frequency IMU data, and full-trajectory ground truth together.
Abstract
from arXiv · showhide
Deep learning approaches for Visual-Inertial Odometry (VIO) have proven successful, but they rarely focus on incorporating robust fusion strategies for dealing with imperfect input sensory data. We propose a novel end-to-end selective sensor fusion framework for monocular VIO, which fuses monocular images and inertial measurements in order to estimate the trajectory whilst improving robustness to real-life issues, such as missing and corrupted data or bad sensor synchronization. In particular, we propose two fusion modalities based on different masking strategies: deterministic soft fusion and stochastic hard fusion, and we compare with previously proposed direct fusion baselines. During testing, the network is able to selectively process the features of the available sensor modalities and produce a trajectory at scale. We present a thorough investigation on the performances on three public autonomous driving, Micro Aerial Vehicle (MAV) and hand-held VIO datasets. The results demonstrate the effectiveness of the fusion strategies, which offer better performances compared to direct fusion, particularly in presence of corrupted data. In addition, we study the interpretability of the fusion networks by visualising the masking layers in different scenarios and with varying data corruption, revealing interesting correlations between the fusion networks and imperfect sensory input data.
1. Introduction
The paper addresses unreliable deep VIO fusion when visual or inertial features are degraded. It proposes selective feature weighting and masking, interprets modality preferences, and studies robustness using corrupted-data benchmarks.
- VIO combines visual and inertial information for robust pose estimation, but real systems face occlusion, low light, inertial noise, drift, synchronization errors, and spatial misalignment.
- Deep VIO methods extract high-level features but commonly feed all features directly into pose-regression modules without explicitly modelling sensor degradation.
- The proposed framework learns feature selection conditioned on measurement reliability, egomotion, and environmental dynamics through deterministic soft fusion and stochastic hard fusion.
- Inertial features contribute more during fast rotation, whereas visual features are preferred during large translations, and selective fusion improves robustness to real-world data corruption.
- Selective fusion masks can be visualized to reveal relative modality strengths and provide insight for further system design.
- The evaluation creates challenging datasets with seven sensor-degradation sources across existing public VIO datasets to study fusion accuracy and robustness.
2. Neural VIO Models with Selective Fusion
The neural VIO architecture encodes monocular images and IMU measurements, selectively fuses their features, models temporal dependencies, and regresses pose transformations. Its fusion component compares direct concatenation with learned soft and hard selection mechanisms.
- The end-to-end architecture processes raw image and IMU sequences through visual and inertial encoders, feature fusion, temporal modelling, and pose regression.
- Feature Encoder: The visual encoder extracts latent representations from two consecutive monocular images using FlowNetSimple to obtain features suited to optical flow.
- Feature Encoder: The inertial encoder applies a two-layer bidirectional LSTM to IMU measurements between image pairs, reflecting the higher temporal frequency of inertial data.
- Fusion Function: The fusion function combines visual features aV and inertial features aI into a representation z for subsequent pose regression.
- Fusion Function: Direct fusion concatenates features uniformly, whereas soft and hard fusion learn feature selection according to environment dynamics and input reliability.
- Temporal Modelling and Pose Regression: A recurrent network models temporal dependencies in the combined features, after which a fully connected regressor maps them to a pose transformation.
3. Selective Sensor Fusion
The paper proposes selective sensor fusion for monocular VIO, explicitly learning which visual and inertial features to retain before pose regression. It compares deterministic soft masks that re-weight features with stochastic hard masks that switch features on or off.
- Monocular vision provides scene appearance and geometry but lacks metric scale, while inertial data offers complementary motion information.
- Direct Fusion: Direct fusion concatenates visual and inertial features, but treating all features as reliable can make incorrect extraction or matching undermine state estimation.
- Soft Fusion (Deterministic): Soft fusion deterministically and differentiably re-weights visual and inertial features using continuous masks conditioned on both modalities.
- Hard Fusion (Stochastic): Hard fusion stochastically generates binary masks that act as switches, propagating or blocking feature components according to learned reliability.
- Hard Fusion (Stochastic): Gumbel-Softmax resampling enables the stochastic hard-fusion layer to be trained end to end despite discrete latent variables.
- Soft fusion is simpler and handles input uncertainty, whereas hard fusion is more difficult to infer but offers an intuitive, interpretable selection mechanism and greater tolerance to imperfect data.
4. Experiments
The experiments evaluate selective fusion against vision-only and direct-fusion baselines across three VIO datasets and multiple sensor degradations. Results show modality-selective fusion improves robustness, with soft fusion favoring angle estimation and hard fusion favoring translation estimation.
- Experimental setup: The study evaluates selective fusion on KITTI, EuRoC, and PennCOSYVIO, using Vision-Only and VIO-Direct as baselines.The experiments include autonomous-driving, MAV, and hand-held scenarios.
- Data corruption: The degradation suite includes occlusion, blur and noise, sensor noise and bias, missing inertial data, spatial misalignment, and temporal misalignment.These perturbations model common visual, inertial, calibration, and synchronization failures.
- Robustness results: Both selective fusion approaches outperform vision-only and direct-fusion baselines under degradation.The comparison reports absolute translational and rotational errors across corruption conditions.
- Robustness results: Soft fusion consistently improves angle estimation, whereas hard fusion consistently improves translation estimation on KITTI and EuRoC.The hand-held setting shows a smaller translation difference, but hard fusion still improves both errors under combined visual and inertial degradation.
- Comparison with classical VIO: Learning-based fusion degrades gracefully under corrupted data, while classical filtering methods can fail abruptly under large noise and misalignment.On KITTI, MSCKF fails with missing images; on EuRoC, classical baselines fail with full sensor degradation caused by temporal misalignment.
5. Related Work
Related work covers classical visual-inertial fusion, learned visual odometry, and attention-based multimodal fusion. The paper distinguishes selective sensor fusion from attention mechanisms by its goal of modality feature selection for sensor fusion.
- Visual-Inertial Odometry: Classical VIO methods use filtering, fixed-lag smoothing, or full smoothing, commonly relying on handcrafted features and explicit visual-inertial error terms.Semi-direct and direct methods reduce reliance on feature extraction pipelines.
- Deep Neural Networks for Localization: Deep localization methods learn pose or motion representations from images and can address conditions difficult to model manually, including missing features and motion blur.Prior work includes CNN-based monocular 6-DoF pose regression and CNN-LSTM combinations.
- Multimodal Sensor Fusion and Attention: The proposed selective fusion differs from sequence-generation attention by selecting multimodal sensor features rather than focusing a decoder on parts of an embedding.Its design intention is multimodal sensor fusion through feature selection.
6. Conclusion
The paper presents end-to-end visual-inertial navigation with deterministic soft fusion and stochastic hard fusion. Experiments interpret how each modality is selected under degradation and self-motion conditions.
- Conclusion: The work proposes deterministic soft fusion and stochastic hard fusion for end-to-end visual-inertial sensor fusion.Soft fusion learns a continuous mask, while hard fusion uses Gumbel-softmax resampling for a stochastic binary mask.
- Conclusion: The experiments investigate how modality influence changes across sensor degradation and self-motion circumstances.The analysis provides interpretations of selective sensor fusion beyond trajectory estimation.
Abstract
The supplementary document adds implementation and data-preparation details, along with further results and discussion of predicted global trajectories.
- Supplementary material: The supplementary document provides detailed network architecture and data preparation and training details.It also includes additional results and discussion of predicted global trajectories.
1. Network Architecture
The architecture uses pretrained visual and bidirectional LSTM inertial encoders, followed by a recurrent pose regressor.
- The experiments use convolutional, fully connected, activation, concatenation, and element-wise multiplication components.Table 1 also defines B as batch size.
- The visual encoder follows FlowNetS and is initialized with FlyingChairs-pretrained weights.Training the visual encoder from scratch experimentally produced worse results.
- Bidirectional LSTMs encode inertial data and process the recurrent part of the pose regressor.The LSTM layers apply dropout regularization on recurrent connections.
2. Dataset Preprocessing Details
The study uses KITTI, EuRoC, and PennCOSYVIO because they provide the imagery, inertial measurements, and trajectory ground truth needed for VIO evaluation, with dataset-specific synchronization and sampling.
- The selected datasets provide camera images, high-frequency IMU data, and ground truth for training and evaluating visual-inertial odometry.Oxford Robotcar lacks inertial data, while TUM VIO lacks ground truth for full trajectories.
- KITTI: KITTI uses manually synchronized raw IMU and image timestamps, with sequences 00, 01, 02, 04, 06, 08, 09 for training and 05, 07, 10 for testing.Sequence 03 is excluded because its corresponding raw file is unavailable; images are resized to 512×256.
- EuRoC: EuRoC uses 20fps grayscale CAM1 images and 200Hz VI-Sensor IMU measurements with tight synchronization.All sequences except MH 04 difficult are used for training, and that sequence is used for testing; images are resized to 512×256.
- PennCOSYVIO: PennCOSYVIO uses 30fps Tango Bottom video and 200Hz VI-Sensor measurements, subsampled to 10Hz images and 100Hz IMU data.Images are cropped and resized to 512×256; sequences as, bf, and bs are used for training, and af for testing.
3. Evaluation of global trajectories on KITTI
On KITTI trajectories, selective fusion reduces error accumulation relative to vanilla VIO, especially under degraded sensing, while vision-only estimation can outperform vanilla fusion when inertial inputs or synchronization are degraded.
- Selective soft and hard fusion outperform vanilla VIO over time on KITTI, with the clearest advantage on challenging Seq 05.The comparison covers global RMSE position errors over Seq 05, Seq 07, and Seq 10 under normal and fully degraded conditions.
- VO heavily underperforms during large angular rotations on Seq 05 and Seq 07.This comparison is reported for global RMSE position error trajectories.
- Under IMU degradation and camera–IMU synchronization errors, VO performs slightly better than vanilla VIO.The result indicates that vanilla fusion can underperform vision-only estimation when sensor inputs are corrupted or misaligned.
- Figure 1 plots global position error in meters against travelled distance in meters on KITTI.The supplementary evaluation shows degraded images, soft and hard masks, and trajectories for four methods on Seq 05.