Source-linked AI summary
Experimental Comparison of Open Source Visual-Inertial-Based State Estimation Algorithms in the Underwater Domain
Bharat Joshi, Sharmin Rahman, Michail Kalaitzakis, Brennan Cain, James Johnson, Marios Xanthidis, Nare Karapetyan, Alan Hernandez, Alberto Quattrini Li, Nikolaos Vitzilaios, Ioannis Rekleitis
TL;DR
Comparisons of visual-inertial estimation have rarely assessed underwater conditions, where visibility, illumination, and image-feature challenges vary. This paper tests ten open-source packages on underwater-robot datasets; across the tested datasets, OKVIS, SVO, ROVIO, and VINS-Mono exhibited the best performance. The results also identify non-deterministic package behavior as a concern for closed-loop control.
Problem
Method
The study compares open-source visual-odometry packages, emphasizing those that use inertial data, on datasets collected with underwater robots.
Source: §VI · pp. 6–7Results
Across the tested datasets, OKVIS, SVO, ROVIO, and VINS-Mono exhibited the best performance, and incorporating IMU measurements improved performance over pure visual odometry.
Source: §VI · pp. 6–7Takeaways & Limitations
The comparison offers insights into package performance limits and qualitatively assesses computational needs for online applications.
Source: §VI · p. 7Package outputs were non-deterministic under the same conditions, sometimes losing tracking or diverging, a concern for closed-loop control.
Source: §VI · p. 7
Abstract
from arXiv · showhide
A plethora of state estimation techniques have appeared in the last decade using visual data, and more recently with added inertial data. Datasets typically used for evaluation include indoor and urban environments, where supporting videos have shown impressive performance. However, such techniques have not been fully evaluated in challenging conditions, such as the marine domain. In this paper, we compare ten recent open-source packages to provide insights on their performance and guidelines on addressing current challenges. Specifically, we selected direct methods and tightly-coupled optimization techniques that fuse camera and Inertial Measurement Unit (IMU) data together. Experiments are conducted by testing all packages on datasets collected over the years with underwater robots in our laboratory. All the datasets are made available online.
I. INTRODUCTION
Visual-inertial state estimation methods have developed alongside wider use of camera and IMU sensors, but comparisons have rarely covered underwater conditions or more than a few packages. This paper evaluates ten open-source packages on underwater-robot datasets to assess their capabilities and identify challenges for robust estimation.
I. INTRODUCTION
- Visual methods include indirect approaches that detect and match features and direct approaches that use pixel intensities to minimize photometric error.The direct approach skips feature detection and matching.Source: §I · p. 1
- Visual state estimation can fuse other sensors: filtering propagates state with IMU measurements and updates with visual features, while tightly coupled optimization jointly minimizes IMU and reprojection errors.The paper examines packages integrating visual and inertial data.Source: §I · p. 1
- Comparisons have typically covered only a few packages, rarely focused on a specific domain, and used higher-quality visual data than underwater settings provide.An earlier comparison found that most packages had higher errors on marine-domain datasets.Source: §I · p. 1
- Ten recent open-source packages integrating visual and inertial data are evaluated on new, publicly available datasets collected with underwater robots.The paper extends an earlier comparison and aims to identify methods’ strengths, weaknesses, and failure points.Source: §I · pp. 1–2
II. RELATED WORK AND METHODS EVALUATED
The evaluated methods span camera configurations, IMU use, direct versus indirect processing, sensor coupling, and loop closure. The comparison covers varied underwater datasets and a selection of methods that is broad across these dimensions but not exhaustive of the literature.
Method taxonomy
- The methods are classified by camera count, IMU use, direct versus indirect processing, sensor coupling, and loop-closing capability.The paper describes its selected algorithms as covering these dimensions, while not claiming an exhaustive literature survey.Source: §II · p. 2
Direct methods
- LSD-SLAM tracks and maps using monocular image intensities, while DSO samples high-gradient pixels and minimizes photometric error over a sliding window.LSD-SLAM allows dense 3D reconstruction; DSO was tested on the TUM monoVO dataset.Source: §II · p. 2
Visual-inertial and feature-based methods
- SVO 2.0 combines direct tracking and triangulation of high-gradient pixels with feature-based joint structure-and-motion optimization, using an IMU prior for image alignment.It can be generalized to multi-camera systems and was tested in a lab setting and on EuRoC and ICL-NUIM.Source: §II · p. 2
- ORB-SLAM2 uses ORB features for tracking, mapping, relocalization, and loop closing, while its IMU-extended system was not available open source.The described system is monocular or stereo and was tested on datasets including KITTI and EuRoC.Source: §II · p. 2
Filtering methods
- REBiVO targets Micro Aerial Vehicles, fusing monocular-camera and IMU data while using an EKF to estimate depth from tracked edges.Source: §II · p. 2
- Monocular and stereo MSCKF implementations use different camera setups; the stereo version is described as comparably costly to monocular solutions with increased robustness.The monocular implementation was tested on EuRoC, while the stereo version was evaluated on EuRoC and a custom UAV dataset.Source: §II · p. 2
- ROVIO tightly fuses image-patch photometric residuals with IMU data using an iterated extended Kalman filter.Source: §II · p. 2
Optimization methods
- OKVIS and VINS-Mono use tightly coupled sliding-window optimization; OKVIS selects keyframes by spacing and marginalizes states outside its window, while VINS-Mono also supports loop detection and relocalization.OKVIS uses one or more cameras, and VINS-Mono uses a monocular camera.Source: §II · p. 2
A. Setup
Packages were run on a common computer platform, with parameters tuned per sequence from defaults. Evaluation considered supported camera and IMU combinations, and reported the best of multiple runs.
Computing platform
- Experiments used an Intel i7-7700 CPU at 3.60 GHz, 32 GB RAM, Ubuntu 16.04, and ROS Kinetic.Source: A. Setup · p. 3
Evaluation procedure
- Each package’s parameters were initialized to defaults and manually tuned per sequence, following available author suggestions; the best result across multiple runs was reported.Source: A. Setup · p. 4
- Supported modes were evaluated across monocular, monocular-plus-IMU, stereo, and stereo-plus-IMU combinations.For datasets with ground truth, trajectories were aligned using sim(3), and translation RMSE was computed.Source: A. Setup · p. 4
B. Evaluation
Across six challenging underwater datasets, package performance ranged from complete tracking to failure, with results varying by sensor configuration, initialization, visual conditions, and loop closure. The evaluation reports both aligned trajectory error and the fraction of each trajectory tracked.
B. Evaluation
- The first four datasets had ground-truth measurements; DPV and Aqua2Reef lacked manual ground truth, and sonar-based reference trajectories were validated with manual measurements.The reported tracking percentage indicates how much of the trajectory was completed before a package failed.Source: B. Evaluation · p. 4
- The qualitative ratings classify green as at least 90% tracked with a fairly accurate trajectory, yellow as mostly localized or complete but substantially deviating, orange as under 50%, and red as failure or under 10%.The table also reports aligned RMSE and tracked-time percentage for each dataset.Source: B. Evaluation · p. 4, Table II · p. 5
- LSD-SLAM, REBiVO, and monocular SVO produced no consistent results and were excluded from the performance table.Source: B. Evaluation · p. 4
- DSO performed well on high-gradient areas and large rotations without full photometric calibration, but tracked low-gradient motion for only seconds and required more computation and memory.Scale changes occurred in low-gradient images; full calibration is needed for best performance.Source: B. Evaluation · p. 4
- SVO 2.0 tracked long trajectories and sparse-feature regions; stereo with IMU generally maintained tracking and produced a good trajectory, unlike stereo without IMU, which had rotation errors.Low-gradient images caused depth-scale changes, especially in monocular mode, where tracking failed.Source: B. Evaluation · p. 4
- OKVIS tracked all datasets in monocular and stereo modes, while stereo ORB-SLAM2 also tracked all datasets with loop closure; ROVIO, VINS-Mono, and Stereo-MSCKF had dataset-specific exceptions.The overall comparison also reports that stereo SVO tracked for most of the time but deviated from ground truth, while Stereo-MSCKF was fairly accurate when it maintained tracking.Source: B. Evaluation · p. 4, Table II · p. 5
- Monocular ORB-SLAM2 failed to initialize on both sunken-bus datasets because haze and low contrast prevented reliable model selection, but loop closure enabled relocalization after track loss.It tracked the full Cave and Aqua2Reef trajectories in monocular mode.Source: B. Evaluation · p. 5
- Mono-MSCKF worked when stationary for IMU initialization; ROVIO was robust except on Aqua2Reef, OKVIS handled haze and low illumination, VINS-Mono loop closure reduced drift, and Stereo-MSCKF struggled with fast starts and DPV.Source: B. Evaluation · pp. 5–6
V. DISCUSSION
Underwater challenges expose limits in feature-dependent tracking, while the experiments show advantages for stereo, loop closure, and inertial data. Reported performance also varies across repeated runs, and monocular scale remains a practical concern.
V. DISCUSSION
- Low visibility, particulates, blur, illumination changes, and sparse features challenge underwater localization; changing image appearance also affects feature tracking.Color loss with depth can reduce sharpness; investigating color restoration's effect on state estimation remains future work.Source: §V · p. 6
- Stereo outperformed monocular, loop closure extended tracking, and inertial data improved scale estimates; direct methods often lost trajectories but reconstructed tracked portions well.DSO had the best reconstructions when it could track the trajectory.Source: §V · p. 6
- Packages produced inconsistent results under identical conditions, partly because frame-rate-driven RANSAC iteration limits can yield widely varying estimates.The paper reports each package's best result across multiple trials, so small variations are not significant.Source: §V · p. 6
- Offline trajectory transformations and scaling can conceal inaccurate scale; VINS-Mono's scale was always inaccurate, which matters for distance-based AUV missions.The paper notes that incorrect scale can cause failure when following a transect or performing a grid search.Source: §V · p. 6
- OKVIS and ROVIO performed well; stereo ORB-SLAM with loop closure and VINS-Mono up to scale also performed well.DSO reconstructed best when it tracked the trajectory.Source: §V · p. 6
VI. CONCLUSION
The paper compares open-source visual odometry packages, emphasizing those using inertial data, across underwater environments. It reports improved performance with IMU measurements and identifies four packages with the best performance on the tested datasets, while noting non-determinism as a concern for closed-loop use.
VI. CONCLUSION
- Across a wide range of underwater environments, incorporating IMU measurements improved performance compared with pure visual odometry packages.The environments included man-made underwater structures, caverns, and coral reefs.Source: §VI · pp. 6–7
- For the tested datasets, OKVIS, SVO, ROVIO, and VINS-Mono exhibited the best performance.The comparison used quantitative and qualitative criteria and qualitatively assessed computational needs for online applications.Source: §VI · p. 7
- Non-deterministic package outputs are a concern for closed-loop robot control because tracking loss or estimator divergence can leave an autonomous vehicle unable to complete its mission.Most packages sometimes produced different outputs for the same input under the same conditions.Source: §VI · p. 7
- Future work will test additional packages and robotic platforms, including BlueROV2, and collect more challenging datasets with more test cases per package.Source: §VI · p. 7