Source-linked AI summary
MFVINS: Multiple Fisheye Camera-Based Visual Inertial System
Eunseong Jang, YuJin Chung, Sang Jun Lee, Jihyun Yoon, HyungGi Jo
TL;DR
Monocular VINS can accumulate errors in occluded, variably lit, or textureless scenes, while limited field of view makes dynamic occlusion especially challenging. MFVINS combines multiple fisheye cameras, an IMU, and learning-based depth estimation; the authors report improved accuracy and robustness against existing VINS frameworks with real-time operation.
Problem
Monocular VINS can accumulate errors under occlusion, changing illumination, and textureless conditions, and limited field of view can cause rapid degradation when dynamic objects block the view.
Source: §1 · p. 2Method
MFVINS tightly combines multiple fisheye cameras and a low-cost IMU, using IMU-aided FAST tracking, distortion-aware outlier filtering, and learning-based depth estimation.
Source: §1 · pp. 2–3Results
MFVINS outperforms existing monocular and multi-camera VINS frameworks in accuracy and robustness on real-world sequences and is implemented in real time without excessive computational overhead.
Source: §1 · p. 3, §6 · p. 25Takeaways & Limitations
The reported results show that multiple fisheye cameras can be used in a real-time VINS that is robust to occlusion and textureless regions.
Source: §1 · p. 3The scalability of cited omnidirectional-camera systems to large-scale scenarios remains unverified.
Source: §2.2 · p. 5
Abstract
from arXiv · showhide
A simultaneous localization and mapping (SLAM) method using a monocular camera and a low-cost inertial measurement unit (IMU) sensor is an effective way to fulfill a low-cost sensor configuration. Using this sensor configuration, visual-inertial system (VINS) focuses on fusing data from a camera and an IMU sensor to estimate the six degrees-of-freedom (DOF) of the sensor pose. Typically, VINS uses only a single camera as visual input, which lead to problems such as error accumulation due to occlusion, various illumination, and textureless environments. In this paper, we propose a new multiple fisheye camera-based visual-inertial system called MFVINS. We present an IMU-aided FAST feature tracker for multiple cameras that enables efficient extraction and robust matching of local features. Then, the proposed method filters out outliers caused by fisheye distortion on the normalized image plane. Subsequently, a new reprojection error with physical validity constraints is proposed for bundle adjustment using learning-based depth estimation. The proposed method is applied to various scenarios, and its effectiveness is demonstrated by comparing previous VINS methods. In particular, MFVINS is implemented in real-time process to leverage the advantages of using multiple cameras -- robustness against occlusion and textureless regions -- while reducing the computational burden.
1 Introduction
MFVINS combines multiple fisheye cameras with a low-cost IMU to address monocular VINS weaknesses in occluded, dynamically lit, and textureless environments. Its feature-tracking and depth-estimation methods support improved accuracy and robustness in real-world tests, with real-time operation reported.
1 Introduction
- MFVINS tightly integrates multiple fisheye cameras with a low-cost IMU and uses learning-based depth estimation on each fisheye image for 3D reconstruction in visually degraded scenes.Source: §1 · p. 2
- The frontend uses grid-based FAST extraction, IMU-aided local feature tracking, and normalized-plane filtering to reject mismatches caused by heavy fisheye distortion.IMU data supplies initial feature locations to improve tracking efficiency; learned depth estimates support subsequent localization.Source: §1 · pp. 2–3
- The system uses loop closure detection and pose graph optimization in its backend to reduce long-term drift.Source: §1 · p. 3
- MFVINS outperforms existing monocular and multi-camera VINS frameworks in accuracy and robustness on real-world sequences, while running in real time without excessive computational overhead.The evaluated sequences include occlusions and low-texture environments.Source: §1 · p. 3
- The paper reports extensive real-world experiments and ablation studies to assess the proposed modules' effects on pose-estimation accuracy.Source: §1 · p. 3
2 Related Works
Related work spans monocular, stereo, RGB-D, and multi-camera visual-inertial systems, with different sensor configurations and approaches to feature extraction and depth. The authors identify limits in existing multi-camera methods, including unverified large-scale scalability and insufficient use of inter-camera geometry.
2.1 Mono/Stereo/RGB-D camera system
- Monocular VINS couples a camera and IMU, while stereo and RGB-D approaches add parallax or direct or learned depth information for localization.The section reviews VINS-Mono and ORB-SLAM3, stereo MSCKF-VIO, and RGB-D or learned-depth systems.Source: §2.1 · pp. 3–4
2.2 Multiple camera system
- Reviewed multi-camera systems use fisheye cameras, stereo-fisheye combinations, or front-stereo and side-monocular configurations to support visual-inertial estimation.Examples include BAMF-SLAM, ROVINS, OmniNxt, and MAVIS.Source: §2.2 · pp. 4–5
- The authors note that cited omnidirectional systems were evaluated in relatively small environments, leaving their scalability to large-scale scenarios unverified.Source: §2.2 · p. 5
- MCVIO processes each camera's features independently, limiting use of inter-camera geometry and feature-depth estimation; its presented code and experiments also lack loop closure and pose graph optimization.Source: §2.2 · p. 5
3 Overview
MFVINS combines multi-fisheye visual-inertial processing with depth estimation, initialization, and pose-graph optimization. The overview also establishes coordinate frames and projection functions for describing the system.
3.1 Proposed Frameworks
- The framework comprises measurement preprocessing, fisheye depth estimation, initialization, and multi-camera VIO with pose-graph optimization.These four components are identified as the system’s primary components.Source: §3.1 · pp. 5–6
3.1.1 Measurement Preprocessing
- IMU angular velocity supports feature tracking, while a predicted rotation matrix improves tracking efficiency and fisheye-distortion outliers are filtered before downstream processing.Camera–IMU timing offset is estimated and compensated through calibration.Source: §3.1.1 · p. 6
3.1.2 Fisheye Depth Estimation
- A deep-learning depth network produces pixel-wise depth maps or inverse-depth values, which are associated with extracted features as geometric constraints for localization.Source: §3.1.2 · p. 6
3.1.3 Initialization
- Structure-from-motion using primarily the front-facing camera establishes the initial position and orientation and estimates parameters for VIO initialization.Source: §3.1.3 · p. 6
3.1.4 Multi-Cam VIO and Pose Graph Optimization (PGO)
- Multi-camera VIO uses sliding-window optimization, then applies loop closure and pose-graph optimization using the front camera to mitigate accumulated drift and improve long-term stability.Source: §3.1.4 · pp. 6–7
3.2 Notations and frame definitions
- The notation defines world, camera, IMU-body, and LiDAR frames, and uses fisheye-model and unit-sphere projection functions to map world points to the image plane.The coordinate-frame transformations are represented by 4×4 matrices in SE(3).Source: §3.2 · p. 7
4 Multiple Fisheye VINS(MFVINS)
MFVINS combines multi-fisheye feature extraction and IMU-aided tracking with distortion-aware filtering, learning-based depth initialization, and pose-graph drift correction. Its components address the feature, depth, and long-term drift challenges of multi-camera VINS.
4.1.1 Visual Feature Extraction
- Grid-based FAST extraction distributes features across each fisheye image, while parallel processing supports efficient multi-camera operation.Features are extracted uniformly within N×M grid cells, with the process applied independently to each camera.Source: §4.1.1 · p. 9
4.1.2 IMU-aided Tracking
- IMU-aided tracking predicts feature locations by rotating back-projected feature directions, then uses those locations to initialize LK optical flow.The method estimates and compensates camera–IMU timing offsets for both current and previous frames.Source: §4.1.2 · pp. 10–11
4.1.3 Outlier Filtering for Fisheye-Induced Distortion
- The filter removes horizontally diverging peripheral features while preserving the valid central region of the normalized plane.It uses the feature’s distance from the normalized-plane center and an empirically set threshold of τ = 6.Source: §4.1.3 · p. 12, Figure 5 · p. 14
4.1.4 Learning-based Depth Estimation and Multi-Camera Visual Inertial Odometry
- Learning-based inverse depth initializes feature depth only when it satisfies physical validity constraints; otherwise, conventional triangulation is used.Depth is then refined by minimizing reprojection error, limiting optimization contamination from large prediction errors while using stable predictions to address fisheye depth uncertainty.Source: §4.1.4 · pp. 14–15
4.2 Pose Graph Optimization for Drift Correction
- Loop closure uses the front-facing camera, and pose-graph optimization aligns revisited locations to reduce long-term drift while preserving local odometry accuracy.Restricting loop-closure detection and optimization to the front camera also reduces computational complexity.Source: §4.2 · p. 15
5 Experiments
The experiments evaluate MFVINS in underground-parking-lot driving, persistent-occlusion, and rotational scenarios using a two-wheeled UGV with four fisheye cameras and an IMU. Across the reported comparisons, MFVINS achieves the highest localization accuracy in driving and persistent occlusion, while component studies examine feature detection, tracking, and distortion filtering.
5.1 Evaluation Metrics
- Localization accuracy is evaluated with RMSE as ATE against an aligned LiDAR-based 6-DoF reference; ablations also measure feature detection time, untracked features, and track duration.Untracked features are counted over 100 seconds; track duration is the average number of frames per successfully tracked feature.Source: §5.1 · p. 19
5.2 Experiments on DRIVING Sequences; 5.3 Experiments on OCCLUSION Sequences
- MFVINS achieves the most accurate localization in driving sequences and the highest localization accuracy under persistent occlusion; omnidirectional configurations outperform front-and-rear-only setups.Table 2 reports localization results for both scenarios.Source: Table 2 · p. 21, §5.2 · p. 21, §5.3 · p. 21
5.2 Experiments on DRIVING Sequences
- In driving sequences, the omnidirectional camera configuration performs better than front-and-rear-only configurations, and MFVINS(D) denotes runs using learning-based depth estimation.The driving comparison includes five sequences and tests the effect of learning-based depth estimation.Source: §5.2 · pp. 19, 21
5.3 Experiments on OCCLUSION Sequences
- Under persistent occlusion, omnidirectional cameras are more robust than front-and-rear-only cameras, and the proposed method retains the highest localization accuracy.Depth estimation was not used in these occlusion experiments.Source: §5.3 · p. 21, Table 2 · p. 21
5.4 Ablation Study
- The ablation study tests feature extraction, IMU-aided tracking, and distortion-outlier filtering with omnidirectional cameras, measuring component effects before pose-graph optimization.The depth-estimation effect is covered in the driving experiments rather than this ablation.Source: §5.4 · p. 22
5.4.1 Feature Detection Algorithm
- FAST reduces computation and yields the fewest untracked features, while VPI-Harris is faster but its overlapping detections increase untracked features and degrade omnidirectional MCVIO localization.MFVINS uses CPU processing, with per-camera running time similar to GPU-based detection.Source: §5.4.1 · pp. 22–23, Table 3 · p. 23
5.4.2 IMU-aided Feature Tracker
- On triangular, square, and infinity-shaped rotational paths, IMU-aided tracking improves localization accuracy and increases the average frames per tracked feature.These paths were selected because rotation is less pronounced in the driving and occlusion sequences.Source: §5.4.2 · p. 24, Table 4 · p. 24
5.4.3 Filtering Method for Distortion-included Feature Outlier
- Experiments on driving and occlusion sequences find that distortion-caused outliers adversely affect localization accuracy, supporting the use of feature filtering.Source: Table 5 · p. 24, §5.4.3 · p. 25
6 Conclusion
MFVINS combines multiple fisheye cameras with IMU-aided FAST feature extraction, distortion-based outlier filtering, learning-based depth estimation, and pose-graph optimization. Real-world experiments report better accuracy and robustness than existing monocular and multi-camera algorithms; future work targets sensor configuration and real-time processing.
6 Conclusion
- MFVINS combines multiple fisheye cameras with grid-based FAST extraction aided by an IMU and distance-based filtering of fisheye-distortion outliers.Source: §6 · p. 25
- Learning-based depth estimation is used to enhance depth accuracy, and pose-graph optimization is used to reduce accumulated errors.Source: §6 · p. 25
- Real-world dataset experiments report better accuracy and robustness than existing monocular and multi-camera algorithms; future work will optimize sensor configurations and real-time processing.Source: §6 · p. 25