Source-linked AI summary
LIC-Fusion: LiDAR-Inertial-Camera Odometry
Xingxing Zuo, Patrick Geneva, Woosik Lee, Yong Liu, Guoquan Huang
TL;DR
LIC-Fusion addresses multimodal odometry by tightly fusing asynchronous IMU, sparse visual, and LiDAR measurements while calibrating sensor spatial and temporal relationships online. Implemented within MSCKF, it combines tracked LiDAR edge and planar features with visual observations and IMU readings; experiments report improved accuracy and robustness across indoor, outdoor, low-light, and aggressive-motion conditions.
Problem
Camera-based odometry is limited by lighting, whereas LiDAR has sparsity, cost, and lower collection-rate limitations, motivating complementary multimodal sensing.
Method
LIC-Fusion tightly couples IMU, sparse visual features, and tracked LiDAR edge and planar features in an MSCKF framework with online spatial and temporal calibration.
Results
LIC-Fusion outperforms state-of-the-art visual-inertial and LiDAR odometry methods in estimation accuracy and robustness to aggressive motions across indoor and outdoor experiments.
Takeaways & Limitations
The multimodal fusion provides accurate and robust 6DOF motion tracking across different environments and under aggressive motions.
Abstract
from arXiv · showhide
This paper presents a tightly-coupled multi-sensor fusion algorithm termed LiDAR-inertial-camera fusion (LIC-Fusion), which efficiently fuses IMU measurements, sparse visual features, and extracted LiDAR points. In particular, the proposed LIC-Fusion performs online spatial and temporal sensor calibration between all three asynchronous sensors, in order to compensate for possible calibration variations. The key contribution is the optimal (up to linearization errors) multi-modal sensor fusion of detected and tracked sparse edge/surf feature points from LiDAR scans within an efficient MSCKF-based framework, alongside sparse visual feature observations and IMU readings. We perform extensive experiments in both indoor and outdoor environments, showing that the proposed LIC-Fusion outperforms the state-of-the-art visual-inertial odometry (VIO) and LiDAR odometry methods in terms of estimation accuracy and robustness to aggressive motions.
I. INTRODUCTION AND RELATED WORK
LIC-Fusion addresses limitations of individual camera and LiDAR sensing by tightly coupling IMU, sparse visual, and sparse LiDAR measurements with online calibration in an MSCKF framework.
- Cameras are limited by lighting, while LiDAR provides robust range measurements but suffers from sparsity, cost, and lower collection rates.
- Prior camera–LiDAR fusion approaches include augmented LiDAR depth, sequential visual–LiDAR odometry, and loosely coupled scan-matching refinement.
- Loosely coupled fusion passes only VIO pose estimates to LiDAR scan matching, preventing direct processing of raw visual-inertial measurements and losing correlation information.
- LIC-Fusion is a fast, tightly coupled, single-thread LIC odometry algorithm with online spatial and temporal calibration in the computationally efficient MSCKF framework.
- The method combines IMU measurements, sparse visual features, and sparse edge and planar LiDAR features for efficient 6DOF pose estimation.
- Experiments in indoor and outdoor environments report greater accuracy and robustness than state-of-the-art visual-inertial and LiDAR odometry methods.
A. State Vector
The LIC-Fusion state contains the IMU state, sensor extrinsics, time offsets, and sliding-window clones at camera and LiDAR measurement times.
- The state vector includes the IMU state, IMU–camera and IMU–LiDAR extrinsics, and clones from past image and LiDAR scan times.
- The IMU state represents rotation, velocity, position, and gyroscope and accelerometer biases in the global and local frames.
- Camera and LiDAR calibration states represent rigid-body transformations between each exteroceptive sensor and the IMU frame.
- The method co-estimates camera and LiDAR time offsets relative to the IMU clock to correct latency, clock skew, and transmission delays.
- The generalized update operation relates the true state, estimated state, and error state, using Euclidean addition for vector components and quaternion composition for rotations.
B. IMU Propagation
LIC-Fusion models IMU angular-rate and acceleration measurements through continuous-time kinematics, with stochastic bias evolution and covariance propagation.
- The IMU propagation model uses angular rate and linear acceleration measurements in continuous-time kinematics.
- Gyroscope and accelerometer biases follow random walks driven by white Gaussian noise, while gravity is represented in the global frame.
- The continuous-time system is integrated and linearized to propagate the state covariance forward in time.
C. State Augmentation
When new camera or LiDAR data arrive, LIC-Fusion propagates the IMU state to the estimated measurement time, clones it, and augments the covariance with calibration dependencies.
- Each new image or LiDAR scan triggers IMU propagation to that sensor’s time, followed by cloning into the corresponding camera or LiDAR state window.
- For a LiDAR scan, the estimated LiDAR time offset determines the IMU propagation endpoint before adding the new LiDAR clone.
- Covariance augmentation uses Jacobians with respect to the current IMU state, calibration parameters, and existing clone states.
- Modeling the LiDAR clone’s dependence on the LiDAR time offset allows measurement models to use clones at the true IMU-clock measurement time.
1) LiDAR Feature Measurement:
LIC-Fusion extracts and tracks sparse LiDAR edge and planar-surf features, converting their geometric relationships into covariance-aware EKF measurements with outlier rejection.
- Feature extraction and tracking: High- and low-curvature LiDAR scan sections provide sparse edge and planar-surf features for estimation.Features are tracked between scans using projection and KD-tree nearest-neighbor indexing.
- Edge features: Edge-feature residuals measure the distance from a projected point to a line formed by two corresponding points in the previous scan.The second edge point is selected from an immediately neighboring scan ring.
- Measurement model: LiDAR distance measurements are linearized at the current estimate, with Jacobians involving cloned poses and IMU–LiDAR calibration.The residual noise is modeled as white Gaussian with variance Cr.
- Robust update: Covariances are propagated from raw LiDAR point uncertainties, and Mahalanobis-distance tests reject measurements that fail a chi-squared consistency test.The resulting accepted residuals are used in the EKF update.
- Planar-surf features: Planar-surf residuals measure the distance from a projected point to a plane formed by three corresponding points assumed to lie on the same physical plane.Their covariance propagation, linearization, and Mahalanobis testing follow the edge-feature procedure.
2) Visual Feature Measurement:
Visual measurements use tracked sparse image features, triangulate them when appropriate, and apply MSCKF nullspace projection before updating the state while retaining online camera–IMU calibration.
- Feature processing: FAST image features are tracked across frames with KLT optical flow and triangulated after being lost or observed through the sliding window.The current camera clones provide the estimates used for triangulation.
- Reprojection update: The visual update uses standard reprojection error for bearing measurements of triangulated 3D features.Measurement and feature Jacobians are evaluated at the current best estimates.
- MSCKF formulation: MSCKF nullspace projection removes dependence on the triangulated 3D feature from the visual measurement model.This produces the projected residual used for the state update.
- Online calibration: A non-zero Jacobian with respect to the IMU–camera rigid transformation allows that extrinsic calibration to be estimated online.
E. Measurement Compression
Measurement compression reduces the cost of jointly processing LiDAR and visual residuals by applying thin QR through Givens rotations before a standard EKF update.
- Motivation: The stacked residuals and Jacobians from LiDAR and visual features create a large measurement-update cost.
- Compression: Assuming independent measurements, Givens rotations perform thin QR to compress the stacked measurement system.The associated orthogonal matrices preserve the information needed for the update.
- Compressed system: The compressed Jacobian is square with dimension equal to the state vector, and the noise is transformed consistently.
- EKF update: The compressed residual is used to update the state estimate and covariance with the standard EKF.
III. EXPERIMENTAL RESULTS
Experiments evaluate LIC-Fusion on a calibrated LiDAR–IMU–camera rig in indoor and outdoor settings against visual-inertial and LiDAR odometry baselines. Outdoor tests use RTK GPS ground truth and report lower trajectory error for LIC-Fusion.
- Experimental setup: The experimental rig combines an Xsens MTi-300 AHRS IMU, Velodyne VLP-16 LiDAR, and monochrome global-shutter Blackfly camera.Sensor extrinsics are calibrated offline and refined during online estimation.
- Outdoor tests: Outdoor evaluation uses an approximately 800-meter, four-minute Ackermann-robot sequence with centimeter-level RTK GPS ground truth.
- Outdoor results: 2.5 meters lower average error than standard MSCKF and 5 meters lower than LOAM were reported for LIC-Fusion.These comparisons use average mean squared trajectory error after best-fit alignment to RTK ground truth.
- Evaluation protocol: Each algorithm was run six times to account for randomness from RANSAC and represent typical performance.
B. Indoor Tests
Indoor tests evaluate LIC-Fusion across normal to low-light conditions and motion profiles ranging from slow to aggressive. The system localizes accurately, including under high motion and low light, while the compared methods fail with large errors in the most aggressive sequence.
- Dataset and conditions: Indoor sequences span normal to low-light lighting conditions and slow to aggressive motion profiles.The sensor rig was held by hand at chest height during collection.
- Evaluation protocol: Indoor evaluation uses average trajectory start-end error because groundtruth was unavailable.The sensor platform was returned to its initial location for this evaluation.
- Indoor performance: LIC-Fusion localizes with high accuracy and handles extreme high-motion and low-light cases through fusion of three sensing modalities.Indoor trajectories are shown for sequences A, B, C, and D.
- Indoor performance: In the high-dynamic Indoor-C sequence, LIC-Fusion localizes while the two compared methods fail with large errors.The sequence contains high angular velocities and high linear accelerations produced by strongly shaking the sensor rig.
- Conclusion: LIC-Fusion provides accurate and robust 6DOF 3D motion tracking across different environments and aggressive motions.The conclusion attributes this capability to exploiting different sensing modalities.