Source-linked AI summary
Online Temporal Calibration for Monocular Visual-Inertial Systems
Tong Qin, Shaojie Shen
TL;DR
Temporal offsets caused by sensor delays and unsynchronized clocks impair visual-inertial fusion. The paper introduces online calibration that jointly optimizes the offset with states and features, achieving accurate calibration and motion estimation while improving visual-inertial performance.
Problem
Triggering and transmission delays and unsynchronized clocks create unknown temporal offsets between camera and IMU measurements, affecting visual-inertial robustness and accuracy.
Method
The method jointly optimizes a constant unknown time offset with camera and IMU states and feature locations in a SLAM-based visual-inertial framework.
Results
The proposed method achieves high accuracy in temporal-offset calibration and motion estimation, with calibration results close to Kalibr and lower standard deviations on four of five reported cases.
Takeaways & Limitations
Online temporal calibration significantly benefits visual-inertial systems and can be adopted in optimization-based visual-inertial frameworks.
Abstract
from arXiv · showhide
Accurate state estimation is a fundamental module for various intelligent applications, such as robot navigation, autonomous driving, virtual and augmented reality. Visual and inertial fusion is a popular technology for 6-DOF state estimation in recent years. Time instants at which different sensors' measurements are recorded are of crucial importance to the system's robustness and accuracy. In practice, timestamps of each sensor typically suffer from triggering and transmission delays, leading to temporal misalignment (time offsets) among different sensors. Such temporal offset dramatically influences the performance of sensor fusion. To this end, we propose an online approach for calibrating temporal offset between visual and inertial measurements. Our approach achieves temporal offset calibration by jointly optimizing time offset, camera and IMU states, as well as feature locations in a SLAM system. Furthermore, the approach is a general model, which can be easily employed in several feature-based optimization frameworks. Simulation and experimental results demonstrate the high accuracy of our calibration approach even compared with other state-of-art offline tools. The VIO comparison against other methods proves that the online temporal calibration significantly benefits visual-inertial systems. The source code of temporal calibration is integrated into our public project, VINS-Mono.
I. INTRODUCTION
Temporal misalignment between camera and IMU measurements undermines visual-inertial state estimation, especially on low-cost systems without hardware synchronization. The paper proposes online calibration integrated with feature-based visual-inertial optimization and evaluates its benefits in simulation and real-world experiments.
- Accurate state estimation supports robot navigation, autonomous driving, virtual reality, and augmented reality.
- Triggering and transmission delays, unsynchronized clocks, and imprecise timestamps create temporal offsets that affect visual-inertial robustness and accuracy.
- The proposed method jointly calibrates the camera–IMU temporal offset with camera and IMU states and feature locations in a SLAM system.
- The calibration approach is presented as a general factor for feature-based visual-inertial optimization frameworks and is applicable beyond the showcased monocular setup.
- Simulation and real-world experiments showcase the significance of online temporal calibration for visual-inertial systems.
- Prior work includes filter-based and batch-optimization visual-inertial methods, while related temporal calibration approaches use initialization, curve alignment, or continuous batch optimization.
III. ALGORITHM
The algorithm models the camera–IMU time offset as an unknown temporal variable and estimates it online within an optimization-based visual-inertial framework. The formulation addresses unsynchronized low-cost sensors while representing camera pose and feature geometry in the optimization.
- The method models temporal offset in a vision factor and estimates it online together with features, IMU states, and camera states.
- The camera pose transformation maps a 3D feature from the camera frame to the global frame.
- Low-cost and self-assembled sensor sets may have timestamp errors from triggering delay, transmission delay, and unsynchronized clocks.
- The paper assumes the camera–IMU time offset td is constant but unknown in the general case.
- The time offset td specifies how far camera timestamps should be shifted to make camera and IMU streams temporally consistent, with either positive or negative values.
B. Feature Velocity on Image Plane
The method represents temporal misalignment through feature motion on the image plane rather than shifting entire sensor sequences. It approximates feature velocity from consecutive frames under a short-interval constant-speed assumption.
- The method shifts feature observations in the timeline instead of shifting the entire camera or IMU sequence.
- Over several milliseconds, camera motion is treated as constant-speed motion, so image-plane feature motion is approximated by constant velocity.
- Figure 2 depicts two consecutive image frames and assumes camera motion from Ck to Ck+1 has constant speed during [tk, tk+1].
- The feature velocity is computed from the feature’s 2D observations on the consecutive image planes.
C. Vision Factor with Time Offset
The visual factor augments classical reprojection with an unknown time offset that shifts feature observations, aligning visual and IMU constraints during optimization.
- Classical reprojection is refactored by adding time offset as a new variable.
- The feature is parameterized by its 3D global position, while camera pose transforms it into the local camera frame before projection.
- Time misalignment makes IMU and vision constraints inconsistent, so feature observations are shifted forward or backward in the timeline.
- Optimizing td finds the camera pose and feature observation in the time domain that match the IMU constraint.
2) Depth Parameterization:
With depth parameterization, reprojection uses a feature depth in one image and incorporates time offset into the observation used for the target-image residual.
- The feature is parameterized by depth λi in image i for reprojection to image j.
- Traditional reprojection projects the feature into the global frame and back onto the local camera frame j.
- The reprojection residual is the displacement between the observation and its back-projected location.
- The depth-based formulation similarly incorporates the time offset variable td.
D. Optimization with Time Offset
The optimization framework augments visual-inertial SLAM states with time offset and jointly minimizes IMU, visual, and prior factors over a bounded local bundle.
- Local bundle adjustment jointly optimizes camera and IMU states together with feature locations.
- The whole state vector is augmented with time offset td alongside IMU states and feature parameters.
- The cost function contains IMU propagation, proposed visual, and prior factors.
- A local bundle retains several camera frames and IMU measurements, with bundle size limited to reduce computational complexity.
- The nonlinear least-squares objective is efficiently optimized using Gauss-Newton methods after marginalizing measurements and states outside the bundle into a prior.
E. Compensation of Time Offset
After each optimization, the system compensates the estimated offset by shifting subsequent visual timestamps and iteratively optimizes the remaining offset.
- After optimization, compensated visual timestamps are shifted as t′cam = tcam + td.
- The system then estimates δtd between compensated visual and inertial measurements in subsequent data streams.
- As δtd decreases, the constant-speed feature-motion assumption over short image-plane intervals becomes more reasonable.
- The iterative process is described as converging to zero even from an initial offset of hundreds of milliseconds.
IV. EXPERIMENT RESULTS
Experiments evaluate temporal calibration and overall VIO performance using simulation, a real sensor, a public dataset, and a real-world setup. The system extends a visual-inertial optimization framework with time offset while retaining real-time operation.
- Evaluation design: The evaluation compares the proposed algorithm against other popular methods across calibration and overall VIO experiments.Calibration uses simulated data and a real sensor; VIO performance uses a public dataset and a real-world experiment.
- Implementation: The implementation augments a visual-inertial optimization framework with time offset and a proposed vision factor.Features are detected and tracked visually, while IMU measurements are locally integrated.
- Simulation evaluation: Simulation calibration results report RMSE and NEES, with RMSE defined as root mean square error and NEES as normalized estimation error squared.The simulation uses predefined time offsets of 5 ms, 15 ms, and 30 ms.
- Real-sensor evaluation: The real sensor is an Intel Realsense ZR300 containing a fisheye global-shutter camera and an IMU.The camera has a 100° × 133° field of view, and the IMU includes gyroscope and accelerometer measurements.
- Implementation: The optimization keeps selected keyframes and feature, pose, velocity, and IMU-bias variables in a bounded local window for real-time computation.Previous keyframes are marginalized, and Ceres Solver is used for the nonlinear optimization.
B. Temporal Calibration Results
Temporal calibration is evaluated with real-sensor exposure changes, manually time-shifted EuRoC sequences, and real-world motion. The proposed method remains consistent across offsets, converges quickly, and improves relative pose and trajectory performance compared with methods without temporal calibration.
- Real-sensor calibration: The ZR300 real-sensor experiment compares estimated offsets from the proposed method and Kalibr across exposure times from 20 ms to 30 ms.Fifteen datasets were collected for each exposure time using chessboard motion with sufficient rotational and accelerated motion.
- Real-sensor calibration: The estimated temporal offset varies linearly with exposure time with a slope around 0.5 for both the proposed method and Kalibr.The offset comprises fixed communication and triggering delays plus half the exposure time.
- Dataset VIO evaluation: On time-shifted EuRoC sequences, VINS-Mono’s RMSE follows a parabolic curve and its tolerance interval is only within 6 milliseconds.The proposed method’s RMSEs remain the same under different time offsets.
- Dataset VIO evaluation: For OKVIS, increasing temporal offset degrades RMSE and can cause divergence at 30 ms, whereas the proposed method’s RMSEs remain almost unchanged within each sequence.The proposed method accurately calibrates the predefined offsets and obviously outperforms OKVIS above 10 ms.
- Convergence and real-world evaluation: The estimated offset converges to a stable value within a few seconds, and real-world relative pose errors are lower for the proposed method than for OKVIS.In the real-world experiment, the calibrated temporal offset is 12.74 ms and the proposed method outperforms OKVIS in translation and rotation.
2) Real-world Experiment:
In a real-world handheld experiment, the proposed online temporal calibration produced lower relative pose error and less drift than OKVIS in translation and rotation. The calibrated camera–IMU offset was 12.74ms.
- The experiment used an Intel Realsense camera at 30Hz and inertial measurements at 350Hz during circular handheld motion.OptiTrack results served as ground truth, and the proposed method was compared with OKVIS.
- OKVIS’s translation and rotation estimates drifted noticeably over time across the five-circle trajectory.The trajectory and axis-wise translation and rotation comparisons were shown in Figures 10–12.
- The proposed method had lower relative pose error than OKVIS, whose error also increased faster over time.The comparison used relative pose error in Figure 13.
- 12.74ms was the calibrated temporal offset, a discrepancy reported as significantly affecting long-run VIO without calibration and compensation.
V. CONCLUSION
The paper presents an online camera–IMU time-offset calibration approach that jointly optimizes temporal offset with states and features. Results indicate accurate calibration and motion estimation, with applicability beyond the demonstrated monocular setup.
- The approach jointly optimizes time offset with IMU and camera states and feature locations.
- The general model can be adopted in optimization-based visual-inertial frameworks.
- Simulation and experimental results indicate high accuracy in time-offset calibration and system motion estimation compared with state-of-the-art offline methods.
- The method demonstrated with a monocular sensor suite can be generalized to multi-camera visual-inertial systems.