PaperlayerView exact revision on arXiv

Source-linked AI summary

MAC-I$^2$: Learned Metrics-Aware Covariance for Robust Visual-Inertial Fusion in Initialization and Calibration

Xiang Fei, Yuheng Qiu, Can Xu, Yutian Chen, Ruogu Li, Xingxing Zuo, Wenshan Wang, Sebastian Scherer

arXiv:2609.07116v1cs.RO

TL;DR

Existing VI fusion methods use predefined covariances despite context-dependent measurement reliability. MAC-I2 learns metrics-aware covariances for visual and inertial measurements and applies them in an initialization and calibration system. The system reports high initialization success on EuRoC and VBR, including where VINS-Mono performs poorly.

  • Problem

    Existing VI methods use predefined covariances that do not capture how visual and inertial measurement reliability varies with local context.

    Source: §I · pp. 1–2

  • Method

    MAC-I2 derives visual pose covariance from learned feature-matching uncertainty and predicts inertial covariance with a learned IMU model and held-out fine-tuning.

    Source: §I · p. 2

  • Results

    99.9% initialization success on EuRoC, and 80% on VBR where VINS-Mono falls below 10%.

    Source: §I · p. 2

  • Takeaways & Limitations

    The initialization and calibration showcase reports robustness and accuracy gains over state-of-the-art methods across challenging environments.

    Source: §VII · p. 8

Abstract

from arXiv · show

Visual-Inertial (VI) fusion is fundamental to accurate and robust state estimation, where camera and IMU measurements are combined according to their respective uncertainties. Existing methods, however, fuse the two modalities with predefined uncertainties, regardless of how reliable each is in the local context, and thus often struggle under challenging environments involving illumination changes, dynamic objects, and textureless regions. In this paper, we present MAC-I$^2$, which achieves robust VI fusion through learned metric-aware covariance for both modalities, so that vision and IMU compete on their own merits rather than relying on predefined uncertainties. Here, metrics-aware means that each predicted covariance faithfully reflects the actual magnitude of the corresponding measurement noise. On the visual side, we propagate learned feature-matching uncertainties into pose covariances for the fusion. On the inertial side, motivated by the observation that integration error accumulates sharply at the early stage and grows slowly afterward, we design a learned IMU model with a learnable initial covariance, and propose a dedicated fine-tuning strategy on a held-out training subset to enable the metrics-aware covariance on unseen sequences. As a showcase, we build a VI initialization and calibration system, since accurate and robust initialization and calibration are the prerequisite for any reliable VI system. Experiments on EuRoC, and VBR show that MAC-I$^2$ substantially outperforms existing methods: it achieves a 99.9% initialization success rate on EuRoC, reducing gravity and velocity errors by about 60% and 42% over the strongest baseline, and maintains 80% success rate on challenging VBR sequences where baseline methods such as VINS-Mono drop below 10%.

I. INTRODUCTION

The introduction argues that fixed or heuristic covariances do not reflect context-dependent measurement reliability, motivating learned metrics-aware covariance for visual and inertial fusion. MAC-I2 applies this approach to initialization and calibration and reports strong results on EuRoC and VBR.

I. INTRODUCTION

  • Fixed covariances cannot represent local reliability changes, so unreliable measurements may be over-weighted and degrade accuracy or cause failure.Visual reliability varies with illumination changes, dynamic objects, and textureless regions; inertial reliability is affected by motion patterns.Source: §I · pp. 1–2
  • MAC-I2 propagates learned feature-matching uncertainty into visual pose covariance and uses a learned IMU model with learnable initial covariance and held-out fine-tuning for unseen sequences.The stated design follows the observation that integration error accumulates sharply early and grows more slowly afterward.Source: §I · p. 2
  • 99.9% initialization success on EuRoC, while MAC-I2 maintains 80% on VBR, where VINS-Mono falls below 10%.The system is presented as a showcase application of MAC-I2.Source: §I · p. 2

A. Traditional Uncertainty Modeling in VI Fusion

  • Fixed and hand-tuned covariances are common, while robust costs, heuristics, and adaptive inertial schemes still rely on residual statistics or heuristic signals rather than actual measurement uncertainty.These approaches relax fixed weighting but do not capture how reliable each measurement is.Source: A. Traditional Uncertainty Modeling in… · p. 2

B. VI Initialization and Calibration

  • Initialization supplies the states and camera–IMU extrinsics needed for VI pose estimation, while existing methods use predetermined trust or hand-tuned noise models.The surveyed approaches include visual-motion alignment, decoupled vision and gyroscope integration, joint optimization, and offline or online extrinsic calibration.Source: B. VI Initialization and Calibration · p. 2

C. Learning-based Uncertainty Modeling

  • Prior learned methods estimate visual or inertial uncertainty, but learned inertial covariance is not metrics-aware and prior work had not fused metrics-aware covariances from both modalities.MAC-I2 addresses this gap with learned metrics-aware covariances for vision and IMU in joint optimization.Source: §I · p. 2, C. Learning-based Uncertainty Modeling · p. 3

A. Notations

  • The initialization and calibration state includes frame poses and velocities, gravity, IMU biases, and camera–IMU extrinsics; the biases are assumed constant during the process.With known camera–IMU extrinsics, the extrinsic can be removed from the state.Source: A. Notations · p. 3

B. IMU Integration and learning-based IMU model

This section describes the IMU preintegration inputs and learned measurement correction and uncertainty estimation, alongside MAC-VO’s use of learned visual uncertainty in pose-graph optimization.

B. IMU Integration and learning-based IMU model

  • IMU preintegration computes rotation, velocity, and translation terms between frames from raw accelerometer and gyroscope measurements.The integration follows a standard approach.Source: B. IMU Integration and learning-based… · p. 3
  • AirIMU estimates corrections and uncertainties from raw IMU data, then applies the learned corrections to the accelerometer and gyroscope measurements.The predicted uncertainties characterize measurement noise.Source: B. IMU Integration and learning-based… · p. 3

C. Learning-based Visual Odometry MAC-VO

  • MAC-VO uses learned metrics-aware uncertainty to select reliable keypoints and weight residuals in pose-graph optimization.Its consecutive-frame pose estimation uses learned 3D keypoint covariances in Mahalanobis residuals.Source: C. Learning-based Visual Odometry MAC-VO · p. 3

IV. METHODOLOGY

MAC-I2 combines learned visual-pose and inertial covariances to weight the two modalities in a joint VI initialization and calibration system. Its visual covariance is derived from feature uncertainty, while its learned IMU model uses a learnable initial covariance and held-out fine-tuning for unseen sequences.

A. Metrics-Aware Visual Pose Covariance

  • Feature-matching uncertainty from MAC-VO is propagated into visual-pose covariance, which closely tracks mean translation and rotation error on EuRoC.The pose covariance is derived using the visual pose-estimation Jacobian at convergence; the model was trained on synthetic TartanAir without fine-tuning on real-world datasets.Source: A. Metrics-Aware Visual Pose Covariance · pp. 3–4

B. Metrics-Aware Inertial Covariance

  • After bias correction, IMU integration error rises sharply at the start of a window and grows more slowly afterward, especially for rotation and velocity.The learned model uses a learnable initial covariance to start propagation in each segment, replacing the zero initial covariance of standard preintegration.Source: B. Metrics-Aware Inertial Covariance · p. 4
  • Held-out fine-tuning of the initial covariance and uncertainty decoder yields metrics-aware IMU uncertainties on unseen EuRoC sequences, while AirIMU remains overconfident.The rest of the network is frozen during fine-tuning.Source: B. Metrics-Aware Inertial Covariance · p. 4, Figure 4 · p. 5

C. VI Initialization and Calibration

  • Learned visual-pose and inertial covariances jointly weight visual and inertial residuals in the VI initialization and calibration system.The system is presented as a showcase application of MAC-I2.Source: §IV · p. 3, C. VI Initialization and Calibration · p. 4
  • The system offers a learned-IMU workflow with corrected preintegration and predicted uncertainties, and a workflow without it using raw IMU data and explicit gyroscope-bias estimation.The second workflow uses pre-calibrated or manually tuned IMU uncertainty.Source: C. VI Initialization and Calibration · p. 5, Figure 5 · p. 5

D. Gyroscope Bias and Extrinsic Rotation Estimation

When the learned IMU model is unavailable, initialization estimates gyroscope bias and calibration estimates camera-IMU rotation by comparing vision and IMU rotation. Joint optimization then weights visual and inertial residuals by their covariances.

D. Gyroscope Bias and Extrinsic Rotation Estimation

  • Without the learned IMU model, gyroscope bias and camera-IMU rotation are estimated by minimizing the difference between rotation from vision and rotation from IMU integration.The gyroscope-bias increment is initialized to zero; the rotation preintegration is first-order bias-corrected.Source: D. Gyroscope Bias and Extrinsic… · p. 5

E. Learned MAC-enhanced Joint Optimization

  • Joint optimization minimizes a weighted sum of visual and inertial residuals, with each weighted by its metrics-aware covariance.The IMU residual uses network-corrected preintegrations when the learned model is available, and first-order bias-corrected standard preintegration otherwise.Source: E. Learned MAC-enhanced Joint… · pp. 5–6
  • The visual residual constrains relative camera pose rather than raw features, retaining feature uncertainty through derived pose covariance while simplifying nonlinear optimization for initialization.Analytical Jacobians on SO(3)/SE(3) are derived for efficient optimization.Source: E. Learned MAC-enhanced Joint… · pp. 5–6

V. EXPERIMENTS

The experiments evaluate VI initialization on EuRoC and VBR and camera–IMU extrinsic calibration on TUM-VI. The setup specifies baseline methods, evaluation measures, and training data.

A. Experiment Setup

  • Initialization uses EuRoC and VBR, while extrinsic calibration uses four TUM-VI IMU calibration sequences with rapid six-DoF motion.Source: A. Experiment Setup · p. 6
  • Initialization baselines include geometric monocular and stereo methods; VINS-Mono is the calibration comparator, with Kalibr used for calibration ground truth.Source: A. Experiment Setup · p. 6
  • Initialization metrics include gravity-direction, velocity, and gyroscope-bias errors and success rate; calibration evaluates extrinsic rotation and translation.Source: A. Experiment Setup · p. 6
  • The visual model is trained on synthetic TartanAir; EuRoC IMU training uses MH 01, MH 05, V2 01, and V2 03, with uncertainty fine-tuned on MH 03 and V1 02.Source: A. Experiment Setup · p. 6

B. Initialization Accuracy and Robustness Evaluation

MAC-I2 improves initialization across EuRoC and VBR and produces stable calibration on aggressive TUM-VI sequences. Ablations report gains from learned covariances and the learned IMU model; the conclusion identifies broader fusion settings as future work.

B. Initialization Accuracy and Robustness Evaluation

  • 0.999 success rate across EuRoC segments accompanies gravity error of 0.418° and velocity RMSE of 0.018 m/s; on VBR, MAC-I2 reaches 0.803 success rate versus ORB-SLAM3 at 0.561.On EuRoC, gravity and velocity errors improve over DRT-t (1.137°) and VINS-Fusion (0.031 m/s) by about 63% and 42%; on VBR, VINS-Mono and sqrt-VINS fall below 0.10 success rate.Source: B. Initialization Accuracy and… · pp. 6–7, Table II · p. 7

C. Calibration Results

VI. ABLATION STUDY

  • Learned covariances reduce body rotation error from 0.362° to 0.215° and gyroscope-bias magnitude error from 3.9% to 2.0%; the learned IMU model also improves orientation metrics, and MAC-I2 outperforms AirIMU on gravity and rotation errors.Against AirIMU, average gravity-direction error falls from 0.611° to 0.418° and body rotation error from 0.254° to 0.215°.Source: §VI · pp. 7–8, Table III · p. 7, Table IV · p. 7

VII. CONCLUSION

  • The paper concludes that learned metrics-aware covariances improve VI fusion robustness and accuracy in the demonstrated initialization and calibration system, with full VI odometry, SLAM, and broader multi-sensor fusion left for future work.Source: §VII · p. 8

APPENDIX I ANALYTICAL JACOBIANS

The appendix summarizes analytical Jacobians used in optimization, using left-multiplicative perturbations on SO(3) and SE(3) and minimal tangent-space coordinates. It also develops first-order expressions for gyroscope-bias and camera–IMU rotation estimation.

A. Notation

  • The appendix uses standard SO(3) and SE(3) Lie definitions and reports derivatives with respect to minimal tangent-space coordinates.Source: A. Notation · p. 10

B. Left perturbation model

C. First-order identities used in the derivations

D. Gyroscope bias and camera-IMU rotation estimation

E. Visual pose-level residual (main paper Eq. (10))

The appendix details Jacobians for the visual pose-level residual and the preintegrated IMU residuals. It distinguishes learned-IMU and no-learned-IMU cases where their dependencies differ.

E. Visual pose-level residual (main paper Eq. (10))

F. IMU residuals (main paper Eqs. (12)–(14))

APPENDIX II VISUAL RESIDUAL COVARIANCE

The visual residual covariance is derived by propagating learned keypoint uncertainty first to the estimated relative pose, then through the Log map defining the residual.

APPENDIX II VISUAL RESIDUAL COVARIANCE

  • Learned 3D keypoint uncertainties are propagated through pose-graph optimization to obtain covariance for the estimated relative pose.The resulting pose covariance is then propagated through the Log map that defines the visual residual.Source: APPENDIX II VISUAL RESIDUAL COVARIANCE · p. 12
  • The Log-map residual covariance completes the propagation from the estimated relative pose to the pose-level visual residual.The section uses a left-perturbation convention and translation-first ordering for δξ.Source: APPENDIX II VISUAL RESIDUAL COVARIANCE · p. 12
  • The derivation reuses the translation-first perturbation ordering δξ = [δρ⊤ δϕ⊤]⊤.This ordering places translation before rotation.Source: APPENDIX II VISUAL RESIDUAL COVARIANCE · p. 12

A. Covariance of the estimated relative visual pose

The appendix derives relative visual-pose covariance from the local PGO objective, propagates it into the pose-level residual, and relates IMU residual covariance to preintegration covariance.

A. Covariance of the estimated relative visual pose

  • The relative visual-pose covariance comes from a Laplace approximation of the local PGO objective, using the inverse of its quadratic form.The derivation approximates the Hessian with the Gauss–Newton/Fisher information term and treats the estimate as stationary; the covariance lies in the tangent space under left perturbation.Source: A. Covariance of the estimated… · p. 13

B. Propagation to the pose-level residual

  • The estimated relative pose is treated as a measurement in the pose-level residual, whose perturbation is propagated through the residual’s Log-map expression.Because the measured pose is the leftmost factor, its perturbation requires no adjoint transport.Source: B. Propagation to the pose-level… · p. 13

APPENDIX III IMU RESIDUAL COVARIANCE

APPENDIX IV LEARNED IMU MODEL

The appendix describes the learned IMU covariance model and reports fusion, generalization, and keyframe-count results. Visual-inertial fusion improves pose accuracy, while covariance models remain metrics-aware in the tested settings.

APPENDIX IV LEARNED IMU MODEL

  • The learned IMU model uses a shared CNN encoder, network-predicted measurement covariances, and a learnable initial covariance to propagate preintegration uncertainty.The model jointly supervises the integrated IMU state and propagated covariance.Source: APPENDIX IV LEARNED IMU MODEL · pp. 13–14

APPENDIX V POSE ESTIMATION IMPROVEMENT VIA VI FUSION

  • 39.5% lower average rotation error (0.319° to 0.193°) and 16.7% lower translation error (0.024 m to 0.020 m) are reported for MAC-I2 with Learned IMU versus MAC-VO across EuRoC sequences.The comparison reports stronger relative improvement in rotation than translation.Source: Table V · p. 14, Appendix V · p. 14
  • On MH02 and V103, covariance weighting shifts trust toward the IMU in segments with degraded visual estimates, improving pose estimates over MAC-VO.The described segments include large rotation errors on MH02 and frequent rotation-error spikes under aggressive motion and challenging visual conditions on V103.Source: Figure 9 · p. 14, Appendix V · p. 14

A. Visual Covariance Generalization

  • Trained on synthetic TartanAir with pinhole cameras and applied without fine-tuning, the visual covariance remains metrics-aware on TartanAirV2 and fisheye-camera TUM-VI.The reported tests span diverse unseen synthetic scenes and handheld motions.Source: A. Visual Covariance Generalization · p. 14, Figure 10 · p. 15

B. IMU Covariance Generalization

  • On held-out TUM-VI room sequences, the IMU covariance tracks actual integration errors across handheld motions distinct from EuRoC MAV trajectories.The model requires per-sensor training; a raw-IMU workflow is also provided when sensor-specific training data is unavailable.Source: B. IMU Covariance Generalization · p. 15, Figure 11 · p. 15

APPENDIX VII SENSITIVITY TO KEYFRAME COUNT

  • All tested keyframe counts (5, 10, and 20) achieve 100% success on EuRoC, with diminishing error reductions as the count increases.With learned IMU, gravity direction error drops 27.4% from 0.576° to 0.418° when increasing from 5 to 10 keyframes, then 4.3% from 0.418° to 0.400° from 10 to 20.Source: Table VI · p. 15, APPENDIX VII SENSITIVITY TO KEYFRAME… · p. 15

Related papers

Loading 2609.07116v1…