PaperlayerView exact revision on arXiv

Source-linked AI summary

DM-VIO: Delayed Marginalization Visual-Inertial Odometry

Lukas von Stumberg, Daniel Cremers

arXiv:2201.04114v1cs.CVcs.RO

TL;DR

Monocular visual-inertial odometry struggles when motion leaves scale unobservable and initialization must balance speed against accuracy. DM-VIO addresses this with delayed marginalization and pose graph bundle adjustment, then continues optimizing scale and gravity direction; it reports state-of-the-art performance, including results exceeding stereo-inertial methods.

  • Problem

    Monocular visual-inertial systems can have unobservable scale during constant-velocity motion, making initialization duration and accuracy difficult to balance.

  • Method

    DM-VIO uses delayed marginalization and pose graph bundle adjustment for IMU initialization, with continued scale and gravity-direction optimization in the main system.

  • Results

    DM-VIO exceeds the state of the art in visual-inertial odometry, including stereo-inertial methods, across flying, handheld, and automotive scenarios.

  • Takeaways & Limitations

    Delayed marginalization supports an IMU initializer that captures full photometric uncertainty and transfers IMU information into the main system.

Abstract

from arXiv · show

We present DM-VIO, a monocular visual-inertial odometry system based on two novel techniques called delayed marginalization and pose graph bundle adjustment. DM-VIO performs photometric bundle adjustment with a dynamic weight for visual residuals. We adopt marginalization, which is a popular strategy to keep the update time constrained, but it cannot easily be reversed, and linearization points of connected variables have to be fixed. To overcome this we propose delayed marginalization: The idea is to maintain a second factor graph, where marginalization is delayed. This allows us to later readvance this delayed graph, yielding an updated marginalization prior with new and consistent linearization points. In addition, delayed marginalization enables us to inject IMU information into already marginalized states. This is the foundation of the proposed pose graph bundle adjustment, which we use for IMU initialization. In contrast to prior works on IMU initialization, it is able to capture the full photometric uncertainty, improving the scale estimation. In order to cope with initially unobservable scale, we continue to optimize scale and gravity direction in the main system after IMU initialization is complete. We evaluate our system on the EuRoC, TUM-VI, and 4Seasons datasets, which comprise flying drone, large-scale handheld, and automotive scenarios. Thanks to the proposed IMU initialization, our system exceeds the state of the art in visual-inertial odometry, even outperforming stereo-inertial methods while using only a single camera and IMU. The code will be published at http://vision.in.tum.de/dm-vio

I. INTRODUCTION

DM-VIO addresses difficult monocular visual-inertial initialization by combining delayed marginalization with pose graph bundle adjustment and continued scale optimization. Evaluated across drone, handheld, and automotive datasets, it is reported to outperform prior visual-inertial and stereo-inertial methods.

  • Motivation: Monocular visual-inertial initialization is difficult because scale can remain unobservable during constant-velocity motion, making premature initialization harmful.This difficulty has historically contributed to stereo-inertial methods outperforming mono-inertial systems.
  • Approach: DM-VIO combines parallel visual-only and IMU initialization with continued optimization of scale and gravity direction in the main system.The combination is intended to retain rapid convergence while improving scale accuracy.
  • Approach: Delayed marginalization maintains a second factor graph that can be readvanced, transfer IMU information into marginalized states, and replace inconsistent marginalization priors after scale changes.The method is proposed to address visual-uncertainty capture, information transfer, and prior consistency simultaneously.
  • Results: DM-VIO is reported to exceed the visual-inertial state of the art, including stereo-inertial methods, across three benchmark domains.The paper’s figure presents pointclouds and trajectories for representative sequences from those benchmarks.
  • Approach: Pose graph bundle adjustment combines pose-graph efficiency with bundle adjustment’s full uncertainty for IMU initialization.It is enabled by adding new IMU factors to the delayed factor graph.
  • System and evaluation: The system uses a multi-stage IMU initializer and dynamically weighted photometric factors within a monocular visual-inertial odometry pipeline.The evaluation covers EuRoC drone, TUM-VI handheld, and 4Seasons automotive scenarios.

II. RELATED WORK

Prior visual-inertial systems commonly separate visual odometry from IMU initialization, while monocular scale observability remains motion-dependent. DM-VIO instead transfers visual uncertainty and continues explicit scale optimization through delayed marginalization.

  • Visual odometry background: Direct visual methods optimize photometric error over dense, semi-dense, or sparse point clouds, providing the visual-method context for DM-VIO.The passage situates direct methods after earlier feature-based visual odometry and SLAM systems.
  • Prior systems: Earlier visual-inertial systems established tight camera–IMU integration as a route to improved odometry accuracy and robustness.The literature includes both tightly coupled visual-inertial odometry and SLAM systems.
  • Monocular initialization: Monocular visual-inertial initialization requires sufficient motion for scale observability, motivating separate visual-only and IMU initialization stages in many systems.Constant-velocity motion is identified as a common degenerate case for scale recovery.
  • Monocular initialization: DM-VIO continues optimizing scale explicitly in the main system rather than stopping scale refinement after initialization.The related-work discussion contrasts this with systems that limit refinement or do not transfer covariances between initializer and main system.
  • DM-VIO positioning: Delayed marginalization allows the initializer to capture full visual uncertainty while continuously optimizing scale in the main system.The paper presents this combination as distinct from the compared systems.

B. Direct Visual-Inertial Bundle Adjustment

DM-VIO jointly optimizes direct photometric and IMU terms over keyframes, using dynamic visual weighting and explicit scale and gravity-direction variables.

  • B. Direct Visual-Inertial Bundle Adjustment: DM-VIO jointly optimizes visual and IMU variables in a bundle-adjustment energy over active keyframes, with IMU data integrated by preintegration.The visual formulation is direct and based on DSO, while inertial errors penalize deviations from preintegrated state predictions.
  • B. Direct Visual-Inertial Bundle Adjustment: The objective combines dynamically weighted photometric error, IMU error, and prior terms, and is optimized with Levenberg-Marquardt.The priors include constraints on the first pose, gravity direction, and marginalization priors.
  • B. Direct Visual-Inertial Bundle Adjustment: A dynamic photometric weight reduces the relative influence of visual residuals when overall image quality is poor, increasing the relative IMU weight.The weight is activated above a threshold; the experiments use θ = 8.
  • B. Direct Visual-Inertial Bundle Adjustment: Scale and gravity direction are optimized explicitly because doing so improves convergence of bundle adjustment.The state also includes keyframe poses and associated visual-inertial variables.

C. Partial Marginalization using the Schur Complement

DM-VIO marginalizes selected variables with the Schur complement, producing a prior over their connected Markov blanket while retaining a bounded keyframe set.

  • C. Partial Marginalization using the Schur Complement: Schur-complement marginalization eliminates selected variables and produces a linear-system prior connecting the remaining Markov-blanket variables.The factors involving marginalized variables and connected variables are linearized at the current state estimate before elimination.
  • C. Partial Marginalization using the Schur Complement: The system keeps at most Nf = 8 keyframes and marginalizes a mixture of newer and older poses rather than always removing the oldest pose.This preserves active points and avoids the fixed-lag strategy used by conventional smoothers.

D. Delayed Marginalization

Delayed marginalization preserves the benefits of marginalization while allowing delayed graph updates, IMU information transfer, and relinearization around changing states.

  • D. Delayed Marginalization: Because ordinary marginalization cannot be reversed and fixes connected-variable linearization points, delayed marginalization addresses consistency problems when scale changes.The paper identifies significant scale changes after the first keyframe is marginalized as a particularly problematic case.
  • D. Delayed Marginalization: The delayed graph can be partially readvanced to capture the full photometric probability distribution for PGBA and update the main-system prior with IMU information.This transfers inertial information from initialization into the main system.
  • D. Delayed Marginalization: Delayed marginalization also permits relinearizing Markov-blanket variables while retaining visual and most inertial information.This is presented as a benefit over approaches that lose substantial prior inertial information when scale changes quickly.
  • D. Delayed Marginalization: Delayed marginalization maintains a second factor graph whose frame marginalization is delayed while point marginalization and the original order are preserved.In practice, the delay is d = 100, and marginalization remains equally fast as in the main graph.

E. Pose Graph Bundle Adjustment for IMU Initialization

PGBA uses the delayed graph for IMU initialization by adding inertial factors to successive connected frames, combining pose-graph efficiency with bundle-adjustment uncertainty.

  • E. Pose Graph Bundle Adjustment for IMU Initialization: PGBA populates the delayed graph with IMU and bias factors from the first frame still directly connected to the newest frame.IMU factors are inserted only across successive frames, avoiding connections between non-successive keyframes.
  • E. Pose Graph Bundle Adjustment for IMU Initialization: With Nf = 8 and delay d = 100, at least 93 IMU factors are available even in the worst case.At most Nf −2 poses lack corresponding IMU variables, and the graph is optimized with Levenberg-Marquardt.
  • E. Pose Graph Bundle Adjustment for IMU Initialization: PGBA combines pose-graph optimization with bundle adjustment by retaining photometric constraints over Nf frames while optimizing poses and inertial variables without updating point depths.Its octonary constraints connect Nf frames and capture the full bundle-adjustment probability distribution.

F. Robust Multi-Stage IMU Initialization

DM-VIO combines coarse IMU initialization, pose graph bundle adjustment, and delayed marginalization to estimate scale, gravity direction, and biases while maintaining consistent priors. The delayed graph supports readvancing, IMU information transfer, and marginalization replacement when scale changes.

  • F. Robust Multi-Stage IMU Initialization: Delayed marginalization maintains a second graph that can be readvanced to produce consistent marginalization priors with updated linearization points.This graph also supports injecting IMU factors into already marginalized states.
  • F. Robust Multi-Stage IMU Initialization: The initialization design follows three principles: optimize unknown variables first, jointly optimize for full covariance, and marginalize only near-optimal connected variables.These principles motivate coarse initialization, PGBA, and marginalization replacement, respectively.
  • F. Robust Multi-Stage IMU Initialization: PGBA jointly captures full photometric covariance to improve initial estimates of scale, gravity direction, and biases.Coarse IMU initialization supplies starting values for the PGBA, which models photometric noise properties.
  • F. Robust Multi-Stage IMU Initialization: The multi-stage pipeline enables early initialization, later reinitialization, and marginalization replacement when new scale information becomes available.Scale continues to be optimized in the main VIO system while the delayed graph supports prior updates.
  • F. Robust Multi-Stage IMU Initialization: The coarse initializer uses the last 100 keyframes with IMU factors while fixing poses and using a single bias.The delayed graph itself remains visual-only to facilitate marginalization replacement.

IV. RESULTS

The evaluation spans EuRoC, TUM-VI, and 4Seasons across drone, handheld, and automotive scenarios. Results use repeated runs, trajectory alignment, cumulative error plots, RMSE/ATE, and drift percentages.

  • IV. RESULTS: DM-VIO is evaluated on flying-drone, handheld, and automotive datasets representing three challenging application domains.The datasets are EuRoC, TUM-VI, and 4Seasons.
  • IV. RESULTS: Experiments generally run in realtime on the same CPU-only MacBook Pro, while ORB-SLAM3 uses a slightly stronger desktop because macOS is unsupported.This hardware distinction applies to the reported comparisons unless otherwise stated.
  • IV. RESULTS: Results use 10 runs per EuRoC sequence and 5 runs per sequence for the other datasets.Trajectories are aligned with ground truth using SE(3), with RMSE/ATE and drift percentages reported.
  • IV. RESULTS: Cumulative error plots show how many sequences remain below each accuracy threshold, while tables compare the method with results from other papers.The reported metric is RMSE, also called absolute trajectory error, with drift in percent for TUM-VI and 4Seasons.

A. EuRoC dataset

On EuRoC, DM-VIO outperforms the evaluated visual-inertial methods in RMSE and reports the lowest average scale error. On TUM-VI, it achieves the best result on 16 sequences and a mean drift of 0.472, while loop closure gives ORB-SLAM3 advantages on some sequences.

  • A. EuRoC dataset: DM-VIO outperforms all compared EuRoC methods in RMSE, with Basalt achieving lower error on only two sequences.The system also reports the lowest average scale error reported on the dataset.
  • B. TUM-VI dataset: DM-VIO achieves the best TUM-VI result on 16 sequences with a mean drift of 0.472, ahead of Basalt’s 8 sequences and 0.939 mean drift.The comparison includes stereo-inertial methods and uses RMSE ATE in meters in Table II.
  • B. TUM-VI dataset: TUM-VI combines large-scale indoor and outdoor handheld scenes with sequences where almost the full image is covered.Long straight-line walking periods give stereo methods an advantage because scale remains observable under constant motion.
  • B. TUM-VI dataset: On TUM-VI, DM-VIO is more robust overall than VI-DSO and ORB-SLAM3, although ORB-SLAM3 is more accurate on some sequences.ORB-SLAM3’s loop closure system provides an advantage in accuracy on those sequences.

C. 4Seasons dataset

4Seasons is challenging for monocular methods because constant-motion stretches make scale unobservable. DM-VIO’s delayed-marginalization-based initializer handles this setting and outperforms stereo-inertial systems without loop closures.

  • C. 4Seasons dataset: All methods use the same procedure for determining IMU noise parameters, including noise inflation factors of 1, 10, 100, and 1000.The best setting is selected for each method before a final evaluation of all 30 sequences, run five times each.
  • C. 4Seasons dataset: The 4Seasons evaluation crops the bottom 96 image pixels to remove the car hood from every method’s input.The same visual preprocessing is applied across the comparison.
  • C. 4Seasons dataset: DM-VIO outperforms stereo-inertial ORB-SLAM3 and Basalt on 4Seasons while using monocular images and no loop closures.The result is attributed to the novel IMU initialization in this automotive scenario.
  • C. 4Seasons dataset: 4Seasons is particularly difficult for monocular visual-inertial methods because its constant-motion stretches make scale unobservable.The evaluation covers an automotive dataset with a time-synchronized visual-inertial sensor.

I. ABLATION IMU INITIALIZER

The ablations show that each IMU-initialization component contributes, with delayed marginalization especially important for PGBA, readvancing, and marginalization replacement. Dynamic photometric weighting improves robustness on difficult TUM-VI sequences, while the initializer adds little regular runtime overhead.

  • IMU initializer: Each successive IMU-initializer ablation significantly worsens 4Seasons results, showing that all tested components contribute.The tested variants remove reinitialization, marginalization replacement, readvancing, or PGBA in stages.
  • IMU initializer: Removing pose graph bundle adjustment and using only the Coarse IMU initializer causes the largest performance degradation on 4Seasons.The ablation identifies PGBA as the most consequential component among the tested initializer variants.
  • Dynamic photometric weight: Dynamic photometric weights improve robustness on TUM-VI slides: the unweighted version fails in 4 of 15 runs, while the proposed method succeeds in all runs.The benefit is most visible when image quality becomes particularly poor.
  • Dynamic photometric weight: On other datasets, dynamic weighting produces only a marginal difference because severe image-quality degradation is rare.The weighting is designed for cases where poor images would otherwise inflate photometric residual influence.
  • Runtime: The initializer runs at 97 FPS for tracking and 19 FPS for keyframe processing on the evaluation machine, while delayed marginalization adds 0.8% of keyframe processing time.Delayed marginalization takes 0.44 ms per keyframe; the initializer’s coarse and PGBA components mainly run in separate threads.
Loading 2201.04114v1…