PaperlayerView exact revision on arXiv

Source-linked AI summary

ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM

Carlos Campos, Richard Elvira, Juan J. Gómez Rodríguez, José M. M. Montiel, Juan D. Tardós

arXiv:2007.11898v2cs.RO

TL;DR

Visual SLAM and visual-inertial odometry have lacked a single system combining broad sensor support, accurate MAP estimation, and reuse of observations across disconnected mapping sessions. ORB-SLAM3 addresses this with integrated visual-inertial estimation, improved-recall place recognition, and an Atlas multi-map system. It reports robust performance and more than doubled accuracy versus competing multi-session methods in the cited comparisons, while low-texture environments remain its main failure case.

  • Problem

    Many VO systems rely only on recent observations, while difficult visual conditions can cause tracking loss and existing place recognition can sacrifice recall for precision.

  • Method

    ORB-SLAM3 combines MAP-based visual-inertial SLAM, improved-recall place recognition, and Atlas multi-map data association across monocular, stereo, and other camera configurations.

  • Results

    ORB-SLAM3 more than doubles accuracy against the compared multi-session methods, with an advantage reaching 3.2 times over VINS-Mono in multi-session evaluation.

  • Takeaways & Limitations

    Reusing short-term, mid-term, long-term, and multi-map associations enables accurate localization using maps built during previous sessions.

  • Takeaways & Limitations

    Low-texture environments are ORB-SLAM3’s main failure case because descriptor matching is less robust for tracking than photometric tracking.

Abstract

from arXiv · show

This paper presents ORB-SLAM3, the first system able to perform visual, visual-inertial and multi-map SLAM with monocular, stereo and RGB-D cameras, using pin-hole and fisheye lens models. The first main novelty is a feature-based tightly-integrated visual-inertial SLAM system that fully relies on Maximum-a-Posteriori (MAP) estimation, even during the IMU initialization phase. The result is a system that operates robustly in real-time, in small and large, indoor and outdoor environments, and is 2 to 5 times more accurate than previous approaches. The second main novelty is a multiple map system that relies on a new place recognition method with improved recall. Thanks to it, ORB-SLAM3 is able to survive to long periods of poor visual information: when it gets lost, it starts a new map that will be seamlessly merged with previous maps when revisiting mapped areas. Compared with visual odometry systems that only use information from the last few seconds, ORB-SLAM3 is the first system able to reuse in all the algorithm stages all previous information. This allows to include in bundle adjustment co-visible keyframes, that provide high parallax observations boosting accuracy, even if they are widely separated in time or if they come from a previous mapping session. Our experiments show that, in all sensor configurations, ORB-SLAM3 is as robust as the best systems available in the literature, and significantly more accurate. Notably, our stereo-inertial SLAM achieves an average accuracy of 3.6 cm on the EuRoC drone and 9 mm under quick hand-held motions in the room of TUM-VI dataset, a setting representative of AR/VR scenarios. For the benefit of the community we make public the source code.

I. INTRODUCTION

ORB-SLAM3 extends visual SLAM with tightly integrated MAP-based visual-inertial estimation, improved-recall place recognition, and multi-map data association. These capabilities support robust, accurate localization while reusing observations across time and mapping sessions.

  • Motivation: SLAM uses short-, mid-, and long-term data association, whereas many VO systems use only recent observations and accumulate drift.Long-term matching can reset drift and correct maps using pose-graph optimization or bundle adjustment.
  • Contributions: ORB-SLAM3 is presented as a complete visual, visual-inertial, and multi-map SLAM library.It supports monocular and stereo configurations and extends prior ORB-SLAM systems.
  • Contributions: Its monocular and stereo visual-inertial systems use MAP estimation throughout, including IMU initialization.The approach integrates visual and inertial estimation and is evaluated on public datasets.
  • Contributions: Improved-recall place recognition checks geometric consistency before local consistency with three covisible keyframes.This increases recall and densifies data association at slightly higher computational cost.
  • Contributions: ORB-SLAM3’s Atlas represents disconnected maps and supports relocalization, loop closure, map merging, and incremental multi-session SLAM.Maps built at different times can be automatically combined.
  • Contributions: The library uses an abstract camera representation supporting pin-hole and fisheye models through projection, unprojection, and Jacobian functions.This makes the SLAM code agnostic to the camera model.

II. RELATED WORK

Prior work spans feature-based, direct, filtering, tightly coupled, and visual-inertial odometry systems, but ORB-SLAM3 emphasizes combining accurate estimation with mid-term, long-term, and multi-map associations. Its contribution extends ORB-SLAM-VI with stereo-inertial SLAM and faster MAP-based initialization.

  • Visual SLAM: Keyframe-based methods enable accurate bundle adjustment, while direct methods optimize photometric error and may trade accuracy or association range differently.PTAM established keyframe-based accuracy, whereas LSD-SLAM used semi-dense maps with pose-graph optimization.
  • Visual SLAM: ORB-SLAM combines ORB features, covisibility graphs, bundle adjustment, and DBoW2-based loop closing and relocalization.Its three association types support short-term, mid-term, and long-term matching.
  • Visual-Inertial SLAM: Visual-inertial sensors improve robustness to poor texture, motion blur, and occlusions while making scale observable in monocular systems.Related systems include tightly coupled filtering, keyframe bundle adjustment, photometric odometry, and stereo-inertial odometry.
  • Visual-Inertial SLAM: ORB-SLAM3 extends ORB-SLAM-VI to stereo-inertial SLAM with MAP initialization that estimates true scale with 5% error in 2 seconds and converges to 1% error in 15 seconds.The authors connect its accuracy to precise initialization and mid-term data associations, including sequences without loops.

C. Multi-Map SLAM

ORB-SLAM3 extends ORB-SLAM into a complete multi-map, multi-session system that maintains disconnected maps and merges them when previously mapped areas are recognized. Its Atlas supports visual and visual-inertial operation, while improved place recognition and local bundle adjustment improve robustness and accuracy.

  • Related systems: Unlike systems that keep sub-maps separate or require manual initialization, ORB-SLAM3 automatically relates and seamlessly merges maps built at different times.The comparison passage contrasts ORB-SLAM3 with earlier keyframe-based and multi-map systems.
  • Evaluation: 2.6 times better accuracy than VINS-Mono is reported for monocular-inertial single-session EuRoC, increasing to 3.2 times in multi-session operation.The reported advantages are attributed to mid-term association, higher-recall place recognition, and more detailed local bundle-adjustment map merging.
  • Multi-map representation: ORB-SLAM3’s Atlas represents disconnected maps and supports relocalization, loop closure, map merging, and incremental multi-session SLAM.A unique DBoW2 database supports relocalization, loop closing, and map merging across the Atlas.
  • Tracking and mapping: The system localizes incoming frames against an active map, grows it with new keyframes, and searches all Atlas maps after tracking loss.Tracking estimates pose in real time and attempts relocalization across the Atlas when tracking fails.
  • Loop closing and map merging: Common regions trigger loop correction within the active map or seamless merging with another map, followed by independent full bundle adjustment after loop correction.Merging makes the other map active; the independent bundle-adjustment thread preserves real-time operation.

IV. CAMERA MODEL

ORB-SLAM3 abstracts camera models throughout the SLAM pipeline, enabling pin-hole and fisheye operation without relying on image rectification. Its stereo formulation treats the rig as two calibrated monocular cameras while incorporating their rigid relationship for scale and pose estimation.

  • Camera-model abstraction: ORB-SLAM3 isolates projection, unprojection, and Jacobian operations into camera modules, supporting pin-hole and Kannala-Brandt fisheye models.This abstraction lets the pipeline use different camera models without changing the rest of the SLAM system.
  • Fisheye cameras: Image rectification is unsuitable for fisheye lenses reaching or exceeding 180° field of view because it enlarges peripheral objects and reduces central resolution.These distortions hinder feature matching, motivating direct support for the fisheye camera model.
  • Relocalization: MLPnP replaces pin-hole-specific ePnP to make camera relocalization independent of the camera model.The change addresses relocalization when tracking fails while retaining calibrated-camera geometry.
  • Stereo formulation: ORB-SLAM3 avoids stereo image rectification by modeling the rig as two monocular cameras with a constant relative SE(3) transformation and optional shared image region.This accommodates divergent or fisheye stereo configurations that make rectification restrictive or infeasible.
  • Scale and pose estimation: Stereo overlap provides true-scale landmarks at first observation, while non-overlapping regions contribute monocular observations triangulated from multiple views.The pipeline estimates a 6 DoF rigid-body pose and incorporates stereo constraints during triangulation and bundle adjustment.
  • Visual-inertial optimization: Visual-inertial optimization combines preintegrated IMU measurements with reprojection residuals and robust Huber weighting for spurious visual matches.The optimized state includes poses, velocities, IMU biases, and 3D points; inertial residuals do not require robustification because miss-associations do not exist.

B. IMU Initialization

ORB-SLAM3 initializes visual-inertial state through staged MAP estimation, beginning with vision-only SLAM and then estimating inertial variables before joint refinement. The method improves initialization speed and accuracy, while slow motion can still limit convergence.

  • Initialization strategy: The initialization method uses three stages: vision-only MAP estimation, inertial-only MAP estimation, and joint visual-inertial refinement.Vision-only SLAM first supplies an up-to-scale trajectory; inertial variables are then estimated before joint optimization.
  • Vision-only MAP Estimation: Vision-only SLAM runs for 2 seconds, inserting keyframes at 4Hz, then optimizes an up-to-scale map with visual bundle adjustment.The resulting trajectory contains k = 10 camera poses and hundreds of points.
  • Inertial-only MAP Estimation: The inertial-only state estimates scale, gravity orientation, constant accelerometer and gyroscope biases, and up-to-scale body velocities using MAP estimation.The estimation uses IMU measurements and accounts for sensor uncertainty and prior information.
  • Joint refinement: After inertial optimization, poses, velocities, and map points are scaled and gravity-aligned, biases are updated, and IMU preintegration is repeated before joint refinement.Stereo-inertial initialization fixes the scale factor to one, removing it from the inertial-only variables.
  • Performance and limitation: The initialization is more accurate than algebraic joint methods and faster than earlier ORB-SLAM-VI and VI-DSO initialization approaches.Earlier methods required 15 seconds or 20–30 seconds to converge, respectively.
  • Performance and limitation: Slow motion may provide insufficient observability, causing initialization to fail to converge accurately within 15 seconds; periodic scale refinement is proposed for robustness.The refinement estimates scale and gravity direction while fixing biases estimated during mapping.

C. Tracking and Mapping

Tracking and mapping use computationally reduced visual-inertial optimizations rather than solving the full problem at every step. Tracking updates recent frame states, while mapping optimizes a keyframe-and-point sliding window with covisible observations.

  • Tracking: Tracking optimizes only the states of the last two frames while keeping map points fixed.This simplified visual-inertial optimization supports the tracking thread’s real-time operation.
  • Mapping: Mapping optimizes a sliding window of keyframes and their points, adding observations from covisible keyframes whose poses remain fixed.This avoids the intractable cost of optimizing the whole map for large environments.

D. Robustness to tracking loss

ORB-SLAM3 handles tracking loss with a short-term IMU-assisted recovery stage followed by new-map creation for longer losses. Maps lost shortly after initialization are discarded to prevent inaccurate accumulation.

  • Loss detection: The system declares visual-inertial tracking lost when fewer than 15 map points are tracked and then applies two recovery stages.The stages are short-term recovery using IMU prediction and long-term recovery through new-map initialization.
  • Short-term lost: During short-term loss, IMU readings predict the body state and guide map-point searches within a large image window.Recovered matches are incorporated into visual-inertial optimization; otherwise, the system advances after 5 seconds.
  • Long-term lost: During long-term loss, the system initializes a new visual-inertial map and makes it active.This lets tracking continue without immediately requiring recovery of the previous map.
  • Map discard: If tracking is lost within 15 seconds after IMU initialization, the current map is discarded to avoid accumulating inaccurate and meaningless maps.The discard rule specifically applies to early post-initialization failures.

VI. MAP MERGING AND LOOP CLOSING

ORB-SLAM3 improves long-term place recognition and uses it for loop closing and multi-map association. Geometric verification, local covisibility windows, and welding bundle adjustment support accurate loop closure and seamless map merging.

  • Loop Closing and Map Merging: A recognized keyframe in the active map triggers loop closure, whereas a match in another Atlas map triggers multi-map association and map merging.The same place-recognition process therefore supports both loop closing and cross-session map reuse.
  • Place Recognition: The system queries an Atlas-wide DBoW2 database for similar keyframes, then performs geometric verification to improve recall while rejecting false matches.Each candidate is checked using ORB descriptor matching, ambiguity filtering, and additional consistency tests.
  • Place Recognition: For each candidate, ORB-SLAM3 builds a local window containing the matching keyframe, covisible neighbors, and their observed map points.The window supports intensive mid-term data-association searches after the relative pose is estimated.
  • Loop Closing and Map Merging: Map merging transforms the active map into the matching map’s reference, assembles a welding window, fuses the maps, and removes duplicate points.The welding window includes keyframes, covisible neighbors, and all map points they observe.
  • Visual-Inertial Welding: Visual-inertial welding bundle adjustment optimizes selected poses, velocities, biases, and map points connected by IMU preintegration terms.The factor graph also contains reprojection-error terms and bias random-walk terms.

D. Loop Closing

ORB-SLAM3 evaluates loop closing and single-session robustness across EuRoC sensor configurations, reporting stronger accuracy and stability than competing systems. Its multi-map capability improves recovery from tracking losses, while inertial integration further boosts accuracy and robustness.

  • Single-session evaluation: ORB-SLAM3 achieves more accurate results than the best available systems across all four EuRoC sensor configurations.Table II reports medians over ten executions.
  • Visual-inertial results: ORB-SLAM3 is five to ten times more accurate than MCSKF, OKVIS, and ROVIO in the monocular-inertial configuration.It also more than doubles the accuracy of VI-DSO and VINS-Mono.
  • Visual-inertial results: ORB-SLAM3 is three to four times more accurate than Kimera and VINS-Fusion in the stereo-inertial configuration.BASALT is the closest result but does not complete sequence V203 because frames from one camera are missing.
  • Robustness: The ten-execution error visualization is used to assess variance and robustness, with comparisons indicating superiority over DSO, ROVIO, and VI-DSO.A median summarizes robust behavior, whereas a non-robust system exhibits high variance.
  • Multi-map robustness: Multi-map processing solves difficult V103 monocular and V203 stereo sequences that ORB-SLAM2 could not solve in most executions.The system creates a new map after tracking loss and later merges it with the global map.
  • Robustness: Inertial integration reduces median ATE relative to pure visual solutions and provides a more stable performance.Stereo-inertial SLAM has a slight robustness advantage over monocular-inertial SLAM, especially on V203.

B. Visual-Inertial SLAM on TUM-VI Benchmark

On the TUM-VI benchmark, ORB-SLAM3 is evaluated with fisheye visual-inertial configurations across indoor, outdoor, and highly feature-poor sequences. It performs especially well in small and medium indoor environments, while long outdoor sequences expose drift-related errors.

  • Benchmark comparison: ORB-SLAM3 is significantly more accurate than competing approaches on TUM-VI visual-inertial benchmark results.The comparison uses RMS ATE for regions with available ground truth.
  • Indoor sequences: Most small and medium indoor room and corridor sequences achieve errors below 10 cm.Continuous reuse of previously mapped regions is identified as a strength in these trajectories.
  • Outdoor sequences: Long outdoor sequences can produce errors from 10 to 70 meters when scarce close visual features cause drift in scale and accelerometer bias.ORB-SLAM3 nevertheless remains the best-performing system on the outdoor sequences.
  • Challenging sequences: The visual-inertial system processes the entire dark-slide sequences despite an almost total lack of visual features and no detectable loop closures.VINS-Mono and BASALT obtain better accuracy on some of these sequences using Lukas-Kanade feature tracking.
  • AR/VR setting: Room sequences provide full-trajectory ground truth and represent typical hand-held or head-mounted AR/VR motion in small environments.Table IV reports RMS ATE for ORB-SLAM3’s four sensor configurations using three-execution medians.

C. Multi-session SLAM

ORB-SLAM3 supports multi-session mapping by creating and merging maps across sequential sessions. Reusing previous maps improves accuracy and enables robust processing of difficult sequences that challenge single-session visual SLAM.

  • Multi-session procedure: Sequential EuRoC sessions are processed with an initial map, followed by new active maps that are quickly merged with the previous map.After merging, ORB-SLAM3 reuses the accumulated map during subsequent processing.
  • EuRoC results: ORB-SLAM3 more than doubles the accuracy of published CCM-SLAM and VINS-Mono multi-session results on EuRoC.The comparison reports global multi-session RMS ATE across the four sensor configurations and three rooms.
  • EuRoC results: Multi-session monocular and stereo SLAM robustly process V103 and V203 by exploiting the previous map.These difficult sequences were not robustly processed in the corresponding single-session comparison.
  • TUM-VI results: A small-room TUM-VI sequence supplies loop closures that reduce errors in longer sequences to centimeter level.Ground truth is unavailable outside the room, so the broader comparison is qualitative.
  • TUM-VI results: Processing outdoors1 after magistrale2 substantially reduces the approximately 60 m drift observed in stereo-inertial single-session processing.The resulting final map is reported as much more accurate.

D. Computing Time

ORB-SLAM3 runs its tracking, mapping, loop-closing, and map-merging operations in real time, while preserving accuracy through inertial local bundle adjustment and separate processing threads.

  • Tracking and mapping: 30–40 frames and 3–6 keyframes per second: ORB-SLAM3 operates in real time during tracking and mapping.The inertial component adds negligible tracking time, while inertial local bundle adjustment achieves better accuracy with similar running time.
  • Loop closing and map merging: 10 ms per keyframe: the novel place-recognition method adds little computational overhead.Map merging and loop closing remain below one second when using pose-graph optimization; full bundle adjustment can take a few seconds depending on map size.
  • Loop closing and map merging: Separate-thread execution prevents loop closing and map merging from interfering with the real-time performance of the rest of the system.This design isolates potentially longer optimization operations from the tracking and mapping threads.
  • Evaluation scope: The paper does not compare running time against other systems because such an evaluation was beyond its scope.The reported timing results therefore establish ORB-SLAM3’s real-time operation but not comparative speed superiority.
Loading 2007.11898v2…