PaperlayerView exact revision on arXiv

Source-linked AI summary

ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras

Raul Mur-Artal, Juan D. Tardos

arXiv:1610.06475v2cs.ROcs.CV

TL;DR

Visual SLAM must address monocular depth and scale ambiguity, initialization challenges, and scale drift. ORB-SLAM2 provides an open-source system for monocular, stereo, and RGB-D cameras using bundle adjustment, loop closing, relocalization, and map reuse, achieving state-of-the-art accuracy in most evaluated cases.

  • Problem

    Monocular visual SLAM lacks observable depth and scale, requires initialization techniques, and can suffer scale drift or fail during pure rotation.

  • Method

    ORB-SLAM2 is an open-source SLAM system for monocular, stereo, and RGB-D cameras combining bundle adjustment with loop closing, relocalization, and map reuse.

  • Results

    ORB-SLAM2 achieves higher accuracy than ICP- or direct-method RGB-D systems and direct stereo SLAM, while providing zero-drift localization in mapped areas.

  • Takeaways & Limitations

    The released system supports real-time operation on standard CPUs and reliable long-term localization through globally consistent maps across varied environments.

Abstract

from arXiv · show

We present ORB-SLAM2 a complete SLAM system for monocular, stereo and RGB-D cameras, including map reuse, loop closing and relocalization capabilities. The system works in real-time on standard CPUs in a wide variety of environments from small hand-held indoors sequences, to drones flying in industrial environments and cars driving around a city. Our back-end based on bundle adjustment with monocular and stereo observations allows for accurate trajectory estimation with metric scale. Our system includes a lightweight localization mode that leverages visual odometry tracks for unmapped regions and matches to map points that allow for zero-drift localization. The evaluation on 29 popular public sequences shows that our method achieves state-of-the-art accuracy, being in most cases the most accurate SLAM solution. We publish the source code, not only for the benefit of the SLAM community, but with the aim of being an out-of-the-box SLAM solution for researchers in other fields.

I. INTRODUCTION

ORB-SLAM2 is presented as an open-source visual SLAM system addressing limitations of monocular sensing while supporting monocular, stereo, and RGB-D cameras. Its contributions include loop closing, relocalization, map reuse, improved stereo and RGB-D accuracy, and lightweight localization.

  • Motivation: Visual SLAM with a monocular camera has unknown scale, requires bootstrapping for initial mapping, suffers scale drift, and may fail during pure rotations.Stereo or RGB-D cameras solve these issues and provide more reliable visual SLAM solutions.
  • Contributions: ORB-SLAM2 is the first open-source SLAM system for monocular, stereo, and RGB-D cameras with loop closing, relocalization, and map reuse.The system builds on the authors’ monocular ORB-SLAM.
  • Results: RGB-D results achieve more accuracy than state-of-the-art methods based on ICP or photometric and depth error minimization by using Bundle Adjustment.This contribution targets improved RGB-D estimation accuracy.
  • Results: Stereo results are more accurate than state-of-the-art direct stereo SLAM by using close and far stereo points together with monocular observations.The method combines stereo and monocular information for stereo estimation.
  • Contributions: A lightweight localization mode reuses the map with mapping disabled.The paper illustrates stereo trajectory reconstruction with multiple successfully detected loop closures and RGB-D keyframe pose estimation.

II. RELATED WORK · A. Stereo SLAM

The related work reviews stereo and RGB-D SLAM, focusing on SLAM approaches, with emphasis on keyframe-based optimization, loop closure, and direct methods. Earlier systems addressed larger environments and close/far stereo points, while later methods traded global consistency or robustness for scalability or feature independence.

  • II. RELATED WORK: The discussion and evaluation focus exclusively on SLAM approaches for stereo and RGB-D cameras.
  • A. Stereo SLAM: Paz et al. introduced an early stereo SLAM system that operated in larger environments and exploited both close and far points.
  • A. Stereo SLAM: Far stereo points with unreliable depth from little disparity were represented using inverse depth parametrization.
  • A. Stereo SLAM: Modern stereo SLAM systems are typically keyframe-based and use local bundle adjustment for scalability, sometimes achieving constant-time complexity without guaranteeing global consistency.
  • A. Stereo SLAM: RSLAM supports loop closure and active-area expansion, whereas S-PTAM lacks large loop closing; both use local optimization without enforcing global consistency.
  • A. Stereo SLAM: Stereo LSD-SLAM is a semi-dense direct method minimizing photometric error in high-gradient regions, but unmodeled rolling shutter or non-lambertian reflectance can severely degrade performance.

B. RGB-D SLAM

Prior RGB-D SLAM systems emphasized dense reconstruction, pose-graph optimization, or surfel-based mapping, with limitations in workspace size or computational complexity. ORB-SLAM2 instead converts RGB-D depth into stereo coordinates, uses bundle adjustment for a globally consistent sparse reconstruction, and targets lightweight, long-term localization on standard CPUs.

  • Prior RGB-D SLAM: KinectFusion fused depth into a dense volumetric model tracked by ICP but was limited to small workspaces without loop closing.Kintinuous extended operation to large environments with a rolling cyclical buffer and place-recognition loop closing.
  • Prior RGB-D SLAM: RGB-D SLAM and DVO-SLAM used feature or visual-odometry constraints with pose-graph optimization and loop closure.RGB-D SLAM combined feature matching and ICP, while DVO-SLAM minimized photometric and depth error for keyframe constraints.
  • Prior RGB-D SLAM: ElasticFusion built a surfel-based map and closed loops through non-rigid map deformation, but its complexity limited the implementation to room-size maps.The approach provided detailed reconstruction and localization accuracy while scaling with the number of surfels.
  • ORB-SLAM2: ORB-SLAM2 synthesizes stereo coordinates from RGB-D depth, making the system agnostic to stereo or RGB-D input.Its tracking thread preprocesses stereo or RGB-D input so the rest of the system operates independently of the sensor.
  • ORB-SLAM2: ORB-SLAM2 uses bundle adjustment to build a globally consistent sparse reconstruction, enabling lightweight operation on standard CPUs and long-term localization.Depth maps can be fused locally or post-processed after full bundle adjustment to obtain accurate 3D models.

III. ORB-SLAM2 · A. Monocular, Close Stereo and Far Stereo Keypoints

ORB-SLAM2 combines feature-based tracking, local mapping, place recognition, and graph-based locality in parallel threads, using ORB features across monocular, stereo, and RGB-D inputs. Stereo information is represented through unified keypoints whose depth-dependent classification determines scale, translation, rotation, and triangulation behavior.

  • III. ORB-SLAM2: III. ORB-SLAM2 uses three parallel threads for tracking, local mapping, and map optimization.Tracking localizes each frame against the local map using feature matches and motion-only bundle adjustment.
  • III. ORB-SLAM2: III. ORB-SLAM2 uses DBoW2 place recognition for relocalization, reinitialization, and loop detection after tracking failure or in mapped scenes.A covisibility graph and minimum spanning tree retrieve local keyframe windows for local processing.
  • III. ORB-SLAM2: III. ORB-SLAM2 uses the same ORB features for tracking, mapping, and place recognition.The features are robust to rotation, scale, camera auto-gain, auto-exposure, and illumination changes, while remaining fast to extract and match.
  • A. Monocular, Close Stereo and Far Stereo Keypoints: A. Monocular, Close Stereo and Far Stereo Keypoints pre-processes images into salient keypoint features and discards the input images afterward.This makes subsequent operations independent of whether the sensor is stereo or RGB-D, while supporting monocular, close stereo, and far stereo keypoints.
  • A. Monocular, Close Stereo and Far Stereo Keypoints: A. Monocular, Close Stereo and Far Stereo Keypoints represent stereo observations as xs = (uL, vL, uR) using rectified-image matching.For stereo cameras, ORB features are extracted in both images and matched along horizontal epipolar lines.
  • A. Monocular, Close Stereo and Far Stereo Keypoints: A. Monocular, Close Stereo and Far Stereo Keypoints approximate the Kinect and Asus Xtion structured-light baseline as 8cm.The baseline is between the structured-light projector and infrared camera.
  • A. Monocular, Close Stereo and Far Stereo Keypoints: A. Monocular, Close Stereo and Far Stereo Keypoints handle stereo and RGB-D features equivalently by representing depth-sensor uncertainty through the virtual right coordinate.This unifies stereo and RGB-D processing in the rest of the system.
  • A. Monocular, Close Stereo and Far Stereo Keypoints: A. Monocular, Close Stereo and Far Stereo Keypoints classify observations as close below 40 times the baseline and far otherwise.Close points provide scale, translation, and rotation from one frame; far points provide accurate rotation but weaker scale and translation, so they are triangulated with multiple-view support. Monocular points are also triangulated from multiple views and contribute rotation and translation without scale.

B. System Bootstrapping · C. Bundle Adjustment with Monocular and Stereo Constraints

For stereo or RGB-D cameras, ORB-SLAM2 initializes directly from depth in the first frame, avoiding monocular structure-from-motion initialization. Its bundle adjustment operates at tracking, local mapping, and post-loop-closure scales using monocular and rectified-stereo reprojection constraints.

  • B. System Bootstrapping: Stereo and RGB-D bootstrapping uses the first frame to create an origin keyframe and an initial map from all stereo keypoints.Depth from one frame removes the need for a specific monocular structure-from-motion initialization.
  • C. Bundle Adjustment with Monocular and Stereo Constraints: Bundle adjustment optimizes camera pose during tracking, covisible keyframes and points during local mapping, and the entire map after loop closure.All three optimizations use the Levenberg–Marquardt method implemented in g2o.
  • C. Bundle Adjustment with Monocular and Stereo Constraints: Motion-only bundle adjustment optimizes camera orientation R ∈ SO(3) and position t ∈ R^3 by minimizing reprojection error for matched 3D points and keypoints.The matches associate world-coordinate points Xi ∈ R^3 with observed keypoints.
  • C. Bundle Adjustment with Monocular and Stereo Constraints: The reprojection objective uses a robust Huber cost and a covariance matrix associated with each keypoint’s scale.The formulation supports monocular and rectified-stereo projection functions.
  • C. Bundle Adjustment with Monocular and Stereo Constraints: Monocular and rectified-stereo projections are defined using calibrated focal lengths, principal point coordinates, and stereo baseline.The calibration parameters are (fx, fy), (cx, cy), and b, respectively.
  • C. Bundle Adjustment with Monocular and Stereo Constraints: Local bundle adjustment optimizes covisible keyframes KL and all points PL observed in them, while other keyframes observing PL contribute to the cost but remain fixed.The optimization is defined over matches Xk between points in PL and keypoints in each keyframe k.
  • C. Bundle Adjustment with Monocular and Stereo Constraints: Full bundle adjustment optimizes all keyframes and points in the map, fixing the origin keyframe to eliminate gauge freedom.It is the specific case of local bundle adjustment in which the optimized sets encompass the entire map.

D. Loop Closing and Full BA

ORB-SLAM2 closes loops by detecting and validating them before pose-graph correction, using rigid-body transformations for stereo/depth data without scale-drift handling. A subsequent full bundle adjustment refines the result in a separate thread while preserving continued mapping and loop detection.

  • Loop Closing: Loop closing first detects and validates a loop, then corrects it by optimizing a pose-graph.The loop-closing procedure is explicitly divided into detection and validation, followed by pose-graph optimization.
  • Loop Closing: Stereo/depth information makes scale observable, so geometric validation and pose-graph optimization use rigid-body transformations instead of similarities.Unlike monocular ORB-SLAM, stereo/depth loop correction does not need to handle scale drift.
  • Full BA: ORB-SLAM2 performs a full BA optimization after the pose-graph to achieve the optimal solution.Because this optimization may be very costly, it runs in a separate thread while map creation and loop detection continue.
  • Full BA: If a new loop is detected during optimization, ORB-SLAM2 aborts the bundle adjustment and proceeds to close the loop.This handles the challenge of merging bundle-adjustment output with the map’s current state.

E. Keyframe Insertion

ORB-SLAM2 inserts keyframes frequently and subsequently culls redundant ones, adding a stereo-depth-aware condition for challenging scenes dominated by far points. This condition ensures enough close points for accurate translation estimation, while far points mainly support orientation.

  • ORB-SLAM2 inserts keyframes very often and culls redundant ones afterwards, following the policy introduced in monocular ORB-SLAM.
  • Distinguishing close from far stereo points enables a new keyframe-insertion condition for challenging environments where much of the scene lies far from the stereo sensor.
  • Close points provide information needed for accurate translation estimation, whereas far points contribute to orientation but weakly constrain translation and scale.In KITTI 01, points nearer than 40 times the stereo baseline are classified as close, while more distant points are far.

F. Localization Mode · IV. EVALUATION · A. KITTI Dataset

ORB-SLAM2 adds a lightweight localization mode for stable, well-mapped environments and is evaluated on KITTI alongside other SLAM systems using repeated runs and standard trajectory metrics. On KITTI, the stereo system detects and reuses loops, generally achieves under 1% relative error, and handles the challenging highway sequence better than the monocular system.

  • F. Localization Mode: Localization Mode disables local mapping and loop closing while continuously localizing through tracking, relocalization, visual-odometry matches, and map-point matches.It is intended for lightweight long-term localization in well-mapped areas without significant environmental changes.
  • IV. EVALUATION: ORB-SLAM2 was evaluated on three popular datasets against state-of-the-art SLAM systems using published results and standard literature metrics.Experiments ran on an Intel Core i7-4790 computer with 16Gb RAM; each sequence was run 5 times and median trajectory accuracy was reported.
  • A. KITTI Dataset: KITTI provides stereo urban and highway sequences recorded from a car with a ∼54cm baseline, 10Hz operation, and 1240 × 376 rectified resolution.Sequences 00, 02, 05, 06, 07, and 09 contain loops; ORB-SLAM2 detects and reuses all except the loop in sequence 09.
  • A. KITTI Dataset: The KITTI evaluation reports estimated trajectories against ground truth for sequences 00, 01, 05, and 07.These trajectories are shown in Fig. 4 with estimated paths in black and ground truth in red.
  • A. KITTI Dataset: ORB-SLAM2 outperforms Stereo LSD-SLAM in most KITTI sequences and generally achieves a relative error lower than 1%.Sequence 01 has slightly worse translation because few close points are trackable at high speed and low frame-rate, while orientation error reaches 0.21 degrees per 100 meters.
  • A. KITTI Dataset: The stereo version processes KITTI sequence 01, where the monocular system failed.Stereo initialization from one stereo keyframe avoids the monocular system’s delayed two-keyframe initialization, preventing tracking loss when close points appear briefly.
  • A. KITTI Dataset: In KITTI 08, stereo ORB-SLAM2 estimates the true trajectory and map scale without scale drift, unlike monocular ORB-SLAM.The monocular system exhibits severe scale drift, especially at turns.

B. EuRoC Dataset · C. TUM RGB-D Dataset

The EuRoC evaluation uses stereo sequences from a micro aerial vehicle to assess map reuse and loop closing across varied flight conditions. The TUM RGB-D evaluation compares ORB-SLAM2 with established methods on indoor sequences spanning diverse scene conditions.

  • B. EuRoC Dataset: EuRoC contains 11 stereo sequences recorded by a micro aerial vehicle flying through two rooms and a large industrial environment.The sequences cover varied indoor and industrial settings.
  • B. EuRoC Dataset: The EuRoC stereo sensor has a ∼11cm baseline and captures WVGA images at 20Hz.These are the sensor characteristics specified for the dataset.
  • B. EuRoC Dataset: EuRoC sequences are classified as easy, medium, or difficult according to MAV speed, illumination, and scene texture.The classification reflects flight and visual conditions.
  • B. EuRoC Dataset: Because the MAV revisits the environment in every EuRoC sequence, ORB-SLAM2 can reuse its map and close loops when necessary.Map reuse and loop closing are demonstrated across all sequences.
  • C. TUM RGB-D Dataset: TUM RGB-D contains indoor sequences recorded with RGB-D sensors and grouped by texture, illumination, and structure conditions.The categories support evaluation of object reconstruction and SLAM/odometry methods.
  • C. TUM RGB-D Dataset: The TUM RGB-D results use a subset of sequences where most RGB-D methods are usually evaluated.The selected subset is intended for comparison with common RGB-D evaluations.
  • C. TUM RGB-D Dataset: Table III compares ORB-SLAM2 against ElasticFusion, Kintinuous, DVO-SLAM, and RGB-D SLAM.These are identified as state-of-the-art comparison methods in the passage.
  • C. TUM RGB-D Dataset: ORB-SLAM2 is the only method in the TUM RGB-D comparison based on the approach described in the passage.The supplied passage ends before specifying the approach.

D. Timing Results

The timing evaluation measures thread-task runtimes across sequences with different image resolutions and sensors, showing that tracking remains fast enough for real-time operation. Runtime differences arise from sequence structure, especially loop keyframes and covisibility-graph density.

  • D. Timing Results: Timing results report means and two-standard-deviation ranges for each thread task across sequences with different image resolutions and sensors.Because each sequence contains one loop, full bundle adjustment and some loop-closing tasks are executed only once.
  • D. Timing Results: The average tracking time per frame is below the inverse of each camera frame-rate, enabling real-time operation.Stereo ORB extraction is parallelized.
  • D. Timing Results: Loop-closing and local-processing costs depend on sequence structure, with denser covisibility graphs making loop fusion, pose-graph optimization, full BA, local-map tracking, and local BA more expensive.KITTI 07 has more loop keyframes, but the indoor fr3 office has a denser covisibility graph, increasing local map size and processing costs.

V. CONCLUSION

ORB-SLAM2 is presented as a real-time, open-source SLAM system for monocular, stereo, and RGB-D sensors that supports relocalization, loop closing, map reuse, and globally consistent mapping. The conclusion reports state-of-the-art accuracy, zero-drift localization in mapped areas, computational advantages of bundle adjustment for RGB-D, and several future extensions.

  • Contributions: ORB-SLAM2 performs relocalization, loop closing, and map reuse in real time on standard CPUs across a wide range of environments.The system focuses on globally consistent maps for reliable, long-term localization.
  • Contributions: Its localization mode provides robust, zero-drift, lightweight localization in known environments by combining relocalization with visual odometry tracks and map matching.The mode is described as useful for applications such as tracking a user viewpoint in virtual reality.
  • Results: ORB-SLAM2 achieves the highest accuracy in most comparisons to state-of-the-art methods.The conclusion specifically identifies it as the best stereo SLAM solution on the KITTI visual odometry benchmark.
  • Results: On KITTI, ORB-SLAM2 provides zero-drift localization in already mapped areas, unlike the stereo visual odometry methods discussed.This result is presented as a crucial advantage over recent stereo visual odometry approaches.
  • Results: For RGB-D localization, bundle adjustment outperforms direct methods and ICP while requiring less computational expense and no GPU for real-time operation.The conclusion frames this advantage around obtaining the most accurate camera localization.
  • Open-source release: The released source code includes examples and instructions for researchers, supports monocular, stereo, and RGB-D inputs, and contains a monocular augmented-reality example.The authors describe ORB-SLAM2 as the first open-source visual SLAM system known to them with all three input modalities.
  • Future work: Future extensions include non-overlapping multi-camera, fisheye, and omnidirectional support, large-scale dense fusion, cooperative mapping, and greater motion-blur robustness.These are presented as example directions rather than completed capabilities.
Loading 1610.06475v2…