PaperlayerView exact revision on arXiv

Source-linked AI summary

Keyframe-based monocular SLAM: design, survey, and future directions

Georges Younes, Daniel Asmar, Elie Shammas, John Zelek

arXiv:1607.00470v2cs.CVcs.RO

TL;DR

Monocular SLAM is attractive because a single camera is inexpensive and broadly deployable, but its bearing-only measurements make mapping and localization challenging. This paper surveys keyframe-based systems, organizes their design components and strategies, and identifies limitations that motivate future work.

  • Problem

    Single-camera SLAM is challenging because monocular images provide bearing-only measurements, while existing surveys give limited exclusive attention to keyframe-based monocular SLAM.

  • Method

    The paper systematically surveys keyframe-based monocular SLAM architectures, implementations, environmental design choices, and open-source and closed-source systems.

  • Results

    The paper provides a structured account of seven KSLAM components and the implementation strategies used across systems.

  • Takeaways & Limitations

    The survey helps practitioners select systems for environmental conditions and helps researchers identify weaknesses for future work.

  • Takeaways & Limitations

    No public dataset supports an unbiased comparison across all surveyed systems because different methods favor different operational conditions.

Abstract

from arXiv · show

Extensive research in the field of monocular SLAM for the past fifteen years has yielded workable systems that found their way into various applications in robotics and augmented reality. Although filter-based monocular SLAM systems were common at some time, the more efficient keyframe-based solutions are becoming the de facto methodology for building a monocular SLAM system. The objective of this paper is threefold: first, the paper serves as a guideline for people seeking to design their own monocular SLAM according to specific environmental constraints. Second, it presents a survey that covers the various keyframe-based monocular SLAM systems in the literature, detailing the components of their implementation, and critically assessing the specific strategies made in each proposed solution. Third, the paper provides insight into the direction of future research in this field, to address the major limitations still facing monocular SLAM; namely, in the issues of illumination changes, initialization, highly dynamic motion, poorly textured scenes, repetitive textures, map maintenance, and failure recovery.

1. Introduction

This paper focuses on keyframe-based monocular SLAM because existing surveys rarely treat it exclusively and newer systems introduce substantial implementation differences. It organizes these systems to guide design choices, survey components, and identify open problems.

  • Motivation: Monocular SLAM uses a single camera to estimate location and build a map, offering low-power, low-cost sensing across indoor and outdoor environments.The paper notes that monocular cameras are bearing-only sensors, making SLAM more challenging than with depth-enabled sensors.
  • Research gap: Earlier surveys generally covered SLAM, visual odometry, or filter-based methods rather than the specific components and strategies of keyframe-based monocular SLAM.The paper also notes that many newer systems appeared after those surveys.
  • Contribution: The paper systematically compares components and implementation nuances across keyframe-based monocular SLAM systems.This organization is intended to help scholars and practitioners choose algorithms and locate weaknesses in existing techniques.
  • Paper scope: It covers architecture, open-source and closed-source systems, component choices under environmental conditions, and current open problems.The architecture discussion includes data association, initialization, pose estimation, map generation, optimization, maintenance, failure recovery, and loop closure.

2. Keyframe-based Monocular SLAM Architecture

Keyframe-based monocular SLAM separates regular-frame localization from keyframe optimization and organizes the system around seven interacting components. The architecture supports map expansion, maintenance, loop closure, and recovery when tracking or association fails.

  • Architecture: Keyframe-based systems separate camera localization on regular frames from optimization on keyframes, unlike filter-based systems that jointly update pose and all landmarks every frame.The paper attributes this separation to improved efficiency and reports a reduction in optimization parameters from twelve to six in the discussed context.
  • Architecture: A KSLAM design treats seven components: visual initialization, data association, pose estimation, map generation, BA/PGO/map maintenance, failure recovery, and loop closure.These components define the generic KSLAM flow presented in the paper.
  • Pipeline: Initialization establishes the first 3D map and camera poses, while data association uses predicted poses to match new frames against the map.Pose optimization then minimizes the error between measurements and associations.
  • Pipeline: Keyframes triangulate new landmarks to expand the map, while maintenance optimizes it, removes outliers, and supports loop-closure correction.Regular frames terminate after pose processing unless selected as keyframes.
  • Direct methods: Direct methods align image regions using brightness consistency, whereas dense and semi-dense variants use all pixels or pixels with significant brightness gradients.The brightness constraint assumes that a world point retains the same intensity across successive images.
  • Direct methods: Direct image alignment uses a nonlinear optimization whose per-iteration complexity is O(n^2N + n^3), with n transform parameters and N image pixels.The paper discusses several alternative alignment formulations with different computational complexities.

b. Feature-based methods

Feature-based methods reduce pixel-processing cost by matching salient image locations represented by descriptors. Their design balances distinctiveness and invariance against computational demands and environmental conditions.

  • Feature-based methods: Feature-based methods reduce computational complexity by matching salient image locations, or keypoints, instead of processing every pixel.Each feature is paired with a descriptor that measures similarity to other keypoints.
  • Feature-based methods: Useful features should be distinctive and invariant to viewpoint and illumination changes while remaining resilient to blur and noise.The paper contrasts these robustness goals with the need for computationally efficient extraction.
  • Feature-based methods: Feature detector selection depends on platform computational power and environment, with examples including Harris, Shi-Tomasi, FAST, and Difference of Gaussian detectors.The survey lists multiple detector families with different rotation, scale, and speed characteristics.
  • Feature-based methods: Descriptors such as BRIEF, BRISK, SURF, SIFT, HoG, FREAK, and ORB provide alternative representations for feature matching.The paper treats detailed extractor and descriptor comparisons as outside its scope.

c. Hybrid methods

Hybrid methods combine direct and feature-based approaches, using their complementary strengths for pose refinement or dense/semi-dense mapping. Within keyframe-based monocular SLAM, data association, initialization, pose estimation, map construction, maintenance, and failure recovery form an interconnected pipeline.

  • Hybrid methods: Hybrid systems combine direct and feature-based methods to refine camera pose estimates or generate dense/semi-dense maps.Direct methods use image intensity information, while feature-based methods match salient image locations.
  • Visual initialization: During initialization, 2D-2D correspondences are established before estimating a Homography or Fundamental matrix from matched image features.The selected model depends on scene geometry, since Homography and Fundamental matrix estimation have different degeneracies.
  • Pose estimation: Pose estimation uses a motion prior to limit data association, then minimizes photometric or landmark re-projection error to estimate the current camera pose.Most systems assume constant velocity or negligible inter-frame pose change when forming the prior.
  • Failure recovery: Failure recovery is essential because abrupt camera motion, motion blur, or featureless regions can cause monocular SLAM to fail.Systems invoke recovery procedures when pose optimization diverges or data association fails.

3. Design choices

This section surveys keyframe-based monocular SLAM systems, distinguishing open-source from closed-source implementations and emphasizing the depth enabled by code access.

  • Open-source systems receive deeper treatment because access to their code provides additional implementation insight.
  • The section covers open-source systems including PTAM, SVO, DT SLAM, LSD SLAM, ORB SLAM, and DPPTAM.
  • Closed-source systems are discussed separately and only briefly compared with the open-source systems.
  • The survey catalogs keyframe-based monocular SLAM systems as either open-source or closed-source.The referenced table lists seven open-source and sixteen closed-source systems.

3.1. Data association

The survey compares feature-based, direct, and hybrid data-association strategies, along with their feature representations, sampling choices, and robustness trade-offs.

  • Feature-based methods associate FAST or ORB features with local patches or descriptors, whereas direct methods use pixels with photometric gradients.
  • DSO subsamples high-gradient pixels within image blocks to reduce redundancy while maintaining spatial distribution and local image information.
  • PTAM uses a 4-level image pyramid to improve feature robustness to scale changes and enlarge the pose-estimation convergence radius.
  • SVO combines direct image alignment with FAST feature matching, using multiscale alignment to establish preliminary correspondences.
  • ORB SLAM extracts FAST corners across eight pyramid levels, computes 256-bit ORB descriptors, and stores them in a visual vocabulary.
  • DSO represents each candidate point with eight surrounding pixels to balance tracking information, computational time, and motion-blur resilience.

3.2. Visual initialization: choices made

Monocular SLAM systems initialize with homographies, Essential matrices, or random depths, making different scene assumptions and imposing different startup requirements.

  • Homography initialization assumes a planar scene, whereas Essential-matrix initialization assumes a non-planar scene.
  • PTAM’s homography-based initialization requires user-captured keyframes and slow, smooth, significant translation between them.
  • SVO avoids a second user input by waiting until tracked features reach a baseline threshold indicating sufficient parallax for homography estimation.
  • LSD SLAM and DSO assign random depths with large variance in a single initial frame, then rely on subsequent processing for convergence.
  • Random-depth initialization can require several added keyframes before a stable configuration is reached.
  • ORB SLAM estimates both a Fundamental matrix and a Homography, discarding failed initialization and restarting with another frame pair when tracking is poor.

3.3. Pose estimation: choices made

Pose-estimation modules predict a new camera pose, associate image measurements, and optimize geometric or photometric error, with systems differing in motion models and recovery strategies.

  • PTAM represents camera pose in a six-parameter minimal SE(3) form, reducing optimization parameters from twelve to six and accelerating optimization.
  • The generic pose-estimation process uses a previous pose and map to form a motion-model guess, restricts association to a search window, and minimizes an error vector.
  • PTAM refines a decaying constant-velocity prior using a Small Blurry Image and restricts tracking to higher pyramid levels during fast motion.
  • SVO estimates pose by minimizing photometric error through inverse-compositional image alignment, then applies motion-only, structure-only, and joint local bundle adjustment.
  • DT SLAM switches among full pose, Essential-matrix, and pure-rotation estimation according to the available matches and observed motion.
  • ORB SLAM combines a constant-velocity prior with optimization and triggers a wider feature search when matched features fall below a threshold.
  • DSO uses multiple motion hypotheses, including constant velocity, scaled velocity, no motion, and small random rotations, for front-end tracking.

3.4. Topological/Metric Map generation: choices made

Keyframe-based monocular SLAM systems use different map-generation strategies, ranging from triangulation and depth filtering to propagated or refined depth maps. Their topological and metric representations also vary in connectivity, landmark parameterization, and optimization of observations.

  • Map-generation strategies: PTAM establishes feature matches between a new keyframe and its nearest keyframe, then triangulates new landmarks while checking existing landmark projections along epipolar lines.Projected-landmark depth is used to constrain the matching process.
  • Map-generation strategies: SVO initializes inverse-depth seeds from high-scoring features and updates them through bounded epipolar searches as uncertainty decreases.Seeds that repeatedly fail to match can be removed during map management.
  • Map-generation strategies: Map generation combines two-view triangulation, depth filtering, inverse-depth estimation, and depth-map propagation or refinement.The surveyed systems differ in whether landmarks are represented directly in 3D, along camera rays, or through inverse depth.
  • Map-generation strategies: Other systems defer triangulation until sufficient parallax, refine depth on regular frames, or generate dense maps from sparse measurements and planar superpixels.DT SLAM uses visual-change criteria for keyframe insertion, while REMODE and DPPTAM support dense scene representations.
  • Topological and metric maps: ORB SLAM separates co-visibility and essential graphs: the former retains available connections, whereas the latter keeps the strongest two edges per keyframe.New landmarks are triangulated from ORB features observed in connected keyframes and filtered using depth, reprojection, and scale checks.
  • Map-generation strategies: DSO uses all frames for map building, maintains candidate points per keyframe, and performs windowed optimization over a fixed active set of keyframes and points.Older keyframes and candidate points are marginalized as new ones enter the active window.

3.5. BA/PGO/Map maintenance: choices made

Map maintenance in keyframe-based monocular SLAM uses local or global optimization, pose-graph methods, structure-only adjustment, and system-specific culling or outlier rejection. These choices trade map accuracy, computational cost, and operational duration.

  • Optimization strategies: KSLAM systems use local bundle adjustment, global bundle adjustment, pose-graph optimization, or structure-only bundle adjustment for different optimization scopes.LBA optimizes a local map subset, GBA the entire map, PGO selected poses, and SOBA landmark structure with fixed poses.
  • Optimization strategies: PTAM performs local and global bundle adjustment, but its cost becomes intractable as the keyframe map grows, restricting it to small workspaces.The limitation follows from optimization cost scaling with map size.
  • Map maintenance: SVO removes distant keyframes to retain real-time performance during prolonged operation over large distances.Its map-management policy maintains only a fixed number of keyframes.
  • Optimization strategies: DT SLAM continuously optimizes the entire map in the background through sparse global bundle adjustment.This runs in a separate thread from the main mapping process.
  • Map maintenance: ORB SLAM uses landmark and keyframe culling to reduce outliers and counter the effects of maintaining many keyframes.Landmarks must satisfy visibility and matching requirements before they are retained.
  • Map maintenance: DSO performs six Gauss-Newton iterations on combined photometric and geometric residuals over active keyframes, dropping new keyframes with large residuals.It also removes points that are indistinct, outside the search region, displaced by more than two pixels, or photometrically inconsistent.

3.6. Failure recovery: choices made

Failure recovery strategies attempt to relocalize lost frames using appearance, image alignment, pose-graph neighbors, or randomized pose hypotheses. The systems differ in whether recovery is global, local, or absent.

  • Recovery mechanisms: PTAM compares incoming-frame appearance representations with keyframes, estimates rotational change, and resumes tracking only after sufficient landmark matches.If matching remains insufficient, the tracker stays lost while acquiring another frame.
  • Recovery mechanisms: SVO aligns a lost frame with the closest keyframe and considers recovery converged when more than 30 features match correctly.Otherwise, it continues attempting relocalization with subsequent incoming frames.
  • Recovery mechanisms: LSD SLAM selects pose-graph keyframes with multiple neighbors and tests alignment against them, replacing candidates when the outlier-to-inlier ratio is unfavorable.Recovery therefore uses graph connectivity to search for a suitable reference keyframe.
  • Recovery mechanisms: ORB SLAM uses bag-of-words place recognition to query candidate keyframes for relocalization, leveraging shared visual vocabulary between observations of the same scene.The same place-recognition module supports both loop detection and failure recovery.
  • Recovery mechanisms: DSO lacks global failure recovery and instead retries photometric tracking from random rotations around the last successfully tracked pose.If coarse-level photometric minimization fails, tracking fails.

3.7. Loop closure: choices made

Loop-closure mechanisms differ in how they detect corresponding places and correct accumulated drift. They include sub-map matching, appearance-based search, and pose-graph optimization, while some systems omit global loop closure.

  • Loop-closure mechanisms: DT SLAM starts a new sub-map after tracking failure and searches for associations across sub-maps in a background loop-closure thread.Once enough associations are found, the sub-maps are merged through similarity-transform optimization.
  • Loop-closure mechanisms: LSD SLAM searches the ten nearest keyframes and uses FABMAP appearance modeling, then distributes loop-edge similarity error through pose-graph optimization.The procedure establishes both ends of a detected loop before optimizing the graph.
  • Loop-closure mechanisms: ORB SLAM uses global place recognition to generate loop hypotheses, validates them through landmark projections and matches, and updates its co-visibility and essential graphs.The queried keyframe and neighboring keyframes provide the candidate landmark associations.
  • Loop-closure mechanisms: DSO permanently discards marginalized keyframes and points, so it has no global localization method for failure recovery or loop-closure detection.Its marginalization policy prevents later reuse of those map elements.

4. Closed source systems

The closed-source systems survey highlights diverse strategies for dense reconstruction, robust tracking, environmental change, rotational motion, loop closure, and map representation. Because many systems are insufficiently documented, the discussion emphasizes concise summaries of their additive contributions.

  • Scope: Closed-source systems are surveyed chronologically, but limited published details prevent fully dissecting their implementations.The section therefore focuses on contributions beyond the open-source-system discussion rather than repeating complete system details.
  • Dense reconstruction: Dense reconstruction systems extend feature-based or direct monocular SLAM toward surface and dense-map representations.One hybrid system refines a feature-based triangulated surface with dense GPU processing, while DTAM performs online dense reconstruction using whole-image alignment.
  • Environmental change and relocalization: CD SLAM targets short- and long-term environmental changes and mixed indoor/outdoor scenes using descriptor updates, outlier weighting, nested loop closures, and scale-drift correction.Its keyframes are organized in an undirected, unweighted pose graph, with relocalization initialized from a matching map keyframe.
  • Motion and scene challenges: Other systems address specific motion and scene challenges, including occlusions and slowly varying dynamics, pure rotations, and small-scene dense reconstruction.Their strategies include parallelized SIFT with alternating KD-trees, local panorama maps, and per-frame dense depth estimation from neighboring keyframes.
  • Mapping and optimization: Several systems vary the map representation and optimization strategy, including hybrid topological/metric maps, local bundle adjustment, and dense-map pose-graph optimization over sim(3).Multi-Level Mapping uses quad-tree keyframe representations to densify low-texture regions, fill depth holes, regularize measurements, and align the map globally.

5. Discussion

The survey finds that monocular SLAM design involves environment-dependent trade-offs across data association, initialization, mapping, and global localization. It therefore evaluates component-level limitations rather than offering a single unbiased benchmark across systems.

  • Benchmarking: No public dataset supports an unbiased comparison across monocular SLAM systems because systems favor different operational conditions.Examples include preferences for high frame rates, planar indoor scenes, or repeated observations.
  • Direct methods: Direct methods are robust to poor texture and blur but remain vulnerable to illumination changes, large motions, and computationally intensive pixel-wise error calculation.Pyramidal alignment and high-frame-rate cameras are used to address motion constraints, while parallel processing and semi-dense filtering improve feasibility.
  • Feature-based methods: Feature-based methods tolerate lighting changes and wider baselines, but can fail under large scene changes, repetitive textures, motion-induced association errors, and textureless regions.Extraction is computationally expensive, creating a trade-off between feature choice, robustness, and real-time operation.
  • Initialization: Initialization methods remain vulnerable to low parallax, violated scene-structure assumptions, inappropriate motion, distant scenes, or slow translation.Random-depth approaches avoid some two-view degeneracies but require subsequent-frame processing and do not guarantee convergence.
  • Map generation: Triangulation-based map generation requires a significant baseline, so pure rotations can cause failure when scenes cannot later be re-observed with greater baseline.DT SLAM addresses this with 2D landmarks, submaps, and attempted cross-submap associations, but tracking resets make prior pose estimates obsolete after failure.
  • Global localization: Global localization methods can fail after large camera displacements and are sensitive to illumination changes, repetitive textures, and unrepresentative bag-of-words training environments.These limitations affect both re-localization and loop detection.

6. Conclusions

The review identifies core monocular SLAM building blocks and highlights unresolved challenges spanning robustness, mapping, and recovery. It also points toward hybrid integration of metric, topological, and semantic representations.

  • Core building blocks: The review covers data association, initialization, pose estimation, map generation, map maintenance, and global localization across monocular SLAM systems.It also discusses open-source systems including PTAM, SVO, DT SLAM, LSD SLAM, ORB SLAM, DPPTAM, and DSO.
  • Open challenges: Robust data association remains challenging under illumination changes, dynamic scenes, and occlusions.
  • Open challenges: Initialization remains challenging because robust methods should operate without an initial scene assumption.
  • Open challenges: Accurate pose estimation remains difficult under sudden motion, blur, noise, large depth variations, and moving objects.
  • Open challenges: Future mapping and maintenance should support dense representation in poorly textured regions and resilience to changing, dynamic environments.
  • Future directions: Failure recovery remains challenging when camera viewpoints change substantially, while semantic integration calls for hybrid metric, topological, and semantic fusion.
Loading 1607.00470v2…