PaperlayerView exact revision on arXiv

Source-linked AI summary

Real-time Monocular Object SLAM

Dorian Gálvez-López, Marta Salas, Juan D. Tardós, J. M. M. Montiel

arXiv:1504.02398v1cs.ROcs.CV

TL;DR

Monocular SLAM maps provide geometry but describe scenes poorly, and object recognition had largely been treated separately from visual SLAM. This paper integrates object recognition with monocular mapping, using object constraints to estimate real scale and database cues to guide detections. It reports improved recognition performance and more accurate maps than state-of-the-art RGB-D systems.

  • Problem

    Object recognition for inserting objects into SLAM maps had largely remained separate from visual SLAM, although geometric maps describe observed scenes poorly.

    Source: §1 · p. 1

  • Method

    The system accumulates object observations, triangulates object features into map anchors, and jointly optimizes camera poses, points, objects, and scale while recognizing objects with bags of binary words.

    Source: §1 · p. 1

  • Results

    The system reports more accurate maps than PTAM and RGB-D SLAM, real-scale estimation, and improved recognition performance in datasets with many objects, repeated instances, occlusion, and clutter.

    Source: §1 · p. 2, §7 · p. 15

  • Takeaways & Limitations

    The authors suggest that embedding other recognition methods in SLAM may also improve recognition when they exploit accumulated object observations.

    Source: §7 · p. 15

    If the first object inserted is based on wrong observations, its incorrect scale estimate can produce a missized map; this case was rare and absent from the experiments.

    Source: §7 · p. 15

Abstract

from arXiv · show

We present a real-time object-based SLAM system that leverages the largest object database to date. Our approach comprises two main components: 1) a monocular SLAM algorithm that exploits object rigidity constraints to improve the map and find its real scale, and 2) a novel object recognition algorithm based on bags of binary words, which provides live detections with a database of 500 3D objects. The two components work together and benefit each other: the SLAM algorithm accumulates information from the observations of the objects, anchors object features to especial map landmarks and sets constrains on the optimization. At the same time, objects partially or fully located within the map are used as a prior to guide the recognition algorithm, achieving higher recall. We evaluate our proposal on five real environments showing improvements on the accuracy of the map and efficiency with respect to other state-of-the-art techniques.

1. Introduction

The introduction presents a monocular SLAM system that integrates object recognition with mapping to estimate object poses and recover real scene scale. It reports more accurate maps than PTAM and RGB-D SLAM while retaining real-time performance.

1. Introduction

  • Visual SLAM maps provide geometry for camera localization but describe scenes poorly, while object recognition had largely remained apart from visual SLAM.Source: §1 · p. 1
  • The system couples object recognition with monocular SLAM, accumulating observations over time to estimate robust object poses and keep objects located in the scene.Source: §1 · p. 1
  • Triangulated object features become map anchor points that constrain bundle adjustment; real-scale object models also provide observations of map scale.Source: §1 · p. 1
  • The back end jointly optimizes camera poses, map points, anchors, objects, and scale, producing a map at real scale.Source: §1 · p. 1
  • The recognition method uses ORB features and bags of binary words to recognize 3D objects in databases of up to 500 models in real time.Source: §1 · p. 1
  • The system recovers real scene scale and reports more accurate maps than PTAM and RGB-D SLAM while retaining real-time performance.Tracking takes 7.6 ms, and recognition takes around 200 ms per image.Source: §1 · p. 2

2. Related work

Prior work combines object recognition and mapping in several ways, but differs in whether objects enter map optimization, how recognition and reconstruction are integrated, and whether systems scale to large object databases. The paper positions its approach against both object-SLAM systems and feature-recognition methods.

2. Related work

  • Castle et al. added recognized objects as augmented-reality overlays rather than optimization variables, using 37 planar SIFT-described pictures; their method was limited to planar objects.Source: §2 · p. 2
  • Bao et al. jointly optimize cameras, points, and objects in SSfM, while this system integrates recognition with incremental SLAM and estimates 6DoF poses; Fioraio et al. also optimize object poses with the map.Source: §2 · p. 2
  • Salas-Moreno et al. combine RGB-D mapping and recognition but show results with four objects, whereas this work reports high-frequency CPU execution with up to 500 objects from a monocular camera.Source: §2 · p. 2
  • Earlier object-recognition methods use visual vocabularies or direct feature matching, with 3D models enabling pose recovery from 2D-to-3D correspondences.Source: §2 · p. 2
  • The proposed direct index balances correspondence count against execution time by matching features at a coarse tree level, unlike finest-word-only or all-level matching approaches.Source: §2 · p. 3
  • Binary ORB features and a vocabulary independent of object count support databases up to 500 objects, with recognition around 200 ms per image.Source: §2 · p. 3

3. System overview

The system tracks camera motion and builds a map while running object recognition in parallel. Consistent object observations are accumulated before triangulation, allowing object anchors to constrain optimization and estimate map scale.

3. System overview

  • The system uses PTAM tracking and parallel recognition and insertion processes, with a redesigned back end jointly optimizing keyframes, map points, objects, and scale.Source: §3 · p. 3
  • SLAM processes every video frame and adds keyframes when distinctive geometry is available, while object recognition searches as many frames as possible.Source: §3 · pp. 3–4
  • Objects enter the map only after consistent observations accumulate; observations without parallax or distinctive geometry are discarded before triangulation.Source: §3 · p. 4
  • Triangulated object points become anchor points, and their observing cameras become semantic keyframes; each triangulation estimates map scale for global optimization.Source: §3 · p. 4

4. Object-aware SLAM

Object observations are accumulated across views and triangulated into map anchors. Object geometry then constrains SLAM optimization, allowing the system to estimate map scale alongside camera, point, and object poses.

4.1. Object insertion within the map

  • Object observations are accumulated until at least five points are seen from two positions with at least 3 degrees of parallax and good geometric conditioning.The points are triangulated as anchor points, and frames offering parallax become semantic keyframes; observations without parallax or new points are disregarded.Source: §4.1 · p. 4
  • Anchor points are retained by map maintenance, updated from new object observations, and propagated across keyframes by patch correlation.The patches are warped to compensate for scale, rotation, and foreshortening.Source: §4.1 · p. 4
  • After an object's first triangulation, its map-frame pose prior is composed from the SLAM camera pose and the detected object pose relative to that camera.The pose computation uses the object-derived scale estimate or the map's previously refined scale estimate.Source: §4.1 · p. 5

4.2. Object SLAM optimization

  • Each anchor point pairs an object-model point with its map location, adding an object-alignment constraint to bundle adjustment.Known-size object points make the shared map scale observable.Source: §4.2 · p. 5
  • Bundle adjustment jointly estimates map scale, camera poses, map points, and object locations by combining reprojection and object-alignment errors.The shared scale becomes observable when map points correspond to known-size objects.Source: §4.2 · pp. 5–6

5. 3D Object recognition with large databases

The recognition pipeline combines prior map information and database retrieval to generate object candidates, then verifies their poses from 2D-to-3D correspondences. Its model and indexing design supports recognition from multiple viewpoints and scales.

5. 3D Object recognition with large databases

  • Online recognition obtains candidates through database queries or prior object locations, verifies them by estimating rigid poses, and returns observations for association with map instances.The process runs in real time, with detected observations subsequently associated to object instances using the current camera pose.Source: §5 · p. 6

5.1. Object models

  • Each object model combines a 3D point cloud with ORB descriptors and an appearance bag of words, representing surface appearance across views and scales.The point cloud is reconstructed from training images, retaining points visible in at least three images; multiple descriptors per point accommodate viewpoint and scale variation.Source: §5.1 · p. 6

5.2. Object model database

  • The database uses a binary-tree visual vocabulary with inverted and direct indices to retrieve shared words and filter likely feature correspondences.The inverted index supports fast access to common words; the direct index filters feature matches during pose verification.Source: §5.2 · pp. 6–7

5.3. Prior knowledge to obtain object candidates

  • Previously observed objects provide expected poses for projecting model points into the current image; successful 2D-to-3D pose estimation produces new observations.Expected poses come from mapped objects or accumulated observations; without estimated map scale, the recent camera-to-object transformation is used as the expected pose.Source: §5.3 · p. 7

5.4. General retrieval of object candidates

  • Region-based retrieval ranked the correct model first with 6 of 9 correspondences as inliers, enabling verification; whole-image retrieval ranked it seventh and failed to verify.The whole-image query produced 24 correspondences, 18 incorrect, and did not verify within 100 iterations; the region query verified in 34 iterations.Source: §5.4 · p. 8, Figure 6 · p. 8

5.5. Object verification and pose estimation

  • DISAC samples correspondences with probability inversely proportional to descriptor Hamming distance, relaxing PROSAC’s fixed ordering to handle low-distance mismatches under a 50-iteration cap.After finding a pose, projected model points guide additional ORB matches in 7 × 7 patches, and the refined pose is scored using reprojection error and inlier count.Source: §5.5 · p. 9

6. Experimental evaluation

The evaluation tests object retrieval and object-aware monocular SLAM across five datasets, with object databases ranging from 7 to 500 models. Results include real-scale, richer maps, real-time processing, and successful operation with clutter, repeated objects, occlusion, and tracking loss.

6.1. Desktop dataset

  • KL divergence gave the best retrieval performance across database sizes, and performance rose through the top 10 candidates before stalling.The top-10 cutoff was selected to limit verification time.Source: §6.1 · p. 10, Figure 11 · p. 10
  • All six desktop objects were located in the map with no false-positive detections.Figure 13 shows the resulting map.Source: §6.1 · p. 10, Figure 13 · p. 10
  • SLAM tracking averaged 3.3 ms and object recognition 138 ms per image; another result reports 7.6 ms tracking and 220 ms recognition per image with a 500-object database.Tracking and recognition ran in parallel, allowing the SLAM map to be created in real time independently of recognition time.Source: §6.1 · p. 11, §6.2 · p. 13

6.2. RGB-D SLAM Dataset

  • Two posters were not recognized in the RGB-D sequence: the smallest yielded no detections, while the middle poster was barely in focus near the trajectory center.The RGB-D dataset supplied camera-pose ground truth; the sequence used different data for object-model training and evaluation.Source: §6.2 · p. 12
  • The reported Object SLAM system estimated the real scene scale and produced maps described as more accurate and richer in information, including both 3D objects and points.The average keyframe-pose error was 3.4 cm in translation and 1.4 degrees in rotation.Source: §6.2 · pp. 12–13

6.3. Aroa’s Room and Snack datasets

  • Across room and snack tests, maps remained consistent with clutter, repeated models, occlusion, and tracking loss; unsupported false instances were not triangulated.In the cluttered snack sequence, early detections provided priors that helped detect all objects in later frames.Source: §6.3 · pp. 13–14

7. Conclusions

The conclusions present object-aware monocular SLAM and 3D recognition for databases of up to 500 models. They connect object-derived geometric constraints to real-scale mapping, while identifying incorrect first-object observations as an unaddressed risk.

7. Conclusions

  • The system combines monocular SLAM with 3D object recognition for databases of up to 500 object models.The authors report improved recognition performance in datasets with many different objects, repeated instances, occlusion, and clutter.Source: §7 · p. 15
  • Object anchor points add geometric constraints to back-end optimization and enable estimation of real map scale.The authors report more accurate maps than state-of-the-art algorithms using RGB-D data.Source: §7 · p. 15
  • A wrongly observed first object can produce an incorrect scale estimate and a missized map; the authors say safety steps made this rare and it did not occur in their experiments.They suggest checking scale-estimate variance across object triangulations or using a rough initial estimate from odometry or an IMU.Source: §7 · p. 15
  • The authors suggest semantic map information as a basis for future reasoning about object mobility and dynamic maps.They describe this as future work.Source: §7 · p. 15

Appendix A. Efficient KL-divergence computation

The appendix describes an efficient comparison score derived from Kullback-Leibler divergence. It regularizes zero entries and removes a query-only term when comparing the query with object models.

Appendix A. Efficient KL-divergence computation

  • The appendix defines Kullback-Leibler divergence for normalized vectors v and w.Both vectors are specified to have L1 norm 1.Source: Appendix A · p. 15
  • When w_i is zero, the computation substitutes a small positive constant ε to avoid undetermined values.The text specifies ε → 0+.Source: Appendix A · p. 15
  • For comparing a query vector v with object-model vectors w, the score removes the divergence addend that depends only on v.The passage identifies v as the query vector and w as object models.Source: Appendix A · p. 15
Loading 1504.02398v1…