Source-linked AI summary
Meaningful Maps With Object-Oriented Semantic Mapping
Niko Sünderhauf, Trung T. Pham, Yasir Latif, Michael Milford, Ian Reid
TL;DR
The paper addresses the gap between geometric and semantic mapping by building maps containing both geometric representations and semantically meaningful object-level entities. It combines RGB-D SLAM, deep-learning object detection, and unsupervised 3D segmentation to build object models on the fly. Indoor evaluations showed that the system correctly identified the majority of objects, with false negatives remaining a prominent failure mode.
Problem
Robots need maps that jointly convey geometric and semantic information, but prior mapping research often handled these properties separately or required known 3D object models.
Method
The system combines sparse feature-based RGB-D SLAM, image-based deep-learning object detection, unsupervised 3D segmentation, data association, and object model updates.
Results
The system correctly identified the majority of objects across indoor scenes, with only two false positive detections reported.
Takeaways & Limitations
Object-oriented maps maintain individual objects as key entities, build 3D models on the fly, and support semantic annotation without a-priori known 3D models.
Abstract
from arXiv · showhide
For intelligent robots to interact in meaningful ways with their environment, they must understand both the geometric and semantic properties of the scene surrounding them. The majority of research to date has addressed these mapping challenges separately, focusing on either geometric or semantic mapping. In this paper we address the problem of building environmental maps that include both semantically meaningful, object-level entities and point- or mesh-based geometrical representations. We simultaneously build geometric point cloud models of previously unseen instances of known object classes and create a map that contains these object models as central entities. Our system leverages sparse, feature-based RGB-D SLAM, image-based deep-learning object detection and 3D unsupervised segmentation.
I. INTRODUCTION
The paper targets maps that jointly represent geometry, semantics, and separate object instances without requiring pre-existing 3D models. Its object-oriented system combines mature SLAM and deep-learning components to build richly annotated maps.
- Existing mapping research typically prioritizes geometric accuracy or limited semantic classes, while neglecting separate object instances or requiring known 3D models.
- The proposed maps represent both where an entity is and what it is, addressing limitations of previous approaches.
- Object instances are central map entities carrying class labels, confidence values, and geometric information independently from non-object map parts.This organization supports reasoning that points belonging to one object move together during manipulation.
- The system combines bounding box-based detection with unsupervised 3D segmentation to support instance-level semantic mapping.
- The approach builds 3D object models on the fly for previously unseen instances and avoids requiring precise a-priori models for all class variations.
II. RELATED WORK
Geometric maps support mapping and localization but do not identify mapped entities or indicate possible actions, motivating semantic representations for interaction.
- Geometric SLAM maps provide points, planes, and surfaces, but these entities do not reveal their identities or possible actions.
- Semantic representation is needed for robots to interact with their environment, with label granularity depending on the task.
A. Semantic Mapping
Semantic mapping attaches meaning to mapped entities, while prior approaches differ in reconstruction, segmentation, object templates, and detection strategies. Instance-level mapping remains important for maintaining separate objects over time.
- A. Semantic Mapping: Semantic mapping attaches categories, identities, or actions to mapped entities while using SLAM to reason about sensor motion and position.
- A. Semantic Mapping: Earlier methods reconstructed maps and then segmented them into semantic concepts, including dense 3D reconstruction followed by point-wise CRF labeling.
- A. Semantic Mapping: Other systems performed online scene reconstruction and segmentation, including functional-space labeling, multimodal semantic mapping, and distributed object detection.
- A. Semantic Mapping: Object-template methods inserted recognized known objects into maps, unlike approaches that build models for previously unseen instances.
- B. Object Detection and Semantic Segmentation: Proposal-based detectors generate bounding-box proposals and classify them over object classes, whereas semantic segmentation produces dense pixel-wise labels without necessarily distinguishing instances.
- B. Object Detection and Semantic Segmentation: The illustrated pipeline combines SSD proposals, unsupervised 3D segmentation, and a map of objects carrying semantic labels, confidence, and geometry.
- B. Object Detection and Semantic Segmentation: Current instance-level RGB segmentation methods were considered insufficiently accurate and fast for the paper’s online framework.
III. OBJECT-ORIENTED SEMANTIC MAPPING
This section introduces the main components of the semantic mapping system and points to figures that depict its processing stages and information flow.
- The system’s main components are outlined here, with Fig. 2 showing key stages and Fig. 3 showing information flow between components.
A. SLAM
The system uses ORB-SLAM2 to provide camera poses and sparse geometric mapping, while SSD detects object instances in keyframes for semantic mapping. This one-way flow makes the system semantic mapping rather than semantic SLAM.
- A. SLAM: ORB-SLAM2 tracks and maps using RGB-D information, supplying camera poses and map structure for semantic labelling.
- A. SLAM: Information flows only from SLAM to semantic mapping: camera poses and the map support coherent semantic labelling, without semantics helping SLAM.
- A. SLAM: Instance-level segmentation methods were considered too slow for the framework, whereas proposal-based detectors provide real-time performance.DeepMask is reported to take about 1.6 s per image.
- A. SLAM: SSD generates bounding-box object proposals with class labels and confidence scores for every keyframe.
- A. SLAM: The COCO-trained SSD network recognizes 80 classes, and its 500×500 variant takes 86 ms per forward pass on a TitanX GPU.
C. 3D Segmentation
The system refines image detections into object-level 3D segments using depth-based unsupervised segmentation. It partitions a weighted supervoxel adjacency graph to obtain accurate object boundaries.
- C. 3D Segmentation: Depth information is used to produce precise object boundaries because image-based bounding boxes are often poorly fitted to objects.
- C. 3D Segmentation: The method over-segments the depth image into supervoxels, builds an adjacency graph, and partitions it into connected components.
- C. 3D Segmentation: Graph edges receive weights based on supporting-plane relations and whether neighboring supervoxels are convex or concave.
- C. 3D Segmentation: The 3D segmentation typically takes 175 ms when executed.
D. Data Association
Data association determines whether each segmented detection corresponds to an existing map object or requires a new object. It first narrows candidates by centroid distance, then applies point-level nearest-neighbor matching.
- D. Data Association: The association pipeline decides whether a detected object is already represented in the map or must be added as a new map object.
- D. Data Association: The system uses an ICP-like matching score to associate detections with existing objects or create new map objects, extending matched models with new 3D structure.
- D. Data Association: Candidate landmarks are selected using Euclidean distance between the detected and landmark point-cloud centroids.
- D. Data Association: A detection is associated with an existing landmark when at least 50% of its 3D points lie within 2 cm of associated landmark points.
E. Object Model Update
Each map object stores segmented colored point clouds, observation-pose references, and accumulated detector confidences. New observations update the object’s geometry and class evidence, while stored clouds support model reconstruction after trajectory changes.
- E. Object Model Update: Every object maintains segmented colored point clouds, pose-graph indices for observation poses, and accumulated per-class detector confidences.
- E. Object Model Update: Class labels are selected by the maximum accumulated class score, with final confidence computed as the maximum score divided by the number of observations.
- E. Object Model Update: Stored point-cloud segments allow object models to be rebuilt after SLAM updates its trajectory estimate, such as following loop closure.Downsampled clouds use 5 mm spatial resolution and also support data association.
F. Map Generation
The system maintains an implicit map from keyframe point clouds and segmented object point clouds linked to the SLAM pose graph. It can regenerate explicit maps at application-dependent resolutions and update object models whenever objects are observed.
- The implicit map stores keyframe 3D point clouds, segmented object point clouds, and pointers into ORB-SLAM’s pose graph.
- Explicit maps are generated by projecting stored points using the current best estimates of their associated poses.This supports uses such as path planning and grasp point selection.
- The generated map uses 1 cm resolution for non-objects and 0.5 cm resolution for objects.The sparse point cloud can be converted into a dense mesh with appropriate algorithms.
- Object point-cloud models are updated whenever the corresponding object is observed, keeping data association current.
IV. EVALUATION AND LESSONS LEARNED
Evaluation across indoor environments shows that the system maps most objects, but false negatives remain the dominant failure mode. Errors arise from instance-segmentation ambiguity, corrupted or noisy depth, detector training-set mismatch, and low camera resolution.
- The evaluation spans indoor scenes from a desk and office to a kitchen and complete lab, comparing mapped instances with ground-truth object counts.Results are summarized in Table I and illustrated in Figures 5–6.
- The system correctly identifies the majority of objects and produces only two false-positive detections across the evaluated scenes.One false positive mistook a window corner for a monitor; another duplicated a monitor because of depth-perception errors.
- False negatives are the more prominent failures, caused either by missed object detections or by discarding detections during later pipeline processing.
- Failing Instance Segmentation: Five lab-sequence monitors were missed because near-parallel dual-monitor setups could not always be separated into distinct objects.Spatial-structure and object-size reasoning could help distinguish two monitors from one.
- Depth and Segmentation: Noisy depth complicated segmentation of flat keyboards, while a reflecting iMac disrupted depth perception and caused segmentation and data-association failures.The keyboard failures affected 8 of 30 keyboards across all sequences.
- Training Set Discrepancies: Books were difficult for SSD because their orientations and appearances differed from the COCO training data, whose 80 classes poorly matched the tested indoor scenes.Only 10 COCO classes appeared in the evaluated environments.
- Low-Resolution Cameras: Small objects are especially challenging with the low-resolution PrimeSense RGB-D sensor because they are less likely to be detected and provide few 3D points for reliable processing.Higher-resolution sensors are identified as the practical remedy.
V. CONCLUSIONS AND FUTURE WORK
The paper concludes that combining SLAM, object detection, instance segmentation, data association, and model updates yields object-centered semantic maps built from on-the-fly 3D models. Future work targets semantic landmarks for SLAM, denser object models, and applications including planning and behavior generation.
- The system combines SLAM, object detection, instance-level segmentation, data association, and model updates while maintaining individual objects as map entities.
- It builds 3D object models on the fly without requiring a-priori models and was evaluated by semantically annotating indoor workplace scenes for automated inventory management.
- Future work will investigate detected objects as semantic landmarks for improving SLAM accuracy and integrating object detections into factor-graph data fusion.
- The authors also plan to replace point-cloud object representations with dense surface models and study semantic-map use in planning, path planning, and behavior generation.