Source-linked AI summary
Volumetric Instance-Aware Semantic Mapping and 3D Object Discovery
Margarita Grinvald, Fadri Furrer, Tonci Novkovic, Jen Jen Chung, Cesar Cadena, Roland Siegwart, Juan Nieto
TL;DR
Robots need object-centric maps that support perception and planning in complex open-set environments, including objects outside fixed training classes. The paper combines geometric and semantic segmentation with cross-frame data association to incrementally construct volumetric instance-aware maps. Evaluations show dataset competitiveness, unseen-object discovery, and qualitative online mapping on a robotic platform.
Problem
Robots require instance-level scene understanding for navigation and interaction, but real-world environments include objects from previously unseen categories.
Method
The framework combines geometric-semantic segmentation, data association across frames, and incremental fusion into a global volumetric map.
Results
The method builds instance-aware semantic maps while discovering objects of unseen categories and demonstrating online robotic mapping qualitatively.
Takeaways & Limitations
The resulting object-level volumetric maps are expected to benefit navigation and manipulation planning by representing recognized and novel objects with geometric information.
Abstract
from arXiv · showhide
To autonomously navigate and plan interactions in real-world environments, robots require the ability to robustly perceive and map complex, unstructured surrounding scenes. Besides building an internal representation of the observed scene geometry, the key insight toward a truly functional understanding of the environment is the usage of higher-level entities during mapping, such as individual object instances. We propose an approach to incrementally build volumetric object-centric maps during online scanning with a localized RGB-D camera. First, a per-frame segmentation scheme combines an unsupervised geometric approach with instance-aware semantic object predictions. This allows us to detect and segment elements both from the set of known classes and from other, previously unseen categories. Next, a data association step tracks the predicted instances across the different frames. Finally, a map integration strategy fuses information about their 3D shape, location, and, if available, semantic class into a global volume. Evaluation on a publicly available dataset shows that the proposed approach for building instance-level semantic maps is competitive with state-of-the-art methods, while additionally able to discover objects of unseen categories. The system is further evaluated within a real-world robotic mapping setup, for which qualitative results highlight the online nature of the method.
I. INTRODUCTION
The paper addresses instance-level mapping for robots operating in open-set, unstructured environments. It combines geometric and semantic perception, tracks instances across frames, and evaluates online mapping on dataset and robotic-platform settings.
- Robots need instance-level discovery, segmentation, tracking, and reconstruction to acquire task-relevant 3D object models for interaction.
- The approach incrementally builds volumetric maps containing object-instance location, 3D shape, and available semantic categories.
- Evaluation uses office sequences from SceneNN and an online robotic mapping setup to demonstrate the framework's mapping capabilities.
- The framework combines geometric segmentation with semantic predictions to extend object detection to previously unseen categories.
- A data association strategy tracks and matches instance predictions across multiple frames.
A. Object detection and segmentation
Existing methods provide strong image-level or 3D semantic segmentation but remain limited by fixed training classes, missing instance structure, or insufficient scene representation. The paper motivates combining open-set discovery with dense volumetric object maps.
- Learning-based instance segmentation requires extensive annotated masks and recognizes only the fixed classes represented during training.
- Image-based open-set methods can segment novel objects but do not alone provide comprehensive 3D scene representations for planning.
- Conventional semantic mapping does not disambiguate individual instances or represent their geometry and relative placement.
- Object-oriented methods may require known geometric templates, while geometry-only methods lack semantic information.
- Surfel-based maps omit explicit free-space distinctions and make visibility and collision reasoning harder for robotic planning.
- The proposed TSDF representation distinguishes observed empty space from unknown space while retaining shape and pose for recognized and unknown objects.
III. METHOD
The method processes each incoming RGB-D frame through geometric segmentation, semantic instance refinement, data association, and global map integration.
- Each RGB-D frame undergoes geometric segmentation, semantic instance-aware refinement, data association, and map integration.
- Geometric segmentation yields contours describing physical boundaries, while Mask R-CNN detects instances and predicts semantically annotated masks.
- The system associates current-frame segments with persistent map labels before integrating the resulting information into the global volume.
A. Geometric segmentation
The frame-level pipeline combines convexity-based depth segmentation with RGB instance masks, using overlap to assign semantic labels and merge over-segmented object parts.
- Geometric segmentation: Depth frames are decomposed into convex 3D segments using surface normals, concavity boundaries, and strong depth discontinuities.
- Semantic refinement: Mask R-CNN detects and classifies individual RGB instances and predicts a binary semantic mask for each.
- Semantic refinement: The pipeline computes normalized 2D overlaps between geometric regions and predicted masks to associate depth segments with instances.
- Semantic refinement: Segments exceeding the overlap threshold receive the matched instance label and category, allowing multiple segments to represent one non-convex articulated object.
- Semantic refinement: Unmatched segments retain a geometric-only label, supporting discovery of elements without a corresponding semantic mask.
C. Data association
Data association resolves the lack of temporal correspondence by assigning persistent labels to geometric segments and object instances across mapping frames.
- Frame-wise segmentation lacks associations between predicted segments in consecutive frames, and mask indices are not persistent across frames.
- Persistent geometric and object-instance labels track corresponding segments and instances throughout the mapping session.
- 3D overlap with visible global-map segments determines geometric correspondences, while unmatched segments receive new persistent labels.The overlap threshold is τπ = 20, preventing poorly overlapping labels from propagating.
- Pairwise counts between persistent segment and instance labels map current-frame instances to existing persistent instances or create new ones.The procedure prevents multiple current labels from mapping to one persistent label.
- After association, each current segment carries persistent segment, object-instance, and predicted semantic-category labels.
D. Map integration
Map integration fuses associated 3D segments into a global TSDF volume while preserving object identity and semantic information when available.
- The framework extends Voxblox’s TSDF-based dense mapping to encode object segmentation alongside volumetric geometry.
- Global-map segments are defined by voxels carrying persistent labels and may additionally store persistent instance labels and semantic classes.Instance and class assignments are selected using the highest pairwise counts for each segment.
IV. EXPERIMENTS
Experiments evaluate the framework on SceneNN and in an online robotic mapping setup, demonstrating recognized-object mapping alongside discovery of previously unseen objects.
- The framework runs on 640x480 RGB-D input using Mask R-CNN GPU inference and CPU-based remaining stages.
- The reconstructed object inventory includes recognized chairs, couches, and tables together with unlabeled objects such as jackets, fans, boxes, and appliances.
- The SceneNN evaluation compares the proposed dense object-level semantic mapping against prior work on real-world indoor scans.
- The online robotic experiment operates at approximately 1 Hz and qualitatively validates online mapping in real-world open-set conditions.
A. Instance-aware semantic segmentation
The experiments assess segmentation accuracy, discovered object models, and execution time, while comparing SceneNN results with a prior 3D instance-segmentation approach.
- SceneNN evaluation computes per-class AP at IoU 0.5 across 10 sequences and compares nine-category mAP with Pham et al.’s results.
- Figure 3 shows densely reconstructed recognized instances and newly discovered scene elements across the evaluated sequences.
- The system runs at approximately 1 Hz on 640x480 input.
- Table II reports execution times averaged over 10 sequences at 640x480 input, with Mask R-CNN on GPU and other stages on CPU.The table uses 1 cm voxels and notes that starred components can run in parallel.
B. Online reconstruction and object mapping
The system performs online object-centric mapping from RGB-D streams on a mobile robotic platform, producing dense volumetric reconstructions with recognized and novel objects. The resulting map also provides free-space information, while reconstruction quality is affected by accumulated localization errors.
- Online robotic setup: A collaborative YuMi robot on a Ridgeback mobile base evaluated the system in a real-world online mapping scenario.The platform used visual-inertial sensing for online localization and RGB-D cameras for mapping.
- Online robotic setup: A 5-minute scan followed an office-floor trajectory, recording RGB-D streams at 1 Hz to emulate real-time onboard operation.The manually steered trajectory exceeded 80 m, according to the resulting reconstruction description.
- Mapping results: The final map densely represented observed surface geometry and included recognized known-class objects alongside novel previously unseen object-like elements.Only a subset of incoming RGB-D frames was integrated into the map volume.
- Mapping results: At 2 cm voxel resolution, the complete map used 605 MB of memory, comparable to bare Voxblox, while also providing free-space information.The free-space representation is relevant to safe navigation and interaction planning.
- Mapping limitations: Accumulated pose-estimation errors reached 0.5 m and affected reconstruction quality, while the impact of inaccurate localization was left for future work.The mapping system assumes camera localization is given.
V. CONCLUSIONS
The paper presents online volumetric instance-aware semantic mapping that combines geometric and semantic cues to discover and map known and novel objects. Real-world experiments support its online operation, but runtime optimization and unified RGB-depth fusion remain open directions.
- V. CONCLUSIONS: The framework incrementally builds volumetric instance-aware semantic maps from RGB-D data by jointly reasoning over geometric and semantic cues.It infers category information for recognized elements and discovers novel objects without prior knowledge of their exact appearance.
- V. CONCLUSIONS: Partial segmentation information is incrementally fused into a global map containing object-level semantic annotations.The resulting maps are expected to benefit navigation and manipulation planning.
- V. CONCLUSIONS: Real-world experiments validate the online nature of the incremental framework.The conclusion identifies this as an experimental validation rather than a quantitative benchmark result.
- V. CONCLUSIONS: Further work must optimize individual component runtimes for real-time capability and investigate unified RGB-depth fusion for detection, discovery, and segmentation.These are stated future research directions.