Source-linked AI summary
Fusion++: Volumetric Object-Level SLAM
John McCormac, Ronald Clark, Michael Bloesch, Andrew J. Davison, Stefan Leutenegger
TL;DR
The paper addresses object-oriented indoor scene understanding and 3D mapping with an online RGB-D SLAM system that reconstructs and persistently maps object instances. It reports high-quality loop-closed object maps, 4–8Hz operation excluding relocalisation and graph optimisation, and identified limitations from tracking accuracy and spurious detections.
Problem
Indoor scene understanding and 3D mapping need map representations suited to important objects for robotic planning, interaction, and human-robot communication.
Method
An online object-oriented SLAM system uses Mask R-CNN detections, variable-resolution per-object TSDF reconstruction, 3D foreground masks, and a persistent object pose graph.
Results
The system produces high-quality object reconstructions within globally consistent loop-closed maps and operates at approximately 4–8Hz excluding relocalisation and graph optimisation.
Takeaways & Limitations
The resulting map focuses on important object elements while supporting consistent instance mapping and classification of numerous previously unknown objects in cluttered indoor scenes.
Takeaways & Limitations
The approach does not achieve state-of-the-art RGB-D SLAM benchmark performance, and spurious detections can accumulate into cluttered partial reconstructions over time.
Abstract
from arXiv · showhide
We propose an online object-level SLAM system which builds a persistent and accurate 3D graph map of arbitrary reconstructed objects. As an RGB-D camera browses a cluttered indoor scene, Mask-RCNN instance segmentations are used to initialise compact per-object Truncated Signed Distance Function (TSDF) reconstructions with object size-dependent resolutions and a novel 3D foreground mask. Reconstructed objects are stored in an optimisable 6DoF pose graph which is our only persistent map representation. Objects are incrementally refined via depth fusion, and are used for tracking, relocalisation and loop closure detection. Loop closures cause adjustments in the relative pose estimates of object instances, but no intra-object warping. Each object also carries semantic information which is refined over time and an existence probability to account for spurious instance predictions. We demonstrate our approach on a hand-held RGB-D sequence from a cluttered office scene with a large number and variety of object instances, highlighting how the system closes loops and makes good use of existing objects on repeated loops. We quantitatively evaluate the trajectory error of our system against a baseline approach on the RGB-D SLAM benchmark, and qualitatively compare reconstruction quality of discovered objects on the YCB video dataset. Performance evaluation shows our approach is highly memory efficient and runs online at 4-8Hz (excluding relocalisation) despite not being optimised at the software level.
1. Introduction
The paper presents an online object-oriented RGB-D SLAM system that reconstructs and maps object instances rather than maintaining a dense scene representation. Its object-centric design supports loop-closed mapping, efficient scaling, and online operation in cluttered indoor scenes.
- Motivation: Object-oriented mapping groups dense geometry into labelled object instances, supporting scene understanding, planning, and interaction.The paper contrasts this representation with independently labelled surfels or points.
- System contributions: The system uses Mask R-CNN detections, variable-resolution per-object TSDFs, and 3D voxel foreground masks for online reconstruction.Missing detections are handled with an existence probability.
- System contributions: The persistent map contains only reconstructed object instances, combining rigid TSDF quality with pose-graph flexibility without intra-TSDF deformation.Separate volumes support object-size-dependent resolution and avoid storing large free-space regions.
- Evaluation: The evaluation targets repeated loops in a cluttered office scene containing many varied objects, including trajectory-error comparison against a baseline.The sequence is designed to test occasional poorly constrained ICP tracking and scaling with many objects.
- Results: The system demonstrates high-quality object reconstruction within globally consistent loop-closed object SLAM maps.This is presented as a primary system result.
2. Related work
Related systems differ in their representations, prior knowledge, scene assumptions, and integration of object discovery with SLAM. Fusion++ emphasizes online discovery, TSDF object landmarks, and tight integration into a pose-graph SLAM system.
- Prior-model approaches: Earlier object-oriented systems required complete, detailed object models known and preprocessed offline before operation.SLAM++ and related approaches tracked learned object models rather than discovering arbitrary unknown shapes online.
- Object discovery: Choudhary et al. used discovered objects as pose-graph landmarks for loop closure, but represented them with point-cloud centroids rather than TSDF volumes and full 6DoF constraints.Their object discovery used unsupervised segmentation rather than a trained object detector.
- Object discovery: Change-detection methods require scene changes before detecting objects, whereas this system is designed for online use without that requirement.The paper describes these methods as complementary because they can provide supervisory signals and database filtering.
- RGB-only methods: RGB-only approaches aggregate visual detection evidence or group landmarks by rigid transformations, rather than reconstructing RGB-D object instances as TSDF landmarks.The cited systems use ORB-SLAM or ORB descriptors for recognition and grouping.
- Closely related systems: Sünderhauf et al. used a separate ORB-SLAM2 system and point-cloud segments, while Fusion++ tightly integrates instances into SLAM and fuses them into separate TSDF volumes.Both approaches use CNN-based detections, but their reconstruction and SLAM integration differ.
- Dynamic-scene methods: Rünz and Agapito reconstruct and track moving instances with ElasticFusion for objects and background, whereas Fusion++ targets high-quality pose-graph landmarks and assumes static objects.The paper notes the two approaches could potentially be combined.
3. Method
Fusion++ builds an object-level online SLAM map from separate, optimisable TSDF volumes, using RGB-D tracking and Mask R-CNN detections to initialise and refine object instances. The method adapts volume resolution and foreground estimation to object geometry while supporting association, rendering, and existence-aware instance management.
- Pipeline: The pipeline tracks RGB-D input with a coarse background TSDF while a parallel Mask R-CNN thread filters detections and matches them to mapped objects.Unmatched detections initialise new TSDF object instances; relocalisation and graph optimisation occur when motion is sufficient or tracking appears lost.
- Object representation: Each mapped object occupies a separate cubic TSDF volume with its own SE(3) pose, making object instances the persistent map representation.Separate volumes support rigid reconstruction and pose-graph optimisation without intra-TSDF deformation.
- Object representation: Object volumes use robust 10th–90th percentile bounds and padding, then allocate fixed 64-voxel resolution so smaller objects retain finer detail than larger ones.Matched volumes can be resized as detections reveal additional object areas while preserving voxel size and increasing resolution where needed.
- Object representation: New instances require a camera distance below 5m and axis-aligned volume IoU below 0.5 with existing map volumes.These conditions constrain initialisation to nearby, non-overlapping object hypotheses.
- Fusion and rendering: Depth measurements are fused into visible TSDF volumes only when tracking quality is sufficient, using weighted updates with truncation distance µ = 4v_o.Integration requires 50% validly tracked TSDF pixels and ICP RMSE < 0.03 to preserve reconstruction quality under drift.
- Fusion and rendering: A voxel foreground probability is updated from instance-mask detections with a beta model, and foreground is rendered when E[p_o(v)] > 0.5.Surface integration still covers the entire volume; the foreground model separately records which voxels belong to the instance.
- Fusion and rendering: The system raycasts TSDF depth, normals, vertices, RGB, and object indices for tracking, association, and visualisation, using foreground zero crossings and background fallback handling.Existence probabilities suppress spurious instances, while semantic labels are refined by averaging associated Mask R-CNN class distributions instead of multiplying them.
4. Experiments
Experiments evaluate Fusion++ on an office RGB-D sequence, YCB reconstruction data, and the RGB-D SLAM benchmark, emphasizing loop-closed mapping, reconstruction quality, trajectory error, memory, and runtime.
- Office sequence: 3,685 frames from a repeated office sequence tested map consistency under planar-floor geometry, ICP drift, and loop closure.The sequence contained many object instances and was used to evaluate memory and runtime scaling.
- Office sequence: 105 landmark object instances were reconstructed, although noisy partially reconstructed sub-objects still accumulated despite filtering.
- Reconstruction quality: YCB sequence 0001 reconstructions were qualitatively compared with ground-truth models, with missing cracker-box geometry attributed to occlusion and missed foreground detection.
- RGB-D SLAM benchmark: Fusion++ improved trajectory error over simple coarse TSDF odometry on all but one evaluated RGB-D SLAM benchmark sequence.The baseline omitted instance layers and loop-closure pose-graph optimisation.
- Limitations: The system did not achieve state-of-the-art performance and would require joint depth and photometric tracking for competitiveness.The experiments prioritized a usable object map over motion-tracking accuracy.
- Memory and runtime: 377MB for 105 objects and approximately 4MB per object were used on the office sequence, illustrating compact object-volume memory usage.The system used dynamically varying object volumes up to dimension 128^3.
- Memory and runtime: 4-8Hz average frame rate was achieved excluding relocalisation, with approximately 1ms of additional computational cost per object.The system was not real-time and the runtime analysis covered scaling by object count.
5. Conclusions
Fusion++ consistently maps and classifies numerous previously unknown objects in cluttered indoor scenes while maintaining a long-term, variable-resolution object map. Remaining limitations include spurious partial reconstructions and separate TSDFs for duplicate objects.
- Fusion++ consistently maps and classifies numerous objects of previously unknown shape in real, cluttered indoor scenes.
- Its online system combines instance segmentation, TSDF fusion, tracking, and pose-graph optimisation into a long-term object-focused map.
- Variable, object size-dependent resolution focuses the map on important object elements while adapting reconstruction fidelity to object scale.
- Spurious detections can accumulate into a growing clutter of partial object reconstructions despite existence probabilities and deletion mechanisms.
- Duplicate objects seen from different views remain in separate TSDFs instead of being combined into a single better model.