Source-linked AI summary
MaskFusion: Real-Time Recognition, Tracking and Reconstruction of Multiple Moving Objects
Martin Rünz, Maud Buffier, Lourdes Agapito
TL;DR
Conventional SLAM largely assumes static scenes and often provides either purely geometric maps or semantic labels without object-instance distinction. MaskFusion combines real-time instance segmentation with object-level dynamic RGB-D SLAM to recognize, track, reconstruct, and label multiple moving objects. It demonstrates object-aware maps and applications in augmented reality and robotics, while remaining limited by recognition classes, rigid-object tracking, and small-object geometry.
Problem
Existing SLAM methods are commonly limited by static-scene assumptions and semantic representations that lack broad, model-free instance-level object mapping.
Method
MaskFusion combines Mask-RCNN instance segmentation with geometry-based boundaries and independently tracks and fuses 3D models for recognized objects and the background.
Results
MaskFusion provides real-time dynamic SLAM that recognizes, segments, semantically labels, tracks, and reconstructs multiple moving rigid objects.
Takeaways & Limitations
The object- and label-level map supports augmented-reality applications and robotics tasks such as tracking manipulated objects during interaction.
Takeaways & Limitations
Recognition is restricted to Mask-RCNN’s trained classes, tracking and reconstruction are limited to rigid objects, and small objects with little geometric information can cause errors.
Abstract
from arXiv · showhide
We present MaskFusion, a real-time, object-aware, semantic and dynamic RGB-D SLAM system that goes beyond traditional systems which output a purely geometric map of a static scene. MaskFusion recognizes, segments and assigns semantic class labels to different objects in the scene, while tracking and reconstructing them even when they move independently from the camera. As an RGB-D camera scans a cluttered scene, image-based instance-level semantic segmentation creates semantic object masks that enable real-time object recognition and the creation of an object-level representation for the world map. Unlike previous recognition-based SLAM systems, MaskFusion does not require known models of the objects it can recognize, and can deal with multiple independent motions. MaskFusion takes full advantage of using instance-level semantic segmentation to enable semantic labels to be fused into an object-aware map, unlike recent semantics enabled SLAM systems that perform voxel-level semantic segmentation. We show augmented-reality applications that demonstrate the unique features of the map output by MaskFusion: instance-aware, semantic and dynamic.
1 INTRODUCTION
MaskFusion addresses two limitations of conventional SLAM: handling independently moving objects and representing scenes with semantic, instance-level structure rather than purely geometric maps.
- Most SLAM systems assume mostly static environments, leaving robust reconstruction of arbitrary dynamic and non-rigid scenes an open challenge.
- Conventional semantic SLAM commonly recognizes only known instances with available 3D models or labels map points without distinguishing object instances.
- MaskFusion recognizes, detects, tracks, reconstructs, segments, and semantically labels multiple moving rigid objects in real time.
- Its instance-aware masks distinguish objects sharing a category and propagate semantic image labels into temporally consistent 3D map labels.
- Unlike prior semantic SLAM systems, MaskFusion handles multiple independently moving objects and maintains their 3D representations instead of treating them as outliers.
- MaskFusion combines semantic information from many object classes with real-time dynamic mapping, including individual object detection, labels, and instance segmentation.
- The resulting object- and label-level dynamic representation supports applications in robotics and augmented reality, including tracking manipulated objects during human interaction.
2 RELATED WORK
Related systems established dense RGB-D, surfel-based, semantic, and dynamic mapping, but prior methods generally lacked MaskFusion’s combination of real-time operation, object instances, scene-wide reconstruction, and broad semantic coverage.
- Dense RGB-D SLAM: Dense RGB-D SLAM methods enabled accurate real-time mapping of indoor environments using inexpensive consumer-grade RGB-D cameras.
- Surfel-based maps: Surfel maps represent local surface properties such as radius and normal alongside point locations, offering a memory-efficient alternative to TSDF-based maps.
- MaskFusion comparison: Compared with previous semantic SLAM, MaskFusion is dynamic and segments object instances; compared with dense non-rigid systems, it reconstructs the entire scene and adds semantic labels.
- Dynamic reconstruction: Earlier dynamic approaches included non-rigid reconstruction and independently moving rigid-object tracking, using both template-based and template-free formulations.
- Comparative scope: Table 1 compares real-time SLAM systems across five important properties.
- Prior dynamic semantic systems: Co-Fusion used semantic cues for object segmentation but was not real-time, while DynSLAM reconstructed only moving vehicles and was also not real-time.
3 SYSTEM OVERVIEW
MaskFusion performs dense dynamic RGB-D SLAM at the object level by independently tracking and fusing object models and the background, using geometric and semantic segmentation.
- System overview: MaskFusion maintains a separate 3D representation for each recognized object and for the background, tracking and fusing each model independently.
- Tracking: Each object’s geometry is represented by surfels, and its six-degree-of-freedom pose is tracked using geometric ICP and photometric brightness-constancy costs.
- Tracking: Only non-static objects are tracked separately to reduce computational demand and improve robustness, with motion inconsistency and human-touch cues used to identify dynamics.
- Pipeline: The paper combines semantic and geometric processing within a real-time pipeline and evaluates the approach quantitatively and qualitatively.
- Segmentation: Geometric segmentation analyzes depth discontinuities and surface normals to produce accurate object boundaries in real time.
4 MULTI-OBJECT SLAM
MaskFusion represents the scene with independent surfel-based 3D models for each recognized object and the background. It tracks each model by aligning current RGB-D observations with rendered model projections using joint geometric and photometric errors, then fuses data within object boundaries.
- Scene representation: MaskFusion maintains an independent surfel-based 3D model for each recognized object and an additional background model.Each model stores surface geometry and appearance, along with class and object labels.
- Model tracking: Each model’s six-degree-of-freedom pose is estimated by aligning the current depth and intensity maps with a rendered projection of the model.The rendered projection is generated from the model’s surfels using OpenGL.
- Model tracking: Pose optimization minimizes a joint geometric and photometric error over the unknown rigid transformation.The geometric term uses projective ICP residuals, while the photometric term uses photoconsistency residuals.
- Optimization: The non-linear alignment cost is optimized with Gauss-Newton using a four-level coarse-to-fine pyramid scheme.The solver has a CUDA-accelerated implementation based on prior open-source releases.
- Fusion: Projective data association updates each model’s surfels while segmentation stencils enforce object boundaries during fusion.New surfels belong to exactly one model, and surfels outside the stencil receive a confidence penalty.
5 SEGMENTATION
MaskFusion combines semantic instance segmentation with fast geometric over-segmentation to obtain accurate object masks at real-time rates. Geometric components are mapped to masks and existing models, with fallback associations when semantic masks are unavailable.
- Segmentation design: The pipeline combines semantic masks with geometric over-segmentation because semantic methods detect objects but provide imperfect boundaries, while geometry supplies accurate boundaries but tends to oversegment.The geometric method uses depth discontinuities and surface normals.
- Semantic instance segmentation: Mask-RCNN provides instance masks and category labels for many object classes, but its 5Hz processing rate requires concurrent execution with tracking and fusion.Frames are buffered so the SLAM pipeline and semantic segmentation can operate at different frequencies.
- Geometric segmentation: Geometric segmentation marks pixels where depth-discontinuity and concavity terms exceed a threshold, then converts the edge map into connected components.The local-neighborhood computation supports fast GPU evaluation, but the convexity premise can cause over-segmentation.
- Merged segmentation: For each frame, geometric components are mapped to available semantic masks and then to existing models; unmatched components can be associated directly with objects.Geometric labels are processed every frame, while mask mapping is skipped when semantic masks are unavailable.
- Mapping geometric labels to masks: A component is assigned to the mask with maximal overlap only when overlap exceeds 65% of the component’s pixels.Multiple components may map to one mask, but each component links to at most one mask.
- Merged segmentation: Model association tests component overlap against rendered object labels, checks class-ID agreement, and uses a 65% overlap fallback for remaining components.A special ignore label prevents selected regions, such as a person’s arm, from being fused.
6 EVALUATION
The section focuses on challenging problems beyond traditional SLAM, particularly dynamic and non-rigid scene reconstruction. It directs readers to prior work for details of MaskFusion’s mapping and tracking components.
- Scope: MaskFusion’s mapping and tracking components are based on prior work.The paper refers readers to the corresponding publications for additional details.
- Evaluation focus: The evaluation focus is on challenging problems that traditional SLAM systems cannot solve.This frames the assessment around the system’s dynamic-scene capabilities.
- Scope: The cited prior publications provide additional details about the mapping and tracking components.The section does not reproduce those implementation details.
6.1 Quantitative results
MaskFusion is evaluated quantitatively for camera and object tracking, reconstruction, and segmentation across dynamic RGB-D sequences. Results show benefits from contextual semantic information and temporally consistent object models, while independent tracking can affect camera estimation.
- Trajectory estimation: MaskFusion is quantitatively compared with other methods on an established RGB-D benchmark using trajectory-error measures.The evaluation reports absolute trajectory error and relative pose error as RMSE-based measures.
- Trajectory estimation: ElasticFusion performs best under slight motion despite assuming a static scene.The authors interpret this as other methods retaining points beneficial for tracking while dynamic or outlier labeling removes them.
- Trajectory estimation: 2.2cm teddy-bear AT-RMSE and 8.9cm camera AT-RMSE are reported when the bear is tracked independently.Treating the bear as background reduces camera AT-RMSE to 7.2cm because the bear occupies a significant field of view.
- Trajectory estimation: Context information is especially useful in highly dynamic scenes and difficult scene beginnings, where semantic segmentation remains robust.These cases can be difficult for energy-minimization approaches.
- Reconstruction: 7.0mm average 3D reconstruction error with 5.8mm standard deviation is measured for a bleach bottle from the YCB dataset.The evaluation compares reconstructed surfels with a ground-truth model of a 250mm-tall, 100mm-wide bottle.
- Segmentation: Combining semantic and geometric cues improves segmentation, while tracking and fusing temporally consistent 3D models achieves still better results.Segmentation quality is measured with intersection-over-union against 2D teddy-bear annotations over 600 frames.
6.2 Qualitative results
Qualitative results demonstrate MaskFusion across dynamic sequences, robotics tasks, and augmented-reality applications. Its object-level semantic reconstructions support interaction-aware tracking, calorie estimation, and virtual-character behavior.
- Qualitative results: MaskFusion is tested on varied dynamic sequences as an effective toolbox for different use cases.The qualitative demonstrations include object manipulation and augmented reality.
- Robotics applications: Dense object detection and reconstruction support grasp-point identification while tracking continues during interaction.The system is described as suitable for robotic grasping because it reconstructs objects densely and maintains tracking when objects move or become occluded.
- Augmented reality: The system's map combines semantic object labels with reconstructed geometry for dynamic-scene applications.Examples include a virtual character interacting with the scene and calorie estimation for groceries.
- Augmented reality: The calorie demo estimates an object's calories from its class and shape using body-volume estimation and class-specific calories-per-volume ratios.Simple primitive fitting is used to estimate body volumes.
- Augmented reality: The skateboard demo keeps a virtual character attached to a moving skateboard after a person kicks it.This requires simultaneous accurate tracking of the skateboard and camera.
6.3 Performance
MaskFusion separates neural masking from the SLAM pipeline to achieve real-time operation, with throughput depending on the number of tracked non-static objects.
- Performance: The masking component runs at 5Hz on a dedicated GPU, while the SLAM pipeline exceeds 30Hz for one tracked model.With three non-static models, performance declines to 20Hz.
7 CONCLUSIONS
MaskFusion provides real-time semantic SLAM with independent 3D models for recognized objects and the background, supporting augmented-reality and robotics applications. Its scope remains bounded by recognition, rigidity, and small-object tracking limitations.
- Conclusions: MaskFusion maintains independent 3D models for each recognized object instance and the background while tracking and mapping multiple objects.The system uses semantic scene understanding within a real-time visual SLAM framework.
- Conclusions: MaskFusion supports novel augmented-reality applications and common robotics tasks.The conclusion presents these as demonstrated uses of the system.
- Limitations: Recognition is limited to Mask-RCNN's trained classes and does not account for object-label misclassification.The current trained set comprises 80 MS-COCO classes.
- Limitations: Tracking and reconstruction are limited to rigid objects, although some non-rigid objects such as humans can be removed from the map.Small objects with little geometric information can also produce errors when no 3D model is available.