MaskFusion: Real-Time Recognition, Tracking and Reconstruction of Multiple Moving Objects

• SLAM has progressed from sparse to dense mapping and enabled augmented-reality applications, but most systems assume mostly static scenes and produce geometric maps.Prior work addressed single non-rigid objects [20] or multiple moving rigid objects, while semantic mapping often relied on known 3D models or labeled map points without distinguishing instances [28] [44].Source: §1 · pp. 1–2
• MaskFusion recognizes, detects, tracks, and reconstructs multiple moving rigid objects, segmenting each instance and assigning it a semantic label.It combines Mask-RCNN category predictions for 80 object classes with depth- and surface-normal-based geometry segmentation to improve object boundaries.Source: §1 · p. 2
• Accurate instance masks let MaskFusion fuse moving objects and the static background while propagating semantic image labels into temporally consistent 3D map labels.Unlike pixel-level semantic segmentation, instance-aware segmentation distinguishes separate objects in the same category instead of treating them as one blob.Source: §1 · p. 2
• By mapping independently moving objects rather than treating them as outliers, MaskFusion adds their detailed geometry and improves object and camera pose prediction and estimation.This addresses the static-scene assumption common in previous semantic SLAM systems [28] [44].Source: §1 · p. 2
• Compared with previous dynamic SLAM systems, MaskFusion adds real-time semantic labels from many object classes and segments individual object instances.Mask-RCNN supports object detection and semantic labeling of corresponding 3D map points.Source: §1 · p. 2
• Object labels support robotics and augmented reality; detecting and segmenting people can help tracking continue when objects are moved or occluded by a hand.The system can ignore people’s pixels and focus on the objects they manipulate.Source: §1 · p. 2

Dense RGB-D SLAM

• Dense RGB-D SLAM enabled accurate real-time mapping, with KinectFusion using TSDF maps and later work extending the approach to large environments.Consumer-grade RGB-D cameras helped stimulate this line of research.Source: §2 · p. 2

Surfel maps

• Surfel-based RGB-D-SLAM represents local surface properties alongside location, yielding memory-efficient maps without TSDF fusion’s representation-switching overhead.Whelan et al. presented a surfel-based system for large environments with local and global loop closure.Source: §2 · p. 2

Scene segmentation

• Geometry-based segmentation can produce accurate object boundaries, but often over-segments scenes and provides no semantic information.These methods commonly segment RGB-D data using surface-normal geometry while assuming objects are convex.Source: §2 · p. 2

Semantic scene segmentation

• Semantic 3D scene segmentation with MRFs requires labelled 3D training data, whose annotation and consistent reconstruction demand substantial effort.The cited works manually annotated training data; isolated RGB-D datasets such as NYUv2 do not provide the required consistent reconstructions [10].Source: §2 · p. 2

Semantic SLAM

• Deep-network semantic SLAM [28] [44] and earlier decision-forest work fuse labelled images into 3D maps, but do not track object instances independently.Using 2D inference avoids the need for annotated 3D data, but without instance awareness multiple models cannot be tracked independently.Source: §2 · p. 2

Dynamic SLAM

• Dynamic SLAM covers deformable surfaces and independently moving rigid objects; template-based approaches require prior observation, while Co-Fusion and DynSLAM were not real-time.Co-Fusion tracked and reconstructed semantically labelled objects; DynSLAM reconstructed moving vehicles only. Table 1 distinguishes MaskFusion as real-time, dynamic, and instance-segmenting, unlike prior semantic SLAM systems [28, 40, 44, 46].Source: §2 · p. 3, Table 1 · p. 3

System architecture

• MaskFusion maintains a separate 3D representation for each recognized object alongside a background model, tracking and fusing each model independently.Object geometry is represented by surfels, and object labels associate surfels with the correct model during fusion.Source: §3 · pp. 3–4

Tracking

• Each model’s six-degree-of-freedom pose is tracked using geometric ICP and photometric brightness-constancy costs, while only non-static objects are tracked separately.Static-versus-dynamic status is determined either by motion inconsistency, similar to, or by treating objects touched by a person as dynamic.Source: §3 · p. 3

Segmentation

• MaskFusion combines Mask-RCNN semantic masks with depth-discontinuity and surface-normal segmentation to balance semantic labels, accurate boundaries, and real-time operation.Geometric segmentation runs in real time but tends to oversegment; semantic segmentation runs at up to 5 Hz and can leak across object boundaries, so geometry is used alone when semantic masks are unavailable and alongside them when available.Source: §3 · pp. 3–4

4 MULTI-OBJECT SLAM

• Each recognized object and the background have independent surfel models with class and object labels, plus per-time static indicators and rigid poses.Each surfel stores position, normal, colour, weight, radius and two timestamps.Source: §4 · p. 4

4.1 Tracking

• Tracking aligns current depth and grayscale images with a rendered model projection by minimizing joint geometric and photometric error.A Gauss-Newton solver uses a four-level coarse-to-fine pyramid; its CUDA implementation builds on and.Source: §4.1 · p. 4

4.2 Fusion

• Fusion associates each model’s surfels with the current RGB-D frame using segmentation-based stencils, assigning every new surfel to exactly one model.A confidence penalty for surfels outside the stencil compensates for imperfect segmentations.Source: §4.2 · p. 4

5 SEGMENTATION

• 2D semantic-to-model correspondences let MaskFusion mask data before fusion, while geometric segmentation sharpens boundaries; the pipeline runs at 30fps despite segmentation latency.Instance segmentation runs concurrently with tracking and fusion; a 12-frame queue adds approximately 400ms delay, and frames without masks are associated with existing models. The segmentation pipeline provides masks at 30Hz or more.Source: §5 · p. 4

5.1 SEMANTIC INSTANCE SEGMENTATION

• Mask-RCNN supplies instance masks, bounding boxes, and class IDs for 80 classes at 5Hz, but its masks can miss object-boundary detail.Its third branch predicts masks independently of classes and boxes, extending Faster-RCNN with a ResNet-based backbone.Source: §5.1 · p. 5

5.2 GEOMETRIC SEGMENTATION

• Geometric segmentation uses depth discontinuities and concavity to place edges, then converts the edge map into connected components.Assuming objects are largely convex enables fast over-segmentation; local-neighborhood edginess computation runs quickly on a GPU.Source: §5.2 · p. 5

5.3 MERGED SEGMENTATION

• Geometric segmentation runs on every SLAM frame; when semantic masks are unavailable, components are associated directly with existing models and mask-mapping steps are skipped.The pipeline maps geometric components to available masks and then to models.Source: §5.3 · p. 5, Figure 5 · p. 6

5.3.1 Mapping geometric labels to masks

• Each geometric component maps to its maximum-overlap semantic mask only when overlap exceeds 65% of the component’s pixels; multiple components may share a mask.A component maps to at most one mask, and assigned components receive the mask ID in the updated labeling.Source: §5.3.1 · p. 6

5.3.2 Mapping masks to models

• Grouped components map to rendered models when overlap exceeds 5% and model and mask classes match; unassigned components can map directly at a 65% overlap threshold.The final labels contain associated model IDs and can mark regions to ignore during fusion, preventing reconstruction of selected classes such as a person’s arm.Source: §5.3.2 · p. 6

6 EVALUATION

• The evaluation targets problems traditional SLAM cannot solve; MaskFusion’s mapping and tracking components are based on.The paper refers readers to those publications for additional implementation details.Source: §6 · p. 6

6.1.1 Trajectory estimation

• Trajectory evaluation uses ATE and RPE RMSE on dynamic benchmark sequences, measuring overall positional offsets and local motion error, respectively.MaskFusion neglects data associated with people because the compared methods cannot reconstruct deformable parts.Source: §6.1.1 · p. 6
• 2.2cm teddy-bear and 8.9cm camera AT-RMSEs show that independent object tracking can affect camera-pose accuracy; treating the bear as background reduces camera AT-RMSE to 7.2cm.The bear occupied a significant part of the field of view in the comparison with ground-truth trajectories.Source: Figure 6 · p. 7, §6.1.1 · p. 7
• Semantic context helps in highly dynamic scenes and difficult starts, while ElasticFusion performs best when motion is slight despite assuming a static scene.The authors suggest other methods may discard points useful for tracking when labeling them dynamic or outliers.Source: §6.1.1 · pp. 6–7

6.1.2 Reconstruction

• 7.0mm average 3D error (5.8mm standard deviation) was measured for the reconstructed 250mm-tall, 100mm-wide bleach bottle.The evaluation used a dynamic sequence and a ground-truth model from the YCB Object and Model Set.Source: §6.1.2 · p. 7

6.1.3 Segmentation

• Combining semantic and geometric cues improves per-frame mask segmentation, while tracking and fusing temporally consistent 3D models improves it further.The evaluation used ground-truth 2D teddy masks over a 600-frame sequence and compared MaskRCNN-only masks with geometric segmentation and the full method.Source: §6.1.3 · p. 7

6.2 Qualitative results

• MaskFusion is tested on varied dynamic sequences to demonstrate its effectiveness across different use cases.Source: §6.2 · p. 7

6.2.1 Grasping

• MaskFusion densely detects and reconstructs objects and continues tracking them during interaction, supporting grasp-point identification on the correct object.In one sequence, it modeled five objects during the first 600 frames while tracking the camera, then tracked a touched spray bottle reliably until it was returned to the table at frame 1100. A hand detector identified touch; known actuators or people can be excluded from reconstruction by training the network.Source: §6.2.1 · p. 7

6.2.2 Augmented reality

• MaskFusion’s augmented-reality demos combine geometric and semantic data in dynamic scenes.Source: §6.2.2 · p. 7
• The calories demo estimates an object’s calories from its class and shape by fitting primitives to body volume and applying class-specific calories-per-volume ratios.Source: §6.2.2 · p. 7
• The skateboard demo keeps a virtual character attached to a skateboard after a person kicks it into motion, requiring simultaneous tracking of the board and camera.Source: §6.2.2 · p. 7

6.3 Performance

• 5Hz masking runs asynchronously on a dedicated GPU; SLAM on a second GPU runs >30Hz for one tracked model and 20Hz for three non-static models.The test system uses two Nvidia GTX Titan X GPUs and an Intel Core i7 at 3.5GHz.Source: §6.3 · p. 8

CONCLUSIONS

• MaskFusion infers semantic labels from 2D images, maintains independent 3D models for object instances and background, and supports augmented-reality applications and robotics tasks.The paper presents it as a real-time visual SLAM system for mapping and tracking multiple objects.Source: §7 · p. 8
• Recognition is limited to classes in MaskRCNN’s training data—currently 80 MS-COCO classes—and the system does not account for object-label misclassification.The paper identifies these as recognition limitations.Source: §7 · p. 8
• MaskFusion can remove some non-rigid objects, such as humans, from the map, but tracking and reconstruction are limited to rigid objects.The paper notes that this leaves non-rigid objects outside its tracking and reconstruction capability.Source: §7 · p. 8
• Tracking small objects with little geometric information can produce errors when no 3D model is available.The paper presents addressing these limitations as an opportunity for future work.Source: §7 · p. 8