PaperlayerView exact revision on arXiv

Source-linked AI summary

MID-Fusion: Octree-based Object-Level Multi-Instance Dynamic SLAM

Binbin Xu, Wenbin Li, Dimos Tzoumanikas, Michael Bloesch, Andrew Davison, Stefan Leutenegger

arXiv:1812.07976v4cs.ROcs.CV

TL;DR

Dynamic environments challenge SLAM systems built on static-world assumptions, motivating robust tracking and object-aware mapping. The paper proposes an octree-based multi-instance RGB-D SLAM system that jointly tracks camera and moving objects while refining and fusing object models. It demonstrates effectiveness across synthetic and real-world indoor sequences and reports CPU operation at 2–3 Hz excluding instance segmentation.

  • Problem

    Static-environment assumptions cause moving objects to be treated as outliers, while robust SLAM for highly dynamic environments remains an open problem.

  • Method

    The system uses object-level octree volumetric models, multimodal mask refinement, camera and object tracking, and probabilistic fusion of geometric, semantic, and foreground information.

  • Results

    The method demonstrates effectiveness in various indoor scenarios, with camera tracking achieving the best results in almost all evaluated sequences among dense tracking methods.

  • Takeaways & Limitations

    The resulting map continuously represents dense geometry, semantics, and object foreground probabilities for dynamic indoor environments and supports indoor robotic applications.

Abstract

from arXiv · show

We propose a new multi-instance dynamic RGB-D SLAM system using an object-level octree-based volumetric representation. It can provide robust camera tracking in dynamic environments and at the same time, continuously estimate geometric, semantic, and motion properties for arbitrary objects in the scene. For each incoming frame, we perform instance segmentation to detect objects and refine mask boundaries using geometric and motion information. Meanwhile, we estimate the pose of each existing moving object using an object-oriented tracking method and robustly track the camera pose against the static scene. Based on the estimated camera pose and object poses, we associate segmented masks with existing models and incrementally fuse corresponding colour, depth, semantic, and foreground object probabilities into each object model. In contrast to existing approaches, our system is the first system to generate an object-level dynamic volumetric map from a single RGB-D camera, which can be used directly for robotic tasks. Our method can run at 2-3 Hz on a CPU, excluding the instance segmentation part. We demonstrate its effectiveness by quantitatively and qualitatively testing it on both synthetic and real-world sequences.

I. INTRODUCTION

The paper addresses the limitations of static-environment SLAM by proposing an object-level dynamic volumetric map for indoor scenes. Its contributions combine robust tracking, multimodal segmentation refinement, and probabilistic octree fusion.

  • Motivation: Static-environment SLAM treats moving objects as outliers, limiting its reliability in constantly changing real-world environments.The paper identifies robust SLAM in highly dynamic environments as an open problem.
  • System overview: The system overview shows RGB-D inputs producing labelled object models and a coloured reconstruction while handling dynamic objects and ignoring people.The output is an object-level dense volumetric map.
  • Proposed system: The proposed system is the first object-level dynamic volumetric map for indoor applications, representing free space and surface connectivity for each object.An octree-based structure improves memory efficiency, while semantic predictions are integrated and refined during map fusion.
  • Contributions: The system introduces RGB-D multi-instance dynamic SLAM with uncertainty-weighted object tracking, multimodal mask refinement, and probabilistic octree fusion.The fusion combines semantic distributions and foreground object probabilities with object models.

II. RELATED WORK

Prior dynamic SLAM methods either deform the world, reconstruct only static backgrounds, or build separate sub-maps for moving objects. The paper differentiates its object-level system through representation, tracking, segmentation, fusion, and computational design.

  • Dynamic SLAM directions: Dynamic SLAM approaches have mainly pursued non-rigid world deformation, static-background reconstruction, or separate sub-maps for potentially rigidly moving objects.The first two directions respectively include deformable objects and intentional removal of moving objects from tracking and reconstruction.
  • Object-level systems: Co-Fusion and MaskFusion reconstruct multiple moving objects with surfels, but surfel maps do not directly provide free-space information or surface connectivity.The paper focuses instead on indoor scenes with many potentially moving objects using a single RGB-D camera.
  • System differences: The proposed system combines uncertainty-weighted photometric and geometric camera tracking, object-coordinate tracking, multimodal mask refinement, and semantic foreground fusion.These components address depth loss, object rotation, mask boundaries, and neural predictions without map-fusion refinement.
  • Computational design: The method runs on CPU at a speed comparable to DynSLAM, whereas the other compared multi-object approaches require one or two powerful GPUs.This comparison concerns computational requirements rather than a shared accuracy metric.
  • Comparison with Fusion++: Compared with Fusion++, the system handles dynamic scenes, jointly tracks camera and object poses, refines masks, and uses memory-efficient octrees instead of discrete voxel grids.Fusion++ generates an object-level volumetric map in static environments and uses geometric camera tracking.

III. NOTATIONS AND PRELIMINARIES

The system represents each detected object in its own coordinate frame and octree-based volumetric model, then processes RGB-D frames through segmentation, tracking, fusion, and raycasting.

  • Object representation: Each detected object is stored in a separate object coordinate frame, with object 0 representing the background.The object index ranges over the total detected objects excluding the background.
  • Object representation: Every object has a canonical static volumetric model, semantic class information, a semantic probability distribution, a world-relative pose, and a motion label.Each object is represented by a separate octree structure.
  • Processing pipeline: The pipeline performs instance segmentation, geometric and motion-based mask refinement, camera tracking, object visibility and association, moving-object tracking, and fusion.Raycasting identifies visible objects and supports associating local masks with existing models.
  • Tracking pipeline: The camera is tracked against vertices outside the human mask, after which visible objects are raycast and moving objects are tracked before camera refinement against the static world.The overview identifies segmentation, tracking, fusion, and raycasting as the four pipeline parts.

B. RGB-D Camera tracking

Camera tracking estimates the live camera pose by combining model- and static-scene tracking with dense geometric and photometric residuals. Robust weighting, coarse-to-fine optimisation, and motion detection support refinement against static objects.

  • RGB-D Camera tracking: Camera tracking first uses all model vertices while masking detected people, then tracks against static scene parts.Both stages estimate the live camera pose.
  • RGB-D Camera tracking: Dense point-to-plane ICP and photometric RGB residuals are jointly minimised and weighted by individual measurement uncertainty.Invalid correspondences, occlusions, and humans are excluded by the mask.
  • RGB-D Camera tracking: Rendered reference depth maps provide photometric alignment and improve robustness when raw input depth is unavailable.The rendered depth is used instead of raw live- or reference-frame depth for de-noised model quality.
  • RGB-D Camera tracking: Measurement uncertainty weights reflect RGB constancy and depth-sensor structure, depth range, and inverse covariance.The ICP weighting uses sensor parameters and depth-measurement uncertainty.
  • RGB-D Camera tracking: Gauss-Newton optimisation runs in a three-level coarse-to-fine scheme, and a second finest-level evaluation identifies moving objects.Objects with an inlier ratio below 0.9 are considered moving, after which camera tracking uses only static objects.

C. Object pose estimation

Moving-object pose estimation uses an object-centric joint dense ICP and RGB tracker. Its canonical object-frame formulation reduces rotational lever-arm effects and supports stable alignment.

  • Object pose estimation: The method estimates the current relative object-camera pose by aligning live and rendered vertex maps expressed in object coordinates.It uses joint dense ICP and RGB tracking with the same uncertainty weighting as camera tracking.
  • Object pose estimation: Each object coordinate frame is assumed to yield a static canonical model, so corresponding point clouds should align.This is the geometric assumption underlying object-pose estimation.
  • Object pose estimation: Centering the object frame makes the rotational derivative term small, reducing the lever-arm effect.The paper connects this parameterisation to more stable tracking.
  • Object pose estimation: The object-pose cost is optimised with Gauss-Newton in a three-level coarse-to-fine scheme, initialised from the reference object-camera pose.

D. Combined semantic-geometric-motion segmentation

Semantic-geometric-motion segmentation combines Mask R-CNN instances, geometric boundary refinement, rendered object masks, and motion residuals. Probabilistic fusion is retained when semantic predictions are uncertain.

  • Combined semantic-geometric-motion segmentation: Mask R-CNN detects semantic instances, geometric refinement corrects leaked boundaries, and raycasting renders existing object masks into the live frame.
  • Combined semantic-geometric-motion segmentation: Mask association does not require local semantic labels to equal existing object classes because semantic predictions may be uncertain.Semantic class probabilities are refined through probabilistic fusion.
  • Combined semantic-geometric-motion segmentation: Unassociated masks initialise new object models, while undetected existing objects retain their rendered masks for subsequent fusion.
  • Combined semantic-geometric-motion segmentation: Motion residuals refine associated masks by filtering pixels whose joint ICP and RGB residuals are too high.The residual evaluation is repeated at the finest level using photometric residuals on the live frame.
  • Combined semantic-geometric-motion segmentation: Foreground and dilated-background masks assign probabilities that reduce spurious integration from incorrect segmentation masks.Information from both foreground and background regions is integrated into object models.

E. Object-level fusion

Object-level fusion incrementally integrates geometric, appearance, semantic, and foreground-probability information into object models. New models use object-centred octree volumes sized from observed point clouds.

  • Object-level fusion: Each frame updates depth, colour, semantics, and foreground probabilities within object models using foreground and background masks.TSDF, colour, and foreground values are updated concurrently, while semantic class distributions are refined by averaging.
  • Object-level fusion: New object models centre their coordinate frames on the object and estimate volume centre and size from back-projected masked points.
  • Object-level fusion: New TSDF volumes are initialised at three times the point-cloud size to accommodate possible occlusions.
  • Object-level fusion: The octree leaves unused voxels uninitialised, preserving memory efficiency during object-volume construction.The initial object translation is placed at the volume's left-side corner with world-aligned orientation.

F. Raycasting

The system reduces raycasting cost by identifying visible objects once per frame, then skipping invisible objects in subsequent operations.

  • Raycasting supports depth rendering, visible-object discovery, IoU calculation, and visualisation.Continuously raycasting every object for all four operations would be computationally expensive.
  • All objects are raycast once to identify which are visible in the current frame.This visibility pass determines which object models need further processing.
  • Subsequent raycasting steps avoid objects classified as invisible on that frame.The optimization reduces repeated raycasting without changing the listed pipeline operations.

V. EXPERIMENTS

Experiments evaluate camera tracking in real dynamic sequences and object reconstruction in a controlled synthetic scene. The results indicate strong dense-tracking performance and accurate object reconstruction, while qualitative tests show robust handling of multiple moving objects.

  • Experimental setup: The evaluation uses a Linux desktop with an Intel Core i7-7700 CPU, 32GB memory, and pre-computed Mask R-CNN segmentation.Each object is represented by a separate octree-based volumetric model modified from Supereight.
  • Camera tracking: Six TUM RGB-D sequences with motion-capture camera trajectories measure dynamic-environment tracking using RMSE of Absolute Trajectory Error.The method is compared with VO-SF, StaticFusion, DynaSLAM, Co-Fusion, and MaskFusion.
  • Camera tracking: The system achieves the best results in almost all sequences among dense tracking methods.It outperforms VO-SF and StaticFusion, which were designed for robust camera tracking in dynamic environments.
  • Object reconstruction: In a synthetic indoor scene with a moving sofa and chair, object pose estimation is evaluated through reconstruction error.The experiments compare the proposed system with ground-truth masks, virtual-camera tracking, and Co-Fusion.
  • Object reconstruction: The system produces more accurate object reconstructions, with negligible difference between ground-truth masks and its own segmentation in the tested example.Higher error with virtual-camera tracking indicates greater reliability of object-centric tracking for large object rotations.

C. Real-world applications

Real-world demonstrations show detailed, non-colliding reconstructions for multiple moving objects, while the CPU implementation retains efficient memory usage but remains insufficiently optimized for high performance.

  • Qualitative results: Separate volumetric maps prevent object models from colliding, supporting detailed multi-instance reconstructions.The system demonstrated robust tracking and reconstruction for more than six moving objects.
  • Qualitative comparison: More than 6 moving objects were simultaneously tracked while maintaining a highly detailed reconstruction.In comparison, Co-Fusion did not successfully segment and reconstruct these objects in the reported scene.
  • Runtime: 400 ms per frame was the average CPU processing time when more than 25 objects were generated in the scene.The evaluation used sequences with approximately 3 to 6 objects being moved.
  • Runtime: 10 ms per object was required for initialization when a new object was detected.Tracking time scales mainly with moving objects, while segmentation, integration, and raycasting scale with visible objects.
  • Limitations: The current system runs only on CPU and is not highly optimized for performance.The authors suggest GPU parallelization could enable a higher frame-rate version.

VI. CONCLUSIONS

The paper presents an octree-based multi-instance dynamic SLAM approach that tracks the camera while estimating dense object geometry, semantics, and foreground probabilities. Experiments in varied scenarios demonstrate effectiveness in indoor environments and motivate indoor robotic applications.

  • Conclusion: The method uses an octree-based volumetric representation for multi-instance dynamic SLAM.It robustly tracks camera pose while continuously estimating dense geometry, semantics, and object foreground probabilities.
  • Conclusion: Experimental results in various scenarios demonstrate the method’s effectiveness in indoor environments.The authors connect the system to robotic applications involving environment change, free space, and object-level information.
Loading 1812.07976v4…