PaperlayerView exact revision on arXiv

Source-linked AI summary

Co-Fusion: Real-time Segmentation, Tracking and Fusion of Multiple Objects

Martin Rünz, Lourdes Agapito

arXiv:1706.06629v1cs.CV

TL;DR

Dynamic-scene SLAM commonly treats moving objects as outliers instead of modeling their geometry and motion. Co-Fusion segments objects using motion or semantic cues, tracks and reconstructs each independently from RGB-D streams, and maintains object-level models in real time for dynamic-scene robotics.

  • Problem

    Existing dynamic-scene approaches often treat moving regions as outliers, leaving their geometry and motion unmodeled despite the need for object-level robot interaction.

  • Method

    Co-Fusion uses motion or semantic segmentation with multiple independently tracked models, fusing only pixels labeled as belonging to each object.

  • Results

    Co-Fusion maintains an up-to-date background map and detailed 3D models for up to 5 objects at 12 frames per second.

  • Takeaways & Limitations

    The resulting object-level scene description could enable robots to maintain and interact with dynamic environments.

Abstract

from arXiv · show

In this paper we introduce Co-Fusion, a dense SLAM system that takes a live stream of RGB-D images as input and segments the scene into different objects (using either motion or semantic cues) while simultaneously tracking and reconstructing their 3D shape in real time. We use a multiple model fitting approach where each object can move independently from the background and still be effectively tracked and its shape fused over time using only the information from pixels associated with that object label. Previous attempts to deal with dynamic scenes have typically considered moving regions as outliers, and consequently do not model their shape or track their motion over time. In contrast, we enable the robot to maintain 3D models for each of the segmented objects and to improve them over time through fusion. As a result, our system can enable a robot to maintain a scene description at the object level which has the potential to allow interactions with its working environment; even in the case of dynamic scenes.

I. INTRODUCTION

Co-Fusion addresses dynamic-scene SLAM by segmenting independently moving or semantically relevant objects, tracking them, and reconstructing their 3D geometry over time. This object-level representation is intended to support robotic interaction with dynamic environments.

  • Dynamic scenes are harder than static environments because multiple objects may move independently, while conventional systems treat moving regions as outliers.
  • Robots need segmentation, independent tracking, and 3D geometry estimation for objects that matter in dynamic scenes.
  • Co-Fusion segments backgrounds and foreground objects using motion or semantic cues while simultaneously tracking and reconstructing their 3D geometry.
  • Motion segmentation groups consistently moving points, whereas instance segmentation detects and masks semantically specified objects, including static objects of interest.
  • Detected objects become active models whose geometry is updated by fusing only pixels labeled as belonging to each object.
  • The system targets object-level scene descriptions that can help robots discover unknown objects, learn their geometry through motion, and interact physically with dynamic scenes.

II. RELATED WORK

Prior dense reconstruction systems largely assume static scenes or focus on a single deformable object, leaving real-time reconstruction of multiple moving objects less explored. Co-Fusion differs by providing an online, real-time approach for multiple models.

  • Traditional dense SLAM and reconstruction systems commonly assume that the scene is largely static.
  • DynamicFusion and related systems reconstruct deformable objects in real time, but their setting centers on a single observed object.
  • Dense reconstruction of dynamic scenes containing multiple moving objects has received more limited attention than single-object or sparse dynamic-scene tracking.
  • Some prior methods track only one moving object, require manual initialization, or rely on human-provided segmentation without real-time operation.
  • Several RGB-only methods perform simultaneous segmentation, tracking, and reconstruction, but use sparse tracks or batch processing.
  • Compared with dense RGB multi-body reconstruction, Co-Fusion is online and real time rather than batch-based and several seconds per frame.

III. OVERVIEW OF OUR METHOD

Co-Fusion processes live RGB-D frames while maintaining a background model and independent models for segmented objects. Its pipeline tracks active models, segments pixels, and fuses object-labeled data into surfel-based geometry.

  • Co-Fusion processes each RGB-D frame in real time while maintaining detailed models for the background and independently tracked objects.
  • The system distinguishes active models, currently visible, from inactive models, previously visible but currently out of view.
  • After initializing with the background, each new frame passes through tracking, segmentation, and subsequent model-update steps.
  • Tracking: Active-model tracking estimates each model’s 6DOF pose using dense geometric ICP alignment and photometric color differences.
  • 12 frames per second is achieved while maintaining an up-to-date background map and detailed models for up to 5 objects.
  • The system maintains each model as a list of surfels containing position, normal, color, weight, radius, and timestamps.

V. TRACKING ACTIVE MODELS

Co-Fusion tracks each active model independently by registering current observations against its stored model. The tracking objective combines geometric ICP alignment with photometric consistency, extending single-model RGB-D tracking to multiple models in real time.

  • Each active model is tracked independently using only map points labeled as belonging to that model.
  • A. Energy: The tracking cost combines point-to-plane ICP alignment with a photometric term measuring brightness differences between projected model colors and the current frame.
  • A. Energy: Unlike prior RGB-D systems that track one static model, Co-Fusion tracks multiple models while maintaining real-time performance.

B. Geometry Term

Co-Fusion estimates each active model’s motion by aligning current depth and color observations with projections of its stored 3D model from the previous frame.

  • B. Geometry Term: The geometry term minimizes point-to-plane ICP error between current live depth vertices and the previous-frame predicted depth map for each model.The transformation aligns model m from frame t−1 with frame t.
  • B. Geometry Term: Tracking independently optimizes each active model’s rigid transformation using only 3D map points labeled as belonging to that model.
  • B. Geometry Term: The photometric term minimizes brightness differences between the live color frame and a synthesized view obtained by projecting stored 3D models.The synthesized view uses the current depth image, model geometry, and estimated rigid motions.
  • B. Geometry Term: A four-layer spatial pyramid embeds the optimization for robustness and efficiency in the GPU implementation.

VI. MOTION SEGMENTATION

Motion segmentation formulates model assignment as a super-pixel labeling problem over active rigid transformations and an outlier label, combining geometric fit with appearance-aware spatial coherence.

  • VI. MOTION SEGMENTATION: The system tracks active models’ global poses before segmentation and performs labeling at the SLIC super-pixel level for real-time CPU operation.Super-pixel position, color, and depth are computed by averaging constituent pixels.
  • VI. MOTION SEGMENTATION: The labeling assigns each current-frame point to an active model’s rigid motion or to an outlier, while the number of active models can change as objects appear or disappear.
  • VI. MOTION SEGMENTATION: Unary costs measure ICP alignment under each label’s rigid transformation, while the outlier receives a low cost only when no model explains the super-pixel motion.Missing projected geometry is assigned a fixed misalignment cost to prevent labels from expanding beyond object bounds.
  • VI. MOTION SEGMENTATION: In the handover sequence, the system models the background, teddy bear, and two arms, separating the left arm from the teddy when their motions diverge.
  • VI. MOTION SEGMENTATION: Pairwise costs encourage neighboring super-pixels with similar feature vectors to share labels using Potts and contrast-sensitive terms.Features encode 2D location, RGB color, and depth, and the contrast-sensitive terms use Gaussian kernels.
  • VI. MOTION SEGMENTATION: Soft label assignments are converted to hard assignments by selecting the maximum label, followed by connected-component merging, separation, and size filtering.

A. Addition of New Models

Co-Fusion adds models when sufficiently large outlier regions indicate new objects and removes duplicate or unsupported reconstructions as objects enter or leave the scene.

  • A. Addition of New Models: An outlier-connected region larger than 3% of the image pixels triggers spawning a new object label.
  • A. Addition of New Models: When a moving object was previously part of the background map, high-ICP-error regions are removed to reduce duplicate reconstruction.
  • A. Addition of New Models: A disappeared label is treated as having left the scene if it fails to reappear within a specified number of frames.
  • A. Addition of New Models: Models that leave the scene are retained only when they contain enough high-confidence surfels; otherwise they are deleted.

VII. OBJECT INSTANCE SEGMENTATION

Co-Fusion also uses semantic cues to segment both moving and static objects, applying an object-instance segmentation network before enforcing temporal consistency across frames.

  • VII. OBJECT INSTANCE SEGMENTATION: SharpMask predicts object proposals and masks simultaneously using feature extraction, segmentation, and objectness-scoring branches.
  • VII. OBJECT INSTANCE SEGMENTATION: SharpMask segmentations are temporally stabilized between consecutive frames before being passed to Co-Fusion.
  • VII. OBJECT INSTANCE SEGMENTATION: Semantic segmentation can target selected object classes, such as all tools on a table, using models pretrained on COCO.

VIII. FUSION

Co-Fusion tracks active object models by aligning their poses to the camera view, then updates each model by fusing newly available RGB-D data.

  • Active models are projected into the camera view using splat rendering to align their individual poses.
  • The fusion stage updates surfel maps by merging each new RGB-D frame into the existing object models.
  • Projective association links image coordinates to corresponding surfels before applying an update scheme similar to prior surfel-based approaches.

IX. EVALUATION

The evaluation uses synthetic and real RGB-D sequences with ground-truth geometry and trajectories, alongside demonstrations of continuous object tracking and refinement.

  • Quantitative evaluation covered synthetic and real sequences with ground-truth data for reconstruction and object trajectories.Synthetic ToyCar3 and Room4 sequences used Kinect-like noise; real scenes used markers, a NextEngine 3D scanner, and OptiTrack motion capture.
  • Table I reports average 3D reconstruction error, its standard deviation, and surfel percentages exceeding 1cm and 5cm errors for Esone1.
  • Table II reports AT-RMSE trajectory errors in millimeters for the synthetic sequences, including two trajectories for the split airship object.
  • Motion segmentation quality was evaluated with per-label, per-frame intersection-over-union on the ToyCar3 and Room4 synthetic sequences.
  • In the final demonstration, Co-Fusion continuously tracked and refined objects placed sequentially on a table.The demonstration highlights successful segmentation and is presented as useful for robotics applications involving actuator-moved objects.

X. CONCLUSIONS

The conclusion presents Co-Fusion as a real-time RGB-D SLAM system that segments, tracks, and models multiple objects using motion or semantic cues while maintaining an environmental model.

  • Co-Fusion segments scenes into multiple objects using motion or semantic cues while tracking and modeling them accurately.
  • Semantic labeling supports scenes that would be difficult for motion-based segmentation, using CNN-generated labels to highlight object labels in reconstruction.
  • The ToyCar3 visualization shows the stages of the Co-Fusion system on a synthetic sequence.
  • Trajectory comparisons and intersection-over-union plots evaluate estimated object trajectories and label quality across ToyCar3 and Room4 sequences.
  • The system maintains an environmental model alongside object models and is demonstrated in robotics and 3D scanning applications.
Loading 1706.06629v1…