PaperlayerView exact revision on arXiv

Source-linked AI summary

FlowFusion: Dynamic Dense RGB-D SLAM Based on Optical Flow

Tianwei Zhang, Huayan Zhang, Yang Li, Yoshihiko Nakamura, Lei Zhang

arXiv:2003.05102v1cs.ROcs.CV

TL;DR

Dynamic objects make visual SLAM unreliable by occluding static features and disrupting camera-motion estimation. FlowFusion jointly estimates ego-motion, segments dynamic and static regions, and reconstructs static backgrounds using optical-flow residuals in a dense RGB-D pipeline. The method reports competitive performance in static and dynamic environments, while its module-free design extracts varied moving objects but is less sensitive to slight or very fast motion.

  • Problem

    Dynamic objects can occlude static environment features, producing insufficient feature associations and camera-motion estimation failures in visual SLAM.

  • Method

    FlowFusion jointly performs dense RGB-D ego-motion estimation, optical-flow-residual-based dynamic/static segmentation, and static-background reconstruction.

  • Results

    FlowFusion achieves competitive performance in static and dynamic environments, while outperforming other module-free dynamic SLAM methods on evaluated sequences.

  • Takeaways & Limitations

    The module-free approach can extract different kinds of moving objects rather than relying on object-specific detection modules.

  • Takeaways & Limitations

    FlowFusion is less sensitive to slight motions and very fast motions, for which optical-flow estimation can be wrong.

Abstract

from arXiv · show

Dynamic environments are challenging for visual SLAM since the moving objects occlude the static environment features and lead to wrong camera motion estimation. In this paper, we present a novel dense RGB-D SLAM solution that simultaneously accomplishes the dynamic/static segmentation and camera ego-motion estimation as well as the static background reconstructions. Our novelty is using optical flow residuals to highlight the dynamic semantics in the RGB-D point clouds and provide more accurate and efficient dynamic/static segmentation for camera tracking and background reconstruction. The dense reconstruction results on public datasets and real dynamic scenes indicate that the proposed approach achieved accurate and efficient performances in both dynamic and static environments compared to state-of-the-art approaches.

I. INTRODUCTION

Dynamic environments undermine visual SLAM because moving objects occlude static features and cause insufficient associations for camera-motion estimation. FlowFusion addresses this without assuming known dynamic object classes by using optical-flow-based segmentation.

  • Dynamic obstacles reduce reliable static visual features, causing insufficient associations and camera-motion estimation failures.
  • Semantic labeling and object-detection approaches can work effectively for particular dynamic objects but may lose robustness when unknown objects appear.
  • Flow methods detect slight motions and track non-rigid surfaces, but require complex penalty settings and can produce unclear segmentation boundaries.
  • FlowFusion uses optical-flow residuals for dynamic/static segmentation, enabling efficient dynamic extraction and accurate static-environment reconstruction without pre-known object hypotheses.

II. RELATED WORKS

Prior dynamic RGB-D SLAM methods extend dense static reconstruction through semantic labeling, outlier or scene-flow detection, and object-detection preprocessing. These approaches differ in their assumptions about which dynamic entities can be recognized.

  • Dense RGB-D fusion methods such as KinectFusion and ElasticFusion enabled real-time, robust, accurate reconstruction in static indoor environments.
  • Semantic labeling: Semantic approaches treat motion segmentation as labeling, with Co-Fusion combining deep segmentation and static dense reconstruction.
  • Outlier and scene-flow methods: Outlier-based methods identify dynamic point clouds within dense fusion, including joint motion segmentation and scene-flow estimation and static-background reconstruction.
  • Object detection: Object-detection pipelines remove potential dynamic objects before reconstructing scenes with static SLAM frameworks, including human-focused PoseFusion and SegNet with ORB-SLAM2.
  • Rigidity semantics: Rigidity-based work defines the static background point-cloud set as stationary and contrasts it with moving objects, while another approach learns background rigidity with deep networks.

III. OPTICAL FLOW BASED JOINT DYNAMIC SEGMENTATION AND DENSE FUSION

FlowFusion processes consecutive RGB-D frames through optical-flow estimation and robust ego-motion initialization, then uses projected 2D scene flow for dynamic segmentation before reconstructing static backgrounds.

  • The framework takes two continuous RGB-D frames and estimates RGB optical flow with PWC alongside an initial camera motion from intensity-depth pairs.
  • It warps frame A using the estimated motion, obtains projected 2D scene flow, and applies that flow to dynamic segmentation.
  • After several iterations, the resulting static backgrounds are used for reconstruction.

A. Visual Odometry in Dense RGB-D Fusion

The dense RGB-D visual-odometry front end combines photometric and geometric alignment, initializes motion over supervoxel clusters, and uses robust optimization before introducing optical-flow residuals for non-rigid motion.

  • The front end optimizes color and depth alignment errors between RGB-D frames.
  • The point cloud is over-segmented into supervoxel clusters, each treated as a rigid body, and an initial ξ is estimated from photometric and depth residuals.
  • Intensity and depth terms are weighted for measurement noise, while geometric residuals come from depth measurements.
  • The transformation T(ξ) represents camera rotation and translation, and W denotes image warping; RGB-D projection uses a pinhole camera model.
  • A Cauchy robust penalty balances optimization robustness and convergence, and coarse-to-fine iteratively reweighted least squares solves the nonlinear objective.
  • Because conventional residuals rely on rigid motion, FlowFusion defines optical-flow residuals to directly indicate non-rigid environmental motion.

B. Optical Flow Residual Estimated by Projecting the Scene Flow

The method removes camera ego-motion from optical flow to obtain projected 2D scene flow, whose residuals distinguish static from dynamic pixels and clusters. Iterative estimation improves scene-flow separation in dynamic scenes.

  • Motivation: Intensity and depth residuals are unreliable because RGB-D sensors suffer from imperfect cross-lens registration, boundary discretization, and range-dependent depth errors.These limitations motivate using optical flow residuals to indicate dynamic level directly.
  • Optical flow residual: Optical flow residuals are defined as projected 2D scene flow after subtracting camera ego-motion from optical flow.Optical flow is obtained from image pairs, while ego-motion is estimated from the RGB-D frames and used to project the scene flow.
  • Optical flow residual: Static pixels have near-zero residuals, whereas dynamic pixels have non-zero residuals whose magnitudes increase with moving speed.This residual behavior provides the dynamic cue used for segmentation.
  • Implementation: The pipeline computes optical flow with GPU-accelerated PWC-net instead of directly using the dense-flow formulation because dense optical-flow computation is time-consuming.The approach combines RGB-D ego-motion estimation, frame warping, and projected scene-flow computation.
  • Figure 3: In the illustrated dynamic scene, blue optical flows represent camera ego-motion; subtracting them reveals scene-flow components, with improved results after 7 iterations.Flow color encodes direction and intensity encodes pixel displacement.

C. Dynamic Clusters Segmentation

The segmentation method combines intensity, depth, and optical-flow residuals at supervoxel-cluster level, then minimizes an energy function to assign dynamic scores while encouraging neighboring clusters to agree.

  • Residual-based scoring: The method computes average cluster residuals and combines intensity, depth, and optical-flow residuals to measure each cluster’s dynamic level.The segmentation proceeds by first combining residuals and then optimizing cluster assignments.
  • Residual-based scoring: Each cluster receives a dynamic score b_i ∈ [0,1], where b_i = 0 denotes a definitely static cluster.The score is based on the cluster’s average residual and threshold relationship.
  • Threshold assignment: The assignment function maps average residuals below θ_b to 0, above θ_t to 1, and intermediate values linearly between the thresholds.This creates a continuous dynamic-static assignment from the residual range.
  • Spatial regularization: The cluster energy function includes adjacency-graph regularization to encourage neighboring clusters with similar properties to share dynamic or static labels.The graph connects supervoxel clusters through adjacency edges.
  • Integration with VO: The optimized dynamic scores modify the camera-motion objective, and the resulting equation is solved with an iteratively re-weighted least-square solver.This incorporates dynamic-static segmentation into visual odometry.

IV. DYNAMIC SLAM EXPERIMENTS AND EVALUATIONS

The experiments compare FlowFusion with dynamic SLAM baselines on TUM and HRPSlam data, evaluating trajectory estimation, dynamic segmentation, and background reconstruction. FlowFusion performs competitively overall, while its optical-flow segmentation remains sensitive to certain motion and scene conditions.

  • Evaluation setup: FlowFusion is evaluated against SF, JF, and PF on public TUM and HRPSlam datasets using VO and mapping results.The evaluation includes Absolute Trajectory Error (ATE) and Relative Pose Error (RPE).
  • Static sequences: In static sequences fr1/xyz and fr1/desk2, the VO performances of the four methods are similar because SF, PF, and FlowFusion are based on the EF framework.These methods converge to EF’s performance in static or slightly dynamic local areas.
  • TUM dynamic sequence: On TUM fr3/walking xyz, PF achieves the smallest trajectory errors, while FlowFusion outperforms the other model-free dynamic SLAM methods.The sequence contains 827 RGB-D images with two moving humans and slight object motions.
  • TUM dynamic sequence: PF obtains an ATE RMSE of 4.1 cm, whereas FlowFusion obtains 12 cm on TUM fr3/walking xyz.
  • Segmentation behavior: FlowFusion’s optical-flow residuals support dynamic foreground point-cloud extraction and static background segmentation without relying on object modules.Unlike PF, FlowFusion can extract different kinds of moving objects, not only humans.
  • Limitations: FlowFusion is non-sensitive to slight motions and very fast motions, because very fast motions can produce incorrect optical-flow estimates.In HRPSlam, fast rotation and shaking complicate optical-flow residual estimation, and sweeping feet may be assigned to the static background through graph connectivity.

V. CONCLUSIONS

The paper presents a dense RGB-D SLAM algorithm that jointly identifies dynamic segments and reconstructs static environments. Experiments on online datasets and real robotic scenes show competitive performance in static and dynamic environments.

  • FlowFusion jointly determines dynamic segments and reconstructs static environments through a dense RGB-D SLAM algorithm.
Loading 2003.05102v1…