Source-linked AI summary
ReFusion: 3D Reconstruction in Dynamic Environments for RGB-D Cameras Exploiting Residuals
Emanuele Palazzolo, Jens Behley, Philipp Lottes, Philippe Giguère, Cyrill Stachniss
TL;DR
Dynamic environments make consistent RGB-D localization and mapping difficult because moving objects can corrupt pose estimation and maps. ReFusion uses TSDF-based direct tracking with geometric dynamic filtering from registration residuals and free-space modeling, and its evaluations show robust tracking and accurate static-world models in highly dynamic scenes.
Problem
Dynamic objects can produce wrong correspondences, degrade pose estimation, and corrupt maps, while most mapping systems focus on static environments.
Method
ReFusion directly tracks RGB-D sensor pose against a color-augmented TSDF and filters dynamics using registration residuals plus explicit free-space representation.
Results
Evaluations on TUM RGB-D and Bonn dynamic datasets show robust sensor tracking and consistent, accurate models of static environments, with performance often matching or exceeding dense SLAM methods.
Takeaways & Limitations
The approach provides class-agnostic dynamic-scene mapping without explicitly tracking moving objects, while also releasing a dynamic dataset, ground truth, and open-source implementation.
Takeaways & Limitations
When out-of-range depth values are not distinguished from invalid measurements, dynamic objects may be incorrectly added to the model.
Abstract
from arXiv · showhide
Mapping and localization are essential capabilities of robotic systems. Although the majority of mapping systems focus on static environments, the deployment in real-world situations requires them to handle dynamic objects. In this paper, we propose an approach for an RGB-D sensor that is able to consistently map scenes containing multiple dynamic elements. For localization and mapping, we employ an efficient direct tracking on the truncated signed distance function (TSDF) and leverage color information encoded in the TSDF to estimate the pose of the sensor. The TSDF is efficiently represented using voxel hashing, with most computations parallelized on a GPU. For detecting dynamics, we exploit the residuals obtained after an initial registration, together with the explicit modeling of free space in the model. We evaluate our approach on existing datasets, and provide a new dataset showing highly dynamic scenes. These experiments show that our approach often surpass other state-of-the-art dense SLAM methods. We make available our dataset with the ground truth for both the trajectory of the RGB-D sensor obtained by a motion capture system and the model of the static environment using a high-precision terrestrial laser scanner. Finally, we release our approach as open source code.
I. INTRODUCTION
ReFusion addresses dynamic-environment SLAM by combining geometric filtering with TSDF-based mapping, producing static models and robust camera tracking. The paper evaluates the approach on existing and newly released highly dynamic RGB-D data.
- Dynamic objects can create wrong correspondences, impair pose estimation, and corrupt maps in SLAM.
- ReFusion is a dense indoor-mapping approach robust to dynamic elements that uses geometric processing rather than explicit semantic interpretation.
- The method detects dynamics from registration residuals and explicit free-space modeling, without relying on object classes or explicitly tracking moving objects.
- The authors evaluate ReFusion on the TUM RGB-D dataset and 24 highly dynamic scenes, releasing motion-capture trajectories, a terrestrial-laser-scanned static model, and open-source code.
- ReFusion achieves camera tracking on par with or better than state-of-the-art dense SLAM and produces a more accurate static 3D model than competing approaches.
II. RELATED WORK
Prior RGB-D mapping work uses TSDFs, surfels, direct frame registration, and several strategies for handling dynamics. ReFusion instead combines registration residuals with detected free space to identify dynamics without class-specific models or explicit object tracking.
- TSDF-based RGB-D mapping has produced detailed maps, while later methods address fixed-grid limits through compressed representations such as octrees.
- Surfel methods represent environments with disks carrying normals and radii, with deformation graphs enabling long-range map corrections through loop closures.
- Other approaches directly register consecutive RGB-D frames by minimizing photometric error and combining depth, color, and normal cues.
- Dynamic objects have been handled through outlier rejection, explicit modeling, segmentation, or explicit motion tracking.
- ReFusion uses registration residuals and detected free space to detect any kind of dynamics without specific classes, models, or explicit object tracking.
III. OUR APPROACH
ReFusion estimates sensor pose directly against a color-enhanced TSDF model, then integrates RGB-D measurements using the refined pose. Its geometric formulation combines depth and intensity alignment without rendering synthetic views.
- Pipeline: The pipeline first estimates pose directly from the TSDF, filters dynamic regions, refines the pose, and integrates depth and color measurements.The processing sequence uses an initial registration, dynamic-element detection, a second registration, and model integration.
- Model representation: The environment model stores truncated signed distances, voxel weights, and projected color, supporting robust updates, texturing, and intensity-based registration.Weights enable running weighted averages, while color supports both mesh texturing and registration.
- Model representation: Voxel hashing allocates only needed voxel blocks, enabling larger mapped scenes without fixing the maximum scene size in advance.Voxel processing is parallelized on the GPU.
- Pose estimation: Incoming RGB-D points are backprojected using camera depth and intrinsics, then aligned directly to the implicit SDF rather than a rendered synthetic view.The pose is represented as an SE(3) transformation with Lie-algebra updates converted through the exponential map.
- Pose estimation: The depth error uses the TSDF distance to the nearest surface, with trilinear interpolation applied to neighboring voxel SDF and intensity values.The model intensity is computed from stored RGB values and used alongside the SDF for alignment.
- Pose estimation: Color improves alignment through a photometric error between current-image intensity and intensity stored in corresponding model voxels.The joint objective weights intensity information relative to depth information and is minimized using coarse-to-fine GPU processing.
C. Dynamics Detection
Dynamics are detected from registration residuals and refined with depth-aware flood fill. The resulting mask excludes dynamic regions from a second registration and subsequent model integration.
- Mask refinement: Flood fill is implemented with a queue initialized by residual-mask pixels and expanded through neighboring pixels.The algorithm repeatedly adds eligible neighbors until the queue is empty.
- Residual-based detection: After initial registration, pixels whose residuals exceed threshold t form a binary mask for candidate dynamic regions.The residual histogram separates most residuals from those associated with dynamic parts of the environment.
- Mask refinement: The initial residual mask is eroded, expanded with depth-aware neighboring regions, and then dilated to recover omitted border pixels.Flood fill adds neighbors while their depth difference remains below threshold θ.
- Filtered integration: The refined mask excludes dynamic pixels during a second registration and during RGB-D integration into the model.The second registration can take up to twice as long as registration without dynamic handling.
D. Carving of Model and Free Space Management
ReFusion manages free space by marking reliably empty voxels and rejecting later occupancy there, while TSDF updates can remove objects that subsequently move away. Virtual depth can address invalid sensor measurements.
- Free-space management: Previously measured free voxels reject later occupied measurements because new occupancy there indicates a dynamic object entered the space.Free space is marked for nonoccluded voxels in the camera frustum outside the truncation region.
- Free-space management: When a previously mapped object moves away, weighted TSDF updates progressively mark its voxels as free after sufficiently persistent empty observations.The representation updates each voxel using a weighted average of current and stored values.
E. Limitations and Handling Invalid Measurements
ReFusion handles invalid RGB-D depth measurements by distinguishing out-of-range from non-measurable zeros, because free-space modeling depends on knowing which regions are empty.
- Free-space marking requires reliable knowledge that regions in the camera frustum are empty.
- Ignoring zero-valued depth measurements works correctly only when the entire scene lies within the sensor’s depth range.
- With out-of-range values, dynamic objects may be incorrectly integrated into the model, reducing reconstruction quality.
- A correction strategy builds a temporary model from consecutive frames and fills zero values with virtual depths generated from registered poses.
- The TUM sequences use correction for out-of-range depths, whereas the authors’ dataset uses the simpler option because its depths remain within sensor range.
IV. EXPERIMENTAL EVALUATION
The evaluation tests ReFusion against geometric and neural-network-assisted dense SLAM methods on TUM RGB-D and a purpose-built highly dynamic dataset, using standard implementations and parameters.
- The experiments claim that ReFusion operates in highly dynamic environments using only geometric information and produces static-only dense models.
- Comparisons include StaticFusion, DynaSLAM, and MaskFusion across dynamic TUM RGB-D scenes and the authors’ highly dynamic dataset.
- All methods use default open-source parameters, while ReFusion’s defaults were selected empirically and similar values produced comparable results.
- The comparison separates geometry-only methods from methods that also use neural networks, emphasizing class-agnostic approaches.
A. Performance on TUM RGB-D Dataset
On the TUM RGB-D dataset, ReFusion generally tracks comparably to dense SLAM methods and consistently reconstructs the environment, though specific dynamic sequences expose failures and artifacts.
- DynaSLAM outperforms the other evaluated methods on six dynamic TUM RGB-D sequences, but does not provide a dense model.
- The dense approaches usually achieve similar results, except StaticFusion loses track on walking halfsphere and ReFusion performs worse on sitting halfsphere.
- ReFusion consistently creates a coherent environmental mesh across the evaluated TUM sequences.
- On walking xyz, a person remains in the model because the stopping location is never revisited, preventing free-space verification.
- ReFusion remains on par with state-of-the-art dense mapping approaches for tracking while using TSDF rather than surfel representations.
B. Performance on Bonn RGB-D Dynamic Dataset
The Bonn RGB-D dataset evaluates dense SLAM methods across highly dynamic scenes, including people, balloons, and box manipulation. ReFusion is strongest in people-crowded scenes and remains comparatively robust across failures.
- Dataset: The dataset contains 24 highly dynamic scenes involving tasks such as manipulating boxes or playing with balloons, which can obstruct the camera.It was recorded with an ASUS Xtion Pro LIVE sensor and Optitrack Prime 13 motion capture system, with a terrestrial laser scanner providing static-environment ground truth.
- Tracking behavior: Tracking error is particularly high during the sequence’s opening four seconds, when the camera tracks the person, and drops substantially afterward.
- Comparative performance: ReFusion performs best on scenes crowded with people, among the most challenging sequences without semantic segmentation.On box-manipulation sequences, ReFusion is better in about half the cases, while DynaSLAM is better in the others.
- Comparative performance: DynaSLAM outperforms dense approaches on uniformly colored balloon scenes because it cannot detect balloon features, which therefore do not affect SLAM performance.
- Comparative performance: ReFusion’s worst performance is on par with DynaSLAM (N+G) and substantially better than other geometric approaches, indicating greater robustness to failure.DynaSLAM (N+G) performs best in most cases, partly because every sequence contains people and its segmentation therefore helps consistently.
C. Model Accuracy
Model accuracy is assessed against a high-resolution terrestrial-laser-scanner point cloud of the static environment. Across both evaluated sequences, ReFusion reconstructs a more accurate model with fewer dynamic artifacts.
- Ground-truth construction: The ground-truth model is a high-resolution point cloud of the static test environment acquired with a Leica BLK360 terrestrial laser scanner.It is aligned to the motion-capture reference frame using tilt-and-turn targets observed by both systems.
- Ground-truth construction: Algorithm-generated models are transformed into the motion-capture reference frame and aligned using environmental markers known in both coordinate frames.
- Qualitative comparison: In the crowd3 qualitative comparison, ReFusion does not show the dynamic-object artifacts visible in the competing model.The competing model contains residual dynamic elements represented by red points.
- Quantitative comparison: Across both evaluated sequences, the cumulative distance plots show that ReFusion’s reconstructed model is more accurate than StaticFusion’s.The comparison uses the cumulative percentage of model points at a given distance from ground truth.
- Overall finding: The evaluation concludes that ReFusion robustly tracks an RGB-D sensor in highly dynamic environments while producing a consistent and accurate static-environment model.
VI. CONCLUSION
The conclusion presents ReFusion as a TSDF-based approach for camera tracking in dynamic environments and consistent reconstruction of the static world. Experiments report improved pose estimation relative to dense SLAM approaches, alongside public dataset and implementation releases.
- Conclusion: ReFusion tracks camera pose in dynamic environments and builds a consistent 3D model of the static world using a TSDF-based mapping approach.
- Conclusion: The method directly exploits TSDF and encoded color information for tracking, while filtering dynamics through registration residuals and explicit free-space representation.
- Conclusion: Experiments on TUM RGB-D and the Bonn RGB-D dynamic dataset show improved pose estimation amid dynamic elements compared with dense SLAM approaches.
- Conclusion: The authors publicly release the Bonn dataset and the open-source implementation.