Source-linked AI summary
DynaSLAM: Tracking, Mapping and Inpainting in Dynamic Scenes
Berta Bescos, José M. Fácil, Javier Civera, José Neira
TL;DR
The paper addresses the limits of static-scene assumptions in visual SLAM for populated environments. DynaSLAM adds dynamic-object detection and background inpainting to ORB-SLAM2 across monocular, stereo, and RGB-D configurations, achieving robust dynamic-scene tracking and static-map construction.
Problem
Static-scene assumptions limit visual SLAM in populated real-world environments where dynamic objects can enter the map and compromise its reuse.
Method
DynaSLAM extends ORB-SLAM2 with CNN-based segmentation, RGB-D multi-view geometry, and background inpainting for monocular, stereo, and RGB-D SLAM.
Results
DynaSLAM achieves robust tracking in dynamic scenarios and, in most comparisons, the highest accuracy while producing a static reusable map.
Takeaways & Limitations
The resulting static maps and inpainted frames support long-term mapping and applications such as augmented or virtual reality.
Abstract
from arXiv · showhide
The assumption of scene rigidity is typical in SLAM algorithms. Such a strong assumption limits the use of most visual SLAM systems in populated real-world environments, which are the target of several relevant applications like service robotics or autonomous vehicles. In this paper we present DynaSLAM, a visual SLAM system that, building over ORB-SLAM2 [1], adds the capabilities of dynamic object detection and background inpainting. DynaSLAM is robust in dynamic scenarios for monocular, stereo and RGB-D configurations. We are capable of detecting the moving objects either by multi-view geometry, deep learning or both. Having a static map of the scene allows inpainting the frame background that has been occluded by such dynamic objects. We evaluate our system in public monocular, stereo and RGB-D datasets. We study the impact of several accuracy/speed trade-offs to assess the limits of the proposed methodology. DynaSLAM outperforms the accuracy of standard visual SLAM baselines in highly dynamic scenarios. And it also estimates a map of the static parts of the scene, which is a must for long-term applications in real-world environments.
I. INTRODUCTION
Visual SLAM supports localization and mapping but commonly assumes a static environment, limiting use in populated dynamic settings. DynaSLAM extends ORB-SLAM2 with dynamic-object handling and background inpainting for reusable static maps.
- SLAM jointly estimates a map and robot pose, allowing continual localization without the drift accumulation of local-window odometry.
- The system removes dynamic content and reconstructs occluded background using information from previous views, producing a reusable static scene map.
- Static-environment assumptions limit visual SLAM in populated real-world environments and allow only small fractions of dynamic content to be treated as outliers.
- Dynamic-scene SLAM must prevent moving objects from influencing tracking or mapping and reconstruct map regions temporarily occluded by them.
- DynaSLAM adds an online front end to ORB-SLAM2 for monocular, stereo, and RGB-D SLAM, using CNN segmentation and RGB-D multi-view geometry.
II. RELATED WORK
Prior dynamic-scene SLAM methods detect changes, track known objects, or use motion and image differences, but their coverage is limited. DynaSLAM combines deep learning with multi-view geometry to address complementary kinds of dynamic content.
- Existing feature-based and direct SLAM methods commonly reject dynamic objects as outliers rather than explicitly modeling dynamic-scene content.
- Earlier approaches use map projection, known-object tracking, depth-edge weighting, scene flow, optical flow, or RGB-D and intensity differences.
- Sequence-only static mapping fails when an a priori dynamic object remains static, such as a parked car or seated person.
- DynaSLAM combines multi-view geometry and deep learning to detect both moving objects and changes involving objects that were previously static.
III. SYSTEM DESCRIPTION
DynaSLAM combines semantic segmentation, multi-view geometry, tracking, mapping, and background reconstruction across monocular, stereo, and RGB-D pipelines. Its synthetic outputs use static information from previous views to fill dynamic-object occlusions.
- A CNN first segments potentially dynamic content pixel-wise before mapping and tracking in the monocular and stereo pipelines.
- In RGB-D, multi-view geometry refines CNN masks and detects moving instances that are static most of the time.
- A low-cost tracker localizes the camera within the existing scene map, after which segmented frames support trajectory and map estimation.
- After dynamic-object detection and camera localization, the system reconstructs occluded background using static information from previous views.
- The stored sparse map supplies data to the pipeline, while inpainting follows tracking and mapping once the camera position is known.
A. Segmentation of Potentially Dynamic Content using a CNN
DynaSLAM uses Mask R-CNN to obtain pixel-wise semantic segmentation of potentially dynamic object classes. Its output combines per-instance binary masks into one dynamic-content segmentation.
- The CNN stage uses Mask R-CNN to obtain pixel-wise semantic segmentation, using pixel masks rather than instance labels in these experiments.
- Mask R-CNN receives the original RGB image and targets classes considered potentially dynamic or movable.
- For an m×n×3 RGB input, the network outputs an m×n×l matrix whose channels provide binary masks for detected objects.
- Combining all output channels yields a segmentation of the dynamic objects appearing in the image.
B. Low-Cost Tracking
DynaSLAM combines learning-based and multi-view geometric segmentation to identify dynamic content before tracking. Its geometric stage compares projected and observed depths, while combined masks address complementary detection failures.
- Low-Cost Tracking: The system excludes features in potentially dynamic regions and tracks the camera using only static image areas.It also removes features near segment contours, which tend to form high-gradient regions.
- Low-Cost Tracking: Five overlapping previous keyframes are selected as a compromise between computational cost and dynamic-object detection accuracy.Selection considers both distance and rotation relative to the current frame.
- Low-Cost Tracking: Multi-view geometry labels a keypoint dynamic when projected depth exceeds current depth by more than threshold τz.The threshold was set to τz = 0.4m by optimizing 0.7×Precision+0.3×Recall on 30 manually tagged TUM images.
- Low-Cost Tracking: Geometric and learning-based methods provide complementary detections: geometry can find moved normally static objects, while learning detects objects missed by geometry.The geometric method can miss distant people, whereas learning may miss movable objects such as carried books.
- Low-Cost Tracking: The final segmentation combines both approaches, using the geometric mask for jointly detected objects and retaining learning detections found only by the CNN.This combination is described as an effective way to improve tracking and mapping accuracy.
D. Tracking and Mapping
The system extracts ORB features only from regions classified as static and produces processed RGB-D outputs in which dynamic content is removed.
- Tracking and Mapping: ORB features are extracted from static image segments, while features at segment contours are removed.The contours are treated as high-gradient areas that can produce problematic keypoints.
- Tracking and Mapping: The qualitative output shows detected dynamic objects removed and the occluded background reconstructed in both RGB and depth channels.The input and output are displayed as RGB-D frame pairs.
E. Background Inpainting
DynaSLAM inpaints removed dynamic regions using static information from previous views, producing synthetic frames for static-scene applications. Some regions remain unreconstructed when prior views lack correspondence or valid depth.
- Background Inpainting: The system projects RGB and depth data from previous keyframes into current dynamic segments to reconstruct the occluded background.The experiments use the last 20 keyframes for this projection.
- Background Inpainting: Inpainted frames can support virtual and augmented reality, relocation, camera tracking, place recognition, and SLAM under a staticity assumption.The paper evaluates the system on the TUM RGB-D and KITTI datasets and compares it with state-of-the-art systems and ORB-SLAM2.
- Background Inpainting: Gaps remain blank when the corresponding scene area has not appeared previously or lacks valid depth information.Geometric methods cannot reconstruct these gaps without a more elaborate inpainting technique.
A. TUM Dataset
On the TUM RGB-D dataset, DynaSLAM combines learning and geometric motion detection to improve dynamic-scene tracking while producing maps without dynamic content.
- Dataset: The TUM RGB-D dataset contains 39 indoor sequences recorded at 30Hz with RGB, depth, and motion-capture ground truth.Walking sequences include two people moving in the foreground and background, making them challenging for standard SLAM.
- Variants: DynaSLAM (N+G), combining Mask R-CNN with multi-view geometry, is the most accurate variant in most evaluated sequences.The geometry-only variant has higher error because its segmentation becomes accurate only after a delay.
- Inpainting: Adding background inpainting before camera localization usually reduces tracking accuracy because reconstruction is strongly correlated with estimated camera poses.The authors therefore recommend performing inpainting after tracking, especially for rotational sequences.
- RGB-D comparison: DynaSLAM outperforms RGB-D ORB-SLAM2 in highly dynamic walking sequences while achieving similar error to ORB-SLAM2 in static scenarios.In low-dynamic sitting sequences, tracking is slightly worse, but DynaSLAM’s map excludes dynamic objects.
- State-of-the-art comparison: DynaSLAM significantly outperforms other state-of-the-art dynamic RGB-D SLAM systems across both high- and low-dynamic sequences, with errors generally around 1–2 cm.Its motion-detection approach also outperforms the other evaluated methods.
- Monocular comparison: In monocular evaluation, DynaSLAM initializes more quickly and can bootstrap with dynamic content, although its accuracy is slightly lower than ORB-SLAM.It produces a map without dynamic content for reuse in long-term applications.
B. KITTI Dataset
On KITTI urban and highway sequences, DynaSLAM shows similar monocular and stereo behavior, with greater benefits for monocular tracking in dynamic scenes.
- Dataset and evaluation: KITTI provides stereo sequences recorded from a car in urban and highway environments, evaluated with trajectory and relative motion error metrics.The study compares stereo DynaSLAM with stereo ORB-SLAM2 and also evaluates monocular variants.
- Configuration comparison: Monocular tracking is more sensitive to dynamic objects than stereo tracking, making DynaSLAM’s additions more influential in the monocular case.The overall results are otherwise similar between monocular and stereo configurations.
- Dynamic content: Removing features from a priori dynamic objects can improve tracking when vehicles in the sequence are moving.KITTI 01 and KITTI 04 are cited as examples where all appearing vehicles are moving.
- Static vehicles and mapping: When most vehicles are parked, absolute trajectory RMSE is usually larger because remaining tracking keypoints are farther away and often lie in low-texture areas.Despite this trade-off, the resulting map contains structural objects and supports more robust loop closure and relocalization.
- Future work: The authors identify distinguishing movable from currently moving objects using only RGB information as future work.A detected but stationary car could contribute to local tracking while remaining excluded from the map.
C. Timing Analysis
DynaSLAM’s dynamic-scene capabilities improve static-map creation but add computational cost and prevent real-time operation in its evaluated form.
- Timing Analysis: DynaSLAM is not optimized for real-time operation, although offline creation of lifelong static-scene maps remains relevant.The system is positioned for offline use when static-map creation is the priority.
- Timing Analysis: Table VII reports DynaSLAM’s average computational time in milliseconds for its different processing stages.The table provides the stage-level timing breakdown used in the analysis.
- Timing Analysis: The multi-view geometry stage adds further delay, mainly because of its region-growth algorithm.Background inpainting also introduces delay and should therefore follow tracking and mapping.
V. CONCLUSIONS
DynaSLAM extends visual SLAM to dynamic environments while producing reusable static maps. It achieves strong accuracy in dynamic RGB-D settings, with more mixed results in monocular and KITTI evaluations.
- Conclusions: DynaSLAM adds motion segmentation to ORB-SLAM and supports monocular, stereo, and RGB-D cameras while creating static, reusable maps.For RGB-D input, it can also synthesize RGB and depth frames with dynamic content removed and occluded backgrounds inpainted.
- Conclusions: DynaSLAM achieves the highest accuracy in most comparisons against state-of-the-art systems.This is the paper’s broad conclusion across its evaluated settings.
- Conclusions: DynaSLAM is the best RGB-D SLAM solution on the TUM Dynamic Objects dataset.In the monocular case, its accuracy is similar to ORB-SLAM while initialization is earlier and the resulting map is static.
- Conclusions: DynaSLAM is slightly less accurate than monocular and stereo ORB-SLAM on KITTI, except when dynamic objects occupy an important part of the scene.Its map contains only structural objects and can therefore be reused for long-term applications.
- Conclusions: Future extensions include real-time performance, an RGB-based motion detector, and more realistic synthesized RGB frames.The proposed improvements target computational speed, motion detection, and image-inpainting realism.