Source-linked AI summary
RoSe-SLAM: Robust Semantic-Aware Gaussian Splatting SLAM from Dynamic Monocular Videos
Wenting Wang, Jiaxin Guo, Wenzhen Dong, Yun-Hui Liu, Charlie C. L. Wang, Yeung Yam
TL;DR
Dynamic scenes undermine SLAM systems that assume rigidity, while dense annotations and unreliable depth hinder practical reconstruction. RoSe-SLAM combines semantic Gaussian fields, spatial-temporal motion masks, occlusion-aware keyframes, and multi-view semantic consistency from uncalibrated monocular RGB video. Experiments on dynamic TUM, Bonn, and Wild-Mocap report superior trajectory estimation and static-scene mapping against existing dynamic RGB SLAM baselines.
Problem
Dynamic objects, occlusions, costly dense annotations, and unreliable depth challenge robust SLAM and static-scene reconstruction in real-world environments.
Method
RoSe-SLAM distills foundation-model semantic features into Gaussian fields and combines motion masks, occlusion-aware keyframes, and multi-view semantic consistency for dynamic monocular SLAM.
Results
Experiments on dynamic TUM, Bonn, and Wild-Mocap report superior camera tracking and static-scene reconstruction compared with existing dynamic RGB SLAM methods.
Takeaways & Limitations
Semantic-aware Gaussian representations support robust pose estimation, dynamic-object filtering, occlusion inpainting, and high-quality static geometry in dynamic scenes.
Abstract
from arXiv · showhide
In dynamic and unstructured environments, conventional SLAM systems generally suffer from significant accuracy degeneration due to their static assumptions. In this work, we propose Robust Semantic-aware Gaussian Splatting SLAM (RoSe-SLAM), to address the dynamic challenge by a holistic semantic scene understanding from uncalibrated monocular inputs, achieving accurate camera tracking and high-quality geometry reconstruction. Unlike conventional semantic SLAM using handcrafted semantic labels, our RoSe-SLAM exploits the semantic feature from 2D foundation model to enhance the dynamic tracking and mapping performance. By distilling the rich semantic features to our Gaussian fields, our method effectively identifies dynamic distractors and achieves semantic-aware multi-view consistency, significantly enhancing the geometric reconstruction and scene inpainting. Specifically, we propose a spatial-temporal motion mask generation module, enabling both long-term motion monitoring and short-term transient dynamics capturing, achieving robust and effective disentanglement of dynamic objects and static backgrounds. During global bundle adjustment, we propose an occlusion-aware keyframe selection mechanism to prioritize the occlusion as metric to pick the keyframes, and a multi-view semantic consistency module to improve the mapping quality in dynamic environments. By combining geometric motion cues with semantic priors, our system dynamically filters unreliable observations and reconstructs accurate static scene geometry. Extensive experiments conducted on benchmark datasets including dynamic TUM, Bonn and Wild-Mocap datasets, demonstrate that our method achieves superior performance in both trajectory estimation and static scene mapping, outperforming existing dynamic RGB SLAM baselines in long-term dynamic indoor environments.
I. INTRODUCTION
RoSe-SLAM addresses dynamic-scene SLAM failures by combining semantic scene understanding from uncalibrated monocular RGB input with motion-aware filtering and semantic multi-view consistency. Its modules target dynamic-object disentanglement, occlusion inpainting, and improved geometric reconstruction.
- Dynamic objects and occlusions violate conventional SLAM’s rigidity assumption, causing tracking failures and chaotic mapping in real-world scenes.
- Dense per-pixel annotations for removing dynamic objects are labor-intensive and difficult to scale to long sequences or large scenes.
- RoSe-SLAM learns semantic Gaussian fields from foundation-model features to promote consistent semantic representations across viewpoints.
- Spatial-temporal motion masks capture long-term motion and short-term dynamics to disentangle dynamic objects from static backgrounds.
- Occlusion-aware keyframe selection and multi-view semantic consistency improve scene completeness, rendering, camera pose estimation, and mapping.
II. RELATED WORK
Prior dynamic SLAM methods face limitations from motion scale, domain gaps, training-data demands, or weak static-map fidelity. RoSe-SLAM uses semantic 3D Gaussians to jointly represent scene appearance, geometry, and semantics for dynamic reconstruction.
- Residual-optimization methods handle small motions but struggle with sustained, large dynamic changes.
- Segmentation- and detection-based methods are more adaptable but can suffer from domain gaps and prediction errors in real-world deployment.
- End-to-end learning methods improve dynamic motion estimation but require large-scale training data and may produce low-fidelity static maps.
- RoSe-SLAM models dynamic scenes with semantic 3D Gaussians to support robust pose estimation and reconstruction.
- Semantic features are rendered alongside color and depth, while low-dimensional rendering followed by decoding reduces the cost of high-dimensional semantic representations.
B. Coarse Initialization
RoSe-SLAM obtains coarse depths, camera poses, and intrinsics from RGB input, then refines monocular depth using a metric-depth foundation model.
- Coarse depths, camera poses, and intrinsics provide initialization but contain noise and lack ground-truth precision.
- A metric-depth foundation model produces fine-grained, scale-consistent pseudo-depth for each frame.
C. Spatial-Temporal Motion Mask
RoSe-SLAM combines long-term keyframe alignment with local motion cues to detect objects that are transiently or persistently movable. It then uses geometric matching and epipolar consistency to construct motion masks that separate dynamic objects from static backgrounds.
- Motion-mask motivation: Spatial-temporal adaptive masking addresses movable objects that remain static across consecutive frames but move over longer periods.The method uses long-term and overlapping frames from a global perspective rather than relying only on short sliding windows.
- Keyframe selection: Keyframe candidates are selected to provide sufficient spatial overlap while maximizing temporal separation from the current frame.The selection balances overlap O(I_t, I_i) and time interval ΔT(i, t) through λ.
- Geometric motion cues: The warp-based mask uses camera intrinsics, relative poses, pseudo-depth, optical flow, and Sampson distance to distinguish ego-motion from dynamic-object motion.The fundamental matrix is estimated from inlier matches, and correspondence errors are evaluated against epipolar lines.
- Geometric motion cues: Correspondences whose Sampson distance exceeds the inlier threshold are marked as violating the static-background epipolar constraint and therefore treated as dynamic.The indicator function produces the warp mask from the thresholded correspondence distance.
- Mask fusion: The final motion-mask construction combines global and local motion information with segmentation-based masking.The supplied passage identifies global, local, and segmentation-based masks as components of the final mask M_t.
1) Occlusion-aware Keyframe Selection:
Random keyframe supervision can undersample frequently occluded regions in dynamic scenes. RoSe-SLAM therefore favors keyframes that expose occluded areas, improving their optimization and inpainting.
- Occlusion-aware selection: Random keyframe selection may provide insufficient supervision for regions that are frequently occluded or only partially visible.Such regions may not appear consistently across randomly selected keyframes, producing suboptimal optimization.
- Occlusion-aware selection: Selecting keyframes where occluded regions are clearly visible enables more effective optimization of those areas from the current viewpoint.The stated consequence is improved overall inpainting performance.
- Occlusion-aware selection: During selection, the current frame is warped into the candidate keyframe, while motion masks are warped and combined for occlusion-aware supervision.The notation defines k as the selected keyframe and ⊙ as element-wise multiplication.
2) Semantic-aware Multiple-view Consistency:
RoSe-SLAM enforces semantic agreement across views by comparing rendered semantic features after camera-based warping. This view consistency also helps identify noisy Gaussian geometry within sufficiently covered regions.
- Semantic consistency: Semantic foundation features can be inconsistent across viewpoints because occlusions and perspective changes alter pixel interpretations.The method addresses this issue with pixel embeddings and view-consistent semantic comparison.
- Semantic consistency: Rendered semantic features from the current image are warped into a previous frame, and their feature-space distance is minimized against the previous rendering.The comparison uses the estimated camera transformation T and targets small semantic distances for corresponding observations.
- Semantic consistency: View-consistent semantics help identify noise points and artifact floaters because adjacent pixels within one semantic category generally have small spatial-distance variations.The comparison is applied to pixels with sufficient Gaussian coverage, represented by the silhouette mask.
E. Semantic-aware SLAM System
The system uses high-level semantic features to strengthen multi-view geometric consistency and improve reconstruction of backgrounds hidden by dynamic objects. Its Gaussian map is progressively expanded in newly observed or low-silhouette regions for efficiency.
- Semantic-aware SLAM system: High-level semantic features enhance multi-view geometry consistency and improve inpainting of occluded background regions.The tracking and mapping processes are jointly optimized in an interleaved system.
- Mapping process: The map is initialized from the first frame and progressively adds Gaussians in newly observed areas and low-silhouette regions.This accelerates optimization while reducing the number of Gaussians, improving time and storage efficiency.
- Mapping process: Rendered depth is compared with pseudo-depth using per-pixel confidence weights in the mapping process.The supplied notation identifies ˆD_t as rendered depth, ˜D_t as pseudo-depth, and W_t as its confidence weight.
2) Tracking Process:
RoSe-SLAM refines camera poses by minimizing color, depth, and semantic rendering losses while keeping Gaussian parameters fixed. It is evaluated on challenging dynamic datasets using trajectory error and qualitative rendered-image/depth comparisons.
- Tracking optimization: During tracking, Gaussian parameters remain fixed while camera poses are optimized through coarse-to-fine relative pose refinement.The pose is updated by minimizing differences between ground-truth and differentially rendered color, depth, and semantic views.
- Evaluation setup: The evaluation uses TUM, Bonn, and Wild-Mocap datasets representing highly dynamic indoor environments with movable humans and objects.Pose estimation uses RMSE of the Absolute Trajectory Error (ATE).
- Qualitative evaluation: Qualitative TUM comparisons assess rendered images and depths, with both geometry and photometric results demonstrating effectiveness against dynamic objects.The comparison focuses on how the method handles dynamic content in rendered RGB and depth outputs.
A. Evaluation of mapping performance
RoSe-SLAM is evaluated for static mapping and tracking in dynamic scenes using quantitative and qualitative comparisons. It reports strong reconstruction and tracking performance, including distractor removal and consistent results across TUM and Wild-Mocap sequences.
- A. Evaluation of mapping performance: Quantitative Bonn evaluations show state-of-the-art reconstruction across accuracy, completeness, and completion ratio against GS-based RGB and RGBD SLAM methods.The reconstructed static maps also exhibit high visual fidelity in qualitative comparisons.
- B. Evaluation of tracking performance: Tracking comparisons cover classical, NeRF-based, Gaussian-splatting, and monocular SLAM systems across dynamic and static environments.The evaluated baselines include ORB-SLAM3, ReFusion, Co-fusion, MID-fusion, EM-fusion, iMAP, NICE-SLAM, Vox-Fusion, Co-SLAM, ESLAM, DGS-SLAM, Rodyn-SLAM, DG-SLAM, GOSLAM, NICER-SLAM, GS-SLAM, and DDN-SLAM.
- A. Evaluation of mapping performance: Bonn qualitative comparisons show that RoSe-SLAM removes distractors and occlusions while rendering high-quality images without blurry artifacts.The comparison is presented through rendered images and depths.
- A. Evaluation of mapping performance: The Bonn reconstruction table reports results on several dynamic sequences and marks tracking failures with “X”.The table is used to compare reconstruction performance across dynamic-scene sequences.
- B. Evaluation of tracking performance: RoSe-SLAM consistently surpasses state-of-the-art methods on Wild-Mocap and demonstrates superior distractor removal on TUM and Bonn.The reported tracking performance is attributed to semantic-aware multi-view consistency and coarse-to-fine camera tracking.
- B. Evaluation of tracking performance: The TUM tracking table reports Absolute Trajectory Error in centimeters, using “X” for tracking failures and “-” for absent mentions.Results are reported for several dynamic scene sequences.
C. Ablation study
RoSe-SLAM’s ablations show that its proposed components improve camera tracking and mapping quality in dynamic TUM and Bonn sequences. The method also reports superior tracking speed under a fixed evaluation protocol.
- Component effectiveness: Ablations on TUM and Bonn sequences show that all proposed methods improve camera tracking, measured by average ATE.The sequences include predictable pedestrian motions and unpredictable motions from chairs and balloons.
- Mapping quality: Semantic-aware multi-view consistency and semantic-guided keyframe selection substantially enhance mapping quality.These components also positively affect overall system capability in the reported ablations.
- Evaluation setup: Table IV reports RoSe-SLAM ablation studies on Bonn and TUM using ATE in centimeters as the camera-tracking metric.
- Time consumption analysis: RoSe-SLAM achieves superior tracking speed compared with existing approaches under an identical protocol with 40 tracking and 60 mapping iterations per frame.The reported per-frame computational cost excludes semantic-segmentation time and uses efficient pose estimation and 3D Gaussian Splatting rendering.
- Conclusion: The overall system is reported to improve pose estimation accuracy and robustness while producing higher-quality reconstructed maps and rendering images in dynamic scenes.