Source-linked AI summary
SGS-SLAM: Semantic Gaussian Splatting For Neural Dense SLAM
Mingrui Li, Shuhong Liu, Heng Zhou, Guohao Zhu, Na Cheng, Tianchen Deng, Hongyu Wang
TL;DR
Neural implicit and sparse semantic SLAM methods face oversmoothing, limited object disentanglement, and constraints in reconstruction quality, speed, and memory usage. SGS-SLAM uses multi-channel Gaussian optimization with semantic supervision and semantic-guided keyframe selection, achieving state-of-the-art tracking and mapping with rapid rendering and precise semantic labeling. Its effectiveness depends on accessible depth and 2D semantic inputs, and large scenes incur high memory consumption.
Problem
Neural implicit and sparse semantic SLAM methods struggle with over-smoothed object edges, object disentanglement, reconstruction speed, high-fidelity modeling, and memory usage.
Method
SGS-SLAM represents scenes with 3D Gaussians, jointly optimizing appearance, geometric, and semantic channels using 2D semantic labels and semantic-guided keyframe selection.
Results
SGS-SLAM achieves state-of-the-art tracking and mapping results while maintaining rapid rendering speeds, precise 3D semantic labeling, and high-fidelity dense reconstruction.
Takeaways & Limitations
The resulting high-quality reconstruction and precise semantic labeling support downstream scene editing and manipulation by grouping Gaussians according to object labels.
Takeaways & Limitations
SGS-SLAM depends on depth and 2D semantic inputs, and its memory consumption is large when deployed to large scenes.
Abstract
from arXiv · showhide
We present SGS-SLAM, the first semantic visual SLAM system based on Gaussian Splatting. It incorporates appearance, geometry, and semantic features through multi-channel optimization, addressing the oversmoothing limitations of neural implicit SLAM systems in high-quality rendering, scene understanding, and object-level geometry. We introduce a unique semantic feature loss that effectively compensates for the shortcomings of traditional depth and color losses in object optimization. Through a semantic-guided keyframe selection strategy, we prevent erroneous reconstructions caused by cumulative errors. Extensive experiments demonstrate that SGS-SLAM delivers state-of-the-art performance in camera pose estimation, map reconstruction, precise semantic segmentation, and object-level geometric accuracy, while ensuring real-time rendering capabilities.
1 Introduction
SGS-SLAM replaces implicit neural scene representations with an explicit Gaussian-based system that jointly optimizes appearance, geometry, and semantics. Semantic supervision and keyframe selection support high-fidelity reconstruction, segmentation, tracking, and object manipulation.
- Motivation: NeRF-based SLAM can oversmooth object edges, hinder object disentanglement, suffer catastrophic forgetting in larger scenes, and be computationally inefficient.These limitations affect fine-grained mapping, scene editing, and scalability.
- Representation: SGS-SLAM uses explicit 3D Gaussians for fast rendering, direct parameter optimization, and dynamic multi-channel feature rendering.The representation supports real-time rendering of appearance, depth, and semantic color.
- Results: Experiments report superiority over neural implicit semantic SLAM systems in rendering speed, reconstruction quality, and segmentation accuracy.The evaluation covers synthetic and real-world benchmarks for mapping, tracking, and semantic segmentation.
- Optimization: SGS-SLAM jointly optimizes appearance, geometric, and semantic signals for camera tracking and scene reconstruction.The system maps 2D semantic priors into 3D and optimizes them with the scene representation.
- Tracking: Semantic and geometric criteria guide keyframe selection by recognizing objects previously observed along the trajectory.This strategy is intended to improve tracking and reduce reconstruction errors from cumulative drift.
- Applications: Disentangled 3D object representations support moving, rotating, or removing objects by grouping Gaussians according to semantic labels.This provides a basis for downstream scene editing and manipulation.
2 Related Work
Semantic SLAM seeks to combine scene understanding with geometric mapping, but sparse semantic representations and neural implicit methods have limitations in complexity, speed, fidelity, and memory. SGS-SLAM adds semantic supervision to Gaussian-based SLAM through keyframe selection and semantic feature loss while preserving real-time processing.
- Semantic SLAM: Semantic SLAM supports applications in robotics and VR or AR by integrating semantic information into 3D geometric representations.The surveyed systems treat semantic information as an important component of SLAM.
- Semantic SLAM: Voxel, point-cloud, and signed-distance-field methods can simplify complex environments and limit reconstruction speed, fidelity, and memory efficiency.Their sparse 3D semantic expressions may not capture the full range of objects and relationships in a scene.
- Neural Implicit SLAM: Neural implicit SLAM methods use representations such as single or multiple MLPs, but single-MLP approaches show limitations in large-scale scenes.Related methods also include hierarchical MLPs, pixel-set tracking, and multi-resolution hash encoding.
- Gaussian-Based SLAM: SGS-SLAM addresses the semantic gap in Gaussian-based SLAM by using semantic maps for keyframe selection and semantic feature loss during tracking and mapping.The method reports higher-quality segmentation while preserving real-time processing performance.
3 Method
SGS-SLAM represents scenes with Gaussian primitives carrying geometric, appearance, and semantic channels, jointly optimized through differentiable rendering. Its tracking and mapping pipeline uses visibility-aware rendering, semantic supervision, multi-channel losses, and geometry- and semantics-guided keyframe selection.
- Multi-Channel Gaussian Representation: Each Gaussian is parameterized by opacity, center position, radius, and RGB color, then projected into images through differentiable splatting and front-to-back volume rendering.Camera intrinsics and extrinsics determine the projected Gaussian center, radius, and depth.
- Multi-Channel Gaussian Representation: Depth, color, and semantic maps are rendered by weighting Gaussian channels with projected influence and front-to-back occlusion, enabling joint multi-channel optimization.Semantic colors are optimized jointly with appearance color and depth during mapping.
- Tracking and Mapping: Visibility-aware silhouette rendering supports camera tracking and identifies pixels lacking sufficient information for introducing new Gaussians during reconstruction.Setting Gaussian depth to one produces a silhouette used to assess visibility; low-information pixels can receive new map elements.
- Tracking and Mapping: Tracking refines camera poses, while mapping densifies and optimizes the Gaussian scene using color, depth, and semantic losses restricted to sufficiently visible pixels.The silhouette threshold excludes pixels with insufficient visibility certainty from the loss calculation.
- Keyframe Selection and Optimization: SGS-SLAM selects keyframes using geometric overlap and semantic-map differences, and its explicit multi-channel representation targets faster rendering and improved reconstruction and segmentation.Geometric filtering removes candidates below a coverage threshold, while semantic filtering prefers views with low semantic overlap.
- Multi-Channel Gaussian Representation: SGS-SLAM models each scene with Gaussian means for geometry, appearance colors, and semantic colors, jointly optimized while camera poses remain fixed during mapping.The representation supports three coordinated channels for reconstruction and semantic understanding.
4 Experiment
Experiments on Replica, ScanNet, and ScanNet++ evaluate reconstruction, tracking, semantic segmentation, novel-view synthesis, scene manipulation, and ablations. SGS-SLAM generally achieves stronger reconstruction, tracking, segmentation, and editing outcomes, while multi-channel and keyframe optimization materially support performance.
- Experimental setup: Experiments use Replica, ScanNet, and ScanNet++ to evaluate reconstruction, camera pose, semantic segmentation, and downstream scene manipulation.Reconstruction uses PSNR, Depth-L1, SSIM, and LPIPS; tracking uses ATE RMSE; segmentation uses mIoU.
- Mapping and localization: 10dB in PSNR separates SGS-SLAM from baseline methods on Replica reconstruction evaluation.The paper reports state-of-the-art reconstruction performance and a 10dB PSNR margin over baselines.
- Mapping and localization: 70% lower depth loss and 34% lower ATE RMSE are reported on Replica tracking evaluation.The comparison uses average outcomes across eight scenes and also evaluates rendering speed using FPS.
- Semantic segmentation: Over 90% mIoU is achieved on four Replica scenes, outperforming semantic NeRF-based approaches.The reported mIoU compares rendered and ground-truth 2D semantic images in training views.
- Keyframe optimization: Keyframe optimization and semantic constraints improve novel-view reconstruction when later inaccurate poses could disrupt earlier object reconstructions.The ScanNet scene0000 example revisits objects including a bike and guitar at different trajectory stages.
- Scene manipulation: Semantic masks support removing, moving, and rotating individual objects or groups without training or fine-tuning.The demonstrated edits include a jar and flowers, with grouped transformations also applied to a table and its objects.
- Ablation study: Removing appearance, depth, or semantic supervision degrades the corresponding rendering, geometric, segmentation, tracking, or mapping capabilities.The ablation identifies depth as important for depth estimation and semantic input as important for segmentation and tracking/mapping.
- Ablation study: Removing either geometric or semantic keyframe constraints, or uncertainty weighting, reduces tracking and mapping performance.The full two-level keyframe selection strategy performs better than variants omitting these components.
5 Conclusion and Limitations
The paper concludes that SGS-SLAM combines Gaussian representation with appearance, geometric, and semantic constraints for dense semantic SLAM. It reports strong tracking, mapping, rendering, labeling, and editing support, while identifying input dependence and large-scene memory use as limitations.
- Conclusion: SGS-SLAM combines appearance, geometric, and semantic constraints in a 3D Gaussian representation for dense semantic SLAM.The system targets semantic segmentation, dense map reconstruction, and camera pose estimation.
- Conclusion: State-of-the-art tracking and mapping results are reported alongside rapid rendering speeds.The conclusion also reports high-quality reconstruction and precise 3D semantic labeling.
- Downstream applications: High-quality reconstruction and precise 3D semantic labeling provide a foundation for scene editing and applications in robotics or mixed reality.The paper frames these outputs as priors for downstream tasks.
- Limitations: SGS-SLAM depends on depth and 2D semantic inputs, and its memory consumption becomes large in large scenes.The paper identifies both issues as objectives for future research.
6 Experiment Settings
The experiments use fixed implementation settings across datasets with dataset-specific iteration counts and standard baseline configurations. Additional tables report ATE RMSE on Replica, ScanNet, and ScanNet++.
- Implementation: Experiments run on an NVIDIA A100-40GB GPU, while studied scenes typically require less than 12 GB of memory.The reported memory requirement is described as compatible with GPUs exceeding that capacity.
- Implementation: Tracking uses Tsil = 0.99, with depth, color, and semantic loss weights λD = 1.0, λC = 0.5, and λS = 0.05.The semantic loss weight is comparatively low because real-world semantic labels are typically noisy.
- Implementation: Tracking and mapping iterations are dataset-specific: 40 and 60 for Replica, 120 and 40 for ScanNet, and 220 and 50 for ScanNet++.The ScanNet++ setting reflects larger camera transitions between frames.
- Baselines: Baselines use their reported default configurations, with consistent tracking and mapping metrics across systems.For unavailable implementations, baseline results are taken from their published papers.
- Additional evaluation: ATE RMSE [cm] is reported for Replica, selected ScanNet scenes, and selected ScanNet++ scenes.The corresponding tables provide dataset-specific comparisons with baselines.
7 Additional Experiment Results
Across Replica, ScanNet, and ScanNet++, SGS-SLAM achieves accurate camera tracking, detailed semantic segmentation, and high-fidelity novel-view rendering. Its semantic Gaussian representation also supports object-level removal, grouping, translation, and rotation, although edits can leave holes where geometry was unobserved.
- Camera Tracking: SGS-SLAM achieves state-of-the-art ATE RMSE tracking on Replica, ScanNet, and ScanNet++, including robust tracking where ESLAM fails on large camera transitions.The evaluation covers comparisons with Vox-Fusion, NICE-SLAM, Co-SLAM, ESLAM, and Point-SLAM.
- Semantic Segmentation: SGS-SLAM produces accurate, detailed semantic segmentation on Replica, while DNS-SLAM shows edge difficulties associated with NeRF oversmoothing.The comparison is visualized against DNS-SLAM using Replica frames.
- Novel View Rendering: Across Replica, ScanNet, and ScanNet++, SGS-SLAM consistently delivers high-quality novel-view rendering and precise segmentation, while ESLAM struggles on ScanNet++.The ScanNet and ScanNet++ comparisons report high-fidelity reconstruction and, for ScanNet++, instance-level segmentation.
- Scene Manipulation: Semantic masks let SGS-SLAM remove individual objects while preserving attached items, or translate and rotate grouped objects together.The demonstrated example removes a table while preserving objects on it, then moves and rotates the group.
- Scene Manipulation: Object edits can leave holes because unobserved geometry is missing from the explicit 3D Gaussian representation.The paper identifies geometry priors or scene inpainting as potential future solutions.