PaperlayerView exact revision on arXiv

Source-linked AI summary

GS-SLAM: Dense Visual SLAM with 3D Gaussian Splatting

Chi Yan, Delin Qu, Dan Xu, Bin Zhao, Zhigang Wang, Dong Wang, Xuelong Li

arXiv:2311.11700v4cs.CV

TL;DR

NeRF-based dense SLAM faces a speed–accuracy trade-off because high-resolution ray-based rendering is time-consuming. GS-SLAM uses 3D Gaussians with differentiable splatting, adaptive expansion, and coarse-to-fine tracking, achieving competitive performance on Replica and TUM-RGBD while running at 8.43 FPS.

  • Problem

    NeRF-based SLAM faces an efficiency–accuracy trade-off because obtaining high-resolution images with ray-based volume rendering is time-consuming.

  • Method

    GS-SLAM represents scenes with 3D Gaussians and differentiable splatting, adaptively expands or deletes Gaussians, and uses coarse-to-fine tracking for camera-pose estimation.

  • Results

    8.43 FPS is reported, with competitive tracking and mapping performance on the Replica and TUM-RGBD datasets.

  • Takeaways & Limitations

    GS-SLAM extends 3D Gaussian representation from static-object synthesis to whole-scene dense RGB-D SLAM with real-time tracking, mapping, and rendering.

  • Takeaways & Limitations

    Reliance on high-quality depth data may limit performance, while high memory requirements constrain use in large scenes.

Abstract

from arXiv · show

In this paper, we introduce \textbf{GS-SLAM} that first utilizes 3D Gaussian representation in the Simultaneous Localization and Mapping (SLAM) system. It facilitates a better balance between efficiency and accuracy. Compared to recent SLAM methods employing neural implicit representations, our method utilizes a real-time differentiable splatting rendering pipeline that offers significant speedup to map optimization and RGB-D rendering. Specifically, we propose an adaptive expansion strategy that adds new or deletes noisy 3D Gaussians in order to efficiently reconstruct new observed scene geometry and improve the mapping of previously observed areas. This strategy is essential to extend 3D Gaussian representation to reconstruct the whole scene rather than synthesize a static object in existing methods. Moreover, in the pose tracking process, an effective coarse-to-fine technique is designed to select reliable 3D Gaussian representations to optimize camera pose, resulting in runtime reduction and robust estimation. Our method achieves competitive performance compared with existing state-of-the-art real-time methods on the Replica, TUM-RGBD datasets. Project page: https://gs-slam.github.io/.

1. Introduction

SLAM jointly builds a scene map and estimates camera motion, but existing representations and NeRF-based methods face efficiency, scalability, or rendering-speed challenges. GS-SLAM addresses these issues with 3D Gaussians, differentiable splatting, adaptive expansion, and coarse-to-fine tracking.

  • SLAM constructs a dense or sparse map of an unknown environment while simultaneously tracking the camera pose.
  • NeRF-based SLAM improves dense-map quality and memory consumption, but ray-based volume rendering makes high-resolution images time-consuming.
  • 3D Gaussian representation and tile-based splatting provide efficient high-resolution rendering and state-of-the-art visual quality for static-object novel-view synthesis.
  • GS-SLAM combines 3D Gaussian scene representation with differentiable splatting for dense RGB-D SLAM, targeting a better balance between speed and accuracy.
  • Adaptive Gaussian expansion reconstructs new observed geometry, while coarse-to-fine tracking selects reliable Gaussians for faster and more accurate camera-pose estimation.
  • 8.43 FPS is reported for GS-SLAM, with competitive tracking and mapping performance on the Replica and TUM-RGBD datasets.

3. Methodology

GS-SLAM represents scenes with 3D Gaussians and differentiable RGB-D splatting for simultaneous camera tracking and dense mapping. Its adaptive expansion and coarse-to-fine tracking strategies add new geometry, suppress noisy Gaussians, and optimize poses efficiently.

  • 3. Methodology: GS-SLAM estimates every-frame camera poses while reconstructing a dense scene map from sequential RGB-D inputs with known camera intrinsics.The system jointly performs tracking and mapping using 3D Gaussian scene representations and RGB-D differentiable splatting.
  • 3.1. 3D Gaussian Scene Representation: The scene is modeled as 3D Gaussians defined by position, covariance, opacity, and spherical-harmonics color coefficients.Each Gaussian includes a 3D position, covariance matrix, opacity, and 12 spherical-harmonics coefficients.
  • 3.1. 3D Gaussian Scene Representation: Given a camera pose, Gaussians are projected into the image plane and rendered by depth sorting with front-to-back α-blending for color and depth.The projected covariance uses the Jacobian of an affine approximation to the projective function; depth uses each Gaussian center’s camera-coordinate z value.
  • 3.2. Adaptive 3D Gaussian Expanding Mapping: At each keyframe, adaptive expansion adds Gaussians for unreliable pixels whose opacity or rendered depth disagrees with sensor observations.Unreliable pixels are back-projected into 3D, and new Gaussians are initialized there to model newly observed areas.
  • 3.2. Adaptive 3D Gaussian Expanding Mapping: The method suppresses floating Gaussians by reducing opacity when their positions are sufficiently far from observed scene surfaces.For a visible Gaussian, opacity is scaled by η when the depth discrepancy exceeds γ; η is much smaller than 1.
  • 3.3. Tracking and Bundle Adjustment: Camera tracking uses analytical pose derivatives and a coarse-to-fine procedure that first estimates pose from sparse half-resolution pixels, then refines it using reliable Gaussians.The fine stage renders full-resolution images while ignoring noisy Gaussians that introduce artifacts.

4. Experiment

GS-SLAM is evaluated against state-of-the-art visual SLAM methods on Replica and TUM-RGBD, covering localization, reconstruction, rendering, runtime, memory, and component ablations. Results show strong accuracy and rendering quality with substantial rendering and tracking speed, while memory usage remains a trade-off.

  • Localization and mapping: GS-SLAM achieves the best or second-best tracking performance in 7 of 8 Replica scenes and outperforms Point-SLAM by 0.4 cm on average at 8.34 FPS.Point-SLAM runs at 0.42 FPS, making it 20× slower than GS-SLAM.
  • Localization and mapping: On Replica reconstruction, GS-SLAM achieves the best average Depth L1 of 1.16cm and Precision of 74.0%, with comparable Recall and F1 to CoSLAM.The reported mesh reconstruction has clear boundaries and details.
  • Rendering evaluation: GS-SLAM outperforms the second-best rendering methods by 1.52 dB in PSNR, 0.027 in SSIM, and 0.12 in LPIPS, while reaching 386 FPS on average.The reported visualizations show improved image quality around detailed object and image boundaries.
  • Runtime and memory analysis: GS-SLAM runs at 8.34 FPS with zero learnable parameters but uses 198.04 MB for Gaussian scene representations, 4× the memory of NICE-SLAM.It is reported as 20 × faster than the similar point-based method Point-SLAM; a light version reduces memory using zero-order spherical harmonics.
  • Ablation study: Removing Gaussian addition causes the mapping implementation to crash, while removing deletion produces redundant noisy Gaussians and undesirable supervision.The full expansion strategy improves performance by 0.1 in ATE and 11.97 in Recall compared with the ablated setting.
  • Ablation study: Coarse-to-fine tracking performs best across tracking, mapping, and rendering metrics, improving over fine tracking by 0.01 in tracking ATE, 2.11 in Recall, and 0.72 in PSNR.The authors associate fine tracking's lower robustness with artifacts and noise in the reconstructed scene.

5. Conclusion and Limitations

GS-SLAM uses 3D Gaussian Splatting for efficient mapping and accurate camera pose estimation, while relying on high-quality depth data and substantial memory for large scenes.

  • GS-SLAM leverages 3D Gaussian Splatting for efficient map construction and accurate camera pose estimation.
  • The adaptive expansion strategy compares reconstruction with and without the delete strategy in Replica #Room0.
  • The system evaluates tracking, rendering, and system FPS on Replica #Office0.
  • High-quality depth data constrains performance in some conditions, while large scenes impose high memory requirements.Future improvements may target memory use through quantization and clustering.

Overview

The supplementary material extends GS-SLAM with proofs, algorithmic details, visualizations, evaluations, and qualitative findings supporting the main conclusions.

  • The supplement provides proofs of gradient computation for pose and depth derivatives.
  • It includes the coarse-to-fine pose optimization algorithm with pseudocode and visualization.
  • Additional evaluations and qualitative findings support the conclusions presented in the main paper.

1. Proofs of Gradient Computation

The gradient derivation represents camera pose with quaternion rotation and translation, then back-propagates RGB-D rendering gradients through projected Gaussian quantities.

  • Camera pose P is represented by a quaternion q for rotation and a translation vector t.The rotation quaternion is converted to rotation matrix R.
  • The derivation treats Xi=[x, y, z]^T in camera coordinates as Xc and computes pose-related Jacobians.
  • Gradients are back-propagated through the intermediate term E=JP^-1.The rotation-quaternion gradients are derived after this intermediate representation is formed.
  • Oversized Gaussians are controlled by deletion, splitting, and cloning so a covariance-dependent intermediate term can be ignored.
  • Depth pose gradients use the same back-propagation procedure as color because point-based depth and color alpha blending are similar.The rasterization involves n 3D Gaussian splats affecting a pixel.

2. Coarse-to-fine Pose Optimization Algorithm

GS-SLAM estimates pose with strict front-to-back alpha blending and a coarse-to-fine selection of reliable Gaussians, removing artifacts that can impair tracking.

  • 2. Coarse-to-fine Pose Optimization Algorithm: The coarse-to-fine pose optimization algorithms are summarized in Algorithm 1.
  • 2. Coarse-to-fine Pose Optimization Algorithm: Splatting-based pose estimation uses strict front-to-back alpha blending on 3D Gaussians.This differs from the volume rendering used by current NeRF-based SLAM methods.
  • 2. Coarse-to-fine Pose Optimization Algorithm: The fine stage removes unreliable 3D Gaussians before pose optimization.Pixels without any 3D Gaussian do not affect pose gradients.
  • 2. Coarse-to-fine Pose Optimization Algorithm: Using all 3D Gaussians can leave artifacts from improperly optimized positions, introducing excessive optimization errors in the loss function.

3. Additional Performance Comparison

GS-SLAM is evaluated against state-of-the-art NeRF-based SLAM methods through rendered environment models on TUM-RGBD and Replica. The comparisons emphasize detailed reconstruction quality, especially edges and textured regions.

  • Render Performance on TUM-RGBD: GS-SLAM is compared with CoSLAM, ESLAM, Point-SLAM, and ground truth using final reconstructed environment models on TUM-RGBD.The comparison presents methods in descending order from top to bottom.
  • Coarse-to-Fine Pose Optimization: The coarse-to-fine pose optimization initializes from previous poses, performs coarse rendering and optimization, selects reliable 3D Gaussians, then performs fine optimization.The procedure uses separate coarse and fine iteration stages with loss computation and Adam updates.
  • Ablation Context: A Replica-wide ablation study evaluates tracking, mapping, and rendering performance for the complete method and alternative module arrangements.The table caption identifies the study as covering the whole Replica dataset.
  • Render Performance on TUM-RGBD: GS-SLAM provides the clearest TUM-RGBD results, with richer detail information and stronger handling of details and edges in the complex #fr3 office scene.
  • Render Performance on Replica: On Replica, GS-SLAM performs superior in all tested scenarios, producing clear edges and richly textured surfaces closely aligned with ground truth.The reported results indicate precise geometric-detail and textural-information reconstruction under complex textures and lighting conditions.

4. Additional Ablation Results

The ablations examine adaptive Gaussian management, coarse-to-fine tracking, and depth supervision. Together, the reported comparisons support the effectiveness of the complete methodology and depth-based geometry constraints.

  • Module Ablation: The complete methodology improves tracking, mapping, and rendering quality across the Replica dataset compared with disabling or altering proposed modules.The ablation contrasts Gaussian deletion, coarse-only tracking, fine-only tracking, and the fully integrated method.
  • Adaptive Gaussian Expansion: The adaptive 3D Gaussian expansion strategy produces a more coherent and accurate reconstructed mesh on Replica Room0 than the version without the delete strategy.
  • Render Comparison: Replica render comparisons show clearer edges and better results in richly textured regions than previous state-of-the-art methods.
  • Depth Supervision: Depth supervision improves tracking and mapping by providing accurate geometry constraints during optimization.
  • Depth Supervision: With depth supervision, GS-SLAM achieves tracking ATE of 0.48, mapping precision of 64.58, and rendering PSNR of 31.56.These values are reported as better than the implementation without depth supervision.

5. Additional Implementation Details

The implementation uses Adam-based optimization for Gaussian parameters and camera pose, with staged pose optimization and TSDF-Fusion for mesh generation. The paper notes a limitation in directly extracting meshes from 3DGS.

  • Mapping Hyper-Parameters: The 3D Gaussian representation and pose are trained using Adam with specified position, spherical-harmonics, opacity, scaling, and rotation learning rates.The position learning rate decays from lrXinit = 1.6e−5 to lrXfinal = 1.7e−7.
  • Mapping Hyper-Parameters: Camera pose optimization is restricted to the latter half of mapping iterations because improper 3D Gaussians can adversely affect optimization.The paper still reports occasional negative optimizations for camera poses.
  • Mapping Mesh Comparison: Mesh generation follows Point-SLAM by applying TSDF-Fusion to predicted poses and depths, while map rendering quality is evaluated separately.
  • Mapping Mesh Comparison: Direct surface or mesh extraction is unavailable for 3DGS-based SLAM because these methods do not represent scenes with density fields for marching cubes.

GS-SLAM + TSDFusion

The paper compares mesh generation from 3DGS with different methods and reports rendered-detail results on TUM-RGBD. Tracking uses staged coarse and fine pose estimation with reliable Gaussians.

  • GS-SLAM + TSDFusion: Figure 10 compares mesh generation from 3DGS using different methods.
  • GS-SLAM + TSDFusion: Pose tracking uses coarse pose estimation in the first five iterations and reliable 3D Gaussians for fine pose estimation later.
  • GS-SLAM + TSDFusion: The tracking hyper-parameters include FusedAdam learning rates lrt = 2e−4 and lrq = 5e−4, with photometric loss weighting 0.8.
  • GS-SLAM + TSDFusion: On TUM-RGBD, GS-SLAM reconstructs dense environment maps with rich and intricate details.The caption attributes this to fast back-propagation of splatting in optimized 3D Gaussians.
Loading 2311.11700v4…