Source-linked AI summary
Gaussian-SLAM: Photo-realistic Dense SLAM with Gaussian Splatting
Vladimir Yugay, Yue Li, Theo Gevers, Martin R. Oswald
TL;DR
Dense neural SLAM has struggled to combine scalable real-world reconstruction with photo-realistic rendering and practical tracking. Gaussian-SLAM uses 3D Gaussians, independently optimized sub-maps, efficient seeding, and photometric-geometric frame-to-model tracking. It reports top results across rendering, camera pose estimation, and scene reconstruction, while motion blur, poor depth, and trajectory drift remain limitations.
Problem
Dense neural SLAM methods remain limited on real-world data by computational demands, slow speed, and difficulty incorporating pose updates.
Method
Gaussian-SLAM uses 3D Gaussians with sub-map organization, efficient seeding and online optimization, and photometric-geometric frame-to-model tracking.
Results
Gaussian-SLAM yields top results in rendering, camera pose estimation, and scene reconstruction across varied synthetic and real-world datasets.
Takeaways & Limitations
The Gaussian representation supports dense SLAM with fast tracking and mapping together with improved rendering results on synthetic and real-world datasets.
Takeaways & Limitations
Tracking remains challenging with heavy motion blur and low-quality depth maps, and frame-to-model tracking inevitably accumulates drift without loop closure or bundle adjustment.
Abstract
from arXiv · showhide
We present a dense simultaneous localization and mapping (SLAM) method that uses 3D Gaussians as a scene representation. Our approach enables interactive-time reconstruction and photo-realistic rendering from real-world single-camera RGBD videos. To this end, we propose a novel effective strategy for seeding new Gaussians for newly explored areas and their effective online optimization that is independent of the scene size and thus scalable to larger scenes. This is achieved by organizing the scene into sub-maps which are independently optimized and do not need to be kept in memory. We further accomplish frame-to-model camera tracking by minimizing photometric and geometric losses between the input and rendered frames. The Gaussian representation allows for high-quality photo-realistic real-time rendering of real-world scenes. Evaluation on synthetic and real-world datasets demonstrates competitive or superior performance in mapping, tracking, and rendering compared to existing neural dense SLAM methods.
1 Introduction
Gaussian-SLAM addresses limitations of dense neural SLAM by using 3D Gaussians for interactive mapping, tracking, and photo-realistic rendering in real-world RGBD scenes. Its contributions cover geometry-aware representation, scalable sub-map optimization, efficient seeding, and photometric-geometric frame-to-model tracking.
- Dense neural SLAM methods achieve high scene-representation quality but remain limited to small synthetic scenes and non-photo-realistic re-rendering.
- Gaussian splatting offers rendering quality comparable to NeRFs while being an order of magnitude faster in rendering and optimization.
- Gaussian-SLAM uses 3D Gaussians in a dense RGBD SLAM system for mapping, tracking, and photo-realistic re-rendering at interactive runtimes.
- The method extends Gaussian splatting to better encode geometry and reconstruct beyond radiance fields in a single-camera setup.
- Online optimization processes the map as sub-maps and introduces efficient seeding and optimization strategies.
- Frame-to-model tracking minimizes photometric and geometric errors using the Gaussian splatting scene representation.
2 Related Work
Prior dense SLAM representations trade off scalability, local updateability, geometric flexibility, or rendering quality. Gaussian-SLAM uses controlled Gaussian-map growth, GPU-independent sub-map processing, sparse seeding, and pose tracking without explicit camera-pose derivatives.
- Traditional dense SLAM methods use grid, point, network, or hybrid scene representations with distinct memory, connectivity, and updateability trade-offs.
- Neural methods render impressively on synthetic data but struggle on real-world data and remain computationally demanding, slow, and difficult to update with poses.
- Grid-based methods provide quick neighborhood lookup and context integration but require a predefined resolution that is difficult to adjust during reconstruction.
- Point-based methods allow scene-adaptive density and surface-focused memory use, but their unstructured connectivity makes neighbor search more complex.
- Gaussian-SLAM avoids retaining all 3D Gaussians in GPU memory, uses sparser color-gradient and masking-based seeding, and preserves state-of-the-art rendering quality.
- Its tracking differs from concurrent Gaussian-SLAM methods by avoiding explicitly computed camera-pose derivatives and using PyTorch.
3 Method
Gaussian-SLAM represents single-camera RGBD scenes with 3D Gaussians, using sub-maps for scalable online mapping and jointly optimizing geometry, color, and tracking objectives. It seeds Gaussians in sparsely covered regions, adds depth supervision, and estimates poses by matching rendered and input frames.
- Overview: Gaussian-SLAM builds a 3D Gaussian scene representation for dense RGBD SLAM in a sequential single-camera setup.The representation supports mapping, tracking, and photo-realistic re-rendering at interactive runtimes.
- Map construction: New anisotropic Gaussians are seeded from subsampled RGBD points in high-gradient or low-alpha regions that lack neighbors within radius ρ.Their scales are initialized from nearest-neighbor distances in the active sub-map.
- Geometry and color encoding: Differential depth rendering supplies gradients for Gaussian means, covariances, and opacities, extending Gaussian splatting to encode geometry.The depth of each pixel is computed from ordered Gaussian means, opacities, and transmittance terms.
- Map construction: Sub-maps partition the input stream into independently represented Gaussian point clouds covering several keyframes, avoiding the need to keep all Gaussians in GPU memory.This design addresses catastrophic forgetting, overfitting, and computational feasibility as the explored area grows.
- Geometry and color encoding: Gaussian parameters are optimized with photometric, depth, and isotropic regularization losses combined as L = λ_color · L_color + λ_depth · L_depth + λ_reg · L_reg.Color supervision combines L1 and SSIM losses, while depth supervision uses an L1 loss; regularization limits excessive elongation from sparse seeding.
- Tracking: Frame-to-model tracking minimizes a loss between rendered and input color and depth using a constant-speed pose initialization and soft alpha masking.The masking excludes or downweights previously unobserved and poorly reconstructed pixels during pose optimization.
4 Experiments
Experiments evaluate Gaussian-SLAM on synthetic and real-world datasets for rendering, tracking, reconstruction, runtime, and tracking-mask effects. The method achieves strong rendering and tracking results, comparable reconstruction, and robustness advantages, while motion blur and poor depth remain challenging.
- Experimental setup: Experiments compare Gaussian-SLAM with state-of-the-art dense neural RGBD SLAM methods on synthetic and real-world datasets.Results are averaged over three runs with different seeds.
- Rendering performance: Gaussian-SLAM outperforms existing dense neural RGBD methods on Replica, TUM-RGBD, and ScanNet rendering metrics.The ScanNet advantage is reported as a significant margin.
- Tracking performance: Gaussian-SLAM outperforms the nearest competitor by 14% on Replica tracking and performs better than all baseline methods on TUM-RGBD.It also performs significantly better than NeRF-based methods on ScanNet++.
- Tracking performance: On ScanNet, tracking exhibits drift because of low-quality depth maps and substantial motion blur.These conditions remain a practical challenge for the method.
- Reconstruction performance: Gaussian-SLAM achieves reconstruction performance on par with existing dense SLAM methods on Replica and is comparable to Point-SLAM.Point-SLAM requires ground-truth depth maps for inference, whereas the comparison describes Gaussian-SLAM as superior to other dense SLAM methods.
- Ablation study: Ablations show that both soft alpha and inlier masks improve tracking, with the soft alpha mask having the larger performance impact.The masks improve tracking and thereby lead to better rendering.
5 Conclusion
Gaussian-SLAM uses 3D Gaussian Splatting for dense SLAM, combining efficient seeding, online sub-map optimization, scalable mapping, and frame-to-model tracking. It achieves strong rendering, pose-estimation, and reconstruction results across synthetic and real-world datasets.
- Gaussian-SLAM represents scenes with 3D Gaussian Splatting to support dense SLAM and high-quality re-rendering.
- The system combines efficient Gaussian seeding, online optimization, sub-map organization, and frame-to-model tracking.
- Gaussian-SLAM achieves faster tracking and mapping than Point-SLAM while obtaining better rendering results on synthetic and real-world datasets.
- Across varied datasets, Gaussian-SLAM yields top results in rendering, camera pose estimation, and scene reconstruction.
B Further Implementation Details
Implementation details define depth-error filtering, alpha-based loss weighting, Gaussian initialization, sub-map merging, and global-map color refinement.
- The inlier mask excludes pixels with depth errors exceeding 50 times the median current re-rendered depth error and pixels lacking valid depth input.
- The soft alpha mask uses M_alpha = α^3 for per-pixel tracking-loss weighting.
- Added Gaussians are initialized with opacity 0.5 and scales based on nearest-neighbor distances.
- After processing the sequence, saved sub-maps are merged into a global map using sequential candidate selection and nearest-neighbor checking.
- Global-map color refinement runs for 10000 iterations as post-processing.
C Additional Experiments
Additional experiments show that isotropic regularization supports accurate tracking and rendering, while Gaussian-SLAM performs strongly on held-out-view synthesis and qualitative reconstruction.
- Isotropic Regularization Ablation: Without isotropic regularization, elongated Gaussians overfit training views and harm pose optimization at novel views.
- Qualitative Reconstruction Results: Gaussian-SLAM recovers more geometric and color details in real-world ScanNet and TUM-RGBD reconstructions.
- Isotropic Regularization Ablation: Isotropic regularization improves both tracking and rendering on the Replica dataset.
- Isotropic Regularization Ablation: On TUM-RGBD, isotropic regularization is critical for tracking accuracy and improves rendering performance.
- Qualitative Reconstruction Results: Gaussian-SLAM achieves comparable reconstruction performance to state-of-the-art dense neural SLAM methods on Replica.
- Novel View Synthesis: 3.6 dB in PSNR is Gaussian-SLAM’s average advantage over concurrent work on held-out ScanNet++ views.