Source-linked AI summary
Gaussian Splatting SLAM
Hidenobu Matsuki, Riku Murai, Paul H. J. Kelly, Andrew J. Davison
TL;DR
Online monocular SLAM needs a representation that supports accurate tracking, mapping, and high-fidelity rendering despite uncertain poses and incremental reconstruction. The paper builds a 3DGS-only SLAM system with direct pose optimisation, geometric regularisation, and Gaussian management, achieving state-of-the-art performance across monocular and RGB-D benchmarks. The method also supports photorealistic reconstruction, including tiny and transparent objects, while remaining limited to room-scale scenes and below hard real-time speed on benchmark data.
Problem
3DGS had previously been used for offline reconstruction with known camera poses, leaving online monocular SLAM without a unified representation for tracking, mapping, and high-fidelity rendering.
Method
The system uses 3D Gaussians as its only SLAM representation, adding analytic pose Jacobians, isotropic shape regularisation, geometric verification, and Gaussian allocation and pruning.
Results
The system achieves state-of-the-art performance across monocular and RGB-D benchmarks, including camera trajectory estimation, mapping, and photorealistic online reconstruction.
Takeaways & Limitations
3D Gaussian Splatting can support a live SLAM system that reconstructs diverse materials, including tiny and transparent objects, from monocular input.
Takeaways & Limitations
The method is tested only on small room-scale scenes and does not achieve hard real-time operation at 30 fps on TUM sequences.
Abstract
from arXiv · showhide
We present the first application of 3D Gaussian Splatting in monocular SLAM, the most fundamental but the hardest setup for Visual SLAM. Our method, which runs live at 3fps, utilises Gaussians as the only 3D representation, unifying the required representation for accurate, efficient tracking, mapping, and high-quality rendering. Designed for challenging monocular settings, our approach is seamlessly extendable to RGB-D SLAM when an external depth sensor is available. Several innovations are required to continuously reconstruct 3D scenes with high fidelity from a live camera. First, to move beyond the original 3DGS algorithm, which requires accurate poses from an offline Structure from Motion (SfM) system, we formulate camera tracking for 3DGS using direct optimisation against the 3D Gaussians, and show that this enables fast and robust tracking with a wide basin of convergence. Second, by utilising the explicit nature of the Gaussians, we introduce geometric verification and regularisation to handle the ambiguities occurring in incremental 3D dense reconstruction. Finally, we introduce a full SLAM system which not only achieves state-of-the-art results in novel view synthesis and trajectory estimation but also reconstruction of tiny and even transparent objects.
1. Introduction
The paper introduces the first online monocular SLAM system using 3DGS as its sole scene representation, extending Gaussian Splatting from offline reconstruction to live tracking, mapping, and rendering. It adds pose optimisation, geometric regularisation, and resource management to support high-fidelity reconstruction and competitive trajectory estimation.
- Motivation: 3DGS provides a unified representation for local detail, large-scale mapping, and direct camera tracking in online SLAM.The paper positions this unified dense representation against layered systems that combine multiple representations or components.
- Technical innovations: Analytic pose Jacobians, Gaussian shape regularisation, and geometric verification enable direct camera estimation and consistent incremental reconstruction.The method also introduces Gaussian resource allocation and pruning to keep geometry clean and support tracking.
- Reported results: The method achieves photorealistic online local reconstruction and state-of-the-art camera trajectory estimation and mapping for larger scenes compared with rendering-based SLAM methods.It uses only monocular RGB input without a pretrained monocular depth predictor or existing tracking modules.
- Evaluation: The system is evaluated in both monocular and RGB-D settings, with competitive performance particularly in real-world scenarios.The RGB-D extension applies when external depth measurements are available.
2. Related Work
Related work frames dense SLAM around detailed map construction and contrasts frame-centric and map-centric approaches. The paper adopts a map-centric design with dynamically allocated 3D Gaussians to support arbitrary spatial distributions and differentiable, high-fidelity rendering.
- Dense SLAM: Dense SLAM reconstructs detailed, interactive 3D maps, whereas sparse SLAM primarily supports pose estimation and localisation.Dense maps are described as useful for broader applications including AR and robotics.
- Frame-centric SLAM: Frame-centric SLAM jointly estimates per-frame depth and camera motion by minimising photometric error across consecutive frames.This approach is distinguished from map-centric representations in the related-work taxonomy.
- Map-centric SLAM: Point-based map representations such as surfel clouds adapt resolution and spatial distribution through dynamic allocation, benefiting online SLAM applications.The related work notes that achieving high-fidelity optimisation with these representations remains challenging.
- Differentiable rendering: Differentiable volumetric rendering addresses the over-smoothing and limited detail of weighted-averaging fusion for photorealistic scene capture.The paper adopts 3D Gaussians as the sole map-centric SLAM representation and dynamically allocates them across the scene.
3. Method
The method uses 3D Gaussians as a unified representation for rendering, camera tracking, mapping, and keyframe management in online SLAM. It combines differentiable rasterisation, analytic pose Jacobians, geometric residuals, keyframe selection, Gaussian maintenance, and isotropic regularisation.
- 3.1. Gaussian Splatting: 3D Gaussian Splatting represents scenes with anisotropic Gaussians whose optical properties include colour and opacity, while volume rendering synthesises pixels without explicit surface extraction.Gaussians are splatted and blended after projection from world to image coordinates.
- 3.3. SLAM: The system uses 3D Gaussians as its only representation across tracking, mapping, keyframe management, and novel view synthesis.This unified representation is the basis of the full SLAM framework.
- 3.2. Camera Pose Optimisation: Camera pose optimisation uses analytically derived Lie-group Jacobians integrated with differentiable rasterisation, avoiding automatic-differentiation overhead.The Jacobians are minimally parameterised to match the camera pose degrees of freedom.
- 3.3.1. Tracking: Tracking optimises the current camera pose against rendered and observed images, and optionally combines photometric and depth residuals when depth observations are available.Affine brightness parameters are also optimised for exposure variation, while low-opacity or non-edge pixels are penalised.
- 3.3.2. Keyframe Management: A small keyframe window is maintained using covisibility and relative-translation criteria, with Gaussian visibility estimated through rasterisation-aware occlusion handling.Keyframes are registered when covisibility falls below a threshold or translation is large relative to median depth.
- 3.3.3. Mapping: Mapping inserts Gaussians for newly visible details, initialises their positions from observed or rendered depth, and prunes recent Gaussians that lack sufficient multiview support.Gaussians unobserved by at least three other frames after insertion within the last three keyframes are removed.
- 3.3.3. Mapping: Isotropic regularisation penalises differences between Gaussian scaling parameters and their mean to reduce elongated viewing-direction artefacts that otherwise hinder continuous SLAM.Depth residuals are added to the mapping objective when depth observations are available.
4. Evaluation
The evaluation spans monocular and RGB-D tracking, rendering, ablations, convergence, and qualitative reconstruction. Across these settings, the system achieves strong tracking and rendering results while preserving fine geometric and visual detail.
- Experimental Setup: The evaluation covers TUM RGB-D and Replica sequences, with TUM supporting both monocular and RGB-D evaluation and Replica supporting RGB-D evaluation only.Replica includes challenging purely rotational camera motions.
- Camera Tracking Accuracy: The method surpasses monocular tracking baselines without deep priors and achieves performance comparable to systems using explicit loop closure.Tracking accuracy is measured by ATE RMSE of keyframes.
- Camera Tracking Accuracy: The RGB-D method outperforms all baseline methods on real-world TUM data and achieves the best result in 6 out of 8 Replica sequences with the single-process implementation.The multi-process implementation, which performs fewer mapping iterations, remains comparable; Gaussian position optimisation compensates for sensor noise on TUM RGB-D.
- Ablative Analysis: Isotropic regularisation, geometric residuals, and keyframe selection improve tracking by constraining geometry when photometric signals are weak.The ablation evaluates these components on the TUM RGB-D dataset.
- Qualitative Results: Qualitative reconstructions show geometrically coherent Gaussian placement, high-quality novel views, and natural handling of transparent objects without explicitly modelling a surface.The examples include monocular SLAM reconstructions and challenging self-captured scenes.
5. Conclusion
The paper presents the first SLAM method using 3D Gaussians as its representation, achieving high-fidelity live reconstruction and state-of-the-art benchmark performance. Future work targets loop closure and explicit surface-geometry extraction.
- The system improves the fidelity and diversity of materials captured by live SLAM through efficient volume rendering.
- The method achieves state-of-the-art performance across monocular and RGB-D benchmarks.
- Future research should integrate loop closure for large-scale scenes and extract geometry such as surface normals.
7. Implementation Details
Implementation uses Adam-based optimisation, residual losses, covisibility-driven keyframing, and Gaussian insertion and pruning to maintain an online map. Dataset-specific thresholds control keyframe selection and window size.
- Optimisation: Adam optimises camera poses and Gaussian parameters, with separate rotation and translation learning rates and a 10× monocular position-rate increase.Camera pose rates are 0.003 for rotation and 0.001 for translation; monocular Gaussian positions use a tenfold increase.
- Losses: Tracking minimises photometric and geometric residuals, while mapping uses a corresponding modified objective.
- Keyframing: Keyframe selection is based on Gaussian covisibility using Intersection over Union and Overlap Coefficient criteria.
- Keyframing: Keyframe-window sizes are 10 for Replica and 8 for TUM.
- Gaussian insertion and pruning: The method inserts monocular Gaussians by sampling rendered-depth distributions and prunes recent Gaussians lacking sufficient visibility or having opacity below 0.7.
8. Evaluation details
Evaluation measures trajectory accuracy, rendering quality and speed, camera localisation, and end-to-end processing across monocular, RGB-D, and synthetic settings. Comparisons use established baselines and dataset-specific training and testing protocols.
- Trajectory evaluation: Monocular trajectory evaluation uses ATE RMSE after scale alignment, whereas RGB-D evaluation aligns trajectories without scale adjustment.
- Baselines: The study benchmarks against classical, learning-based, and rendering-based SLAM or visual-odometry methods using reported or locally reproduced results.
- Rendering performance: Rendering evaluation reports PSNR, SSIM, LPIPS, and forward-rendering speed, distinguishing rendering fps from end-to-end system fps.The reported rendering speed is 769 fps, versus 2.17 fps for VoxFusion.
- Camera localisation: Convergence-basin experiments render synthetic Replica training and test views with camera frustums distributed across specified spatial radii.
- Training setup: Training protocols include depth and no-depth Gaussian-map initialisation, fixed training poses, and baseline networks trained with their specified losses.
- Runtime and rendering evaluation: Tables report rendering comparisons and total execution time or fps for Replica and TUM experiments, including multiprocessing and single-process RGB-D variants.
- Camera localisation: Camera localisation fixes the Gaussian map and measures success by convergence within 1cm of the target view in 1000 iterations.
9. Further Ablation Analysis (Table 3)
Ablations show that pruning improves monocular tracking and isotropic regularisation is especially valuable for Replica RGB-D scenes. Additional analyses cover spherical harmonics, large-scale stereo input, memory, and whole-system runtime.
- Pruning ablation: Gaussian pruning significantly enhances monocular camera tracking by removing unstable randomly initialised Gaussians that can create incorrect geometry.
- Isotropic-loss ablation: Isotropic regularisation has marginal effect on TUM RGB-D tracking but significantly improves Replica tracking by preventing view-axis Gaussian overstretching.
- Spherical-harmonics ablation: Enabling spherical harmonics slightly improves rendering metrics without significant runtime change but increases Gaussian-map size and GPU memory usage.
- Tracker comparison: Using an off-the-shelf tracker with 3DGS mapping produces no significant rendering-metric difference, although ORB-SLAM has the best ATE in the cited RGB-D comparison.
- Large-scale scenes: The method is competitive on easier EuRoC Machine Hall sequences, while performance drops on more difficult, longer sequences.
- Memory analysis: Pruning lowers memory by retaining well-constrained Gaussians and omitting spherical harmonics compared with original Gaussian Splatting.
- Runtime analysis: Whole-system processing time is measured across all frames to reflect realistic operating conditions rather than isolated components.
10. Camera Pose Jacobian
The paper derives camera-pose Jacobians on the Lie group for optimizing poses against a 3D Gaussian map. The derivation separates rotational and translational effects and assembles the resulting matrix derivatives into the pose Jacobian.
- The method derives minimal analytical camera-pose Jacobians on the Lie group for direct optimization against a 3D Gaussian map.
- The Jacobian with respect to W considers only the rotational component because translation does not contribute.
- The derivation defines skew-symmetric matrix derivatives for individual rotation components such as θx.
- Basis vectors e1, e2, and e3 provide the coordinate directions used in the rotation-derivative expressions.
- Column-wise vectorization and horizontal stacking of the component derivatives produce the final Jacobian in Eq. (6).
11. Additional Qualitative Results
Additional qualitative results visually compare Gaussian Splatting SLAM with differentiable-rendering SLAM methods across supplementary sequences.
- The supplementary results compare Gaussian Splatting SLAM visually with Point-SLAM and ESLAM using differentiable rendering.
12. Limitation of this work
The method is currently limited to small room-scale scenes and does not yet achieve benchmark hard real-time operation. Larger scenes can accumulate trajectory drift, while TUM sequences do not reach 30 fps.
- The method has been tested only on small room-scale scenes, limiting its demonstrated scope.
- Larger real-world scenes can exhibit inevitable trajectory drift without a loop-closure module.
- Interactive live operation is achieved, but benchmark hard real-time operation at 30 fps on TUM sequences is not.