Source-linked AI summary
SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM
Nikhil Keetha, Jay Karhade, Krishna Murthy Jatavallabhula, Gengshan Yang, Sebastian Scherer, Deva Ramanan, Jonathon Luiten
TL;DR
Dense SLAM needs representations that support efficient tracking, mapping, and novel-view synthesis beyond the limitations of existing explicit and implicit approaches. SplaTAM uses differentiably rendered 3D Gaussians with online tracking and densification, achieving state-of-the-art results while remaining sensitive to challenging inputs and requiring known intrinsics and dense depth.
Problem
Existing explicit dense SLAM methods depend on rich geometry and coverage, while implicit volumetric methods are computationally inefficient and difficult to edit.
Method
SplaTAM uses 3D Gaussians with differentiable rendering to jointly optimize RGB-D camera poses and an explicit volumetric scene map.
Results
SplaTAM achieves state-of-the-art camera pose estimation, scene reconstruction, and novel-view synthesis, including a trajectory error reduction from 0.52cm to 0.36cm on Replica.
Takeaways & Limitations
The results support 3D Gaussian Splatting as a framework for fast rendering, dense optimization, explicit map extent, and structured map densification in SLAM.
Takeaways & Limitations
SplaTAM is sensitive to motion blur, large depth noise, and aggressive rotation, and requires known camera intrinsics and dense depth input.
Abstract
from arXiv · showhide
Dense simultaneous localization and mapping (SLAM) is crucial for robotics and augmented reality applications. However, current methods are often hampered by the non-volumetric or implicit way they represent a scene. This work introduces SplaTAM, an approach that, for the first time, leverages explicit volumetric representations, i.e., 3D Gaussians, to enable high-fidelity reconstruction from a single unposed RGB-D camera, surpassing the capabilities of existing methods. SplaTAM employs a simple online tracking and mapping system tailored to the underlying Gaussian representation. It utilizes a silhouette mask to elegantly capture the presence of scene density. This combination enables several benefits over prior representations, including fast rendering and dense optimization, quickly determining if areas have been previously mapped, and structured map expansion by adding more Gaussians. Extensive experiments show that SplaTAM achieves up to 2x superior performance in camera pose estimation, map construction, and novel-view synthesis over existing methods, paving the way for more immersive high-fidelity SLAM applications.
1. Introduction
Dense visual SLAM depends strongly on map representation, yet existing explicit and implicit approaches retain important efficiency, coverage, and viewpoint-synthesis limitations. SplaTAM explores explicit volumetric 3D Gaussians to combine fast differentiable rendering with dense optimization and high-fidelity mapping.
- Dense SLAM estimates a vision sensor’s pose and an environmental map, making map representation consequential for system design and downstream tasks.
- Explicit representations require rich 3D geometric features and high-framerate captures, while reliably explaining observed scenes mainly with dense view coverage.
- Unobserved or novel viewpoints remain important for applications such as mixed reality and high-fidelity 3D capture.
- Implicit volumetric SLAM methods provide high-fidelity global maps and dense photometric losses, but are computationally inefficient and difficult to edit.
- SplaTAM uses explicit volumetric 3D Gaussians for rendering, tracking, and mapping, enabling rendering up to 400 FPS and dense photometric optimization in real time.
- 3D Gaussians provide direct, almost linear projective gradient flow from physical scene parameters to dense photometric loss and camera parameters.
- Across simulated and real data, SplaTAM achieves state-of-the-art results in camera pose estimation, map reconstruction, and novel-view synthesis.
2. Related Work
Prior dense SLAM systems use explicit, neural, or implicit representations, while recent Gaussian Splatting methods assume known camera poses. SplaTAM removes that constraint by jointly estimating poses and fitting a Gaussian map.
- Dense SLAM research has used explicit representations including images, signed-distance functions, Gaussian mixture models, and circular surfels.
- Neural-network approaches have integrated predicted depth, latent depth codes, or learned depth cost volumes with traditional SLAM.
- Implicit scene-representation methods such as iMAP, NICE-SLAM, and iSDF use neural or hierarchical structures for tracking, mapping, and signed-distance computation.
- Gaussian Splatting supports extremely fast differentiable rendering and has been extended to static and dynamic scene representations.
- Prior Gaussian Splatting approaches require accurately known 6-DOF camera poses for representation optimization.
- SplaTAM removes the known-pose constraint by simultaneously estimating camera poses and fitting the underlying Gaussian representation.
3. Method
SplaTAM builds an online RGB-D SLAM system around differentiable rendering of an explicit 3D Gaussian map. It alternates camera tracking, Gaussian densification, and map updating using rendered color, depth, and silhouette information.
- SplaTAM represents the world as 3D Gaussians and jointly optimizes camera poses and a volumetric map through differentiable rendering and gradient-based optimization.
- Each Gaussian uses view-independent color and isotropic geometry, parameterized by RGB color, 3D center, radius, and opacity.
- Differentiable Rendering via Splatting: Gaussian splatting sorts primitives front-to-back and alpha-composites their 2D projections to render RGB images efficiently.
- Differentiable Rendering via Splatting: Projected Gaussian centers and radii depend on camera intrinsics, camera extrinsics, focal length, and camera-coordinate depth.
- Differentiable Rendering via Splatting: The renderer also produces differentiable depth images that can be compared with input depth to return gradients to the 3D map.
- SLAM System: A rendered silhouette identifies pixels containing information from the current map and guides visibility-aware optimization and map expansion.
- SLAM System: For each new RGB-D frame, the system tracks camera pose, adds Gaussians using silhouette and depth, then updates Gaussian parameters over selected overlapping keyframes.
- SLAM System: Tracking minimizes RGB and depth reconstruction errors only over well-optimized visible pixels, while map updating optimizes Gaussian parameters with fixed camera poses over all pixels.
4. Experimental Setup
SplaTAM is evaluated across four RGB-D datasets with varied scene, image-quality, trajectory, and novel-view conditions. The experiments use established rendering and pose metrics and compare against radiance-field, dense, and traditional SLAM baselines.
- Datasets and Evaluation Settings: The evaluation covers ScanNet++, Replica, TUM-RGBD, and the original ScanNet, following prior radiance-field-based SLAM procedures for the last three.ScanNet++ is added because the other benchmarks lack hold-out novel-view evaluation.
- Datasets and Evaluation Settings: ScanNet++ provides high-quality RGB-D data and separate hold-out capture loops, but its large inter-frame motion makes pose estimation difficult.The frame displacement is approximately equivalent to a 30-frame gap on Replica.
- Datasets and Evaluation Settings: Replica is the simplest benchmark, whereas TUM-RGBD and original ScanNet are harder for dense methods because of sparse depth, missing data, low-quality color, and motion blur.Replica uses synthetic scenes with accurate complete depth and small consecutive-pose displacements.
- Evaluation Metrics: RGB rendering is measured with PSNR, SSIM, and LPIPS, depth rendering with Depth L1 loss, and tracking with ATE RMSE.These metrics follow the evaluation protocol used by Point-SLAM.
- Baselines: Comparisons include Point-SLAM as the main baseline, older dense methods, and traditional systems selected for TUM-RGBD, Replica, and ScanNet++.Traditional baselines include Kintinuous, ElasticFusion, ORB-SLAM2, DROID-SLAM, and ORB-SLAM3.
5. Results & Discussion
Across four datasets, SplaTAM delivers strong camera tracking and high-fidelity rendering, including held-out novel views, while ablations identify critical design choices and experiments expose practical limitations.
- Camera pose estimation: SplaTAM consistently outperforms dense baselines on ScanNet++, Replica, and TUM-RGBD, while remaining competitive on Orig-ScanNet.Table 1 reports online camera-pose estimation using ATE RMSE.
- Camera pose estimation: 1.2cm average trajectory error lets SplaTAM track ScanNet++ sequences despite large camera displacements that cause Point-SLAM and ORB-SLAM3 to fail.The ScanNet++ scans are also texture-less, causing repeated ORB-SLAM3 reinitialization.
- Camera pose estimation: SplaTAM reduces Replica trajectory error by more than 30%, from 0.52cm to 0.36cm, relative to Point-SLAM.It also provides better or competitive performance than DROID-SLAM.
- Camera pose estimation: SplaTAM reduces TUM-RGBD trajectory error by almost 40%, from 8.92cm to 5.48cm, although feature-based methods still outperform dense methods there.The benchmark contains sparse depth and severe motion blur, which challenge volumetric methods.
- Rendering quality: On ScanNet++, SplaTAM achieves 24.41 average PSNR on novel views and 27.98 PSNR on training views.Novel-view evaluation uses ground-truth novel-view poses for alignment, while Point-SLAM fails to estimate poses and construct a good map.
- Ablations: Using both RGB and depth losses is necessary for excellent tracking and reconstruction, while silhouette masking, forward-velocity propagation, and thresholding are critical tracking choices.Depth-only tracking fails, RGB-only tracking has more than 5x the error, removing velocity propagation raises error more than 10x, and threshold 0.99 gives a 5x error reduction versus 0.5.
- Runtime: SplaTAM renders roughly 1.2 million pixels per iteration yet has runtime comparable to NICE-SLAM and Point-SLAM.The comparison uses an Nvidia RTX 3080 Ti; competing methods render far fewer sampled pixels.
- Limitations: SplaTAM remains sensitive to motion blur, large depth noise, and aggressive rotation, and requires known camera intrinsics and dense depth input.The authors identify temporal modeling and removing these input dependencies as future work.
6. Conclusion
SplaTAM uses 3D Gaussians as an explicit map representation for fast rendering, dense optimization, spatial extent awareness, and streamlined densification. It achieves state-of-the-art results in camera pose estimation, scene reconstruction, and novel-view synthesis.
- SplaTAM achieves state-of-the-art results in camera pose estimation, scene reconstruction, and novel-view synthesis.
S1. Overview of Supplementary Material
The supplementary material provides online resources, qualitative demonstrations, and interactive visualizations of SplaTAM's SLAM and novel-view rendering results.
- The project website and open-source code accompany the supplementary qualitative and quantitative results.
- The website includes an edited video, interactive rendering demos, qualitative SLAM and novel-view videos, and camera-tracking loss visualizations.
- Qualitative online reconstructions are also demonstrated using RGB-D data from an iPhone with a commodity camera and time-of-flight sensor.
S2. Additional Qualitative Visualizations
Additional visualizations show high-fidelity Gaussian reconstructions and camera trajectories that closely align with ground-truth poses, even on challenging sequences with large inter-frame displacement.
- SplaTAM's estimated poses precisely align with ground-truth poses while providing a high-fidelity reconstruction on ScanNet++ S2.
- Reconstructions on two ScanNet++ sequences are high quality in both geometry and visual appearance.
- SplaTAM accurately solves ScanNet++ sequences despite the large displacement between sequential camera poses.
S3. Additional Quantitative Results
Supplementary quantitative results examine Gaussian-distribution choices and compare SplaTAM's novel-view synthesis with original 3D Gaussian Splatting under differing pose-estimation and mapping settings.
- Gaussian Distribution Ablation: Isotropic Gaussians provide faster speed and better memory efficiency than anisotropic Gaussians, with only marginal SLAM performance differences on thin-structured ScanNet++ S1.
- Comparison to 3D Gaussian Splatting: SplaTAM provides novel-view synthesis performance comparable to original 3D Gaussian Splatting while estimating unknown camera poses online.