PaperlayerView exact revision on arXiv

Source-linked AI summary

LoopSplat: Loop Closure by Registering 3D Gaussian Splats

Liyuan Zhu, Yue Li, Erik Sandström, Shengyu Huang, Konrad Schindler, Iro Armeni

arXiv:2408.10154v2cs.CVcs.RO

TL;DR

Existing 3DGS-based SLAM methods lack global consistency through loop closure or global bundle adjustment. LoopSplat uses 3DGS submaps and direct registration for online loop closure, and reports competitive or superior tracking, mapping, and rendering across four datasets.

  • Problem

    Existing coupled 3DGS SLAM methods lack global-consistency strategies for maps and poses.

    Source: §1 · p. 1

  • Method

    LoopSplat performs dense RGB-D mapping and frame-to-model tracking with 3DGS submaps, directly registering them to derive loop constraints for pose graph optimization.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    Evaluation on four datasets shows competitive or superior tracking, mapping, and rendering performance.

    Source: §5 · pp. 8–9

  • Takeaways & Limitations

    LoopSplat achieves global consistency through online loop closure using direct 3DGS submap registration.

    Source: §5 · p. 8

    When the number of submaps exceeds 100, pairwise registrations during pose graph optimization significantly increase computational demands and reduce loop-closure efficiency.

    Source: G. Limitations and Future Work · p. 16

Abstract

from arXiv · show

Simultaneous Localization and Mapping (SLAM) based on 3D Gaussian Splats (3DGS) has recently shown promise towards more accurate, dense 3D scene maps. However, existing 3DGS-based methods fail to address the global consistency of the scene via loop closure and/or global bundle adjustment. To this end, we propose LoopSplat, which takes RGB-D images as input and performs dense mapping with 3DGS submaps and frame-to-model tracking. LoopSplat triggers loop closure online and computes relative loop edge constraints between submaps directly via 3DGS registration, leading to improvements in efficiency and accuracy over traditional global-to-local point cloud registration. It uses a robust pose graph optimization formulation and rigidly aligns the submaps to achieve global consistency. Evaluation on the synthetic Replica and real-world TUM-RGBD, ScanNet, and ScanNet++ datasets demonstrates competitive or superior tracking, mapping, and rendering compared to existing methods for dense RGB-D SLAM. Code is available at loopsplat.github.io.

1. Introduction

Dense RGB-D SLAM methods either decouple tracking from mapping or couple them without maintaining global consistency, leaving pose errors and map distortions unresolved. LoopSplat uses 3DGS submaps for coupled tracking and mapping, with direct 3DGS registration for online loop closure and global consistency.

1. Introduction

  • Coupled 3DGS SLAM methods lack global-consistency strategies, while decoupled methods do not use the dense map for tracking.Decoupling can cause inefficient information sharing and increased computational overhead; the lack of global consistency can lead to accumulated pose errors and distorted maps.Source: §1 · p. 1
  • Prior global-consistency methods can require costly map retraining or optimization and saving all mapped frames, limiting scalability.GO-SLAM retrains hash-grid features, while Photo-SLAM optimizes Gaussian parameters after pose updates; these re-integration approaches require storing all mapped frames.Source: §1 · pp. 1–2
  • LoopSplat uses 3DGS submaps for coupled RGB-D tracking and dense mapping, with online loop closure and pose graph optimization.Its loop edge constraints come from a registration method that directly operates on 3DGS, rather than traditional global-to-local point-cloud registration.Source: §1 · p. 2
  • The proposed registration is reported to outperform traditional techniques in speed and accuracy, while the system improves tracking and reconstruction across real-world datasets.The loop closure module uses both 3D geometry and visual scene content for loop detection and closure.Source: §1 · p. 2

2. Related Work

Related work spans dense mapping, globally consistent SLAM, and registration of point clouds, NeRFs, and 3D Gaussian splats. LoopSplat is distinguished by direct 3DGS registration for submap loop constraints and loop detection that combines image matching with submap overlap.

2. Related Work

  • Dense mapping progressed from truncated signed distance functions and voxel methods to point-based representations, including neural points and 3D Gaussians.Globally consistent dense SLAM commonly divides the map into submaps and uses pose graph optimization, sometimes with global bundle adjustment.Source: §2 · p. 2
  • Among RGB-D 3DGS SLAM methods, prior approaches lacked global consistency, while Loopy-SLAM used submaps and loop closure with traditional point-cloud registration.LoopSplat instead registers 3DGS directly, avoids separate meshing for registration, and combines image matching with submap overlap for loop detection.Source: §2 · p. 2
  • Point-cloud registration aligns fragments through rigid transformations, typically using hand-crafted descriptors and RANSAC, while learning-based methods use local descriptors.The passage notes that point clouds lack NeRFs’ continuous, view-dependent, multiscale representation capabilities.Source: §2 · p. 2
  • LoopSplat’s pipeline couples Gaussian-splat tracking and mapping with submap finalization, loop detection, registration-derived pose-graph edges, and pose graph optimization.The front end starts a new submap after the camera passes a predefined threshold; the back end monitors for revisits and optimizes camera poses and submaps after a loop.Source: Figure 2 · p. 3
  • NeRF and earlier 3DGS registration methods assume ground-truth training-view camera poses and explore pairwise registration in small scenes.LoopSplat operates without training or preprocessing on estimated front-end poses and is integrated into online loop closure.Source: §2 · p. 3

3. LoopSplat

LoopSplat uses RGB-D input to build 3D Gaussian submaps and track frames against them, then registers submaps and optimizes a pose graph for global consistency. Its registration treats a submap as a rigid body and localizes selected keyframes through rendering.

3.1. Gaussian Splatting SLAM

  • Submaps grow as the scene expands, with new Gaussians added in sparsely covered regions and optimized against keyframe rendering losses.New submaps are triggered when displacement or rotation from the first keyframe exceeds a threshold; optimization combines color, depth, and regularization losses.Source: §3.1 · pp. 3–4

3.2. Registration of Gaussian Splats

  • Registration treats each Gaussian submap as a rigid body and localizes selected keyframes by matching rendered RGB-D views.NetVLAD similarities select top-k overlapping keyframe pairs; transformations are estimated in both directions, then combined using residual-weighted rotation averaging and weighted-mean translation.Source: §3.2 · pp. 4–5

3.3. Loop Closure with 3DGS

  • Loop closure is triggered when a new submap is created; detected loops add registration-based constraints to a robust pose graph that corrects submap poses.Visual similarity proposes revisits, while a geometric overlap ratio greater than 0.2 filters loop candidates; pose corrections update camera poses and Gaussian geometry.Source: §3.3 · p. 5
  • LoopSplat performs best on Replica tracking and achieves the highest accuracy on ScanNet++, including under large camera motions.The supplied table captions identify the datasets and tracking results but do not provide numerical values.Source: Table 1 · p. 5, Table 2 · p. 5

4. Experiments

Experiments evaluate LoopSplat on synthetic Replica and real-world TUM-RGBD, ScanNet, and ScanNet++ datasets, comparing tracking, reconstruction, rendering, runtime, memory, and registration ablations. Results show strong tracking and rendering, with reconstruction and efficiency trade-offs across methods.

4.1. Tracking

  • LoopSplat achieves 10% higher tracking accuracy than the second-best method on Replica, and improves tracking accuracy by 14% on TUM-RGBD and 33% on ScanNet++.It obtains the third-best performance on ScanNet; on longer sequences with loops and motion blur, errors accumulate for the evaluated 3DGS baselines.Source: §4.1 · p. 6

4.2. Reconstruction

  • On Replica, LoopSplat outperforms all 3DGS-based reconstruction baselines but trails Loopy-SLAM and Point-SLAM, which require ground-truth depth for ray-marching.On ScanNet, its reconstructed meshes recover more geometric detail, including chair shapes.Source: §4.2 · p. 7, Table 5 · p. 7, Figure 4 · p. 8

4.3. Rendering

  • LoopSplat surpasses competing methods in PSNR and LPIPS on Replica and ScanNet, and is competitive with SplaTAM on TUM-RGBD.The comparison merges submaps into a global map and optimizes it with estimated camera poses to avoid local submap overfitting.Source: §4.3 · p. 7, Table 6 · p. 7

4.4. Memory and Runtime Analysis

  • LoopSplat’s Gaussian-splat embedding is 8× smaller than SplaTAM’s, and it uses the least GPU memory for a room-sized scene.Its per-frame tracking and map-optimization time is slower than the fastest baselines, while its loop-edge registration is significantly faster than Loopy-SLAM’s.Source: §4.4 · p. 7, Table 7 · p. 8

4.5. Ablations

  • On Replica, FPFH+ICP applied directly to 3D Gaussian centers yields less accurate loop edges and deteriorates loop closure; its preprocessing makes more than 8× slower than our method.The preprocessing re-renders and back-projects 3DGS, then downsamples and voxelizes the point clouds.Source: §4.5 · p. 8
  • Multi-view optimization and rotation averaging improve registration accuracy, while overlap-based view selection is crucial for informative viewpoints and system efficiency.The ablation study reports that every registration component contributes to final performance.Source: §4.5 · p. 8, Table 8 · p. 8

5. Conclusion

LoopSplat is a dense RGB-D SLAM system built around 3D Gaussian Splat submaps. Its evaluation across four datasets shows competitive or superior tracking, mapping, and rendering performance.

5. Conclusion

  • LoopSplat uses 3D Gaussian Splats as its scene representation and achieves global consistency through loop closure.It supports dense mapping, frame-to-model tracking, and online loop closure through direct 3DGS submap registration.Source: §5 · p. 8
  • The system performs dense mapping, frame-to-model tracking, and online loop closure using 3DGS submaps.It rigidly aligns submaps to achieve global consistency.Source: §5 · p. 8
  • Across four datasets, LoopSplat achieves competitive or superior tracking, mapping, and rendering performance.Source: §5 · p. 9

Abstract

The supplement documents LoopSplat’s implementation and additional evaluations. In a ScanNet scene, online loop closure corrects accumulated drift when the camera revisits its first room.

Abstract

  • The supplementary material adds implementation details, loop-closure and pose-graph statistics, qualitative results, ablations, and discussion of limitations and future work.Source: Abstract · p. 13

A. Video

  • Online loop closure corrects accumulated error when the ScanNet camera revisits its first room, despite substantial drift in the bathroom and storage room.The accompanying video shows the reconstructed mesh, trajectory colored by average translation error, and current-frame point cloud.Source: A. Video · p. 13

B. Implementation Details

Implementation details specify dataset-dependent submap initialization, tracking and mapping settings, loop detection and registration, and final submap merging. Loop edges are filtered by geometric overlap and temporal separation.

Submap Initialization

  • ScanNet and ScanNet++ trigger submaps every 50 and 100 frames, respectively, while other datasets use motion thresholds of 0.5 m and 50°.Source: B. Implementation Details · p. 13

Frame-to-model Tracking

  • Tracking optimizes the relative pose between consecutive frames and renders from the previous camera pose after transforming the submap.The tracking loss excludes invalid-depth pixels and outliers; ScanNet++ can reinitialize pose with ICP when initialized rendering loss is much higher than its running average.Source: B. Implementation Details · p. 13

Submap Expansion and Update

  • Submap expansion samples qualifying pixels at dataset-specific counts, while newly added Gaussians are initialized locally and pruned after mapping optimization.Sampling is 30K points for TUM-RGBD and ScanNet, 100K for ScanNet++, and all qualifying points for Replica; the opacity pruning threshold is 0.1 for Replica and 0.5 elsewhere.Source: B. Implementation Details · p. 13

Submap Merging

  • After processing the sequence, submaps are merged through TSDF mesh extraction, Gaussian-center initialization from mesh vertices, and global-map color refinement.The global map is optimized from scratch using RGB-D input and estimated camera poses.Source: B. Implementation Details · p. 14

Loop Detection

  • Loop detection selects top-2 overlapping viewpoint pairs per dataset, then checks geometric overlap; loops below the dataset-specific threshold or too close in submap time are removed.Overlap thresholds are 0.1 m on Replica and 0.2 m on TUM-RGBD, ScanNet, and ScanNet++; minimum submap intervals are listed in Table B.2.Source: B. Implementation Details · p. 14, Table B.2 · p. 14

Loop-closure statistics

  • ScanNet scenes average about 4,000 frames, and challenging scenes require over 100 submaps and more than 30 pose-graph optimizations.Replica averages about four loop closures per scene, roughly one every 500 frames; its shorter, single-room sequences explain the lower frequency.Source: B. Implementation Details · p. 14, Table B.3 · p. 14

C. Datasets

The evaluation spans five ScanNet++ DSLR sequences, limited to their first 250 frames because of sudden large motions. ScanNet++ has about 10× the average frame motion of the other datasets, and LoopSplat achieves the best reported NVS results there.

C. Datasets

  • ScanNet++ average frame motion is about 10× larger than in the other evaluated datasets, making accurate pose estimation challenging.The evaluation uses five DSLR sequences and only their first 250 frames because sudden large motions occur in these sequences.Source: C. Datasets · pp. 14–15, Table C.1 · p. 15

D. Novel View Synthesis

  • LoopSplat yields the best ScanNet++ NVS results on held-out test views distant from training views.PSNR is evaluated over all test views after 10K iterations of global-map refinement at 876×584 resolution.Source: Table D.1 · p. 15, D. Novel View Synthesis · p. 15

E. Additional Analysis

Additional analyses examine scene-level rendering, online loop closure, and map size. Online loop closure is especially important in complex ScanNet and TUM RGB-D scenes, while a room-sized global map averages around 300K Gaussian splats.

Rendering Performance at Scene Level

Online LC

  • Online loop closure is crucial to superior performance on complex ScanNet and TUM RGB-D scenes, where it constantly corrects map drift and prevents cumulative errors.For smaller Replica scenes, online loop closure does not significantly improve performance because loops are limited.Source: E. Additional Analysis · p. 15

Average Number of Gaussians Per Scene

  • A room-sized global map contains around 300K Gaussian splats on average.The count depends on scene scale, mesh vertices used to initialize Gaussians, and densification iterations during 3DGS optimization.Source: E. Additional Analysis · p. 15, Table E.5 · p. 16

F. Additional Qualitative Results

The section presents qualitative evidence on overlap-based loop selection, submap registration, and mesh reconstruction. LoopSplat’s registrations and meshes compare favorably in the illustrated results.

Overlap Ratio

  • The overlap ratio is used to decide whether a detected loop is added to the pose graph.The figure presents representative submap pairs with descending overlap ratios.Source: F. Additional Qualitative Results · p. 16

3DGS Registration

  • Visual inspection shows LoopSplat’s submap alignments closer to, or sometimes better than, ground truth, while odometry is most misaligned.The comparison includes additional ScanNet registration examples.Source: F. Additional Qualitative Results · p. 16

Mesh Reconstruction

  • In geometrically complex regions, LoopSplat produces higher-quality, more consistent mesh reconstructions than baseline methods.The additional ScanNet examples focus on regions with high geometric complexity.Source: F. Additional Qualitative Results · p. 16

G. Limitations and Future Work

LoopSplat’s limitations include increased loop-closure cost as submaps grow and dataset-specific hyperparameters that may hinder generalization to in-the-wild data. Suggested future directions target mesh extraction, uncertainty estimates, and reconstruction in overlapping regions.

Limitations

  • Above 100 submaps, pairwise registrations during pose-graph optimization significantly reduce loop-closure efficiency.The authors also cite iterative Gaussian and camera-pose optimization as a speed constraint.Source: G. Limitations and Future Work · p. 16
  • Dataset-specific submap-construction hyperparameters may hinder generalization to in-the-wild data.The authors note that using different hyperparameters across datasets is standard practice in SLAM.Source: G. Limitations and Future Work · p. 16

Future Work

  • Future work includes advanced 3DGS mesh extraction, viewpoint uncertainty estimates, and refinement of reconstruction in overlapping submap regions.The authors suggest SuGAR and GOF as examples of mesh extraction methods that operate directly on 3DGS.Source: G. Limitations and Future Work · p. 16

Qualitative Results

  • The illustrated low-overlap submap pair is rejected as a loop because insufficient overlap can lead to inaccurate registration or failure.The figure labels this example OR = 0.1.Source: Figure G.1 · p. 17
  • Additional registration examples show LoopSplat alignments closer to or visually better than ground truth, while odometry is most misaligned.The figure highlights differences among odometry, LoopSplat, and ground truth.Source: Figure G.2 · p. 18
  • On the illustrated ScanNet scenes, LoopSplat produces smoother, more complete meshes than the strongest baseline, Loopy-SLAM.The examples show colored meshes and corresponding normals.Source: Figure G.3 · p. 19
Loading 2408.10154v2…