PaperlayerView exact revision on arXiv

Source-linked AI summary

CGS-SLAM: Collaborative Gaussian Splatting based SLAM for Multi-Agent Reconstruction

Jean-Daniel de Ambrogi, Aladine Chetouani, Vincent Nguyen, Aurélien Chateigner

arXiv:2608.26868v1cs.CVcs.RO

TL;DR

CGS-SLAM addresses the lack of depth sensors and limited collaborative 3DGS support on consumer-grade smartphones. It combines monocular inertial local mapping with metric depth, dynamic keyframing, and server-side VGGT alignment. The system reports improved reconstruction quality, reduced tracking error, RGB-D-comparable quality, and accurate collaborative alignment using RGB and inertial data.

  • Problem

    Consumer-grade smartphones lack dedicated depth sensors, while collaborative 3DGS SLAM remains limited and existing depth-dependent methods constrain monocular deployment.

  • Method

    CGS-SLAM uses decentralized local RGB-inertial reconstruction with metric monocular depth, broadcasts keyframe encodings for dynamic overlap-aware keyframing, and centrally aligns submaps.

  • Results

    The method achieves superior reconstruction quality and reduced tracking error versus other monocular 3DGS methods, comparable quality to RGB-D systems, and accurate submap alignment.

  • Takeaways & Limitations

    RGB and inertial data can support collaborative Gaussian-Splatting SLAM with globally consistent mapping in GNSS-denied, resource-constrained environments.

  • Takeaways & Limitations

    Depth Pro is too heavy for the authors' laptop and is offloaded to a centralized server; the pipeline is otherwise MMDE-agnostic.

Abstract

from arXiv · show

Recent advances in SLAM have leveraged 3DGS for photorealistic reconstruction and novel view synthesis. However, most methods rely on RGB-D input, which is unavailable on consumer-grade smartphones, and few integrate 3DGS within a collaborative framework. Therefore, we present CGS-SLAM, a hybrid decentralized/centralized system enabling multi-agent 3DGS SLAM using only RGB and inertial data. Each agent performs local tracking with inertial data as a motion prior and reconstructs a scaled map using a metric monocular depth estimator (Depth Pro). Keyframe encodings are shared among agents, enabling dynamic keyframing in regions of spatial overlaps with other agents, enhancing submap alignment. Afterwards, a central server aligns submaps using VGGT as a view alignment model. This bidirectional communication keeps communication cost low during mapping and global reconstruction in difficult GNSS-denied environments. Experiments on multiple datasets demonstrate competitive tracking performance, improved rendering quality over state-of-the-art methods, and accurate submap alignment.

1 Introduction

CGS-SLAM targets collaborative SLAM in GNSS-denied, sensor-constrained environments where consumer smartphones provide only RGB and inertial data. It combines monocular inertial mapping, metric depth, dynamic keyframing, and server-side submap alignment.

  • Existing depth-dependent SLAM methods can require substantial computation, time, and parallax-inducing motion during initialization.
  • Consumer-grade tactical smartphones typically provide RGB cameras and IMUs but lack LiDAR or ToF depth sensing.
  • CGS-SLAM reconstructs dense, continuous local maps and aligns them on a server into a globally consistent shared map.
  • The method uses monocular inertial local SLAM with a metric monocular depth estimator to produce metrically scaled 3DGS maps from the first frame.
  • Agents exchange keyframe encodings to increase sampling density in spatially overlapping regions, improving subsequent submap alignment.
  • VGGT provides rapid submap alignment and immediate pose estimation for local submaps.

2 Related Work

Related work establishes 3DGS as a photorealistic, high-frame-rate representation but identifies dependence on depth priors and limited development of collaborative neural SLAM. CGS-SLAM addresses the combination of monocular sensing and multi-agent Gaussian reconstruction.

  • 3DGS provides photorealistic rendering at high frame rates and has become a mapping representation in SLAM systems.
  • Many 3DGS SLAM methods depend on depth priors, while relative monocular depth can drift and cause catastrophic map collapse.
  • Monocular SLAM methods address consumer-device constraints but can be less robust under sparse texture, changing illumination, or rapid camera motion.
  • Collaborative SLAM remains relatively underdeveloped because inter-agent data association, communication, and map merging are technically challenging.
  • Prior neural collaborative systems include point-based approaches, while Gaussian-Splatting collaboration has used RGB-D inputs requiring LiDAR.

3 Method

CGS-SLAM uses a hybrid decentralized–centralized architecture: agents independently build local Gaussian maps from RGB, IMU, and metric depth, while exchanging keyframe encodings and later merging submaps centrally.

  • Architecture: Each agent independently reconstructs a local map and periodically exchanges fixed-size keyframe encodings through a central server.
  • Sensor Fusion: IMU preintegration chains high-frequency inertial transformations to initialize each new camera pose before visual pose optimization.
  • Depth and Initialization: A metric monocular depth estimator initializes scale-consistent scene geometry and supplies metric depth at keyframes.
  • Global Reconstruction: After exploration, the agent sends keyframes, Gaussian parameters, and camera intrinsics to the server for global reconstruction.
  • Tracking: Tracking optimizes camera pose with a masked combination of photometric and depth losses while holding the Gaussian map constant.
  • Keyframe Selection: Keyframes balance temporal spacing and information gain, including geometric coverage, temporal consistency, and computational efficiency.
  • Collaborative Keyframing: Similarity between incoming and local keyframe encodings detects overlap and increases keyframe density there to improve global consistency during fusion.

4 Experiments

Experiments evaluate CGS-SLAM’s monocular tracking, rendering, IMU contribution, and collaborative submap alignment under consumer-grade computational constraints. The method achieves competitive tracking, improved rendering over monocular 3DGS baselines, and accurate multi-agent alignment.

  • Monocular single-agent exploration: Metric depth estimation improves Gaussian-map rendering consistency over consumer LiDAR artifacts and reduces horizontal linear distortion.The comparison uses LiDAR, MiDaS, and Depth Pro on UT-MM’s fast-straight scene.
  • Monocular single-agent exploration: The method predicts trajectory scale close to ground truth, whereas MM3DGS produces a raw trajectory approximately one-third the ground-truth size.The raw trajectories are only origin-aligned, without rotation, scale, or translation fitting.
  • Monocular single-agent exploration: The approach achieves competitive ATE-RMSE without depth measurements, performing comparably to RGB-D 3DGS SLAM methods and competitively with Magic-SLAM.Table 1 evaluates RGB-D and monocular configurations across multiple TUM sequences.
  • Monocular single-agent exploration: Rendering consistently outperforms MM3DGS across evaluated sequences and achieves quality similar to RGB-D methods on most sequences, while Magic-SLAM leads on fr1/room.Evaluation uses PSNR, SSIM, and LPIPS between original keyframes and renderings at estimated camera poses.
  • IMU contribution: IMU pre-integration reduces Fast-straight ATE from 12.85 cm to 6.07 cm, a 53% reduction, while improving PSNR by 2.73 dB; Square-1 ATE falls from 55.43 cm to 42.33 cm.The comparison is against a constant-velocity tracking baseline in monocular settings.
  • Global reconstruction: Server-side submap alignment succeeds in most UT-MM cases and achieves translation error below 0.40 m while supporting more than two agents.Trajectories are evaluated after transformation by the server alignment process within a shared coordinate frame.

5 Conclusion

CGS-SLAM combines metric monocular depth, inertial data, and VGGT-based submap alignment to build scale-consistent collaborative Gaussian maps from RGB and inertial inputs. It reports improved reconstruction quality and tracking error over monocular 3DGS methods, with quality comparable to RGB-D systems.

  • 5 Conclusion: CGS-SLAM uses Depth Pro to construct scale-consistent 3D models without dedicated depth sensors, while integrating IMU data for tracking robustness.The system also introduces VGGT-based rapid submap alignment for collaborative mapping.
  • 5 Conclusion: The method achieves superior reconstruction quality and reduced tracking error versus other monocular 3DGS methods, with quality comparable to RGB-D systems using only RGB and inertial data.Future work targets a unified, computationally lighter model for fully onboard smartphone execution.

Supplementary Material: CGS-SLAM — Collaborative Gaussian Splatting based SLAM for Multi-Agent Reconstruction

The pipeline represents scenes as parametric 3D Gaussians, projects them onto the image plane, and differentiably optimizes their parameters against reference images.

  • Gaussian representation: Each Gaussian is parameterized by a 3D position, covariance, opacity, and color.The covariance is represented through rotation and scaling matrices.
  • Gaussian representation: Gaussian covariance is decomposed as Σ = R S S^T R^T to preserve symmetry and positive semi-definiteness during optimization.R is orthonormal and S contains scaling factors, supporting numerical stability and physically plausible Gaussian shapes.
  • Projection and rendering: Splatting projects 3D Gaussians onto the 2D image plane, using the world-to-view transform and projection Jacobian to obtain the 2D covariance.The projection is the basis for rendering Gaussian contributions in image space.
  • Projection and rendering: Pixel color is computed by depth-ordered alpha blending of projected Gaussians, with each contribution weighted by color and local opacity.The final pixel opacity is defined analogously to the color compositing process.
  • Optimization: Because rendering is differentiable, Gaussian parameters are optimized by minimizing the L1 difference between rendered and reference images.The objective is L_photo = ||I - C(G, T_c)||_1.

B Message passing between clients and server

The system coordinates agents through lightweight message passing while using inertial measurements to estimate inter-frame camera motion and support client-side tracking.

  • Message passing: A scalable message-passing framework coordinates agents without centralized mapping or full trajectory sharing.The server receives client initialization messages, including intrinsic camera parameters that accommodate heterogeneous sensors.
  • Message passing: The communication protocol uses custom messages to transmit keyframe encodings over low-bandwidth links.The message architecture is designed for efficient inter-agent communication.
  • Inertial motion prior: The IMU provides accelerations and angular velocities that support relative displacement and rotation estimation between consecutive frames.Displacement uses integrated velocity and acceleration, while rotation uses angular velocity over the timestep.
  • Inertial motion prior: The relative camera transformation is obtained by combining the IMU-frame motion with the static IMU-to-camera transformation.This converts inertial motion estimates into the camera coordinate frame.

D Implementation Details

Implementation details specify the camera transformation used in tracking and the optimization settings used for training, tracking, and global mapping.

  • Tracking: The relative camera transformation is constructed by composing the IMU motion with the fixed IMU-to-camera transform.This provides the camera-frame motion relation used by the system.
  • Optimization and mapping: Mapping uses balanced depth, color, and structural loss weights plus densification controls to maintain fidelity while limiting Gaussian growth.Tracking additionally applies opacity and covisibility constraints, while global optimization uses separate position and rotation learning rates.

E Two-Stage Procrustes Alignment

The alignment procedure constructs augmented camera-position point sets and estimates similarity transforms in two Procrustes stages to align VGGT coordinates and local submaps.

  • Point-set construction: Each client’s camera-position point set is augmented with a virtual point along the camera up-axis to reduce rotation ambiguity.The offset scale λ is the mean inter-camera distance within the point set.
  • Stage 1: VGGT to sub-map 1: Stage 1 estimates a similarity transform aligning client 1’s VGGT camera positions with its corresponding sub-map poses.The transform minimizes squared point-position error and is solved using Procrustes analysis.
  • Stage 2: transformed VGGT to sub-map 2: Stage 2 applies the first transform to client 2’s VGGT poses and estimates a second similarity transform aligning them with sub-map 2.The transformed pose alignment is refined using gradient descent on full pose error, including orientation.
  • Composition: The composed transform T2 ◦ T1 maps VGGT coordinates to sub-map 2, from which the sub-map-2-to-sub-map-1 alignment is derived.This composition connects the two-stage coordinate transformations.

F Experiments with Depth Anything V3

The experiments compare Depth Pro with Depth Anything V3 as the monocular depth and pose configuration. Depth Anything V3 is competitive on small TUM sequences but degrades on larger UT-MM scenes, motivating retention of the dual-model setup.

  • Model comparison: Depth Anything V3 produces both metric depth maps and camera pose estimates, potentially replacing Depth Pro and VGGT in the pipeline.The tested configuration aimed to reduce dependence on two deep neural network models.
  • Conclusion: Lower local-mapping quality with Depth Anything V3 contaminated submap alignment, so the authors retained the dual-model approach.The degradation was especially evident on larger UT-MM scenes.
  • Model comparison: Depth Anything V3 is competitive and occasionally better on the small TUM sequences but degrades sharply on large UT-MM scenes and fails on Square-2.These tracking and rendering comparisons are reported in Table S2.
  • Model comparison: Depth Pro achieves better tracking than Depth Anything V3, particularly in handling turns.The trajectory comparison is shown in Figure S2.

G Tests on Replica Multiagent Dataset with the constant velocity hypothesis

Replica Multiagent lacks inertial measurements, so CGS-SLAM is evaluated under a constant-velocity assumption instead of its IMU prior. This substitution can produce severe, trajectory-dependent failures when near-field depth estimates are unreliable.

  • Evaluation setting: Replica Multiagent provides no inertial measurements, requiring replacement of the IMU prior with a constant-velocity assumption.The resulting evaluation is reported separately from the main evaluation.
  • Evaluation setting: The constant-velocity substitution costs 7.05 cm of average ATE on UT-MM.This quantifies the cost of removing inertial measurements.
  • Tracking results: Replica results are bimodal: some sequences remain within IMU-based performance, while Apt-1 and Apt-2/agent-1 diverge above one meter.The separation follows trajectories rather than scenes, including agents observing the same Apt-2 environment.
  • Failure analysis: Near-field depth estimation errors at doorframes inject mis-scaled geometry into the Gaussian map, and without inertial constraints tracking cannot recover.The reported failure is attributed to depth estimation rather than the collaborative back-end.
  • Qualitative results: Figures S3 and S4 illustrate the global submap trajectories and rendering examples associated with the Replica evaluation.Figure S3 shows top-view trajectories, while Figure S4 shows scenes from Apt-0 and Apt-1.

H Supplementary Results

Supplementary experiments extend single-agent tracking comparisons across UT-MM and provide additional rendering results. CGS-SLAM completes all eight UT-MM sequences, while sharp-turn sequences remain difficult for every evaluated system.

  • Single-agent tracking: CGS-SLAM is the only evaluated system to complete all eight UT-MM sequences.Magic-SLAM diverges on Square-2, while MAC-Ego3D diverges on Ego-Drive.
  • Single-agent tracking: On straight and localized sequences, CGS-SLAM remains within the range of RGB-D baselines without using a depth sensor.The Square sequences are exceptions because they chain four sharp turns.
  • Single-agent tracking: MAC-Ego3D records 40.07/77.24 cm and MM3DGS records 47.05/68.50 cm on the difficult Square sequences.The reported values are ATE results for the compared systems in Table 3.
  • Failure analysis: Sharp turns provide little translational parallax, allowing heading error to accumulate and making rotation-dominated motion a key improvement target.Magic-SLAM recovers on some Square sequences through loop closure and global bundle adjustment, which CGS-SLAM omits for bandwidth reasons.
  • Rendering results: Supplementary Figure S5 shows monocular single-agent rendering on UT-MM and TUM with Depth Pro, including good results with small artifacts.The metric monocular depth estimator also bypasses some poorly detailed UT-MM depth-ground-truth areas.
Loading 2608.26868v1…