PaperlayerView exact revision on arXiv

Source-linked AI summary

VGGT-SLAM: Dense RGB SLAM Optimized on the SL(4) Manifold

Dominic Maggio, Hyungtae Lim, Luca Carlone

arXiv:2505.12549v2cs.CV

TL;DR

VGGT-SLAM addresses projective ambiguity that similarity transforms cannot fully resolve when aligning submaps from uncalibrated monocular cameras. It optimizes submap alignment on the SL(4) manifold, enabling large-scale dense mapping and performance competitive with or superior to other learning-based SLAM approaches.

  • Problem

    Similarity transformations cannot fully resolve projective ambiguity in VGGT reconstructions from uncalibrated cameras, which introduces shear, stretch, and perspective degrees of freedom.

  • Method

    VGGT-SLAM incrementally and globally aligns VGGT submaps by optimizing a factor graph on the SL(4) manifold using relative homographies and loop-closure constraints.

  • Results

    VGGT-SLAM extends VGGT to large-scale scenes using uncalibrated monocular RGB cameras without additional training, achieving performance competitive with or superior to other state-of-the-art learning-based SLAM approaches.

  • Takeaways & Limitations

    Projective SL(4) optimization provides a framework for handling cases where similarity transformations are insufficient while globally aligning dense submaps.

  • Takeaways & Limitations

    Full 15-DOF homography estimation is degenerate for planar points and vulnerable to outliers, producing unstable solutions in planar scenes.

Abstract

from arXiv · show

We present VGGT-SLAM, a dense RGB SLAM system constructed by incrementally and globally aligning submaps created from the feed-forward scene reconstruction approach VGGT using only uncalibrated monocular cameras. While related works align submaps using similarity transforms (i.e., translation, rotation, and scale), we show that such approaches are inadequate in the case of uncalibrated cameras. In particular, we revisit the idea of reconstruction ambiguity, where given a set of uncalibrated cameras with no assumption on the camera motion or scene structure, the scene can only be reconstructed up to a 15-degrees-of-freedom projective transformation of the true geometry. This inspires us to recover a consistent scene reconstruction across submaps by optimizing over the SL(4) manifold, thus estimating 15-degrees-of-freedom homography transforms between sequential submaps while accounting for potential loop closure constraints. As verified by extensive experiments, we demonstrate that VGGT-SLAM achieves improved map quality using long video sequences that are infeasible for VGGT due to its high GPU requirements.

1 Introduction

VGGT-SLAM extends VGGT to long, uncalibrated monocular sequences by aligning submaps with projective transformations rather than relying only on similarity alignment. The SL(4)-based factor graph addresses projective ambiguity and supports global alignment with loop closures.

  • Motivation: VGGT produces dense reconstructions and camera estimates but is limited to approximately 60 frames on a 24 GB RTX 4090.Reconstructions requiring hundreds or thousands of frames are therefore infeasible in a single VGGT inference.
  • Motivation: Similarity alignment cannot fully resolve the projective ambiguity introduced by uncalibrated cameras, which includes shear, stretch, and perspective degrees of freedom.The issue becomes especially pronounced when disparity between frames is small.
  • Method: VGGT-SLAM rectifies this ambiguity by estimating 4 × 4 homographies in SL(4) and optimizing submap alignment as a factor graph with sequential and loop-closure constraints.The SL(4) formulation estimates the full 15-degrees-of-freedom projective transformation between submaps.
  • Contribution: The system extends VGGT to large-scale scenes using only monocular RGB cameras, without known or consistently calibrated intrinsics and without additional training.It incrementally creates and aligns submaps rather than reconstructing the entire sequence in one inference.
  • Contribution: SL(4)-based alignment addresses cases where conventional Sim(3) transforms fail to resolve scale and alignment issues caused by projective ambiguity.Figure 1 illustrates this difference on six VGGT submaps from Clio apartment and cubicle scenes.

2 Related Work

The related work spans classical geometric SLAM, feed-forward scene reconstruction, scene-coordinate regression, and Lie-group optimization. VGGT-SLAM builds on feed-forward reconstruction while introducing SL(4) factor-graph optimization for submap alignment.

  • Classical Scene Reconstruction: Classical SLAM estimates camera poses and 3D scenes using feature extraction, matching, robust pose estimation, and bundle adjustment over SE(3) transformations.These methods generally rely on geometric features and multi-view constraints.
  • Feed-forward Scene Reconstruction: DUSt3R and MASt3R infer dense point maps from image pairs, while MASt3R additionally provides descriptors for pairwise correspondence generation.These feed-forward methods support pose and scene reconstruction from image pairs.
  • Feed-forward Scene Reconstruction: Spann3R and Cut3R incrementally reconstruct multiple images but remain limited to short sequences, whereas Pow3R incorporates optional camera and depth estimates.Splatt3R extends the DUSt3R approach to Gaussian Splatting.
  • Feed-forward Scene Reconstruction: MASt3R-SLAM provides real-time dense monocular SLAM without known calibration using Sim(3) optimization and loop closures.VGGT-SLAM instead builds on VGGT’s ability to process arbitrary numbers of frames, subject to computational limits.
  • Scene Coordinate Regression: Scene-coordinate regression methods such as ACE and DSAC* estimate world points relative to a global scene frame using scene-specific trained networks.This represents an alternative to feed-forward reconstruction and classical geometric pipelines.
  • Optimization over the Special Linear Group: Prior SL(3) optimization addresses 8-DOF image homographies, while VGGT-SLAM formulates point-cloud alignment on the 15-DOF SL(4) manifold.The cited prior applications include panoramic stitching and dense SLAM.

3 Review: VGGT

VGGT processes image sets with transformer-based attention and predicts camera geometry, dense depth, point maps, and tracking features. VGGT-SLAM uses its dense depth and confidence outputs to construct filtered submaps.

  • VGGT Outputs: VGGT tokenizes an image set with a fine-tuned DINO backbone and applies alternating global and local attention.Its heads estimate camera intrinsics and poses, dense depth, point maps, and point-tracking features.
  • VGGT Outputs: VGGT outputs camera poses and intrinsics together with dense depth maps, dense point maps, feature tracks, and confidence estimates.The point maps are defined relative to the first camera in the input image set.
  • Submap Construction: VGGT-SLAM uses dense depth and confidence maps, then inverse-projects depth with camera-head projection matrices to obtain the dense submap point cloud.The resulting point cloud is defined in the coordinate frame of the first camera in the image set.
  • Submap Construction: Points with confidence below a threshold relative to the average confidence are pruned before submap alignment.Each VGGT image-set output is treated as a submap and a node in pose-graph optimization.

4 VGGT-SLAM

VGGT-SLAM incrementally builds VGGT submaps, aligns them with projective homographies, and globally optimizes those transformations on the SL(4) manifold, including loop closures.

  • Submap construction: VGGT-SLAM incrementally groups incoming images into submaps using disparity-based keyframes, a frame limit w, prior frames, and retrieved loop-closure frames.Keyframes are selected when disparity from the previous keyframe exceeds τ_disparity; each submap combines Ilatest with Mprior and up to wloop retrieved frames.
  • Local submap alignment: For overlapping submaps, VGGT-SLAM estimates a relative 4×4 homography that aligns their point clouds in homogeneous coordinates.The shared-frame construction supplies dense point correspondences without explicit association estimation.
  • Projective ambiguity: Uncalibrated-camera reconstructions can differ from metric geometry by a 15-DOF projective transformation in SL(4), so SE(3) or Sim(3) alignment may be insufficient.SL(4) comprises real-valued 4×4 matrices with unit determinant and is distinct from the 8-DOF planar homography group SL(3).
  • Local submap alignment: Relative homographies are estimated from a homogeneous linear system using a minimum five-point solver with RANSAC, then normalized to unit determinant.This adds robustness to incorrect VGGT depth measurements while producing transformations in SL(4).
  • Loop closures: Loop closures use SALAD image retrieval to find frames in earlier submaps, then add relative homography constraints between the retrieved frames and their originating submaps.The same direct homography estimation procedure is used because identical frames provide dense correspondences without separate association estimation.
  • Backend optimization: A nonlinear MAP factor graph globally estimates absolute submap homographies from odometry and loop-closure constraints, optimizing 15-dimensional SL(4) tangent-space states.Levenberg–Marquardt updates the Lie-group states using H ← H Exp(δ̂).

5 Experiments

VGGT-SLAM is evaluated for pose estimation, dense reconstruction, qualitative mapping, and ablations across standard RGB SLAM benchmarks. The experiments show competitive quantitative performance, improved dense accuracy and Chamfer distance, successful long-loop mapping, and benefits from SL(4) alignment and loop closure, while planar scenes remain challenging.

  • Evaluation setup: VGGT-SLAM evaluates camera poses on 7-Scenes and TUM RGB-D, and dense mapping on 7-Scenes using ATE, accuracy, completion, and Chamfer distance.DROID-SLAM, MASt3R-SLAM, and Spann3R serve as comparison methods in the reported evaluations.
  • Pose estimation: 0.053 m average error: the SL(4) version of VGGT-SLAM performs best overall on the TUM dataset.On 7-Scenes, its average APE is approximately the same as the top-performing MASt3R-SLAM baseline.
  • Limitations: Planar scenes can cause homography degeneracy and divergence, while approximately pure rotation is especially difficult for smaller submaps in TUM 360.The TUM 360 case is handled well with w = 32.
  • Dense reconstruction: Best accuracy and Chamfer distance: VGGT-SLAM achieves the strongest reported dense reconstruction results on 7-Scenes, with comparable performance across methods.The dense evaluation follows the MASt3R-SLAM protocol.
  • Qualitative results: 22 submaps over a 55 meter office-corridor loop are joined into a globally consistent map with loop closure at the trajectory end.The qualitative example uses window size w = 16 and displays mapped camera poses by submap color.
  • Ablations: Loop closures improve pose accuracy, yield larger ATE reductions as submap counts increase, and τconf = 25% balances dense accuracy against completion.These ablations cover window sizes w = 8, 16, and 32.

6 Limitations

VGGT-SLAM’s full SL(4) homography estimation introduces several stability risks, including degeneracy, outlier sensitivity, and additional scene drift.

  • Planar points make full 15-DOF homography estimation degenerate, potentially producing unstable solutions.This was observed in the planar floor scene of the TUM dataset.
  • VGGT-derived homography estimation is vulnerable to outliers, including high or adversarial outlier ratios that can cause incorrect estimates.A 5-point RANSAC is used to reduce this issue, but does not eliminate the vulnerability.
  • The 15-DOF parameterization creates additional opportunity for scene drift, although loop closures substantially correct drift.

7 Conclusion

The paper concludes that VGGT-SLAM incrementally builds dense maps from uncalibrated monocular cameras by aligning VGGT submaps on SL(4), while identifying future work toward adaptive transformation choices.

  • VGGT-SLAM incrementally constructs dense maps from uncalibrated monocular cameras by locally and globally aligning VGGT submaps through loop closures.
  • The general case requires projective transformations, motivating the first factor-graph SLAM system optimized on the SL(4) manifold.
  • Future work will study when Sim(3) suffices and how to combine Sim(3) and SL(4) for more robust real-time SLAM.
  • SL(4) has 15 generators because its tangent space contains all 4 × 4 real matrices with zero trace.
  • The relation between the Lie algebra sl(4) and Lie group SL(4) is used to formulate the optimization on the manifold.

B Extra Quantitative Results

Additional experiments evaluate 7-Scenes and TUM RGB-D under different submap sizes and report submap and loop-closure counts.

  • Additional results evaluate VGGT-SLAM on the 7-Scenes and TUM RGB-D datasets with varied submap sizes.
  • The appendix reports the number of submaps and loop closures for each scene.

B.1 Evaluation with different submap sizes

The appendix studies submap-size effects, reports trajectory and dense-reconstruction evaluations, and shows qualitative cases where SL(4) resolves alignment failures that Sim(3) cannot.

  • Evaluation with different submap sizes: Submap sizes w = 8, 16, and 32 are evaluated on both 7-Scenes and TUM RGB-D, while w = 1 is additionally evaluated on 7-Scenes.
  • Evaluation with different submap sizes: For w = 1, TUM floor and 360 scenes are excluded because the backend becomes numerically unstable and cannot estimate an alignment.
  • Evaluation with different submap sizes: Planar views make full 15-DOF homography estimation degenerate, while pure rotation can reduce VGGT depth accuracy and relative homography accuracy.
  • Evaluation with different submap sizes: The evaluations report RMSE of absolute trajectory error on 7-Scenes and TUM RGB-D, plus dense reconstruction quality on 7-Scenes.
  • Evaluation with different submap sizes: The appendix also records submap and loop-closure counts for different window sizes.
  • Extra examples of SL(4) versus Sim(3): Qualitative examples show SL(4) correcting projective alignment failures that leave Sim(3) reconstructions inconsistent.Examples include hallucinated duplicate bananas and clustered bollards caused by Sim(3) misalignment.

C.2 7-Scenes Qualitative Results

Additional visualizations show VGGT-SLAM scene reconstructions on 7-Scenes using SL(4) with default parameters. The fire and heads scenes each use two submaps, with camera pose estimates colored by submap.

  • Additional 7-Scenes reconstructions are shown for VGGT-SLAM with SL(4) using the default parameters from Sec. 5.
  • The fire scene reconstruction uses two submaps, with camera pose estimates colored by submap.
  • The heads scene reconstruction uses two submaps, with camera pose estimates colored by submap and part of the scene cropped for visual clarity.

C.3 TUM RGB-D Qualitative Results

Additional visualizations show VGGT-SLAM scene reconstructions on TUM RGB-D using SL(4) with default parameters. The displayed scenes use either six or two submaps, and camera pose estimates are colored by submap.

  • Additional TUM RGB-D reconstructions are shown for VGGT-SLAM with SL(4) using the default parameters from Sec. 5.
  • The TUM room scene reconstruction uses six submaps, with camera pose estimates colored by submap and part of the scene cropped for visual clarity.
  • The TUM 360 scene reconstruction uses six submaps, with camera pose estimates colored by submap and part of the scene cropped for visual clarity.
  • The TUM xyz scene reconstruction uses two submaps, with camera pose estimates colored by submap.
Loading 2505.12549v2…