Source-linked AI summary
VGGT-Long: Chunk it, Loop it, Align it -- Pushing VGGT's Limits on Kilometer-scale Long RGB Sequences
Kai Deng, Zexin Ti, Jiawei Xu, Jian Yang, Jin Xie
TL;DR
Large-scale monocular RGB reconstruction is limited by foundation models’ memory demands and the difficulty of maintaining consistency over long sequences without calibration. VGGT-Long processes overlapping chunks, aligns them, and applies lightweight loop closure, achieving scalable reconstruction across KITTI, Waymo, and Virtual KITTI. The results support using foundation models for kilometer-scale outdoor reconstruction without a complex backend.
Problem
Monocular RGB reconstruction remains difficult for kilometer-scale, uncalibrated driving sequences, while foundation models are constrained by memory and long-sequence drift.
Method
VGGT-Long independently processes overlapping sequence chunks with VGGT, aligns adjacent chunks using confidence, and corrects accumulated drift through loop closure.
Results
VGGT-Long achieves accurate, scalable reconstruction across KITTI, Waymo, and Virtual KITTI, including sequences where other foundation models fail from memory overflow.
Takeaways & Limitations
The framework demonstrates that foundation models can serve as a frontend for large-scale monocular reconstruction without camera calibration or a complex backend.
Takeaways & Limitations
Some central-intersection regions remain misaligned when no explicit loop-closure constraint covers the input image.
Abstract
from arXiv · showhide
Foundation models for 3D vision have recently demonstrated remarkable capabilities in 3D perception. However, extending these models to large-scale RGB stream 3D reconstruction remains challenging due to memory limitations. In this work, we propose VGGT-Long, a simple yet effective system that pushes the limits of monocular 3D reconstruction to kilometer-scale, unbounded outdoor environments. Our approach addresses the scalability bottlenecks of existing models through a chunk-based processing strategy combined with overlapping alignment and lightweight loop closure optimization. Without requiring camera calibration, depth supervision or model retraining, VGGT-Long achieves trajectory and reconstruction performance comparable to traditional methods. We evaluate our method on KITTI, Waymo, and Virtual KITTI datasets. VGGT-Long not only runs successfully on long RGB sequences where foundation models typically fail, but also produces accurate and consistent geometry across various conditions. Our results highlight the potential of leveraging foundation models for scalable monocular 3D scene in real-world settings, especially for autonomous driving scenarios. Code is available at https://github.com/DengKaiCQ/VGGT-Long.
1. Introduction
Monocular RGB reconstruction is important for autonomous driving but remains difficult at kilometer scale without calibration, partly because foundation models cannot fit long sequences in GPU memory. VGGT-Long addresses this gap with overlapping chunk processing, alignment, and loop-closure drift correction rather than a complex backend.
- Monocular RGB reconstruction must handle kilometer-scale, uncalibrated driving sequences with sparse correspondence, dynamic objects, and difficult outdoor conditions.
- Transformer foundation models unify camera pose estimation, intrinsic regression, and point-map scene representation in one learned framework.
- VGGT processes only 60 to 80 images on a 24 GiB RTX 4090, making a roughly 4,600-frame KITTI trajectory exceed available GPU memory.
- VGGT-Long asks whether VGGT’s large-scale reconstruction capability can be unlocked with minimal overhead instead of another full system.
- VGGT-Long processes overlapping chunks, aligns adjacent reconstructions, and uses loop closure to correct accumulated drift without a graph-based optimization backend.
- The proposed system targets kilometer-scale, unbounded outdoor reconstruction without camera calibration or depth supervision, while addressing foundation-model memory limits and Sim(3) drift.
2. Related Work
Prior work spans classical SfM and SLAM, learning-based systems, and Transformer foundation models for dense 3D reconstruction. VGGT-Long adopts a minimalist chunk-and-align strategy to extend these models to long, large-scale sequences without a complex system.
- Classical SfM estimates camera poses and sparse 3D structure through keypoint detection, feature matching, and bundle adjustment.
- Traditional and learning-based SLAM methods either scale poorly to long sequences or require pre-calibrated cameras.
- Transformer foundation models estimate camera parameters and dense geometry from overlapping, uncalibrated images, with VGGT producing stable local 3D maps from raw RGB.
- Their computational and memory costs restrict Transformer-based 3D vision models to short image sequences.
- VGGT-Long uses a simple chunk-and-align framework to extend VGGT to long-sequence, large-scale scenarios with minimal overhead.
3. Method
VGGT-Long extends VGGT to long monocular RGB sequences by processing overlapping chunks, aligning them with confidence-aware robust estimation, and correcting accumulated drift through loop closure and global optimization.
- Sequence chunking: Long monocular RGB sequences are decomposed into overlapping chunks, each independently processed by VGGT to produce local camera poses, point maps, and confidence values.The chunk size is L and overlap size is O; chunk k spans frames from (k−1)(L−O) to (k−1)(L−O)+L.
- Confidence-aware alignment: Confidence-aware alignment down-weights or discards points from dynamic objects and sky regions while emphasizing high-confidence static structures such as buildings.Points below 0.1× the median confidence are directly discarded in the implementation.
- Chunk-wise alignment: Adjacent chunks are aligned using overlapping 3D point correspondences and an IRLS optimization with Huber loss and weighted Umeyama updates.The weights combine VGGT confidence with a robustness term, reducing the influence of outliers.
- Loop closure: Loop closure detects non-adjacent chunks depicting the same scene, then estimates their Sim(3) relationship through a temporary loop-centric reconstruction.A DINOv2-based visual place recognition model proposes candidates using descriptor similarity and temporal separation.
- Global optimization: Global Levenberg-Marquardt optimization jointly adjusts chunk transformations using adjacent-chunk and loop-closure constraints to reduce accumulated drift.The optimization works directly with a small set of Sim(3) variables rather than constructing a complex factor graph.
4. Experiments
VGGT-Long is evaluated on KITTI, Waymo, and Virtual KITTI using trajectory, reconstruction, robustness, runtime, and ablation analyses. Across these settings, chunk-wise processing, alignment, and loop optimization enable scalable reconstruction without camera calibration, while foundation-model baselines often fail on long sequences.
- Evaluation Setup: VGGT-Long is evaluated on KITTI, Waymo, and Virtual KITTI using ATE and point-map reconstruction metrics.ATE measures long-term trajectory consistency; reconstruction evaluation uses accuracy, completeness, and Chamfer Distance after alignment and confidence-based filtering.
- KITTI Results: VGGT-Long runs successfully on all KITTI sequences, whereas Fast3R, CUT3R, and VGGT fail from memory overflow.It also maintains competitive accuracy without calibrated camera intrinsics and outperforms learning-based methods including DROID-SLAM and DPVO.
- Waymo Results: 1.996m average ATE is achieved across ten 200-frame Waymo segments, with significantly lower errors than CUT3R and MASt3R-SLAM.The method produces more accurate and complete reconstructions on segments with strong viewpoint diversity.
- Virtual KITTI Results: VGGT-Long maintains stable ATE across Virtual KITTI weather and lighting conditions, including fog, rain, and sunset, without retraining or domain adaptation.DROID-SLAM occasionally fails to track, while CUT3R exhibits large drifts under these synthetic domain shifts.
- Efficiency: 2.6–2.8s per chunk and about 0.2s for Sim(3) alignment demonstrate efficient processing, while loop optimization converges within 3 iterations on average.Each solver step takes less than 15ms, supporting real-time or near-real-time operation on kilometer-scale sequences; disk latency is reported as insignificant overall.
- Ablation Study: 58.69m ATE on KITTI Seq. 00 results when loop closure is removed, while disabling IRLS causes a 13% performance drop.The best performance is obtained with loop closure, IRLS, and confidence-weighted alignment enabled.
5. Conclusion
The paper presents VGGT-Long as a framework for extending foundation-model-based monocular RGB reconstruction to long, unbounded video sequences. Experiments across real-world and synthetic environments show accurate and scalable reconstruction without camera calibration.
- Conclusion: VGGT-Long extends monocular RGB-only 3D reconstruction to long, unbounded video sequences using foundation models.The framework is presented as a simple approach that addresses the GPU memory limitations of existing 3D vision models without camera calibration.