Source-linked AI summary
FastVGGT: Training-Free Acceleration of Visual Geometry Transformer
You Shen, Zhipeng Zhang, Yansong Qu, Xiawu Zheng, Jiayi Ji, Shengchuan Zhang, Liujuan Cao
TL;DR
Long-sequence 3D geometry inference is limited by VGGT’s costly Global Attention and redundant token interactions. FastVGGT introduces training-free, 3D-specific token merging with protected spatial anchors and salient tokens, achieving up to 4× speedup on 1000-image inputs while maintaining competitive accuracy and reducing error accumulation.
Problem
VGGT’s dense global token interactions make long-sequence inference computationally inefficient, with Global Attention becoming the primary bottleneck.
Method
FastVGGT applies training-free token merging with a 3D-specific partitioning strategy that protects the initial-frame reference and salient correspondence tokens.
Results
4× speedup on 1000-image inputs is achieved over VGGT while mitigating error accumulation in long-sequence reconstruction.
Takeaways & Limitations
Token merging shows potential as a scalable approach for feed-forward 3D vision systems across camera pose estimation and 3D reconstruction.
Takeaways & Limitations
The original VGGT encounters OOM errors around 300 frames, so long-sequence scalability relies partly on the memory-optimized VGGT* variant.
Abstract
from arXiv · showhide
Foundation models for 3D vision have recently demonstrated remarkable capabilities in 3D perception. However, scaling these models to long-sequence image inputs remains a significant challenge due to inference-time inefficiency. In this work, we present a detailed analysis of VGGT, a state-of-the-art feed-forward visual geometry model and identify its primary bottleneck. Visualization further reveals a token collapse phenomenon in the attention maps. Motivated by these findings, we explore the potential of token merging in the feed-forward visual geometry model. Owing to the unique architectural and task-specific properties of 3D models, directly applying existing merging techniques proves challenging. To this end, we propose FastVGGT, which, for the first time, leverages token merging in the 3D domain through a training-free mechanism for accelerating VGGT. we devise a unique token partitioning strategy tailored to 3D architectures and tasks, effectively eliminating redundant computation while preserving VGGT's powerful reconstruction capacity. Extensive experiments on multiple 3D geometry benchmarks validate the effectiveness of our approach. Notably, with 1000 input images, FastVGGT achieves a 4x speedup over VGGT while mitigating error accumulation in long-sequence scenarios. These findings underscore the potential of token merging as a principled solution for scalable 3D vision systems. Code is available at: https://mystorm16.github.io/fastvggt/.
1. Introduction
VGGT provides accurate feed-forward 3D reconstruction but becomes difficult to scale to long image sequences because Global Attention incurs rapidly increasing computation. FastVGGT addresses this redundancy with training-free token merging tailored to 3D geometry, achieving faster long-sequence inference while preserving reconstruction capabilities.
- Motivation: VGGT directly regresses camera parameters, depth maps, and point tracks, but dense cross-view token interactions create quadratic time complexity O(n^2d).Flash-Attention reduces memory complexity from O(n^2) to O(nd), while the underlying time complexity remains quadratic.
- Motivation: Global Attention increasingly dominates VGGT inference as the number of input frames grows.Its cost escalates rapidly with sequence length, eventually dominating the runtime profile.
- Motivation: Attention maps show strong similarity across tokens, indicating substantial redundancy in Global Attention computation.This observation motivates reducing redundant token processing without compromising VGGT’s reconstruction capability.
- Approach: FastVGGT introduces training-free token merging for feed-forward visual geometry models, preserving initial-frame tokens as global references and protecting salient tokens.The strategy is tailored to VGGT’s cross-image correspondences and reconstruction requirements.
- Results: 4× speedup on 1000-image inputs is achieved over VGGT while mitigating error accumulation in long-sequence reconstructions.The optimized VGGT variant also handles inputs exceeding 1000 images, whereas the original encounters OOM errors beyond roughly 300 images.
- Results: The paper reports acceleration on large-scale inputs while preserving reconstruction quality and mitigating long-sequence error accumulation.These contributions include identifying VGGT’s bottleneck and introducing token merging into feed-forward visual geometry architectures.
2. Related work
Prior work established end-to-end neural approaches for robust 3D reconstruction and training-free visual token merging for efficient transformer inference. FastVGGT builds on these lines by extending token merging beyond primarily image-domain applications toward long-sequence 3D settings.
- End-to-end 3D reconstruction: DUSt3R initiated end-to-end 3D reconstruction by directly regressing view-consistent point maps from two RGB images without camera calibration.Subsequent methods, including MASt3R and related works, extend this learning-based reconstruction direction.
- Visual token merging: Visual token merging partitions tokens, matches each source with its most similar destination, averages them, and concatenates the resulting representations.The training-free approach improves Vision Transformer throughput and has been extended to diffusion, video, language understanding, and video editing.
- Visual token merging: Most token-merging applications remain focused on images, while long-form video remains comparatively underexplored despite its spatiotemporal structure.This limitation motivates adapting token merging to multi-view 3D geometry models.
3. Method
FastVGGT compresses redundant tokens during VGGT’s Global Attention while preserving spatial anchors, salient correspondences, uniform frame coverage, and dense per-token outputs. Its design combines 3D-specific token partitioning, similarity-based merging, and unmerging for reconstruction-compatible acceleration.
- 3.1. Visualization Findings: VGGT Global Attention maps contain highly similar patterns across many tokens and blocks, motivating token merging to reduce redundant computation.Each image contains 1,041 tokens: one camera token, four register tokens, and 1,036 patch tokens.
- 3.1. Visualization Findings: Direct random or fixed-stride token merging reduces inference time but significantly increases Chamfer Distance and degrades reconstruction accuracy.These results show that conventional 2D-oriented partitioning is unsuitable without adaptation to VGGT’s 3D reconstruction structure.
- 3.2. Token Partitioning: FastVGGT preserves cross-frame correspondence, uniform intra-frame coverage, and representative tokens through a 3D-specific partitioning strategy.The first frame supplies destination tokens as the global spatial anchor; salient tokens are protected, and region-based sampling balances selection within each frame.
- 3.3. Token Merging Procedure: Source tokens are assigned to their most similar destination tokens, reducing the sequence length processed by Global Attention.The matching uses cosine similarity between source and destination token representations, after which source information is merged into the selected destination.
- 3.4. Token Unmerging Procedure: Unmerging replicates merged representations to restore the original token resolution and produce dense outputs for every patch.An explicit source–destination mapping makes this restoration deterministic while retaining Global Attention’s compressed computation.
- 3.5. Memory Optimization: VGGT* discards unused intermediate outputs during inference, reducing memory consumption for longer input sequences.Although VGGT has 24 encoder blocks, only layers 4, 11, 17, and 23 are required for inference outputs.
4. Experiments
Experiments evaluate FastVGGT on point-map reconstruction and camera-pose estimation across multiple datasets and sequence lengths. Results show substantial acceleration, preserved reconstruction quality, reduced long-sequence pose error, and sensitivity to token-partitioning and merging choices.
- Benchmark setup: FastVGGT is evaluated on ScanNet-50, 7 Scenes, and NRGBD for camera pose estimation and point-map reconstruction.ScanNet-50 uses 50 sampled scenes; the architecture follows VGGT with 24 frame and global attention layers and Flash-Attention2.
- 3D reconstruction: FastVGGT maintains robust 3D reconstruction performance on 7 Scenes and NRGBD under long-sequence evaluation.The reported metrics are accuracy, completeness, and normal consistency, with keyframes sampled every 3 or 10 frames.
- Token partitioning: Salient-token protection performs best among tested partitioning strategies, while using the first frame as destination tokens provides substantial gains.Random sampling performs poorly, and region-based uniform sampling improves performance but remains suboptimal.
- Merging location and intensity: A 90% merging ratio from block 0 reduces inference time with only minor Chamfer Distance fluctuations, providing the adopted accuracy-efficiency balance.The study varies both the starting block and merging ratio.
5. Conclusion
FastVGGT accelerates VGGT inference through training-free, strategically designed token merging that targets global-attention redundancy. Across multiple benchmarks, it preserves competitive accuracy, mitigates long-sequence error accumulation, and reaches up to 4× speedup on 1000-image inputs.
- Conclusion: FastVGGT accelerates VGGT inference through training-free token merging tailored to feed-forward visual geometry models.The method targets the global-attention bottleneck and observed attention-map similarity.
- Conclusion: Across camera-pose and 3D-reconstruction tasks, FastVGGT maintains competitive accuracy while mitigating error accumulation in long-sequence settings.The conclusion reports results across multiple benchmarks.
- Conclusion: Up to 4× speedup on 1000-image inputs highlights token merging as a scalable approach for visual geometry models.The authors describe the method as immediately practical for real-world applications.