Source-linked AI summary
InfiniteVGGT: Visual Geometry Grounded Transformer for Endless Streams
Shuai Yuan, Yantai Yang, Xiaotian Yang, Xupeng Zhang, Zhonghao Zhao, Lingming Zhang, Zhipeng Zhang
TL;DR
Existing offline and streaming systems do not jointly provide scalable, stable geometry understanding over truly unbounded sequences, and long-term evaluation has lacked continuous benchmarks. InfiniteVGGT introduces a training-free rolling-memory mechanism with adaptive token pruning, while Long3D evaluates continuous estimation on sequences of about 10,000 frames. The paper reports bounded-memory infinite-horizon streaming, improved reconstruction accuracy and robustness, and resolution of baseline out-of-memory failures on long sequences.
Problem
Offline methods are ill-suited to online or unbounded sequences, while streaming methods can fail on infinite horizons or suffer long-term drift; continuous benchmarks have also been lacking.
Method
InfiniteVGGT uses a training-free rolling memory with key-space redundancy detection and adaptive, layer-wise token-budget allocation, without materializing attention weights.
Results
InfiniteVGGT surpasses existing explicit- and implicit-state methods in reconstruction accuracy and robustness while maintaining bounded memory and computational efficiency for infinite-horizon streaming.
Takeaways & Limitations
Long3D enables rigorous evaluation of extended-sequence 3D geometry estimation, including sequences of about 10,000 frames.
Abstract
from arXiv · showhide
The grand vision of enabling persistent, large-scale 3D visual geometry understanding is shackled by the irreconcilable demands of scalability and long-term stability. While offline models like VGGT achieve inspiring geometry capability, their batch-based nature renders them irrelevant for live systems. Streaming architectures, though the intended solution for live operation, have proven inadequate. Existing methods either fail to support truly infinite-horizon inputs or suffer from catastrophic drift over long sequences. We shatter this long-standing dilemma with InfiniteVGGT, a causal visual geometry transformer that operationalizes the concept of a rolling memory through a bounded yet adaptive and perpetually expressive KV cache. Capitalizing on this, we devise a training-free, attention-agnostic pruning strategy that intelligently discards obsolete information, effectively ``rolling'' the memory forward with each new frame. Fully compatible with FlashAttention, InfiniteVGGT finally alleviates the compromise, enabling infinite-horizon streaming while outperforming existing streaming methods in long-term stability. The ultimate test for such a system is its performance over a truly infinite horizon, a capability that has been impossible to rigorously validate due to the lack of extremely long-term, continuous benchmarks. To address this critical gap, we introduce the Long3D benchmark, which, for the first time, enables a rigorous evaluation of continuous 3D geometry estimation on sequences about 10,000 frames. This provides the definitive evaluation platform for future research in long-term 3D geometry understanding. Code is available at: https://github.com/AutoLab-SAI-SJTU/InfiniteVGGT
1. Introduction
3D geometry understanding must reconcile offline accuracy with online scalability and long-term stability. InfiniteVGGT addresses this tension with rolling memory, while Long3D enables evaluation on approximately 10,000-frame sequences.
- The scalability–stability dilemma: Offline methods exploit multi-view constraints but require prohibitive GPU memory for online or unbounded sequences, whereas streaming methods target sequential infinite-length processing.The two paradigms trade geometric fidelity and memory demands against immediate online operation.
- The scalability–stability dilemma: Contiguous camera trajectories create substantial KV-cache redundancy because minimal viewpoint shifts add approximately 1,000 tokens per frame.The cache can reach O(10^5) tokens within 100 frames, motivating token-level reduction.
- InfiniteVGGT: InfiniteVGGT introduces a rolling memory that dynamically refreshes token-level contents through a multi-level retention strategy.The approach avoids unbounded history accumulation while mitigating information drift from implicit state compression.
- Long3D benchmark: Long3D provides rigorous continuous 3D geometry evaluation on sequences of about 10,000 frames, addressing the shortage of long-term benchmarks.The benchmark is intended to support future research in long-term 3D scene understanding and reconstruction.
- Contributions: The paper contributes InfiniteVGGT as an explicit dynamic-memory architecture, reports state-of-the-art long-sequence performance, and introduces Long3D for long-term evaluation.It also claims robust infinite-horizon reconstruction without memory overflow.
2. Related Work
3D reconstruction research has moved from globally optimized offline pipelines toward unified feedforward and streaming architectures. These approaches differ in how they balance geometric accuracy, computational demands, and preservation of long-term dependencies.
- Classical offline and online reconstruction: Classical offline reconstruction relies on SfM pipelines such as COLMAP and global Bundle Adjustment across views and points for high global accuracy.The batch optimization is computationally demanding and supports subsequent multi-view processing.
- Classical offline and online reconstruction: Online reconstruction systems incrementally estimate camera trajectories using feature-based, direct, or dense approaches.Their sequential operation distinguishes them from globally optimized offline pipelines.
- Learning-based offline 3D reconstruction: Learning-based offline methods evolved from pairwise pointmap regression toward unified feedforward prediction of camera poses, depth, and feature tracks.VGGT performs these predictions jointly in a single forward pass, while earlier methods require global alignment.
- Streaming reconstruction: Streaming methods pursue infinite-length reconstruction through hierarchical dynamic rolling memory that preserves long-term dependencies and reduces drift and catastrophic forgetting.This line of work directly targets the stability challenges of extended sequences.
3. Method
InfiniteVGGT adapts VGGT for streaming 3D geometry by replacing global batch interaction with causal temporal attention and a bounded rolling KV memory. Its cache compression preserves immutable initial-frame anchors, retains diverse tokens, and allocates storage adaptively across layers.
- Online adaptation: VGGT processes image batches with frame and global interactions across 24 self-attention layers to estimate cameras, depth, point maps, and tracking features.The estimated quantities are camera parameters, depth maps, point maps, and point-tracking features.
- Online adaptation: StreamVGGT enables incremental frame processing by replacing global interaction with causal temporal attention over a KV cache of prior-frame context.The cache is contiguously updated by combining new keys and values.
- Rolling memory: The KV cache grows as O(t), motivating a fixed-size rolling memory that evicts redundancy while preserving valuable information.The method targets the unsustainable memory demands caused by explicit history accumulation.
- Motivation and analysis: Attention-based eviction is impractical because obtaining attention scores would require materializing the full matrix that FlashAttention avoids.This would negate the low-latency inference required for streaming systems.
- Diversity-aware retention: InfiniteVGGT uses key-space diversity, measured by negative cosine similarity to the mean normalized key, as a query-independent redundancy proxy.Keys with higher diversity scores are retained because they represent more geometrically distinct features.
- Diversity-aware retention: The rolling memory preserves all initial-frame KV tokens as immutable anchors and applies token retention independently per layer and head.The first frame defines the canonical global coordinate reference, so pruning its tokens would compromise geometric consistency.
- Adaptive budget allocation: Layer-wise budgets are allocated nonuniformly from measured information diversity, then enforced with TopK selection before recombining retained candidates with anchors.The final compressed cache is the union of retained candidate tokens and the immutable anchor set.
4. Long3D Benchmark
Long3D addresses the lack of continuous, long-term benchmarks for 3D geometry estimation with five challenging sequences spanning about 2,000 to 10,000 frames. It pairs uninterrupted RGB streams with global ground-truth point clouds for dense-view streaming reconstruction.
- Long3D provides five continuous sequences, each ranging from about 2,000 to 10,000 frames, across diverse indoor and outdoor environments.
- The benchmark addresses prior evaluation limits from short sequences or discontinuous clips that prevent rigorous long-term assessment.
- Each scene includes a global ground-truth point cloud and the corresponding uninterrupted RGB image sequence.
- Long3D evaluates dense-view streaming reconstruction by processing the entire image stream to generate a complete global point cloud.
5. Experiments
The experiments evaluate InfiniteVGGT across 3D reconstruction, video depth estimation, and camera pose estimation, including sequences approaching 10,000 frames. Results show robust long-term performance, limited temporal error accumulation, and benefits from the proposed memory design.
- Evaluation Setup: Experiments cover 3D reconstruction, video depth estimation, and camera pose estimation using long contiguous sequences and the Long3D benchmark.The evaluation compares InfiniteVGGT with CUT3R and TTT3R and focuses on long-sequence scenarios.
- 3D Reconstruction: On 7-Scenes and NRGBD, InfiniteVGGT maintains state-of-the-art reconstruction accuracy as sequence length increases, while competing methods encounter memory or degradation issues.VGGT and StreamVGGT fail on long inputs because of memory constraints; TTT3R degrades on NRGBD, and Point3R uses perpetually increasing memory.
- Long3D Benchmark: On Long3D sequences of approximately 2,000, 4,500, 6000, and nearly 10,000 frames, InfiniteVGGT outperforms CUT3R and TTT3R on most metrics while limiting temporal error propagation.The method underperforms the baselines on the mean of the Comp. metric, identified as future work.
- Video Depth Estimation: The method is evaluated on Bonn continuous sequences ranging from 200 to 500 frames to assess per-frame depth quality and inter-frame depth consistency.The passage introduces the evaluation setting but does not provide the reported numerical outcomes.
- Token Selection: Cosine-similarity token selection is compared with attention-based selection, alongside measurements of reconstruction quality, inference time, and peak GPU memory.The ablation examines chamfer distance, normal consistency, per-frame inference time, and peak GPU memory consumption.
- Memory Ablations: Smaller per-head token budgets degrade reconstruction quality, while dynamic layer-wise allocation improves point-cloud accuracy and normal consistency.The budget study compares 300- and 500-frame inputs, and the allocation study compares uniform limits with dynamic allocation.
- Anchor Frame: Retaining all tokens from the first frame prevents error accumulation and significantly improves reconstruction quality compared with pruning the initial state.The first frame serves as a global reference for the canonical coordinate system.
6. Discussion
InfiniteVGGT is designed for streaming 3D reconstruction over long sequences without unbounded memory growth. On shorter sequences it performs similarly to StreamVGGT, while on longer sequences its rolling memory avoids out-of-memory errors and limits temporal error accumulation.
- Short Sequences: On 50–100-frame inputs, InfiniteVGGT and StreamVGGT show negligible CD and NC differences, with a slight NC precision advantage for InfiniteVGGT.These sequences are short enough that StreamVGGT operates without memory constraints.
- Rolling Memory: The diversity-aware rolling memory preserves varied historical information, improving robustness to noisy data and preventing redundant later inputs from saturating memory.The mechanism retains more diverse information from early stages as sequences grow.
- Long Sequences: For long sequences, InfiniteVGGT resolves baseline out-of-memory errors and curtails temporal error accumulation, motivating the focus on demanding long-sequence scenarios.The discussion frames these benefits as critical when the input length increases.
7. Conclusion
InfiniteVGGT uses a rolling-memory strategy to mitigate unbounded memory growth and long-term drift in streaming 3D geometry understanding, while supporting bounded, efficient online processing and improved reconstruction robustness.
- InfiniteVGGT mitigates the trade-off between unbounded memory growth and long-term drift in streaming 3D geometry understanding.
- A training-free strategy identifies memory redundancy using key cosine similarity and applies adaptive, layer-wise budget allocation.
- FlashAttention compatibility supports bounded memory and computational efficiency for online streaming over infinite-horizon sequences.
- InfiniteVGGT surpasses existing explicit- and implicit-state methods in reconstruction accuracy and robustness.
- The Long3D benchmark supports rigorous evaluation of extended-sequence performance.