PaperlayerView exact revision on arXiv

Source-linked AI summary

Streaming 4D Visual Geometry Transformer

Dong Zhuo, Wenzhao Zheng, Jiahe Guo, Yuqi Wu, Jie Zhou, Jiwen Lu

arXiv:2507.11539v2cs.CVcs.AIcs.LG

TL;DR

Online 3D reconstruction from video calls for low latency and temporal consistency, while global-attention approaches limit efficient streaming. StreamVGGT uses causal processing, cached memory, and teacher-to-student distillation; it matches offline VGGT with small performance degradation and surpasses online state-of-the-art models across multiple tasks.

  • Problem

    Global self-attention approaches achieve strong reconstruction accuracy but limit efficient processing of streaming inputs, despite demand for interactive systems where latency and temporal consistency matter.

    Source: §1 · p. 2, §3.1 · p. 4

  • Method

    StreamVGGT uses causal frame-by-frame processing with cached historical-token memory and teacher-to-student distillation.

    Source: §3.2 · pp. 4–5, §3.3 · p. 6

  • Results

    StreamVGGT achieves accuracy comparable to offline VGGT with small performance degradation and surpasses current online state-of-the-art models across multiple tasks.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The findings mark a step toward scalable, low-latency streaming 3D vision systems.

    Source: §5 · p. 10

    StreamVGGT faces scalability challenges on very long sequences, for which windowed streaming or K-nearest-frames caching bounds inference memory and latency.

    Source: §4.6 · p. 10

Abstract

from arXiv · show

Perceiving and reconstructing 3D geometry from videos is a fundamental yet challenging computer vision task. To facilitate interactive and low-latency applications, we propose a streaming visual geometry transformer that shares a similar philosophy with autoregressive large language models. We explore a simple and efficient design and employ a causal transformer architecture to process the input sequence in an online manner. We use temporal causal attention and cache the historical keys and values as implicit memory to enable efficient streaming long-term 3D reconstruction. This design can handle low-latency 3D reconstruction by incrementally integrating historical information while maintaining high-quality spatial consistency. For efficient training, we propose to distill knowledge from the dense bidirectional visual geometry grounded transformer (VGGT) to our causal model. For inference, our model supports the migration of optimized efficient attention operators (e.g., FlashAttention) from large language models. Extensive experiments on various 3D geometry perception benchmarks demonstrate that our model enhances inference speed in online scenarios while maintaining competitive performance, thereby facilitating scalable and interactive 3D vision systems. Code is available at: https://github.com/wzzheng/StreamVGGT.

1 INTRODUCTION

StreamVGGT enables low-latency online 3D reconstruction by incrementally processing video with causal attention and historical-token memory. It distills geometric knowledge from bidirectional VGGT and reports reduced long-sequence inference overhead with a slight performance trade-off.

1 INTRODUCTION

  • StreamVGGT targets streaming reconstruction where low latency and temporal consistency are crucial for online interactive visual systems.The introduction motivates this need by the growing demand for on-the-fly reconstruction from streaming inputs.Source: §1 · p. 2
  • StreamVGGT reduces long-sequence inference overhead with only a slight performance trade-off, while supporting fast current-frame inference compared with VGGT.Figure 2 compares inference time and memory for the current frame across varying sequence lengths in the online setting.Source: Figure 2 · p. 2, §1 · p. 2
  • Causal temporal attention and implicit historical-token memory let StreamVGGT process incoming frames incrementally, avoiding reprocessing the entire sequence after each new frame.The design enables progressive scene updates in an online streaming manner.Source: §1 · p. 2
  • For training, StreamVGGT distills global geometric understanding from bidirectional VGGT (Wang et al., 2025a), reducing training cost while aiming for accuracy comparable to full-sequence inference.Teacher-generated pseudo-GT unifies multi-task supervision without per-dataset engineering; soft targets and confidence estimates act as regularizers that improve robustness and generalization.Source: §1 · p. 2
  • Equipped with FlashAttention-2 (Dao, 2023), StreamVGGT supports fast inference for the current frame compared with VGGT, while reducing inference overhead in long-term sequences with only a slight performance trade-off.Source: §1 · p. 2

2 RELATED WORK

The review traces the limits of offline reconstruction and dense-view transformers, then contrasts prior streaming memory designs with the paper’s causal implicit cache.

Conventional 3D Reconstruction

  • Conventional reconstruction combines SfM’s sparse geometry with dense MVS and NeRF, but offline optimization limits dynamic-scene robustness and low-latency incremental updates.SfM uses feature matching and bundle adjustment; MVS relies on accurate camera poses and photometric consistency, while NeRF fits a volume-rendering equation.Source: §2 · pp. 2–3

Learning-Based 3D Reconstruction

  • Learning-based methods improve robustness and generalization; DUSt3R regresses point maps from two uncalibrated images, while MASt3R adds confidence-weighted losses to approximate metric scale (Wang et al., 2024a; Leroy et al., 2024).Feed-forward transformers enable dense-view interaction in a single pass and deliver state-of-the-art accuracy in a few seconds (Yang et al., 2025; Wang et al., 2025a).Source: §2 · p. 3
  • Fast3R processes over 1000 images, while VGGT jointly predicts cameras, dense depth, point maps, and tracking features but must re-encode every frame when new images arrive (Yang et al., 2025; Wang et al., 2025a).VGGT’s quadratic token-pair complexity, memory footprint, and noncausal processing preclude streaming reconstruction and limit low-latency use.Source: §2 · p. 3

Streaming 3D Reconstruction

  • Prior streaming systems use token-addressable memory, recurrent scene state, or geometry-aligned pointers, whereas this work uses causal attention to cache historical visual tokens implicitly.Spann3R (Wang & Agapito, 2024) can drift on long or dynamic sequences; CUT3R degrades far from observed regions and requires heavy recursive training; Point3R (Wu et al., 2025) provides low-drift real-time updates.Source: §2 · p. 3

3 PROPOSED APPROACH

StreamVGGT uses causal attention and cached historical tokens to reconstruct 3D geometry frame by frame, while distilling geometric knowledge from bidirectional VGGT to train the causal student efficiently. Its task heads predict scene geometry and camera attributes in a shared reference frame.

3.1 STREAMING 3D GEOMETRY RECONSTRUCTION

  • Causal attention limits each frame to itself and its predecessors, reducing latency to O(N) and enabling low-latency 3D perception.This addresses the streaming inefficiency of global self-attention, which relies on all-to-all frame interaction.Source: §3.1 · p. 4, Figure 3 · p. 4

3.2 CAUSAL ARCHITECTURE WITH CACHED MEMORY TOKEN

  • During streaming inference, StreamVGGT caches historical tokens and attends to them alongside current-frame image tokens; this matches training-time causal attention and achieves performance comparable to full-sequence inference.The architecture processes inputs and predicts outputs frame by frame.Source: §3.2 · pp. 4–5, Figure 4 · p. 5
  • Its task heads predict camera pose, point maps, depth maps, and point tracks, using a learnable camera token to align subsequent frames to the first frame’s shared coordinate system.This enables consistent streaming reconstruction without post-processing.Source: §3.2 · p. 5

3.3 DISTILLATION-BASED TRAINING

  • Distillation transfers bidirectional VGGT’s soft predictions and confidence estimates for camera parameters, depth, point maps, and tracks, enabling teacher-level performance with fewer GPU hours than training on full annotations.The teacher’s pseudo-labels provide uncertainty-aware supervision across tasks and datasets; removing distillation under the same training time and compute budget degrades accuracy.Source: §3.3 · pp. 6–7

4 EXPERIMENTS

Across reconstruction, depth, and camera-pose benchmarks, StreamVGGT delivers competitive results with streaming inputs; experiments also examine its training setup and efficiency trade-offs, including strategies for long sequences.

4.1 IMPLEMENTATION DETAILS; 4.2 TRAINING DATASETS

  • StreamVGGT uses 24 temporal and spatial attention layers, FlashAttention-2, and VGGT pre-trained weights, with approximately 950 million parameters fine-tuned for 10 epochs.Training used a curated 13-dataset collection spanning indoor and outdoor domains, temporal scales, synthetic data, and real-world captures.Source: §4.1 · p. 7, §4.2 · p. 7

4.4 SINGLE-FRAME AND VIDEO DEPTH ESTIMATION

  • On four training-excluded datasets, StreamVGGT matches overall best single-frame depth performers and outperforms the state-of-the-art streaming model; Table 4 reports Abs Rel and δ1.25.The evaluation covers KITTI, Sintel, Bonn, and NYU-v2, spanning dynamic and static, indoor and outdoor scenes.Source: Table 4 · p. 8, §4.4 · p. 8
  • Under per-sequence scale alignment, StreamVGGT exceeds CUT3R on Sintel and Bonn video depth benchmarks and is comparable to offline VGGT.Video depth evaluation assesses both per-frame depth quality and cross-frame consistency.Source: Table 5 · p. 8, §4.4 · p. 8

4.3 3D RECONSTRUCTION

  • On 7-Scenes and NRGBD, StreamVGGT surpasses streaming model CUT3R; on ETH3D it surpasses DUSt3R and MASt3R and matches offline VGGT, while showing strong results on dynamic TUM-dynamics scenes.Evaluations use sparse inputs on 7-Scenes and NRGBD, randomly sampled frames on ETH3D, and 50 frames per TUM-dynamics sequence.Source: §4.3 · p. 8

4.5 CAMERA POSE ESTIMATION

  • On ScanNet, Sintel, and TUM-dynamics, StreamVGGT performs on par with the best existing camera-pose methods while supporting streaming-input prediction.Pose errors are evaluated after Sim(3) alignment using ATE, RPE trans, and RPE rot.Source: Table 6 · p. 9, §4.5 · p. 9

4.6 EXPERIMENTAL ANALYSIS

  • For long sequences, cached tokens increase StreamVGGT’s inference time and memory use; windowed streaming, K-nearest-frames caching, and FlashAttention-2 help bound resource costs or reduce overhead.The pruning strategies bound memory and latency while maintaining high accuracy; cached memory tokens reduce inference time, while FlashAttention-2 reduces memory overhead.Source: Table 7 · p. 9, §4.6 · pp. 9–10, Table 8 · p. 10
  • Under the same training budget, distillation brings StreamVGGT close to VGGT across 3D reconstruction metrics, while the non-distilled model has higher reconstruction errors.Qualitative visualizations show scene reconstructions with geometric fidelity and fewer outliers in complex environments.Source: Figure 5 · p. 9, Table 9 · p. 10, §4.6 · p. 10

5 CONCLUSION

StreamVGGT uses causal temporal attention and cached token memory for incremental 3D scene updates with long-term spatial consistency. It achieves accuracy comparable to offline VGGT with small degradation, surpasses current online state-of-the-art models across multiple tasks, and motivates future work on dynamic-scene adaptation and lightweight memory compression.

5 CONCLUSION

  • StreamVGGT achieves accuracy comparable to offline VGGT with small performance degradation while surpassing current online state-of-the-art models across multiple 3D and depth-estimation tasks.The passage names 3D reconstruction, single-frame depth estimation, and depth estimation.Source: §5 · p. 10
  • Causal temporal attention replaces global self-attention, while cached token memory enables incremental scene updates and preserves long-term spatial consistency.Source: §5 · p. 10
  • Future work will explore dynamic scene adaptation and lightweight memory compression to bridge the gap between offline precision and mobile deployment requirements.Source: §5 · p. 10
  • The work aims to advance scalable, low-latency streaming 3D vision systems for applications such as autonomous navigation and immersive AR/VR.Source: §5 · p. 10

A APPENDIX · A.1 RECONSTRUCTION PERFORMANCE IN COMPLEX SCENARIOS

The appendix evaluates StreamVGGT in complex reconstruction scenarios, reporting robust temporal consistency on long sequences and usable performance under loop closure, missing frames, and noisy inputs.

A.1 RECONSTRUCTION PERFORMANCE IN COMPLEX SCENARIOS

  • The appendix assesses reconstruction in complex conditions, including long sequences and dynamic scenes.It provides quantitative results for these scenarios.Source: §A.1 · p. 15
  • StreamVGGT maintains robust temporal consistency on 7-Scenes sequences with over 70 frames, evaluated at every 10th frame.The per-frame results are shown in Figure 6.Source: §A.1 · p. 15
  • In simulated loop closure on 7-Scenes, StreamVGGT achieves lower drift and usable performance when matching the first and last frames.Each sequence uses the first 50 frames in forward-and-reverse order, with pose error measured between the repeated frame.Source: §A.1 · p. 15
  • When the middle 25 frames are removed from the first 50 frames of each 7-Scenes sequence, StreamVGGT handles the interruption better than CUT3R.The interrupted-stream scenario is evaluated quantitatively.Source: §A.1 · p. 15
  • On 7-Scenes inputs corrupted by randomly masking regions, StreamVGGT retains correct reconstruction, indicating robustness to noisy frames.The noisy-frame results are reported in Table 13.Source: §A.1 · p. 15

A.2 MORE ANALYSIS

The analysis examines cached memory, reconstruction and pose visualizations, failure cases, and comparisons with a concurrent method. It reports benefits for streaming consistency and pose estimation, while identifying conditions that can cause reconstruction failures.

Cached Memory Token

  • Cached memory tokens accelerated streaming inference while preserving temporal consistency and perceptual accuracy on ETH3D.The evaluation compared full-sequence and streaming input; latency was measured for the final frame of a 40-frame stream.Source: §A.2 · p. 15, Table 14 · p. 16

More Visualizations

  • On 7-Scenes, StreamVGGT’s reconstructions showed fewer outliers and superior spatial consistency compared with CUT3R and the ground truth.The reconstruction comparisons are shown in Figure 7.Source: §A.2 · p. 16, Figure 7 · p. 16

Failure Cases

  • Severe dynamic occlusion and aggressive ego-motion caused reconstruction drift or localization failure in some cases.These failure cases are visualized in Figure 8.Source: §A.2 · p. 16, Figure 8 · p. 16

Visualizations of Pose Estimation

  • On ScanNet pose visualizations, StreamVGGT accurately estimated camera poses across diverse scenarios and outperformed CUT3R on camera-pose trajectories.The visualizations are shown in Figures 9 and 10.Source: §A.2 · p. 16, Figure 9 · p. 17, Figure 10 · p. 17

More Quantitative Results

  • StreamVGGT achieved comparable results to STream3R using 13 training datasets and 4 A800 GPUs instead of 29 datasets and 8 A100 GPUs, with both trained for seven days.The authors attribute the reduced training-resource demand and faster training process to knowledge distillation.Source: §A.2 · pp. 16–17, Table 15 · p. 18, Table 16 · p. 18

A.3 FURTHER DISCUSSIONS

StreamVGGT’s cached token memory supports low-latency incremental 3D reconstruction with long-term spatial consistency, but its memory and computational costs grow with sequence length, limiting lightweight deployment.

Limitations

  • Cached token accumulation rapidly increases memory use and computational overhead over long sequences, challenging deployment on lightweight or mobile devices.Improving memory efficiency while preserving accuracy remains a future optimization need.Source: §A.3 · p. 18

Broader Impacts

  • Temporal attention and cached token memory enable low-latency incremental multi-view processing while maintaining long-term spatial consistency for dynamic applications such as navigation, robotics, and AR/VR.These applications rely on timely, accurate 3D scene understanding.Source: §A.3 · p. 18

Related papers

Loading 2507.11539v2…