PaperlayerView exact revision on arXiv

Source-linked AI summary

VGGT-World: Transforming VGGT into an Autoregressive Geometry World Model

Xiangyu Sun, Shijie Wang, Fengyi Zhang, Lin Liu, Caiyan Jia, Ziying Song, Zi Huang, Yadan Luo

arXiv:2603.12655v1cs.CV

TL;DR

Geometry foundation models recover observed structure but do not forecast its evolution, and video-generation predictions can remain geometrically inconsistent. VGGT-World autoregressively predicts frozen VGGT geometry features; on KITTI and Cityscapes, it outperforms prior depth-forecasting models, with mean AbsRel reductions up to 21% and 32%.

  • Problem

    Geometry foundation models infer geometry from available views but do not predict its temporal evolution, while video-generation approaches can produce geometrically inconsistent predictions.

    Source: §1 · p. 1

  • Method

    VGGT-World treats frozen VGGT features as the world state and autoregressively predicts their future trajectory with a temporal flow transformer.

    Source: §1 · p. 3, §3.1 · p. 5

  • Results

    On KITTI and Cityscapes depth forecasting, VGGT-World reduces mean AbsRel by up to 21% and 32% over the strongest prior baselines, respectively.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results suggest geometry-centric latent forecasting is a promising alternative to full-frame generation.

    Source: §5 · p. 14

    Scalability to large, heterogeneous datasets and long-term generalization remain to be established.

    Source: D Limitations and Future Work · p. 26

Abstract

from arXiv · show

World models that forecast scene evolution by generating future video frames devote the bulk of their capacity to photometric details, yet the resulting predictions often remain geometrically inconsistent. We present VGGT-World, a geometry world model that side-steps video generation entirely and instead forecasts the temporal evolution of frozen geometry-foundation-model (GFM) features. Concretely, we repurpose the latent tokens of a frozen VGGT as the world state and train a lightweight temporal flow transformer to autoregressively predict their future trajectory. Two technical challenges arise in this high-dimensional (d=1024) feature space: (i) standard velocity-prediction flow matching collapses, and (ii) autoregressive rollout suffers from compounding exposure bias. We address the first with a clean-target (z-prediction) parameterization that yields a substantially higher signal-to-noise ratio, and the second with a two-stage latent flow-forcing curriculum that progressively conditions the model on its own partially denoised rollouts. Experiments on KITTI, Cityscapes, and TartanAir demonstrate that VGGT-World significantly outperforms the strongest baselines in depth forecasting while running 3.6-5 times faster with only 0.43B trainable parameters, establishing frozen GFM features as an effective and efficient predictive state for 3D world modeling.

1 Introduction

Geometry foundation models recover observed scene structure but do not forecast its evolution, while video-generation approaches can produce geometrically distorted predictions. VGGT-World predicts frozen VGGT features autoregressively and outperforms prior depth-forecasting models on KITTI and Cityscapes.

1 Introduction

  • Geometry foundation models infer scene structure from available views but do not predict how geometry evolves over time.Video-generation models instead predict future RGB in VAE latents, where appearance-focused objectives can leave recovered geometry distorted.Source: §1 · pp. 1–2
  • VGGT-World uses frozen VGGT features as the world state and autoregressively predicts future camera and patch tokens for geometry forecasting.A frozen VGGT decoder and heads produce geometric outputs such as depth, point maps, and camera-motion cues.Source: §1 · p. 3
  • High-dimensional VGGT features and autoregressive exposure bias make direct flow-based forecasting difficult to optimize.The approach uses z-prediction and a two-stage flow-forcing curriculum to address these challenges.Source: §1 · p. 3
  • VGGT-World reduces mean AbsRel by up to 21% and 32% over the strongest prior baselines on KITTI and Cityscapes, respectively.It is 3.6–5× faster than Cosmos (12B) and Gen3R (9B), with 0.43B trainable parameters.Source: §1 · p. 3

2 Related Work

Prior work spans observational geometry foundation models, video-generation world models, and latent feature-based prediction. The paper positions geometry-foundation features as a route to forecasting with explicit 3D structure.

Geometry Foundation Model

  • Geometry foundation models such as VGGT infer camera poses, depth, and point clouds from images, but remain observational rather than forecasting future states.Earlier multi-view models such as DUSt3R infer geometry in a single forward pass from unposed image collections.Source: §2 · pp. 3–4

Video Generation

  • Video world models generate future RGB in pixel or compressed latent spaces, while geometric-prior variants still center on RGB reconstruction.These representations lack explicit 3D structure or entangle geometry and appearance, limiting geometric consistency over long horizons.Source: §2 · p. 4

Latent World Model

  • Latent world models forecast features from vision encoders or BEV spaces, but DINO features emphasize semantic invariance rather than explicit 3D geometry.The paper contrasts those representations with geometry foundation models that encode multi-view geometry, spatial consistency, and metric structure.Source: §2 · p. 4

3 Our Approach

VGGT-World forecasts frozen VGGT geometry states rather than RGB frames, separating spatial representation from temporal dynamics. It addresses high-dimensional prediction and rollout mismatch with clean-target prediction and staged self-conditioning.

3 Our Approach

  • A frozen VGGT encoder supplies geometry states, a lightweight temporal flow transformer predicts their evolution, and the frozen decoder and 3D heads produce geometric outputs without a video VAE.A two-stage latent flow-forcing curriculum exposes the model to its own generated histories during training.Source: §3 · pp. 4–5

3.1 Geometry World Modeling via Autoregressive Flow

  • Layer-4 VGGT features are the earliest decoder-compatible state; the model predicts fixed-size chunks from a sliding k-state context, then jointly decodes history and forecasts, improving observed-step precision.The forecasted future serves as a temporal regularizer when the observed context is sparse.Source: §3.1 · pp. 5–6

3.2 Chunk Transition in High-Dimensional Geometry Space

  • In 1024-dimensional VGGT space, standard v-prediction remains noisy because Gaussian corruption perturbs off-manifold directions, while clean-target z-prediction yields higher SNR and recovers structured latents.The target space is weakly compressed, and velocity errors can drive trajectories away from the geometry manifold.Source: Figure 2 · p. 5, §3.2 · p. 7

3.3 Latent Flow Forcing Curriculum

  • Stage 1 learns with ground-truth history; Stage 2 conditions on interpolated ground-truth and partially denoised self-rollouts, with mixing weight λ annealed from 0 to 1 as rollout error decays.The mixed condition gradually shifts training toward the model's inference-time history distribution.Source: §3.3 · pp. 8–9

4 Experiments

VGGT-World is evaluated on depth forecasting across KITTI and Cityscapes and point-cloud forecasting on TartanAir. Further analyses examine rollout behavior, horizon length, current perception, camera motion, and computational efficiency.

4 Experiments

  • Experiments use Cityscapes, KITTI, and TartanAir, evaluating depth with AbsRel and δ1 and point clouds with Accuracy, Completeness, and Chamfer Distance.Depth horizons vary by dataset; point-cloud predictions are aligned to ground truth before metric computation.Source: §4 · pp. 9–10

4.1 Depth Forecasting

4.2 Point Map Forecasting

  • On TartanAir, VGGT-World outperforms Gen3R on all geometry metrics without pose/text conditioning and remains competitive with explicit pose conditions.The qualitative comparison shows more coherent geometry and closer adherence to ground-truth topology.Source: §4.2 · pp. 11–12, Table 3 · p. 12, Figure 4 · p. 12

4.3 Further Analysis

  • Without forcing, short-term performance is strong but degrades during rollout; the proposed linear schedule increases λ during training to alleviate this.Source: §4.3 · p. 12
  • For short-horizon foresight on TartanAir, VGGT-World improves completeness and overall CD over current-frame VGGT, while accuracy changes only marginally.Source: Table 4 · p. 13, §4.3 · p. 13
  • On TartanAir, reconstructed geometry improves initially with longer foresight, reaches its best CD around frame 7, then degrades with distortion.The predicted observations are aggregated with the current frame and passed to VGGT.Source: Figure 5 · p. 13, §4.3 · p. 14
  • Predicted VGGT features preserve camera-motion cues: after Umeyama alignment, trajectories show relatively consistent global structure without explicit pose supervision.The qualitative study covers KITTI and TartanAir; quantitative ATE/RTE/RRE results are reported in the Appendix.Source: Figure 6 · p. 13, §4.3 · p. 13
  • VGGT-World takes 1.9 s/frame, 5× faster than Cosmos-12B and 3.6× faster than Gen3R, while training 0.43B parameters with its 1.3B VGGT backbone frozen.The comparison uses normalized latency in seconds per generated frame.Source: Figure 7 · p. 14, §4.3 · p. 14

5 Conclusion and Future Work

VGGT-World forecasts high-dimensional geometric latents without ground-truth depth or point-cloud supervision. The authors present it as a promising alternative to full-frame generation and identify richer conditioning as future work.

5 Conclusion and Future Work

  • VGGT-World forecasts geometry directly in a high-dimensional latent space and requires no ground-truth depth or point-cloud supervision.The authors describe it as an early attempt at this form of world modeling.Source: §5 · p. 14
  • The authors report that VGGT-World surpasses substantially heavier video-generation world models in forecasting quality and efficiency.They characterize geometry-centric latent forecasting as a promising alternative to full-frame generation.Source: §5 · p. 14
  • Future work could add camera-pose and action conditioning, with action-conditioned forecasting potentially supporting more controllable and interactive rollouts for embodied agents.The authors frame these as directions for future work.Source: §5 · p. 14

Supplementary Material

The supplementary material provides theoretical analysis, implementation details, and extended experiments, organized across Sections A–D.

Supplementary Material

A Theoretical Analysis

The theoretical analysis derives a Sequential ELBO for trajectory-consistent flow forcing, using mixed conditions to express the likelihood bound across autoregressive steps. The supplementary material also specifies the latent architecture used to produce geometric outputs.

A Theoretical Analysis

  • Optimizing the Trajectory-Consistent Flow Forcing objective maximizes a Sequential ELBO on the true data likelihood.The analysis also relates rollout-error variance to step-wise KL divergence to justify the λ curriculum schedule.Source: A Theoretical Analysis · p. 20
  • Under strict causality in the forward corruption process and generative model, the likelihood bound factors over autoregressive steps, with a reconstruction term and a transition KL penalty.The KL compares the artificial forward corruption distribution with the learned autoregressive transition prior.Source: A Theoretical Analysis · pp. 20, 22
  • The derivation introduces mixed conditions as auxiliary latent variables and applies Jensen’s inequality to obtain a variational lower bound.The forward corruption distribution is parameterized by the curriculum schedule λ and generates the conditions via ODE trajectory interpolation.Source: A Theoretical Analysis · pp. 20–21

B.1 Model Architecture

  • The model extracts 1024-dimensional latents from VGGT’s DINO encoder and first five aggregator layers, then decodes predicted tokens with the remaining layers and task-specific heads.A Flow Transformer with eight double-stream and eight single-stream blocks predicts latent representations directly, with 3DRoPE encoding temporal order.Source: §B.1 · p. 22

B.2 Evaluation Datasets

The evaluation datasets cover short- and mid-horizon depth prediction on driving videos and point-cloud forecasting on TartanAir. Training uses a two-stage chunk curriculum, while the reported inference configuration uses chunk size four.

B.2 Evaluation Datasets

  • Cityscapes and KITTI evaluate pose-free depth forecasting at short and mid-term horizons, measuring frame 19 on Cityscapes and frame 13 on KITTI.The short-term look-ahead is 187.5 ms for Cityscapes and 200 ms for KITTI; the mid-term look-ahead is 562.5 ms and 600 ms, respectively.Source: §B.2 · p. 23
  • On TartanAir, the study evaluates short- and long-horizon point-cloud forecasting without pose conditions; for long-horizon forecasting, the evaluation target is the point cloud aggregated from the first seven frames.Evaluation samples 10% of scenes and 49 frames from each selected scene.Source: §B.2 · p. 23

B.3 Chunk Size Details

  • Training first predicts two target frames from two conditioning frames in chunks of four, then uses length-five chunks for trajectory-consistent flow forcing.During the second stage, the model rolls out using a ground-truth frame and a previously predicted latent; gradients pass only through this subsequent rollout stage.Source: §B.3 · pp. 23–24
  • Inference autoregressively generates with chunk size four and a sliding-window stride of one; Fig. 8 shows Cityscapes short- and mid-term depth results.The figure caption notes that its Upper Bound uses real future images and that DINO-Foresight predicts depth in a different scale.Source: §B.3 · p. 24, Figure 8 · p. 24

C.1 Qualitative Results on Cityscapes

On Cityscapes, VGGT-World has the best overall generation quality among the compared methods. The qualitative comparison contrasts its geometric-space dynamics with artifacts in video-generation and latent-space baselines.

C.1 Qualitative Results on Cityscapes

  • VGGT-World achieves the best overall generation quality among the compared methods.The comparison includes Cosmos, DINO-Foresight, and an upper bound derived from VGGT.Source: §C.1 · p. 24
  • Cosmos suffers severe artifacts in depth prediction, while DINO-Foresight retains geometric inaccuracies, particularly in distant regions.The passage attributes Cosmos's artifacts to pixel-driven generation prioritizing perceptual plausibility over geometric consistency.Source: §C.1 · p. 24
  • VGGT-World explicitly models dynamics in geometric space, in contrast to the video-generation and latent-space baselines.Source: §C.1 · p. 24

C.2 Qualitative Results on TartanAir

On TartanAir, VGGT-World's autoregressive forecasts maintain more reasonable point-cloud structure and better individual-frame alignment than Gen3R. The qualitative comparison covers full sequences and a third-frame single-frame view.

C.2 Qualitative Results on TartanAir

  • Without pose conditioning, Gen3R deviates from the original point cloud, leading to global-alignment failures and incomplete structures, especially around windows and walls.The comparison covers the first seven frames.Source: §C.2 · p. 25, Figure 9 · p. 25
  • VGGT-World follows the trajectory implied by conditioning frames, growing the point cloud along the correct geometric direction and maintaining more reasonable structure.Source: §C.2 · p. 25
  • In the third-frame single-frame comparison, VGGT-World preserves better individual-frame alignment, while Gen3R's sequence-wide alignment errors can accumulate into drift.Gray regions denote target-frame ground-truth geometry; RGB-colored regions are predictions.Source: Figure 10 · p. 25, §C.2 · p. 26

C.3 Quantitative Results of Camera Parameter Estimation

Camera prediction errors are analyzed on KITTI and TartanAir using ATE, RTE, and RRE. The reported results indicate reasonable camera consistency and preservation of camera evolution without explicit pose supervision.

C.3 Quantitative Results of Camera Parameter Estimation

  • VGGT-World achieves reasonable camera consistency on KITTI and TartanAir across ATE, RTE, and RRE.The quantitative camera-parameter study is reported in Table 5.Source: §C.3 · p. 26, Table 5 · p. 26
  • The learned latent dynamics preserve camera evolution without explicit pose supervision.Source: §C.3 · p. 26
  • The findings support the framework's ability to capture temporally coherent structure and motion cues in a unified latent space.Source: §C.3 · p. 26

D Limitations and Future Work

The authors identify unestablished scalability to large, heterogeneous datasets and limited precise control as current constraints. They propose larger and more diverse evaluations and additional control signals as future directions.

D Limitations and Future Work

  • Scalability to large, highly heterogeneous datasets and long-term generalization remain to be fully established.The authors call for further validation across larger and more diverse datasets and complex real-world environments.Source: D Limitations and Future Work · p. 26
  • Avoiding external supervision such as ground-truth depth or camera poses limits precise control over generation.Source: D Limitations and Future Work · p. 26
  • Future work proposes adding pose or action signals to support more flexible, directionally controllable generation.The authors identify interactive prediction and planning, including autonomous driving and embodied AI, as downstream applications.Source: D Limitations and Future Work · p. 26

Related papers

Loading 2603.12655v1…