PaperlayerView exact revision on arXiv

Source-linked AI summary

VGGT: Visual Geometry Grounded Transformer

Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, David Novotny

arXiv:2503.11651v1cs.CV

TL;DR

3D reconstruction often relies on task-specific models and costly visual-geometry optimization. VGGT directly predicts multiple scene attributes in one forward pass and achieves state-of-the-art results across several 3D tasks.

  • Problem

    Existing 3D reconstruction methods remain reliant on visual-geometry optimization, while neural networks often specialize in individual tasks.

  • Method

    VGGT is a feed-forward transformer that jointly predicts camera parameters, depth maps, point maps, and 3D point tracks from one to hundreds of views.

  • Results

    VGGT achieves state-of-the-art performance across camera estimation, multiview depth, dense point-cloud reconstruction, and 3D point tracking.

  • Takeaways & Limitations

    Its simplicity and efficiency make VGGT suitable for real-time 3D applications without task-specific post-processing optimization.

  • Takeaways & Limitations

    VGGT does not support fisheye or panoramic images, degrades under extreme rotations, and fails with substantial non-rigid deformation.

Abstract

from arXiv · show

We present VGGT, a feed-forward neural network that directly infers all key 3D attributes of a scene, including camera parameters, point maps, depth maps, and 3D point tracks, from one, a few, or hundreds of its views. This approach is a step forward in 3D computer vision, where models have typically been constrained to and specialized for single tasks. It is also simple and efficient, reconstructing images in under one second, and still outperforming alternatives that require post-processing with visual geometry optimization techniques. The network achieves state-of-the-art results in multiple 3D tasks, including camera parameter estimation, multi-view depth estimation, dense point cloud reconstruction, and 3D point tracking. We also show that using pretrained VGGT as a feature backbone significantly enhances downstream tasks, such as non-rigid point tracking and feed-forward novel view synthesis. Code and models are publicly available at https://github.com/facebookresearch/vggt.

1. Introduction

VGGT addresses the challenge of direct 3D scene reconstruction by using a feed-forward transformer to predict comprehensive 3D attributes from one to hundreds of views without relying substantially on geometry post-processing. It achieves state-of-the-art performance across 3D tasks while sharing one backbone across interrelated predictions.

  • Motivation: VGGT advances beyond DUSt3R and MASt3R [62], which process only two images at once and require post-processing to reconstruct larger image sets.The broader goal is to remove the need for iterative visual-geometry optimization such as Bundle Adjustment (BA).
  • Contribution: VGGT predicts camera parameters, depth maps, point maps, and 3D point tracks from one, a few, or hundreds of views in a single forward pass.The network performs reconstruction in seconds and produces predictions directly usable for downstream 3D tasks.
  • Method: VGGT is built from a standard large transformer trained on many publicly available datasets with 3D annotations, using alternating frame-wise and global attention rather than specialized 3D inductive biases.Code and models are publicly available to support further research.
  • Contribution: VGGT uses one shared backbone to predict interrelated 3D quantities together, which enhances overall accuracy despite potential redundancies.This contrasts with models such as DepthAnything, MoGe, and LRM, which focus on individual 3D tasks.
  • Results: VGGT achieves state-of-the-art results across 3D tasks, often substantially improving quality even over methods specialized for subsets of those tasks.Its predictions are highly competitive with, and usually better than, methods that use slow post-processing optimization.

2. Related Work

This section situates VGGT within established lines of 3D vision research: Structure from Motion, Multi-view Stereo, and Tracking-Any-Point. It contrasts their traditional or task-specialized formulations with VGGT’s demonstrated use as a feature backbone for point tracking.

  • Structure from Motion: Structure from Motion estimates camera parameters and sparse point clouds from images, traditionally using image matching, triangulation, and bundle adjustment.COLMAP is identified as the most popular framework based on the traditional pipeline.
  • Multi-view Stereo: Multi-view Stereo densely reconstructs scene geometry from overlapping images, typically assuming camera parameters estimated with SfM, using traditional, optimization-based, or learning-based methods [37].DUSt3R and MASt3R [62] are introduced among recent learning-based approaches, though the supplied passage is truncated before describing them further.
  • Tracking-Any-Point: Tracking-Any-Point predicts 2D correspondences for queried points across video frames, including dynamic motion, with benchmarks from TAP-Vid [23] and later methods such as TAPIR [24] and CoTracker [55].The task originated with Particle Video and was revived by PIPs.
  • Tracking-Any-Point: VGGT’s features achieve state-of-the-art tracking performance when coupled with existing point trackers, unlike the specialized trackers described above [13].The related methods include LocoTrack [13], which extends pointwise features to nearby regions.

3. Method

VGGT is a large transformer that maps a sequence of images of one 3D scene to per-frame camera parameters, depth maps, point maps, and dense tracking features. Its alternating frame-wise and global attention architecture uses the first image as a reference frame, while separate heads produce dense geometric predictions and jointly trained tracking outputs.

  • Problem formulation: VGGT maps each image to camera parameters, depth, a viewpoint-invariant point map, and dense features used to recover 2D tracks across all images.Tracking is computed by a separate module from the query point and transformer-produced dense features; the transformer and tracking module are trained jointly end-to-end.
  • Coordinate frame: The first input image defines the reference coordinate frame, with its extrinsics fixed to identity; predictions are otherwise permutation equivariant to image order.Special learnable camera and register tokens distinguish the first frame from the remaining frames and express 3D predictions in its coordinate system.
  • Training setup: Explicitly predicting related cameras, depths, and point maps during training substantially improves performance, while combining independently estimated depths and cameras improves inference-time 3D points.VGGT normalizes training scenes to select a canonical coordinate and scale variant for reconstruction.
  • Architecture: Alternating-Attention alternates frame-wise and global self-attention, balancing within-image activation normalization with information integration across images, using 24 layers by default.The architecture processes DINO image tokens and uses no cross-attention layers.
  • Prediction heads: Camera predictions use a dedicated attention-and-linear head, while DPT-based dense heads produce depth maps, point maps, tracking features, and aleatoric depth uncertainty.The camera representation concatenates a rotation quaternion, translation vector, and field of view, assuming the principal point lies at the image center.

4. Experiments

VGGT achieves strong feed-forward performance across camera pose, multiview depth, point-map, tracking, and downstream tasks, often matching or surpassing methods that rely on optimization or known cameras. Its performance is supported by native multiview reasoning, task decomposition, multitask supervision, and reusable pretrained features.

  • Point Tracking: VGGT’s tracking head surpasses Roma on ScanNet-1500 two-view matching despite not being specialized for the two-view setting.Two-view matching is evaluated as a rigid point-tracking benchmark under the standard ScanNet protocol.
  • Camera Pose Estimation: VGGT outperforms competing camera-pose methods across all metrics on RealEstate10K and CO3Dv2 while requiring only 0.2 seconds per input set in feed-forward inference.The comparison includes methods using Global Alignment or Bundle Adjustment, which typically require more than 10 seconds; the learnable methods were trained on CO3Dv2 rather than RealEstate10K.
  • Dense MVS Estimation: VGGT reduces DTU Overall error from 1.741 to 0.382 versus DUSt3R and approaches methods given ground-truth cameras at test time.The authors attribute the gain to multiimage training that enables native multiview triangulation rather than ad hoc averaging of pairwise camera triangulations.
  • Point Map Estimation: Constructing point clouds from VGGT’s predicted depth and camera heads is more accurate than directly using its point-map head.The authors attribute this to decomposing point-map estimation into simpler depth and camera prediction subproblems, despite joint supervision of all outputs.
  • Qualitative Results: VGGT generalizes well to challenging out-of-domain scenes, including oil paintings, non-overlapping frames, and repeating or homogeneous textures.Qualitative comparisons with DUSt3R are presented for in-the-wild scenes.
  • Downstream Applications: Pretrained VGGT features transfer to downstream tasks, yielding competitive novel-view synthesis with less training data than LVSM and improving TAP-Vid RGB-S tracking from 78.9 to 84.0.The feature extractor is reused for feed-forward novel-view synthesis and dynamic point tracking; the latter result demonstrates generalization to rapid-motion videos despite the model not being explicitly designed for that setting.

5. Discussions

The discussion highlights VGGT’s broad flexibility and single-view capability, while noting limitations in image formats, extreme rotations, substantial non-rigid deformation, and computationally expensive differentiable bundle adjustment. It also describes practical runtime–memory trade-offs and design choices for patchifying, prediction normalization, and scalable inference.

  • Limitations: The model does not support fisheye or panoramic images, degrades under extreme rotations, and fails with substantial non-rigid deformation, although fine-tuning on targeted datasets can address these limitations with minimal architectural changes.This adaptability contrasts with existing approaches that typically require extensive test-time re-engineering for specialized scenarios.
  • Runtime and Memory: The camera head adds approximately 5% runtime and 2% GPU memory relative to the feature backbone, while a DPT head averages 0.03 seconds and 0.2 GB per frame.Measurements use a single NVIDIA H100 GPU with flash attention v3 at 336 × 518 resolution; Table 9 reports backbone runtime and peak memory across input-frame counts.
  • Runtime and Memory: Multiple frames can be processed in one forward pass when memory permits, whereas resource-constrained users can predict frame by frame because inter-frame relationships are confined to the backbone and DPT heads predict independently.Tensor parallelism, as used by Fast3R [141], is identified as a directly applicable way to accelerate inference across multiple GPUs.
  • Patchifying: DINOv2 patchifying outperforms a 14 × 14 convolutional tokenizer, stabilizes early training, and is less sensitive to learning-rate or momentum changes, making it the default choice.These findings follow the patchifying alternatives explored in Sec. 3.2.
  • Differentiable BA: Differentiable bundle adjustment, explored as in VGGSfM, showed promise but made PyTorch training with Theseus roughly 4 times slower, so it was excluded despite potential value as supervision without explicit 3D annotations.The authors leave customized acceleration for future work and identify large-scale unsupervised training as a promising application.
  • Single-view Reconstruction: VGGT supports single-image reconstruction natively through frame-wise attention and achieves surprisingly good results despite not being explicitly trained for that setting.Unlike DUSt3R and MASt3R, VGGT does not need to duplicate an image to form a pair.

6. Conclusions

VGGT is a feed-forward neural network that directly estimates key 3D scene properties from hundreds of input views. It achieves state-of-the-art results across several 3D vision tasks while using a neural-first approach rather than traditional optimization and post-processing.

  • Conclusions: VGGT directly estimates key 3D scene properties from hundreds of input views using a feed-forward neural network.The estimated properties include camera parameters, multiview depth, dense point clouds, and 3D point tracks.
  • Conclusions: VGGT achieves state-of-the-art results in camera parameter estimation, multiview depth estimation, dense point cloud reconstruction, and 3D point tracking.
  • Conclusions: The approach is neural-first and departs from traditional visual geometry methods that rely on optimization and post-processing.

Appendix

The appendix supplies formal definitions, implementation details, additional experiments, qualitative single-view reconstruction examples, and an expanded related-work review.

  • Appendix A: Appendix A formally defines the paper’s key terms.
  • Appendix B: Appendix B provides comprehensive implementation details, including the architecture and training hyperparameters.
  • Appendix C: Appendix C presents additional experiments and discussions.
  • Appendix D: Appendix D provides qualitative examples of single-view reconstruction.
  • Appendix E: Appendix E expands the review of related work.

A. Formal Definitions

The formalization defines camera geometry relative to the first camera’s world frame and specifies how scene surfaces yield per-pixel depths and 3D points.

  • Camera Geometry: Camera extrinsics use the first camera’s coordinate system as the world reference frame, with γ applying rigid transformations and π applying perspective projection.γ maps a world-frame 3D point to camera coordinates, while π maps the transformed point to a 2D image location.
  • Scene Representation: The scene is modeled as regular surfaces S_i ⊂ R3 that may vary with the i-th input image over time.This image-dependent formulation accounts for scene changes across time.
  • Depth and Point Maps: For each pixel, depth is the minimum depth among scene points projecting there, and the point map selects the corresponding minimizing 3D point.D_i(y) is defined by the minimum projected depth, while P_i(y) is the camera-frame point achieving that minimum.

B. Implementation Details

VGGT uses a 24-block attention architecture with 1024-dimensional, 16-head attention layers, flash attention, and QKNorm [4]. Training samples scenes and 2–24 frames per scene while maintaining 48 frames per batch, with standardized image geometry.

  • Architecture: VGGT comprises 24 attention blocks, each combining frame-wise and global self-attention layers.
  • Architecture: Each attention layer uses feature dimension 1024 and 16 heads, following the ViT-L model in DINOv2, with flash attention and QKNorm [4].
  • Training: Training randomly selects a dataset and uniformly samples a scene, then chooses 2–24 frames while keeping 48 total frames per batch.
  • Training: RGB frames, depth maps, and point maps are isotropically resized to a 518-pixel longer dimension and cropped around the principal point.

C. Additional Experiments

On the IMC phototourism camera-pose benchmark, VGGT achieved state-of-the-art performance while offering substantially faster feed-forward inference than competing methods. Comparisons included classical incremental SfM, VGGSFmv2, DUSt3R, and MASt3R [62].

  • Camera Pose Estimation on IMC: The evaluation compared direct VGGT predictions and VGGT + BA refinements against incremental SfM and recently proposed deep methods.VGGT outputs camera poses directly, whereas VGGT + BA adds a Bundle Adjustment stage.
  • Camera Pose Estimation on IMC: DUSt3R and MASt3R [62] were trained on much of MegaDepth, whose scenes overlap with IMC despite using different images.
  • Camera Pose Estimation on IMC: VGGT achieved state-of-the-art camera-pose performance on challenging IMC phototourism data, outperforming VGGSFmv2 [125].
  • Camera Pose Estimation on IMC: AUC@10 was 71.26 for VGGT versus 76.82 for VGGSFmv2, while runtime was 0.2 versus 10 seconds per scene.
  • Camera Pose Estimation on IMC: VGGT significantly outperformed MASt3R [62] and DUSt3R across all accuracy thresholds while being much faster.DUSt3R and MASt3R require costly global alignment because their feed-forward predictions process only frame pairs.

D. Qualitative Examples

The paper presents qualitative examples of single-view reconstruction in Fig. 7.

  • D. Qualitative Examples: Qualitative examples demonstrate single-view reconstruction results in Fig. 7.

E. Related Work

This section situates the work within Vision Transformers and camera pose estimation. It highlights the adoption and evolution of Vision Transformers alongside SfM and regression-based approaches for multi-view camera poses.

  • Vision Transformers: Vision Transformers became dominant in computer vision after ViT, supported by simplicity, capacity, flexibility, and long-range dependency modeling [4] [12].The Transformer originated in language processing [6] [22], while subsequent work improved training, self-supervised features, and attention dynamics [10] [48].
  • Camera Pose Estimation: Structure from Motion remains the dominant approach to multi-view camera pose estimation, encompassing both incremental and global methods [17].Recent methods formulate pose estimation as regression and report promising results in sparse-view settings [65] [109] [112].
Loading 2503.11651v1…