Source-linked AI summary
How Much 3D Do Video Foundation Models Encode?
Zixuan Huang, Xiang Li, Zhaoyang Lv, James M. Rehg
TL;DR
The paper asks whether large-scale video training can produce useful 3D understanding despite limited high-quality 3D data and uncertainty about what video models encode. It probes frozen VidFM features with shallow readouts for 3D properties and finds strong 3D awareness in frontier video generators, with temporal reasoning playing a key role.
Problem
High-quality 3D data at scale remains a bottleneck, and it is unclear whether video data alone can induce strong 3D awareness in general-purpose models.
Source: §1 · pp. 1–2Method
The paper introduces a model-agnostic framework that trains shallow readouts on frozen VidFM features to predict 3D points, camera poses, and depth.
Source: §1 · p. 2Results
Takeaways & Limitations
VidFM features may be better suited for 3D reconstruction under limited 3D data, and their evaluation provides observations for developing scalable 3D models.
Source: §1 · p. 2The study uses publicly released checkpoints, so it cannot strictly attribute awareness differences to factors such as data or training strategy.
Source: §4.5 · p. 8
Abstract
from arXiv · showhide
Videos are continuous 2D projections of 3D worlds. After training on large video data, will global 3D understanding naturally emerge? We study this by quantifying the 3D understanding of existing Video Foundation Models (VidFMs) pretrained on vast video data. We propose the first model-agnostic framework that measures the 3D awareness of various VidFMs by estimating multiple 3D properties from their features via shallow read-outs. Our study presents meaningful findings regarding the 3D awareness of VidFMs on multiple axes. In particular, we show that state-of-the-art video generation models exhibit a strong understanding of 3D objects and scenes, despite not being trained on any 3D data. Such understanding can even surpass that of large expert models specifically trained for 3D tasks. Our findings, together with the 3D benchmarking of major VidFMs, provide valuable observations for building scalable 3D models.
1. Introduction
The paper asks whether large-scale video training can yield useful 3D awareness despite the scarcity of 3D data and confounds in prior work. It introduces a model-agnostic probe and reports strong 3D understanding in frontier video generators across several axes.
1. Introduction
- Limited high-quality 3D data constrains data-driven 3D foundation models, while video is more readily available at scale but its 3D value remains uncertain.Prior work involving 3D fine-tuning and task-specific engineering leaves unclear whether video alone produces strong 3D awareness in general-purpose models.Source: §1 · pp. 1–2
- The paper proposes shallow feedforward readouts of pretrained video features to estimate 3D attributes without fine-tuning the base model.The probe predicts points, camera poses, and depth maps from frozen spatial-temporal features.Source: §1 · p. 2
- Frontier video generators show strong 3D understanding; temporal reasoning is critical, while 3D fine-tuning, model scaling, and feature localization have mixed or consistent effects.3D fine-tuning improves in-domain awareness but may hurt generalization; scaling effects differ by model, and mid-layer features at early-but-not-first timesteps perform best.Source: §1 · p. 2
- The study presents a systematic, model-agnostic benchmark for evaluating VidFMs’ emerging 3D properties across multiple axes.Its findings are intended to inform the development of VidFMs.Source: §1 · p. 2
2. Related Works
Prior approaches to 3D from video include classical geometry, data-driven prediction, and video-prior methods, but the 3D information already encoded in general-purpose video models remains insufficiently quantified.
2. Related Works
- Video foundation models achieve strong pixel synthesis, yet their internal representations are not well understood.This motivates examining how much 3D understanding those representations contain.Source: §2 · p. 2
- Classical Structure from Motion and Multi-view Stereo rely on feature matching, while data-driven 3D prediction models still face scaling and generalization challenges in dynamic or cluttered scenes.The latter challenges are linked to reliance on annotated data.Source: §2 · pp. 2–3
- Video-generation priors have been used for 3D control, attribute prediction, memory, optimization, and feedforward reconstruction, but base-model 3D information remains unclear quantitatively.Fine-tuned models can retain 3D-inconsistency artifacts, especially outside their fine-tuning data.Source: §2 · p. 3
- Existing 3D-awareness evaluations include semantic or coarse-estimation tasks and dense probes of image models, often using depth or multi-view consistency.The paper argues that indirect depth and multi-view consistency probes are not necessarily the best measures across model families, and instead probes video models with 3D attributes.Source: §2 · p. 3
3. Approach
The approach evaluates 3D awareness by training a lightweight probe on frozen VidFM features to recover 3D structure and camera motion from sampled video frames.
3. Approach
- 3D awareness is operationalized as recovering 3D structure and ego-motion from frozen video features, with lower reconstruction error indicating stronger awareness under fixed probe capacity and training data.The probe predicts dense 3D point maps, depth maps, and camera poses relative to the first frame.Source: §3 · pp. 3–4
3.1. Feature Extraction
- For diffusion generators, features come from hidden activations after one denoising step at a chosen timestep and layer; other models use last-layer spatial features from a standard forward pass.The layer and timestep are fixed across experiments; image-to-video models are conditioned on the first frame, and an empty text embedding is used.Source: §3.1 · p. 4
- Models with short context windows process longer videos in chunks that share the first-frame reference, with an index map matching raw frames to chunk features.The map is used to gather features for the probe’s sampled input frames.Source: §3.1 · p. 4
3.2. 3D Awareness Probe
- The probe uses four frames, four alternating-attention blocks, and three heads to predict point maps, depth maps, and camera poses.Frame attention mixes tokens within each frame and global attention mixes tokens across frames; point maps and poses use the first frame’s coordinate reference.Source: §3.2 · p. 4
- Training combines point-map, depth, and camera losses, using confidence-weighted L2 losses for maps and a Huber loss for poses.Ground-truth scenes are normalized before map-loss calculation to remove scale ambiguity; all three loss weights default to 1.Source: §3.2 · p. 4
4. Experiments
Video foundation model features show strong 3D awareness across datasets, with WAN2.1-14B surpassing a 3D expert on DL3DV; temporal feature choices and 3D fine-tuning affect results. Under limited 3D data, frozen WAN features also improve feedforward reconstruction, though the study cannot isolate some factors behind model differences.
4.1. Evaluation
- Evaluation uses video-level 9:1 splits on filtered 11k CO3Dv2 videos and the first 6k DL3DV splits, with VGGT-generated point, depth, and pose targets.Point-map and depth errors are reported after scene normalization; pose is evaluated using joint rotation-and-translation accuracy and AUC.Source: §4.1 · p. 4
4.2. 3D Awareness Benchmark
- WAN2.1-14B ranks second only to Fast3R across CO3Dv2 metrics but surpasses Fast3R across DL3DV metrics; Open-Sora2.0 is consistently strong.The benchmark covers video generators, self-supervised encoders, 3D experts, and per-frame image models; point-map errors are multiplied by 10 in Table 1.Source: Table 1 · p. 5, §4.2 · p. 7
- Aether improves over CogVideoX on DL3DV but slightly underperforms its base on CO3Dv2, so 3D-aware fine-tuning does not guarantee gains across datasets.The authors suggest Aether's mostly synthetic game and simulator scenes may relate to this discrepancy.Source: §4.2 · p. 7
- Qualitative reconstructions track benchmark rankings: Fast3R, WAN2.1-14B, and Open-Sora2.0 preserve fine CO3Dv2 details, while WAN2.1-14B is sharpest overall on DL3DV.On DL3DV, DINOv2 can fail catastrophically, while top video generators often retain coherent geometry.Source: Figure 3 · p. 5, Figure 4 · p. 6, §4.2 · p. 7
4.3. Ablations
- Point-map error falls from 0.0468 to 0.0360 (−23%) as WAN scales from 1.3B to 14B, but rises from 0.0576 to 0.0590 (+2%) as CogVideoX scales from 2B to 5B.The authors hypothesize that additional training data may help explain the differing scaling results.Source: §4.3 · pp. 7–8
4.4. VidFM Features for Feedforward 3D
- With frozen WAN2.1-14B features, VGGT outperforms the original DINO-feature VGGT by a large margin across metrics on CO3Dv2 and DL3DV under matched compute.Table 2 compares the variants; the authors frame the result as an advantage for feedforward reconstruction when 3D data are limited.Source: Table 2 · p. 8, §4.4 · p. 8
4.3. Ablations
- Across tested diffusion models, mid-network layers with early-but-not-first denoising steps yield the strongest 3D awareness.The study attributes this to a balance between retaining global 3D cues and limiting noise effects; Figure 5 reports point-map error across layers and timesteps.Source: Figure 5 · p. 7, §4.3 · p. 8
4.5. Limitations
- The study uses public checkpoints rather than controlled model variants, preventing strict attribution of 3D-awareness differences to factors such as data or training strategy.The authors also cannot isolate data scale or train large reconstruction models from scratch on massive datasets with VidFM features.Source: §4.5 · p. 8
5. Conclusion
The paper probes video foundation models for 3D awareness using direct 3D prediction tasks. It finds strong, generalizable 3D awareness in state-of-the-art video generators and examines factors affecting it.
5. Conclusion
- State-of-the-art video generators exhibit strong, generalizable 3D awareness, even compared with domain experts.The experiments also examine temporal reasoning, 3D fine-tuning, model scaling, and diffusion feature-extraction choices.Source: §5 · p. 8
- The study evaluates video-model features through direct 3D prediction tasks rather than 2.5D or optimization-based proxies used for image models.It also presents an evaluation protocol and benchmark for existing video foundation models.Source: §5 · p. 8
- VidFM features show promise for 3D reconstruction in limited-data settings.The paper hopes its findings will support understanding and building scalable 3D world models.Source: §5 · p. 8
Supplementary Material
The supplement tests probe-capacity robustness, data scaling for VidFM-VGGT, and the relationship between probe performance and multiview feature consistency. It reports that probe rankings remain stable across sizes and that cross-view correspondence can be a biased proxy for 3D awareness.
Supplementary Material
- The supplement finds that probe rankings and conclusions remain unchanged when comparing the original and smaller probes on DL3DV.It also reports benefits from strong video-generator features in limited-data or challenging reconstruction settings, and warns that cross-view correspondence alone can be biased when comparing model families.Source: Table 3 · p. 12, Supplementary Material · p. 12
- Strong video-generator features are especially beneficial for feedforward 3D reconstruction with limited 3D data or challenging learning scenarios.The supplement examines how performance scales with the amount of 3D training data.Source: Supplementary Material · p. 12
- Cross-view correspondence alone can be a biased proxy for true 3D awareness, especially across different model families.The supplement analyzes this relationship alongside 3D probe performance.Source: Supplementary Material · p. 12
A. Ablation on Probe Size
The ablation checks whether the study's 3D-awareness conclusions depend on probe capacity. A narrower probe preserves relative performance rankings on DL3DV.
A. Ablation on Probe Size
- Relative performance remains stable when probe width is halved from 1024 to 512 channels on DL3DV.The main experiments use shallow probes with 4 layers and 1024 channels.Source: A. Ablation on Probe Size · p. 12
- The smaller probe does not change the conclusion that state-of-the-art video-generation features, including WAN2.1-14B, show strong 3D awareness compared with other model categories.The comparison follows the main paper's experimental protocol.Source: A. Ablation on Probe Size · p. 12
- The probe-size ablation evaluates 3D awareness on DL3DV using probes of different capacities.The study compares the original probe with a smaller probe.Source: A. Ablation on Probe Size · p. 12
B. Data Scaling for VidFM VGGT
The data-scaling experiment compares VidFM-VGGT with original VGGT as 3D training data varies. VidFM-VGGT typically exceeds the full-data baseline with less than 10% of the training data, with a particularly large gap on DL3DV.
B. Data Scaling for VidFM VGGT
- Under the same training data, VidFM-VGGT significantly outperforms original VGGT in the limited-resource comparison.The data-scaling experiment extends this comparison by varying available training data.Source: B. Data Scaling for VidFM VGGT · p. 12
- With less than 10% of the 3D training data, VidFM-VGGT typically surpasses original VGGT trained with 100% of the data across all metrics.The scaling curves cover CO3Dv2 and DL3DV.Source: B. Data Scaling for VidFM VGGT · p. 12
- The VidFM-VGGT advantage is especially large on DL3DV, whose scenes are more diverse and cluttered.The passage links this result to benefits in diverse and challenging domains.Source: B. Data Scaling for VidFM VGGT · p. 12
- The authors could not extrapolate the curves to the scale of original VGGT's training set, which pools most available 3D data.They identify such extrapolation as a future direction.Source: B. Data Scaling for VidFM VGGT · p. 12
C. Analysis on Multi-view Consistency
The analysis finds that multi-view consistency does not reliably track 3D awareness across model families. Video diffusion models show a positive relationship between the two measures, while feedforward models can have stronger correspondence despite weaker 3D probing.
C. Analysis on Multi-view Consistency
- Cross-view correspondence error is the mean pixel distance between predicted and ground-truth matches, with predictions obtained by nearest-neighbor search in feature space.Ground-truth matches come from reprojection of sampled anchor-view pixels using 3D, excluding occluded pixels.Source: C. Analysis on Multi-view Consistency · pp. 12–13
- DINOv2 and V-JEPA achieve strong multi-view correspondence yet perform significantly worse in 3D probing, so correspondence may not suffice as a cross-family proxy for 3D awareness.The figure compares 3D Probe Error with Cross-view Correspondence Error; lower is better on both axes.Source: Figure 7 · p. 13, C. Analysis on Multi-view Consistency · p. 13
- Within video diffusion models, lower 3D probe error accompanies lower correspondence error; CogVideoX is worst on both, while WAN2.1-14B is best.Open-Sora2.0 and WAN2.1-1.3B are intermediate.Source: C. Analysis on Multi-view Consistency · p. 13
- Noise and denoising-specific features in diffusion models can make corresponding pixels differ across frames, suppressing raw feature consistency even when shallow probes recover 3D structure.The account attributes this mismatch to noise injection into VAE features and a single denoising step.Source: C. Analysis on Multi-view Consistency · p. 14
- The authors hypothesize that local-motion channels make same-point features vary across frames, so temporal information can aid 3D decoding while weakening nearest-neighbor consistency.They identify feature consistency alone as a potentially biased measure of 3D awareness.Source: C. Analysis on Multi-view Consistency · p. 14