Which Pretraining Paradigm Better Serves Spatial Intelligence? An Empirical Comparison of Vision-Language and Video Generation Models

Haozhan Shen; Tiancheng Zhao; Kangjia Zhao; Jianwei Yin

Source-linked AI summary

Which Pretraining Paradigm Better Serves Spatial Intelligence? An Empirical Comparison of Vision-Language and Video Generation Models

Haozhan Shen, Tiancheng Zhao, Kangjia Zhao, Jianwei Yin

arXiv:2605.28132v1May 27, 2026cs.CV

TL;DR

The paper asks which pretraining scheme better encodes spatial information before downstream fine-tuning. It compares frozen VLM and VGM features with lightweight probes across semantics, instances, and geometry, finding complementary strengths and effective simple fusion.

Problem
It remains unclear whether VLMs or VGMs provide better frozen representations for spatial intelligence, which requires both semantic objects and geometric structure.
Method
The study freezes representative VLMs and VGMs and trains identical lightweight probes for semantic tagging, instance grouping, and 3D geometry prediction.
Results
VLMs are stronger on semantic tagging and instance grouping, whereas VGMs provide more accessible dense geometry and camera-motion signals; simple fusion combines these strengths.
Takeaways & Limitations
Spatial-intelligence backbones may benefit from integrating language-aligned object semantics with video-generation geometric representations.
Takeaways & Limitations
The evaluation covers only selected axes and mainly ScanNet and DL3DV, so conclusions may not fully transfer to outdoor, dynamic, or robot-collected environments.

Abstract

from arXiv · show

Spatial intelligence requires visual representations that capture both semantic objects and geometric structure in the physical world. To support this, two major pre-training schemes are now widely used as foundation backbones: Vision-Language Models (VLMs), which use language supervision to align visual observations with semantic concepts, and Video Generation Models (VGMs), which learn from temporally evolving visual worlds. However, it still remains unclear which pre-training scheme provides a better representation substrate for spatial intelligence. In this paper, we present the first systematic frozen-feature probing study of VLMs and VGMs across three representative axes of spatial intelligence: semantic tagging, instance grouping, and 3D geometry prediction. Using the lightweight probe, our framework enables a controlled comparison of what information is already encoded in frozen representations from two model families. Experimental results reveal a clear complementarity: VLMs are stronger at semantic tagging and instance grouping, while VGMs provide more accessible signals for dense geometry and camera motion. Moreover, a naive fusion of the two already yields a representation that excels at both geometry and semantics, suggesting a promising direction for building stronger spatial-intelligence backbones by effectively integrating features from both model families. Our code is available at \href{https://github.com/om-ai-lab/Probing-VLM-VGM}{https://github.com/om-ai-lab/Probing-VLM-VGM}.

1 Introduction

The paper asks whether VLMs or VGMs provide better frozen representations for spatial intelligence, which requires both semantic objects and geometric structure. A controlled probing study finds complementary strengths: VLMs favor semantics and instances, VGMs favor geometry, and fusion combines them.

Spatial intelligence requires representations that capture semantic objects and geometric structure for understanding and acting in physical environments.
VLMs align visual observations with semantic concepts through language supervision, whereas VGMs learn from temporally evolving visual worlds.
Existing embodied-policy comparisons cannot isolate pretraining effects because action decoders, robot data, post-training, inference, and benchmarks are entangled.
The study freezes foundation models and trains identical lightweight probes across semantic tagging, instance grouping, and 3D geometry prediction.
VLMs outperform VGMs on semantic tagging and instance grouping, while VGMs recover dense geometry and camera motion more accurately.
Simple feature-level fusion preserves VLM semantic advantages while recovering much of VGM geometric strength.

2 Related Work

Related work establishes VLMs and VGMs as visual backbones and motivates comparing their frozen representations directly. This paper places semantic, instance, and geometry probing in one controlled framework.

VLMs are used in embodied agents and spatial reasoning because language supervision aligns representations with object categories, attributes, and instructions.
VGMs learn from temporally evolving observations and may acquire priors about dynamics and geometric consistency.
Prior frozen-feature studies show that lightweight readouts can reveal information accessible in visual representations, including 3D geometry from video features.
The paper's novelty is comparing VLMs and VGMs across semantic tagging, instance grouping, and 3D geometry prediction within one probing framework.

3 Probing Framework

The framework freezes each VLM or VGM, extracts temporally aligned video features, and reads them with a shared probing backbone. Separate task heads predict semantics, instance identities, dense geometry, and camera motion.

3 Probing Framework: Frozen representation probing defines spatial intelligence by what semantic categories, object instances, and 3D scene geometry can be recovered without foundation-model fine-tuning.
3.1 Frozen Feature Extraction: The framework extracts temporally aligned features from frozen models, samples frame features, and applies task-specific heads after a shared probing backbone.
3.1 Frozen Feature Extraction: VGM features come from internal denoising-transformer activations, while VLM features come from visual-token hidden states reshaped into matching frame-level grids.
3.2 Unified Probing Backbone: The three task probes use separate heads and are trained independently despite sharing the backbone architecture.
3.2 Unified Probing Backbone: The shared backbone projects model-specific channels to a common width, prepends camera tokens, and alternates within-frame and across-frame attention.
3.3 Task-Specific Readout Heads: Semantic tagging predicts video-level object-category labels, instance grouping learns normalized embeddings with multi-view pull-push supervision, and geometry prediction estimates points, depth, confidence, and camera pose.

4 Experiments

Under a unified frozen-feature probing protocol, VLMs outperform VGMs on semantic tagging and instance grouping, while VGMs lead on all evaluated 3D geometry metrics. Simple feature fusion combines these complementary strengths, and probe-depth results indicate that rankings largely reflect information already encoded in frozen features.

Experimental Setup: The evaluation uses ScanNet20 semantic tagging, multi-view instance grouping, and 3D geometry metrics including P-map Err., depth AbsRel, and camera AUC@30.Models are evaluated with the same 76-frame context, while geometry samples 4 frames and semantic tagging and instance grouping sample 8.
Semantic Tagging: VLMs improve family-average semantic tagging mAP from 69.89 to 92.08 and APmid from 58.63 to 87.28 compared with VGMs.The average Mid Ratio is also higher for VLMs, 0.948 versus 0.838, indicating stronger performance on less frequent categories.
Instance Grouping: VLMs improve family-average instance grouping from 13.24 to 22.66 in T-mIoU and from 4.35 to 11.23 in T-SR.The results indicate that grouping across views benefits from object-centric semantics in addition to spatial consistency.
3D Geometry: VGMs outperform VLMs on all three geometry metrics: P-map Err. 0.152 vs. 0.223, depth AbsRel 0.072 vs. 0.113, and camera AUC@30 0.527 vs. 0.330.Qualitative results show sharper VGM depth structure and more faithful shelf geometry, whereas VLM predictions capture only coarse layout.
Feature Fusion: Feature-level fusion improves semantic tagging and instance grouping over WAN2.1-T2V-14B while lowering depth AbsRel and raising camera AUC@30 relative to Qwen3-VL-8B.The fused representation slightly exceeds WAN2.1-T2V-14B on both reported geometry metrics and preserves strong VLM semantic performance.
Robustness and Qualitative Analysis: Probe-depth ablations preserve the relative ranking of representative models across instance grouping and 3D geometry, indicating that probes mainly read out frozen representations.Qualitative examples likewise show VLM advantages in category recognition and object separation, and VGM advantages in depth structure.

5 Conclusion

The study finds complementary strengths rather than a single superior pretraining family: VLMs expose semantic and object-centric information, while VGMs expose geometry. Simple feature fusion combines these advantages.

VLMs provide stronger semantic and object-centric representations, whereas VGMs make dense geometry and camera motion more recoverable.
Simple feature-level fusion combines VLM semantic performance with strong VGM geometry.The fusion experiment is presented as evidence that the two representation families’ strengths are complementary.

6 Limitations

The evaluation covers important but incomplete aspects of spatial intelligence, uses mainly indoor and reconstructed-video data, and depends on several design choices. The simple fusion study is only a proof of concept.

The three evaluated axes do not exhaust spatial reasoning, leaving physical dynamics, affordances, active exploration, and long-horizon embodied reasoning for future work.
Conclusions may not fully transfer beyond ScanNet and DL3DV to outdoor, highly dynamic, or robot-collected environments.
Results depend on selected feature layers, frame sampling, spatial resolution, and VGM denoising timesteps, despite controlled protocols and probe-depth ablations.
Feature-level fusion is intentionally simple and remains a proof of concept rather than a final fusion architecture.

A VGM Temporal Compression

WAN compresses an 81-frame video into latent temporal positions using a causal spatiotemporal VAE. The probing setup retains positions aligned with its 76-frame context.

WAN encodes RGB video with a causal spatiotemporal VAE before the denoising transformer.
The VAE treats the first frame separately and processes remaining frames in temporal chunks.
Two 3D downsampling blocks implement temporal compression using causal convolutions with kernel size (3, 1, 1) and stride (2, 1, 1).
An 81-frame window produces one latent position for the first frame and 20 positions for the remaining 80 frames.
The probing context uses the first 76 frames, so the last WAN position is discarded and a 20-position VGM feature bank is retained.

B Detailed Training Objectives

The probes use frozen foundation-model features and task-specific objectives for semantic tagging, instance grouping, and 3D geometry prediction. Geometry supervision combines confidence-weighted map losses with iterative camera-pose refinement.

Shared probing setup: All foundation models remain frozen while a shared probing backbone and task heads are optimized for the three spatial-intelligence axes.The backbone produces stage tokens; tagging and grouping use final-stage patch tokens, while geometry uses selected dense stages and the final-stage camera token.
Semantic tagging: Semantic tagging constructs binary class targets from visible pixels in sampled frames rather than from the full video clip.A class is labeled using pixel- and frame-visibility thresholds, with τpix = 200 and τfrm = 1.
Semantic tagging: The semantic head predicts one class logit per category and uses asymmetric multi-label loss with γneg = 4 and γpos = 0.The negative-probability shift is m = 0, so no probability clipping is applied; easy negatives are down-weighted while rare positives are not focal-suppressed.
Instance grouping: Instance grouping maps final patch tokens to normalized per-pixel embeddings, trains contrastive pull-push relationships, and clusters embeddings with HDBSCAN at evaluation.Training samples 2048 valid pixels per scene, using margin µ = 1.0 and equal pull and push weights; evaluation compares clusters with ground-truth instance masks.
3D geometry prediction: The geometry probe predicts point maps, depth maps, and camera-pose encodings using VGGT-derived supervision aligned to the first sampled frame.Point-map and depth losses are confidence-weighted; point maps receive additional valid-pixel distance normalization, while camera predictions use iterative Huber refinement with later-step weighting.
3D geometry prediction: The final geometry objective combines point-map, depth, and camera losses with λP = λD = λC = 1.The multi-scale gradient regularizer is logged but excluded from the optimized objective because its main-experiment weights are zero.

C Implementation Details

Implementation uses task-specific training schedules, optimization settings, and fixed intermediate feature layers to compare models under controlled probing conditions.

Optimization: Probes use AdamW with linear-warmup cosine learning rates, batch sizes 8 for ScanNet tasks and 10 for DL3DV geometry.Training lasts 10, 40, and 60 epochs for semantic tagging, instance grouping, and geometry, respectively.
Geometry implementation: The geometry probe uses learning rate 10^-4, weight decay 0.05, ten warmup epochs, a depth-4 width-1024 backbone, and DPT heads.Point-map, depth, and camera losses are weighted equally.
Feature selection: Each model is evaluated at a fixed intermediate layer rather than selecting the best layer separately for each metric.VGM features use specified denoising timesteps, while the main runs use fixed layers for WAN, CogVideoX, InternVL3, InternVL3.5, Qwen2.5-VL, and Qwen3-VL.

D Probe-Depth Ablation Details

The probe-depth ablation shows that absolute scores vary with probing depth, but the relative ordering of representative models remains stable.

Probe-depth ablation: Relative model ordering remains stable across probe depths despite variation in absolute scores.Table 4 provides the full numerical results for this ablation.

E Additional Qualitative Example

Additional qualitative examples compare VLM and VGM predictions for instance grouping, depth, and point-cloud reconstruction. The examples show finer object regions in Qwen3-VL grouping and sharper geometric structure in WAN and CogVideoX outputs.

Instance grouping: Qwen3-VL preserves several meaningful object-level regions across views, whereas OpenSora and CogVideoX tend to merge large scene areas.The Qwen3-VL regions include tables, chairs, shelves, and foreground objects, although they remain coarser than ground truth.
Quantitative reporting: Table 4 reports instance grouping with T-mIoU percentages and geometry with P-map Err. on its original scale, where lower P-map Err. ranks higher.WAN, Cog, Intern, and Qwen identify the representative model families used in the ablation table.
Depth prediction: WAN and CogVideoX follow greenhouse depth more closely on large table planes and retain sharper discontinuities than InternVL3 and Qwen3-VL.The VLM predictions recover coarse near-far layout but smooth or blur local depth changes near table boundaries and overhead beams.
Point-cloud reconstruction: WAN preserves a roomlike bookstore point-cloud structure with visible shelf planes and a clearer aisle layout, while InternVL3 and Qwen3-VL are more compact and fragmented.CogVideoX is noisier but retains much of the elongated shelf structure.