PaperlayerView exact revision on arXiv

Source-linked AI summary

Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding

Xianjin Wu, Dingkang Liang, Tianrui Feng, Kui Xia, Yumeng Zhang, Xiaofan Li, Xiao Tan, Xiang Bai

arXiv:2603.19235v3cs.CVcs.RO

TL;DR

MLLMs remain weak at fine-grained spatial reasoning, while existing remedies often require explicit 3D supervision or complex geometric scaffolding. VEGA-3D reuses implicit geometric priors from a video diffusion model and adaptively fuses them with semantic features, yielding consistent improvements across scene understanding, spatial reasoning, and manipulation tasks. The approach avoids extra 3D supervision but increases inference cost.

  • Problem

    MLLMs have strong semantic capabilities but suffer from spatial blindness, and existing geometric solutions commonly rely on explicit 3D supervision or complex scaffolding.

  • Method

    VEGA-3D treats a pretrained video diffusion model as a Latent World Simulator and integrates its spatiotemporal features with semantic tokens through adaptive gated fusion.

  • Results

    VEGA-3D consistently improves 3D scene understanding, spatial reasoning, and manipulation without extra 3D supervision.

  • Takeaways & Limitations

    Video-generation priors provide a scalable source of geometric anchors for MLLMs and can transfer as video-generation models advance.

  • Takeaways & Limitations

    Using a frozen video diffusion backbone increases memory and inference cost, although feature caching alleviates the overhead in practice.

Abstract

from arXiv · show

While Multimodal Large Language Models demonstrate impressive semantic capabilities, they often suffer from spatial blindness, struggling with fine-grained geometric reasoning and physical dynamics. Existing solutions typically rely on explicit 3D modalities or complex geometric scaffolding, which are limited by data scarcity and generalization challenges. In this work, we propose a paradigm shift by leveraging the implicit spatial prior within large-scale video generation models. We posit that to synthesize temporally coherent videos, these models inherently learn robust 3D structural priors and physical laws. We introduce VEGA-3D (Video Extracted Generative Awareness), a plug-and-play framework that repurposes a pre-trained video diffusion model as a Latent World Simulator. By extracting spatiotemporal features from intermediate noise levels and integrating them with semantic representations via a token-level adaptive gated fusion mechanism, we enrich MLLMs with dense geometric cues without explicit 3D supervision. Extensive experiments across 3D scene understanding, spatial reasoning, and embodied manipulation benchmarks demonstrate that our method outperforms state-of-the-art baselines, validating that generative priors provide a scalable foundation for physical-world understanding. Code is publicly available at https://github.com/H-EmbodVis/VEGA-3D.

1 Introduction

Video generation models learn geometry-consistent spatiotemporal priors that can provide spatial guidance to MLLMs. VEGA-3D fuses these generative features with semantic representations, improving geometric understanding across multiple tasks.

  • Motivation: Video generators encode geometry-consistent structure and motion because temporally coherent synthesis requires persistent identities, depth-dependent motion, and consistent interactions.These learned priors emerge without explicit 3D supervision.
  • Spatial Priors: Multi-view consistency provides spatial guidance: generative priors sharpen scattered baseline attention into precise target-object localization.Figure 2 links stable representations across camera views with improved attention localization.
  • Method: VEGA-3D repurposes a video generation model as a Latent World Simulator and combines its spatiotemporal priors with semantic features.The framework uses token-level adaptive gated fusion to integrate heterogeneous representations.
  • Feature Analysis: Generative and semantic features are complementary, with their fusion producing consistent gains rather than replacing semantic representations.The feature analysis reports synergy between the two streams.
  • Results: Experiments across 3D scene understanding, spatial reasoning, and manipulation benchmarks show consistent gains over larger spatially enhanced models.The most informative spatial cues arise from intermediate representations and mid-denoising stages.

2 Related Work

Prior 3D-MLLM methods use explicit 3D inputs, multi-view lifting, geometric supervision, or linguistic reasoning strategies. VEGA-3D instead treats spatial reasoning as a representational problem by importing implicit geometric priors from video generators.

  • Explicit 3D Inputs: Point-cloud methods align 3D encoders with language models but depend heavily on high-quality 3D data.Examples include PointLLM, Point-Bind, and GPT4Point.
  • Multi-view Lifting: Multi-view approaches project 2D features into 3D space using positional embeddings or bird’s-eye-view rendering.Video-3D LLM and GPT4Scene exemplify this direction.
  • Geometric Supervision: Auxiliary geometric methods rely on reconstruction, depth, camera-pose, or pretrained-3D-backbone supervision, often requiring complex multi-stage pipelines.Ross3D, 3DRS, and ThinkWith3D follow this paradigm.
  • Reasoning-based Methods: Other approaches scale spatial instruction data or use mental simulation and chain-of-thought, treating spatial reasoning primarily as a linguistic or logical problem.These directions address spatial reasoning without directly supplying a physically consistent representation.
  • VEGA-3D: VEGA-3D treats spatial reasoning as a representational problem, grounding MLLM reasoning in generative video priors rather than explicit geometric supervision.The approach leverages temporal coherence and spatiotemporal regularities learned by video generators.

3 Preliminaries

The preliminaries describe standard MLLM visual-token processing and language-model training, then introduce video diffusion models trained to predict latent-space transport dynamics. Sparse vocabulary-level supervision can leave geometric errors weakly constrained, motivating richer spatial representations.

  • Multimodal Language Models: An MLLM maps visual inputs into visual embeddings through a visual encoder and projector before conditioning language generation.The visual sequence is v = f_proj(f_enc(V)).
  • Multimodal Language Models: The MLLM is trained to maximize the likelihood of response tokens conditioned on text and visual embeddings.Its cross-entropy objective sums token-level negative log probabilities.
  • Spatial Blindness: Vocabulary-space supervision treats spatial mistakes such as predicting “left” instead of “right” as generic token mismatches.Because geometric metric constraints are absent, discriminative encoders may focus on semantic presence rather than precise spatial structure.
  • Video Diffusion Models: Video diffusion models sample noise and time around a clean latent video and train a flow network with mean-squared error to predict target velocity.The formulation uses a continuous-time transport field in latent space.
  • Video Diffusion Models: Implementation uses a discrete timestep index k in {0, . . . , K} with K=1000.The timestep is normalized for the diffusion process.

4 Method

VEGA-3D combines a frozen video diffusion model’s implicit 3D priors with semantic features through adaptive token-level fusion. It extracts multi-view-consistent generative representations during denoising and uses them to support geometry-sensitive reasoning.

  • Framework overview: VEGA-3D uses a dual-branch encoder combining discriminative semantic features with dense 3D priors from a generative video diffusion model.The framework repurposes the generative model as a Latent World Simulator and integrates both streams for MLLM-based scene understanding.
  • 3D awareness analysis: Multi-view feature consistency is used as an indicator of geometric capability and is positively correlated with downstream 3D performance.The correspondence score projects features into shared voxels and averages cosine similarity across views; higher scores indicate stronger implicit alignment.
  • 3D awareness analysis: DiT-based video models achieve consistency above 96% and show superior downstream 3D understanding compared with lower-consistency UNet-based models.The analysis attributes this architectural difference to global attention in DiT models versus the local inductive bias and limited scale of UNet-based models.
  • Generative feature extraction: VEGA-3D activates a frozen video model by adding noise to encoded video latents and extracting intermediate DiT features using an empty text prompt.The method samples a timestep, constructs a noisy latent, and selects an intermediate layer balancing spatial precision with abstract spatiotemporal context.
  • Bridging feature spaces: Independent MLP projectors map generative and semantic streams into the LLM hidden dimension before token-level adaptive gated fusion.For each spatial token, a sigmoid gate weighs the two streams, producing a convex combination rather than simply averaging conflicting signals.
  • Bridging feature spaces: The gate acts as a semantic-geometric arbitrator, shifting emphasis between recognition-oriented semantic cues and spatially informative generative knowledge.This integration yields dense 3D understanding without explicit geometric supervision while preserving discriminative semantic cues.

5 Experiments

The experiments evaluate VEGA-3D across 3D scene understanding, spatial reasoning, and manipulation, finding consistent benefits from generative spatial priors. Ablations indicate that performance depends on complementary semantic–generative fusion and appropriate diffusion features.

  • 3D Scene Understanding: VEGA-3D improves Video-3D LLM across five 3D scene-understanding benchmarks, especially localization tasks.ScanRefer Acc@0.5 rises from 51.7 to 56.2, while SQA3D EM rises from 58.6 to 61.3.
  • 3D Scene Understanding: Generative priors yield notable gains in grounding and spatial question answering but slightly reduce Scan2Cap CIDEr.The authors attribute this pattern to a semantic–geometry trade-off addressed by token-wise adaptive weighting.
  • 3D Scene Understanding: Without explicit 3D annotations, VEGA-3D achieves the best overall average rank and leads on every grounding-oriented metric.Ross3D attains higher ScanQA and SQA3D scores through BEV supervision, whereas VEGA-3D remains strongest overall for localization-centric understanding.
  • Spatial Reasoning: On VSI-Bench, adding generative priors to Qwen2.5VL-7B produces consistent gains across the overall average and multiple spatial-reasoning categories.The plug-and-play augmentation remains lightweight compared with specialized geometry-aware mechanisms.
  • Embodied Manipulation: VEGA-3D transfers to real manipulation, improving Click Bell success from 40% (8/20) to 55% (11/20).The simulation evaluation uses LIBERO average success rate, while the real-world test uses Agilex Piper arms under the RoboTwin protocol.
  • Ablation Studies: Adaptive Gated Fusion outperforms lightweight fusion alternatives on most metrics because fixed fusion weights cannot reliably bridge semantic–generative distribution differences.Using only generative features causes a substantial drop; naive addition reaches SQA3D EM 61.8 versus 61.3 but is weaker on grounding and captioning.

6 Conclusion

VEGA-3D repurposes video generation models as Latent World Simulators and aligns their geometric priors with semantic tokens through adaptive fusion. The framework improves spatial understanding and manipulation without extra 3D supervision, while increasing inference cost.

  • Conclusion: VEGA-3D uses video generation models as Latent World Simulators to mitigate MLLMs’ spatial blindness.The training pipeline constructs semantic and generative branches, aligns and fuses them, then passes the resulting visual tokens to the language model.
  • Limitations and Future Work: The video diffusion backbone increases inference cost, motivating future distillation into lightweight encoders and extension to dynamic scenes and larger Vision-Language-Action models.The authors characterize the performance–cost trade-off as acceptable in practice.
  • Conclusion: Noise injection and Adaptive Gated Fusion activate generative priors and align them with semantic tokens, improving scene understanding, spatial reasoning, and manipulation without extra 3D supervision.The algorithmic pipeline includes generative feature caching or encoding, projection, fusion, and insertion of fused visual tokens into the language model.

B.1 Training Datasets

The experiments use established datasets across 3D scene understanding, spatial reasoning, and robotic manipulation, with visual features aligned into a common token grid and analyzed through voxel-based multi-view correspondence.

  • Training Datasets: Training covers 3D scene understanding, spatial reasoning, and robotic manipulation using ScanNet-derived tasks, VG-LLM’s S1 mixture, and the standard LIBERO benchmark.The 3D setting combines five public benchmarks; spatial reasoning uses SPAR-7M and the LLaVA-Hound split; manipulation uses four LIBERO suites.
  • 3D scene understanding: 3D scene understanding jointly trains on ScanRefer, Multi3DRefer, Scan2Cap, ScanQA, and SQA3D, all built from ScanNet scenes converted into video-style multi-view inputs.These tasks cover visual grounding, dense captioning, and question answering.
  • Spatial reasoning: Spatial reasoning uses VG-LLM’s S1 mixture without additional VLM-3R data, isolating the gain associated with generative priors under the core data setting.S1 combines sampled SPAR-7M instances with the LLaVA-Hound split of LLaVA-Video-178K.
  • Robotic manipulation: LIBERO evaluation follows OpenVLA-OFT across LIBERO-Spatial, LIBERO-Object, LIBERO-Goal, and LIBERO-Long.The suites vary spatial layouts, object identities, goal conditions, and long-horizon task compositions.
  • Backbone configurations: Features from compared visual backbones are generally aligned to 14×14 = 196 spatial tokens before fusion with the semantic branch.The backbone configurations include discriminative, 3D foundation, and generative encoders.
  • Multi-view correspondence analysis: The correspondence analysis samples up to 32 frames, pairs pooled feature tokens with 3D coordinates, groups them into 0.1 m voxels, and compares prototypes across distinct views.Same-view tokens within a voxel are averaged and L2-normalized; only voxels observed by at least two views contribute.

C.2 Normalized Overall Score (NOS)

The Normalized Overall Score summarizes heterogeneous downstream metrics after separate min-max normalization for discriminative and generative model groups, including the relevant baseline.

  • Definition: NOS normalizes each evaluation metric to [0, 1] and averages the normalized values across metrics.The final score is intended to summarize overall downstream performance with a single scalar.
  • Group-wise normalization: Normalization is performed separately for discriminative and generative models, with the corresponding baseline included in each group’s per-metric minimum and maximum.The group is either the discriminative-model group or the generative-model group together with the baseline.
  • Evaluation metrics: The NOS averages nine metrics spanning grounding, referring expression, captioning, and question-answering performance.The metrics include ScanRefer, Multi3DRefer, Scan2Cap, ScanQA, and SQA3D measures.
  • Reporting: The reported summary table contains NOS and multi-view correspondence scores in percentage, while the baseline row reports two NOS values in discriminative/generative normalization order.Bold formatting identifies the best result in each group.

D Probe analysis of the 3D priors in Generation Models

The probe analysis tests how generative priors vary with diffusion timestep and DiT depth, finding that intermediate timesteps generally provide stronger and more stable downstream performance.

  • Timestep and depth analysis: Intermediate diffusion timesteps generally yield stronger and more stable downstream performance than very late timesteps.This trend is reported for the main analysis and additional SEVA and Vmem timestep ablations.
  • Timestep and depth analysis: The analysis also varies DiT depth to examine how the extracted generative prior changes across network blocks.Exact downstream results for timestep and DiT-depth experiments are reported in supplementary tables.

E Additional Qualitative Results

Qualitative comparisons show stronger spatial grounding and more reliable video-based spatial reasoning, while a representative failure case indicates that fine-grained instance disambiguation remains challenging.

  • VSI-Bench: On VSI-Bench, the method handles appearance order, relative direction, and relative distance more reliably than the baseline.The examples cover temporal ordering, directional quadrants, and near/far relationships.
  • ScanRefer: VEGA-3D more precisely localizes referred objects in cluttered indoor scenes, including cases involving occlusion and ambiguous referring expressions.The qualitative ScanRefer comparisons describe stronger spatial grounding relative to the baseline.
  • Failure case: A representative ScanRefer failure shows that VEGA-3D can remain near the ground-truth region while struggling to distinguish among nearby similar objects.The limitation concerns fine-grained instance disambiguation rather than the absence of a useful spatial anchor.
  • Appearance Order: The appearance-order example reports VEGA-3D matching the ground truth while the baseline selects a different option.The ground-truth answer is B, VEGA-3D predicts B, and the baseline predicts D.
  • Relative Direction: The relative-direction example reports VEGA-3D matching the ground truth while the baseline chooses a different direction.The ground-truth answer is D, VEGA-3D predicts D, and the baseline predicts A.
  • Relative Distance: The relative-distance comparison indicates that VEGA-3D distinguishes near/far relationships and depth ordering more effectively than the baseline.The accompanying example has VEGA-3D matching the ground-truth option while the baseline selects another object.

F Limitations

The study identifies limitations involving task dependence, backbone suitability, computational overhead, extraction choices, and evaluation scope.

  • Generative priors help most on localization-centric and geometry-sensitive tasks, while captioning gains are less consistent.
  • DiT-based backbones show stronger multi-view consistency than several UNet-based alternatives, making transfer quality architecture- and pretraining-dependent.
  • A frozen video generator increases memory and inference cost, although feature caching reduces the practical overhead.
  • The best extraction setting currently requires manually selected intermediate timesteps and feature layers.
  • Experiments remain centered on indoor multi-view settings, leaving dynamic and open-world environments for future work.
Loading 2603.19235v3…