Source-linked AI summary
Learning Geometric Representations from Videos for Spatial Intelligent Multimodal Large Language Models
Haibo Wang, Lifu Huang
TL;DR
MLLMs are strong at 2D semantics but lack intrinsic 3D awareness and geometric consistency, while large-scale 3D training data remain scarce. GeoVR learns from pure 2D videos by restructuring internal representations with four geometric objectives and reports state-of-the-art spatial reasoning performance without inference-time overhead.
Problem
MLLMs trained primarily with language supervision lack explicit geometric signals for pose, depth, scale, and multi-view structural consistency.
Method
GeoVR restructures MLLM representations using training-time camera-pose, dense-depth, metric-scale, and multi-scale 3D representation objectives distilled from a frozen 3D teacher.
Results
GeoVR achieves state-of-the-art spatial reasoning performance, including average scores of 69.1 and 69.6 for GeoVR-2B and GeoVR-4B, respectively.
Takeaways & Limitations
GeoVR demonstrates that geometric awareness can be learned from purely 2D videos while preserving zero additional inference-time computational overhead.
Takeaways & Limitations
The reported default and ablation studies use Qwen3-VL-2B-Instruct, VGGT, about 374K video samples, one epoch, and at most eight frames per video.
Abstract
from arXiv · showhide
Multimodal Large Language Models (MLLMs) excel at 2D semantic understanding but lack intrinsic 3D awareness, resulting in representations that fail to maintain geometric and spatial consistency across video frames. Given the scarcity of large-scale 3D data, we present GeoVR, a novel framework that learns geometric representations using purely 2D video sequences. This approach effectively restructures the semantic latent space within MLLMs to unlock spatial intelligence. Rather than employing superficial feature mixing, GeoVR reshapes the internal representations of the MLLM by distilling geometry knowledge from pre-trained 3D foundation models. This is accomplished through a multi-objective learning strategy driven by four complementary geometric targets: (1) estimating inter-frame camera poses to embed varying viewpoint dynamics, (2) regressing dense depth maps to anchor physical distances, (3) predicting a metric scale factor for real-world calibration, and (4) distilling multi-scale 3D features to align the intermediate feature space. Guided by these explicit physical and geometric constraints, the model's internal representations naturally develop strong 3D awareness. Extensive experiments on spatial reasoning benchmarks demonstrate that GeoVR achieves state-of-the-art performance, establishing a new paradigm for endowing foundation models with spatial intelligence.
1. Introduction
GeoVR addresses the geometric representation deficiency of MLLMs by learning from pure 2D videos and using training-only geometric constraints to restructure their latent space.
- Motivation: MLLMs trained with 2D semantic supervision often lack intrinsic representations of pose, depth, scale, and multi-view consistency.This limits their robustness for dynamic viewpoint shifts and physical-world reasoning.
- Limitations of Existing Paradigms: Existing 3D-data approaches depend on scarce explicit annotations, limiting scalability and potentially weakening standard 2D visual generalization.
- Limitations of Existing Paradigms: Feature-fusion approaches add external 3D features to 2D tokens but do not fundamentally instill geometric awareness and impose inference overhead.
- GeoVR: GeoVR learns geometric representations directly from pure 2D video sequences without manual 3D annotations by restructuring the MLLM’s internal semantic space.
- GeoVR: GeoVR uses camera-pose estimation, depth prediction, metric-scale calibration, and multi-scale geometric representation alignment as training-only constraints.
- Results: Extensive experiments report state-of-the-art performance on spatial reasoning benchmarks without additional inference-time computational overhead.
2. Related Work
Prior work either relies on scarce explicit 3D inputs, uses only semantic supervision for 2D videos, or passively fuses 3D foundation-model features. GeoVR instead restructures MLLM representations with training-time geometric objectives.
- 3D Scene Understanding: Explicit-3D methods project point clouds or reconstructed meshes into MLLM embeddings but face severe scarcity of large-scale, high-quality 3D-text data.
- 2D-Based Spatial Reasoning: 2D image- and video-based methods avoid explicit 3D inputs but use semantic supervision that lacks true physical depth and multi-view consistency.
- 3D Reconstruction: Feed-forward 3D reconstruction methods efficiently predict attributes such as 3D points, dense depths, and camera parameters from image or video inputs.
- MLLMs with 3D Foundation Models: Recent MLLM approaches use frozen 3D foundation models through patch-level addition or cross-attention to fuse 3D features with 2D tokens.
- GeoVR: GeoVR uses multi-objective geometric learning to restructure the MLLM’s internal semantic latent space into geometry-aware representations.
3. Method
GeoVR restructures an MLLM’s internal latent space into geometry-aware representations by learning from pure 2D videos with targets generated by a frozen 3D foundation model. Its multi-objective framework combines camera pose, dense depth, metric scale, and multi-scale representation alignment to introduce explicit geometric constraints.
- Framework Overview: GeoVR learns geometry-aware MLLM representations directly from 2D video sequences by distilling targets from a frozen 3D foundation model.The 3D teacher generates geometric pseudo-labels from arbitrary videos, avoiding reliance on manually annotated 3D datasets.
- Problem Formulation: The standard language-only objective lacks explicit geometric supervision, leaving visual tokens unable to represent pose, depth, scale, and multi-view consistency.Comparisons with VGGT show weaker cross-view correspondences and greater semantic ambiguity in the MLLM representations.
- Camera Pose Estimation: Camera pose learning uses per-frame camera tokens and a lightweight MLP head to regress each frame’s 9-dimensional camera parameter vector.Self-attention lets camera tokens aggregate multi-view context, while the pose constraints encourage representations of a consistent 3D scene under viewpoint changes.
- Depth Map Prediction: Dense depth learning aggregates multi-scale MLLM features with a DPT head to predict high-resolution depth maps supervised by L1 and gradient-based losses.The objective grounds visual tokens in spatial layout and physical distances while retaining both structural detail and semantic context.
- Metric Scale Calibration: Metric scale calibration adds a video-level scale token whose exponentially activated MLP head predicts a positive absolute scale factor.The scale objective addresses the ambiguity that remains when camera pose and depth capture only relative scene structure.
- Geometric Representation Alignment: Multi-scale distillation aligns intermediate MLLM features with structured 3D teacher features after spatial resizing and channel projection, while the total objective jointly combines language and geometric losses.Alignment is enforced across multiple intermediate layers, and the loss balances language modeling with camera, depth, scale, and representation-alignment terms.
4. Experiments
GeoVR is evaluated on spatial reasoning benchmarks and through ablations of its geometric teachers, objectives, alignment strategy, and depth heads. Results show strong performance from purely 2D video inputs, with complementary geometric supervision and multi-scale alignment providing key gains.
- Benchmark Evaluation: GeoVR-2B and GeoVR-4B achieve average VSI-Bench scores of 69.1 and 69.6, respectively, surpassing GPT-5 and LLaVA-OneVision-72B.GeoVR-4B matches SpaceMind-8B’s 69.6 using zero additional architectural overhead and half the parameters.
- Benchmark Evaluation: GeoVR-4B records 63.9 EM-1 on SQA3D, establishing a state-of-the-art result among video-input models.GeoVR-2B also surpasses LLaVA-3D on ScanQA with CIDEr 103.3 and EM-1 61.4 without explicit point clouds during inference.
- 3D Foundation Model Backbone: The base VGGT teacher outperforms VGGT-Ω, while both VGGT variants consistently surpass DA-3 in the 3D foundation-model comparison.The analysis attributes VGGT-Ω’s lower performance to reduced fine-grained spatial correspondences from register attention.
- Multi-task Geometric Learning: The full four-objective model reaches 62.1 average VSI-Bench performance, exceeding text-only supervision at 56.7 and partial objective combinations.Camera pose, depth, and metric scale each provide gains, while their combination with representation alignment performs best.
- Alignment at Different Layers: Uniform multi-scale alignment at layers [5] [12] [18] [24] reaches 59.67, outperforming single-layer alignment and incomplete two-layer pairings.The ablation links evenly distributed distillation with progressively internalized spatial priors.
- Depth Prediction Heads and Loss: The Dense depth head with L1 supervision achieves the highest performance at 60.30, leading the adopted depth-prediction design.The comparison includes DPT, MLP, and Dense heads alongside L1 and SILog losses.
- Feature Visualization: PCA visualizations show less noisy, more geometrically consistent visual tokens after multi-objective training, while predicted depths and poses reconstruct plausible 3D point clouds.The reconstructed scenes exhibit spatial fidelity comparable to the 3D foundation model.
5. Conclusion
GeoVR restructures MLLM representations into geometry-aware features using purely 2D videos and multiple geometric learning targets. Experiments show improved spatial reasoning, while future work targets larger models, datasets, and more complex tasks.
- Conclusion: GeoVR estimates camera poses, predicts dense depths, calibrates metric scale, and distills multi-scale geometric priors from a pretrained 3D teacher.These objectives restructure the MLLM’s semantic latent space into geometry-aware representations using 2D video sequences.
- Conclusion: Extensive experiments demonstrate that GeoVR significantly enhances MLLMs’ spatial reasoning capabilities.The paper identifies scaling to larger architectures and datasets, plus more complex spatial-intelligence tasks, as future directions.