Source-linked AI summary
VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction
Zhiwen Fan, Jian Zhang, Renjie Li, Junge Zhang, Runjin Chen, Hezhen Hu, Kevin Wang, Huaizhi Qu, Shijie Zhou, Dilin Wang, Zhicheng Yan, Hongyu Xu, Justin Theiss, Tianlong Chen, Jiachen Li, Zhengzhong Tu, Zhangyang Wang, Rakesh Ranjan
TL;DR
Existing VLMs struggle with geometric spatial understanding and often depend on depth sensors or pre-constructed 3D maps. VLM-3R addresses this with monocular-video 3D reconstructive instruction tuning, and experiments show robust static and temporal 3D spatial reasoning.
Problem
Current VLMs struggle with geometric spatial skills, while existing spatial methods rely on depth sensors or pre-constructed 3D maps that limit monocular-video scalability.
Method
VLM-3R combines image semantics and metric-scale geometric information into implicit spatial and view tokens, supported by reconstructive instruction tuning and spatio-temporal QA generation.
Results
VLM-3R reasons about static spatial context and temporal 3D changes induced by camera motion across benchmarks, including VSTI-Bench’s five temporal tasks.
Takeaways & Limitations
The framework enables 3D spatial understanding directly from monocular video without depth sensors or pre-computed 3D maps, while supporting temporal spatial evaluation.
Abstract
from arXiv · showhide
The rapid advancement of Large Multimodal Models (LMMs) for 2D images and videos has motivated extending these models to understand 3D scenes, aiming for human-like visual-spatial intelligence. Nevertheless, achieving deep spatial understanding comparable to human capabilities poses significant challenges in model encoding and data acquisition. Existing methods frequently depend on external depth sensors for geometry capture or utilize off-the-shelf algorithms for pre-constructing 3D maps, thereby limiting their scalability, especially with prevalent monocular video inputs and for time-sensitive applications. In this work, we introduce VLM-3R, a unified framework for Vision-Language Models (VLMs) that incorporates 3D Reconstructive instruction tuning. VLM-3R processes monocular video frames by employing a geometry encoder to derive implicit 3D tokens that represent spatial understanding. Leveraging our Spatial-Visual-View Fusion and over 200K curated 3D reconstructive instruction tuning question-answer (QA) pairs, VLM-3R effectively aligns real-world spatial context with language instructions. This enables monocular 3D spatial assistance and embodied reasoning. To facilitate the evaluation of temporal reasoning, we introduce the Vision-Spatial-Temporal Intelligence benchmark, featuring over 138.6K QA pairs across five distinct tasks focused on evolving spatial relationships. Extensive experiments demonstrate that our model, VLM-3R, not only facilitates robust visual-spatial reasoning but also enables the understanding of temporal 3D context changes, excelling in both accuracy and scalability.
1. Introduction
Current VLMs and LMMs remain limited in geometric spatial understanding, while existing approaches often depend on depth sensors or pre-constructed 3D maps. VLM-3R addresses this with monocular-video 3D reconstructive instruction tuning and a benchmark for temporal spatial reasoning.
- Current VLMs and LMMs struggle with geometric spatial skills, including simple distance estimation, when interacting with the physical world.
- Depth sensors constrain scalability to sensor-equipped environments and limit the use of abundant monocular video data.
- Pre-constructed 3D maps make multi-stage pipelines slow to adapt and brittle when reconstruction ignores real-world scene scale.
- VLM-3R extracts implicit spatial and view tokens from monocular video and fuses them with visual features for language-aligned spatial understanding.
- VSTI-Bench evaluates temporal changes in spatial relations caused by camera motion across five temporal tasks defined on static 3D scenes.
2. Related Work
Related work extends multimodal models toward embodied and spatial reasoning, but geometry-augmented approaches still face scale ambiguity and reconstruction-efficiency challenges. Existing benchmarks and reconstruction pipelines motivate more scalable temporal spatial evaluation.
- Large Multimodal Models have expanded from joint image-text representations toward embodied agents, grounding, and general-purpose task solving.
- Geometry-aware VLMs fuse multi-view geometry tokens with vision and language features, but normalized-depth predictions remain scale-ambiguous for metric distance and object-size reasoning.
- Spatial reasoning benchmarks assess configurational, measurement-estimation, and spatiotemporal abilities in real-world videos.
- Classical SfM+MVS reconstruction pipelines are highly accurate but notoriously slow to optimize in practice.
- VSTI-Bench organizes 138.6K QA samples by primary and sub-category distributions and illustrates representative spatio-temporal task types.
3. Scalable Spatial and Temporal Data
The paper builds scalable spatial and temporal supervision from existing 3D datasets, simulators, and automated scene-graph procedures. It also introduces VSTI-Bench to evaluate spatial configurations evolving over time.
- VSTI-Bench contains 138.6K samples focused on spatial configurations evolving over time.
- Earlier spatial benchmarks contain about 5,000 manually or semi-automatically created QA pairs, limiting scalable supervision for robust spatial reasoning.
- An automated pipeline generates over 200K diverse spatial reasoning QA pairs and 4,225 simulator-based embodied route-planning instances.
- A temporal scene graph represents frames as temporal nodes and object instances with global, local, and semantic attributes for automatic QA generation.
- Habitat simulation generates accurate and plausible navigation routes at scale by sampling navigable paths in 3D scenes.
- The benchmark evaluates camera motion, camera-object interactions, and evolving spatial configurations across approximately 138,600 QA pairs.
4. VLM-3R Architecture
VLM-3R processes monocular video with visual and geometry encoders, representing scene structure and camera pose as implicit 3D tokens. Spatial-Visual-View Fusion injects these tokens into the VLM alongside language instructions for spatial and temporal reasoning.
- Overview: Monocular video frames and language instructions are encoded into visual, geometric, and camera-pose tokens aligned with language representations through instruction tuning.
- 3D Reconstructive Tokenization: CUT3R processes frames recurrently to produce context-aware image tokens and pose-related output tokens for 3D reconstructive tokenization.
- 3D Reconstructive Tokenization: Prediction heads use enriched tokens to produce metric-scale 3D point maps and relative camera poses, avoiding normalized-scale outputs.
- Training: The visual and CUT3R-based spatial encoders are frozen during subsequent training, while LoRA updates the fusion-attention and projection layers.
- Spatial-visual View Fusion: Spatial and camera-view tokens form a unified 3D representation that cross-attends with native visual tokens, preserving appearance through a residual connection.
- Spatial-visual View Fusion: The fused visual tokens are projected into the language-model input space, concatenated with instruction tokens, and passed to the transformer backbone.
5. Experiments
VLM-3R is evaluated across spatial, temporal, online embodied, and general video/image benchmarks, with strong gains from its 3D fusion and reconstructive training. Ablations further show that spatial, view, and 3D fusion components contribute to performance.
- Benchmark evaluations: Relative Direction improves from 42.4 to 80.5, while Absolute Distance rises from 20.2 to 49.4 and Room Size from 12.3 to 67.1.These comparisons are between the 2D-only baseline and the full VLM-3R model.
- Generalization: Video-MME performance increases from 59.9% to 62.1% (+2.2 pp) when 30k general videos are mixed with 200k domain-specific spatial data.The mixed model remains within 1pp of the original LLaVA-NeXT-Video-7B, while VLM-3R also improves Video-MME Spatial Perception by +3.7%.
- Benchmark evaluations: VLM-3R ranks first among open-source models on VSI-Bench after training with 200K spatial reasoning QA pairs.VSI-Bench covers eight spatial reasoning tasks using multiple-choice and numerical-answer formats.
- Benchmark evaluations: VLM-3R achieves leading VSTemporalI-Bench performance and captures camera motion, camera–object interactions, and evolving spatial configurations.The benchmark evaluates temporal reasoning in monocular videos, including tracking static objects under moving cameras.
- Benchmark evaluations: VLM-3R outperforms its base model across OST-Bench categories, particularly for Agent State, Agent–Object Spatial Relationship, and Estimation tasks.This evaluates online, temporally grounded reasoning in an actively exploring embodied setting.
- Ablation studies: Removing spatial tokens lowers the overall score from 60.90 to 59.46, removing view tokens lowers it to 50.09, and 2D–2D fusion averages 58.12 versus 60.90.Direct point-cloud fusion also scores 57.87 versus 60.90, while the full model gains 3.2 points over the LLaVA-NeXT-Video fine-tuned baseline.
6. Conclusion
VLM-3R combines reconstructive instruction tuning with Spatial-Visual-View fusion to support 3D understanding from monocular video. The paper also introduces a large temporal benchmark while identifying reconstruction accuracy, 4D data collection, and dynamic environments as remaining boundaries.
- Contributions: VLM-3R uses over 200K curated training instances and Spatial-Visual-View fusion to derive 3D spatial understanding directly from monocular video.The framework eliminates depth sensors and pre-computed 3D maps while incorporating camera-view context.
- Contributions: The approximately 138.6K-pair benchmark targets temporal reasoning that is often overlooked in current evaluations.Its broader applicability remains tied to large-scale 4D data collection and accurate end-to-end 3D reconstruction.
- Limitations: Current datasets prioritize static indoor scenes, leaving dynamic and extreme environments for future work.The conclusion identifies this as a scope boundary for the present system and data.
Supplementary Material
The spatial encoder maps monocular image sequences to dense 3D and camera representations, while implicit geometry and view tokens are fused with 2D visual tokens before language processing.
- Spatial encoder: The feed-forward spatial encoder maps RGB image sequences to estimated camera intrinsics, extrinsics, and point maps associating pixels with 3D coordinates.The design uses end-to-end dense 3D reconstruction from monocular video.
- Spatial encoder: Implicit encoder tokens are aligned instead of using global point clouds, reducing computational and cross-scene scaling difficulties.CUT3R processes monocular frames and outputs dense point maps and relative camera poses for each view.
- Fusion: Spatial-Visual-View Fusion combines geometry tokens encoding scene structure, camera-view tokens encoding global motion, and 2D appearance tokens.The geometry and view tokens have dimensions 729 × 768 and 1 × 768, while visual tokens have dimension 729 × 1152.
- Fusion: Concatenated 3D tokens interact with visual tokens through one-layer cross-attention, whose output is residually added to the original visual representation.The resulting features are projected and concatenated with language instruction tokens for LMM processing.
A.2. Training Details
Training initializes from LLaVA-Video-7B-Qwen2 and uses parameter-efficient adaptation with frozen spatial and visual encoders. The configuration specifies LoRA, distributed training, and a CUT3R-based cross-attention spatial module.
- Model configuration: Training initializes from the LLaVA-Video-7B-Qwen2 checkpoint with a SigLIP vision tower.The vision tower is google/siglip-so400m-patch14-384.
- Optimization: Training uses 16 H200 GPUs for approximately five hours with DeepSpeed ZeRO stage 2 and torchrun-based distributed execution.The run uses the accelerate library and SLURM environment variables for distributed setup.
- Optimization: LoRA uses rank 128 and alpha 256, with a nominal five-epoch schedule that concluded after the first epoch.The per-device batch size is 1, gradient accumulation is 8, and the learning rate is 2 × 10^-5.
- Spatial module: The spatial module uses CUT3R with 768-dimensional features and a cross_attention fusion block.The spatial tower is frozen, while the fusion block and multimodal MLP adapter are tunable.
B. Dataset Curation and Benchmark Design
The paper builds scalable 3D reconstructive instruction data from structured scene and frame metadata, covering configurational, measurement, spatiotemporal, and route-planning reasoning tasks. These QA pairs support training models for visual-spatial intelligence.
- Structured metadata files encode scene dimensions, room centers, object counts, 3D bounding boxes, camera intrinsics, frame information, and camera motion.
- Configurational Tasks: Configurational tasks cover object counting, relative distance, relative direction, and route-plan action completion.
- Measurement Estimation Tasks: Measurement tasks estimate object size, absolute distance, and room size using numerical answers.
- Spatiotemporal Task: Spatiotemporal tasks determine the first-appearance order of four object categories in a video sequence.
- Route Planning: Habitat-generated trajectories and nearest-object anchor descriptions produce multiple-choice route-planning QA pairs.
- 207,779 QA pairs form the VSI-Bench training-data distribution across multiple spatial reasoning task types.
B.2. VSTemporalI-Bench: New Evaluation Benchmark (138.6K Set)
VSTemporalI-Bench evaluates whether models can reason about evolving spatial relationships in monocular-video sequences, especially changes induced by camera motion. It organizes approximately 138.6K QA pairs into five temporal reasoning tasks.
- The benchmark targets camera displacement, camera-object distance and direction changes, and frame-dependent object-object relative positions.
- Task Organization: Its two main temporal scopes are Frame-Level tasks and Sequence-Level tasks.
- VSTI-Bench contains approximately 138.6K QA pairs across five distinct question types.
- Frame-Level Tasks: Frame-Level tasks include camera-object absolute distance, camera-object relative distance, and object-object relative position questions.
- 138,610 QA pairs comprise the VSTemporalI-Bench training and testing splits.
C.1. Task-wise Analysis on VSI-Bench
VLM-3R’s gains are concentrated on tasks requiring explicit geometric perception and metric-aware reasoning, while object counting remains comparable to the baseline. Camera and geometry tokens both contribute to absolute-scale distance estimation.
- The strongest gains occur on reasoning categories that require explicit geometric perception and metric-aware reasoning.
- 49.38 on Absolute Distance exceeds the LLaVA-NeXT-Video ft (w/o C&G Tok.) baseline’s 43.67.
- Removing camera tokens or geometry tokens lowers Absolute Distance performance to 48.66 and 49.27, respectively.
- 70.16 on Object Counting is comparable to the baseline’s 70.64 and the no-geometry-token variant’s 70.30.
C.2. Ablation on Pretrained Geometry Encoders
Pretrained geometry encoders improve relative spatial reasoning, but encoder choice matters for temporal ordering and metric estimation. CUT3R best matches these requirements and delivers the strongest overall VSI-Bench performance.
- Both CUT3R and VGGT improve relative spatial reasoning over the finetuned base model.
- VGGT underperforms on Appearance Order, decreasing from 38.5 to 34.5, and Room Size, decreasing from 63.7 to 54.0.
- 60.9% overall performance with CUT3R is the best reported result, including Appearance Order at 40.1%, Room Size at 67.1%, and Route Plan at 45.4%.
- OpenEQA Transfer: On OpenEQA spatial questions, VLM-3R improves over its base model from 49.95% to 51.60%.
- OpenEQA Transfer: VLM-3R reaches 61.7% overall zero-shot accuracy on OpenEQA and outperforms the listed general-purpose multimodal baselines.