Source-linked AI summary
From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D
Jiahui Zhang, Yurui Chen, Yanpeng Zhou, Yueming Xu, Ze Huang, Jilin Mei, Junhui Chen, Yu-Jie Yuan, Xinyue Cai, Guowei Huang, Xingyue Quan, Hang Xu, Li Zhang
TL;DR
LVLMs still have limited spatial perception, while existing 3D-enhanced methods face data, representation, and integration challenges. The paper addresses this gap with a 3D-grounded 2D data pipeline, SPAR-7M, SPAR-Bench, and multi-view 3D grounding. Training with the resulting data achieves state-of-the-art performance on 2D spatial benchmarks and competitive results after 3D task-specific fine-tuning.
Problem
LVLMs struggle with spatial relationships in complex 3D scenes, while 3D-representation approaches face scarce data, difficult encoders, inaccessible representations, and integration challenges.
Method
The paper generates diverse 2D spatial QA from 3D-ground-truth scenes, constructs SPAR-7M and SPAR-Bench, and adds multi-view 3D grounding.
Results
Training on SPAR-7M with large-scale 2D data achieves state-of-the-art performance across multiple 2D spatial benchmarks, while 3D fine-tuning yields competitive results with 2D-only inputs.
Takeaways & Limitations
Spatial supervision from 3D-grounded 2D data supports spatial reasoning while remaining compatible with VLM inputs and subsequent 3D task fine-tuning.
Takeaways & Limitations
The study is static and image-centric, lacks continuous interactive embodied or long-horizon video reasoning, and relies on 3D ground truth that may be unavailable in some domains.
Abstract
from arXiv · showhide
Recent advances in LVLMs have improved vision-language understanding, but they still struggle with spatial perception, limiting their ability to reason about complex 3D scenes. Unlike previous approaches that incorporate 3D representations into models to improve spatial understanding, we aim to unlock the potential of VLMs by leveraging spatially relevant image data. To this end, we introduce a novel 2D spatial data generation and annotation pipeline built upon scene data with 3D ground-truth. This pipeline enables the creation of a diverse set of spatial tasks, ranging from basic perception tasks to more complex reasoning tasks. Leveraging this pipeline, we construct SPAR-7M, a large-scale dataset generated from thousands of scenes across multiple public datasets. In addition, we introduce SPAR-Bench, a benchmark designed to offer a more comprehensive evaluation of spatial capabilities compared to existing spatial benchmarks, supporting both single-view and multi-view inputs. Training on both SPAR-7M and large-scale 2D datasets enables our models to achieve state-of-the-art performance on 2D spatial benchmarks. Further fine-tuning on 3D task-specific datasets yields competitive results, underscoring the effectiveness of our dataset in enhancing spatial reasoning.
1 Introduction
The paper addresses limited LVLM spatial perception by training models on spatially relevant 2D data derived from 3D-grounded scenes. It introduces SPAR-7M and SPAR-Bench to cover spatial perception, reasoning, and single- and multi-view evaluation.
- Current VLMs struggle to perceive spatial relationships and reason about complex 3D scenes, despite advances in vision-language understanding.
- Existing 3D-representation approaches face scarce and unevenly distributed data, difficult 3D encoders, inaccessible complete representations, and architectural integration challenges.
- The proposed approach seeks to acquire spatial understanding directly from large-scale 2D observations without explicit 3D supervision, while also supporting 3D tasks through multi-view grounding.
- SPAR-7M contains 33 tasks organized into low, medium, and high levels spanning spatial perception and spatial reasoning, with numerical and semantic supervision.
- The 2D generation pipeline derives precise 3D-ground-truth supervision, improving task diversity and reliability compared with datasets relying solely on 2D annotations.
- SPAR-Bench contains 7,207 manually validated questions spanning basic perception to complex reasoning with both single-view and multi-view inputs.
- Training with SPAR-7M and large-scale 2D data achieves state-of-the-art performance across multiple 2D spatial benchmarks, while 3D fine-tuning remains competitive with 2D-only inputs.
2 Related works
Prior spatial datasets and benchmarks provide incomplete coverage of advanced spatial understanding. They commonly emphasize single-view or high-level reasoning settings and lack precise 3D annotations or sufficient multi-view evaluation.
- Existing multimodal datasets are mostly single-view or limited multi-view and target retrieval or basic VQA, making them insufficient for advanced 3D reasoning.
- Internet-collected image datasets often lack precise 3D annotations, while other resources use image sequences or 3D representations for narrower spatial tasks.
- Existing spatial benchmarks mainly focus on high-level reasoning and often neglect fundamental spatial perception tasks.
- Most benchmarks rely on single-view images or video inputs and give limited consideration to multi-view settings that better represent local 3D structures.
3 Method
The method converts 3D scene data into diverse 2D spatial QA and benchmark examples through filtering, unified scene organization, automatic task generation, and human verification. SPAR-7M and SPAR-Bench support perception, reasoning, and multi-view evaluation, alongside a text-generation-based 3D grounding procedure.
- 3 Method: The pipeline uses diverse 3D scenes from ScanNet, ScanNet++, and Structured3D to generate spatial understanding data.
- 3.1 3D-driven data generation pipeline: Figure 2 depicts filtering redundant images, extracting 3D information into a unified structure, and generating QA tasks from metadata.
- 3.1 3D-driven data generation pipeline: Redundant images are removed when camera position and orientation remain within distance and angular thresholds of retained images.
- 3.1 3D-driven data generation pipeline: The unified scene structure links image records and object records in world coordinates for cross-view retrieval and downstream annotation.
- 3.1 3D-driven data generation pipeline: Complete 3D scene information automatically produces diverse task types and QA pairs using human-created, GPT-expanded, and manually refined templates.
- 3.2 SPAR-7M: SPAR-7M contains 4,000+ indoor scenes, 33 task types, and over 7 million QA pairs paired with camera parameters and depth.
- 3.3 Spatial benchmark construction: The benchmark covers single-view, multi-view, and video inputs, including depth estimation, distance comparison, object relations, and spatial imagination.
- 3.4 Data quality check: Manual verification retained 7,207 questions from 8,000 sampled questions after approximately 140 human hours, yielding a 90.1% acceptance rate.
4 Experiments
Experiments evaluate SPAR-7M through existing spatial benchmarks, SPAR-Bench, 3D task fine-tuning, grounding probes, and data-ratio ablations. Results show stronger spatial performance and out-of-domain transfer, while multi-view reasoning and human-level performance remain challenging.
- Experimental setup: The experiments assess SPAR-7M on existing spatial benchmarks, SPAR-Bench, 3D tasks, implicit 3D grounding, and SPAR-ratio ablations.The evaluation includes image-based benchmarks, an in-domain benchmark, supervised fine-tuning on 3D tasks, and meter-level coordinate prediction from multi-view RGB.
- Existing spatial benchmarks: SPAR-mix achieves state-of-the-art results on CV-Bench 3D and VSI-Bench while keeping general-benchmark drops mostly within 1%.The comparison uses Base VLM + EMOVA-2M as the proper baseline because both systems include the same base model and general QA data.
- Existing spatial benchmarks: Pretraining improves spatial understanding across out-of-domain datasets, including an 18-point gain on the outdoor nuScenes dataset.The reported transfer extends beyond the indoor scenes used to construct most of the dataset.
- SPAR-Bench: Human performance exceeds all models on SPAR-Bench, with viewpoint changes, spatial imagination, and indirect metric reasoning remaining especially difficult.Qwen2.5-VL-72B reaches 35.4% on low-level tasks and 23.1% on middle-level tasks, while performance partially rebounds to 48.4% on high-level tasks.
- SPAR-Bench: Fine-tuning InternVL2.5-8B on SPAR-mix reaches 65.53, 63.01, and 61.29 on low-, mid-, and high-level tasks, respectively.View Change Inference improves from 6.32 to 37.14, and the fine-tuned model surpasses human performance in depth prediction.
- Ablation study: Reducing the SPAR-7M ratio from 60% to 40% slightly improves CV-Bench 2D from 73.00 to 74.13 but lowers CV-Bench 3D from 91.10 to 89.09 and harms SPAR-Bench performance.The largest reported SPAR-Bench declines occur on middle-level tasks, from 44.04 to 42.21, and high-level tasks, from 51.86 to 48.19.
5 Conclusion
SPAR-7M and SPAR-Bench target spatial reasoning through large-scale data and broader evaluation across single- and multi-view inputs. Pretraining achieves state-of-the-art results on VSI-Bench and CV-Bench 3D, while 3D-task fine-tuning remains competitive, though the tasks are primarily static QA.
- SPAR-7M is constructed from thousands of scenes to enhance spatial reasoning.
- SPAR-Bench provides a comprehensive spatial-understanding evaluation supporting both single-view and multi-view inputs.
- Pretraining on SPAR-7M alongside general datasets achieves state-of-the-art performance on VSI-Bench and CV-Bench 3D.
- Fine-tuning on 3D task-specific datasets yields competitive results.
- The tasks primarily use static QA and do not yet capture continuous spatial reasoning in embodied or interactive settings.
6 Limitation
The study is primarily static and image-centric, leaving continuous interactive reasoning for embodied or long-horizon video outside its scope. Metric 3D tasks also face scale ambiguity without additional cues, and the pipeline depends on 3D ground truth that may be unavailable in some domains.
- The study does not capture continuous, interactive reasoning for embodied or long-horizon video.
- Without metric 3D, monocular reconstruction suffers scale ambiguity.
- Metric tasks therefore require extra cues such as known sizes or stereo, or must use ordinal or normalized targets.
- The data pipeline relies on 3D ground truth, including depth, intrinsics, and poses, which may be unavailable in some domains.
7 Appendix section
The appendix describes spatial-task construction, benchmark coverage, 2D inference efficiency, and projection-based image-item generation from 3D scene data.
- 7.1 Inference Efficiency Analysis: 13×448×448 inputs require 0.14 s for feature extraction, while the 2D pipeline avoids 3D preprocessing overhead.The comparison covers feature extraction latency; 3D pipelines range from sub-second to minutes.
- 7.1 Inference Efficiency Analysis: 245.6 ms inference and CIDEr 87.6 result from reducing resolution from 13×448×448 to 13×224×224.The higher-resolution setting reports 771.1 ms inference and CIDEr 90.7.
- 7.1 Inference Efficiency Analysis: Peak memory can exceed that of some 3D models, although frame count and resolution provide configurable cost controls.The approach retains competitive accuracy while trading sequence length for latency and memory budgets.
- 7.2 Task Coverage: Single-view tasks cover depth, distance, object-center distance, spatial relations, and spatial imagination.These tasks infer spatial properties from one image, including absolute or relative depth and unseen relationships.
- 7.2 Task Coverage: Multi-view tasks reason across images through depth, distance, object matching, camera pose, spatial relations, and viewpoint-change inference.The listed tasks include predicting camera movement needed to obtain a second perspective.
- 7.3 Image Subsampling: Camera-pose filtering removes approximately 90% of redundant images while preserving sufficiently distinct viewpoints.Images are filtered using translation and rotation thresholds, with dataset-specific settings; Structured3D is not filtered.
- 7.4 Image Item Construction: The pipeline loads scene meshes, camera parameters, and annotations, rasterizes visible faces, projects 3D objects into images, and computes bounding boxes.Projection validity uses visibility, projected-area, and depth-range constraints before downstream task generation.
- 7.5 Task Data Generation: Task generation transforms object coordinates into a common camera view and derives relationships using above-below, left-right, near-far, and front-behind factors.The resulting structured question-answer pairs support object spatial-relation tasks.