Source-linked AI summary
OpenSpatial: A Principled Data Engine for Empowering Spatial Intelligence
Jianhui Liu, Haoze Sun, Wenbo Li, Yanbing Zhang, Rui Yang, Zhiliang Zhu, Yijun Yang, Shenghe Zheng, Nan Jiang, Jiaxiu Jiang, Haoyang Huang, Tien-Tsin Wong, Nan Duan, Xiaojuan Qi
TL;DR
Spatial reasoning in multimodal models remains limited, while spatial data production is often narrow and closed. OpenSpatial introduces an open, 3D box-centric data engine and OpenSpatial-3M, whose training data yields strong cross-benchmark gains and supports controlled analysis of spatial data design.
Problem
MLLMs still struggle with accurate spatial perception, while limited diversity and closed, under-specified data pipelines constrain spatial generalization.
Method
OpenSpatial uses 3D box-centric grounding, 3D lifting, and scene-graph-driven synthesis to create scalable, high-quality, task-diverse supervision and a 3-million-sample curriculum.
Results
OpenSpatial-3M training produces state-of-the-art performance across diverse spatial benchmarks, with a 14.1% average improvement and maximum gain of 19% over baseline.
Takeaways & Limitations
The open-source engine and dataset establish transparent, controllable infrastructure for reproducible spatial-data scaling and analysis across architectures.
Takeaways & Limitations
Performance remains bottlenecked in some desktop-level and outdoor scenarios because of skewed data distribution.
Abstract
from arXiv · showhide
Spatial understanding is a fundamental cornerstone of human-level intelligence. Nonetheless, current research predominantly focuses on domain-specific data production, leaving a critical void: the absence of a principled, open-source engine capable of fully unleashing the potential of high-quality spatial data. To bridge this gap, we elucidate the design principles of a robust data generation system and introduce OpenSpatial -- an open-source data engine engineered for high quality, extensive scalability, broad task diversity, and optimized efficiency. OpenSpatial adopts 3D bounding boxes as the fundamental primitive to construct a comprehensive data hierarchy across five foundational tasks: Spatial Measurement (SM), Spatial Relationship (SR), Camera Perception (CP), Multi-view Consistency (MC), and Scene-Aware Reasoning (SAR). Leveraging this scalable infrastructure, we curate OpenSpatial-3M, a large-scale dataset comprising 3 million high-fidelity samples. Extensive evaluations demonstrate that versatile models trained on our dataset achieve state-of-the-art performance across a wide spectrum of spatial reasoning benchmarks. Notably, the best-performing model exhibits a substantial average improvement of 19 percent, relatively. Furthermore, we provide a systematic analysis of how data attributes influence spatial perception. By open-sourcing both the engine and the 3M-scale dataset, we provide a robust foundation to accelerate future research in spatial intelligence.
1 Introduction
OpenSpatial addresses uneven spatial competence and limited, closed data pipelines by introducing an open, reusable engine for scalable, diverse spatial supervision. Its 3M-sample dataset improves benchmark performance across architectures and supports controlled analysis of data-design choices.
- Motivation: MLLMs often describe scenes convincingly but struggle with accurate distance perception, multi-view consistency, and spatial cognitive maps.
- Motivation: Limited data diversity and closed, under-specified generation pipelines constrain robustness and contribute to “spatial myopia.”
- Approach: OpenSpatial uses 3D box-centric grounding, 3D lifting, and scene-graph-driven synthesis to produce high-quality, scalable, and task-diverse supervision.
- Dataset: 3 million samples span Spatial Measurement, Spatial Relationship, Camera Perception, Multi-view Consistency, and Scene-Aware Reasoning in a curriculum-style training suite.
- Results: 14.1% average improvement and a maximum gain of 19% over the baseline are reported across architectures and spatial benchmarks.
- Contributions: The open-source engine and dataset provide transparent, controllable infrastructure for principled ablations, diagnostic analyses, and reproducible scaling.
2 Related Work
Related work has advanced general multimodal intelligence and spatial reasoning through model architectures, datasets, and training paradigms, but sophisticated geometric and spatial configuration reasoning remains difficult.
- Large Vision-Language Models: Large vision-language models progressed from image-text alignment toward instruction-following systems through visual feature and training-pipeline advances.
- Spatial Reasoning: Existing LVLMs still struggle with spatial reasoning requiring geometric transformations and interpretation of intricate spatial configurations.
- Spatial Reasoning: Spatial-reasoning research has pursued architectural augmentation, large-scale dataset curation, and advanced training paradigms.
3 Implementation Principles of OpenSpatial
OpenSpatial builds a unified spatial data pipeline around world-coordinate 3D oriented bounding boxes, then converts scene annotations into indexed attributes and diverse single- and multi-view QA. OpenSpatial-3M organizes 3M entries across five capabilities and 19 sub-tasks.
- Data Engine: OpenSpatial generates supervision from a unified 3D box-centric representation using human annotation for accuracy and automated 3D lifting for scalable expansion.
- Dataset Organization: OpenSpatial-3M contains 3M entries across five foundational capabilities and 19 sub-tasks for general-purpose spatial understanding.
- Data Pipeline: The pipeline converts multi-view images or video keyframes into scene-level oriented boxes, frame-level attributes, masks, and object–frame indices.
- 3D Box-Centric Design: Oriented bounding boxes provide compact, world-coordinate, viewpoint-invariant anchors for cross-frame association, projection, visibility filtering, and mask refinement.
- QA Synthesis: Scene graphs enumerate objects, attributes, and relations to generate balanced single-view and multi-view QA covering measurement, relations, camera changes, consistency, and scene reasoning.
- Data Sources: The dataset combines sources including ScanNet, Matterport3D, ARKitScenes, ScanNet++, and Hypersim, while excluding SUN-RGBD for lower annotation fidelity.
4 Experiments
Experiments evaluate OpenSpatial through benchmark comparisons, module ablations, scaling studies, data-source expansion, task-diversity analysis, and pipeline-efficiency measurements. The results show broad spatial gains, benefits from the box-centric design and scaling, and remaining limitations from skewed data coverage.
- Quality Evaluation: OpenSpatial models achieve state-of-the-art spatial benchmark performance while preserving general multimodal versatility, with typical gains of 5.4–9.5 points and over 10 points on BLINK, AllAngles, and MMSI.The comparison uses identical evaluation settings across specialized spatial and general multimodal benchmarks.
- Limitations: Performance remains bottlenecked in some desktop-level and outdoor scenarios because the current data distribution is skewed toward insufficient coverage of complex environments.The authors identify broader data coverage as a future direction.
- Quality Evaluation: OpenSpatial and VST show the narrowest mean gaps across benchmarks, at −2.5 and −2.8 respectively, while Cambrian-S and SenseNova-SI fluctuate more.SenseNova-SI nevertheless contributes localized strengths, motivating its integration for specialized scenarios.
- Quality Evaluation: Box-centric representations outperform point-cloud-centric alternatives because partial point clouds fail to capture complete object geometry and produce inaccurate supervision.Occluded cases can introduce hallucinations when they are not filtered.
- Scalability Evaluation: 3D-Avg increases consistently with data volume, although individual benchmarks are not strictly monotonic and gains diminish as scale grows.The observed diminishing returns suggest that further improvements require exponentially larger datasets.
- Scalability Evaluation: An automated 3D-lifting pipeline reconstructs in-the-wild video scenes while producing semantic tags, masks, and 3D bounding boxes, extending scene and source diversity.The pipeline is illustrated on an uncurated outdoor video and is paired with parallel, asynchronous, and feature-reuse optimizations.
- Scalability Evaluation: Larger Qwen2.5-VL models, evaluated at 3B, 7B, and 32B parameters under identical data configurations, show upward trends across nearly all metrics.The model-scaling study examines representative parameter sizes under fixed data conditions.
- Task Diversity: Task contributions are heterogeneous across benchmarks: Spatial Measurement benefits metric-heavy evaluations, while Camera Perception has different performance footprints.The task-diversity analysis uses a benchmark-by-task heatmap to examine complementarity.
5 Conclusion
OpenSpatial shifts spatial-intelligence research from static datasets toward transparent, scalable infrastructure built around 3D box-centric supervision. Its OpenSpatial-3M dataset and open engine support diverse, extensible spatial-data production for embodied AI and robotics.
- Conclusion: OpenSpatial establishes a transparent, scalable infrastructure that replaces a static-dataset focus with a principled data-engine approach.The engine is designed as an open-source foundation for spatial intelligence.
- Conclusion: A 3D box-centric paradigm bridges sparse 2D visual cues and intrinsic 3D metric properties through viewpoint-invariant supervision.The representation supports data synthesis across five hierarchical task categories.
- Conclusion: OpenSpatial-3M achieves state-of-the-art performance across diverse multimodal model architectures.The dataset is synthesized by the OpenSpatial engine.
- Conclusion: Open-sourcing the engine aims to democratize high-quality spatial-data creation and provide a foundation for embodied AI and robotics research.The infrastructure is intended to support continuous expansion and refinement of spatial understanding.