Source-linked AI summary
Scaling Spatial Intelligence with Multimodal Foundation Models
Zhongang Cai, Ruisi Wang, Chenyang Gu, Fanyi Pu, Junxiang Xu, Yubo Wang, Wanqi Yin, Zhitao Yang, Chen Wei, Qingping Sun, Tongxi Zhou, Jiaqi Li, Hui En Pang, Oscar Qian, Yukun Wei, Zhiqian Lin, Xuanke Shi, Kewang Deng, Xiaoyang Han, Zukai Chen, Xiangyu Fan, Hanming Deng, Lewei Lu, Liang Pan, Bo Li, Ziwei Liu, Quan Wang, Dahua Lin, Lei Yang
TL;DR
Multimodal foundation models remain deficient in spatial intelligence, partly amid scarce, fragmented, and imbalanced spatial data. The paper scales taxonomy-guided diverse data across several multimodal foundations, producing SenseNova-SI models that achieve strong spatial benchmark performance while retaining general capabilities. It also examines scaling, generalization, shortcuts, spatial chain-of-thought, and downstream robotic manipulation.
Problem
Multimodal foundation models still struggle with spatial intelligence, while high-quality, diverse spatially grounded data remains scarce, fragmented, and imbalanced.
Method
The paper uses a taxonomy-guided, data-centric approach to scale diverse spatial collections across InternVL3, Qwen3-VL, and Bagel without altering their original architectures.
Results
SenseNova-SI achieves state-of-the-art performance across major spatial-intelligence benchmarks while retaining strong general multimodal understanding, including 84.9% on MMBench-En.
Takeaways & Limitations
The models provide a strong open research baseline, with validated spatial capability acquisition and potential utility for embodied robotic manipulation.
Takeaways & Limitations
Performance gains gradually diminish as training data increases, and data scaling alone is unlikely to achieve human-level spatial intelligence.
Abstract
from arXiv · showhide
Despite remarkable progress, multimodal foundation models still exhibit surprising deficiencies in spatial intelligence. In this work, we explore scaling up multimodal foundation models to cultivate spatial intelligence within the SenseNova-SI family, built upon established multimodal foundations including visual understanding models (i.e., Qwen3-VL and InternVL3) and unified understanding and generation models (i.e., Bagel). We take a principled approach to constructing high-performing and robust spatial intelligence by systematically curating SenseNova-SI-8M: eight million diverse data samples under a rigorous taxonomy of spatial capabilities. SenseNova-SI demonstrates unprecedented performance across a broad range of spatial intelligence benchmarks: 68.8% on VSI-Bench, 43.3% on MMSI, 85.7% on MindCube, 54.7% on ViewSpatial, 47.7% on SITE, 63.9% on BLINK, 55.5% on 3DSR, and 72.0% on EmbSpatial, while maintaining strong general multimodal understanding (e.g., 84.9% on MMBench-En). More importantly, we analyze the impact of data scaling, discuss early signs of emergent generalization capabilities enabled by diverse data training, analyze the risk of overfitting and language shortcuts, present a preliminary study on spatial chain-of-thought reasoning, and validate the potential downstream application. All newly trained multimodal foundation models are publicly released.
1 Introduction
Multimodal foundation models still struggle with spatial intelligence because spatially grounded data is scarce, fragmented, and imbalanced. SenseNova-SI addresses this through taxonomy-guided data scaling and reports strong spatial benchmark performance while retaining general multimodal abilities.
- Spatial intelligence remains difficult for advanced multimodal models despite being fundamental to perceiving and interacting with three-dimensional space.
- SenseNova-SI scales diverse spatial data across multimodal foundations without changing their original architectures.Its taxonomy-guided SenseNova-SI-8M collection spans five key spatial domains.
- 68.8% on VSI-Bench, 43.3% on MMSI, 85.7% on MindCube, 54.7% on ViewSpatial, 47.7% on SITE, 63.9% on BLINK, 55.5% on 3DSR, and 72.0% on EmbSpatial were achieved while MMBench-En reached 84.9%.
- The study analyzes scaling laws, emergent generalization, overfitting and language shortcuts, spatial chain-of-thought, and downstream robotic manipulation.
- Controlled experiments and circular test designs validate that the models acquire spatial capabilities rather than relying on memorization, annotation bias, or unintended shortcuts.
- The SenseNova-SI series is presented as a strong, robust baseline for future spatial-intelligence research.
2 Related Works
Prior work enhances spatial intelligence mainly through 3D experts or spatial-specific datasets. The paper motivates improving widely used open-source multimodal models through scalable spatial training.
- GPT-5 has strong planar reasoning but still lags humans in spatial intelligence, while open-source and closed-source performance gaps are relatively small.
- Existing approaches primarily add 3D expert encoders or integrate geometry and camera-view information into multimodal models.
3 Data
The dataset targets spatial-data scarcity and imbalance by organizing capabilities under a principled taxonomy and expanding coverage beyond dominant metric and relation tasks. It combines existing resources with newly generated, scene-diverse data.
- High-quality, diverse spatial data is scarce, so the paper expands coverage toward holistic spatial intelligence rather than merely increasing volume.
- Task Taxonomy: The taxonomy covers five capabilities: Metric Measurement, Spatial Relations, Perspective-Taking, Mental Reconstruction, and Comprehensive Reasoning.
- Task Taxonomy: Metric Measurement includes estimating camera-to-object and inter-object distances, plus object and scene sizes across scales.
- Task Taxonomy: Spatial Relations models 3D-coordinate reasoning through front–back, left–right, up–down, near–far, and relative-scale comparisons.
- Task Taxonomy: Mental Reconstruction infers three-dimensional object structure from limited two-dimensional observations by identifying which object side is visible.
- Task Taxonomy: Perspective-Taking progresses from view correspondence to camera-motion reasoning and allocentric transformation, encouraging representations of viewpoint-dependent changes.
- Task Taxonomy: Comprehensive Reasoning coordinates multiple spatial capabilities with extended memory and multistep reasoning, but existing datasets are reused as a lightweight complement because such data is scarce.
- The collection combines general QA, community spatial datasets, and 4.5M newly generated pairs to address underrepresented perspective-taking, mental-reconstruction, and viewpoint-transformation tasks.The resulting corpus totals 8.5M QA pairs according to the data-construction passage.
4 Training
The study trains three multimodal foundation models, including visual-language and unified understanding-generation architectures, using a shared training scheme and evaluates them on spatial and general benchmarks.
- The study adopts Qwen3-VL, InternVL-3, and Bagel as its three multimodal foundation models.
- Qwen3-VL scales a language foundation to vision or audio, while InternVL-3 jointly trains vision and language from scratch.
- Evaluation uses the official EASI-8 protocol across key spatial-intelligence and general multimodal benchmarks.
- Bagel provides a unified understanding-and-generation architecture used to examine whether unified models can acquire strong spatial understanding.
- Each model trains for one epoch on the same dataset using 128 GPUs, batch size 2048, AdamW, and a 5×10−6 learning rate.
5 Experiments
Experiments show that scaling diverse spatial data improves performance across benchmarks, generalizes beyond training conditions, reduces shortcut reliance, and supports embodied manipulation, while gains eventually saturate and text CoT remains limited.
- Main Results: SenseNova-SI outperforms general and dedicated spatial-intelligence models across benchmarks, including proprietary GPT-5, and improves consistently across InternVL3, Qwen3-VL, and Bagel families.The authors attribute gains over comparable baselines partly to extensive perspective-taking data.
- Effectiveness: Data scaling steadily improves key spatial capability dimensions, with data mixing enabling a 2B model to surpass existing 7B spatial-intelligence baselines under comparable data budgets.Performance trajectories diverge by model size on perspective-taking tasks, where the 2B model may lack capacity for viewpoint transformations.
- Saturation: Performance gains gradually diminish as training data increases, and the authors state that data scaling alone is unlikely to achieve human-level spatial intelligence.The models are open-sourced so future work can build algorithmic advances on a spatially capable foundation.
- Saturation: Controlled single-dataset experiments show spatial understanding transfers across tasks and visual domains, suggesting possible meta-tasks underlying related spatial capabilities.Examples include transfer from ego-exo association to imagined first-person perspectives and from camera rotation to distinct cross-view questions.
- Extrapolation: SenseNova-SI generalizes from at most 16 training frames to 32 or more inference frames and remains comparable to Cambrian-S despite using fewer frames.It does not continue extrapolating beyond 64 frames, unlike Cambrian-S trained with longer context windows.
- Overfit and Shortcut Analysis: SenseNova-SI relies less on language shortcuts: it drops from 85.6 to 52.5 without MindCube images and loses 10 points under hard circular testing.MindCube-RawQA-SFT drops nearly 30 points in the hard circular test and scores 50.7 without images, nearly matching its visual-input score.
- Spatial Chain-of-Thought: CoT variants produce limited absolute gains, while reinforcement learning provides no clear improvement over a strong baseline for spatial reasoning.Long spatial CoT is prone to inconsistency and internal mistakes, adding computational overhead without reliable benefits.
- Downstream Task: SenseNova-SI substantially improves robot manipulation success under both official and spatial-intelligence-oriented prompts without finetuning.Representative rollouts show integration of language and visual spatial information into coherent trajectories and action sequences.
6 Conclusion
The study scales diverse spatial data across multimodal foundation models and reports consistent gains across spatial capabilities while retaining general multimodal abilities. It also provides a unified data pipeline that converts heterogeneous sources into spatial annotations and QA/CoT labels.
- Scaling spatial data yields consistent performance gains across a wide range of spatial intelligence benchmarks.
- SenseNova-SI improves generalization, including abilities that do not emerge without large-scale, diverse training data.
- The evaluation spans Perspective-taking, Spatial Relations, Metric Measurement, and Comprehensive Reasoning.
- Radar charts normalize model scores so the best model maps to 1.0, the worst to 0.2, and axes range from 0.0 to 1.0.
- The unified pipeline converts diverse sources into reliable QA and Chain-of-Thought labels.
- Heterogeneous raw data are standardized into camera and object poses, visibility annotations, and semantic human–object interaction labels.
B.1.2 Dataset-specific Processing.
Dataset-specific processing standardizes 3D and multi-view evidence, filters objects and views for reliable associations, and constructs diverse QA and CoT data. The pipeline controls ambiguity, overlap, connectivity, difficulty, and textual or visual redundancy.
- Dataset-specific Processing: ScanNet and ScanNet++ provide 3D poses, object boxes, and point clouds used to derive 2D visibility and projected boxes.
- Dataset-specific Processing: SUN RGB-D and CA-1M orientations are standardized, inconsistent categories discarded, and hypothetical human–object interactions annotated.
- Dataset-specific Processing: MessyTable cross-view instance IDs are supplemented with VLM-generated descriptions of object appearance.
- Object Selection: Object selection retains recognizable objects with informative details while filtering weak geometry, ambiguous positions, and insufficient visible size or visibility ratio.
- Image Selection: Multi-view selection removes extreme camera poses and enforces connected image sets with bounded association scores to preserve challenge.
- Image Selection: Scan-based selection greedily maximizes scene point coverage while maintaining connectivity and difficulty constraints, with at least 16 frames ensured by insertion when needed.
- Quality Control: Quality control removes ambiguous object references and directions that cannot map uniquely to spatial sectors.
- Data Diversity: Balanced data construction varies equivalent wording and object combinations while limiting samples per intent and image set.
C Full Results of Single Dataset Training
Single-dataset training produces uneven benchmark strengths, showing that individual datasets cover different subsets of spatial reasoning. The results motivate mixed-data training, while capability-transfer experiments identify cross-capability synergy.
- Single-dataset training often performs strongly on some benchmarks while sacrificing performance on others.
- 64.0 MRA accuracy on VSI is achieved by a model trained solely on VSI-590K, but MMSI, MindCube, and SITE results drop noticeably.
- No single dataset provides comprehensive spatial intelligence coverage, making mixed-data training crucial for balanced models.
- Dataset balancing remains an open challenge because datasets bias models toward particular subsets of spatial reasoning skills.
- Training on capability-specific data reveals cross-capability synergy, with Perspective-taking potentially benefiting other capabilities.
E Impact of Scaling on Benchmarks
Scaling effects are examined both by core spatial capability and by benchmark. Across both views, performance improves steadily as more high-quality, diverse spatial data are introduced.
- Model performance improves steadily as more training data is introduced across core spatial capabilities and benchmarks.
- The scaling analysis attributes the observed gains to high-quality, diverse spatial data addressing key spatial-intelligence knowledge deficiencies.
- Table 8 compares scaling on key benchmarks between the 0M InternVL3-8B base model and 8M SenseNova-SI InternVL3-8B.
F Retention of General Capabilities
Continued training on spatial-intelligence data largely preserves SenseNova-SI’s general multimodal capabilities, with some variants slightly improving on MMBench-En. The evaluation covers broad understanding, diagram reasoning, OCR, documents, perception, grounding, and related benchmarks.
- General capability retention: Minimal performance drops on MMBench-En were observed for SenseNova-SI Qwen3-VL-8B and InternVL3-2B relative to their base models.The comparison evaluates retention after continued training on SenseNova-SI-8M.
- General capability retention: SenseNova-SI Bagel-7B-MoT and InternVL3-8B showed slight MMBench-En improvements over their respective base models.
- General capability retention: Only marginal declines were observed across the remaining three general multimodal benchmarks.
- General capability retention: SenseNova-SI maintains competitive general visual understanding compared with other open-source spatial-intelligence models.Prior work suggests additional general visual-understanding data may further preserve or enhance this capability.
G Text-only Training
Text-only training produces substantially smaller gains than training with visual inputs, indicating that visual information is important for the measured spatial-intelligence improvements. The downstream case study uses a simulated robot setting with structured spatial inputs and executable gripper actions.
- Text-only training: Text-only training yields substantially smaller gains than training that retains visual inputs.The experiment removes all visual inputs and trains on text-only SSI-800K, described as 10% of SSI-8M.
- Text-only training: MMSI-Bench is highly resistant to language shortcuts, showing only minimal improvements under text-only training.
- Downstream manipulation: The downstream agent controls a simulated Franka Panda robot from language instructions and visual scene states without finetuning.The spatial manipulation evaluation uses EmbodiedBench’s spatial subset.
- Downstream manipulation: Figure 6 shows successful executions alongside failures caused by incorrect object recognition or limited manipulation precision.Incorrect output information is highlighted in red in the figure.
- Downstream manipulation: The official prompt represents each object by a 3D discrete position and requires a seven-dimensional gripper action containing position, orientation, and gripper state.Object coordinates use [X, Y, Z], while actions use [X, Y, Z, Roll, Pitch, Yaw, Gripper state].
- Downstream manipulation: The task output JSON contains visual-state description, reasoning and reflection, language plan, and executable plan fields.
I Detail Results on Key Benchmarks
Detailed benchmark tables break down SenseNova-SI performance across subsets and question types for eight spatial-intelligence benchmarks. The reported tables use benchmark-specific metrics and evaluation protocols.
- Benchmark coverage: Per-benchmark results cover VSI-Bench, MMSI-Bench, MindCube-Tiny, ViewSpatial-Bench, SITE, BLINK, 3DSRBench, and EmbSpatialBench.The appendix tables provide finer-grained results than the aggregated main-text scores.
- Benchmark metrics: MMSI-Bench reports accuracy scores and distinguishes positional relationships involving cameras, objects, and regions.
- Benchmark metrics: MindCube-Tiny, ViewSpatial-Bench, BLINK, 3DSRBench, and EmbSpatialBench report accuracy scores.The 3DSRBench results use the circ_eval setting.
- Benchmark metrics: SITE reports CAA scores and follows its original protocol, with multiple-choice questions answered by direct question answering.
J Case Study on Text-based CoT
The case study illustrates both successful and failed text-based spatial chain-of-thought reasoning. Although a structured CogMap can support accurate trajectories, local inconsistencies may accumulate and produce incorrect answers.
- Textual CoT does not appear to yield genuinely stronger 3D reasoning capabilities.
- A successful case maintains consistent and accurate reasoning through images before reaching the correct final answer.
- A failure case shows multiple local inconsistencies accumulating through the reasoning trajectory and ultimately producing an incorrect answer.
- The reasoning pipeline identifies queried objects, connects views through corresponding objects, and constructs a scene layout in a reference coordinate system.