PaperlayerView exact revision on arXiv

Source-linked AI summary

SpatialTree: How Spatial Abilities Branch Out in MLLMs

Yuxi Xiao, Longfei Li, Shen Yan, Xinhang Liu, Sida Peng, Yunchao Wei, Xiaowei Zhou, Bingyi Kang

arXiv:2512.20617v2cs.CV

TL;DR

Spatial abilities in MLLMs are difficult to study systematically because existing evaluations are fragmented and task-centric. SpatialTree organizes these abilities hierarchically and evaluates them through a capability-centric benchmark and training interventions. The results report both cross-level transfer and limitations of naive RL on low-level skills.

  • Problem

    Existing task-centric spatial benchmarks remain fragmented, making it difficult to study the structure and cross-level dependencies of spatial capabilities.

    Source: Mental Simulation · p. 2

  • Method

    SpatialTree organizes spatial abilities into a capability hierarchy, builds a benchmark from reorganized datasets and agentic data, and probes transfer using SFT and RL.

    Source: Mental Simulation · p. 3

  • Results

    B+Dist. increases Geom. abilities by +3.2 and Goal Exec. by +3.4, while decreasing Motion, Rel., and Local. by -2.0, -5.8, and -4.6, respectively.

    Source: §6.1 · p. 10

  • Takeaways & Limitations

    The framework offers a way to analyze how spatial abilities emerge, compose, and transfer, and to explore scaling them through strategically selected data.

    Source: §7 · p. 12

    When naive RL optimizes a single low-level ability, it can overfit its reward and yield siloed gains that fail to generalize or harm higher-level capabilities.

    Source: §6.2 · p. 11

Abstract

from arXiv · show

Cognitive science suggests that spatial ability develops progressively-from perception to reasoning and interaction. Yet in multimodal LLMs (MLLMs), this hierarchy remains poorly understood, as most studies focus on a narrow set of tasks. We introduce SpatialTree, a cognitive-science-inspired hierarchy that organizes spatial abilities into four levels: low-level perception (L1), mental mapping (L2), simulation (L3), and agentic competence (L4). Based on this taxonomy, we construct the first capability-centric hierarchical benchmark, thoroughly evaluating mainstream MLLMs across 27 sub-abilities. The evaluation results reveal a clear structure: L1 skills are largely orthogonal, whereas higher-level skills are strongly correlated, indicating increasing interdependency. Through targeted supervised fine-tuning, we uncover a surprising transfer dynamic-negative transfer within L1, but strong cross-level transfer from low- to high-level abilities with notable synergy. Finally, we explore how to improve the entire hierarchy. We find that naive RL that encourages extensive "thinking" is unreliable: it helps complex reasoning but hurts intuitive perception. We propose a simple auto-think strategy that suppresses unnecessary deliberation, enabling RL to consistently improve performance across all levels. By building SpatialTree, we provide a proof-of-concept framework for understanding and systematically scaling spatial abilities in MLLMs.

1 Introduction

SpatialTree frames spatial intelligence as a hierarchy, addressing fragmented task-centric evaluations and enabling analysis of how abilities relate and transfer. The paper uses benchmark construction and targeted SFT and RL interventions to examine those relationships and training trade-offs.

1 Introduction

  • Spatial abilities span perceiving, understanding, reasoning about, and interacting with 2D and 3D space, but existing evaluations remain fragmented and task-centric.Prior research moved from single-image tasks to 3D and then multi-view and video settings.Source: §1 · p. 1, Mental Simulation · p. 2

Mental Simulation

  • The hierarchy distinguishes native spatial perception, language-aligned mental mapping, internal spatial simulation, and spatial action by an agent.The four levels are L1 through L4, respectively.Source: Mental Simulation · p. 2
  • Foundational abilities function independently, while higher-level capabilities strongly rely on them; SFT experiments also identify foundational perception as a stepping stone to agentic competence.The paper reports intra-level interference alongside cross-level transfer and multi-ability synergy.Source: Mental Simulation · p. 3
  • Extensive reasoning helps complex tasks but can harm intuitive perception, motivating auto-think to suppress deliberation for immediate perception and encourage it for complex planning.The reported example of harmed perception is numerical estimation.Source: Mental Simulation · p. 3
  • SpatialTree organizes spatial abilities into a capability hierarchy and establishes a benchmark to evaluate mainstream MLLMs across a broad range of abilities.The benchmark combines reorganized prior datasets with additional data and task formulations, including agentic interactions.Source: Mental Simulation · p. 3

2 Related Work

Related work traces spatial intelligence from hierarchical cognitive theories to increasingly diverse MLLM benchmarks. SpatialTree organizes that landscape around capabilities and levels of spatial ability.

Spatial Cognitive Modeling

  • Classical accounts describe spatial abilities as hierarchical, from perception and sensorimotor interaction to reasoning and planning, motivating SpatialTree’s multi-level organization.The cited foundations include Piaget’s developmental progression, Tolman’s cognitive maps, and Kuipers’s hierarchical spatial representation.Source: §2 · p. 3

Benchmarks for Spatial Intelligence in MLLMs

  • Figure 2 contrasts relation tasks across levels: basic spatial relations in perception, object attributes and relationships in understanding, and visual-cue-based causal reasoning.The examples progress from L1 through L3.Source: Figure 2 · p. 4
  • MLLM benchmarks progressed from single-image spatial understanding to multi-frame and video reasoning, with later work adding geometry puzzles, dynamic reasoning, and inverse rendering.The cited examples include BLINK, SpatialEval, 3DSR-Bench, VSI-Bench, MMSI-Bench, Omnispatial, SITE, and IR3D-Bench.Source: §2 · p. 4

3 The SpatialTree Taxonomy

SpatialTree divides spatial ability into levels that progress from perception through language-aligned memory and internal simulation to agentic action. Each level emphasizes a distinct kind of spatial processing.

3 The SpatialTree Taxonomy

  • The taxonomy distinguishes lower-level intuitive perception from higher-level abilities requiring richer knowledge and more complex reasoning.The levels and their emphasis are illustrated through relation tasks in Figure 2.Source: §3 · p. 4

3.1 Perception

  • L1 covers primitive perception: geometry, motion, orientation, spatial relations, and localization, before linguistic abstraction.These include distance, size, shape, egocentric and allocentric motion, gravity and object orientation, topology and correspondence, detection and grounding.Source: §3.1 · pp. 4–5

Mental Mapping

  • L2 aligns spatial primitives with language and memory, covering scene understanding and the retention and retrieval of spatial information over time.Its abilities include spatial captioning, semantic relations, perspective taking, affordances, cognitive maps, and memory retrieval.Source: Mental Mapping · p. 5

3.3 Mental Simulation

  • L3 models spatial interactions and cause-effect chains, then converts causal insights into goal-directed plans expressed in language.Its two directions are causal reasoning and sequential planning.Source: §3.3 · pp. 5–6

3.4 Agentic Competence

  • L4 translates internal plans into executable action sequences in dynamic environments, integrating multimodal observations, memory, and action selection.The described settings include games, simulators, and the physical world; MLLMs use language as their interface to environments.Source: §3.4 · p. 6

4 Instantiating the SpatialTree Benchmark

SpatialTree-Bench combines reorganized datasets with newly curated data and level-specific QA construction. Its evaluation uses task-appropriate metrics and formats.

4.1 Data Curation

  • Level-specific data engines construct QAs, using expert perceptual representations, 3D-derived maps, reasoning templates, and action data for different abilities.The benchmark’s curation pipeline is shown in Figure 3.Source: §4.1 · p. 6, Figure 3 · p. 7
  • The benchmark reorganizes prior datasets by capability and adapts complex reasoning tasks to a hybrid multi-option and LLM-as-a-Judge evaluation format.The evaluation adaptation is described for CameraBench and MMSI-Bench.Source: §4.1 · p. 6
  • SpatialPlus fills remaining capability gaps, targeting underrepresented abilities such as L1 Orientation, L1 Shape, and L2 Spatial Caption, with emphasis on L4 agentic competence.Its annotations are generated from diverse video sources using SpatialEngine.Source: §4.1 · p. 7

4.2 Evaluation Metrics Designs

  • The benchmark uses metrics tailored to downstream tasks, including multiple-choice questions, numeric accuracy, LLM-as-a-Judge, and task-specific numeric measures.Multiple-choice questions account for 70.7%.Source: §4.2 · p. 7

5 A Hierarchical Analysis of Spatial Capabilities

The section compares MLLMs across SpatialTree-Bench and describes the models and level-specific metrics used for evaluation. Gemini 3 Flash leads the benchmark, while Qwen3VL-235B leads among open-source models.

5 A Hierarchical Analysis of Spatial Capabilities

  • Across 41 benchmark tasks, multiple-choice questions account for 70.7% of evaluation metrics, alongside numeric measures and LLM-as-a-Judge protocols.Task-specific numeric measures include cognitive map accuracy.Source: Figure 4 · p. 8

5.1 Models and Metrics

  • The evaluation spans thinking, non-thinking, and open-source MLLMs, with model scales ranging from 32B to 72B parameters.Examples include Gemini 3 Pro, GPT-4o, Qwen3-VL, and Kimi-VL.Source: §5.1 · p. 8
  • Metrics vary by ability level: L1–L2 use accuracy-based measures, while L3–L4 use task success rates, with L4 also assessed on action errors.L1–L2 examples include classification accuracy, MSE, and angular difference; L4 action measures include positional and orientation error.Source: §5.1 · p. 8

5.2 Overall Performance

  • Gemini 3 Flash achieves 57.8, while Qwen3VL-235B leads open-source models with 40.0 on SpatialTree-Bench.The reported overall benchmark results are summarized in Table 1.Source: §5.2 · p. 8

6 Exploring Ability Dependencies and Hierarchical Transfer

Correlation analysis finds largely independent L1 abilities but stronger links among L3/L4 capabilities. Targeted SFT and hierarchy-aware RL reveal distinct transfer patterns and gains across the hierarchy.

6 Exploring Ability Dependencies and Hierarchical Transfer

  • L1 abilities show weak correlations, whereas L3/L4 capabilities correlate more strongly, suggesting that complex tasks rely on overlapping foundational sub-skills.The analysis uses Pearson correlations computed from benchmark scores.Source: §6 · p. 9, Figure 5 · p. 9

6.1 Probing Cross-Ability Transfer via SFT

  • Blended Distance, Size, and Correspondence SFT gains +1.1 overall, exceeding individual contributions and reversing L1.Motion’s best individual change of -2.0 to +0.7.Individual overall changes are B+Dist. -0.5, B+Corr. +0.2, and B+Size. -1.5 relative to baseline.Source: Table 2 · p. 10, §6.1 · p. 10
  • Single-ability SFT raises Geom. abilities by +3.2 but lowers Motion, Rel., and Local. by -2.0, -5.8, and -4.6; Underst. and Goal Exec. rise by +2.0 and +3.4.These are the reported changes for B+Dist.Source: §6.1 · p. 10

6.2 Reinforcement Learning

  • The hierarchy-aware strategy outperforms the baseline and naive GRPO variants across SpatialTree-Bench.Table 3 compares the baseline Qwen2.5-VL-7B with a version enhanced by RL on Goal-Exec. tasks.Source: Table 3 · p. 11, §6.2 · p. 12
  • Naive RL on a single low-level ability tends to overfit its reward, producing siloed gains with negligible or detrimental transfer to higher-level capabilities.The reported issue is particularly associated with generalization in low-level skills.Source: §6.2 · p. 11
  • Uniformly mixing datasets across levels yields only marginal gains, as the model struggles to balance the benchmark’s diverse requirements.The authors conclude that a one-size-fits-all RL strategy cannot effectively span perception through agentic planning.Source: §6.2 · p. 12
  • The hierarchy-aware reward removes thinking rewards and penalizes length for intuitive perception, while retaining and amplifying reasoning rewards for complex tasks.The strategy is motivated by the contrast between direct visual perception and reasoning-intensive tasks.Source: §6.2 · p. 12
  • GRPO-tuned Qwen2.5-VL-7B improves across multiple hierarchy levels despite a domain gap and a mismatch between training and evaluation tasks and metrics.Robotic-arm training samples are separate from benchmark test scenes and object configurations; training maximizes discrete MCQ reward while evaluation uses diverse metrics.Source: §6.2 · p. 12

7 Conclusion and Future works

SpatialTree organizes spatial intelligence into four hierarchical layers and offers a framework for studying how abilities emerge, compose, and transfer. The authors propose using this structure to guide scaling with different types of data.

7 Conclusion and Future works

  • SpatialTree organizes spatial intelligence into four hierarchical layers, providing a framework for analyzing how abilities emerge, compose, and transfer.The authors suggest leveraging different data types to identify abilities useful for pre-training, reinforcement learning, or real-world interaction.Source: §7 · p. 12
  • The authors frame SpatialTree as a promising path toward advancing spatial intelligence in MLLMs through strategic use of different types of data.They suggest identifying abilities suited to pre-training, reinforcement learning with minimal additional reasoning data, or acquisition through real-world interactions.Source: §7 · p. 12

Appendix

  • The benchmark is constructed by reorganizing existing datasets and mapping them to the capability tree, with SpatialPlus added to ensure capability coverage.Source: Appendix · p. 18

A Visualization of Data Sources

Figure A shows how different datasets contribute to the SpatialTree evaluation.

A Visualization of Data Sources

B Evaluation Metrics Details

SpatialTree uses evaluation formats matched to different kinds of spatial tasks, from multiple-choice and numeric answers to open-ended judgments and interactive agent tasks. Its measures include accuracy, relative error, GPT-based assessment, and embodied-task scores.

B Evaluation Metrics Details

  • Multi-option QA is evaluated by accuracy against the ground-truth choice, covering spatial relations, object properties, and causal dynamics in L1–L3.Source: B Evaluation Metrics Details · p. 18
  • Numeric QA predicts continuous quantities such as distances, angles, or 3D coordinates and is evaluated with relative error, which scales predictions across magnitudes and emphasizes precision.Source: B Evaluation Metrics Details · p. 18
  • Figure B contrasts gravity-field annotations estimated by GeoCalib with those from OrientAnything.Source: B Evaluation Metrics Details · p. 19
  • A GPT-based judge assesses open-ended or complex responses, optionally assigning partial correctness, especially for mid- and high-level L3–L4 capabilities.Source: B Evaluation Metrics Details · p. 19
  • Interactive agent evaluation measures navigation and manipulation through goal success, relative translation accuracy, and directional alignment, aggregating per-step scores into a long-horizon task measure.Each action-step score combines relative distance and cosine similarity of movement vectors and is normalized to [0, 1].Source: B Evaluation Metrics Details · p. 19

C SpatialPlus: Complementary Data Annotations for SpatialTree

The SpatialPlus orientation annotations cover gravity direction and 3D object orientation, using separate datasets and annotation tools for each task. Gravity evaluation uses uncertainty-aware similarity scores for predicted parameters.

C.1 Orientations (L1)

  • Orientation annotations cover gravity direction and 3D object orientation, using GeoCalib for gravity estimates and OrientAnything for object poses.Gravity annotations use 500 images sampled from TartanAir, while object orientations use Co3dv2.Source: §C.1 · p. 19
  • For gravity evaluation, the model returns pitch, roll, and vFOV in structured JSON; pitch and roll describe forward–backward and sideways camera tilt.Source: §C.1 · pp. 19–20
  • Figure C provides the prompt template for orientation estimation.Source: Figure C · p. 20
  • A Gaussian-kernel similarity score evaluates each predicted parameter against Geocalib ground truth, with slower score decay when annotation uncertainty is larger.The task score averages the pitch, roll, and vFOV scores; object orientation is evaluated on azimuth, polar, and rotation angles.Source: §C.1 · p. 20

C.2 Agentic Competence (L4)

Spatial Action Mapping defines tractable action spaces for navigation and manipulation, supporting evaluation of MLLMs’ planning and execution in 3D. Explicit low-level visual prompts also improve Gemini2.5-pro’s goal-driven navigation accuracy.

C.2 Agentic Competence (L4)

  • Navigation maps camera motion to six motion primitives, translating high-level instructions into a structured low-level action space.Camera trajectories are discretized into navigation actions using a speed threshold.Source: §C.2 · p. 21
  • Manipulation uses gripper open/close and wrist-level 6-DoF actions, or push, pull, and grab gestures, to create a unified, tractable action space.The benchmark draws annotated data from human-hand videos, navigation games, robotic-arm datasets, and simulations.Source: §C.2 · p. 21
  • Navigation accuracy combines relative distance and directional alignment into a normalized [0, 1] score, with action sequences capped at 10 steps.For manipulation, evaluation combines translational accuracy, normalized rotational accuracy, and binary gripper-state accuracy.Source: §C.2 · pp. 22, 24

D Ability Transfer via Prompting

  • 7.1% for correspondence, 5.5% for distance, and 2.1% for size prompting improve Gemini2.5-pro’s goal-driven navigation accuracy.The prompts derive from matching visualizations, depth, and size context.Source: D Ability Transfer via Prompting · p. 24

E Benchmark Metric Aggregation

SpatialTree’s overall score is computed by recursively aggregating weighted child scores, with weights informed by the cognitive hierarchy and observed dependencies among abilities.

E Benchmark Metric Aggregation

  • Higher weights go to abilities with strong, widespread correlations, refining the theoretical hierarchy with observed model-performance dependencies.The weighting scheme also prioritizes L1 perception as foundational for L2 mental mapping and L3 mental simulation.Source: E Benchmark Metric Aggregation · p. 25
  • The final spatial-intelligence score is a bottom-up weighted sum, recursively combining each node’s immediate children up to the root.This yields one score for each MLLM.Source: E Benchmark Metric Aggregation · p. 25

F More Visualizations for QAs in SpatialTree Bench

A B C

  • A cube-stack task asks which sequence of block moves transforms an initial arrangement into a target arrangement under specified movement and gravity rules.Source: A B C · p. 32

L4 Agentic Competence

  • L4 examples ask an agent to select actions across video frames and judge task completion, or identify which image shows the most progress.Source: L4 Agentic Competence · p. 33

L4 Agentic Competence (Continued)

  • A continued L4 example requires navigating between visual states with elementary actions and step counts, output as matching JSON arrays.Translation and rotation step sizes are specified.Source: L4 Agentic Competence (Continued) · p. 34
Loading 2512.20617v2…