Defining and Evaluating Visual Language Models' Basic Spatial Abilities: A Perspective from Psychometrics

• Spatial intelligence is organized into five Basic Spatial Abilities: Spatial Perception, Spatial Relation, Spatial Orientation, Mental Rotation, and Spatial Visualization.Source: §1 · p. 1
• Existing VLM spatial evaluations lack a comprehensive theoretical framework and human benchmarks, limiting cross-study comparison and assessment of the VLM–human gap.Source: §1 · p. 1
• The proposed psychometric framework uses standardized human spatial tests to evaluate VLMs’ Basic Spatial Abilities and establish benchmarks.Source: §1 · p. 1

2.1 Psychometric Studies on Human Spatial Intelligence

• Psychometric accounts place Basic Spatial Abilities beneath domain-specific spatial intelligence and use them to decompose it into measurable subskills.Source: §2.1 · p. 2
• This study focuses on that BSA level, adopting established human experimental paradigms to evaluate VLMs.Source: §2.1 · p. 2

2.2 Evaluation of LLMs and VLMs’ Spatial Abilities

• Prior VLM evaluations largely target specialized 3D tasks, while text-based LLM studies assess abstract spatial relations through verbal descriptions.Source: §2.2 · p. 2
• Existing studies lack theory-grounded comparisons, often omit Mental Rotation and Spatial Visualization, and provide neither hierarchical BSA mapping nor human baselines.Source: §2.2 · p. 2
• The paper evaluates all five BSAs with standardized human experiments, enabling comparisons across models and between humans and AI.Source: §2.2 · p. 2

3.1 Definition and Composition of Basic Spatial Abilities

• Nine classic psychometric tests cover five Basic Spatial Abilities, with existing human experiment results used as benchmarks.Source: §3.1 · p. 3

3.2 Tests for Basic Spatial Abilities

• The selected tests were screened for unambiguous BSA coverage, complete questions and answers, and reproducible human results; the researchers retained classic tests rather than arbitrarily balancing ability weights.Source: §3.2 · p. 3

3.2.1 Spatial Perception Tests

• The MGMP Spatial Visualization Test uses square-cube transformations to assess imagined 2D and 3D views, cube alterations, and cube counts.Source: §3.2.1 · p. 3

3.2.3 Spatial Orientation Tests

• The Money Road-Map Test assesses allocentric-to-egocentric right/left discrimination along a map path with 32 turns.Source: §3.2.3 · p. 4

3.2.4 Mental Rotation Tests

• Mental rotation is tested with cube-figure rotations and matching objects under a demonstrated rotation pattern.Source: §3.2.4 · p. 4

3.2.5 Spatial Visualization Tests

• Spatial visualization tests ask participants to imagine cross-sections of cut solids or determine whether one Rubik’s cube can be rotated into another.Source: §3.2.5 · p. 4

4.1 Settings

• The study tested 13 mainstream open-source and commercial VLMs, feeding each 312 questions and averaging scores across five abilities for the overall score.Each BSA score averages tests under that ability; the overall score averages the five BSA scores.Source: §4.1 · pp. 4–5

4.2 Results

• Table 3 reports model scores by test, with each test repeated three times and standard deviations summarizing performance stability.Its final rows give average scores and standard deviations across models alongside human experiment results.Source: §4.2 · p. 5

4.2.1 Human vs VLM

• VLMs averaged 24.95 overall versus 68.38 for humans, and both groups ranked spatial orientation highest and mental rotation lowest.Across the 13 models, overall scores ranged from 16.31 to 31.22.Source: §4.2.1 · p. 5, Figure 4 · p. 6

4.2.2 VLM vs VLM

• Spatial relation and spatial visualization vary more across models than spatial perception and mental rotation, while spatial orientation is relatively strong for most models.Three models fail completely on spatial orientation; Qwen2-VL-7B and GPT-4o are among those with relatively strong overall results.Source: §4.2.2 · p. 6

4.2.3 Correlation of Basic Spatial Abilities

• Pearson correlations between any two abilities are below 0.4, indicating weak or no association and supporting the treatment of the five abilities as basic.The authors infer independence from the lack of correlation because the events satisfy a two-dimensional normal distribution.Source: Figure 5 · p. 6, §4.2.3 · p. 6

4.3.1 VLMs’ Overall Performance

• VLMs remain significantly below human benchmarks across all dimensions, while ranking highest in spatial orientation and lowest in mental rotation.The authors relate stronger 2D coordinate reasoning and weaker dynamic 3D manipulation to this gradient, and suggest basic 2D reasoning may scaffold advanced 3D capabilities.Source: §4.3.1 · p. 7

4.3.2 Impact of VLM Manufacturer and Size

• Qwen leads overall at 30.82, while InternVL2 trails at 19.6; individual ability profiles also diverge from aggregate scores.Gemini-1.5-flash peaks in spatial relation at 43.78 but scores 16.67 in mental rotation and has no spatial orientation capability; Llama performs strongly in orientation but weakly in visualization.Source: §4.3.2 · p. 7
• Parameter count showed no positive correlation with BSA performance, and smaller models often outperformed larger counterparts, particularly in the Qwen and Llama series.Around 10B-parameter models in those series surpassed larger variants; Gemini-1.5-flash-8B matched Gemini-1.5-pro, with similar anomalies in InternVL2.Source: §4.3.2 · p. 7

4.3.4 Constraints in VLMs’ Basic Spatial Abilities

• The authors identify weak metric geometry encoding, shallow reasoning without dynamic 3D simulation, poor cross-modal grounding, and overreliance on pretraining patterns as constraints on VLM spatial abilities.They attribute these limitations partly to architectural deficiencies and suggest hybrid architectures with geometric priors and neurosymbolic reasoning may help bridge the gap.Source: §4.3.4 · p. 8

4.3.3 Impact of CoT and Example-Based Training

• CoT alone raised baseline accuracy by 0.100, one-shot added 0.017, and example gains reached 0.159 at 1 shot and 0.209 at 3 shots before plateauing beyond 5 examples.The five-step CoT protocol covered understanding the 3D shape, analyzing the plane, determining the cross-section, matching it to options, and answering.Source: §4.3.3 · pp. 7–8, Figure 8 · p. 8

5 Conclusion

• VLMs average 24.95 versus humans’ 68.38, mirror human ability hierarchies, excel in 2D orientation, and struggle with 3D mental rotation.The study reports statistical independence among the BSAs.Source: §5 · p. 8
• Compact models such as Qwen2-VL-7B and Gemini-1.5-flash-8B frequently outperform larger counterparts, suggesting spatial reasoning depends more on architectural optimization than parameter size.Qwen-series models show superior cross-BSA integration, while InternVL2’s fragmented performance suggests training-data biases.Source: §5 · p. 8
• CoT prompting improves baseline accuracy by 0.100 and 5-shot training by 0.259, but both approaches plateau amid limits in metric encoding and dynamic 3D simulation.The study describes these limits as architectural constraints beyond training-data limitations.Source: §5 · p. 8
• The identified barriers include weak geometric priors, absent dynamic simulation, disjointed cross-modal grounding, and over-reliance on static pre-training patterns.The authors suggest hybrid architectures combining neurosymbolic reasoning and biologically inspired spatial processing may be needed.Source: §5 · p. 9
• The study links psychometric BSAs quantitatively to VLM capabilities and presents a diagnostic toolkit and roadmap for advancing embodied AI systems.It calls for collaboration between machine learning and cognitive science communities.Source: §5 · p. 9

Ethics Statement

• The study used OpenAl, DeepInfra, Google, and SiliconFlow API services in compliance with their terms of service and usage policies.The passage spells the provider name “OpenAl.”Source: Ethics Statement · p. 9

A Data Statistics

• The original VLM BSA evaluation statistics are provided in Table 3 as a complement to Section 4.2.The passage describes Table 3 as listing results for 13 VLMs and human recipients.Source: A Data Statistics · p. 12, Table 3 · p. 12
• Table 4 covers VLM performance in the Santa Barbara Solids Test with and without explicit answer-choice separation.Source: Table 4 · p. 12

B Prompts for VLMs

• Prompts are provided for all nine BSA tests used to obtain results from VLMs.Source: B Prompts for VLMs · p. 13

C CoT Method and Example-based Training Demonstration

• The experiments used GPT-4o and the SBST test to assess CoT and example-based training for spatial problem-solving.Figure 9 presents the CoT method and example-based training framework.Source: C CoT Method and Example-based… · p. 16, Figure 9 · p. 16
• The first experiment compared direct answering, example-based training, CoT, and their combination, using a single example question.The CoT process covered understanding the 3D shape, analyzing the plane, determining the cross-section, matching it to options, and answering.Source: C CoT Method and Example-based… · p. 16
• The second experiment measured response accuracy with 1, 3, 5, or 10 example questions and no CoT.Both experiments were repeated multiple times and their results averaged; Figure 9 illustrates the framework.Source: C CoT Method and Example-based… · p. 16, Figure 9 · p. 16

D Sample Data Demonstration

• Qwen2-VL-7B, identified as the highest-scoring model, is shown responding to nine sample questions from different tests.Figure 10 shows data samples from tests for BSA2.Source: D Sample Data Demonstration · p. 17, Figure 10 · p. 17