Source-linked AI summary
SPACENUM: Revisiting Spatial Numerical Understanding in VLMs
Jianshu Zhang, Yijiang Li, Huifeixin Chen, Haoran Lu, Letian Xue, Bingyang Wang, Han Liu
TL;DR
It remains unclear whether VLMs genuinely ground numerical outputs in spatial perception. SpaceNum evaluates this through bidirectional tasks spanning dynamic transitions and static layouts, finding that models largely fail and often perform near random guess.
Problem
It remains unclear whether VLMs understand numbers as metric quantities grounded in spatial properties across spatial settings.
Method
SpaceNum evaluates bidirectional numerical-spatial mapping across dynamic transitions and static layouts using controlled simulator-generated data.
Results
Across both settings, VLMs largely fail to ground numerical values, performing near random guess and relying on shallow spatial cues rather than stable coordinate-aware representations.
Takeaways & Limitations
Spatial numerical understanding remains substantially limited, while tuning partially improves performance and transfers to related spatial reasoning benchmarks.
Takeaways & Limitations
The study focuses mainly on controlled spatial settings with discrete candidate-based evaluation and simulated environments, leaving open-ended real-world and continuous prediction settings for future work.
Abstract
from arXiv · showhide
Vision-Language Models (VLMs) are increasingly deployed in embodied environments, where they need produce numerical outputs such as action magnitudes and spatial coordinates. Although these numbers appear meaningful, it remains unclear whether these numerical outputs are genuinely grounded in spatial perception. Therefore, in this work, we revisit spatial numerical understanding through SpaceNum, a unified framework that captures two complementary settings: numbers as dynamic transitions during spatial exploration, and numbers as static layouts in spatial reasoning. We formulate two bidirectional tasks, Num2Space and Space2Num, to evaluate how well VLMs map between vision-side spatial structure and language-side numerical representations. We systematically study whether current VLMs truly understand numerical values in spatial settings. Across dynamic transitions and static layouts, we find that models largely fail to ground numbers in spatial meaning and often perform close to random guess. Through error analysis, reasoning trace analysis, and controlled interventions, we show that current VLMs rely heavily on shallow spatial cues, struggle to build stable coordinate-aware representations, and fail to abstract structured spatial layouts from visual observations. We further show that explicit reasoning provides only marginal gains, while tuning can partially improve spatial numerical understanding and transfer to external spatial reasoning benchmarks.
1 Introduction
SPACENUM examines whether VLMs ground numerical values in spatial meaning across dynamic transitions and static layouts. Using bidirectional Num2Space and Space2Num tasks, it finds that current models largely fail and often perform near random guessing.
- Two spatial settings: SPACENUM studies numbers as dynamic transition magnitudes in spatial exploration and as static relative layouts in spatial understanding.In exploration, values specify state-change magnitudes; in static layouts, numbers encode relative relationships among multiple objects.
- Bidirectional evaluation: Num2Space maps language-side numerical values to visual outcomes, while Space2Num infers numerical values from spatial configurations.Together, the tasks evaluate bidirectional mapping between vision-side spatial structure and language-side numbers.
- Main finding: 18 VLMs largely fail to ground numerical values in spatial meaning across dynamic transitions and static layouts, often performing close to random guess.The evaluation covers both spatial exploration and spatial understanding scenarios.
- Error patterns: The study further reveals strong asymmetries between vision-to-number and number-to-vision grounding across scenarios and mapping directions.These differences emerge from analysis of how failures vary across the two spatial settings and task directions.
2 SpaceNum Data Curation
SpaceNum curates controllable dynamic-transition and static-layout data through simulator-based pipelines, with validity and spatial-reference controls. It defines bidirectional Num2Space and Space2Num tasks and provides 3,800 benchmark samples plus 77,412 training samples.
- Data sources: Dynamic-transition data is generated in AI2-THOR, while static-layout scenes use NVIDIA Isaac Sim with BlenderKit assets and ground-truth spatial annotations.The two pipelines support embodied actions and controlled cognitive-map construction, respectively.
- Dynamic-transition curation: Dynamic data controls action coverage, transition continuity, visual anchoring, and validity to produce informative, non-ambiguous samples.Frames with fewer than 3 object instances are discarded, and occupancy maps constrain valid initial and post-action states.
- Dynamic-transition tasks: Dynamic Num2Space selects the resulting observation from (o_t, a, n), whereas Space2Num infers n from (o_t, o_t+1, a).Distractors vary the numerical value while preserving the initial observation and action type.
- Static-layout curation: Static-layout scenes establish an unambiguous coordinate frame using two fixed anchor objects, with one defining the origin and their relative position defining direction.The coordinate frame is fixed up to scale and remains consistent across samples in a scene.
- Static-layout tasks: Static Num2Space selects an observation consistent with cognitive map M, while Space2Num infers target coordinates p from the layout observation.Static-layout distractors vary object positions or sizes while preserving the same reference frame.
- Dataset scale: 3,800 samples comprise the benchmark, supplemented by 77,412 automatically generated training samples for later training-based explorations.The larger training-set breakdown is reported separately in Figure 2.
3 Experiments
Experiments show that current VLMs largely fail to ground numerical values in spatial meaning, performing near random and relying on shallow cues rather than stable coordinate-aware representations. The failures span dynamic transitions and static layouts, while reasoning offers marginal gains and targeted tuning partially improves performance and transfer.
- Overall performance: 30.0% random-baseline performance and a 39.8% best-model average show that current VLMs struggle to understand numerical values in spatial settings.Several models fall below the random baseline, indicating shallow spatial-number correlations rather than grounded numerical meaning.
- Scenario differences: Dynamic transitions remain difficult across action types, with strong models reaching only around 40.0%, while static layouts follow a different difficulty structure.Dynamic performance is about 10 points above the 30.0% random baseline, and models show little action preference or specialization.
- Mapping directions: SPACE2NUM consistently outperforms NUM2SPACE for dynamic transitions and nearly every action type, whereas static layouts reverse this directional preference.Models are better at grounding numbers from observed visual changes than predicting future visual outcomes from numerical actions.
- Error analysis: Static-layout errors favor joint position-and-size mistakes, showing that failures across spatial attributes are strongly coupled.Once models fail to establish a coherent layout, errors tend to propagate jointly across position and size.
- Reasoning analysis: Reasoning changes SPACENUM performance by typically within 1%, while traces reveal premature reliance on coarse cues, image-space priors, and inadequate motion-magnitude comparison.Models may stop after noticing a plausible cue, map image-left directly to a smaller x value, or fail to reason counterfactually about whether a change supports a candidate magnitude.
- Representation probes: Symmetric transformations cause substantial performance drops, while structured visual abstractions substantially improve SPACE2NUM more than NUM2SPACE, indicating unstable numerical representations and weak spatial-structure extraction.Static layouts can also be partially solved through language-side priors or shortcut patterns without fully grounding the visual scene; numerical reformulation provides only limited gains for larger transition models, and layout reasoning remains largely unchanged.
- Tuning: 25% transition data and 75% layout data yield the best overall tuning performance, while dimension-specific tuning improves the matching dimension most and RL provides limited gains.Increasing training data under the same mixture improves performance, and graded rewards perform slightly better than strict exact-match rewards.
4 Related Works
Prior work evaluates VLM spatial reasoning in dynamic embodied environments and structured visual layouts, including spatial relations, metric structure, and 3D configurations. SPACENUM instead directly examines whether numerical values are grounded as spatial quantities across dynamic transitions and static layouts.
- Spatial reasoning in dynamic and embodied environments: Prior benchmarks assess dynamic spatial reasoning through action consequences, object movement, perspective taking, spatial aiming, camera motion, and interaction-centered scenarios.SAT and OmniSpatial represent complementary evaluations of spatial changes caused by actions, motion, and embodied interactions.
- Spatial understanding and structured spatial reasoning: Other benchmarks test spatial relations, metric structure, and 3D layouts, finding that VLMs often struggle with spatial prepositions despite strong object recognition.These evaluations include relations such as left/right, above/below, and object-centric configurations.
- SPACENUM's distinction: SPACENUM directly studies whether VLMs ground numerical values as meaningful spatial quantities across dynamic transitions and static layouts.It also analyzes asymmetry, failure patterns, reasoning behaviors, and tuning characteristics of spatial numerical grounding.
5 Conclusion
SPACENUM shows that current VLMs largely fail to ground numerical values in spatial meaning, relying on shallow cues rather than stable spatial reasoning. Future work should extend evaluation to realistic continuous settings and clarify the models’ internal spatial reasoning mechanisms.
- Conclusion: SPACENUM experiments show that current VLMs largely fail to ground numbers in spatial meaning and rely on shallow spatial cues instead of stable spatial reasoning.The benchmark covers both dynamic transitions and static layouts.
- Limitations and future work: The study is limited to controlled spatial settings with discrete candidate-based evaluation and simulated environments.
- Limitations and future work: Future work should examine open-ended real-world scenes, embodied interactions, and continuous spatial prediction settings.
- Limitations and future work: How VLMs internally perform spatial reasoning remains largely unexplored because severe attention collapse limits conclusions from preliminary attention-based analyses.