Source-linked AI summary
Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models
Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Yixuan Li, Neel Joshi
TL;DR
Spatial understanding and reasoning remain under-explored in LLMs and VLMs despite their importance for human cognition and real-world intelligent systems. The paper introduces SpatialEval, a benchmark covering diverse spatial tasks, and evaluates competitive language and vision-language models. The results show that spatial reasoning is difficult, visual inputs often do not outperform language-only backbones, and multimodal models rely less on vision when sufficient textual clues are available.
Problem
Spatial understanding and reasoning are under-explored in LLMs and VLMs despite being fundamental to human cognition and everyday activities.
Method
SpatialEval benchmarks LLMs and VLMs across relationship, navigation, position-understanding, and object-counting tasks using image, text, or combined representations.
Results
Across evaluated models, spatial reasoning remains challenging; VLMs often fail to surpass LLMs with visual inputs, rely less on vision when textual clues suffice, and often perform better than LLMs with text-only inputs.
Takeaways & Limitations
The findings challenge the view that current VLMs perform strongly across vision-text tasks and identify visual-information handling as a target for improving spatial reasoning.
Takeaways & Limitations
The study is primarily empirical and focuses on analysis rather than developing new training strategies or adaptation algorithms.
Abstract
from arXiv · showhide
Large language models (LLMs) and vision-language models (VLMs) have demonstrated remarkable performance across a wide range of tasks and domains. Despite this promise, spatial understanding and reasoning -- a fundamental component of human cognition -- remains under-explored. We propose SpatialEval, a novel benchmark that covers diverse aspects of spatial reasoning such as relationship understanding, navigation, and counting. We conduct a comprehensive evaluation of competitive language and vision-language models. Our findings reveal several counter-intuitive insights that have been overlooked in the literature: (1) Spatial reasoning poses significant challenges where competitive models can fall behind random guessing; (2) Despite additional visual input, VLMs often under-perform compared to their LLM counterparts; (3) When both textual and visual information is available, multi-modal language models become less reliant on visual information if sufficient textual clues are provided. Additionally, we demonstrate that leveraging redundancy between vision and text can significantly enhance model performance. We hope our study will inform the development of multimodal models to improve spatial intelligence and further close the gap with human intelligence.
1 Introduction
Spatial reasoning is under-explored in LLMs and VLMs despite its importance for human cognition and real-world intelligent systems. SpatialEval benchmarks diverse spatial skills and reveals that current models often struggle, with visual inputs frequently failing to improve over language-only backbones.
- Motivation: Spatial understanding and reasoning remain under-explored in LLMs and VLMs despite their importance for navigation, pattern recognition, and spatial relationships.These abilities support everyday activities, survival, and higher-order cognition.
- Benchmark: SpatialEval evaluates LLMs and VLMs across relationship understanding, navigation, position understanding, and object counting.The benchmark contains four tasks: Spatial-Map, Maze-Nav, Spatial-Grid, and Spatial-Real.
- Evaluation design: Each benchmark problem provides both an image and a sufficient text representation, enabling vision-only, text-only, and combined vision-text evaluation.The paper denotes these settings VQA, TQA, and VTQA, respectively.
- Findings: Some competitive VLMs perform worse than random guessing on spatial reasoning tasks.This finding comes from a systematic evaluation of open-source and proprietary LLMs and VLMs.
- Findings: Without detailed textual descriptions, multimodal models rarely surpass their LLM backbones using visual inputs alone.When both modalities are available, multimodal models also tend to rely less on visual information when sufficient textual clues exist.
- Findings: VLMs often outperform their LLM counterparts with text-only inputs, suggesting benefits from multimodal training for their language backbones rather than comparable benefits from visual components.The authors surmise that overly simplistic visual-information handling contributes to VLM spatial-understanding limitations.
2 Dataset and Task Construction
SpatialEval constructs four spatial-reasoning tasks and evaluates them through text-only, vision-only, and combined vision-text representations across language and vision-language models.
- SpatialEval covers spatial relationships, navigation, position understanding, and object counting across four tasks.The tasks are Spatial-Map, Maze-Nav, Spatial-Grid, and Spatial-Real.
- Each benchmark problem has image and text representations sufficient to answer its spatial-understanding question.The modalities are TQA for text-only, VQA for vision-only, and VTQA for combined vision and text.
- Spatial-Map: Spatial-Map models arbitrary object locations using pairwise relations and questions about relationships and spatially constrained counts.The number of objects K is configurable.
- Maze-Nav: Maze-Nav represents routes with colored blocks or ASCII symbols and tests navigation from start S to exit E along the marked path.Movement is allowed in the four cardinal directions.
- Spatial-Grid: Spatial-Grid places object images in rigid grid cells, with textual rows as an alternative representation for spatial reasoning and counting.It contrasts structured grids with the arbitrary positioning of Spatial-Map.
- Spatial-Real: Spatial-Real extends evaluation to real images from Densely Captioned Images, whose captions average more than 1,000 words, with curated spatial questions.Questions address object counting, relations, and position understanding.
3 Main Results and Analysis
The evaluation finds that spatial reasoning remains difficult for competitive models, with vision-only VLMs often near random guessing and frequently weaker than matched LLMs.
- Most competitive multimodal models using vision-only inputs perform at levels similar to or barely above random guessing.The evaluation averages accuracy across questions and uses VQA for VLMs.
- For Spatial-Map and Maze-Nav, only a few models outperform random guessing by a notable margin.
- Text-only input generally does not significantly enhance competitive models’ spatial reasoning, despite containing essential spatial information.On Spatial-Grid, Llama-3 reaches 71.9% accuracy and Mistral-7B-Instruct reaches 62.1%, both surpassing random guessing.
- On Spatial-Map and Spatial-Grid, most VLMs perform worse than their LLM counterparts with the same language backbone.For Spatial-Grid, Mixtral-7B achieves 62.1% average accuracy while LLaVA-v1.6-Mistral-7B achieves 47.1%, a 15% decrease.
4 Delving Into Spatial Reasoning for Vision-Language Models
Controlled comparisons show that VLMs often benefit more from textual descriptions than from visual inputs in spatial reasoning. Across open-source and proprietary models, mismatched or absent images can improve performance, while multimodal fine-tuning strengthens text-only spatial reasoning.
- Seeing Without Understanding: Removing the original image improved VLM performance across architectures, including a 20.1% gain for LLaVA-1.6-34B on Spatial-Grid.The comparison retained the textual description and removed only the visual input.
- Seeing Without Understanding: Replacing the original image with Gaussian noise also improved accuracy, including a 6.5% increase for LLaVA-1.6-Vicuna-13B on Maze-Nav.The result suggests models often prioritize textual information when visual cues are irrelevant.
- Seeing Without Understanding: Replacing the original image with a random image improved Maze-Nav performance across diverse VLM architectures rather than consistently causing degradation.The random image could conflict with the textual description, yet the expected harm did not reliably occur.
- Redundancy Helps: Adding textual descriptions to the same visual inputs enhanced VLM accuracy compared with vision-only VQA across architectures.The comparison indicates that redundant textual information can improve spatial reasoning performance.
- Text-Only Comparison: Most VLMs outperformed their corresponding LLM backbones with text-only inputs, except CogVLM.The authors associate this pattern with enhanced spatial reasoning in the language-model backbones of VLMs.
5 Related Work
Prior work established strong language and vision-language model families but has comparatively under-covered detailed spatial reasoning. Existing spatial benchmarks often use simpler settings, whereas this work emphasizes diverse and more complex spatial tasks.
- Large Language Models: LLMs have achieved strong results across domains including finance, bioinformatics, law, education, coding, and creative tasks.The related-work discussion situates spatial reasoning within the broader expansion of foundation models.
- Vision-Language Models: Vision-language research has built multimodal systems by combining Transformer-based vision components with powerful language models.Examples include Flamingo, LLaMA-Adapter, LLaVA, BLIP, MiniGPT-4, Qwen, Gemini, and MM1.
- Spatial Reasoning: Earlier spatial evaluations mainly emphasized textual or relational reasoning, used simpler navigation settings, or explored spatial reasoning only narrowly.SpatialEval instead targets diverse spatial tasks with more complex environments and in-depth analysis.
6 Discussion and Conclusions
The study finds that current VLMs struggle with spatial reasoning and often fail to exploit visual inputs effectively. The authors attribute this limitation to translating vision into language before reasoning and argue for architectures that treat vision as a first-class information source.
- Discussion and Conclusions: VLMs struggle with spatial reasoning, while visual inputs rarely outperform text-only alternatives and multimodal systems can rely less on vision when text is available.The authors interpret this pattern as evidence that current vision-to-language translation approaches limit visual reasoning.
- Discussion and Conclusions: Future models should treat vision as a first-class source of information and reason jointly across modalities to narrow the gap with human performance.This is the authors’ proposed architectural direction based on the observed limits of translation-based approaches.
A How Does SpatialEval Expand Beyond Traditional VQA?
SpatialEval expands traditional VQA by testing spatial reasoning across optional modality settings, requiring generative explanations, and pairing images with dense textual representations. Its tasks are designed as cognitive tests whose synthetic components remain highly solvable by humans.
- Task Scope and Focus: SpatialEval broadens VQA by evaluating TQA with LLMs, TQA with VLMs, and VTQA, allowing images or texts to be optional.Traditional benchmarks such as Visual Genome, GQA, and BLINK generally focus on image-required VQA.
- Evaluation Scheme: The benchmark primarily evaluates generative models that provide detailed step-by-step reasoning rather than treating VQA as purely discriminative.This leaves open whether prior VQA observations transfer to foundation models pretrained on web-scale data.
- Textual Representation of Images: Each image receives a long, dense caption that fully conveys its content, making answers less easily inferable than in benchmarks with brief or answer-implying descriptions.The captions include numerous objects and dense visual information.
- IQ Test for VLMs and LLMs: SpatialEval functions as an IQ test for multimodal models because humans solve its synthetic visual tasks with near-perfect accuracy.The authors use this human performance to characterize the tasks as within basic cognitive capabilities.
B Limitations and Societal Impact
The paper provides an extensive empirical evaluation but identifies formal theory and new training or adaptation strategies as important future directions. It also discusses potential benefits of improved spatial reasoning for safety and user experience.
- Limitations: The study is primarily empirical, leaving formal theoretical analysis of pretrained multimodal language models for future work.The authors describe theoretical study as challenging but potentially enriching for understanding these models.
- Limitations: The paper focuses on analysis rather than developing training strategies or adaptation algorithms to improve spatial reasoning.
- Societal Impact: The evaluations could catalyze algorithms that enhance LLM and VLM spatial reasoning for sectors requiring robust spatial understanding and navigation.The stated potential benefits include more reliable and efficient systems that improve safety and user experience.
- Dataset Illustrations: The appendix provides complete illustrations of Spatial-Map and Maze-Nav textual descriptions for clearer understanding of the tasks.Three questions, Q1 to Q3, accompany each sample in Spatial-Map, Maze-Nav, and Spatial-Grid.
- Dataset Illustrations: Maze-Nav represents walls, paths, starts, exits, and designated routes with ASCII symbols and defines movement through the four cardinal directions.The task also operationalizes right and left turns as 90-degree clockwise and anticlockwise changes in direction.
D Further Ablation Studies and Discussions
Further analyses compare text-only and vision-text inputs, prompting styles, and decoding temperatures. They find no universal winner between TQA and VTQA, while step-by-step prompting and lower temperature generally perform better.
- TQA vs. VTQA: No definitive winner emerges between TQA and VTQA across all tasks, model sizes, and architectures.For example, VTQA surpasses TQA for InstructBLIP-Vicuna-13B on Maze-Nav, while TQA wins on Spatial-Map.
- TQA vs. VTQA: TQA retains an advantage in some spatial reasoning tasks despite the availability of vision-text input.
- Prompting: The step-by-step explanation prompt consistently outperforms the simpler completion prompt.The compared instruction asks for a concise answer followed by a detailed, step-by-step explanation.
- Decoding Temperature: Most models underperform at temperature 1.0 compared with the default temperature of 0.2.The ablation motivates the lower-temperature default by examining response diversity and performance.
E Results on Spatial-Real task
Spatial-Real preserves the trends observed on synthetic tasks while achieving higher overall accuracy across input modalities. The modality gap is substantially larger for natural images.
- Spatial-Real Results: Overall accuracy increases across TQA, VQA, and VTQA on Spatial-Real compared with synthetic tasks.The same modality comparisons remain present on real images.
- Spatial-Real Results: 30.0% versus 7.0%: the VTQA–VQA accuracy gap widens substantially on Spatial-Real compared with synthetic tasks.The authors interpret this as a more pronounced performance disparity on natural images.
F Detailed Experimental Results
The paper reports detailed results for proprietary models and for individual synthetic spatial tasks. These breakdowns compare language and vision-language models using matched language-model backbones and input modalities.
- Proprietary Models: Proprietary-model results are summarized in Table 7, with detailed per-question comparisons reported separately.The detailed analyses cover Spatial-Map, Maze-Nav, and Spatial-Grid.
- Spatial-Map: Spatial-Map results compare LLMs and VLMs with the same language-model backbone across vision-only and vision-text inputs.Averaged results are summarized in Figures 6 and 11.
- Maze-Nav: Maze-Nav results compare matched-backbone LLMs and VLMs under vision-only and vision-text input formats.The table provides detailed results while averaged comparisons appear in Figures 6 and 11.
- Spatial-Grid: Spatial-Grid results provide detailed task performance, with averaged comparisons summarized in Figures 6 and 11.