Source-linked AI summary
Visual Spatial Reasoning
Fangyu Liu, Guy Emerson, Nigel Collier
TL;DR
Current vision-language benchmarks provide limited controlled evidence about spatial reasoning, despite its importance and difficulty for VLMs. VSR addresses this gap with a large natural image-text dataset covering diverse spatial relations and evaluates several models. Models remain substantially below humans, with orientation and concept-zero-shot reasoning especially challenging.
Problem
Existing benchmarks conflate error sources, while spatial reasoning remains particularly challenging for current vision-language models.
Method
VSR is a controlled dataset of >10k natural image-text pairs covering 66 English spatial relations, evaluated with true/false classification and concept-zero-shot splits.
Results
Models perform more than 25% below the human ceiling; on the concept zero-shot split, tested VLMs struggle to reach 60% accuracy, with orientation especially difficult.
Takeaways & Limitations
By-relation performance has little correlation with training-example counts, while orientation and concept-dependent proximity remain challenging for VLMs.
Takeaways & Limitations
Synthetic datasets may miss real-world challenges because orientations and scene context affect spatial-relation interpretation.
Abstract
from arXiv · showhide
Spatial relations are a basic part of human cognition. However, they are expressed in natural language in a variety of ways, and previous work has suggested that current vision-and-language models (VLMs) struggle to capture relational information. In this paper, we present Visual Spatial Reasoning (VSR), a dataset containing more than 10k natural text-image pairs with 66 types of spatial relations in English (such as: under, in front of, and facing). While using a seemingly simple annotation format, we show how the dataset includes challenging linguistic phenomena, such as varying reference frames. We demonstrate a large gap between human and model performance: the human ceiling is above 95%, while state-of-the-art models only achieve around 70%. We observe that VLMs' by-relation performances have little correlation with the number of training examples and the tested models are in general incapable of recognising relations concerning the orientations of objects.
1 Introduction
VSR addresses limitations in existing vision-language benchmarks by providing controlled evaluation of spatial reasoning in natural image-text pairs. It reveals substantial gaps between human and VLM performance, including weaknesses with orientation and concept generalisation.
- Existing benchmarks conflate multiple error sources, hindering controlled analysis of specific linguistic and cognitive properties.
- VSR is a controlled dataset for testing spatial reasoning in natural image-text pairs.Each example pairs an image with a natural-language spatial description and requires binary true/false classification.
- 66 spatial relations and >10k data points are included, using 6,940 MS COCO images.
- Human ceiling exceeds 95%, whereas four tested VLMs struggle to reach 70% accuracy.The analysis also identifies the importance of positional encodings, weak relation-frequency correlation, orientation difficulties, and poor unseen-concept generalisation.
2 Related Work
Prior spatial-reasoning benchmarks provide either limited control, restricted expressivity, entangled challenges, or insufficient grounding. VSR is positioned to probe a broader range of spatial relations in natural images while retaining controlled evaluation.
- Synthetic datasets offer control but simplify real-world spatial reasoning through bounded relations, absent orientations, and limited scene context.In real images, relation interpretation can depend on object orientation and scene context.
- Spatial relations often depend on viewpoint, object fronts, and orientations, so rule-based determination is generally inadequate.
- Natural-image benchmarks such as NLVR2 and VQA entangle spatial relations with other linguistic and syntactic challenges.
- Visual Genome contains only positive relation statements, preventing binary classification, while VALSE has 535 zero-shot examples and no training set.
- Text-only spatial datasets cannot evaluate how models connect language with grounded visual information.
3 Dataset Creation
VSR is constructed through contrastive caption generation and human validation over natural MS COCO images, with controlled relations, explicit reference-frame guidance, and challenging dataset splits.
- Data collection: VSR construction uses two phases: contrastive caption generation followed by second-round human validation.
- Caption generation: Contrastive generation pairs images containing the same two concepts and asks annotators to select a relation true for one image but false for the other.
- Relation inventory: The fixed relation inventory contains 71 candidates, of which 66 appear in the final dataset.
- Validation: Each data point is reviewed by at least three validators, and instances with fewer than two validators agreeing with the original label are excluded.
- Reference frames: Validation accepts a caption as true when it holds under either intrinsic or relative reference frames.This accommodates cases where the same scene supports different interpretations depending on the chosen frame.
- Dataset splits: The concept zero-shot split prevents concept overlap across train, development, and test sets, requiring compositional learning of concepts and relations.
4 Dataset Analysis
The dataset analysis reports validation-based filtering, relation and concept distributions, annotator disagreement, and reference-frame usage. It also shows that ambiguity varies by relation and that human references in images are associated with more intrinsic-frame interpretations.
- Dataset statistics: 12,809 raw data points were reduced to 10,972 final examples after validation-agreement filtering.Every example received at least three validation labels; instances below the 2/3 agreement threshold were excluded.
- Dataset statistics: Basic relations such as “behind,” “in front of,” “on,” “under,” and left/right relations are frequent, while concepts largely follow MS COCO distributions.Animals and indoor objects are especially common, and concept distributions at the two entity positions are generally similar.
- Annotator disagreement: Around 70 of 100 sampled disagreements were genuinely ambiguous, while around 30 were caused by annotation errors.The unfiltered dataset is therefore retained for studying cognitive and linguistic phenomena, including disagreement patterns.
- Annotator disagreement: Relations requiring intrinsic orientation judgments, such as “at the side of” and “in front of,” show high disagreement, as do vague proximity relations such as “near” and “close to.”Part-whole and in/out relations have the least disagreement.
- Reference frames: The relative reference frame is most common among the predominantly native-English, university-educated annotators, consistent with prior findings for standard English varieties.The dataset examines reference-frame usage across many spatial relations and natural images.
- Reference frames: 198 of 616 cases involved a person; these used intrinsic frames in 16% of cases versus 9% without a person, a statistically significant difference.Relative-frame usage was 78% with a person and 87% without one; 6% of person-involved cases supported both frames.
5 Experiments
The experiments evaluate four VLM baselines on VSR and analyze performance across relations, splits, reference frames, and training-data amounts. Models perform substantially below humans, with failures concentrated in orientation- and concept-sensitive spatial reasoning.
- Baselines and configurations: Four VLMs are evaluated on VSR using finetuned VisualBERT, LXMERT, and ViLT, plus off-the-shelf CLIP with prompted negations.The finetuned models receive image-text pairs, while CLIP compares similarity for original and negated or antonym captions.
- Experimental results: Around 70% accuracy is reached by the best models on the random split, but ViLT reaches only 63.0% on the zero-shot split.CLIP obtains around 55% on both splits, while overall performance remains more than 25% below the human ceiling.
- Experimental results: Explicit positional information matters: LXMERT and ViLT outperform VisualBERT by more than 10% on both splits.LXMERT encodes object coordinates, and ViLT uses positional encodings for image patches; VisualBERT has no explicit position encoding.
- Experimental results: Performance declines substantially from random to zero-shot splits, indicating that disentangling concept and relation representations is challenging.The authors argue that this gap is not likely due to training-example counts but to the difficulty of concept zero-shot learning.
- Performance by relation: Relation frequency has little apparent correlation with performance, while orientation relations such as “facing” and “parallel to” remain difficult.Left-right relations are also challenging when they require interpreting an intrinsic reference frame and object orientation.
- Performance by relation: Orientation is the worst-performing relation group, and Proximity falls from close to 75% accuracy on the random split to chance level on the zero-shot split.The authors attribute the Proximity decline to concept-dependent meanings of relations such as “near” and “far from.”
- Reference frames: Reference-frame prediction transfers positively to VSR performance on left/right relations, supporting reference-frame recognition as an intermediate reasoning step.The case study excludes statements interpretable in both intrinsic and relative frames and initializes the VSR model with the frame-prediction checkpoint.
6 Conclusion and Future Directions
VSR is a controlled probing dataset for evaluating spatial reasoning in natural image-text pairs, revealing substantial model weaknesses and motivating broader investigation of spatial reasoning across architectures, languages, and cultures.
- VSR probes VLM recognition and reasoning about spatial relations in natural image-text pairs while capturing linguistic variability, reference frames, and ambiguity.
- More than 25% below the human ceiling, the tested VLMs underperformed humans on VSR.
- Below 60% accuracy on the concept zero-shot split, the tested VLMs struggled and plateaued despite increased training examples.
- ViLT and LXMERT outperformed VisualBERT among finetuning-based VLMs, while explicit positional information was identified as crucial.
- Orientation was the most difficult relation category, and proximity was also challenging, especially in the zero-shot setup.
- Future work includes comparing dual- and cross-encoders, testing larger VLMs, extending VSR across languages and cultures, and studying spatial language across populations.
A Appendix
The appendix documents genuinely ambiguous spatial annotations, dataset relation-selection decisions, and the interfaces used for annotation and validation.
- Ambiguous cases include “below” when horizontal lower position differs from being directly underneath, and “above” when an object exceeds only part of another object.
- The standard VSR splits exclude ambiguous cases to ensure human consensus, while the raw dataset retains them for further study.
- Relations were excluded when rarely used or redundant, and additional frequent MS COCO relations such as “at the edge of” and “touching” were included.
- Figures 15 and 16 show the Label Studio interfaces for caption generation and validation.
- Tables 6 and 7 provide the original relation inventory and the modified set used in VSR.