Source-linked AI summary
What's "up" with vision-language models? Investigating their struggle with spatial reasoning
Amita Kamath, Jack Hessel, Kai-Wei Chang
TL;DR
The paper asks whether powerful vision-language models reliably understand basic spatial relations, a capability that existing benchmarks do not isolate cleanly. It constructs three controlled benchmarks, evaluates 18 models, examines LAION-2B, and tests remedies; models perform poorly, while the data analysis finds spatial prepositions rare and unreliable and the interventions insufficient. The authors conclude that these benchmarks expose a persistent challenge for precise spatial reasoning, with scope limited by benchmark scale and focus.
Problem
Vision-language models do not reliably capture basic spatial factors, while existing spatial benchmarks often conflate spatial reasoning with other reasoning types.
Method
The paper curates COCO-spatial, GQA-spatial, and tightly controlled What’sUp benchmarks, evaluates 18 VL models, studies LAION-2B, and tests modeling improvements.
Results
All 18 evaluated models perform poorly on the benchmarks; LAION-2B spatial prepositions are rare, ambiguous, or extraneous, and tested remedies do not substantially improve performance.
Takeaways & Limitations
The benchmarks provide controlled tests of basic spatial relations and support further investigation into why VL models struggle with precise spatial reasoning.
Takeaways & Limitations
The benchmarks are smaller than ARO and GQA, the improvement methods do not significantly improve all benchmarks, and the study is restricted to spatial reasoning.
Abstract
from arXiv · showhide
Recent vision-language (VL) models are powerful, but can they reliably distinguish "right" from "left"? We curate three new corpora to quantify model comprehension of such basic spatial relations. These tests isolate spatial reasoning more precisely than existing datasets like VQAv2, e.g., our What'sUp benchmark contains sets of photographs varying only the spatial relations of objects, keeping their identity fixed (see Figure 1: models must comprehend not only the usual case of a dog under a table, but also, the same dog on top of the same table). We evaluate 18 VL models, finding that all perform poorly, e.g., BLIP finetuned on VQAv2, which nears human parity on VQAv2, achieves 56% accuracy on our benchmarks vs. humans at 99%. We conclude by studying causes of this surprising behavior, finding: 1) that popular vision-language pretraining corpora like LAION-2B contain little reliable data for learning spatial relationships; and 2) that basic modeling interventions like up-weighting preposition-containing instances or fine-tuning on our corpora are not sufficient to address the challenges our benchmarks pose. We are hopeful that these corpora will facilitate further research, and we release our data and code at https://github.com/amitakamath/whatsup_vlms.
1 Introduction
Vision-language models perform well on complex benchmarks yet struggle with basic spatial relations. The paper introduces controlled benchmarks, investigates training-data explanations, and tests interventions that do not substantially resolve the problem.
- Existing VL models still fail to reliably capture basic spatial factors needed for more precise reasoning benchmarks.
- The paper studies “left of” and “right of” while separating spatial reasoning from other reasoning types conflated by existing benchmarks.
- Three benchmarks isolate spatial relations, with What’sUp using 205 four-image sets that vary only the preposition between two objects.
- Models select the correct caption from options differing only by preposition, and 18 VL models are evaluated across architectures, objectives, and training data.
- All models perform poorly across benchmarks, often only a few points above random chance and far below human performance.
- Common spatial prepositions occur in less than 0.2% of LAION-2B captions and may be ambiguous, extraneous, or guessable without the image.
- Probability renormalization, replacing “behind” with “in the background,” and several fine-tuning strategies do not dramatically improve spatial-relation understanding.
- The paper contributes benchmarks, training-data analysis, and intervention studies, releasing code and data for further research.
2 Benchmarks
The paper introduces three benchmarks that isolate basic spatial reasoning, then evaluates diverse vision-language models and finds persistent weaknesses despite strong performance on broader tasks. Controlled image-caption designs and additional analyses probe model architecture, scale, supervision, fine-tuning, and representation quality.
- 92% of GQA validation questions conflate spatial reasoning with object relationships or detection, whereas these benchmarks require spatial reasoning about only one or two objects.
- What’sUp contains 820 controlled photographs with balanced coverage of four spatial prepositions for each object pair.Subset A covers objects relative to furniture; Subset B covers objects relative to another tabletop object.
- The three benchmarks present an image with caption options differing only by preposition, requiring selection of the caption with the correct spatial relation.What’sUp provides four options, while COCO-spatial and GQA-spatial provide two.
- Experts estimate human performance at 97.3% on COCO-spatial, 99% on GQA-spatial, and 100% on What’sUp.
- All evaluated models perform poorly on basic spatial relations, with many near random chance and all far behind human-estimated performance.Failure cases include repeatedly predicting only one or two prepositions or predicting seemingly randomly.
- Scaling model size does not necessarily improve spatial reasoning, while XVLM and BLIP2 perform better zero-shot and XVLM is the highest-performing evaluated model.The reported comparisons also find no consistent advantage for generative over contrastive objectives or for downstream fine-tuning.
- CLIP-based models achieve 9% average analogy accuracy for prepositions versus 61% for edited color analogies, suggesting stronger learning of color attachments than spatial relations.Their direct color-option evaluation reaches 100% accuracy, while direct spatial evaluation averages 31%.
- The color analogy result has a methodological caveat because even JPEG compression can substantially change CLIP image representations.
3 Why do they struggle? Studying LAION
The paper studies why vision-language models learn little about basic spatial relations from image-caption pretraining, focusing on LAION-2B’s sparse and unreliable spatial language.
- Caption writers may omit spatial prepositions when relations seem obvious, ambiguous, or irrelevant to describing an image holistically.They may also use casual alternatives such as “next to” instead of “to the left of.”
- Spatial prepositions in LAION can be ambiguous because captions use inconsistent viewer- or subject-centered reference frames.The benchmarks instead use the viewer’s perspective, following CLEVR.
- Contrastive learning rarely requires distinguishing prepositions because identical descriptions seldom occur for two images in one batch, while common object-relation combinations are biased.This can let models satisfy the training objective without learning fine-grained spatial relations.
4 Data-informed attempts at improvement
The paper tests data-informed interventions addressing caption priors, terminology, and additional supervision. These interventions produce limited or narrow gains rather than broadly solving spatial reasoning.
- 4.1 Incorporating Caption Priors: Caption-prior normalization does not drastically improve performance on any benchmark.The method is slightly helpful on COCO-spatial and GQA-spatial, with only slight improvement on some What’sUp accuracies.
- 4.2 Better prompts: don’t fall (for) “behind”: 67%: OpenCLIP models average 67% with “background” prompts versus 52% with “behind” prompts.The “behind” result is only two points above random chance, while “background” performs substantially better.
- 4.2 Better prompts: don’t fall (for) “behind”: The “background” case suggests spatial information may be present but not accessible through every spatial term.The paper cautions that this terminology effect does not generalize to all spatial reasoning tasks.
- 4.3 Finetuning: 14.6 accuracy points: finetuning on COCO-spatial and GQA-spatial improves those benchmarks on average.However, performance on What’sUp drops by 4.3 accuracy points.
- 4.3 Finetuning: Even after supervised in-distribution finetuning, COCO-spatial and GQA-spatial remain approximately 50 accuracy points below human performance.This limits the extent to which the observed gains indicate robust spatial understanding.
- 4.3 Finetuning: Finetuning on preposition-containing LAION data yields only marginal improvements, while switched-preposition hard negatives produce high training loss.Across the tested models and data fractions, scaling model size or data does not help; denser supervision or an inductive bias may be needed.
5 Related work
Prior spatial-reasoning benchmarks often mix spatial understanding with other reasoning and use cluttered, biased images. This paper motivates tighter controls to expose models’ spatial weaknesses.
- Existing benchmarks such as VQAv2, GQA, NLVR2, CLEVR, and ShapeWorld include spatial questions but often conflate several reasoning types.Consequently, their scores can mask difficulty with spatial understanding specifically.
- Non-synthetic spatial benchmarks commonly use COCO or Visual Genome images containing many objects in cluttered environments.Such scenes can confuse models trained with only image-level supervision.
- 27 accuracy points above random chance: text-only GPT-1 scores this much on VALSE spatial questions, indicating that dataset priors can support performance without image attention.The paper contrasts this with What’sUp’s uncluttered, unambiguous photographs containing all four preposition options.
- Text-to-image models also struggle with spatial relations, and DALL-E 2 uses a frozen CLIP backbone that itself struggles with spatial reasoning.This connects vision-language spatial weaknesses to a broader generation problem.
6 Conclusion
The paper introduces three benchmarks for evaluating basic spatial relations and finds that vision-language models perform poorly, while several studied remedies do not substantially improve performance. It closes by proposing future directions for addressing these challenges.
- The paper proposes What’sUp, COCO-spatial, and GQA-spatial to evaluate basic spatial relations across varied environments.
- All 18 evaluated models perform poorly on the three benchmarks in the zero-shot setting.
- LAION captions contain spatial prepositions rarely, ambiguously, and often in extraneous contexts.
- The investigated remedies do not dramatically improve models’ understanding of spatial relations, and CLIP models fail to fit a large-scale precise-spatial-reasoning training set at the studied scale.
- Future directions include hard-negative generation during pretraining, more expressive fine-tuned models, and scaling experiments relating compute to benchmark performance.
Limitations
The paper’s benchmarks are limited in scale, its proposed improvement methods do not significantly improve performance across all benchmarks, and its investigation is restricted to spatial reasoning.
- The proposed benchmarks, especially What’sUp, are smaller in scale than ARO and GQA.
- The improvement methods grounded in the investigation do not significantly improve model performance across all benchmarks.
- The study focuses on spatial reasoning rather than a wide range of reasoning types.
A Appendix
The appendix provides detailed benchmark results, preposition frequencies in LAION-2B, caption-prior experiments, and finetuning loss curves. These materials reinforce that models struggle with spatial relations and that the relevant training signal is sparse.
- Table 3 details zero-shot and downstream-task-finetuned model results, with poor performance especially on pair and set accuracy.
- The spatial relations studied represent less than 0.22% of English LAION-2B training data after noise removal.
- Caption-prior experiments improve performance in some cases, but not by a large margin overall, and some results remain below random chance.
- Figures 5 and 6 compare finetuning losses with and without hard negatives on full and half-sized LAION-4M-prep datasets.
- Without hard negatives, the train loss is about 500x lower than with hard negatives targeting prepositions.
- Table 7 reports detailed results for different finetuning methods applied to CLIP ViT-B/32.