Source-linked AI summary
Investigating Relational Reasoning in VLMs
Adhithya Laxman Ravi Shankar Geetha, Aulia Kharis Rakhmasari, Haleema Ramzan, Xander Yap
TL;DR
The paper asks whether VLMs genuinely understand visual relations or exploit language cues and priors. It combines controlled synthetic scenes, layer-wise probing, and masked-input tests, finding that model behavior relies on both visual processing and shortcuts, with substantial dependence on explicit visual evidence and internal regularities.
Problem
The study addresses whether strong VLM visual-reasoning performance reflects genuine understanding of visual relations or shortcuts based on language cues and data priors.
Method
The paper uses a synthetic geometric-shape benchmark, layer-wise linear probes, and masked-input variants to analyze visual representations and sensitivity to visual evidence.
Results
VLMs combine visual reasoning with shortcut strategies: masking causes −17% to −23% drops on hybrid tasks, while recognition and spatial tasks change by ±5%.
Takeaways & Limitations
The findings indicate that apparent VLM understanding can be fragile and substantially shaped by language cues, data priors, and internal statistical regularities.
Takeaways & Limitations
Complex queries require separate linear probes for multiple sub-labels, and reported results average performance across those sub-labels.
Abstract
from arXiv · showhide
Vision-Language Models (VLMs) achieve strong performance in visual reasoning tasks, but it remains unclear whether they understand visual relations, or simply employ shortcuts such as language cues or priors. To investigate this, we use the Qwen3-VL-4B (Bai et al., 2025), a modern VLM, to decode how visual information is encoded across depths. For this, we propose a synthetic dataset of simple geometric shapes for controlled analysis, along with queries crafted to precisely test language cues. Furthermore, the dataset is modified to test causal reliance on visual evidence. Our results show that current VLMs combine genuine visual reasoning with shortcut strategies primarily rooted in language cues.
1. Introduction
Visual reasoning depends on representing objects, attributes, and relations that are implicit rather than directly marked in pixels. Despite strong VLM task performance, their internal relational understanding remains uncertain, motivating controlled tests of visual evidence and language-cue reliance.
- Visual information combines primitives, attributes, and relationships, with relations such as left-of requiring interpretation of relative space rather than explicit pixel demarcations.
- Strong performance by modern VLMs has not resolved how they internally understand visual information beyond explicit representation.
- Prior work reports that VLMs often struggle with relational reasoning beyond entity recognition on diagrammatic and compositional reasoning benchmarks.
- The study examines depth-wise visual representations and prediction sensitivity to modified visual evidence and language cues, finding depth-dependent encoding, inconsistent relational patterns, and reliance on explicit visual elements.
2. Related Work
Prior research uses mechanistic attention analysis and relational reasoning benchmarks to examine whether VLMs capture structured visual relations rather than relying on shortcuts. This work builds on those directions with controlled relational tasks and internal diagnostics.
- Mechanistic interpretability analyzes transformer attention patterns as structured dependencies relevant to syntax and compositional reasoning in language models.
- In VLMs, attention analysis has more commonly addressed object localization and text–image grounding than mechanistic diagnosis of inter-entity relations.
- GQA, CLEVR, and ChartQA provide complementary evaluations of relational, visual, and logical reasoning over structured graphics, while prior work motivates controlled shortcut tests.
- The related-work framing positions controlled relational tasks and attention-based diagnostics as extensions of existing benchmark and shortcut-reliance research.
3. Methodology
The methodology combines a controlled geometric-shape benchmark, multiple-choice queries, layer-wise linear probes, and masked image variants to analyze information encoding and visual reliance in a VLM.
- The synthetic benchmark contains 5,424 geometric-shape images with explicit ground-truth relations, reducing linguistic ambiguity and enabling controlled complexity.
- Each scene uses 2–5 entities with discrete shapes, colors, and sizes, arranged in clean non-overlapping layouts for precise attention analysis.
- Scenes encode centroid-based axis-aligned relations and split relationships evenly between explicit arrows and implicit position-only variants.
- Masked counterparts remove one entity or relation through full re-rendering, creating visual perturbations for testing reliance on visual evidence.
- Multiple-choice prompts are generated from image-linked JSON content, ensuring query-to-image alignment while using dataset entities and relations for incorrect options.
- Layer-wise linear probes predict query answers from extracted visual attention features, diagnosing which information is decodable at different depths.
- Probe performance is complemented by masked-input accuracy changes because decodability alone does not establish causal reliance on visual evidence.
4. Experiments and Discussion
Qwen3-VL-4B performs well on directly tangible recognition queries, while relational performance varies with the explicitness of textual cues. Layer-wise probes reveal distinct depth-dependent encoding patterns, and masking exposes hybrid visual–shortcut strategies.
- Overall Query Performance: Recognition accuracy is high when queries map directly to tangible visual information.The model performs consistently well on recognition tasks such as color, shape, and count.
- Overall Query Performance: Shape accuracy is lower than Color accuracy, while combining Shape and Color improves performance with more textual information.The result indicates stronger color understanding in the overall query evaluation and a benefit from explicit textual information for conjunctions.
- Overall Query Performance: Arrow Direction significantly outperforms Arrow Existence and Implicit Position because its query provides more explicit, tangible information.Arrow Existence requires discovering connections without textual clues, while Implicit Position lacks tangible visual data.
- Linear Probing: Layer-wise probes use pooled attention maps to train logistic regressions for each query, with 20% of data retained for testing.Attention maps contain 5 steps, 36 layers, and 32 heads; mean pooling across steps and heads yields 36 maps per query.
- Accuracies by Query Type: Count becomes more linearly decodable toward middle-to-late layers, while shape and color show asymmetric positive- and negative-case separability.Shape has stronger positive-case performance, color confirms queried colors better than it rejects absent ones, and Shape-and-Color has low negative-case accuracy.
- Accuracies by Query Type: Arrow Existence is stable across depth, Arrow Direction remains near chance, and Implicit Position has the weakest separability.Implicit Position errors appear less driven by missing linearly accessible evidence than by ambiguity, noise, or nonlinear dependencies.
- Attention Patterns: Early layers encode salient evidence and coarse statistics, middle layers provide the strongest task-relevant signals, and late layers inconsistently recover higher-level relational structure.Late layers often do not recover object identity or compositional binding, and some global signals can become less linearly accessible.
- Ablation Study: Masking reveals a hybrid strategy: count and arrow tasks drop by 17%–23% under distribution shift, whereas recognition and spatial tasks change by about ±5%.The ablation frames high unmasked accuracy of 80%–99% as potentially fragile and shows that masking can expose reliance on internal biases.
5. Conclusion
The study concludes that VLMs lack true visual understanding and remain limited to explicitly observable information, relying on language cues and data priors.
- VLMs do not exhibit true visual understanding, according to the study’s conclusion.
- Their comprehension is strictly limited to what is explicitly observable.
- The results show clear reliance on language cues and data priors.