PaperlayerView exact revision on arXiv

Source-linked AI summary

VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchors

Haz Sameen Shahgir, Xiaofu Chen, Yu Fu, Erfan Shayegani, Nael Abu-Ghazaleh, Yova Kementchedjhieva, Yue Dong

arXiv:2604.02486v3cs.CVcs.CL

TL;DR

VLMs often fail at fine-grained visual tasks even when their internal representations retain the needed information, leaving the cause of this gap unresolved. This paper tests whether semantic labeling explains the gap across correspondence tasks and finds that naming improves performance, while task-specific finetuning supports stronger visual generalization without relying on semantic anchors.

  • Problem

    VLMs fail on fine-grained visual perception despite retaining relevant information internally, and the mechanism behind this hidden-in-plain-sight gap remains unexplained.

  • Method

    The paper evaluates semantic, synthetic shape, and face correspondence tasks, using performance comparisons, representation probing, Logit Lens analysis, naming interventions, and task-specific finetuning.

  • Results

    Across three correspondence settings, VLMs perform better on nameable entities; arbitrary naming improves novel-shape performance, while direct task finetuning outperforms name-trained models with lower semantic discernibility.

  • Takeaways & Limitations

    Many current VLM visual failures reflect a learned shortcut rather than a fundamental limitation of multimodal reasoning, and direct finetuning can elicit more general visual skills.

  • Takeaways & Limitations

    The Jaccard Distance measure underestimates semantic discernibility because it uses only top-1 tokens, ignores synonyms, and counts shared tokens among similar entities.

Abstract

from arXiv · show

Vision-language models (VLMs) have achieved impressive performance across a wide range of multimodal tasks. However, they often fail on tasks that require fine-grained visual perception, even when the required information is still present in their internal representations. Prior work has attributed this ``hidden-in-plain-sight'' gap to the language model, but the cause remains unexplained. In this work, we demonstrate that this gap arises from the language model's lack of semantic labels for fine-grained visual details: when visual entities can be mapped to known concepts, VLMs bypass visual comparison and reason through language; when they cannot, VLMs resort to brittle and hallucinated descriptions. We verify this across semantic correspondence, synthetic shape matching, and face matching, and find that VLMs perform much better when the relevant entities are nameable than when they are unnamable. Mechanistically, Logit Lens analysis confirms that VLMs explicitly recover semantic labels for nameable entities and surface more unique tokens compared to unnameable entities. Furthermore, we show that this limitation can be addressed: teaching completely arbitrary names for unknown entities improves performance. More importantly, task-specific finetuning yields even stronger generalization without relying on language priors, i.e., through real visual perception. Our findings suggest that current VLM failures on visual tasks reflect a learned shortcut rather than a fundamental limitation of multimodal reasoning. Code and datasets are available at https://github.com/Patchwork53/VLMs-Need-Words-COLM2026.

1 Introduction

The paper argues that VLMs fail at fine-grained visual perception because they rely on semantic labels: nameable entities support language-mediated reasoning, while unknown entities expose a hidden representation-to-output gap. Across correspondence settings, naming improves performance, and task-specific finetuning instead elicits direct visual comparison.

  • VLMs can retain enough visual information internally to solve difficult tasks even when their verbal outputs fail.
  • The proposed mechanism is that VLMs map visual entities to semantic labels, bypassing pixel-level comparison for nameable entities and hallucinating descriptions for unknown ones.
  • The study tests this hypothesis across real visual correspondence, synthetic familiar-versus-procedural shapes, and recognizable-versus-generated faces.
  • Across these settings, VLMs perform better on nameable entities than unnameable ones despite sufficient information in both representations.
  • Teaching arbitrary names to novel shapes substantially closes the performance gap by enabling matching through learned labels rather than pixels.
  • Direct task-specific finetuning outperforms name training while showing lower semantic discernibility, indicating a distinct direct-visual-comparison mechanism.

2 The Performance Gap Between Textual Response and Representation Space is Larger for Unknown Entities

Across real-image and synthetic correspondence tasks, textual performance falls further behind representation probing for entities without clear semantic names. Controlled experiments and Logit Lens analysis link this gap to weaker semantic discernibility and misleading verbal descriptions for unknown entities.

  • 2.1 Testing Nameable and Unnameable Key Points in Semantic Correspondence: VLM accuracy is lower for No-Name than Named keypoints across model sizes, model families, and both Direct and CoT settings.
  • 2.1 Testing Nameable and Unnameable Key Points in Semantic Correspondence: 13.7% versus 3.3%: Qwen3VL-8B’s R−max(D, C) gap is larger for No-Name than Named keypoints.The representation probe nearly closes the textual gap for Named keypoints but not for No-Name keypoints.
  • 2.1 Testing Nameable and Unnameable Key Points in Semantic Correspondence: Qwen3VL-2B gains +20.8% from CoT on Named keypoints but only +9.8% on No-Name keypoints.CoT chains explicitly generate point names when available, converting correspondence into verbal string matching; the difference narrows with scale.
  • 2.2 Confirming the Effect of Semantic Anchors on Shape and Face Correspondence: 29.0% versus 74.2%: Qwen3VL-2B’s Direct accuracy on unknown shapes trails its Representation Probing accuracy by 45.2 percentage points.Known shapes are nearly saturated, with 100% Representation Probing and 97.3% CoT accuracy.
  • 2.2 Confirming the Effect of Semantic Anchors on Shape and Face Correspondence: CoT can reduce performance on unknown entities, including a 19.4-point drop for Qwen3VL-8B on unknown shapes.The results suggest that verbal reasoning without a semantic anchor can produce hallucinated descriptions that mislead the task.
  • 2.3 Logit Lens Reveals VLMs Explicitly Recover Semantic Anchors: Known shapes and faces produce higher Mean Jaccard Distance than unknown entities across layers in Gemma3-12B and Qwen3VL-8B.Higher distance indicates more differentiable Logit Lens token sets; known entities surface more unique semantic tokens.

3 Teaching Arbitrary Names

Teaching arbitrary names to unknown shapes improves VLM correspondence accuracy by giving the models semantic anchors. Logit Lens results link this improvement to increased semantic discernibility and learned label-based matching.

  • Training and Results: The models are trained on three name types: ordinary object names, human names, and randomly generated six-character strings.These name sets differ in tokenization length, from one token for ordinary names to about 4.7 tokens for random names.
  • Training and Results: Teaching arbitrary names improves unknown-shape correspondence accuracy for both Qwen3VL-2B and Gemma3-4B.The intervention teaches names through single-image tasks distinct from the original correspondence task.
  • Training and Results: Qwen3VL-2B reaches 86.0% with ordinary names, 70.2% with human names, and 62.8% with random names, versus 29.0% baseline.Ordinary names outperform human and random names for Qwen3VL-2B.
  • Mechanism: Finetuned Qwen models close the Mean Jaccard Distance gap between unknown and known shape baselines.The ordering of semantic discernibility—Ordinary > Human > Random—matches the ordering of VQA accuracy.
  • Mechanism: Chain-of-thought reasoning shows that name-trained VLMs match shapes through learned semantic anchors rather than fine-grained pixel comparison.The learned names let the models short-circuit visual comparison.

4 Is Semantic Alignment A Requirement or A Side Effect?

The paper tests whether semantic alignment is necessary for visual correspondence by finetuning VLMs directly on shape comparison and evaluating them on unseen shapes and tasks. Strong out-of-distribution performance, including across shape families, indicates that task-specific finetuning can teach direct visual comparison rather than reliance on internal labels.

  • Experimental Setup: Task-specific finetuning trains VLMs on squiggle correspondence and evaluates them on held-out squiggles, mazes, face correspondence, and semantic correspondence.The setup uses squiggles with n=30 anchor points and procedurally generated test shapes, including mazes with no geometric resemblance to training shapes.
  • Results: Both Qwen3VL-2B and Gemma3-4B show strong out-of-distribution generalization, including to mazes unseen during training.Gemma3-4B reaches 99.3% on complex n = 5 maze structures after training only on squiggles.
  • Results: Finetuning on squiggles improves face and semantic correspondence despite both being highly out-of-distribution.Qwen3VL-2B improves by 16% on face correspondence and 10.7% on semantic correspondence; Gemma3-4B improves by 8.7% and 5.8%, respectively.
  • Mechanism: Task-specific finetuning achieves 98.7% VQA accuracy versus 86.0% for teaching ordinary names, despite lower Mean Jaccard Distance.This performance pattern suggests a mechanism beyond semantic discernibility.
  • Conclusion: Semantic anchoring is sufficient but not necessary: VLMs can learn direct visual comparison when training requires fine-grained matching.The paper treats semantic anchoring and direct visual comparison as distinct pathways for closing the hidden-in-plain-sight gap.

5 Related Work

Related work identifies persistent VLM failures on fine-grained visual reasoning and evidence that visual representations become more language-like inside the language model. This paper connects these observations by comparing language-mediated semantic anchoring with direct visual comparison.

  • Fine-Grained Visual Reasoning: Recent studies report VLM weaknesses in chart and diagram understanding, optical illusions, and subtle visual discrimination requiring local evidence.These tasks depend on comparing small visual differences rather than relying on coarse object semantics.
  • Cross-Modal Representations: Prior work argues that mathematically meaningful cross-modal similarity requires visual and textual modalities to share a dimensionally consistent coordinate system.Other studies find that image-conditioned hidden states become progressively more interpretable in vocabulary space.
  • This Work: This paper links fine-grained visual failures to the availability of semantic anchors and compares arbitrary-name teaching with direct task finetuning.The two interventions respectively strengthen language-mediated shortcuts and visual comparison without the same semantic recoding.

6 Conclusion

VLMs use visual information more effectively when entities have semantic labels, but direct task finetuning can support visual skills with less reliance on semantic anchoring. These findings suggest that many failures reflect a learned shortcut rather than a fundamental multimodal reasoning limitation.

  • Across semantic, synthetic, and face correspondence tasks, VLMs perform substantially better on familiar, nameable entities than on semantically unknown ones, even when representations retain sufficient information.
  • Logit Lens analyses and naming interventions indicate that semantic labels let models shortcut direct visual comparison by mapping correspondence problems into language.
  • Direct task finetuning enables stronger visual skills that transfer to unseen shapes and families while relying less on semantic anchoring.
  • Many current VLM failures reflect a learned shortcut rather than a fundamental limitation of multimodal reasoning.

Supplementary Material: Appendices

The supplementary examples illustrate how language-mediated reasoning succeeds for named visual entities but becomes error-prone when entities lack concrete semantic labels.

  • Chain-of-Thought reasoning can recover semantic labels for named points and convert correspondence into a verbal problem.
  • When points lack concrete semantic labels, transcription becomes harder and can produce hallucinated descriptions, such as mislabeling point B.

B Representation Probing Performance Per Layer For All Tasks

Layer-wise probing evaluates representations for nameable versus unnameable entities across semantic correspondence, face correspondence, and procedurally generated shape tasks.

  • Figure 8 reports layer-wise representation probing performance for nameable and no-name reference points in semantic correspondence.
  • Figure 9 reports layer-wise representation probing performance for known celebrities and unknown AI-generated faces.
  • Figure 10 reports layer-wise representation probing performance for 2D known and procedurally generated squiggle shapes.

C Faces Dataset Experimental Setup

The face correspondence dataset uses generated celebrity images and model-specific knowledge probes to define known and unknown identities for evaluation.

  • Figure 11 contrasts known celebrities with AI-generated faces in the face correspondence task.
  • The dataset contains four generated images per celebrity with varied expressions, lighting, poses, clothing, and hairstyles at uniform resolution.
  • A celebrity is designated known to a VLM if at least one of four generated images is identified correctly when prompted for the person’s identity.
  • Table 5 presents the initial celebrity set and model-specific subsets used for the face correspondence dataset.

D InternVL3.5’s Facial Recognition Failure and Results on Shape Correspondence

InternVL-3.5 fails to recognize most evaluated celebrities, preventing facial correspondence evaluation, but shape correspondence results corroborate the broader known-versus-unknown pattern.

  • Facial Recognition Failure: InternVL-3.5-14B recognized only two celebrities, while its 2B and 8B variants were similarly limited.The authors therefore excluded InternVL-3.5 from facial correspondence evaluation.
  • Shape Correspondence: InternVL-3.5 was evaluated on shape correspondence because it recognizes common geometric shapes such as squares, circles, and stars.
  • Shape Correspondence: For unknown shapes, InternVL-3.5 shows a larger gap between representation-probing accuracy and verbal output.
  • Shape Correspondence: Chain-of-Thought reasoning benefits InternVL-3.5 only when shapes are known, corroborating results for Gemma3 and Qwen3VL.
  • Finetuning Tasks: The experiments evaluate naming, yes/no, choice, comparison, and description tasks using arbitrary, human, or ordinary-object names.

H.1 Transfer to Open-Ended Visual Change Description

Correspondence finetuning transfers to open-ended visual change description on CLEVR-Change, which differs from training in both visual domain and output format.

  • CLEVR-Change requires free-form descriptions of subtle changes in synthetic scenes rather than selecting annotated candidates or producing fixed labels.
  • Qwen3VL-2B finetuning on squiggle correspondence raises the mean Qwen3-8B judge score from 4.507 to 5.087.
  • The share of high-scoring responses, defined as scores 8–10, increases from 24.7% to 33.1% after finetuning.
  • The improvement provides evidence that correspondence finetuning transfers to broader visual comparison behavior rather than only teaching the multiple-choice answer format.

I Art Style Matching

Art-style matching extends the semantic-anchoring pattern to holistic visual comparison: models generally perform better when they can recognize the painters associated with artworks.

  • Motivation: Art-style matching compares global properties such as texture, color usage, brushwork, and composition, while recognizable painters may provide semantic shortcuts.
  • Dataset Construction: Known artworks come from painters the model identifies, whereas Unknown artworks come from painters it does not identify.
  • Evaluation: The evaluation uses Direct and Chain-of-Thought textual strategies plus MaxSim and Gram-matrix-distance representation comparisons.
  • Results: 73.0% versus 37.2%: Qwen3VL-8B’s CoT accuracy is higher on Known than Unknown artworks.
  • Results: Representation-based methods generally also perform better on Known artworks for Qwen3VL-2B, Qwen3VL-8B, and Gemma3-12B.
  • Results: Gemma3-4B is an exception in textual evaluation, although both representation-based strategies still perform substantially better on known artworks.
  • Limitation: Fixed, training-free comparison functions may miss holistic and spatially distributed style features, so learned probes could extract them more effectively.
  • Conclusion: Overall, performance is generally stronger when artworks can first be mapped to recognizable language concepts.
Loading 2604.02486v3…