PaperlayerView exact revision on arXiv

Source-linked AI summary

Hidden in plain sight: VLMs overlook their visual representations

Stephanie Fu, Tyler Bonnen, Devin Guillory, Trevor Darrell

arXiv:2506.08008v1cs.CVcs.AIcs.LG

TL;DR

The paper asks whether VLMs can use visual information as effectively as their visual encoders on tasks solvable from images alone. It compares VLM evaluation with direct visual readouts and finds that VLMs often fall to chance, with the LLM’s use of visual information emerging as a key bottleneck.

  • Problem

    On tasks that rely on image evidence rather than background knowledge, VLMs often perform near chance despite prior reports that their embedded vision models excel on these tasks.

    Source: §1 · p. 2

  • Method

    The paper compares VLMs with direct visual-encoder readouts across vision-centric benchmarks and analyzes representation quality, prompt sensitivity, and the LLM’s use of visual information.

    Source: §1 · p. 2

  • Results

    VLM evaluation causes a universal performance drop, often to chance, and does not preserve vision-model rankings across pretraining strategies.

    Source: §1 · p. 2

  • Takeaways & Limitations

    VLMs underuse visual representations accessible throughout the model, and improving the LLM’s ability to use them offers the most promising returns for vision-centric capabilities.

    Source: §1 · p. 2

    The paper notes that current visual encoders still have shortcomings, including on 3D shape inference, where humans outperform vision models.

    Source: §6 · p. 9

Abstract

from arXiv · show

Language provides a natural interface to specify and evaluate performance on visual tasks. To realize this possibility, vision language models (VLMs) must successfully integrate visual and linguistic information. Our work compares VLMs to a direct readout of their visual encoders to understand their ability to integrate across these modalities. Across a series of vision-centric benchmarks (e.g., depth estimation, correspondence), we find that VLMs perform substantially worse than their visual encoders, dropping to near-chance performance. We investigate these results through a series of analyses across the entire VLM: namely 1) the degradation of vision representations, 2) brittleness to task prompt, and 3) the language model's role in solving the task. We find that the bottleneck in performing these vision-centric tasks lies in this third category; VLMs are not effectively using visual information easily accessible throughout the entire model, and they inherit the language priors present in the LLM. Our work helps diagnose the failure modes of open-source VLMs, and presents a series of evaluations useful for future investigations into visual understanding within VLMs.

1 Introduction

The paper investigates why VLMs often perform poorly on vision-centric tasks despite strong visual encoders, comparing VLM outputs with direct readouts of their visual representations. It finds that the LLM’s use of visual information is a key bottleneck, while prompting alone does not close the gap.

1 Introduction

  • The study compares VLMs with visual encoders on vision-centric tasks spanning visual abilities, input types, and visual prompting strategies.It analyzes representation quality, prompt sensitivity, and the language model’s ability to use visual representations.Source: §1 · p. 2
  • VLM evaluation causes a universal performance drop, often to chance, and does not preserve vision-model rankings across pretraining strategies.The paper compares standard visual probing with VLM-based evaluation across vision-centric benchmarks.Source: §1 · p. 2, Figure 2 · p. 2
  • Prompt-tuning improves performance only marginally and does not close the gap with standard vision evaluation.The paper also reports that visual representations remain usable through projector and LLM layers when evaluated with visual probing.Source: §1 · p. 2
  • Across tasks, finetuning the LLM yields higher accuracy than finetuning the projector or vision encoder, implicating the LLM’s use of visual representations as a bottleneck.VLM answer distributions largely reflect their blind answers, and LLM finetuning best overcomes these biases.Source: §1 · p. 2
  • The paper concludes that VLMs underuse visual representations accessible throughout the model, and that improving the LLM’s ability to use them offers the most promising returns for vision-centric capabilities.VLMs also inherit language priors, complicating conclusions about their underlying vision representations.Source: §1 · p. 2

2 Evaluation Setup

The evaluation compares visual encoders with VLMs on tasks designed to test abilities grounded in visual input rather than domain expertise. The tasks cover depth, correspondence, object awareness, and art style.

2 Evaluation Setup

  • The setup evaluates four vision backbones, including self-supervised, supervised, and vision-language-pretrained encoders, with paired Vicuña-based VLMs and additional open-source models.The paired setup keeps the vision encoder weights identical to the original ViTs, while the additional models have encoders trained for their VLM context.Source: §2 · p. 3

2.1 Depth Estimation

  • Depth estimation tests which of two boxed objects is closer to the camera, using a DPT head trained on NYUv2 to read out encoder depth predictions.The predicted depth map is cropped to each box, and average depths are compared.Source: §2.1 · p. 3

2.2 Correspondence-based Evaluations

  • Correspondence tasks test matching by semantic similarity, shared object function, or low-level visual patterns across images.Semantic correspondence uses SPair-71k and BLINK keypoints; functional matching uses FunKPoint, while low-level matching uses HPatches image pairs and homographies.Source: §2.2 · pp. 3–4
  • Across tasks, visual evaluation outperforms both VLM and blind evaluation; DINOv2 leads in 5 of 6 encoder tasks but in no VLM task.The table highlights a rank-order shift between standalone visual encoders and VLMs.Source: Table 1 · p. 4

2.3 3D Object Awareness

  • MOCHI tests object-centric representations by asking models to identify the image containing a different object among three or four options.The encoder readout uses pairwise cosine similarity of CLS embeddings and selects the image with the lowest average similarity.Source: §2.3 · p. 4

2.4 Art Style

  • The Art Style benchmark tests visual style matching without requiring artist-name or historical-context knowledge.The encoder compares Gram-matrix style features derived from last-layer ViT patch features, while the VLM uses multiple-choice VQA.Source: §2.4 · p. 4

3 Initial Observations

Large vision encoders perform well on vision-centric tasks, but VLM evaluations often fall from near-ceiling to chance and can reorder which encoders appear strongest. VLM answer choices also largely reflect blind-baseline biases.

3.1 Large-scale vision encoders perform well

  • Large vision models perform above chance on every task, reaching 88.7% on depth estimation; DINOv2 leads tasks requiring spatial awareness.The evaluations are zero-shot except for depth estimation, where a DPT head is trained.Source: §3.1 · p. 4

3.2 VLMs perform far worse than their encoders

  • VLM evaluation universally lowers performance relative to visual readout, often from near-ceiling to chance; low-level matching drops 45.5%, while depth estimation drops 21.7%.Source: §3.2 · p. 5
  • The encoder-to-VQA performance drop also appears in InternVL, Phi-3-V, and QwenVL; InternVL cannot rescue tasks its vision representations cannot easily complete.Source: §3.2 · p. 5

3.3 VLMs change rank ordering of vision encoders

  • DINOv2 leads almost every direct-readout task, but its disproportionate VLM decline changes encoder rankings and makes CLIP-style pretraining conclusions potentially premature.Source: §3.3 · p. 5

3.4 VLM choices reflect their ‘blind’ baselines

  • With-image and blank-image answer distributions are uncannily similar, and VLM choices largely reflect the biases of their blind baselines.Source: Figure 4 · p. 5, §3.4 · p. 5

4 Analysis of VLM performance

The analyses find that task-relevant visual information remains available through VLM layers and that prompt tuning adds little. LLM tuning yields the largest gains, while attention and answer-distribution analyses point to underuse of visual features and reliance on language priors.

4.1 Vision features maintain their integrity throughout VLM layers

  • Vision representations retain task-relevant information through the projector and LLM, so representation degradation does not explain the performance drop under the tested readouts.The authors describe this preservation as a sufficient, but not necessary, condition for the information to remain accessible.Source: Figure 5 · p. 6, §4.1 · p. 6

4.2 Prompt-tuning results in minimal improvements

  • Adding one prompt prefix improves performance only marginally, and tuning five or ten prefixes does not continue to improve it.The prompt-tuning setup freezes the original VLM and prompt while tuning prefixes on 1,000 task examples.Source: §4.2 · p. 6, Figure 6 · p. 6

4.3 The language model underutilizes its vision representations

  • LLM tuning gives the largest gains over equal-parameter ViT or projector tuning, while attention shifts toward salient regions in correspondence tasks.The attention increase is most salient in layers 4–6; the paper notes weaker gains for 3D Object Awareness.Source: Figure 7 · p. 7, §4.3 · p. 7, Figure 8 · p. 7

4.3.1 Tuning the LLM alleviates the language prior

  • LLM-tuned models produce answer distributions closest to ground truth, while the original VLM already closely matches it on Depth Estimation.The authors conclude that inaccurate predictions reflect both limited attention to salient visual regions and reliance on multiple-choice answer biases.Source: §4.3.1 · p. 7, Table 2 · p. 8
  • The authors use task fine-tuning to localize failures and test whether better LLM use of visual representations can reduce language-prior effects, not as an overarching solution.They report that LLM tuning brings multiple-choice answer distributions closer to ground truth, with Depth Estimation as an exception because the original VLM already closely matched that distribution.Source: §4.3.1 · pp. 7–8, Table 2 · p. 8

5 Related Work

Related work studies VLM architectures and component choices, alongside benchmarks that recast visual tasks as language-mediated questions. This paper focuses on whether those evaluations reflect vision representations and visual perception ability.

5 Related Work

  • VQA-style benchmarks can recast computer-vision tasks such as multi-view correspondence, depth estimation, and object affordances as questions answered from images and text prompts.Vision-centric benchmarks seek to probe visual perception with tasks decoupled from language.Source: §5 · p. 8, §5.2 · p. 9

5.1 Vision-language models

  • Late-fusion VLMs commonly combine a vision backbone, a language model, and an adapter, but differ in backbone, adapter, and instruction-tuning choices.Prior work also explores ensembling backbones to provide complementary visual representations.Source: §5.1 · p. 8

5.2 Studying VLM components

  • Prior studies vary backbones, adapters, instruction tuning, and resolution, but do not agree on whether VLM performance tracks the strength of the vision or language backbone.Some works find stronger LLMs do not necessarily improve VLM scores, while others identify the LLM as more of a bottleneck.Source: §5.2 · pp. 8–9

5.2.1 Vision-centric VLM benchmarks.

  • Vision-centric benchmarks including BLINK and CV-Bench recast visual tasks as multiple-choice questions, but their findings about encoder rankings contrast with direct visual evaluations.CV-Bench reports language-supervised encoders outperforming self-supervised ones, whereas cited visual-evaluation work finds the inverse relationship.Source: §5.2.1 · p. 9

6 Discussion

The discussion attributes poor vision-centric VLM performance to underuse of visual representations and reliance on language priors, rather than to encoder weakness alone. It cautions against using language benchmarks as direct measures of encoder ability while retaining a role for language as an evaluation interface.

6 Discussion

  • VLMs underuse vision-encoder representations on vision-centric tasks and rely on language priors, despite those representations performing well when evaluated independently.Source: §6 · p. 9
  • Encoder ensembling is unlikely to address the underlying failure to use visual representations, and VLM-based evaluations can produce near-chance results and alter encoder rankings.The discussion cautions against inferring encoder capability from such results, including rankings that favor CLIP/SigLIP over DINOv2.Source: §6 · p. 9
  • The finding that integration limits VLM performance does not establish that current vision encoders suffice for every visual task; the authors note ongoing shortcomings in 3D shape inference.The discussion cites humans consistently outperforming vision models on 3D shape inferences.Source: §6 · p. 9
  • Language can specify diverse and complex visual tasks, but language-benchmark results may underestimate or misrepresent vision-encoder capabilities.Source: §6 · p. 9

A Overview of task examples

The examples illustrate failure cases shared by VLMs and vision encoders, including confusion from similar local structure and difficulty encoding small objects or parts.

A Overview of task examples

  • Both VLMs and vision encoders can confuse similar local structure in correspondence tasks or miss small objects and object parts.Examples include stone walls and colorful squares, dog ears, and small vehicles.Source: Figure 9 · p. 14

B.1 Evaluation setup

The evaluation tests multi-image VLMs on 3D Object Awareness, using each model’s native multi-image prompting. These architectures offer an upper-bound view on some tasks but lack released checkpoints with multiple vision backbones for direct comparison.

B.1 Evaluation setup

  • The evaluated architectures accept multiple images natively, avoiding image stitching but lacking released checkpoints with multiple vision backbones for direct comparison.The authors focus on 3D Object Awareness to examine their performance as an upper bound on some vision-centric tasks.Source: §B.1 · p. 15
  • LLaVA-Next, Phi-3, and Qwen-VL are evaluated on 3D Object Awareness using each model’s codebase-specific multi-image prompting method.The prompt asks which image is the odd one out when the others show the same object from different angles.Source: §B.1 · p. 15
  • The shared prompt is adjusted for the number of input images and requests an answer by choice letter.It asks the model to identify the image that does not depict the same object from a different angle as the others.Source: §B.1 · p. 15

B.2 Results

In the multi-image 3D Object Awareness evaluation, Qwen-VL is the only tested model reported to perform considerably above chance, and one-shot prompting improves its performance. These results do not account for the visual-readout/VLM gap in the main paper.

B.2 Results

  • Qwen-VL performs considerably better than random chance (= 0.325), and improves slightly in the 1-shot setting, but these results do not account for the main-paper visual-VLM gap.The comparison covers LLaVA-Next, Phi-3, Qwen-VL, and Qwen-VL 1 Shot on 3D Object Awareness.Source: Table 3 · p. 15, §B.2 · p. 15
  • Only Qwen-VL performs considerably better than random chance (= 0.325) across the explored models and tasks.The table covers LLaVA-Next, Phi-3, Qwen-VL, and Qwen-VL 1 Shot on 3D Object Awareness.Source: §B.2 · p. 15, Table 3 · p. 15
  • Using one in-context example further improves the best-performing approach, Qwen-VL.The improvement still does not account for the gap between VLM and visual evaluation strategies.Source: §B.2 · p. 15, Table 3 · p. 15

C Evaluation details

The evaluation details specify the VLM components, preprocessing, and task-specific depth readouts. For depth estimation, the visual encoder is assessed with a trained decoder, while VLM examples are formulated as object-distance choices.

C Evaluation details

  • The evaluation uses four VLMs paired with corresponding vision backbones; each combines a backbone, Vicuna v1.5 language model, and 2-layer MLP projector.The vision and language components are frozen, while the projector is trained on the LLaVA v1.5 pre-training set.Source: C Evaluation details · p. 15
  • VLM images use default letterbox padding to target dimensions, whereas encoder-only inputs are naively resized with model-specific normalization.VLM answer letters are extracted using fuzzy matching to allow some output-format variation.Source: C Evaluation details · pp. 15–16

C.1 Depth Estimation

  • Depth Estimation asks which of two boxed objects is closer to the camera, using prompts from Tong et al. (2024a).Images are resized to 1024px high for this evaluation.Source: §C.1 · p. 16
  • The visual depth readout trains a DPT decoder on NYUv2, then compares average predicted depth within the two query boxes.The boxes may include non-object pixels, which the authors note could underestimate visual-readout performance.Source: §C.1 · p. 16
  • Additional VLM fine-tuning examples are made by randomly sampling Omni3D images and boxing two labeled objects.The vision readout predicts which object is closer by comparing the scalar depths averaged within each box.Source: §C.1 · p. 16

C.2 Correspondence-based Evaluations

The correspondence-based evaluations use multiple-choice visual matching tasks and task-specific vision readouts. The 3D Object Awareness evaluation compares views of an object, while Art Style uses texture-focused features to compare paintings.

C.2 Correspondence-based Evaluations

  • Correspondence-based tasks ask VLMs to match a reference point to a choice, while encoder readouts use feature similarity to select a match.The section covers semantic correspondence, object affordance, and low-level matching; additional fine-tuning examples use sampled keypoints and the benchmark-style circles and labels.Source: §C.2 · pp. 16–17

C.3 3D Object Awareness

  • 3D Object Awareness asks which view is the odd one out; the vision readout selects the image with the lowest average pairwise CLS-token similarity.Fine-tuning examples use ShapeNet renderings in a three-way multiple-choice task; four-image cases add a fourth option.Source: §C.3 · p. 17

C.4 Art Style

  • Art Style VLM prompts compare paintings, while the vision readout uses Gram matrices of ViT patch features to capture texture-focused style information.The method can produce blocky style-transferred outputs, but the authors retain it because it captures the texture information needed for the task; fine-tuning pairs are drawn from WikiArt.Source: §C.4 · p. 17

C.5 Few-shot evaluation details

The appendix details few-shot evaluation procedures, visualizes attention differences on correspondence tasks, and presents additional benchmark failure examples.

C.5 Few-shot evaluation details

  • For 3D Object Awareness, a logistic regression on pairwise CLS-embedding differences trains on 75% of each condition and evaluates one data point, repeated 10 times.The reported standard deviation is zero.Source: §C.5 · pp. 17–18
  • Neural style transfer adjusts image pixels to minimize mean squared error between the original image and style-reference features, transferring texture while retaining landscape structure.The figure shows the original landscape, style reference, and resulting image.Source: Figure 10 · p. 18
  • For VLM few-shot evaluation, one prompt embedding is trained on 75% of each condition for 10 epochs, with 10 trials per data point.The embedding is prepended after the <BOS> embedding; optimization uses Adam at a 3e-3 learning rate.Source: §C.5 · p. 18

C.6 Additional attention visualizations

  • Attention-difference visualizations show AttentionOrig - AttentionLLM for Functional Correspondence and Semantic Correspondence.Source: Figure 11 · p. 19, §C.6 · p. 19

D Additional failure examples

Related papers

Loading 2506.08008v1…