PaperlayerView exact revision on arXiv

Source-linked AI summary

Retrieval Heads Meet Vision: Uncovering How VLMs Locate and Extract Visual Information

Chanho Park, Daehyeon Choi, Jihyun Lee, Minhyuk Sung

arXiv:2608.27417v1cs.CV

TL;DR

The paper asks whether VLMs contain causal, sparse, and general visual-retrieval heads analogous to retrieval heads in LLMs. It introduces VRHs and detects them by scoring attention from output tokens to ground-truth visual regions, finding that masking 20 VRHs can reduce grounding accuracy by up to 80 percentage points while preserving output format but corrupting localization. The heads also generalize across visual-reference tasks and transfer across VLMs sharing an LLM backbone.

  • Problem

    The internal mechanism by which VLMs locate text-referred image regions and route the corresponding visual evidence to their outputs is not understood.

  • Method

    The paper introduces VRHs and identifies them by scoring attention from output prediction tokens to ground-truth visual regions in a unified head-scoring framework.

  • Results

    Masking the top 20 VRHs, about 1.7–2.6% of heads, reduces grounding accuracy by up to 80 percentage points, while random-head masking has little effect across eleven VLMs and five benchmarks.

  • Takeaways & Limitations

    VRHs support visual reference resolution beyond object grounding, remaining causal for attribute recognition, spatial reasoning, counting, and visual math and transferring across related VLMs.

  • Takeaways & Limitations

    The attention-weight analysis primarily reveals where visual evidence is routed, not what information is read out or transformed, and experiments focus on single-image inputs.

Abstract

from arXiv · show

Vision-language models (VLMs) can locate an image region referred to by a text prompt and route the corresponding visual evidence to the output, yet the internal mechanism behind this behavior is not understood. Inspired by retrieval heads in large language models, we ask whether VLMs contain an analogous mechanism for visual retrieval. We answer affirmatively by introducing Visual Retrieval Heads (VRHs), a small subset of attention heads (about 1.7-2.6%) that are causally responsible for grounding text descriptions to image regions. To find them, we recast existing head-scoring methods under a unified design space over query tokens, key aggregation, and cross-sample aggregation. We then show that scoring attention from output prediction tokens with a sum over the ground-truth referent region most reliably identifies causal heads. Across eleven VLMs and five referring-expression benchmarks, masking only the top 20 VRHs reduces grounding accuracy by up to 80 percentage points, while masking the same number of random heads has little effect. Beyond replicating the causal-sparse-universal triad established for text retrieval heads, VRHs exhibit several properties not previously reported: they generalize across visual reference tasks, remaining causal on attribute, spatial, counting, and visual-math benchmarks despite being discovered through bounding-box prediction; they are functionally specific, preserving output format while corrupting localization; and they are architecturally shared, transferring causally across VLMs that share an LLM backbone but differ in vision encoder, projector, and instruction tuning.

1 Introduction

The paper introduces Visual Retrieval Heads (VRHs), a sparse and causal attention-head mechanism for grounding text descriptions to image regions. It identifies VRHs through visual-grounding attention scores and shows that they generalize across models and visual-reference tasks.

  • VRHs are a small subset of attention heads causally responsible for grounding text descriptions to image regions.
  • Visual grounding makes the retrieval target spatially explicit because the ground-truth bounding box identifies the visual tokens the model should consult.
  • Existing head-scoring methods can be unified by choices of query tokens, key aggregation, and cross-query aggregation.
  • Masking the top 20 VRHs, about 1.7–2.6% of all heads, reduces grounding accuracy by up to 80 percentage points, whereas masking random heads has negligible effect.
  • VRHs remain causal across attribute, spatial reasoning, counting, and visual-math benchmarks despite discovery through bounding-box prediction.
  • VRHs transfer causally across VLMs sharing an LLM backbone but differing in vision encoder, projector, and instruction-tuning recipe.

2 Related Work

Prior specialized-head studies primarily target textual evidence, including text rendered in images, while general-image localization studies do not establish causal necessity. The paper positions VRHs as the first general-image visual-retrieval heads validated causally.

  • Retrieval-head methods identify specialized heads from inference-time attention patterns in LLMs and VLMs.
  • OCR Heads and VERA retrieve textual evidence from images, but their targets remain strings in visual-text documents.
  • Localization Heads address general images but characterize attention-map alignment without verifying causal effects on retrieval performance.
  • The paper reports that VRHs are the first attention heads specialized for visual retrieval from general images while exhibiting causality in visual-retrieval performance.

3 Toward Identifying Visual Retrieval Heads

The paper introduces Visual Retrieval Heads (VRHs), a sparse and causal mechanism for retrieving visual information relevant to text prompts, and develops a unified framework for detecting them. It uses visual grounding to compare scoring variants and validates the selected heads through targeted masking.

  • Visual Retrieval Heads: VRHs are a small subset of VLM attention heads causally responsible for retrieving visual information relevant to an input text prompt.The paper characterizes them as causal, universal, and sparse, and presents them as specialized for visual retrieval from general images.
  • 3.1 Visual Grounding as Probing Task: Visual grounding makes the retrieval target spatially explicit because the ground-truth bounding box identifies the visual tokens corresponding to the referred object.A prediction is correct when its predicted box has IoU with the ground-truth box above the task threshold, typically 0.5.
  • 3.2 Unified Formulation of Head-Scoring Methods: The unified head-scoring formulation varies the query-token set, key aggregation, and cross-sample aggregation, producing 23 candidate scoring variants.Query choices include input or output tokens; key choices include argmax or region-sum aggregation; cross-sample choices include average or frequency aggregation.
  • 3.3 Causal Validation of Head-Scoring Variants: The selected default, (Qout s , Φsum, Ωmean), scores attention from output tokens to the target region and identifies heads whose masking most consistently reduces grounding accuracy.Causal validation across four VLMs shows output query tokens are the most important design choice, while attention alignment alone does not establish functional necessity.
  • 3.3 Causal Validation of Head-Scoring Variants: Masking VRHs produces larger grounding-accuracy drops than masking random heads or previously identified specialized heads on RefCOCO.The comparison includes Retrieval Heads, OCR Heads, VERA, and Localization Heads, supporting VRHs as a distinct causally important set.

4 Visual Retrieval Heads Are Universal and Sparse

Across eleven VLMs and five grounding datasets, VRHs consistently form a sparse, causal visual-retrieval mechanism rather than a model- or dataset-specific artifact.

  • Universality: VRHs emerge as a causal visual-retrieval unit across eleven VLMs, eight language-model backbones, and five grounding datasets.Masking top-ranked VRHs consistently reduces grounding accuracy, unlike masking the same number of random heads.
  • Sparsity: Fewer than 15% of heads receive non-negligible VRH scores, while only 0–4% receive high scores above 5×10^-3.The thresholds and ranges vary by model, indicating that visual retrieval is concentrated in a small specialized subset.
  • Sparsity: The RefCOCO score distribution across four VLMs shows that only a small fraction of heads receive non-negligible scores.The distribution provides a direct visual summary of VRH sparsity.

5 Characterizing Visual Retrieval Heads

VRHs extend beyond grounding to diverse visual-reference tasks, selectively disrupt localization, share a language-backbone circuit, and remain stable across detection choices and sample sizes.

  • Universality: Masking top-k VRHs causes large grounding-accuracy drops across four VLMs and three benchmark families, while random-head masking has little effect.The figure varies the number of masked heads k and compares VRH masking with random-head controls.
  • 5.1 VRHs Generalize Across Visual Reference Tasks: VRHs detected on RefCOCO remain causal across seven VQA benchmarks covering attributes, spatial reasoning, counting, vision-centric understanding, and visual math.Random-head masking has little effect across these tasks and models.
  • 5.2 VRHs are Functionally Specific: Masking VRHs increases mislocation while parse failures remain rare, preserving bounding-box syntax but corrupting visual localization.This is the visual analogue of a fluent-but-not-factual failure mode.
  • 5.3 VRHs are Architecturally Shared: VRHs detected in one VLM remain causally critical in another sharing its LLM backbone despite differences in vision encoder, projector, and instruction tuning.The result suggests a reusable language-model circuit rather than an idiosyncratic VLM implementation.
  • 5.4 VRHs are Stably Detectable: VRH detection remains stable across output-token choices: masking heads found from one random output token or all output tokens causes comparable 7.2% and 7.5% accuracy drops.Both choices outperform masking the same number of random heads.
  • 5.4 VRHs are Stably Detectable: Only a few grounding annotations recover nearly the same top-ranked VRH set as a full 200-example scoring set.This indicates a repeatable attention signature rather than a pattern requiring extensive averaging.

6 Conclusion

The paper concludes that VRHs are sparse, causal, and broadly reusable for visual reference resolution, while identifying mechanistic and multimodal scope limits for future work.

  • Conclusion: VRHs are causally necessary for visual reference behavior and remain important for attribute recognition, spatial reasoning, counting, and visual math.Their role therefore extends beyond the bounding-box grounding task used for detection.
  • Limitations and future work: The analysis primarily tracks where visual evidence is routed, not what information is read out or transformed.The authors propose activation patching and causal mediation for finer-grained mechanistic analysis.
  • Limitations and future work: Experiments focus on single-image VLM inputs, leaving multi-image, video, and interleaved image-text contexts for future study.These richer settings may reveal additional visual-evidence routing behavior.
  • Visual evidence routing: Attention visualizations preserve concentration on ground-truth referents when only VRHs are kept but largely lose it when VRHs are excluded.This pattern appears on RefCOCO and Toloka grounding examples.
  • Visual question answering: On VAW and Spatial457, keeping only VRHs preserves attention to visual evidence, whereas masking them yields fluent but visually unfaithful answers.The failure preserves answer fluency while losing visual grounding.
  • Downstream use: VRHs can also serve as targets for DPO-based downstream optimization using masked versus unmasked policies.The appendix applies this idea to visual grounding rather than treating VRHs only as an analytic construct.

B.2 Results

VRHs are actionable beyond analysis: using them to construct DPO negatives produces modest but consistent visual-grounding gains over the evaluated benchmarks.

  • Results: 0.78 percentage-points average improvement is observed across three visual grounding benchmarks when VRH-based negatives are used for DPO.The result is described as modest but consistent.
  • Results: VRH dropping provides stronger downstream improvements than random-head dropping as the DPO negative policy.Perturbing causally relevant heads supplies a more informative preference signal for optimization.
  • Results: The reported main-paper results average across RefCOCO/+/g, while the appendix provides per-dataset breakdowns and per-seed variability.The detailed breakdown supplements the averaged results.

C.1 Per-dataset Visual Grounding Results under Head Masking

This section reports per-dataset grounding accuracy under VRH and random-head masking and documents the evaluated models and masked-head selections. Results are organized across five referring-expression benchmarks and four VLMs.

  • Per-dataset results: Table A1 reports visual grounding accuracy for RefCOCO, RefCOCO+, RefCOCOg, Toloka, and RefSpatialBench across four VLMs and multiple masking levels.The masking levels are k ∈ {0, 1, 3, 5, 10, 20} for both VRHs and random heads.
  • VRH selections: The appendix lists the top-20 VRHs separately for each VLM–benchmark pair, ordered by descending head score.These lists are provided for RefCOCO, RefCOCO+, RefCOCOg, Toloka, and RefSpatialBench.
  • Evaluated models: The evaluation covers Qwen2.5-VL-7B, Qwen3-VL-8B, InternVL3-8B, and InternVL3.5-8B from two model families.The models form same-backbone pairs that differ in vision encoder, projector, and instruction-tuning recipe.
  • Model architecture: Architectural specifications report layers, heads per layer, and total heads used to calculate sparsity ratios.For example, 20 / 784 = 2.6% for Qwen2.5-VL-7B.

E.2 Benchmarks

The evaluation uses five referring-expression grounding datasets and seven VQA benchmarks spanning attributes, spatial reasoning, vision-centric understanding, counting, and visual math. Causal head contribution is measured by zeroing an individual attention head during the forward pass.

  • Grounding benchmarks: Five grounding datasets cover standard COCO-derived expressions, spatial relations, and diverse crowd-sourced referring styles.The datasets are RefCOCO/+/g, RefSpatialBench, and Toloka VQA.
  • Grounding metric: Grounding predictions are correct when IoU(ˆB_s, B_s) ≥ τ = 0.5.This threshold is used consistently with the main paper.
  • Cross-task benchmarks: Seven VQA benchmarks test attribute prediction, spatial reasoning, vision-centric understanding, counting, and visual math.They include VAW, Spatial457, SpatialRGPT, CV-Bench, MMStar, CountBenchQA, and MathVista.
  • Causal evaluation: Ablation zeroes one head’s attention output before residual addition while leaving other heads, MLPs, and layer norms unchanged.This convention matches the masking procedure used for text retrieval heads.
  • Generation and hardware: All evaluations use greedy decoding and the default chat template, with grounding boxes parsed from model-specific output formats.Experiments were conducted on a single NVIDIA RTX A6000 with 48 GB VRAM.

F Ablation on the Combined Query Token Set

The combined query-token setting uses the union of query tokens and achieves comparable causal effectiveness for some models, but the original output-token setting is more robust and efficient across the evaluated models.

  • Combined query set: Using the union of query-token sets creates a VRH detection setting that can be compared with the original Qout_s setting.The combined setting uses a superset of the original query tokens.
  • Causal effectiveness: The combined setting achieves comparable causal effectiveness for some models.Its larger query-token set does not uniformly improve causal head detection.
  • Model comparison: The original Qout_s setting performs better on Qwen3-VL, InternVL3, and InternVL3.5, with a particularly noticeable gap on Qwen3-VL.These comparisons are reported for the four-model ablation.
  • Robustness and efficiency: The original setting is more robust across models while using substantially fewer query tokens for head detection.The appendix interprets this as an efficiency advantage over the combined setting.
  • Scoring ablation: At k ∈ {10, 20, 50}, the original scoring design consistently ranks first or second in causal effect, whereas k=5 results are weaker or inconsistent.The ablation compares Qout_s, Φsum, and Ωmean against one-component variants and random-head masking.

H Head Detection and Evaluation Splits

The default protocol separates head detection from evaluation using disjoint grounding samples, and cross-task evaluations use independently sampled non-grounding benchmarks. VRH detection costs vary with visual-token count and attention implementation.

  • Split protocol: The default protocol uses a fixed detection set and evaluates masked models on the remaining grounding samples.For RefCOCO, 200 validation examples are used for detection and 8,611 remaining examples for evaluation.
  • Split validation: Results do not meaningfully differ when the 200 RefCOCO head-scoring samples are included in evaluation rather than excluded.The paper nevertheless adopts disjoint detection and evaluation sets by default.
  • Cross-task evaluation: Heads detected from 200 RefCOCO samples are also evaluated on separate non-grounding benchmarks.This cross-task setup reduces concerns about similarity between scoring and evaluation data.
  • Detection cost: VRH detection cost grows with the number of visual tokens, which varies with input resolution and image count.Peak memory and computation time are measured using both eager and fused attention.
  • Cost evaluation: Table A11 measures VRH detection memory and computation time on InternVL3-8B with dynamic tiling using a 48 GB A6000 GPU.The measurements vary visual-token counts and compare eager with fused attention.

J Evaluated VLMs: Protocols and Per-Model Results

The evaluation extends VRH detection and masking across additional VLMs using a fixed RefCOCO protocol, and tests whether VRHs transfer between models sharing an LLM backbone.

  • Evaluation protocol: Seven additional VLMs detect VRHs on 200 RefCOCO samples and evaluate masking on a disjoint 500-sample set.The random-head control masks k=20 heads outside the detected VRH set.
  • Evaluation protocol: The random-head control uses the same k=20 masking budget as the VRH intervention.
  • Cross-model transfer: Magma-8B and LLaVA-NeXT-Llama3-8B share Llama-3-based language backbones and are evaluated for cross-model VRH transfer.The corresponding score maps and full k-sweep are reported in Figure 7, with exact values in Table A13.
  • Cross-model transfer: Cross-masking the top 20 VRHs causes a significant drop in RefCOCO visual grounding accuracy in both transfer directions.
Loading 2608.27417v1…