Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding

1. Introduction

• Visual grounding localizes image objects from textual descriptions, while existing LVLM approaches require fine-tuning and added components to produce boxes or masks.The paper presents its identification of localization properties in specific LVLM attention heads as the first of its kind.Source: §1 · pp. 1–2
• Average LVLM text-to-image attention maps are sparse and noisy, but a small subset of heads consistently highlights regions corresponding to the referring text.The paper calls these heads localization heads; examples include L14 H24 and L14 H13 in LLaVA-1.5-7B [35].Source: Figure 1 · p. 1, §1 · p. 2
• Localization heads are selected by image-attention sum and spatial entropy, identifying heads that attend to the image and focus on a specific region.The authors validate that selected heads consistently capture objects associated with the text.Source: §1 · p. 2
• Attention maps from a few localization heads in frozen LVLMs predict bounding boxes or masks without additional fine-tuning; only three heads are sufficient.The authors report this as a contrast to methods that fine-tune LVLMs and add components for grounding outputs.Source: §1 · pp. 1–2
• Across ten LVLMs, the framework outperforms existing training-free methods and performs comparably to fine-tuned LVLM grounding methods.The evaluation covers models with varying parameter counts, architectures, and training datasets.Source: §1 · p. 2

Visual Grounding

• Visual grounding localizes image regions from free-form language, with REC producing bounding boxes and RES requiring pixel-level masks.These tasks extend detection and segmentation to more realistic language-guided scenarios.Source: §2 · p. 2

Application of LVLMs in Grounding Tasks

• Existing LVLM grounding methods use training data and added localization components, whereas this work directly uses LVLMs for training-free grounding.LISA [26] adds a [SEG] token and mask decoder, while F-LMM [64] trains mask-refinement modules despite using frozen-LVLM attention.Source: §2 · p. 3

Training-Free Visual Grounding

• Training-free grounding has used CLIP proposals ranked by text similarity or diffusion-process residues such as attention maps; this work introduces the first LVLM-based framework.CLIP approaches select bounding boxes [52] or masks [53] [71], while diffusion-model approaches predict segmentation masks [3] [40].Source: §2 · p. 3

• An LVLM encodes an image into visual embeddings and text into token embeddings, concatenates both sequences, and feeds them to the language model.The visual sequence has P^2 tokens and the text sequence has L tokens.Source: §3 · p. 3
• Decoder blocks contain self-attention and feed-forward modules; attention heads enable token interactions through query, key, and value projections, with weights reflecting query-key similarity.Source: §3 · p. 3
• Because autoregressive decoding lets the final input text token encapsulate sentence context, its query is used to inspect attention to image tokens; the first P^2 weights are denoted LℓHh.For instance, L5 H3 denotes the third attention head in the fifth layer.Source: §3 · p. 3

4.1. Criteria to Find Localization Heads

• Localization heads are selected by requiring high image-attention sum and low spatial entropy, favoring heads whose attention forms spatially compact clusters.Spatial entropy is calculated from connected components in an attention map binarized at its mean; the attention-sum threshold uses maximum curvature and results remain robust across reasonable thresholds.Source: §4.1 · p. 4

4.2. Finding Localization Heads via Criteria

• Spearman correlations between selection frequency and IoU exceed 0.7 for all LVLMs, identifying consistently top-ranked heads as localization heads and an innate LVLM property.Higher selection frequency tends to correspond to higher average IoU between pseudo-masks and ground-truth masks.Source: Figure 6 · p. 5, §4.2 · p. 5
• Selection frequency measures how often each head exhibits the 10-lowest spatial entropy across samples and determines its rank; pseudo-mask IoU against ground-truth masks is analyzed only for heads selected at least 1% of the time.For LLaVA-1.5-7B [35], head L14 H24 ranks first, followed by L14 H13 in second place.Source: §4.2 · p. 5

• The method uses localization heads to solve visual-grounding tasks without additional training.It builds on criteria for identifying text-referring localization heads.Source: §5 · p. 5
• For a given LVLM, the method ranks heads by selection frequency, selects the k highest-ranked heads, and extracts their attention maps for the input image-text pair.Source: §5 · p. 5
• Gaussian smoothing reduces potential random noise in each attention map; the maps are summed and binarized into a pseudo-mask, whose largest enclosing rectangle yields a bounding box that can prompt SAM for segmentation.Source: §5 · pp. 5–6

6.2. Main Results

• The framework substantially improves on training-free baselines and performs comparably to fine-tuned LVLMs on REC and RES, with results on par with Shikra, Ferret, and LISA.The REC comparison is with Shikra and Ferret using the same base LLMs; the RES comparison is with LISA.Source: Table 2 · p. 6, §6.2 · p. 7

6.1. Experimental Setup

• The experiments evaluate REC, RES, and ReasonSeg using RefCOCO, RefCOCO+, RefCOCOg, and ReasonSeg, with Acc@0.5 for REC and cIoU for segmentation.The approach is applied across ten LVLMs with k = 3, and compared against fine-tuning-based and training-free methods.Source: §6.1 · pp. 6–7, §6 · p. 6

6.2. Main Results

• On ReasonSeg, the method performs comparably to LISA and sometimes outperforms it when both use LLaVA-1.5.The authors interpret this as evidence that localization heads extend to grounding tasks requiring complex reasoning or world knowledge.Source: Table 3 · p. 7, §6.2 · p. 7
• Grounding performance improves with larger models and with architecture or training-data updates, including the progression from LLaVA to LLaVA-1.5.The evaluated models range from 1.3B to 13B parameters.Source: §6.1 · p. 6, §6.2 · p. 7
• LVLMs identify referred regions in challenging cases, including similar objects and off-center targets, and predict boxes or masks without CLIP-style post-processing.The paper attributes this capability to LVLM text comprehension; Fig. 8 presents qualitative comparisons.Source: §6.2 · p. 7, Figure 8 · p. 7

6.3. Ablation Studies

• Performance generally improves as the number of localization heads rises from one to three, but adding more does not guarantee gains and may introduce noise or redundancy.The optimal-k trend remains consistent across LVLMs, suggesting similar numbers of heads localize referred objects across differing architectures.Source: §6.3 · pp. 7–8, Table 4 · p. 8
• Using either attention sum or spatial entropy alone significantly reduces performance, while greedy per-sample selection performs worse than fixed selection based on selection frequency.The authors explain that frequency-based selection favors heads that consistently focus on text-related regions rather than merely producing localized clusters.Source: §6.3 · p. 8, Table 5 · p. 8

6.4. Understanding LVLMs When They Fail

• In a failure to identify the third banana from the right, a localization-head attention map focuses on both the third and fourth bananas, exposing difficulty pinpointing the exact object.The authors present these maps as a way to understand where an LVLM focuses when grounding fails.Source: §6.4 · p. 8, §6.3 · p. 8

7. Conclusion

• The proposed training-free framework achieves competitive visual grounding performance compared with fine-tuning-based methods.Source: §7 · p. 8
• Criteria identify localization heads across various LVLMs that show strong grounding capabilities in response to textual queries.Source: §7 · p. 8
• The framework assembles text-to-image attention maps from a few localization heads to predict image regions relevant to text queries.Source: §7 · p. 8
• Its predictions include bounding boxes and segmentation masks for text-relevant image regions.Source: §7 · p. 8
• The authors conclude that LVLMs can act as text-referring visual localizers through inherent properties of their attention mechanisms.Source: §7 · p. 8
• The work aims to open new possibilities for analyzing and using LVLM attention mechanisms.Source: §7 · p. 8

A. Experimental Details

• Experiments use only model inference, without fine-tuning or training, on a single NVIDIA GeForce RTX A6000 48GB GPU.Source: A. Experimental Details · p. 13
• Head analysis uses RefCOCO training data to prevent validation leakage, estimating selection frequency from 1,000 image-text pairs averaged over five trials and mean-thresholded attention-mask IoU over 1,000 pairs.Source: A. Experimental Details · p. 13
• Evaluation datasets include RefCOCO, RefCOCO+, RefCOCOg, and ReasonSeg, with concise expressions in the first two and more descriptive expressions in RefCOCOg.RefCOCO has 19,994 images and 142,210 expressions; RefCOCO+ has 19,992 images and 141,564 expressions; RefCOCOg has 26,771 images and 104,560 expressions. ReasonSeg has 1,218 image-instruction-mask samples, split into 239 training, 200 validation, and 779 test samples.Source: A. Experimental Details · p. 13
• Referring-expression comprehension and segmentation use RefCOCO, RefCOCO+, and RefCOCOg validation and test splits, with all evaluations following the UNC split.The evaluated splits are RefCOCO validation, test-A, and test-B; RefCOCO+ validation, testA, and test-B; and RefCOCOg validation and test.Source: A. Experimental Details · p. 13
• ReasonSeg evaluates validation and test performance on short, long, and overall queries, which require more contextual and world knowledge than simple referring phrases, following LISA [26].Queries include nuanced descriptions and longer sentences; the benchmark is described as more challenging than referring-expression segmentation.Source: A. Experimental Details · p. 13
• Ablations select heads using either the highest Sℓ,h_img or lowest H(Aℓ,h) values, while greedy validation chooses the top-k heads per sample using either criterion.The criterion-only comparisons select 10 heads and measure their selection frequency; greedy selection does not use overall selection frequency.Source: A. Experimental Details · p. 13

B.1. Spatial Entropy

• Spatial entropy thresholds attention above its mean, identifies 8-connected components, and uses normalized component areas in Shannon entropy to measure attention dispersion from 0 (focused) to 1 (evenly distributed).The calculation follows prior work on spatial entropy in vision-transformer image attention maps.Source: §B.1 · pp. 13–14

B.2. Details of Our Framework

• The framework binarizes attention above its mean, smooths it with a 7-pixel kernel at σ = 1.0, then uses the largest convex hull to produce a tight bounding box [13].When multiple convex hulls occur, only the largest is retained.Source: §B.2 · p. 14

C.1. Extended Analysis Across More LVLMs

• Across InternVL [8], LLaVA, Mini-Gemini [30], ShareGPT4V [7], and Yi-VL [69], relatively few heads contribute strongly to text-image interaction, consistent with the average-attention-sum trend.The analysis extends the main paper’s average Simg attention results to these additional LVLMs.Source: §C.1 · p. 14
• Across the additional LVLMs, attention-head selection frequency and its relationship to IoU support the same localization-head observations.The analysis reports selection frequency and selection-frequency-rank-versus-IoU plots for each head.Source: §C.1 · p. 14

C.2. Robustness of Localization Head Selection

• The same localization heads are selected across plausible threshold values τ and the maximum-curvature threshold, indicating selection is robust to τ.Maximum curvature is used to reduce manual tuning; other plausible thresholds are also tested.Source: §C.2 · p. 14
• The same top-3 localization heads are consistently identified for N from 1 to 14, relative to the default N = 10 selection-frequency setting.The default analysis selects the 10 heads with lowest H(Aℓ,h) values across 1,000 image-text pairs.Source: §C.2 · p. 14

D.1. Comparison with Baseline Models

• Baseline LVLMs have poor localization accuracy because they focus on describing objects, and their localization heads provide only indirect support during ordinary text generation.Accurate object or region coordinates are difficult to produce unless localization-head information is explicitly scrutinized; integrating it into the proposed method makes the head practically useful.Source: §D.1 · pp. 14–15

D.2. Comparision with F-LMM

• On RES, F-LMM [64] outperforms our method with DeepSeek-VL-1.3B [38] and Mini-Gemini-2B [30], while results are comparable with a slight gap for DeepSeek-VL-7B [38] and LLaVA-1.5-7B [35].The authors interpret this as evidence that localization heads have competitive potential against specialized mask decoders, especially in larger LVLMs.Source: Table 7 · p. 15, §D.2 · p. 15

D.3. Gaussian Smoothing Ablation

• The Gaussian-smoothing ablation evaluates LLaVA-1.5-13B [35] on the RefCOCO validation set (UNC split) across standard deviations σ and kernel sizes κ.The study varies smoothing parameters to assess framework robustness.Source: §D.3 · p. 15
• Gaussian smoothing is an auxiliary post-processing step that refines localization-head attention maps.Source: §D.3 · p. 15
• Gaussian smoothing consistently improves performance in almost all tested cases, with robustness across the tested σ and κ values.Source: §D.3 · p. 15
• Without Gaussian smoothing, performance remains competitive, dropping only 1.9% from the best case.The unsmoothed settings are σ = 0 or κ = 1.Source: §D.3 · p. 15

D.4. Multi-Object Grounding Tasks

• On the PNG benchmark, noun-token-based attention maps achieve comparable results and improve on F-LMM.The approach uses spaCy to extract noun tokens for generating attention maps.Source: §D.4 · p. 15
• The authors suggest this approach may extend to other tasks.They identify this as a promising direction, not a demonstrated result.Source: §D.4 · p. 15

E. More Qualitative Results

• Only a few selected localization heads accurately localize query-referred objects in REC, RES, and reasoning segmentation results.The qualitative results are shown in Figs. 15–17.Source: E. More Qualitative Results · p. 16
• The qualitative evaluation covers 10 LVLMs [7] [8] [30] [35] [38] [69] with parameter counts from 1.3B to 13B.Source: E. More Qualitative Results · p. 16
• The framework effectively localizes objects across varied scenarios in its qualitative results.Source: E. More Qualitative Results · p. 16

F.1. Real World Application

• In real-world images, localization heads capture the region or object of interest from the provided expressions, demonstrating robustness across data types.This is illustrated in Fig. 18.Source: §F.1 · p. 16

F.2. Image Editing

• For image inpainting, the frozen LVLM generates an expression-matched segmentation mask, which is combined with an additional text prompt as input to SDXL.The process is illustrated in Fig. 19.Source: §F.2 · p. 16
• Masks produced by a small number of localization heads can guide diffusion models, enabling their use in image editing.Source: §F.2 · p. 16

G. Limitations

• Multi-object grounding is a demonstrated possibility, but a formal, streamlined pipeline that is practical, user-friendly, and adaptable remains undeveloped.The authors identify this as an avenue for future research.Source: G. Limitations · p. 16, Figure 10 · p. 16
• The framework is less suitable for LVLMs or methods that pool away spatial information [1] [19] [28], because extracting attention maps requires reconstructing image-token input order.The authors leave applying the framework to these methods for future exploration.Source: G. Limitations · p. 16