Source-linked AI summary
Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding
Seil Kang, Jinyeong Kim, Junhyeok Kim, Seong Jae Hwang
TL;DR
Visual grounding requires localizing image regions described by free-form text, yet LVLMs generally need fine-tuning and extra components to output boxes or masks. This paper identifies a few localization heads in frozen LVLMs and uses their text-to-image attention maps for training-free grounding. Only three heads suffice for competitive localization, and the framework performs comparably to fine-tuned methods across evaluated LVLMs and grounding tasks.
Problem
Existing LVLM-based visual grounding methods require fine-tuning and additional components to generate bounding boxes or segmentation masks.
Method
The framework identifies localization heads using image-attention concentration and spatial entropy, then assembles their text-to-image attention maps to predict boxes or masks.
Results
Only three localization heads are sufficient for localization, while the framework outperforms training-free baselines and performs comparably to fine-tuned methods.
Takeaways & Limitations
Frozen LVLMs can act as text-referring localizers through inherent attention mechanisms without additional training.
Takeaways & Limitations
Average LVLM attention maps can be sparse and noisy, and head selection uses a threshold whose robustness is evaluated across a range of values.
Abstract
from arXiv · showhide
Visual grounding seeks to localize the image region corresponding to a free-form text description. Recently, the strong multimodal capabilities of Large Vision-Language Models (LVLMs) have driven substantial improvements in visual grounding, though they inevitably require fine-tuning and additional model components to explicitly generate bounding boxes or segmentation masks. However, we discover that a few attention heads in frozen LVLMs demonstrate strong visual grounding capabilities. We refer to these heads, which consistently capture object locations related to text semantics, as localization heads. Using localization heads, we introduce a straightforward and effective training-free visual grounding framework that utilizes text-to-image attention maps from localization heads to identify the target objects. Surprisingly, only three out of thousands of attention heads are sufficient to achieve competitive localization performance compared to existing LVLM-based visual grounding methods that require fine-tuning. Our findings suggest that LVLMs can innately ground objects based on a deep comprehension of the text-image relationship, as they implicitly focus on relevant image regions to generate informative text outputs. All the source codes will be made available to the public.
1. Introduction
LVLMs typically require fine-tuning and added components for visual grounding, but this work finds that a few frozen-model attention heads inherently localize text-relevant image regions. It introduces a training-free framework that uses these localization heads to predict boxes or masks and achieves performance comparable to fine-tuned methods.
- Visual grounding localizes objects described by text, while existing LVLM methods typically fine-tune models and add components to generate boxes or masks.
- Average LVLM text-to-image attention maps are sparse and noisy, making them difficult to use directly for pinpointing relevant objects.
- Specific attention heads consistently produce precise maps for regions corresponding to the referring text; these are termed localization heads.Examples include L14 H24 and L14 H13 in LLaVA-1.5-7B.
- The framework identifies localization heads using image-attention concentration and spatial entropy, then assembles their maps to predict bounding boxes or masks without training.Only three localization heads are sufficient to localize the referred object.
- Across ten LVLMs, the training-free framework outperforms existing training-free methods and performs comparably to fine-tuned visual-grounding methods.
- The work identifies localization properties in specific LVLM attention heads and proposes using them for training-free grounding.
2. Related Works
Related work frames visual grounding as a language-conditioned localization task with REC and RES variants. Prior LVLM approaches add training data and localization components, while training-free methods commonly rely on CLIP or diffusion-model features and attention maps.
- Visual Grounding: Visual grounding identifies image regions from free-form natural-language expressions and includes Referring Expression Comprehension and Referring Expression Segmentation.REC outputs a bounding box, whereas RES additionally requires segmentation.
- Application of LVLMs in Grounding Tasks: LVLM-based grounding methods incorporate grounding datasets during training and add components that extract localization information.LISA, for example, introduces a [SEG] token for mask embedding.
- Training-Free Visual Grounding: Training-free grounding methods typically use internal features or attention maps from CLIP or text-to-image diffusion models to generate or select regions.
3. Background
LVLMs process concatenated visual and textual embeddings through decoder blocks with multi-head self-attention. The analysis focuses on text-to-image attention from the final text token to image tokens.
- Model architecture: LVLMs comprise a vision encoder, projector, and language model that concatenate visual and textual embeddings as input.The image becomes P^2 visual tokens and the text becomes L token embeddings.
- Multi-head self-attention: Multi-head self-attention is the component where tokens interact, using projected queries, keys, and values in each layer and head.Attention weights reflect similarity between query and key matrices.
- Text-image interaction: Because decoding is autoregressive, the final text token is treated as a representative query for the whole sentence.This enables examining how the complete text description attends to image regions.
- Text-image interaction: The analysis extracts the first P^2 attention components, corresponding to image tokens, and denotes them as L_l H_h.For example, L5 H3 denotes the third head in the fifth layer.
4. Towards Discovering Localization Heads
The paper identifies localization heads by combining image-focused attention with spatially concentrated attention, then ranks heads by how consistently they satisfy these criteria. Higher-ranked heads show stronger object alignment, supporting a small-head, training-free grounding approach.
- Criteria: The image-attention threshold is set at the graph’s maximum-curvature point, while the selection method remains robust across reasonable threshold values.For LLaVA-1.5-7B, the example threshold is τ = 0.24.
- Criteria: Low spatial entropy indicates that a binarized attention map forms a concentrated cluster around relevant object regions.Connected components are computed after thresholding attention-map elements at the mean.
- Validation: Higher selection-frequency ranks correspond to higher average IoU, with Spearman correlation coefficients above 0.7 for all evaluated LVLMs.Pseudo-masks from attention maps are compared with ground-truth masks, considering heads selected at least 1% of the time.
- Grounding framework: The framework assembles localization-head attention maps into a combined map for defining the referred object’s bounding box or segmentation mask.The broader framework uses the selected heads to perform training-free visual grounding.
- Validation: The consistent cross-model trend leads the paper to characterize localization heads as an innate property of LVLMs.The claim is based on recurring relationships between selection frequency and object-alignment performance across various LVLMs.
5. Visual Grounding with Localization Heads
The framework identifies localization heads, extracts and combines their attention maps, and converts the result into pseudo-masks and bounding boxes without training.
- Head identification: Localization heads are identified before inference by ranking attention heads according to selection frequency and choosing the top k heads.The selected heads are then used to extract attention maps for a new image-text pair.
- Map processing: Gaussian smoothing preserves localization detail while reducing random noise in each extracted attention map.
- Evaluation: Table 1 compares REC performance against fine-tuning-based and training-free methods, with best results marked separately for each category.
- Map processing: Smoothed maps are summed element-wise and binarized into a pseudo-mask for the referred object.
- Outputs: The largest rectangle enclosing the pseudo-mask provides a bounding box, which can also prompt SAM for segmentation.
6. Experiments
Experiments across LVLMs and visual grounding benchmarks show competitive performance without training, including challenging reasoning settings, while ablations examine head count and selection criteria.
- Experimental setup: The evaluation spans ten LVLMs, REC and RES benchmarks, and the more challenging ReasonSeg benchmark requiring complex reasoning or world knowledge.REC uses Acc@0.5, while RES and ReasonSeg use cIoU.
- Main results: The framework substantially improves over training-free methods and performs comparably to fine-tuned LVLMs on REC and RES.On REC, its best performance is on par with Shikra and Ferret; on RES, a similar finding is reported relative to LISA.
- Scaling: Visual grounding performance consistently improves as model size increases from 1.3B to 13B and with architectural or training-data updates.The comparison includes changes such as LLaVA to LLaVA-1.5.
- Qualitative results: The method accurately localizes objects in challenging images containing multiple similar objects, non-central referred objects, and complex spatial relations.
- Reasoning Segmentation: The method performs comparably to LISA and sometimes outperforms it on ReasonSeg using the same LLaVA-1.5 backbone.This supports applicability to grounding tasks involving complex reasoning or world knowledge.
- Ablation studies: Performance generally improves as the number of heads increases from 1 to 3, but additional heads can introduce noise or redundancy.The optimal trend remains consistent across different LVLMs.
- Ablation studies: Using either attention sum or spatial entropy alone significantly reduces performance, indicating that both criteria are needed to identify localization heads.
- Ablation studies: Greedy per-sample head selection performs worse than fixed selection because localized heads may not be text-referred.Selection frequency favors heads that consistently focus on text-related regions.
7. Conclusion
The paper identifies localization heads across LVLMs and uses their attention maps in a training-free framework to predict text-relevant bounding boxes and segmentation masks.
- Conclusion: The framework assembles attention maps from a few localization heads to predict bounding boxes and segmentation masks for text-relevant image regions.
- Conclusion: Its performance is competitive with fine-tuning-based visual grounding methods.
A. Experimental Details
The appendix details inference-only evaluation, head-selection ablations, spatial-entropy computation, and attention-map post-processing.
- Experimental details: All experiments use inference only on a single NVIDIA GeForce RTX A6000 48GB GPU, without fine-tuning or training.
- Experimental details: Reasoning-segmentation queries include nuanced descriptions and longer sentences requiring contextual and world knowledge.
- Head selection: Criterion ablations select heads using either the highest attention sum or the lowest spatial entropy, separately.
- Head selection: Greedy selection chooses the top-k heads independently for each sample using attention sum or spatial entropy.
- Spatial entropy: Spatial entropy reshapes attention weights into a P × P map, thresholds values above the mean, and identifies 8-connected components.
- Spatial entropy: Spatial entropy uses component probabilities based on area and ranges from 0 for single-region focus to 1 for even image-wide attention.
- Post-processing: Attention maps are binarized above the mean and Gaussian-smoothed with kernel size k = 7 and standard deviation σ = 1.0.
C.1. Extended Analysis Across More LVLMs
Extended analyses show that localization-head selection remains consistent across additional LVLMs and is competitive with F-LMM, especially for larger models.
- Extended analysis across more LVLMs: Across InternVL, LLaVA, Mini-Gemini, ShareGPT4V, and Yi-VL, few attention heads consistently dominate text-image interaction.The average attention-sum trend and selection-frequency observations remain consistent across these LVLMs.
- Robustness of head selection: Localization-head selection is not sensitive to the threshold τ or the number of selected heads N.The top-3 localization heads remain consistent across different τ and N settings.
- Comparison with F-LMM: F-LMM outperforms the proposed method on smaller LVLMs, while the methods are comparable on relatively larger LVLMs.The comparison concerns referring expression segmentation, and F-LMM uses fine-tuned mask decoder modules.
- Comparison with F-LMM: The localization heads have competitive potential with specialized mask decoder modules, particularly in relatively larger LVLMs.This conclusion follows the reported RES comparison between F-LMM and the proposed method.
D.3. Gaussian Smoothing Ablation
Gaussian smoothing generally improves localization-head attention maps, but the framework remains competitive without this auxiliary post-processing step.
- Gaussian smoothing ablation: Gaussian smoothing consistently enhances performance across almost all tested combinations of standard deviation σ and kernel size κ.The ablation evaluates LLaVA-1.5-13B on the RefCOCO validation set using the UNC split.
- Gaussian smoothing ablation: 1.9% drop is observed without Gaussian smoothing compared with the best case, while performance remains competitive.The unsmoothed settings are σ = 0 or κ = 1.
- Interpretation: Gaussian smoothing serves as an auxiliary refinement step rather than a prerequisite for competitive localization.This conclusion is supported by the competitive performance of the basic localization-head attention map.
- Multi-object grounding: The pipeline also shows promise for multi-object grounding, obtaining comparable PNG results with improvements relative to F-LMM.spaCy extracts noun tokens to generate attention maps for multiple objects.
E. More Qualitative Results
Qualitative evaluations across diverse LVLMs and scenarios show that a few localization heads can identify text-referred objects and support downstream image editing.
- Qualitative results: Across 10 LVLMs ranging from 1.3B to 13B parameters, a few localization heads accurately localize objects from text queries.The qualitative results cover referring expression comprehension, referring expression segmentation, and reasoning segmentation.
- Real-world image segmentation: Localization heads capture regions or objects of interest in real-world images across varied data types.Figure 18 reports real-world image segmentation using text expressions.
- Diffusion-model integration: Segmentation masks from a small number of localization heads can guide SDXL to generate desired image edits.The frozen LVLM produces a mask from the expression, which is combined with a text prompt for diffusion-based inpainting.
G. Limitations
The paper identifies two main limitations: multi-object grounding lacks a formalized, streamlined pipeline, and the method is less suitable for models that discard spatial image information.
- Multi-object grounding: Multi-object grounding remains insufficiently formalized and streamlined for practical, user-friendly, and broadly adaptable use.The authors identify this as a future research direction.
- Spatial information: The method is less suitable for LVLMs or methods that do not preserve spatial information, such as pooling-based approaches.Explicit attention-map extraction becomes challenging in these settings, and applying the framework is left for future exploration.