PaperlayerView exact revision on arXiv

Source-linked AI summary

Angular Supervision for Visual Object Grounding in Autoregressive Vision-Language Models

Parsa Esmaeilkhani, Longin Jan Latecki

arXiv:2610.32220v1cs.CV

TL;DR

Autoregressive VLMs make it difficult to determine whether generated words are grounded in the correct image regions. The paper proposes Angular Relevance and Angular Object-Aware Loss to measure and supervise visual-token alignment. The approach yields strong referring-expression segmentation results, improves with AOL finetuning, and transfers to object categories absent from training.

  • Problem

    In autoregressive VLMs, it remains difficult to determine whether generated words are grounded in the correct visual evidence, especially for object-level localization.

    Source: §1 · p. 1

  • Method

    The paper measures cosine alignment between processed visual tokens and vocabulary directions, then uses AOL with next-token prediction to supervise object-aligned visual embeddings.

    Source: §1 · p. 3, §3.1 · p. 4, §5.1 · p. 6

  • Results

    Angular Relevance outperforms attention and Logit Lens maps for training-free RES, while NTP+AOL improves localization and transfers to unfamiliar object categories.

    Source: §5.2 · pp. 6–7, §8 · p. 8

  • Takeaways & Limitations

    The VLM’s visual tokens can support grounding without a trained segmentation head, and the learned grounding preserves general VQA performance.

    Source: §8 · p. 8

    Angular Relevance alone is diagnostic and does not improve the model’s internal grounding.

    Source: §1 · p. 2

Abstract

from arXiv · show

Autoregressive vision-language models (VLMs) struggle to ground generated text in the correct image regions, and existing explainability methods, such as attention maps and Logit Lens, often produce poor spatial explanations due to weak text-to-vision attention and sink tokens. We propose \textbf{Angular Relevance} ($R_L$), a simple visualization measure based on cosine similarity between visual token embeddings and unembedding-matrix rows. Because $R_L$ is invariant to token norm, it produces substantially sharper and more faithful heatmaps than attention maps or Logit Lens on out-of-the-box models. Without any training, $R_L$ yields competitive referring-expression segmentation (RES), outperforming prior diagnostic maps and CLIP-based training-free methods. To preserve localized object information during cross-modal mixing, we introduce \textbf{Angular Object-Aware Loss (AOL)}, an auxiliary loss that directly supervises visual token embeddings to align angularly with the vocabulary directions of their corresponding object concepts. Finetuning with AOL alongside the standard next-token prediction objective significantly sharpens object localization in heatmaps, showing that the VLM's own visual tokens can be used directly for grounding and improving RES, without training any segmentation heads and without degrading general VQA performance. We further demonstrate that the learned grounding also transfers to category-level segmentation of object categories not seen during training.

1. Introduction

Autoregressive VLMs process visual and text tokens jointly, but whether generated words are grounded in the right image regions remains difficult to determine. The section introduces angular diagnostics and supervision to improve localization while preserving language ability.

1. Introduction

  • Object-level grounding requires localizing the visual evidence for generated concepts, but attention can be diffuse, unstable, sink-dominated, or misplaced.Logit Lens can also distort spatial explanations through vocabulary competition and high-norm visual tokens.Source: §1 · p. 1
  • Angular Relevance measures cosine alignment between final-layer visual embeddings and queried vocabulary directions, making its score invariant to visual-token rescaling.Unlike Logit Lens probabilities, this measure focuses on semantic alignment rather than embedding norm.Source: §1 · p. 2, Figure 1 · p. 2
  • Angular Relevance remains diagnostic: it reveals object representation but does not itself improve internal grounding, while cross-modal mixing can diffuse localized information.The paper asks whether localized visual information can be preserved in visual-token representations without sacrificing language and instruction-following ability.Source: §1 · p. 2
  • Angular Object-Aware Loss supervises visual tokens overlapping an object to align with its vocabulary direction and discourages background tokens from aligning.For a “cat” query, it aligns cat-patch tokens with “cat” and pushes non-cat patches away.Source: §1 · p. 2
  • Combined with next-token prediction, AOL preserves local image content during cross-modal mixing and improves referring-expression segmentation without degrading language and instruction-following ability.The approach uses object masks for per-token supervision of existing visual embeddings rather than training a separate mask-prediction head.Source: §1 · p. 3
  • The proposed design differs from segmentation-oriented VLMs by supervising existing visual embeddings instead of attaching a mask decoder or dense prediction module.The cited comparison notes that dense prediction pathways can introduce tension with preserving the original model’s language ability.Source: §1 · p. 3

2. Preliminaries

Preliminaries describe how projected image tokens enter the language model and how their localized information can diffuse during processing. They then introduce Logit Lens as applying the unembedding layer to visual-token embeddings to inspect vocabulary probabilities.

2. Preliminaries

  • Projected image tokens join text tokens as LLM inputs, and attention-layer mixing can diffuse localized visual information or weaken connections to image regions.The passages also note that visual information may transfer to language tokens or be ignored, and that removing half the visual tokens does not reduce VLM performance.Source: §2 · p. 3

2.1. Core Idea of Logit Lens

  • Logit Lens applies the unembedding matrix and softmax to a deep visual-token embedding to obtain a probability distribution over vocabulary tokens.The unembedding matrix maps a text-token representation to vocabulary logits; the method applies it instead to visual-token representations from VLM layers.Source: §2.1 · p. 3

3. Methodology

Angular Relevance scores visual tokens by cosine alignment with vocabulary directions, while Angular Object-Aware Loss directly trains final-layer visual embeddings to align with queried-object concepts and separate them from background patches.

3.1. Relevance Measure for Explainability

  • RL measures cosine similarity between a visual-token embedding and the unembedding row for the queried text token, avoiding Logit Lens softmax and norm effects.The final layer is used because it is directly aligned with the unembedding matrix for vocabulary prediction.Source: §3.1 · pp. 3–4
  • For multiword referring expressions, the method averages subword rows into word directions, averages content-word directions into one query direction, then maps patch scores to a smoothed image heatmap.ReLU suppresses negative cosine similarities before light Gaussian smoothing.Source: §3.1 · p. 3
  • Unlike CLIP patch-to-sentence similarity, RL measures visual tokens after they have been processed and mixed with language tokens inside the LLM.This places relevance in the model’s internal unembedding space.Source: §3.1 · p. 4

3.2. Angular Object-Aware Loss

  • AOL avoids Logit Lens softmax competition, which can suppress synonyms, and uses an angular margin after plain cosine alignment gradients fade near alignment.The paper sets the margin to m = 0.5.Source: §3.2 · p. 4
  • AOL uses projected ground-truth masks to identify object-overlapping patches, pulls their final-layer embeddings toward the queried token’s vocabulary direction, and pushes background patches away.The loss supervises the model’s existing visual embeddings rather than training a separate mask-prediction head.Source: §3.2 · p. 4

3.1. Relevance Measure for Explainability

  • RL produces more localized, semantically aligned heatmaps than attention and Logit Lens, and AOL fine-tuning further sharpens object localization.The figure compares the base-model maps with RL after NTP+AOL fine-tuning.Source: §3.1 · p. 4, Figure 2 · p. 5

4. Direct Gradient to Visual Tokens with AOL

AOL directly updates visual tokens through the unembedding path, whereas NTP reaches them only indirectly through self-attention. Its angular-margin gradient remains nonzero near alignment, unlike the attenuated NTP signal.

4. Direct Gradient to Visual Tokens with AOL

  • AOL gives every visual token a direct gradient through the unembedding matrix, while NTP reaches visual embeddings only indirectly through self-attention weights that are small for most patches.At the relevant layer, visual embeddings do not participate directly in next-token prediction, so their NTP gradient is zero.Source: §4 · p. 5
  • The AOL margin keeps object-token gradients nonzero near alignment, while NTP has no input-independent lower bound because its visual-token gradient vanishes when relevant attention weights are small.The angular gradient gives low-norm tokens larger updates than high-norm ones; fewer high-norm visual tokens with NTP+AOL is reported as an observation, not a direct norm-shrinking effect.Source: §4 · p. 5

5. Experimental Results

Experiments evaluate heatmap-based referring-expression segmentation on RefCOCO/+/g and transfer to ImageNet-S50. Angular Relevance performs strongly without training, while AOL improves supervised localization and transfers to COCO-disjoint classes.

5.1. Tasks and Evaluation Protocol

  • RefCOCO/+/g tests RES on two VLM backbones; heatmaps become masks by thresholding or by prompting frozen SAM2 with a box from the largest connected component.For ImageNet-S50, the query is the category name rather than a referring expression.Source: §5.1 · pp. 5–6
  • Frozen SAM2 refines spatial extent within a heatmap-derived box but performs neither text-conditioned grounding nor object recognition.Thus, semantic localization reflects the heatmaps rather than a text-grounding module in SAM2.Source: §5.1 · p. 6

5.2. Segmentation Results

  • Training-free relevance maps outperform attention and Logit Lens across dataset splits and backbones, and CLIP-based methods; frozen SAM2 refinement further improves over thresholding and training-free SOTAs.The approach uses angular alignment without head selection or architectural modification.Source: §5.2 · p. 6, Table 1 · p. 6
  • AOL raises average RES cIoU over base by +7.7 (LLaVA, Thr), +4.0 (LLaVA, SAM2), +9.4 (Qwen, Thr), and +6.5 (Qwen, SAM2) points.The gains use visual-token supervision without adding a trainable segmentation head.Source: §5.2 · pp. 6–7, Table 2 · p. 7
  • AOL raises ImageNet-S50 mIoU from 36.9 to 57.4 on all classes and from 33.5 to 51.6 on COCO-disjoint classes; the disjoint gain is +18.1 versus +22.6 on overlap classes.Qwen2.5-VL was trained on RefCOCO; disjoint classes have no corresponding MS-COCO category.Source: §5.2 · p. 7, Table 3 · p. 7

6. Ablation Studies

The ablations test whether Angular Relevance identifies patches the model uses, how AOL affects visual-token norms, whether general VQA behavior is preserved, and whether pseudo-mask supervision can support AOL.

Causal faithfulness

  • Replacing the top-k Angular Relevance patches reduced referring-expression comprehension accuracy from 91.8% to 77.4%, versus 87.6% after replacing the same number of random patches.The intervention used the ground-truth object's patch count for k.Source: §6 · pp. 7–8

Effect of AOL on visual embedding norm

  • NTP+AOL produced a more compact visual-token norm distribution with fewer high-norm outliers than NTP-only finetuning.The authors report that these outliers are associated with sink-token behavior and magnitude-dominated explanations, and that their reduction makes Angular Relevance heatmaps more faithful and localized.Source: §6 · p. 8, Figure 4 · p. 8

General VQA and hallucination

  • NTP+AOL did not degrade Qwen2.5-VL's VQAv2 question-answering or POPE object-hallucination performance relative to the base model or NTP-only finetuning, and slightly improved both metrics.Table 4 reports general VQA and hallucination performance for Qwen2.5-VL-7B.Source: §6 · p. 8, Table 4 · p. 8

AOL with pseudo-mask supervision

  • Pseudo-masks from a specialized segmentation model improved RefCOCO validation cIoU from 66.2 before AOL training to 69.6, approaching 73.4 with ground-truth masks.The pseudo-masks define positive and negative visual tokens during training.Source: §6 · p. 8

7. Related Work

Related work spans grounding-specific segmentation architectures, coordinate or token prediction, global cross-modal alignment, and visual-preservation losses. This paper instead targets grounding within autoregressive VLMs through vocabulary-aligned visual tokens and angular relevance heatmaps.

7. Related Work

  • Instruction-tuned autoregressive VLMs perform well on captioning, retrieval, and VQA, but can hallucinate, underuse visual tokens, and exhibit attention-sink behavior [42] [47].Prior work reports that removing visual tokens inside ground-truth regions strongly reduces performance.Source: §7 · p. 8
  • Segmentation-oriented VLMs add mask or region prediction mechanisms, while referring-expression methods often use task-specific box- or mask-grounding architectures.Other approaches use coordinate or token prediction, global cross-modal alignment, or visual-preservation losses.Source: §7 · p. 8
  • AOL differs by aligning internal visual-token embeddings with object-specific vocabulary directions, while Angular Relevance generates heatmaps from visual-embedding alignment with vocabulary directions.The heatmaps provide spatial or token-level cues for analysis and grounding.Source: §7 · p. 8

8. Conclusion

The paper proposes Angular Relevance for visual-token heatmaps and AOL for angular supervision of visual embeddings. Experiments report training-free grounding, improved RES after AOL finetuning, transfer to unseen object categories, and preserved general VQA performance.

8. Conclusion

  • Angular Relevance produces more faithful and localized heatmaps than attention maps and Logit Lens, while AOL directly supervises visual embeddings to preserve localized object information during cross-modal mixing.RES experiments show training-free grounding and consistent gains after NTP+AOL finetuning without segmentation heads; grounding transfers to unseen object categories while preserving general VQA performance.Source: §8 · p. 8

A. Implementation and Evaluation Details

The implementation uses a shared bounding-box prompt for training and evaluation and selects heatmap thresholds on RefCOCO training data before fixing them across evaluation datasets. It also specifies AOL hyperparameter selection and finetuning configuration.

A. Implementation and Evaluation Details

  • Training and evaluation use the same prompt, which asks the VLM to output only four pixel-coordinate values for the referred object's bounding box.The coordinates are (x1, y1, x2, y2).Source: A. Implementation and Evaluation Details · p. 11
  • Angular Relevance masks use threshold τ = µ + σ, while Logit Lens masks use τ = µ; both thresholds were selected on RefCOCO train and fixed across evaluation datasets.The heatmap is ReLU-activated and binarized using per-image statistics; tested thresholds were µ − σ, µ, and µ + σ.Source: A. Implementation and Evaluation Details · p. 11
  • On a held-out RefCOCO train set with Qwen2.5-VL, AOL weight λ = 2 and angular margin m = 0.5 achieved the best performance and were used in the main experiments.The tested weights were 0.25, 0.5, 0.75, 1, 1.5, and 2; tested margins were 0.1, 0.3, 0.5, and 0.7 radians.Source: A. Implementation and Evaluation Details · p. 11
  • Finetuning used LLava-v1.5-7B and Qwen2.5-VL-7B-Instruct with frozen vision encoders except trainable projection layers, LoRA rank 16, and a frozen language head for AOL.Training used four A100 GPUs, effective global batch size 256, initial learning rate 5 × 10^-5, cosine decay, and 3% linear warm-up.Source: A. Implementation and Evaluation Details · p. 11

B. Layer Ablation for AOL Supervision

AOL supervision performs best at the final layer, where hidden representations map most directly to vocabulary logits.

B. Layer Ablation for AOL Supervision

C. AOL Component and Margin Ablations

The ablations favor an angular margin and combined foreground-background supervision; further analyses examine query transfer, gradient behavior, class partitioning, and pseudo-mask training.

C. AOL Component and Margin Ablations

D. Generalization across related text queries.

  • Training AOL with “cat” also improves Angular Relevance localization for related queries including “kitten”, “feline”, and “tabby”.The authors interpret this as transfer to synonymous and closely related queries, not overfitting to the supervised word.Source: Figure 1 · p. 12, D. Generalization across related text… · p. 12

E.1. Lower bound of AOL gradient

  • With bounded hidden-state norms and typical near-orthogonality, AOL gradients have a positive lower bound for most visual tokens.At fixed angle, larger-norm tokens receive smaller gradients; the scale-invariant gradient is orthogonal to the embedding, while NTP+AOL empirically reduces high-norm outliers.Source: §E.1 · pp. 12–13

E.2. Absence of a non-trivial lower bound for NTP

  • NTP has no non-trivial unconditional lower bound on most visual-token gradient norms because its signal reaches visual tokens only through text-token attention.When text-to-visual attention is small, the gradient can approach zero.Source: §E.2 · p. 13

F. ImageNet-S50 Class Partition

  • The ImageNet-S50 partition contains 18/50 fine-grained COCO-overlap classes, 8/50 loose-overlap classes, and 24/50 COCO-disjoint classes.The mapping was manually constructed, cross-checked, and manually verified; ambiguous cases were assigned to loose overlap.Source: F. ImageNet-S50 Class Partition · p. 13

G. AOL with pseudo-mask supervision.

  • 69.6 cIoU with SAM3-I pseudo-mask AOL supervision improves on 66.2 without it, approaches ground-truth-mask AOL at 73.4, and is below SAM3-I direct segmentation at 71.7.SAM3-I supplies training pseudo-masks only; inference uses Angular Relevance heatmaps and binarization without a segmentation model.Source: Table 5 · p. 14, G. AOL with pseudo-mask supervision · p. 14

H. Energy Pointing Game Evaluation

I. Referring Expression Comprehension task

The paper evaluates referring expression comprehension using bounding-box prediction from the VLM’s output text. NTP+AOL matches or slightly improves recognition accuracy while enhancing grounding.

I. Referring Expression Comprehension task

  • The REC evaluation uses the same prompt as RES, so performance differences reflect visual-grounding quality rather than task-specific prompting.Source: §I · p. 14
  • NTP+AOL matches or slightly improves upon the base VLM and NTP-only models across all REC datasets, with results comparable to state-of-the-art finetuned VLMs.No task-specific grounding head is introduced.Source: Table 7 · p. 14, §I · p. 14
  • REC requires predicting a bounding box for the referred object from the VLM’s output text, and performance is measured by IoU@0.5.Source: Table 7 · p. 14, §I · p. 14

Related papers

Loading 2610.32220v1…