Towards Interpreting Visual Information Processing in Vision-Language Models

1 INTRODUCTION

• The paper addresses uncertainty about how visual inputs are encoded, whether object details are spatially localized or dispersed, and whether text-model processing mechanisms generalize to visual inputs.Source: §1 · pp. 1–2
• The study investigates LLaVA 1.5 7B’s language-model component, also testing LLaVA-Phi1 and Qwen2-VL on object-identification tasks.The authors focus on the language model because it comprises more than 95% of the parameters in models like LLaVA 1.5 7B.Source: §1 · pp. 1–2
• Object information is highly localized to visual-token positions corresponding to the object’s original image location.The authors investigate this with ablation techniques.Source: §1 · p. 2
• Visual-token representations are refined toward interpretable vocabulary embeddings despite the language model not being explicitly trained to do so.The authors extend the logit lens technique to analyze this refinement.Source: §1 · p. 2
• Attention blocking suggests the model extracts object information from object tokens in middle-to-late layers.Source: §1 · p. 2
• The authors describe their findings as first steps toward understanding VLM internal mechanisms and more interpretable, controllable multimodal systems.Source: §1 · p. 2

2 BACKGROUND

• Adapter-style VLMs map image-encoder outputs into a language model, which processes input tokens autoregressively to predict the next token.LLaVA connects CLIP and Vicuna with a trained linear projection or MLP and fine-tunes for multimodal conversation without image-text next-token pretraining.Source: §2 · pp. 2–4
• Object information localizes to a subset of visual tokens, whose representations become more interpretable across layers before some information reaches the last token for object identification.The figure summarizes these findings for adapter-style VLMs.Source: Figure 1 · p. 3

2.1 NOTATION

• LLaVA processes 576 image patches with CLIP, maps the resulting feature map into the language model’s input space, and concatenates visual and text embeddings.The paper calls the adapter-mapped image embeddings visual tokens.Source: §2.1 · p. 4

2.2 OPEN QUESTIONS AND HYPOTHESES

• Because adapter outputs are soft prompts without vocabulary semantics, the paper asks how visual content is represented and processed in the language model.It also notes that visual and textual representations have a modality gap and that LLaVA is fine-tuned on visual question-answering prompt-response pairs.Source: §2.2 · p. 4, §2.2.2 · p. 4

2.2.1 LOCALIZATION OF OBJECT-SPECIFIC INFORMATION

• The paper tests whether object information is localized to spatially corresponding patch tokens or carried mainly by global register tokens using object-identification ablations.The competing hypotheses draw on prior work in classical vision transformers and CLIP variants.Source: §2.2.1 · p. 4

2.2.2 PROCESSING OF OBJECT-SPECIFIC REPRESENTATION

• To study visual-token processing, the paper applies a logit lens across layers and examines information flow to the last token, against prior proposals of direct extraction or text-token summarization.Prior language-model work describes refine-then-extract for factual associations, while preliminary VLM evidence suggested image information may first be summarized in text tokens.Source: §2.2.2 · p. 5

3 INVESTIGATING THE REPRESENTATIONS IN MAPPED VISUAL TOKENS

• 4,318 COCO Detection Training images remained after filtering for a single target instance, target size of 1,000–2,000 square pixels, fewer than four annotated objects, and failure to identify the target after masking.The masking filter retains images where the model identifies the target in the original but not the masked image.Source: §3 · p. 5
• The experiments replace hypothesized object-relevant tokens with the mean embedding across visual tokens from 50,000 ImageNet validation images, preserving token norms and keeping replacements in-distribution.The modified embeddings are then used to assess the impact of ablating selected token subsets.Source: §3 · p. 5
• Identification was evaluated by checking target mentions in generated descriptions, testing object-presence answers, and comparing next-token responses on manually curated visual questions.Binary polling measures whether ablation changes the next token from “Yes” to “No.”Source: §3 · pp. 5–6, Figure 2 · p. 6
• The tested subsets included object tokens, one- or two-position buffers, register tokens, random tokens, and the highest-attribution Integrated Gradients tokens.The gradient baseline ranks tokens by attribution to the “Yes” logit for an object-presence prompt.Source: §3 · p. 6
• For comparable numbers of ablated tokens, object-token ablation consistently produced larger performance decreases than gradient-based and random baselines across all three settings and both LLaVA models.The results suggest that object information is localized to the region of the object token.Source: §3 · p. 6

4.1 ANALYZING THE RESIDUAL STREAM THROUGH EMBEDDING SPACE

• The logit lens projects each visual-token activation through the unembedding into vocabulary probabilities and selects the highest-logit token at each layer.The authors note that this works despite VLMs being fine-tuned on multimodal tasks rather than pretrained for visual next-token prediction.Source: §4.1 · pp. 6–7
• 23.7% of object-patch token positions matched the correct class token on average at each image’s best-performing layer, which averaged layer 25.7 of 33.The late-layer activations corresponded to token embeddings describing the original patch object.Source: §4.1 · p. 7
• Decoded visual tokens sometimes represented details more specific than the object itself and concepts such as “year” and “month” in Chinese or Korean.The paper gives sweater patterns as an example of finer-than-object detail.Source: §4.1 · p. 7
• In Qwen2VL, 6.5% of visual tokens matched the correct object class at the best-performing layer, averaging layer 25.1 of 29; LLaVA-Phi-3 attention results closely mirrored LLaVA’s.Qwen2VL also showed non-zero correspondence in early layers, possibly due to its cross-attention adapter.Source: §4.1 · pp. 7–8, §4.2 · p. 10
• Count-like features sometimes appeared in background tokens, persisted after their ablation, and may reflect language-model processing rather than direct visual information.The authors suggest this may help explain why CLIP-based VLMs can underperform CLIP on classification tasks.Source: §4.1 · p. 8

4.2 TRACING ATTENTION FLOW

• Attention knockout selectively blocks token-to-token attention connections across layer windows, and the study measures changes in accuracy and correct-token prediction probability.The experiments test object tokens and buffers, non-object tokens, and visual tokens connecting to the last visual row on a 100-image VQA dataset.Source: §4.2 · p. 9
• Blocking object-token attention to the final token in mid-to-late layers degraded performance, while blocking visual-token attention to the last visual row had minimal effect.Blocking non-object-token attention to the final token in early layers also reduced performance, consistent with earlier integration of broader image context.Source: Table 2 · p. 9, §4.2 · p. 9

5 RELATED WORK

• Input-attribution methods such as Grad-CAM and Integrated Gradients, and VLM extensions such as LVLM-Interpret, complement this work’s focus on internal processing mechanisms.The cited methods are GRAD-Cam (Selvaraju et al., 2017), Integrated Gradients (Sundararajan et al., 2017), and LVLM-Interpret.Source: §5 · p. 10
• Causal tracing and sparse coding have advanced mechanistic interpretability for LLMs, and this work extends some such techniques to visual processing in multimodal contexts (Meng et al., 2022; Marks et al., 2024).The paper notes that VLMs are not trained on next-token prediction for visual inputs (Liu et al., 2023b).Source: §5 · p. 10
• Prior multimodal interpretability studies cover CLIP and generative VLM components, while this work focuses on visual-input representations in generative VLMs.Examples include multimodal neurons and interpretable CLIP features, as well as visual-concept neurons and causal tracing in generative models (Schwettmann et al., 2023; Palit et al., 2023).Source: §5 · p. 10
• Basu et al. (2024) proposed that late visual tokens may summarize image information in VQA; this paper tests that possibility in a more straightforward object-identification setting.Concurrent work reported similar logit-lens findings, while other work observed comparable patterns across visual and audio modalities (Wu et al., 2024).Source: §5 · p. 10

• Object information is highly localized in visual tokens corresponding to the object's spatial location, challenging reliance on global features in register tokens.Source: §6 · p. 10
• Visual token representations align with interpretable textual concepts across layers, suggesting refinement toward language-like representations without next-token prediction pretraining for visual inputs.Source: §6 · p. 10
• Attention-blocking experiments suggest VLMs may extract object information from relevant visual tokens in mid-to-late layers.Source: §6 · p. 10
• The study focuses on object identification and LLaVA-type models, so findings may not fully generalize to complex reasoning, open-ended question answering, or significantly different architectures.Source: §6 · p. 10

• Figure 4 presents example dataset images for the object-identification tasks.Source: Figure 4 · p. 15

• Figure 5 shows curated images and questions; answers begin with “It is a,” and the correct answer is underlined.Source: Figure 5 · p. 16

• 23.7% of object patch token positions correspond to the correct object class token at the best-performing layer on average across 170 COCO images, with the peak layer averaging 25.7/33.Source: Figure 6 · p. 17
• Qwen2VL-2B peaks at layer 25.1/29 with 6.5% correct token-to-object mapping.Source: Figure 7 · p. 17
• Unlike LLaVA, Qwen2VL-2B exhibits non-zero token-to-object correspondence in early layers.Source: Figure 7 · p. 17

D LLAVA-PHI-3 BLOCKING

• Blocking attention from object tokens to the last token position has its strongest effect in middle-to-late layers.The figure reports a pattern similar to LLaVA-1.5.Source: Figure 8 · p. 18
• Blocking attention from visual tokens to the last visual row has minimal impact throughout the network.Source: Figure 8 · p. 18
• The object-token blocking pattern is similar to that observed for LLaVA-1.5.Source: Figure 8 · p. 18