PaperlayerView exact revision on arXiv

Source-linked AI summary

LatentLens: Revealing Highly Interpretable Visual Tokens in LLMs

Benno Krojer, Shravan Nayak, Oscar Mañas, Vaibhav Adlakha, Desmond Elliott, Siva Reddy, Marius Mosbach

arXiv:2602.00462v5cs.CVcs.AI

TL;DR

Visual-token interpretability in VLMs is difficult to assess because standard lenses compare representations with fixed embedding or vocabulary spaces. LatentLens instead retrieves nearest contextualized token representations from a large text corpus, and across 15 VLMs it finds most visual tokens interpretable across layers. The method provides fine-grained language descriptions, while its storage requirements, corpus dependence, behavioral relevance, and architectural generality remain bounded.

  • Problem

    Existing interpretability methods provide limited evidence about what visual-token representations encode across LLM layers, making their language-like meaning difficult to assess.

  • Method

    LatentLens stores contextualized token representations from a large text corpus and retrieves nearest-neighbor contextual representations as descriptions for visual tokens.

  • Results

    Across 15 VLMs and all studied layers, 68% of visual tokens are judged interpretable with LatentLens, compared with 32% for EmbeddingLens and 24% for LogitLens.

  • Takeaways & Limitations

    The findings provide evidence that visual tokens are consistently interpretable and align with semantic intermediate language representations, including at the LLM input.

  • Takeaways & Limitations

    LatentLens requires storing a large contextual-embedding corpus, may be influenced by Visual Genome sentences, has unstudied effects on downstream behavior, and may not generalize beyond substantially different transformer architectures.

Abstract

from arXiv · show

Transforming a large language model (LLM) into a vision-language model (VLM) can be achieved by mapping the visual tokens from a vision encoder into the embedding space of an LLM. Intriguingly, this mapping can be as simple as a shallow MLP transformation. To understand why LLMs can so readily process visual tokens, we need interpretability methods that reveal what is encoded in the visual token representations at every layer of LLM processing. In this work, we introduce LatentLens, a novel approach for mapping latent representations to descriptions in natural language. LatentLens encodes a large text corpus and stores contextualized token representations for each token in that corpus. Visual token representations are then compared to these contextualized representations and the top-nearest neighbor representations serve as descriptions of the visual token. We evaluate this method on 15 different VLMs, showing that commonly used methods, such as LogitLens, substantially underestimate the interpretability of visual tokens. With LatentLens instead, the majority of visual tokens are interpretable across all studied models and all layers. Qualitatively, we show that the descriptions produced by LatentLens are semantically meaningful and provide more fine-grained interpretations for humans compared to individual tokens. More broadly, our findings contribute new evidence on the alignment between vision and language representations and open up new directions for analyzing the latent representations of LLMs.

1. Introduction

The paper asks why frozen LLMs can process visual tokens and whether those representations are interpretable as meaningful language. It introduces LatentLens and finds substantially higher interpretability than existing training-free lenses.

  • Motivation: Frozen LLMs can process visual inputs after a simple linear or shallow-MLP mapping into the LLM embedding space.This motivates studying how visual representations are integrated inside the LLM.
  • Motivation: Existing embedding-space nearest neighbors rarely make visual tokens interpretable at the LLM input level.LogitLens has also been used to analyze visual-token representations.
  • Approach: LatentLens compares visual representations with nearest contextualized token representations from intermediate LLM layers to produce fine-grained sentence-level descriptions.The method is training-free and can yield descriptions such as a building token matching “building with many stories.”
  • Results: 68% of visual tokens are judged interpretable with LatentLens across 15 VLMs and all studied layers, versus 32% with EmbeddingLens and 24% with LogitLens.The comparison indicates that existing methods substantially underestimate visual-token interpretability.
  • Results: The study finds interpretable alignment even with linear projections and observes a Mid-Layer Leap toward semantic intermediate representations.Input and early visual-token layers align most strongly with contextualized representations from mid-layers such as layers 8–16.
  • Implications: The findings challenge assumptions about visual-token interpretability and provide evidence about vision–language representation alignment.The authors also release an interactive demo and a package providing access to contextual embeddings.

2. Background

This section describes how VLMs project vision-encoder outputs into an LLM, how visual tokens are interpreted with existing lenses, and why those lenses have important limitations.

  • Turning LLMs into VLMs: A VLM concatenates projected vision-encoder embeddings with token embeddings and processes the resulting multimodal sequence with an LLM.The projection maps vision features into the LLM dimensionality, while the LLM produces latent representations across positions and layers.
  • Turning LLMs into VLMs: Training commonly minimizes cross-entropy on target caption tokens, with only the projection trained while the vision encoder and LLM remain frozen.The projection is typically linear, an MLP, or attention-based.
  • Interpreting VLM representations: The central interpretability question is whether latent visual-token representations correspond to language-like tokens.The paper focuses on training-free approaches that directly leverage the LLM representation space.
  • Interpreting VLM representations: EmbeddingLens compares visual or latent representations with rows of the LLM embedding matrix and returns the top-k vocabulary items by cosine similarity.Its candidate descriptions are individual vocabulary items.
  • Interpreting VLM representations: LogitLens multiplies a latent representation by the unembedding matrix to obtain vocabulary logits, then selects the highest-scoring vocabulary items.It is conceptually similar to EmbeddingLens but operates in the output embedding space.

3. LATENTLENS

LatentLens maps latent representations to natural-language descriptions by nearest-neighbor search over contextualized corpus representations. It supports layer-wise, sentence-level interpretation and automated semantic evaluation.

  • Method: LatentLens treats contextualized LLM representations as the natural comparison space for latent representations and uses descriptions richer than individual sub-word tokens.This addresses the restricted candidate set and fixed reference spaces of EmbeddingLens and LogitLens.
  • Method: The method encodes a large sentence corpus with an LLM and stores contextualized token representations from every sequence position and model layer.These stored vectors form the reference set for later nearest-neighbor comparisons.
  • Method: For each visual-token latent representation, LatentLens computes cosine similarity with reference vectors and returns the sequences associated with the top-k matches.A reference match may come from a different layer than the visual token being analyzed.
  • Method: LatentLens uses multiple contextualized reference vectors per sentence, one for each token and model layer, rather than one fixed vector per description.This permits cross-layer matching between visual tokens and textual representations.
  • Interpretation: The resulting sentence descriptions support interpretation at every model layer and provide more fine-grained information than individual-token labels.A visual token can match a contextualized token such as “clocks” in “stone tower with gold clocks.”
  • Evaluation: Interpretability evaluation requires judging whether descriptions semantically match the image region, because matches may be concrete, abstract, or global.A VLM judge receives the image region, surrounding visual tokens, and top-5 descriptions to classify interpretability.

4. Experiments

Across controlled and off-the-shelf VLMs, LATENTLENS consistently finds visual tokens interpretable across layers, outperforming embedding- and unembedding-based baselines. Early visual tokens often align with contextual representations from later middle LLM layers, revealing a Mid-Layer Leap.

  • Experimental setup: The experiments evaluate visual-token interpretability across controlled vision–LLM combinations and off-the-shelf VLMs using LATENTLENS, EmbeddingLens, and LogitLens.The controlled setup uses three LLMs and three vision encoders, while the off-the-shelf evaluation spans six models.
  • Additional evaluation: DINOv2 models score 4.4/10 on average captioning, versus 6.8 for CLIP and SigLIP models, while DINOv2 visual tokens still show consistently high interpretability.The paper attributes the captioning difference likely to DINOv2’s lack of language supervision during pre-training.
  • Controlled results: At lower layers, LogitLens labels less than 25% of visual token representations as interpretable, whereas LATENTLENS generally reaches 60–85% across layers.Some OLMo variants fall to around 53% at the final layer, and reducing the mapping to a linear function or using shorter captions does not substantially change LATENTLENS interpretability.
  • Mid-Layer Leap: Early visual tokens, including input-layer tokens, most often align with contextualized representations from later middle LLM layers rather than the same layer.For OLMo-7B + SigLIP, input-layer tokens most often align to layer 8; later processing produces a diagonal alignment pattern.
  • Off-the-shelf VLMs: LATENTLENS achieves the highest interpretability on all six off-the-shelf VLMs, with average scores ranging from 33–35% for the largest evaluated models to 86% for Molmo-7B-D.All six off-the-shelf models also exhibit the Mid-Layer Leap.

5. Qualitative results

Qualitative examples show that LATENTLENS produces semantically meaningful, fine-grained descriptions across image patches and rendered text, while LogitLens often returns plausible but less faithful token predictions.

  • LATENTLENS vs. LogitLens: LATENTLENS descriptions are semantically more meaningful than LogitLens descriptions across qualitative image-patch examples.For a church-tower patch, LATENTLENS retrieves descriptions such as “large tower with clocks” across layers.
  • LATENTLENS vs. LogitLens: LATENTLENS can merge matching contextualized subwords into full words, whereas LogitLens often produces unrenderable tokens, subwords, punctuation, or unrelated non-English tokens.Higher LogitLens scores do not necessarily correspond to more interpretable descriptions.
  • Visually rendered text: For rendered text, LATENTLENS’s highest-scoring contextualized representations correspond exactly to words in the image, while LogitLens predicts plausible next tokens instead.In one example, LATENTLENS predicts “couch” where LogitLens predicts “es” or “potato”; another prediction is “Tomato.”

6. Related Work

Related work studies how vision and language representations connect, how VLMs can be interpreted, and how non-discrete LLM representations behave. Prior approaches often use limited models, closed labels, or embedding and unembedding spaces rather than contextual embeddings.

  • Connecting vision and language models: Prior work shows that frozen LLMs can process multimodal inputs through small MLP or attention-based mappings, motivating analysis of their cross-modal alignment.This literature also frames LLMs as systems capable of processing diverse data with minimal weight updates.
  • Interpreting VLMs: VLM interpretability studies characterize cross-modal concepts, circuits, attention mechanisms, and visual-token representations using methods adapted from unimodal models.The cited literature includes applications of LogitLens and related interpretability techniques.
  • Interpreting VLMs: Existing studies often quantify interpretability on small model sets or use closed object-class labels, while prior work had not considered contextual embeddings for this purpose.The paper positions LATENTLENS as addressing this gap with contextualized token representations.
  • Representations in LLMs: Related representation studies examine modality gaps, narrow-cone effects, soft prompts, hidden states, and contextual word embeddings in language models.These works provide broader context for interpreting non-discrete or intermediate representations.

7. Discussion and Conclusion

The paper concludes that visual tokens are often interpretable throughout LLM processing when compared with contextualized textual representations, including at the input layer. This supports a closer alignment between vision and language representations and motivates broader applications of LATENTLENS.

  • Discussion and conclusion: Visual tokens are consistently interpretable across LLM layers, including at the input, when compared with contextualized textual token representations.The conclusion contrasts this with comparisons against input or output embedding layers.
  • Discussion and conclusion: The Mid-Layer Leap shows that input visual tokens align with semantic intermediate language representations around layers 8–16 rather than word-level embeddings.This observation offers one piece of the explanation for why separately trained vision encoders and LLMs can be connected with minimal updates.
  • Broader implications: The findings contribute evidence about how visual and linguistic representations interface in neural systems and whether they share structural correspondences.The paper frames this as progress toward understanding relations between physical-world information and symbolic processing.
  • Future work: LATENTLENS may extend to soft prompts, latent thinking, speech, and natively multimodal models, while future work could examine deeper structural similarities.The paper also identifies hallucination mitigation and causal ablations as possible downstream directions.

Limitations

The authors identify storage overhead and unresolved scope questions as limitations of LATENTLENS. They also note that the method’s interpretations may depend on the corpus used.

  • LATENTLENS requires pre-computing and storing a large contextual-embedding corpus, unlike LogitLens, which relies solely on model weights.The authors identify this storage overhead as a limitation of their approach.
  • Interpretability judgments may be influenced by the Visual Genome corpus used to generate contextual embeddings.

Impact Statement

The authors position LATENTLENS as a tool for understanding internal representations in vision-language models. They view such fundamental interpretability research as a necessary first step toward more controllable and safer AI systems.

  • LATENTLENS contributes tools for understanding how vision-language models process visual information.
  • The authors regard fundamental research into model internals as a necessary first step toward more controllable and safer AI systems.

A. Limitations

The paper documents limitations involving storage, corpus dependence, interpretation bias, model scope, and evaluation subjectivity. These boundaries constrain how broadly LATENTLENS findings should be interpreted.

  • Storage requirements: LATENTLENS incurs storage overhead because it pre-computes and stores contextual embeddings, with cost scaling by corpus size and analyzed layers.Float8 compression reduces storage to approximately 25% of float32 size, but does not remove the scaling cost.
  • Corpus constraints: Visual Genome’s coverage constrains interpretation quality, so specialized scientific imagery and non-Western cultural contexts may yield less meaningful descriptions.
  • Noun bias: Nouns comprise approximately 45–50% of nearest-neighbor results, and this dominance may partly reflect Visual Genome’s corpus bias.Visual Genome region descriptions emphasize objects and entities over actions or relations.
  • Model scope: Findings from 15 VLM configurations may not generalize to substantially different architectures, connectors, or training paradigms.The study focuses on transformer-based models with frozen LLMs and MLP connectors.
  • Evaluation subjectivity: Interpretability judgments remain subjective despite LLM-judge validation against human annotations with κ = 0.68.The binary interpretable/non-interpretable classification may miss nuances in interpretation quality.

D. Ablations

Ablations show that LATENTLENS interpretability is broadly robust to connector and captioning changes, while language-based objectives remain important. Training dynamics indicate that alignment emerges first at the LLM input and later propagates deeper.

  • Training objectives: Changing captioning to spatial prediction tests whether language-based objectives underlie the observed alignment of visual tokens with the LLM embedding space.
  • Training variations: Changing the connector from a 3-layer MLP to a linear layer preserves nearly the same interpretability, supporting linear alignment between vision and language spaces.
  • Training variations: Replacing detailed captions with single-sentence captions maintains nearly the same interpretability, with 1.8 of 5 baseline neighbors overlapping.
  • Training variations: +5.5% interpretable visual tokens result when the LLM is unfrozen during training.
  • Training dynamics: At step 1000, early layers reach 67% interpretability while deeper layers remain at 17%; by step 6000, all layers approach final performance.These dynamics suggest alignment forms first at the LLM input and later propagates to deeper representations.
  • Tuned Lens comparison: Tuned Lens provides little or no improvement over LogitLens and can reduce interpretability by 9.1 percentage points on OLMo+CLIP.

I. Results for off-the-shelf VLMs

Across off-the-shelf VLMs, visual tokens show consistent mid-layer alignment and remain predominantly concrete and object-related in their LatentLens interpretations. Qwen2-VL exhibits the same broad pattern, with modest later-layer abstraction and stronger token transformation.

  • Mid-Layer Leap: All six off-the-shelf VLMs show a mid-layer leap followed by diagonal alignment from mid-processing onward.The pattern matches the controlled setup, indicating it is not limited to the controlled training configuration.
  • Mid-Layer Leap: Qwen2-VL visual tokens leap to LLM layer 4 at layer 0, then follow a clear diagonal alignment pattern.Visual tokens change more across the LLM than in the frozen-LLM controlled setup, but less than text tokens.
  • Interpretation Types: Concrete interpretations dominate across models and layers at 70–75%, while abstract and global interpretations each contribute 11–15%.These proportions remain remarkably stable across layers in the frozen-LLM models.
  • Interpretation Types: Qwen2-VL shows 62–78% concrete interpretations, with concrete interpretations decreasing to 62% in layers 26–27 as abstract interpretations increase.The later-layer shift may reflect abstraction learned by its unfrozen LLM.
  • Parts of Speech: Nouns comprise approximately 45–50% of nearest neighbors, followed by proper nouns at 10–20%, verbs at 10–15%, and adjectives at around 5%.The distribution is relatively stable across layers, with some model-specific variation.
  • Visual Attributes: Color words occur at around 5–6% in early layers and around 3% later, while shape and texture words remain below 1%.The pattern suggests raw color information is more prominent in early visual token representations.
  • Phrase-Level Context: Preceding context improves interpretation over the word alone in 64% of sampled cases, is neutral in 28%, and misleading in 8%.Phrase context can provide richer spatial and material information than an isolated token.

L. Dynamic Corpus Generation

Dynamic phrase generation searches for better contextual framings of LatentLens matches. On a small evaluation set, it usually improves similarity and can replace the original top match with a better-contextualized token.

  • Method: Dynamic generation uses evolutionary search to modify words before the target token while retaining the target at the phrase end.The search runs six rounds with 20 variations per round and keeps the top five phrases, evaluated on 20 interpretable visual tokens.
  • Results: 85% of cases (17/20) achieve higher cosine similarity, with an average improvement of +0.017.Evolved phrases tend to be more concise and visually specific.
  • Results: 35% of cases (7/20) promote a non-top-1 original-corpus token to the best match after evolution.The search can discover a better target token when surrounding context is more appropriate.
  • Limitation: The fixed corpus provides at most 20 contextual embeddings per vocabulary token, which can leave semantically fitting tokens underrepresented because of suboptimal context.Dynamic generation is presented as a way to address this fixed-context limitation.
  • Qualitative Comparison: Dynamic, fixed-corpus, and lowest-scoring contexts provide progressively different levels of phrase richness for the same visual-token interpretation.The examples compare dynamic context with fixed-corpus phrases and lower-scoring contexts.

M. Captioning Quality Evaluation

The paper evaluates caption quality with DCScore and compares trained model combinations with Qwen2-VL. It also presents qualitative captions and interpretable-token analyses, while noting unresolved faithfulness and resource-cost limitations.

  • Captioning Quality Evaluation: The evaluation covers 300 PixMo-Cap validation images and compares nine trained model combinations with off-the-shelf Qwen2-VL-7B-Instruct.Qwen2-VL serves as an upper-bound reference in the caption-quality comparison.
  • Captioning Quality Evaluation: DCScore rates captions on faithfulness, detail accuracy, hallucinations, and completeness, each scored from 1–10.The judge receives the full image and generated caption and returns structured sub-scores.
  • Captioning Quality Evaluation: Qwen2-VL achieves 8.5 average DCScore, compared with 6.0 for the trained models.The gap reflects different training setups: the trained models use captioning with a frozen LLM, whereas Qwen2-VL receives multimodal instruction tuning.
  • Captioning Quality Evaluation: Qwen2-7B reaches the highest average score, 6.13, among the trained models, while DINOv2 models score 4.3–4.5 versus 6.6–7.1 for CLIP and SigLIP models.The paper attributes the DINOv2 gap to its lack of language supervision during pretraining.
  • Interpretability Analysis: Across randomly sampled examples, LATENTLENS contextual phrases describe visual content more accurately than isolated baseline tokens from EmbeddingLens and LogitLens.The qualitative comparison uses 20 examples across five layers and shows top-3 predictions from each method.
  • Limitations: The authors acknowledge that LATENTLENS faithfulness to model behavior remains unresolved and that its memory and time costs exceed those of simpler lenses.Downstream token-ablation experiments were preliminary, and the paper does not claim much about downstream performance.
Loading 2602.00462v5…