PaperlayerView exact revision on arXiv

Source-linked AI summary

A Concept-Based Explainability Framework for Large Multimodal Models

Jayneel Parekh, Pegah Khayatan, Mustafa Shukor, Alasdair Newson, Matthieu Cord

arXiv:2406.08074v3cs.LGcs.AIcs.CLcs.CV

TL;DR

LMM internal representations remain difficult to interpret, especially across multimodal inputs. CoX-LMM applies dictionary learning to token representations, producing concepts grounded in text and vision. The learned concepts provide meaningful multimodal grounding and help interpret test-sample representations, with Semi-NMF balancing interpretability and disentanglement.

  • Problem

    Understanding internal representations of LMMs remains limited, despite their growing use and the need for human-comprehensible model interpretation.

    Source: §1 · p. 1

  • Method

    CoX-LMM decomposes representations of a target token from a pretrained LMM into a dictionary of concepts, grounded through associated words and maximally activating images.

    Source: §1 · p. 2, §3.4 · p. 5

  • Results

    Semi-NMF generally outperforms other baselines on visual-textual alignment metrics, while providing a balance between useful test-image interpretation and diverse, disentangled concepts.

    Source: §4.2 · p. 8

  • Takeaways & Limitations

    The resulting multimodal concepts offer a way to understand LMM representations through coherent visual and textual evidence.

    Source: §4.2 · p. 9, §5 · p. 10

    The current concept dictionaries are token-specific and do not encode shared concepts across different tokens.

    Source: H Limitations · p. 25

Abstract

from arXiv · show

Large multimodal models (LMMs) combine unimodal encoders and large language models (LLMs) to perform multimodal tasks. Despite recent advancements towards the interpretability of these models, understanding internal representations of LMMs remains largely a mystery. In this paper, we present a novel framework for the interpretation of LMMs. We propose a dictionary learning based approach, applied to the representation of tokens. The elements of the learned dictionary correspond to our proposed concepts. We show that these concepts are well semantically grounded in both vision and text. Thus we refer to these as ``multi-modal concepts''. We qualitatively and quantitatively evaluate the results of the learnt concepts. We show that the extracted multimodal concepts are useful to interpret representations of test samples. Finally, we evaluate the disentanglement between different concepts and the quality of grounding concepts visually and textually. Our code is publicly available at https://github.com/mshukor/xl-vlms

1 Introduction

LMMs extend powerful language and vision systems to multimodal tasks, but their internal computations remain difficult to understand. CoX-LMM addresses this gap with dictionary-learned concepts that can be grounded in both visual and textual domains.

1 Introduction

  • LMM interpretability remains underexplored because prior work focused mainly on CNN-based visual classification, leaving multimodal transformer representations less understood.The paper positions LMMs at the intersection of these comparatively less-studied domains.Source: §1 · p. 1
  • CoX-LMM learns a concept dictionary by decomposing token representations from a pretrained LMM, with each dictionary element representing a concept.For a target token such as “Dog,” the method builds a representation matrix and applies dictionary learning.Source: §1 · p. 2
  • Each learned concept is grounded simultaneously in vision through maximally activating images and in text through associated decoded words or tokens.This dual grounding motivates the paper’s notion of multimodal concepts.Source: §1 · p. 2
  • The framework is presented as a concept-based explainability method for understanding internal representations of large multimodal models.The authors describe it as the first effort targeting multimodal models at this scale, to the best of their knowledge.Source: §1 · p. 2

2 Related work

Existing interpretability methods mainly address CNNs, while newer work studies multimodal representations, neurons, and decompositions in related models. CoX-LMM instead learns token-specific, multimodally grounded concept dictionaries for LMM representations.

2 Related work

  • LMMs combine LLMs, visual encoders, and connector modules, but how frozen language models generalize to multimodal inputs remains insufficiently understood.The paper studies internal LLM representations when processing multimodal inputs.Source: §2 · p. 2
  • CAV-based methods and related dictionary-learning approaches extract concepts mainly for CNNs and classification, whereas CoX-LMM targets LMM architectures and introduces Semi-NMF.The paper frames Semi-NMF as a previously unconsidered variant for concept extraction.Source: §2 · p. 2
  • CoX-LMM grounds each learned concept in both modalities by decoding textual associations and selecting visually activating samples.The pipeline decomposes token representations into concepts whose visual and textual grounding can be inspected together.Source: Figure 1 · p. 3, §2 · p. 3
  • Prior VLM/LMM studies examine multimodal token alignment, neurons, causal tracing, or CLIP decomposition, while CoX-LMM discovers a concept structure directly in token representations.Its analysis is limited to a token of interest, enabling finer token-specific detail.Source: §2 · p. 3

3 Approach

CoX-LMM interprets LMM token representations by extracting relevant samples, decomposing their representations into a concept dictionary, and grounding concepts in visual and textual modalities.

3.1 Background for Large Multimodal Models (LMMs)

  • CoX-LMM models an LMM with a visual encoder, connector, and LLM, using residual-stream token representations produced during autoregressive image captioning.The framework applies to models with either frozen or fine-tuned LLMs, while experiments focus more on frozen-LLM settings.Source: §3.1 · pp. 3–4

3.2 Method overview

  • The pipeline selects samples relevant to target token t, decomposes their representations into a concept dictionary and activations, then grounds concepts in vision and text.It is intended to understand internal representations rather than directly interpret model outputs.Source: §3.2 · p. 4

3.3 Representation extraction

  • For each target token, CoX-LMM selects images where the token is both predicted and present in the ground-truth caption, then extracts its deep-layer residual representation into Z.The selected representation aggregates information computed across earlier layers and combines multimodal information in the token space.Source: §3.3 · pp. 4–5

3.4 Decomposing the representations

  • Semi-NMF decomposes Z ≈ UV into low-rank concept vectors U and nonnegative activations V, with sparse activations encouraged because samples are expected to contain few concepts.Semi-NMF permits mixed values in Z and U while constraining V to be nonnegative; concept activations for new images are obtained by nonnegative projection.Source: §3.4 · p. 5

3.5 Using the concept dictionary for interpretation

  • Concepts are grounded visually by their maximally activating samples and textually by high-probability decoded vocabulary tokens, while image interpretations use concepts with largest activation magnitudes.Figure 2 shows five activating samples and five decoded words for an example concept; concept-importance estimation can be added, but is outside this pipeline’s focus.Source: §3.5 · pp. 5–6, Figure 2 · p. 6

4 Experiments

The experiments evaluate CoX-LMM concepts through test-sample interpretation, concept disentanglement, and visual/text grounding. Semi-NMF generally provides the strongest balance of correspondence, diversity, and interpretability, with multimodal structure emerging in deeper layers.

4 Experiments

  • The experiments use a frozen DePALM captioning model, 20 concepts, layer L = 31 representations, five activating images, and top decoded tokens for grounding.The main study uses COCO data and evaluates common object tokens including Dog, Bus, Train, Cat, Bear, Baby, Car, and Cake.Source: §4 · p. 6

4.1 Evaluation setup

  • The evaluation measures test-sample interpretation with top activating concepts, concept overlap through shared grounded words, and multimodal grounding with CLIPScore and BERTScore.The setup compares dictionary-learning variants and baselines including Rnd-Words, Noise-Imgs, and Simple.Source: §4.1 · pp. 6–7

4.2 Results and discussion

  • Semi-NMF generally outperforms competing baselines on CLIPScore and BERTScore, while Semi-NMF, K-Means, and Simple outperform random-word, noise-image, and PCA baselines.These results indicate that the learned concepts associate grounded words with the visual content of test images.Source: §4.2 · pp. 7–8, Table 1 · p. 7
  • Semi-NMF strikes the best balance between low concept overlap and useful, diverse representations, motivating its use in subsequent CoX-LMM experiments.PCA has almost no overlap, whereas K-Means and Simple often exceed 40% overlap; Semi-NMF retains some overlap while supporting interpretation.Source: §4.2 · p. 8, Table 2 · p. 8
  • Semi-NMF concepts show stronger visual/text correspondence than random words for most concepts, with the majority of CLIPScore and BERTScore points above the y = x line.The comparison uses the same maximum-activating samples for grounded and random words.Source: §4.2 · p. 8, Figure 3 · p. 8
  • Qualitative visualizations show that concepts for “Dog” capture colors, hair length, puppy-like size, actions, scenes, correlated objects, and the distinction from “hot dog.”The figures display activating samples and probable decoded words for selected concepts.Source: §4.2 · pp. 8–9, Figure 4 · p. 8
  • Semi-NMF concept activations provide local interpretations for Dog, Cat, and Bus samples by showing normalized activations and multimodal grounding for the three strongest concepts.A dog example activates concepts corresponding to long hair, small or puppy-like size, and black-and-white color.Source: §4.2 · p. 9, Figure 5 · p. 9
  • Multimodal grounding becomes noticeably stronger around layers L = 20 to L = 25, supporting the use of deeper token-representation layers.Early layers perform no better than random words, while deeper layers align grounded words more strongly with visual samples.Source: §4.2 · p. 9, Figure 6 · p. 9

5 Conclusion

The paper presents a dictionary-learning framework for interpreting LMM representations through multimodal concepts grounded in both text and vision. Additional experiments support the method’s applicability across LMM architectures and visual encoders.

5 Conclusion

  • The framework decomposes token representations into dictionary elements that are grounded in both textual and visual domains as multimodal concepts.The authors propose a Semi-NMF variant for concept dictionary learning.Source: §5 · p. 10
  • Quantitative and qualitative evaluations show that the multimodal grounding is meaningful and that learned concepts help interpret test-sample representations.Source: §5 · p. 10

A.1 Experiments with LLaVA

The framework generalizes to LLaVA and to DePALM variants using non-CLIP visual encoders, while also supporting visual saliency maps for localized concept activations.

A.1 Experiments with LLaVA

  • Semi-NMF on LLaVA extracts balanced concept dictionaries with high multimodal correspondence, low overlap, and meaningful grounding, matching DePALM’s observed trends.Source: §A.1 · p. 14, Table 3 · p. 14, Table 4 · p. 14
  • LLaVA’s 576 visual tokens preserve image-patch representations, enabling saliency maps computed from concept-vector inner products with visual tokens.Source: §A.1 · p. 14
  • Figures provide multimodal grounding examples for Dog, Cat, and Train concepts and saliency maps for two Dog concepts.Source: Figure 7 · p. 15, Figure 8 · p. 15, Figure 9 · p. 16

A.2 DePALM with ViT visual encoders

  • DePALM variants with ImageNet- and MAE-trained ViT-L encoders achieve similar test CLIPScores and retain meaningful multimodal grounding compared with the CLIP-based system.Source: §A.2 · p. 15, Table 5 · p. 15, Table 6 · p. 16

B Analyzing polysemanticity in the learnt concepts

The study examines whether learned concept vectors are monosemantic or polysemantic by measuring how specifically their high-activation samples match manually defined semantic sets.

B Analyzing polysemanticity in the learnt concepts

C Further implementation details

The implementation details specify dictionary-learning procedures, multimodal evaluation metrics, computational resources, and the rationale for selecting and varying dictionary size.

C.1 Dictionary learning details

  • Semi-NMF uses scikit-learn’s DictionaryLearning with positive codes and coordinate descent for dictionary learning and test projection.Source: §C.1 · p. 18

C.2 CLIPScore/BERTScore evaluation

  • CLIPScore compares CLIP image embeddings with text embeddings of up to ten grounded words after filtering non-English and stop words.Source: §C.2 · p. 18
  • BERTScore evaluation uses LLaMA-3-8B Instruct to turn grounded words into coherent phrases before comparing them with captions using RoBERTa-large embeddings.Source: §C.2 · p. 18, Table 8 · p. 19

C.3 Resources

  • For approximately 3,000 samples, DePALM processing takes 10–15 minutes, while LLaVA representation extraction takes around 90 minutes on the reported GPUs.Source: §C.3 · pp. 18–19

C.4 Choice for number of concepts K

  • K = 20 is selected as the minimal size generally producing at least a 50% reconstruction-error drop from K = 0.Source: §C.4 · p. 19
  • Increasing K from 10 to 50 preserves consistently high CLIPScore, while overlap generally decreases and larger dictionaries increase user overhead when K << M.Source: §C.4 · p. 19

D Evaluation and extension to more tokens

Across additional tokens and multi-token words, Semi-NMF produces concept dictionaries that balance meaningful visual/textual grounding with diversity. Qualitative analyses likewise show Semi-NMF best balances coherent grounding and low overlap.

D Evaluation and extension to more tokens

D.1 Extending to multi-token words

  • For multi-token words, extracting the last token’s representation and searching for the complete sequence preserves the method’s high-CLIPScore, low-overlap behavior.The remaining dictionary-learning procedure is unchanged, and the paper reports consistent results for this adaptation.Source: §D.1 · p. 22, Table 12 · p. 22, Table 13 · p. 23

E.1 Concept grounding

  • Across Dog, Cat, Bus, and Train, K=20 concepts from layer L=31 yield diverse concepts with coherent visual and textual grounding.The reported grounding covers multiple target tokens rather than only the main Dog analysis.Source: §E.1 · p. 23

E.2 Local interpretations

  • Semi-NMF, K-Means, and Simple ground image-associated words, but Semi-NMF best balances coherence and diversity because it decomposes representations across the full dictionary.K-Means and Simple exhibit overlapping grounded words, whereas PCA is diverse but less image-coherent.Source: §E.2 · p. 23

F Qualitative analysis for different layers

Multimodal grounding emerges around layers L=20–25, but textual grounding aligns strongly with visual activations only in deeper layers.

F Qualitative analysis for different layers

  • Multimodal token representations begin appearing around L=20–25, while grounded words correspond well to visual activations mainly in deeper layers.Visual samples remain well separated across concepts even before textual and visual grounding become strongly aligned.Source: F Qualitative analysis for different… · p. 23

G Analysis for visual tokens

The framework also applies to visual-token representations, where deep-layer visual tokens exhibit multimodal semantic structure despite not attending to text. Similar visual and textual concepts suggest information may propagate into predicted text-token representations.

G Analysis for visual tokens

  • The visual-token analysis extracts K=20 concepts from L=31 using 1,752 samples, and reports multimodal semantic structure without text-token attention.The method selects visual-token representations associated with the target token and otherwise follows the original dictionary-learning procedure.Source: G Analysis for visual tokens · p. 24
  • Several concepts have similar visual and textual grounding across visual- and text-token dictionaries, suggesting visual-token processing may propagate information into predicted text-token representations.The paper specifically identifies concepts 3, 7, 10, 12, 17, and 19 as visually and textually similar to concepts found for text tokens.Source: G Analysis for visual tokens · pp. 24–25

H Limitations

The paper identifies scope and evaluation limitations, including token-specific dictionaries, concentration on visual captioning models, simple grounding methods, and imperfect CLIPScore/BERTScore metrics.

H Limitations

  • The learned concept dictionaries are currently token-specific rather than shared across different target tokens.The paper suggests exploring dictionaries that encode concepts shared by multiple tokens.Source: H Limitations · p. 25
  • The study focuses mainly on visual captioning models, so application to other LMM types remains to be explored and confirmed.The authors expect the key ideas may generalize but do not establish that generalization here.Source: H Limitations · p. 25
  • The visual and textual grounding procedures are simple, leaving room for improved activation localization and more sophisticated text-grounding methods.The paper mentions enhanced localization and tuned lens as possible directions.Source: H Limitations · p. 25
  • CLIPScore and BERTScore are useful but imperfect interpretability metrics because they inherit limitations from the models producing image and text embeddings.The paper identifies better interpretability metrics as an open research question, especially for LLMs and LMMs.Source: H Limitations · p. 26
Loading 2406.08074v3…