PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding Information Storage and Transfer in Multi-modal Large Language Models

Samyadeep Basu, Martin Grayson, Cecily Morrison, Besmira Nushi, Soheil Feizi, Daniela Massiceti

arXiv:2406.04236v1cs.CV

TL;DR

Prior work has studied information storage and transfer in LLMs, but these mechanisms have not yet been extended to MLLMs. This paper uses constraint-based factual VQA tools to investigate MLLM internals and finds earlier causal layers and consistent visual-token transfer, which it targets with an editing method for correcting and adding facts.

  • Problem

    Information storage and transfer have been studied in LLMs but not extended to multi-modal inputs, leaving these mechanisms in MLLMs less understood.

    Source: §1 · p. 1

  • Method

    The paper combines constraint-based factual VQA, MULTIMODALCAUSALTRACE, and a 9.7K-question constraint-annotated dataset to study LLaVa and multi-modal Phi-2.

    Source: §1 · p. 2

  • Results

    MLLMs use earlier causal MLP layers than LLMs, and a consistent subset of visual tokens transfers image information to those blocks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MULTEDIT targets early causal MLPs to correct errors and insert new long-tailed information into MLLMs.

    Source: §1 · p. 2

    The findings are limited to factual questions about natural images with short-form answers, and the work does not map visual tokens to specific image concepts.

    Source: A Limitations and Ethical Considerations · p. 13

Abstract

from arXiv · show

Understanding the mechanisms of information storage and transfer in Transformer-based models is important for driving model understanding progress. Recent work has studied these mechanisms for Large Language Models (LLMs), revealing insights on how information is stored in a model's parameters and how information flows to and from these parameters in response to specific prompts. However, these studies have not yet been extended to Multi-modal Large Language Models (MLLMs). Given their expanding capabilities and real-world use, we start by studying one aspect of these models -- how MLLMs process information in a factual visual question answering task. We use a constraint-based formulation which views a visual question as having a set of visual or textual constraints that the model's generated answer must satisfy to be correct (e.g. What movie directed by the director in this photo has won a Golden Globe?). Under this setting, we contribute i) a method that extends causal information tracing from pure language to the multi-modal setting, and ii) VQA-Constraints, a test-bed of 9.7K visual questions annotated with constraints. We use these tools to study two open-source MLLMs, LLaVa and multi-modal Phi-2. Our key findings show that these MLLMs rely on MLP and self-attention blocks in much earlier layers for information storage, compared to LLMs whose mid-layer MLPs are more important. We also show that a consistent small subset of visual tokens output by the vision encoder are responsible for transferring information from the image to these causal blocks. We validate these mechanisms by introducing MultEdit, a model-editing algorithm that can correct errors and insert new long-tailed information into MLLMs by targeting these causal blocks.

1 Introduction

The paper addresses the limited study of information storage and transfer in MLLMs by examining factual visual question answering. It introduces constraint-based tools for studying these mechanisms and reports earlier causal layers and visual-token transfer, then edits those layers to correct or add facts.

1 Introduction

  • The paper studies MLLM information storage and transfer because these mechanisms remain less understood than their LLM counterparts despite MLLMs’ expanding use.The authors identify factual VQA as a common MLLM use case [35].Source: §1 · pp. 1–2
  • The paper introduces a constraint-based factual VQA framework, MULTIMODALCAUSALTRACE, and VQA-Constraints, a dataset of 9.7K questions annotated with constraints.The study applies these tools to LLaVa and multi-modal Phi-2.Source: §1 · p. 2
  • MLLMs use earlier causal MLP layers than LLMs, while middle-layer self-attention transfers image information from a consistent subset of visual tokens to the final token.For LLaVa, the causal MLPs are in layers 1–4; the reported visual subset includes roughly the last 36 CLIP tokens.Source: §1 · p. 2, Figure 1 · p. 2
  • MULTEDIT edits early causal MLPs with a closed-form update to correct errors and insert new long-tailed facts into MLLMs.The method is evaluated on VQA-Constraints and Encyclopedic VQA [25].Source: §1 · p. 2
  • MULTIMODALCAUSALTRACE studies information storage by corrupting a question constraint and copying clean-model activations into the corrupted model until its output probability is restored.The framework addresses a gap: prior work studied factual storage and transfer in LLMs, but not for multi-modal inputs.Source: §1 · p. 2, Figure 2 · p. 3

2 Related Works

Prior work covers two broad MLLM architectures and has studied only a few aspects of MLLM interpretability. This paper focuses on the visual-token projection family and investigates how vision and text inputs relate causally to model internals.

Multimodal Large Language Models

  • MLLMs integrate vision either through cross-attention at each language-model layer or by projecting vision outputs into visual tokens; this paper focuses on the latter family.The cited examples include Flamingo and BLIP for cross-attention, and LLaVA and Bunny for visual-token projection.Source: §2 · p. 3

Interpretability of MLLMs

  • Earlier MLLM interpretability work visualized attention or examined CLIP shortcomings, but did not study how both vision and text inputs influence internal states causally.The paper’s projection-layer editing approach complements cross-attention-layer insertion baselines.Source: §2 · p. 3

3 A Constraint-Based Framework for Studying Information Storage and Transfer in MLLMs

The framework treats factual visual questions as sets of visual and textual constraints the answer must satisfy. It combines causal tracing and attention contributions with a constraint-annotated VQA test-bed to study information storage and transfer in MLLMs.

3.2 MULTIMODALCAUSALTRACE: Studying Information Storage in MLLMs

  • Single-constraint information is retrieved mainly from early MLP and self-attention blocks, while visual-and-textual constraints involve early and middle blocks.The figures report these patterns for LLaVa; Figure 3 uses a window size of 3, and Figure 4 shows the OK-VQA dataset.Source: Figure 3 · p. 4, Figure 4 · p. 5

3.1 A Multi-modal Constraint-based Framework

  • Visual questions are classified as single-constraint or multi-constraint depending on whether they require a visual constraint alone or both visual and textual constraints.A visual constraint refers to an entity in the image; for example, “this director” paired with the textual constraint “2006.”Source: §3.1 · p. 4

3.2 MULTIMODALCAUSALTRACE: Studying Information Storage in MLLMs

  • MULTIMODALCAUSALTRACE replaces visual-constraint token IDs to corrupt the model, then restores clean activations and tracks the output-probability increase as the indirect estimation effect.The method avoids adding Gaussian noise across the large set of visual tokens; for questions with textual constraints, those tokens can be corrupted with noise or replacement.Source: §3.2 · p. 5

3.3 Studying Information Transfer in MLLMs with Attention Contributions

  • Attention contributions track information transfer from visual tokens to causal layers and from the last visual-constraint token to the question’s final token.The analysis uses the last token of the visual constraint as the destination for the first transfer and as the source for the second.Source: §3.3 · p. 6

3.4 VQA-Constraints: A Constraint Annotated Test-Bed for VQA

  • VQA-Constraints contains 9.7K natural images paired with factual questions annotated for visual and textual constraints.It draws on OK-VQA, Movies, and Known; GPT-4 annotates constraints in new OK-VQA and Known questions using 100 manually annotated examples per dataset, while Movies uses templated constraints.Source: §3.4 · p. 6

4 Key Findings in how MLLMs Store and Transfer Information

The paper traces how MLLMs retrieve information for visual questions and how that information moves through the model. It reports early-layer retrieval, visual-token transfer, and mid-layer attention patterns, including a signal associated with answer correctness.

4.1 Finding 1: Early MLPs and self-attention layers are causal

  • MLLMs retrieve visual-question information mainly from early MLP and self-attention blocks; LLaVa’s causal MLPs are in layers 1–4 versus LLaMA’s layers 4–7, with consistent traces at window size 3.LLaVa yields causal traces at window size 1, while LLaMA requires at least 5 for significant traces; window size 3 gives consistent traces across VQA-Constraints datasets.Source: §4.1 · pp. 6–7
  • For questions with both visual and textual constraints, textual information is retrieved from early and mid-layer blocks and requires a window size of at least 6.The authors suggest this indicates that meeting both constraints requires more parametric memory.Source: §4.1 · p. 7

4.2 Finding 2: Only a subset of visual tokens are involved in transferring information from the image to the early causal MLP layers.

  • Late visual tokens transfer image information to early causal MLPs through the first self-attention layer, with the highest contributions from LLaVa tokens 540–576.This pattern holds across all three VQA-Constraints datasets; the authors hypothesize these tokens may summarize question-relevant image information.Source: §4.2 · p. 7, Figure 5 · p. 7

4.3 Finding 3: Mid-layer self-attention layers are involved in transferring information from the early causal layers to the question’s final token

  • Middle-layer self-attention propagates information from early causal layers to the question’s final token; LLaVa’s layers 16–17 are most active.The answer is sampled at the final token, and the authors note that this transfer pattern resembles LLMs’ use of mid-layer attention.Source: §4.3 · p. 7

4.4 Finding 4: Mid-layer self-attention contributions can be used to predict whether a MLLM will generate a correct answer, but model confidence is a more reliable predictor

  • For LLaVa, middle-layer attention contributions are higher for correct answers, but confidence predicts correctness better: AUROC 0.63 versus 0.76 on VQA-Constraints’ Known dataset.The attention signal uses contributions between the last constraint token and the last prompt token in layers 16–17; the authors describe it as a coarse early failure detector.Source: §4.4 · pp. 7–8

5 Correcting and Inserting Long-Tailed Information in MLLMs

MULTEDIT edits early causal MLPs to correct wrong answers and insert long-tailed information. The authors report efficacy and generalization on both tasks, while also testing effects on unrelated questions and comparing against fine-tuning baselines.

5.1 MULTEDIT

  • MULTEDIT treats the projection matrix as a key-to-value memory: it extracts constraint-token keys, optimizes output values to raise correct-answer probability, and updates the matrix while limiting weight deviation.The weight constraint is intended to preserve performance on unrelated VQA pairs.Source: §5.1 · pp. 8–9

5.2 Experimental details

  • Tests cover about 450 incorrectly answered Known questions and rare-landmark questions from Encyclopedia-VQA, comparing two fine-tuning baselines using efficacy, paraphrase generalization, and unrelated-question specificity.The baselines fine-tune all layers with a language-modeling objective or fine-tune with a constraint on the weights.Source: §5.2 · p. 9

5.3 Results

  • For incorrect answers, MULTEDIT raises the average correct-answer probability from 0.07 to 0.82 and reaches 0.76 on paraphrased questions, outperforming both fine-tuning baselines across metrics.Accuracy on unrelated image-question pairs drops by 1.5%.Source: §5.3 · p. 9
  • For long-tailed information, MULTEDIT achieves editing efficacy of 0.83 and paraphrase efficacy of 0.79, with a 1.4% drop on unrelated questions and less impact than other methods.The long-tailed questions come from Encyclopedia-VQA and concern rare landmarks.Source: §5.3 · p. 9, §5.2 · p. 9
  • Ablations show that editing early causal MLP layers yields better editing efficacy than editing middle or later MLP layers.Source: §5.3 · p. 9

6 Conclusion

The paper develops tools and insights for studying multimodal information processing, and introduces MULTEDIT to target early causal MLPs when correcting or adding knowledge.

6 Conclusion

  • The work introduces multimodal causal tracing, the VQA-Constraints test-bed, insights into MLLM information retrieval and transfer, and MULTEDIT for editing early causal MLPs.The authors report that MULTEDIT can fix errors or introduce long-tailed knowledge through a closed-form update.Source: §6 · p. 9

A Limitations and Ethical Considerations

The findings are scoped to factual questions about natural images with short-form answers; the authors identify other question types, image domains, and answer lengths for future study.

A Limitations and Ethical Considerations

  • The findings cover factual questions on natural images with short-form answers, leaving subjective questions, charts, and longer answers for future investigation.The authors also note that visual tokens are not mapped to specific image concepts and that MULTEDIT could add false or harmful information.Source: A Limitations and Ethical Considerations · p. 13

B VQA-Constraints Details

VQA-Constraints distinguishes visual and textual constraints and focuses mainly on single-constraint questions. The appendix reports that standard causal tracing fails, while MULTIMODALCAUSALTRACE recovers early causal states with a window size of 1.

B VQA-Constraints Details

  • VQA-Constraints contains 9.7K single-constraint questions and 650 multi-constraint questions, with the study primarily focused on the single-constraint set.Visual constraints refer to question words identifying an image entity; text constraints combine with the visual constraint to answer the question.Source: B VQA-Constraints Details · p. 13

C Standard Causal Tracing Does Not Recover Causal States

D MULTIMODALCAUSALTRACE - Qualitative Plots

E Multimodal Phi-2 Tracing Results

Multimodal Phi-2 shows early causal MLP and self-attention layers, consistent with the paper's LLaVa observations. A window size of 1 is sufficient to recover causal states.

E Multimodal Phi-2 Tracing Results

F Multi-constraint Questions Plots

For multi-constraint OK-VQA questions, tracing at the text-constraint position finds causal states across early and middle MLP and self-attention layers. Recovering those states requires a larger window, at least size 6.

F Multi-constraint Questions Plots

G Model Editing Formulation and Further Results

The section reports stronger constraint-token attention for correct answers and compares editing and early failure-detection results. It also describes MULTEDIT’s closed-form update and its implementation choices.

G Model Editing Formulation and Further Results

  • MULTEDIT’s closed-form objective maps old keys to optimized values intended to drive the model toward a higher target-output probability, with regularization.The section states that this objective can be solved with a closed-form update.Source: G Model Editing Formulation and… · p. 17
  • The closed-form editing update can be solved on a CPU given access to keys and values, and the authors describe it as their first use of such an update for MLLMs.They note prior use of closed-form updates for text-to-image models.Source: G Model Editing Formulation and… · p. 17
  • Among early causal layers 0–5, editing Layer 2 gave the best efficacy, using learning rate 0.1 and regularization factor 0.01.The regularization value was selected after a grid search.Source: G Model Editing Formulation and… · p. 17
  • Editing early causal MLP layers is more effective than editing middle or later layers, though those other layers also yield non-trivial editing efficacy.The figure locates editing at the last token of the visual constraint.Source: Figure 14 · p. 17, G Model Editing Formulation and… · p. 18

H Information Transfer and Early Failure Mode Detection - Qualitative Plots

I Editing Dataset

The editing dataset combines failed Multimodal Known answers with long-tailed Encyclopedia-VQA questions, primarily about landmarks; experiments used Nvidia-A6000 and A5000 GPUs.

I Editing Dataset

  • The editing set combines Multimodal Known questions answered incorrectly by LLaVa-7B with long-tailed Encyclopedia-VQA questions, primarily about landmarks.The figure presents qualitative examples from the long-tailed subset, with questions involving landmarks sourced from Encyclopedia-VQA.Source: I Editing Dataset · p. 19, Figure 18 · p. 20
  • The long-tailed editing examples concern landmark questions from Encyclopedia-VQA.These are qualitative examples from the long-tailed knowledge-editing subset.Source: I Editing Dataset · p. 19, Figure 18 · p. 20

J Compute

Loading 2406.04236v1…