PaperlayerView exact revision on arXiv

Source-linked AI summary

A Multimodal Automated Interpretability Agent

Tamar Rott Shaham, Sarah Schwettmann, Franklin Wang, Achyuta Rajaram, Evan Hernandez, Jacob Andreas, Antonio Torralba

arXiv:2404.14394v2cs.AIcs.CLcs.CV

TL;DR

Understanding model behavior is slow and expensive, and existing automated interpretability methods often provide limited, low-precision hypothesis generation. MAIA combines a pretrained vision-language model with tools for iterative experiments; its neuron descriptions outperform baselines and are often on par with human labels, with applications to model-level tasks.

  • Problem

    Understanding model behavior is slow and expensive, while existing automated interpretability methods often characterize limited inputs and produce low-precision hypothesis generation.

    Source: §1 · p. 1

  • Method

    MAIA equips a pretrained vision-language model with tools to compose experiments, test hypotheses, and explain other systems’ behavior.

    Source: §1 · p. 1, §3 · p. 4

  • Results

    MAIA’s neuron descriptions are more predictive than baseline methods and often on par with human labels; its experimental approach is also applied to spurious-feature removal and bias identification.

    Source: §1 · p. 3, §4.1 · p. 6

  • Takeaways & Limitations

    Initial experiments show promise for interpretability agents, which the authors anticipate will become increasingly useful as they grow more sophisticated.

    Source: §6 · p. 9

    MAIA still requires human steering to avoid confirmation bias and conclusions drawn from small samples.

    Source: §1 · p. 3

Abstract

from arXiv · show

This paper describes MAIA, a Multimodal Automated Interpretability Agent. MAIA is a system that uses neural models to automate neural model understanding tasks like feature interpretation and failure mode discovery. It equips a pre-trained vision-language model with a set of tools that support iterative experimentation on subcomponents of other models to explain their behavior. These include tools commonly used by human interpretability researchers: for synthesizing and editing inputs, computing maximally activating exemplars from real-world datasets, and summarizing and describing experimental results. Interpretability experiments proposed by MAIA compose these tools to describe and explain system behavior. We evaluate applications of MAIA to computer vision models. We first characterize MAIA's ability to describe (neuron-level) features in learned representations of images. Across several trained models and a novel dataset of synthetic vision neurons with paired ground-truth descriptions, MAIA produces descriptions comparable to those generated by expert human experimenters. We then show that MAIA can aid in two additional interpretability tasks: reducing sensitivity to spurious features, and automatically identifying inputs likely to be mis-classified.

1. Introduction

Understanding model behavior is slow and expensive, while existing automated interpretability methods often offer limited, low-precision hypothesis generation. MAIA combines a pretrained vision-language model with experimental tools, producing neuron descriptions that outperform baseline methods and often match human labels while supporting model-level tasks.

1. Introduction

  • MAIA uses a pretrained vision-language model and experimental tools to design experiments, test hypotheses, and explain other systems’ behavior.Its modular API supports novel test-image synthesis and editing for direct hypothesis testing.Source: §1 · pp. 1, 3
  • MAIA’s neuron descriptions are more predictive of behavior than baseline methods and are often on par with human labels.The comparison covers synthetic neurons and neurons in the wild.Source: §1 · p. 3
  • MAIA’s iterative approach is applied to model-level tasks including spurious-feature removal and bias identification in a classifier.The applications demonstrate adaptation to task-specific experiments composed from the API.Source: §1 · p. 3
  • MAIA still requires human steering to avoid confirmation bias and conclusions drawn from small samples.The paper says full automation will require more advanced tools and agents better able to reason about their use.Source: §1 · p. 3

2. Related work

Prior work describes deep features using visual exemplars or generated language labels, but automated labels can be unreliable without further experimentation. MAIA combines iterative experiments with multimodal tools, extending automated interpretability beyond single-pass labeling.

Related work

  • Deep-feature interpretation has used visualizing selected features or categorizing maximally activating real-world inputs, with earlier descriptions also drawn from fixed vocabularies or expressed as programs.The cited approaches include Zeiler & Fergus, 2014; Girshick et al., 2014; Karpathy et al., 2015; Mahendran & Vedaldi, 2015; Olah et al., 2017; Bau et al., 2017; Mu & Andreas, 2021.Source: §2 · p. 3
  • Open-ended feature labels can be unreliable as causal descriptions of model behavior without further experimentation (Huang et al., 2023).Earlier labels were curated by humans or generated directly by learned models.Source: §2 · p. 3
  • The Automated Interpretability Agent protocol used language-only input exploration, whereas MAIA adds interpretability tools and built-in vision capabilities for iterative experiments.MAIA’s modular design supports analyses from individual features to whole-network sweeps and more complex subcomponents (Conmy et al., 2023).Source: §2 · p. 3
  • MAIA uses an agent architecture to generate and test hypotheses about vision networks, building on tool-using language-model agents and multimodal models.The authors describe MAIA as, to their knowledge, the first multimodal agent equipped with tools for interpreting deep networks.Source: §2 · p. 3

3. MAIA Framework

MAIA uses a pretrained multimodal model to compose interpretability subroutines into Python experiments, testing hypotheses and updating them from observed outcomes. Its API exposes systems for probing model components and tools for testing behavior with real or synthetic images and summarizing results.

3. MAIA Framework

  • MAIA runs experiments that test specific hypotheses, observes outcomes, and updates its hypotheses to answer interpretability queries.The system is implemented with a pretrained multimodal model and composes interpretability subroutines into Python programs.Source: §3 · p. 4
  • The VLM receives the MAIA API specification and a task prompt, then writes Python programs that are executed to run experiments.The programs are returned as function-definition strings and executed internally with a Python interpreter.Source: §3 · p. 4

3.1. System API

  • The System class instruments a model’s subcomponents so MAIA can query neuron activations and obtain image masks highlighting maximally activating regions.It supports querying arbitrary vision systems without retraining, unlike specialized approaches that train task-specific models.Source: §3.1 · p. 4

3.2. Tool API

  • The Tools class combines common interpretability procedures into modular functions that MAIA composes to test hypotheses about system behavior.Functions can call pretrained models; for example, text2image generates images from MAIA-written prompts.Source: §3.2 · p. 4
  • MAIA can retrieve 15 ImageNet validation images that maximally activate the system it is interpreting.MAIA often begins experiments with this dataset-exemplar tool.Source: §3.2 · p. 4
  • Text-to-image generation and image editing let MAIA test fine-grained visual differences, contextual selectivity, and hypotheses involving images outside real-world data distributions.The described tools use Stable Diffusion v1.5 and Instruct-Pix2Pix, respectively.Source: §3.2 · p. 4
  • A separate GPT-4V instance can describe highlighted regions or summarize shared image content, while MAIA can log experiment results for later use.The description tool is used when prior hypotheses fail or when images contain complex visual combinations; MAIA chooses what results to log.Source: §3.2 · pp. 4–5
  • The predictive evaluation protocol tests whether candidate neuron labels select prompts whose generated images produce high positive activations relative to neutral examples.Prompts from all methods are pooled before a new language model selects positive and neutral prompts for image generation.Source: Figure 3 · p. 5

4. Evaluation

The evaluation tests MAIA’s neuron descriptions on trained and synthetic vision neurons, then examines how API tools affect description quality and documents image-generation and backbone constraints.

4.1. Neurons in vision models

  • The study samples 100 units from each of ResNet-152, DINO, and CLIP and compares MAIA with MILAN and expert descriptions.Experts label a random 25% subset using the MAIA API.Source: §4.1 · p. 5
  • MAIA labels predict neuron activations better than MILAN across tested models and often match expert predictions.Positive activation values outperform MILAN and are comparable to human descriptions; neutral activations are comparable across methods.Source: Figure 4 · p. 6, §4.1 · p. 6

4.2. Synthetic neurons

  • Synthetic neurons encode single concepts, disjunctions, or conditional concepts, but current text-guided segmentation limits representable concepts to simple ones.The synthetic set contains 85 neurons, and the concepts were verified as segmentable with Grounded DINO and SAM.Source: §4.2 · pp. 6–7
  • In 2AFC judgments, MAIA labels better matched synthetic-neuron ground truth than MILAN and were slightly preferred over expert labels; activation prediction was on par with human labels.The expert-label preference applies to the subset of neurons experts described.Source: Table 1 · p. 7, §4.2 · p. 7

4.3. Tool ablation study

  • The full API performs best with real and synthetic inputs, and DALL-E 3 improves performance over Stable Diffusion.Neither exemplar-only nor generated-only ablations achieve the full system’s average accuracy.Source: §4.3 · p. 7, Figure 7 · p. 8

4.4. MAIA failure modes

  • SD-v1.5 and InstructPix2Pix sometimes fail to faithfully generate or edit requested images, while proprietary alternatives pose API cost and rate-limit constraints.Free, non-proprietary VLM backbones also lag GPT-4V in the reported tests.Source: §4.4 · p. 8

5. Applications

MAIA applies iterative experiments to model-level auditing, including spurious-feature removal and bias identification. In the Spawrious experiment, its selected features improved robustness under distribution shift; its bias experiments surfaced broad failure categories.

5. Applications

  • The applications demonstrate MAIA’s use for model-level spurious-feature removal and bias identification, alongside its broader ability to automate understanding from individual features to failure modes.The applications use the API described in Section 3; an additional experiment evaluates whether MAIA descriptions help humans predict system behavior.Source: §5 · p. 8
  • MAIA’s image editing and generation tools can fail by misinterpreting or incompletely following instructions, limiting the reliability of its experiments.Editing may change too much or too little; generation may deviate from prompts, particularly for negative instructions.Source: Figure 8 · p. 8

5.1. Removing spurious features

  • MAIA’s selected neurons improved accuracy under distribution shift, approaching fine-tuning on balanced data without access to unbiased examples or grouping annotations.The study uses ResNet-18 trained on Spawrious; its training breed-background correlations differ from test pairings.Source: §5.1 · p. 8, Figure 9 · p. 9, Table 2 · p. 9
  • Using ℓ1 regularization on the unbalanced dataset, MAIA evaluated 50 informative neurons and selected 22 robust neurons; an unregularized model on that subset significantly improved accuracy, while MILAN-based procedures achieved significantly lower accuracy.Unbalanced ℓ1 performance dropped as the subset shrank from 50 to 22 neurons; balanced-data ℓ1 fitting on all neurons achieved performance comparable to MAIA’s subset, although MAIA lacked balanced data.Source: §5.1 · p. 9

5.2. Revealing biases

  • MAIA probes ResNet-152 class outputs with synthetic inputs to find settings where class-related images receive lower probabilities or the classifier prefers a subset.Figure 10 shows results for a subset of ImageNet classes; the experiment surfaces broad failure categories.Source: §5.2 · p. 9, Figure 10 · p. 9

6. Conclusion

MAIA composes pretrained modules to conduct experiments that test hypotheses about other systems. Initial experiments show promise, but human supervision remains necessary.

6. Conclusion

  • MAIA automates feature interpretation and bias discovery by composing pretrained modules to make and test hypotheses about other systems.The authors anticipate interpretability agents will become more useful as they grow in sophistication.Source: §6 · p. 9

Impact statement

The impact statement presents MAIA as a prototype for helping people audit AI systems as they take on higher-stakes roles. It augments rather than replaces human oversight, and its explanations do not formally verify performance.

Impact statement

  • MAIA is a prototype intended to help human users ensure AI systems are transparent, reliable, and equitable as scalable reliability auditing becomes important.Source: Impact statement · p. 10
  • MAIA augments rather than replaces human oversight; its explanations do not formally verify system performance, and human supervision is needed to catch mistakes.The authors caution that absence of evidence from MAIA is not evidence of absence.Source: Impact statement · p. 10

Acknowlegements

This section documents MAIA’s neuron-interaction resources and experiment setup, including image tools and an iterative-analysis prompt, alongside funding acknowledgements and competing-interest disclosures.

Acknowlegements

  • The authors acknowledge support from multiple funders and report no competing interests; the funders had no role in the study or manuscript preparation.The named support includes the MIT-IBM Watson AI Lab, Open Philanthropy foundation, Hyundai Motor Company, ARL, Intel, NSF, the Zuckerman STEM Leadership Program, and the Viterbi Fellowship.Source: Acknowlegements · p. 10

A. MAIA Library

  • MAIA’s library computes neuron activations and highlights high-activation image regions, while its ImageNet exemplar tool returns the 15 strongest examples.Exemplars are drawn from real images and masked to emphasize the regions producing maximal activation.Source: A. MAIA Library · p. 15
  • The edit_images tool supports controlled, localized tests of how changing one image aspect affects neuron activation.It generates image prompts and corresponding edited versions to test hypotheses about neuron behavior.Source: A. MAIA Library · p. 15
  • The library also generates and summarizes images, describes unmasked content without access to current hypotheses, and records experiment results for later use.Image summaries focus on shared visual concepts in unmasked regions, while descriptions are intended to provide an unbiased account of image content.Source: A. MAIA Library · pp. 15–16
  • MAIA’s library supports text-to-image generation and summarization of the visual concepts shared across image sets.The examples apply these tools to test neuron activations and summarize dataset exemplars.Source: A. MAIA Library · p. 17
  • The library describes highlighted image regions independently of current hypotheses and logs experiment results for use in later experiments.Logs can include activation values, images, titles, and textual information; results remain available for future experiments.Source: A. MAIA Library · p. 18

B. MAIA user prompt: neuron description

  • MAIA’s prompt instructs the agent to summarize activating inputs, propose multiple possible selectivity hypotheses, and iteratively test them against neuron outputs.It directs experiments to test local and global image differences and says to update hypotheses only when activations exceed prior experiments.Source: B. MAIA user prompt: neuron description · p. 20
  • The prompt requires a final neuron description that is selective and complete, listing distinct concepts with “OR” when a neuron responds to multiple concepts.It also asks for a concise label based on the hypotheses best supported after experimentation.Source: B. MAIA user prompt: neuron description · p. 20

C. Evaluation experiment details

The evaluation appendix describes layer-wise sampling and an activation-normalization procedure whose results follow the same trends as before normalization.

C. Evaluation experiment details

C1. Predictive evaluation with activation normalizing.

C2. Ablation studies

The appendix reports an ablation favoring DALL-E 3 over SD-v1.5 and documents human comparisons, synthetic-neuron construction and limits, and image-tool failures.

C2. Ablation studies

  • Using DALL-E 3 improves performance over SD-v1.5 in the ablation evaluation.The ablation studies use the subset of 25% of neurons labeled by human experts.Source: C2. Ablation studies · p. 21

C3. Human expert neuron description using the MAIA tool library

C4. Synthetic neurons

  • The synthetic-neuron set covers monosemantic, polysemantic, and conditional responses using concepts from 60K human annotations.Source: C4. Synthetic neurons · p. 22
  • Synthetic neurons use Grounded DINO and SAM to detect and segment text-prompted concepts, with similarity scores serving as activation values.Bounding-box accuracy and text-similarity thresholds are 0.3 and 0.25, respectively.Source: C4. Synthetic neurons · p. 22
  • Compound synthetic neurons model polysemantic responses with logical disjunction and conditional responses with logical conjunction.Source: C4. Synthetic neurons · p. 22
  • Text-guided segmentation cannot capture some low-level procedures or higher-level perceptual similarities, limiting synthetic-neuron coverage.Examples include edge detection and radial wheel-and-spoke patterns.Source: C4. Synthetic neurons · p. 22

D1. Tool Failures

  • Image-editing and generation failures sometimes confuse MAIA and lead it to make the wrong prediction.Instruct-Pix2Pix struggles with relative or unusual edits, while Stable Diffusion may assign attributes to the wrong image parts.Source: D1. Tool Failures · p. 24

D2. Confirmation Bias

Tests of alternative MAIA backbones show initial ability to understand neuron-interpretation tasks, but also reveal limitations in experimentation and reliability. MAIA can also stop at an overly specific description after a single generated example.

D2. Confirmation Bias

  • A single higher-activating generated image can lead MAIA to stop testing and output an overly specific neuron label.The example is an underwater image; MAIA does not test whether a more general label fits.Source: D2. Confirmation Bias · p. 24
  • Both LLaVA-Next and Gemini show initial task competence, but have significant shortcomings compared with GPT-4V as MAIA backbones.They understand the task, use the API classes, and write experiment code.Source: D3. Failure modes of non-proprietary… · p. 24
  • Both alternative backbones may describe experiment images instead of updating behavioral hypotheses, and sometimes generate hypotheses or results unrelated to the observed outcomes.Gemini, for example, reported an ungenerated cat image and a hallucinated activation value.Source: D3. Failure modes of non-proprietary… · p. 25
  • Both models sometimes copy API examples rather than designing new experiments, while LLaVA code is often not executable and its one-image input restricts experimentation.Both models generated the example prompt, “a dog standing on the grass”; LLaVA may also focus on the last image when given several in a row.Source: D3. Failure modes of non-proprietary… · p. 25

E. Utility of MAIA descriptions for human users

The study tests whether people can use neuron descriptions to predict image activations. Across three models, MAIA descriptions are more useful than MILAN descriptions for identifying strongly activating image sets.

E. Utility of MAIA descriptions for human users

  • Participants used a neuron description to choose which of four image sets contained strongly activating images; chance performance was 25%.The other three sets were randomly sampled distractors representing baseline activity.Source: E. Utility of MAIA descriptions for… · p. 25
  • MAIA descriptions are more useful than MILAN descriptions for predicting neuron behavior in all three models studied.The task asks participants to distinguish strongly activating from weakly activating images, not rank activity across the full distribution.Source: E. Utility of MAIA descriptions for… · p. 25
  • The table reports human subject predictions from neuron descriptions across interpretation procedures, with 95% confidence intervals for each model.The study used the same subset of 25 neurons per model and 10 participants per task.Source: Table A7 · p. 25, E. Utility of MAIA descriptions for… · p. 25

F. Spurious feature removal experiment

The experiment uses Spawrious to evaluate MAIA’s selection of neurons associated with spurious features. Its appendix describes the dataset, comparison procedures, example interactions, and prompting criteria.

F1. Dataset Selection

  • Spawrious provides a more complex classification task than Waterbirds or CelebA, and its images were generated with Stable Diffusion v1.4.The dataset generator differs from the Stable Diffusion v1.5 model used in MAIA’s tool API.Source: F1. Dataset Selection · pp. 25–26

F2. Experiment Details

  • The experiment compares neuron-selection approaches using a ResNet-18 trained on O2O-Easy and logistic-regression evaluations, including balanced-validation and unbalanced-validation settings.The balanced-validation procedures use a single balanced set for fine-tuning; MAIA’s chosen 22-neuron sparsity is also matched in comparison conditions.Source: F2. Experiment Details · pp. 26–27
  • Unbalanced-validation comparisons include ℓ1-selected neurons, random neuron subsets, MILAN captions, GPT-4V captions, and MAIA-selected neurons.MAIA selected 22 of 50 neurons; the MILAN and GPT-4V caption filters each retained 23.Source: F2. Experiment Details · pp. 26–27

F3. Example MAIA Neuron Robustness Interactions

F4. MAIA Prompting Details

  • The prompt supplies the image domain and definitions of selective and spurious neurons, but does not tell MAIA which correlations are spurious.A selective neuron must activate for one breed across varied contexts; multi-breed activation or context dependence counts as spurious.Source: F4. MAIA Prompting Details · pp. 27–28
  • MAIA is instructed to test hypotheses through repeated tool-based experiments and classify a neuron as spurious when evidence indicates multi-breed or non-dog activation.The prompt advises further tests when uncertain and requires robust selectivity across breeds and contexts for a selective label.Source: F4. MAIA Prompting Details · p. 28

G. Bias identification experiment

The bias-identification prompt asks MAIA to find contexts in which a class-sensitive neuron activates relatively more or less. It directs the agent to form and test bias hypotheses before producing a description.

G. Bias identification experiment

  • The task asks MAIA to identify contexts where a class-sensitive neuron has relatively higher or lower activation, including potential demographic or context dependencies.The prompt asks for biases that cover only part of a class or reduce activation despite the class being present.Source: G. Bias identification experiment · p. 29
  • After observing neuron outputs, MAIA is instructed to summarize shared input features and generate multiple hypotheses about possible biases.The prompt calls for testing local and global image differences with the available tools.Source: G. Bias identification experiment · p. 29
  • The agent is directed to test all hypotheses through further experiments and describe the neuron only after proving or disproving them across multiple experiments.The prompt says to update hypotheses only when later image activation values exceed earlier ones.Source: G. Bias identification experiment · p. 29
Loading 2404.14394v2…