Source-linked AI summary
Multimodal Neurons in Pretrained Text-Only Transformers
Sarah Schwettmann, Neil Chowdhury, Samuel Klein, David Bau, Antonio Torralba
TL;DR
The paper asks whether vision-to-language transfer happens in the projection layer or deeper inside a frozen text transformer. It analyzes a model with separately learned vision and language components, identifying neurons that translate image semantics. Ablating top-scoring neurons reduces target-token probability, linking them to caption output.
Problem
The study asks whether visual inputs are translated into related text by the projection layer or aligned with language representations inside the transformer.
Source: §1 · p. 1Method
The authors analyze a frozen text transformer augmented with a self-supervised visual encoder and learned projection, using gradient-based attribution to identify neurons contributing to caption tokens.
Source: §1 · p. 1, §2.1 · p. 2Results
80% average decrease in target-token probability followed ablation of top-scoring units; the paper also finds multimodal neurons consistently translate image semantics into language.
Source: §3.3 · p. 4, §4 · p. 4Takeaways & Limitations
The authors suggest crossmodal representational alignment could underlie language models’ utility as general-purpose interfaces for sequential-modeling tasks.
Source: §4 · p. 4The study examines one model, LiMBeR-BEIT; transfer to other architectures or modalities remains to be investigated, and cross-modal vector-space structure is incompletely understood.
Source: §5 · p. 5
Abstract
from arXiv · showhide
Language models demonstrate remarkable capacity to generalize representations learned in one modality to downstream tasks in other modalities. Can we trace this ability to individual neurons? We study the case where a frozen text transformer is augmented with vision using a self-supervised visual encoder and a single linear projection learned on an image-to-text task. Outputs of the projection layer are not immediately decodable into language describing image content; instead, we find that translation between modalities occurs deeper within the transformer. We introduce a procedure for identifying "multimodal neurons" that convert visual representations into corresponding text, and decoding the concepts they inject into the model's residual stream. In a series of experiments, we show that multimodal neurons operate on specific visual concepts across inputs, and have a systematic causal effect on image captioning.
1. Introduction
The paper asks where crossmodal alignment arises when vision and language components are trained separately, and reports that translation occurs inside the transformer through semantically selective neurons that can affect captions.
- The study examines a machine analogue of crossmodal transfer: vision and language components learned separately can still support competitive image captioning.Source: §1 · p. 1
- Translation between visual and textual representations occurs inside the transformer, rather than in image prompts cast into its embedding space.Source: §1 · p. 1
- Multimodal neurons within the transformer respond to particular image semantics.Source: §1 · p. 1
- Modulating multimodal neurons can remove concepts from image captions, showing their causal effect on model output.Source: §1 · p. 1
2. Multimodal Neurons
The section investigates whether neurons in a text transformer can translate visual representations into text when vision and language are learned separately. It identifies candidate neurons through gradient-based attribution and examines their language contributions and image responses.
2. Multimodal Neurons
2.1. Detecting multimodal neurons
- A gradient-based attribution procedure estimates each neuron's contribution to generated caption tokens, with high scores requiring both strong activation and a large effect.The score is a rough first-order heuristic relative to ablating the neuron, and can be computed for all neurons in one backward pass.Source: §2.1 · p. 2
2.2. Decoding multimodal neurons
- For each image, the analysis selects neurons most attributed to its first generated noun and decodes their language contributions.Only interpretable neurons, defined by at least seven dictionary words among their ten top tokens, are retained; this keeps 50% of high-scoring neurons.Source: §2.2 · p. 3
- Decoded neuron tokens score competitively with GPT image captions on CLIPScore and outperform randomized image-neuron pairings on BERTScore.Table 1 reports scores for 1,000 COCO validation images; BERTScore averages five human annotations per image.Source: §2.2 · p. 3, Table 1 · p. 3
- Most top-scoring multimodal neurons occur in GPT-J layers 5–10, and Figure 2 shows their maximally active image regions.GPT-J has 28 layers; the layer pattern is described as consistent with earlier-layer MLP knowledge contributions [26].Source: §2.2 · p. 3
- Decoded image prompts show no significant difference from random embeddings on CLIPScore and BERTScore, unlike the semantic alignment found for neuron-derived tokens.The comparison includes 1,000 randomly sampled COCO images.Source: Figure 3 · p. 3, Table 2 · p. 3
3. Experiments
Image prompts do not readily encode interpretable semantics, while multimodal neurons show visual specificity across inputs and influence generated captions. Ablating top-scoring neurons sharply reduces the probability of the caption’s first noun.
3.1. Does the projection layer translate images into semantically related tokens?
- For image prompts and random embeddings, CLIPScore distributions do not differ significantly (D = .037, p > .5), unlike scores for COCO nouns (D > .9, p < .001).The comparison covers 1000 randomly sampled COCO images.Source: §3.1 · p. 3
- Real and random image prompts have negligibly different BERTScores against human COCO annotations, indicating that prompt inputs do not readily encode interpretable semantics.The comparison also includes COCO nouns and GPT captions.Source: §3.1 · p. 3
3.2. Is visual specificity robust across inputs?
- Across 12 COCO categories, multimodal-neuron receptive fields segment objects better than randomly sampled neurons in the same layers.The neurons also preferentially activate on images from particular categories.Source: Figure 5 · p. 4, §3.2 · p. 4
3.3. Do multimodal neurons causally affect output?
- Across COCO validation images, ablating up to 6400 top-scoring multimodal neurons decreases the probability of token c by 80% on average, whereas the same number of randomly selected units in the same layers largely leaves it unaffected.Ablating multimodal neurons also significantly changes the semantics of GPT-generated captions.Source: §3.3 · p. 4
4. Conclusion
The paper finds multimodal neurons in text-only transformer MLPs that consistently translate image semantics into language. Its findings suggest that translation occurs inside the transformer and that studying individual units may help investigate cross-task generalization.
- Multimodal neurons in text-only transformer MLPs consistently translate image semantics into language.Soft-prompt inputs do not map onto interpretable output-vocabulary tokens, suggesting translation occurs inside the transformer.Source: §4 · p. 4
- The capacity to align representations across modalities could underlie language models’ utility as interfaces for sequential tasks, including game next-move prediction and protein design.Source: §4 · p. 4
- Understanding individual computational units can serve as a starting point for investigating how transformers generalize across tasks.Source: §4 · p. 4
5. Limitations
The study examines one model, LiMBeR-BEIT, whose vision and language components were learned separately. It leaves open how multimodal concepts arise from upstream representations and whether similar neurons emerge in other architectures or modalities.
- The study covers only LiMBeR-BEIT, whose vision and language components were learned separately.Source: §5 · p. 5
- The study does not investigate how individual-unit concepts are assembled from upstream representations, and the structure of cross-modal vector spaces remains incompletely understood.Source: §5 · p. 5
- Whether similar neurons emerge with raw-pixel visual input or a pretrained speech autoencoder remains an open question.Source: §5 · p. 5
S.1. Implementation details
The implementation projects self-supervised BEIT image patches into GPT-J’s embedding space, then supplies them with a text prefix. Neuron attribution ranks units by their contribution to the first noun in a generated caption.
- BEIT encodes each resized image as 196 patches of dimension 1024, and a linear projection maps them to GPT-J soft prompts of dimension 4096.Source: §S.1 · p. 7
- GPT-J receives the image prompts followed by “A picture of” as a (199, 4096) input sequence.Source: §S.1 · p. 7
- The attribution score gk,c ranks a neuron’s contribution to the first noun token in a generated caption, selected using temperature T = 0.If the noun has multiple tokens, c is its first token.Source: §S.1 · p. 7
S.2. Example multimodal neurons
The examples decode high-attribution neurons for COCO images, with interpretable units identified by their decoded tokens.
S.2. Example multimodal neurons
- For sampled COCO validation images, the table lists the top 20 neurons across MLP layers and highlights units whose top tokens meet the paper’s interpretability criterion.A unit is interpretable when at least 7 of its top 10 tokens are English-dictionary words of at least 3 letters; units decoding mainly to fragments or punctuation are excluded from object-focused analyses.Source: §S.2 · p. 7
S.3. Evaluating agreement with image captions
The section tests whether image soft prompts already align with language before GPT-J’s MLP layers. Their nearest-token representations score no better than random prompts, supporting the conclusion that translation happens deeper in the transformer.
S.3. Evaluating agreement with image captions
- BERTScore (F1) evaluates how well a token list corresponds to an image caption’s semantic content.The section uses it to test whether projection-layer outputs already map visual features to related language before transformer MLPs.Source: §S.3 · p. 7
- Because projected vectors are not discrete tokens, each is analyzed through its five nearest token vectors in GPT-J’s embedding space.This mapping supplies the token lists used in the prompt comparison.Source: §S.3 · p. 7
- Image soft prompts have BERTScores indistinguishable from random soft prompts, suggesting they fit the GPT-J embedding space but do not yet map image features to related language.The comparison uses random 4096-dimensional vectors; Table S.2 gives one image and Figure 3 shows the distribution across COCO images.Source: §S.3 · p. 7
S.4. Selectivity of multimodal neurons
The analyses test whether multimodal neurons respond selectively to image semantics rather than acting as generally visually attentive units. Results across ImageNet classes and COCO examples support concept-specific activity.
S.4. Selectivity of multimodal neurons
- Across 10 ImageNet classes, selected neurons activate more frequently on validation images from their own category than on other classes, indicating semantic selectivity rather than general visual attention.The analysis selects 20 frequently appearing units per class from top-scoring interpretable units identified on 200 training images per class.Source: §S.4 · p. 7, Figure S.2 · p. 13
- Decoded top-scoring units provide examples of neurons associated with image concepts, with interpretable units distinguished from those dominated by fragments or punctuation.The table includes attribution scores and activation heatmaps for sampled COCO images.Source: Table S.1 · p. 10
- Activation masks from individual multimodal neurons and their mean are compared with COCO object annotations using intersection over union.Masks are upsampled to 224 pixels and thresholded at the 0.95 percentile before comparison.Source: §S.4 · p. 7, Figure S.1 · p. 12
S.5. Ablating Multimodal Neurons
Ablating multimodal neurons reduces caption similarity relative to random-neuron ablation. Their distribution is concentrated in earlier GPT-J layers, consistent with semantic translation occurring early in the transformer.
S.6. Distribution of Multimodal Neurons
- Most top-100 multimodal neurons across COCO validation images occur in GPT-J layers 5–10 of 28, suggesting cross-modal semantic translation occurs in earlier layers.Figure S.4 counts unique neurons by layer and marks interpretable units using the top-logit word criterion.Source: §S.6 · p. 14, Figure S.4 · p. 14
S.5. Ablating Multimodal Neurons
- Across COCO validation images, ablating multimodal neurons produces a strong decrease in caption similarity compared with ablating random neurons from the same layers.The analysis measures BERTScore against the original caption after incremental ablations; the trend also holds when only interpretable multimodal neurons are removed.Source: §S.5 · p. 14, Figure S.3 · p. 14, Table S.3 · p. 16