Source-linked AI summary
Compositional Explanations of Neurons
Jesse Mu, Jacob Andreas
TL;DR
The paper addresses whether neurons are better understood through compositional concepts than atomic labels. It searches logical forms over primitive concepts to approximate neuron behavior and applies them across vision and NLI. The analyses reveal task-dependent links between interpretability and performance and enable predictable copy-paste manipulation of model behavior.
Problem
Atomic-label explanations may oversimplify polysemantic or rule-like neurons, while existing tools cannot automatically surface their compositional concepts.
Method
The method searches logical forms composed from primitive concepts to match neuron behavior and uses these explanations to analyze and target individual neurons.
Results
The method identifies abstract and polysemantic vision concepts, shallow NLI heuristics, task-dependent performance correlations, and predictable copy-paste adversarial effects.
Takeaways & Limitations
Compositional explanations localize abstractions, polysemanticity, and spurious correlations to units and support predictable modification of model behavior.
Takeaways & Limitations
The method requires technical expertise and annotated datasets, and its explanations cannot establish complete concept coverage or definitive causal chains.
Abstract
from arXiv · showhide
We describe a procedure for explaining neurons in deep representations by identifying compositional logical concepts that closely approximate neuron behavior. Compared to prior work that uses atomic labels as explanations, analyzing neurons compositionally allows us to more precisely and expressively characterize their behavior. We use this procedure to answer several questions on interpretability in models for vision and natural language processing. First, we examine the kinds of abstractions learned by neurons. In image classification, we find that many neurons learn highly abstract but semantically coherent visual concepts, while other polysemantic neurons detect multiple unrelated features; in natural language inference (NLI), neurons learn shallow lexical heuristics from dataset biases. Second, we see whether compositional explanations give us insight into model performance: vision neurons that detect human-interpretable concepts are positively correlated with task performance, while NLI neurons that fire for shallow heuristics are negatively correlated with task performance. Finally, we show how compositional explanations provide an accessible way for end users to produce simple "copy-paste" adversarial examples that change model behavior in predictable ways.
1 Introduction
The paper automates neuron explanation by searching compositional logical concepts rather than relying only on atomic labels, enabling broader analysis of learned abstractions, performance relationships, and predictable behavior manipulation.
- Existing interpretability methods often rely on manual inspection or probing datasets with atomic concepts such as dog or verb.
- Atomic labels can oversimplify neurons that combine multiple concepts or detect unrelated features, leaving compositional behavior difficult to characterize automatically.
- The proposed method searches logical forms composed from primitive concepts to produce more expressive explanations of neuron behavior.
- The study examines learned concepts across vision and NLI, their relationship to model performance, and whether explanations support predictable behavior manipulation.
- The introduction reports meaningful perceptual abstractions and unrelated-feature detectors in vision, shallow heuristics in NLI, performance correlations, and copy-paste adversarial examples.
2 Generating compositional explanations
The method represents neuron activations and predefined concepts as comparable binary masks, then searches compositions of primitive concepts for the logical form with highest overlap.
- A neuron is explained by identifying a concept whose activation pattern matches the same inputs as the neuron.
- Predefined atomic concepts are binary functions over inputs, including segmentation masks for image regions such as water.
- Continuous neuron activations can be thresholded into binary masks, which are compared with concepts using Intersection over Union.
- Compositional search: Compositional search expands the explanation space with logical forms built from primitive concepts using operators such as OR, AND, and NOT.
- Compositional search: The search selects the highest-IoU logical form from a structured space, approximated in practice by iteratively constructing formulas with beam search.
3 Tasks
The paper applies compositional neuron explanations across vision and NLI, using task-specific representations and concept inventories to examine learned behavior. These experiments extend atomic concept analysis with compositional visual and sentence-level explanations.
- Image Classification: Vision experiments use a Places365-trained ResNet-18's final 512-unit convolutional layer and 1,105 ADE20k concepts from Broden annotations.The concepts span Scene, Object, Part, and Color categories.
- Image Classification: Vision neuron masks are constructed from bilinearly upsampled activation maps and thresholded using neuron-specific percentiles over the image dataset.The threshold is set so 0.5% of activations exceed it.
- Natural Language Inference: NLI experiments analyze 1,024 penultimate MLP neurons whose inputs are premise-hypothesis pairs rather than intermediate RNN word features.The baseline encodes both sentences with a bidirectional RNN before MLP classification.
- Natural Language Inference: NLI probing uses 10K SNLI validation examples with part-of-speech, common-word, and premise-hypothesis lexical-overlap concepts.Overlap concepts represent thresholds above 0%, 25%, 50%, and 75% IoU between unique words.
- Natural Language Inference: NLI explanations compose AND, OR, and NOT with a NEIGHBORS operator based on the five closest words in GloVe embedding space.The operator is intended to capture firing for groups of semantically similar words.
4 Do neurons learn compositional concepts?
Compositional explanations reveal both meaningful abstractions and unrelated polysemantic behavior in vision, while NLI neurons often encode shallow lexical heuristics. Explanation quality improves with formula length, though substantial behavior remains unexplained.
- Mean IoU rises from 0.059 at N = 1 to 0.099 at N = 10, a statistically significant 68% increase.Scores show diminishing returns after length 10, which is used for subsequent analysis.
- 69% of 128 inspected vision neurons learned meaningful concept combinations, while 31% were polysemantic and fired for unrelated concepts.The meaningful group contained 88 neurons.
- Among meaningful neurons, 50 (57%) learned lexically coherent abstractions, 28 (32%) learned non-coherent abstractions, and 10 (12%) showed specialization.Specialization has the form L1 AND NOT L2 and detects more specific variants of annotated concepts.
- Composition of available annotations explains a majority of learned abstractions, but noisy activations, misclassifications, and absent concepts leave behavior unexplained.
- NLI neurons frequently encode shallow lexical rules, including gender cues, label-associated verbs, and premise–hypothesis lexical overlap.Some neurons were not well explained by the available feature set.
5 Do interpretable neurons contribute to model accuracy?
Interpretability relates differently to accuracy across tasks: more interpretable vision neurons are associated with higher accuracy, whereas more explainable NLI neurons are associated with lower accuracy. The relationship depends on the concept space used for explanation.
- In image classification, neuron IoU correlates positively with model accuracy on inputs where the neuron is active (r = 0.31, p < 1e −13).
- Figure 7 compares neuron IoU with active-input accuracy and shows how their Pearson correlation changes with maximum formula length.The vision analysis uses length 10 and the NLI analysis uses length 3 in the top plots.
6 Can we target explanations to change model behavior?
The paper uses compositional neuron explanations to craft transparent “copy-paste” adversarial examples that predictably alter vision and NLI model predictions.
- Method: Neuron-to-class connection weights identify influential neurons whose explanations guide input modifications toward or away from target predictions.The method probes the final hidden representation before softmax, allowing activation or inhibition of targeted neurons to be related to class contributions.
- Vision: Vision examples modify images using bold length-3 explanations, changing predictions across several Places365 models.Examples include changing non-blue water to blue, which switches swimming-hole predictions toward grotto in many models.
- Vision: Polysemantic explanations expose sensible and spurious scene cues, enabling “copy-paste” examples that predictably alter classification.An igloo pasted into a corridor shifts prediction toward clean room in one model, but not consistently across models.
- NLI: NLI attacks target neurons associated with shallow lexical heuristics, such as “nobody,” couch-related words, or “sitting.”Adding “nobody” changes an entailment example’s true label to neutral while the model predicts contradiction.
- Interpretation: The approach offers a transparent hand-crafted alternative to neuron-visualization search and black-box optimization for constructing adversarial inputs.The paper relates these examples to image-patch attacks, adversarial NLI inputs, and counterfactual NLI data collection.
7 Related Work
The paper extends neuron-level interpretability with globally consistent compositional explanations, distinguishing its scope and grounding from related explanation methods.
- Interpretability: Neuron-level explanations complement representation-level analyses by localizing abstractions, polysemanticity, and spurious correlations to individual units.The paper also notes that neuron-level methods cannot detect concepts distributed across multiple neurons.
- Complex explanations: Unlike Anchors, the method explains globally consistent neuron behavior across a dataset rather than local model behavior in one input neighborhood.It also uses conjunctions and other domain-tailored compositions, not only conjunctions.
- Complex explanations: Logical explanations are grounded in data features and evaluated with explicit IoU quality measures, unlike language explanations generated from black-box methods.Related natural-language approaches primarily explain representations or individual prediction rationales.
- Dataset biases and adversarial examples: The work connects dataset biases and adversarial examples by examining how learned artifacts can make complex models brittle.The related literature describes failures on out-of-domain or bias-reduced datasets and susceptibility to adversarial modifications.
8 Discussion
The discussion summarizes compositional explanations as tools for characterizing neurons, relating them to downstream performance, and manipulating model behavior, while identifying open research directions.
- Conclusions: Compositional explanations more precisely characterize neuron behavior through higher IoU and examples of perceptual abstractions and spurious correlations.They localize these phenomena to specific units in deep representations.
- Conclusions: They can distinguish higher- and lower-quality neurons with respect to downstream performance and support predictable “copy-paste” behavior changes.These conclusions summarize the paper’s performance and intervention findings.
- Open questions: The analysis is limited to penultimate hidden layers, leaving concept formation and composition in intermediate layers unresolved.The paper asks whether other layers can be probed to better understand how concepts form and compose.
- Open questions: Open questions include whether pruning removes lower-quality neurons and whether explanations can replace earlier feature extraction through distillation.The proposed distillation target is a simple classifier over binary concept detectors.
- Open questions: The observed interpretability–accuracy relationship motivates testing neuron interpretability as a training regularization signal.The paper asks whether encouraging interpretable abstractions improves downstream task performance.
Broader Impact
The paper frames compositional explanations as potentially useful for exposing spurious correlations and biases, but emphasizes usability, dataset, and causal-interpretation limitations.
- Potential impact: The explanations may help practitioners understand decisions by revealing richer concepts associated with spurious correlations and problematic biases.This motivation is especially relevant to high-stakes applications of black-box models.
- Limitations: The method requires technical expertise and annotated datasets, limiting non-expert usability while introducing annotation cost and feature-coverage bias.Features absent from the annotated dataset cannot appear in an explanation.
- Limitations: Neuron explanations should not be treated as causal proof or as evidence that an absent explanation means a concept or bias is absent.The paper states that neuron-level explanations cannot identify all concepts encoded in representations or establish definitive causal chains.
- Scope of evidence: The supplementary analysis counts repeated concepts as explanation formula length increases, while Table S1 reports concept occurrence and uniqueness statistics.These materials characterize concept repetition and uniqueness rather than establish causal or impact claims.
A.1 Image Classification
Longer compositional explanations reveal increasing specialization in both vision and NLI neurons. For vision copy-paste examples, adversarial success depends on sub-image size, position, and the specific source-target pair.
- Formula specialization: 97% of concepts are unique at formula length 3, compared with 42% at length 1, revealing greater specialization than NetDissect.All concepts are unique by length 10; at length 1, the mean number of occurrences is 2.61.
- Formula specialization: Longer formula lengths also reveal significantly more specialization in the NLI model's neuron functions.The NLI occurrence statistics are summarized in Table S1, with repeated concepts listed in Table S3.
- Copy-paste examples: Copy-paste adversarial sensitivity depends on the example, with small sub-images often leaving the original class unchanged.The igloo →clean room example is relatively reliable, whereas the street →fire escape example is less reliable.
- Copy-paste examples: Figure S2 codes intended-class changes in green, other-class changes in yellow, and unchanged predictions in red.It varies both the size and position of pasted sub-images.
C Additional image classification examples
The additional image-classification examples are presented as an enumeration rather than a cherry-picked selection.
- Example selection: The examples enumerate neurons 0–39 rather than selecting examples selectively.
- Example categories: The listed examples include physical and perceptual visual concepts associated with image-classification neurons.
D Additional NLI examples
The additional NLI examples are selected from neurons that fire reliably across the validation dataset, with previously illustrated neurons omitted.
- Example categories: The section groups additional NLI examples by semantic and behavioral categories.
- Example selection: The examples use the first 25 neurons that fire reliably, defined as at least 500 activations across the validation dataset.Neurons already illustrated in the main paper are skipped.
- Example presentation: The examples include individual premise-hypothesis cases paired with model predictions.