Natural Language Descriptions of Deep Visual Features

Evan Hernandez; Sarah Schwettmann; David Bau; Teona Bagashvili; Antonio Torralba; Jacob Andreas

Source-linked AI summary

Natural Language Descriptions of Deep Visual Features

Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, Jacob Andreas

arXiv:2201.11114v2Jan 26, 2022cs.CVcs.AIcs.CLcs.LG

TL;DR

Existing neuron-labeling methods cover limited, predefined concepts and cannot systematically reveal novel behaviors. MILAN generates open-ended natural-language descriptions by maximizing pointwise mutual information between candidate descriptions and activating image regions, achieving human-aligned annotations across models and supporting analysis, auditing, and editing. Its descriptions remain partial and potentially noisy because they rely on exemplar-based explanations, human annotations, and a learned captioning model.

Problem
Existing automated neuron-labeling methods require fixed candidate labels, cover only subsets of neurons, and cannot surface novel or unexpected behaviors.
Method
MILAN represents neurons with activating image exemplars and searches for natural-language descriptions that maximize pointwise mutual information with those exemplars.
Results
MILAN obtains higher agreement with human annotations than baselines on held-out networks and supports analysis, auditing, and editing across models and tasks.
Takeaways & Limitations
Fine-grained automatic neuron descriptions can surface meaningful and actionable information about model behavior.
Takeaways & Limitations
The descriptions are partial and potentially noisy because exemplar explanations, human annotations, and the learned captioning model have known limitations.

Abstract

from arXiv · show

Some neurons in deep networks specialize in recognizing highly specific perceptual, structural, or semantic features of inputs. In computer vision, techniques exist for identifying neurons that respond to individual concept categories like colors, textures, and object classes. But these techniques are limited in scope, labeling only a small subset of neurons and behaviors in any network. Is a richer characterization of neuron-level computation possible? We introduce a procedure (called MILAN, for mutual-information-guided linguistic annotation of neurons) that automatically labels neurons with open-ended, compositional, natural language descriptions. Given a neuron, MILAN generates a description by searching for a natural language string that maximizes pointwise mutual information with the image regions in which the neuron is active. MILAN produces fine-grained descriptions that capture categorical, relational, and logical structure in learned features. These descriptions obtain high agreement with human-generated feature descriptions across a diverse set of model architectures and tasks, and can aid in understanding and controlling learned models. We highlight three applications of natural language neuron descriptions. First, we use MILAN for analysis, characterizing the distribution and importance of neurons selective for attribute, category, and relational information in vision models. Second, we use MILAN for auditing, surfacing neurons sensitive to human faces in datasets designed to obscure them. Finally, we use MILAN for editing, improving robustness in an image classifier by deleting neurons sensitive to text features spuriously correlated with class labels.

1 INTRODUCTION

MILAN addresses the limits of fixed-label neuron characterization by generating expressive, open-ended natural-language annotations. It uses mutual-information-guided search to produce descriptions that support analysis, auditing, and editing of vision models.

Existing automated methods use predefined label sets, cover only some neurons, and miss novel or unexpected behaviors.
MILAN generates expressive, compositional, open-ended natural-language descriptions for individual neurons.
MILAN searches for descriptions that maximize pointwise mutual information with the image regions activating each neuron.
The method supports analysis, auditing, and editing by characterizing neuron classes, surfacing sensitive features, and identifying spurious text correlations.

2 RELATED WORK

Prior work explains neurons through visualization or fixed concept categories, while natural-language explanation methods generally target individual decisions. MILAN instead provides model-level descriptions of individual features with broader expressive scope.

Earlier neuron-interpretability methods visualize selected inputs or categorize neurons using predefined concepts.
Natural-language explanation methods mainly explain individual classification decisions and often require task-specific datasets or specialized training.
MILAN generates compositional natural-language descriptions for individual features rather than input-conditional decisions or representations.
MILAN is designed to operate independently of the model being described, making it modular, portable, and to a limited extent task-agnostic.

3 APPROACH

MILAN represents neurons with activating image exemplars and selects descriptions using an information-theoretic objective. Human annotations train the required description models, while constrained search makes generation practical.

Neurons and exemplars: A neuron is represented by input regions whose activation exceeds a fixed threshold.
Neurons and exemplars: MILAN uses exemplar regions because they provide actionable information about neuron behavior despite not fully characterizing broader network function.
Exemplars and descriptions: Unlike fixed candidate inventories, MILAN selects descriptions probabilistically to capture neuron-specific behavior rather than generic image content.
Exemplars and descriptions: The method optimizes pointwise mutual information using models of p(d | E) and p(d), then searches for a high-quality description in natural language.
Collecting human annotations: MILANNOTATIONS contains annotations for 20k neurons from seven vision models, with each exemplar set described by three human participants.
Searching in the space of descriptions: Weighted PMI and beam search rerank likely captions, using λ = .2 and a beam size of 50 in the experiments.

4 DOES MILAN GENERALIZE?

MILAN generalizes across networks and tasks, producing descriptions that agree more closely with human annotations than baseline labeling methods. Its transfer is uneven, and failures arise for unfamiliar concepts, vague descriptions, or context-focused descriptions.

MILAN obtains higher agreement with human annotations on held-out networks than NetDissect, Compositional Explanations, and ordinary image captioning.It also outperforms the p(d | E) decoder, supporting pointwise mutual information as the description objective.
MILAN surfaces highly specific behaviors absent from fixed concept sets, such as neurons responding to splashes of water.This illustrates the expressive scope of open-ended descriptions relative to pre-defined labeling schemes.
MILAN generalizes to novel architectures, including fully connected vision-transformer neurons, but transfer varies across models, datasets, and tasks.Generalization is easiest for new GAN neurons within the same network and hardest from GANs to classifiers; classifier-to-GAN transfer is stronger in the reverse direction.
MILAN sometimes mislabels unfamiliar concepts, produces vague descriptions, or describes surrounding context instead of highlighted regions.The authors present this evaluation primarily as a sanity check of the learned models, not direct evidence of interpretability-tool reliability.

5 ANALYZING FEATURE IMPORTANCE

MILAN descriptions reveal that neurons encoding attributes and relational features are relatively important to classifier behavior, whereas neurons detecting dissimilar concept combinations appear less important. Description-based criteria also vary across network layers.

MILAN enables layerwise analysis by scoring neurons according to syntactic and structural properties of their generated descriptions before ablation.
Neurons captioned with many adjectives or prepositions are relatively important, because ablating them rapidly reduces ImageNet validation performance.These linguistic categories correspond to attribute and relational features.
Adjective-selective neurons are most prevalent in early layers, while neurons with large semantic differences are more prevalent in late layers.
Neurons whose descriptions contain highly dissimilar words are less important, because ablating them hurts performance less than ablating random neurons.Such units can detect non-semantic concept compositions associated with non-robust behavior.

6 AUDITING ANONYMIZED MODELS

MILAN audits whether face-blurring reduces models’ reliance on human-face features. Blurring lowers the number of detected face-selective neurons but does not eliminate neurons responding to unblurred or demographically specific faces.

Blurring reduces but does not eliminate neurons selective for unblurred faces across model architectures.
Models trained on ordinary ImageNet contain 213 MILAN-identified neurons selective for human faces, compared with 142 in models trained on blurred ImageNet.
Some face-selective neurons in blurred models appear to identify female faces or Asian faces, not merely faces in general.These responses are preferential rather than exclusive.
MILAN supports both quantitative and qualitative human-in-the-loop auditing of models trained on privacy-oriented blurred datasets.

7 EDITING SPURIOUS FEATURES

MILAN identifies text-sensitive neurons created by a spurious training correlation and enables targeted deletion of some of them. Removing a small number of such neurons improves adversarial accuracy, though the text reliance is only partially mitigated.

58.8% adversarial accuracy contrasts with 69.9% accuracy on non-spurious test images after training with correlated text labels.
MILAN identifies 300 text-related convolutional units among 1024 examined, showing substantial model capacity devoted to text labels.
Deleting 13 text-sensitive neurons improves adversarial test accuracy by 4.9%, reducing overall error by 12%.Sorting all neurons by validation-accuracy impact improves accuracy by less than 1%.
The editing procedure partially mitigates, rather than completely eliminates, the classifier’s reliance on spurious text features.

8 CONCLUSIONS

MILAN provides a foundation for automatically labeling neuron behavior with natural-language descriptions, while neuron ablation can improve robustness when targeting neurons selective for spurious text.

Ablating neurons that select for spurious text improves ResNet18 adversarial-test accuracy by 4.9 points, exceeding the smaller gain from zeroing arbitrary neurons.Neurons are ablated incrementally after sorting by validation accuracy under single-neuron ablation.
MILAN descriptions support analysis, auditing, and editing workflows for users of deep network models.The procedure selects descriptions by maximizing pointwise mutual information with the image regions where neurons are active.
Without adversarial data to choose the pruning count, robustness improvement of 3.1% remains achievable.

IMPACT STATEMENT

The paper describes MILAN’s human-annotation dataset and its captioning architecture, which converts masked, multi-resolution features from neuron exemplars into descriptions. The dataset spans multiple models, while annotations vary systematically by model layer.

MILANNOTATIONS: Annotators describe highlighted regions in top-activating neuron images without explicitly mentioning the highlighted regions.
MILANNOTATIONS: MILANNOTATIONS contains neuron annotations collected from three distinct human participants for neurons in seven networks.
MILANNOTATIONS: Workers passed a qualification test selecting the most descriptive caption for manually chosen VGG-16 neurons.
Annotation statistics: Annotations near the image generally contain more adjectives, whereas annotations farther from the image contain more nouns and often longer descriptions.The description-length trend is not consistent across all models; in Places365, middle layers are longest on average.
Captioning model: MILAN’s captioner pools masked feature maps from every convolutional layer, concatenates one vector per image, and supplies them to decoder attention.The architecture adapts Show, Attend, and Tell to sets of top-activating images and activation masks.
Captioning model: Masked pooling downsamples each activation mask, multiplies it with each layer’s feature map, and sums spatially to produce a channel vector.
Captioning model: The decoder uses a ResNet101 ImageNet encoder without fine-tuning, a single LSTM, and a maximum decoding length of 15 steps.

C GENERALIZATION EXPERIMENT DETAILS

The generalization experiments evaluate MILAN descriptions on held-out neurons and compare their corpus statistics with human annotations. Generated descriptions are shorter, use fewer unique words, and contain adjectives at different rates than human descriptions.

MILAN descriptions on held-out neurons average 2.7 tokens versus 4.5 for human annotations.
Figure 12 provides randomly chosen examples of MILAN-generated descriptions from the generalization experiments.
MILAN descriptions use 1k unique words versus 4.6k for human annotations.
Adjectives occur in 9.4% of MILAN description words versus 19.8% of human-description words.

D ANALYSIS EXPERIMENT DETAILS

The analysis experiments use MILAN descriptions to select neurons by linguistic criteria and examine their ablation and adversarial behavior. The selected examples include both interpretable patterns and documented description failures.

Method: MILAN annotates every neuron in ResNet18-ImageNet and scores descriptions using four part-of-speech criteria and three structural criteria.The criteria include nouns, verbs, prepositions, adjectives, caption length, parse-tree depth, and maximum word difference.
Results: Examples selected by the criteria include MILAN failure cases, such as brass instruments incorrectly categorized as flutes and a disfluent adjective description.
Results: The analysis examples are intended to confirm the kinds of neurons selected by each linguistic scoring criterion despite the observed failures.
Results: A max-word-difference neuron is directly connected to the final output layer and strongly influences amphibian, hermit crab, and jeep predictions.The neuron’s description mentions dissimilar animal and vehicle concepts.
Results: Cut-and-paste adversarial inputs are constructed by copying and pasting an object mentioned in the max-word-difference neuron’s description.The experiment tests the hypothesis that such neurons correspond to non-robust model behavior.

E EDITING EXPERIMENT DETAILS

The editing experiment uses validation accuracy to choose how many neurons to remove, while tracking the resulting adversarial accuracy. Choosing 8 neurons by validation behavior still increased adversarial accuracy by 3.1%.

E EDITING EXPERIMENT DETAILS: Validation accuracy on the training-distributed held-out dataset starts decreasing earlier than adversarial accuracy after neuron removal.Figure 15 reports validation accuracy and marks initial accuracy with a dotted line.
E EDITING EXPERIMENT DETAILS: Choosing the largest neuron count before validation accuracy permanently decreases selects 8 neurons.This selection rule avoids incrementally testing the model on an adversarial set.
E EDITING EXPERIMENT DETAILS: 8 neurons still produce a 3.1% increase in adversarial accuracy.The result shows that the validation-based selection rule can support the editing experiment’s robustness objective.