Source-linked AI summary
What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, James Glass
TL;DR
Deep NLP models remain difficult to interpret because prior work mostly studied whole representations rather than individual neurons. This paper proposes supervised and unsupervised neuron-correlation methods, evaluates them through ablations on NMT and NLM, and finds structured, partly distributed linguistic information with overlapping salient neurons across methods.
Problem
Prior research largely analyzed linguistic information in whole vector representations, leaving individual dimensions in end-to-end NLP models comparatively understudied.
Method
The paper proposes Linguistic Correlation Analysis using supervised classification and Cross-model Correlation Analysis using similarities across independently trained networks.
Results
Ablation experiments support the neuron rankings and show that linguistic information varies in distribution, with overlapping salient neurons identified by both methods.
Takeaways & Limitations
Neuron-level analysis can reveal dimensions associated with particular linguistic properties and may support model distillation, architecture search, and behavior manipulation.
Takeaways & Limitations
The experiments use standard architectures and leave Transformer and QRNN variants for future work.
Abstract
from arXiv · showhide
Despite the remarkable evolution of deep neural networks in natural language processing (NLP), their interpretability remains a challenge. Previous work largely focused on what these models learn at the representation level. We break this analysis down further and study individual dimensions (neurons) in the vector representation learned by end-to-end neural models in NLP tasks. We propose two methods: Linguistic Correlation Analysis, based on a supervised method to extract the most relevant neurons with respect to an extrinsic task, and Cross-model Correlation Analysis, an unsupervised method to extract salient neurons w.r.t. the model itself. We evaluate the effectiveness of our techniques by ablating the identified neurons and reevaluating the network's performance for two tasks: neural machine translation (NMT) and neural language modeling (NLM). We further present a comprehensive analysis of neurons with the aim to address the following questions: i) how localized or distributed are different linguistic properties in the models? ii) are certain neurons exclusive to some properties and not others? iii) is the information more or less distributed in NMT vs. NLM? and iv) how important are the neurons identified through the linguistic correlation method to the overall task? Our code is publicly available as part of the NeuroX toolkit (Dalvi et al. 2019).
Introduction
The paper shifts interpretability analysis from whole representations to individual neurons, proposing methods to identify salient dimensions and examining how linguistic information is distributed across NMT and NLM models.
- Motivation: Prior analyses often examined linguistic information in whole vector representations while neglecting individual embedding dimensions.The paper positions neuron-level analysis as a response to this representation-level focus.
- Motivation: The paper studies individual neurons post hoc to identify dimensions associated with particular linguistic properties and increase model transparency.The proposed analysis may also support model distillation, neural architecture search, and bias mitigation.
- Methods: Linguistic Correlation Analysis uses supervised classification to extract neurons associated with task-relevant linguistic properties.Neuron activations are extracted from labeled examples for properties such as morphology or semantics.
- Methods: Cross-model Correlation Analysis searches independently trained networks for neurons sharing similar activation patterns without requiring annotated data.It is intended to highlight neurons important to the model itself.
- Findings: The analysis finds that open-class properties are more distributed than closed-class properties, while Cross-model and Linguistic Correlation methods can identify overlapping salient neurons.Position is one example of a property for which both methods identified the same salient neurons.
- Evaluation: The methods are evaluated on neural machine translation and neural language modeling using POS, morphological, and semantic tagging test cases.The evaluation includes identifying important neurons and analyzing distributed versus focused information across the two model types.
Related Work
Prior work mainly analyzed linguistic properties in whole representations, with relatively limited neuron-level analysis; this paper extends the focus using intrinsic and extrinsic methods.
- Representation-level analysis: Earlier studies analyzed word and sentence embeddings, recurrent states, and NMT representations for morphological, semantic, and syntactic properties.These approaches generally probe what information is present in a representation.
- Neuron-level analysis: Only limited prior work examined specific neurons, including neurons associated with length and sentiment.The paper describes neuron-level analysis in neural language models and machine translation as relatively uncommon.
- Cross-domain precedent: Computer-vision research provides precedents for analyzing individual units, including ablation studies testing their contribution to object-class prediction.The paper adapts this unit-level perspective to deep NLP models.
Methodology
The paper analyzes individual neurons in latent representations using supervised linguistic correlations and unsupervised cross-model correlations, then evaluates rankings through ablation.
- Neuron Analysis: A neural model maps input features to latent representations, whose individual dimensions are treated as neurons for analysis.For an NMT system, the model may be the encoder, with word embeddings as input and hidden states as output.
- Linguistic Correlation Analysis: Linguistic Correlation Analysis extracts neuron activations, trains a supervised classifier for labeled properties, and uses classifier weights to rank salient neurons.The method targets properties such as morphology or semantics and uses a linear classifier for interpretability.
- Linguistic Correlation Analysis: Elastic-net regularization balances individual-neuron sparsity with selection of groups of correlated features during ranking.This is intended to support analysis of both localized and distributed information.
- Neuron Ranking: Neurons are ranked by descending absolute classifier weights, selecting those whose cumulative contributions reach specified percentages of total weight mass.An iterative procedure builds an ordering across labels by appending newly identified top neurons.
- Cross-model Correlation Analysis: Cross-model Correlation Analysis ranks neurons by their strongest Pearson correlation with neurons in independently trained models for the same task.The models share architecture and settings but differ in training data and initialization.
- Neuron Ablation: Ablation evaluation zeros ranked neurons incrementally and measures performance degradation using BLEU for NMT or perplexity for NLM.Classification experiments compare top and bottom neuron subsets, with additional retraining using only selected neurons.
Experimental Settings
Experiments use attention-based NMT and LSTM language models across several language pairs and tagged linguistic properties, with separate classifier training and test data.
- Scope: The experiments focus on standard architectures and leave Transformer and QRNN variants for future work.This bounds the architectural scope of the reported evaluation.
- Data: The study evaluates English↔French and German→English translation using subsets of 2 million sentences, splitting each parallel corpus into three subsets for separate models.Language-model training uses the source side of the parallel corpora.
- Language Properties: Linguistic evaluations cover part-of-speech, morphological, semantic, positional, and month-of-year properties.POS and morphology capture word structure, while semantic tagging captures nuanced meaning.
- Classifier Data: Classifier training uses 20k source-side sentences, while testing uses 4k sentences from official test sets with standard property tags.The tagged data support evaluation of classifiers trained on neural-model activations.
Evaluation
The evaluation tests whether neuron rankings identify task- or model-relevant dimensions through classifier masking, visualization, and ablation in NMT and NLM. Results support both interpretable property-specific neurons and model-level rankings, with top-ranked Cross-model neurons strongly affecting performance.
- Linguistic Correlation Analysis: Classifier accuracies generally exceed the local majority baseline, indicating that model activations contain information about the evaluated linguistic properties.The exception is French POS tagging with the NLM classifier.
- Linguistic Correlation Analysis: Masking all but the top-ranked neurons preserves substantially more classification performance than retaining the bottom-ranked neurons.The comparison is performed by masking activations in the trained classifier; analogous trends were also found for morphological tagging.
- Linguistic Correlation Analysis: Visualizations show that individual neurons can focus on specific properties, including verb or article categories, sentence position, months, negation, and numbers.The method also supports inspecting highly activating words to interpret arbitrary neurons in a network.
- Cross-model Correlation Analysis: Cross-model rankings are evaluated by incrementally ablating neurons from the top or bottom of the ranking in the original model.NMT performance is measured with BLEU, while NLM quality is measured with perplexity.
- Cross-model Correlation Analysis: 15-20 BLEU points are lost when only the top 50 neurons, or 2.5%, are ablated, compared with 0.5 BLEU points for the bottom 50.Across languages, ablating from the top causes an earlier performance drop than ablating in reverse order, validating the ranking.
- Cross-model Correlation Analysis: NLM perplexity increases significantly more when top-ranked neurons are erased than when bottom-ranked neurons are ablated.This reproduces the ordering trend observed in the NMT ablation results.
Analysis and Discussion
The analysis compares neuron rankings and ablations across linguistic properties, models, and correlation methods. It finds structured but uneven information distribution, with stronger concentration in NLM than NMT.
- Ranking Evaluation: Top-ranked neurons are evaluated through masking, removal, and classifier retraining to test whether the rankings identify important dimensions.The analysis compares both linguistic-correlation and cross-model rankings.
- Focused versus Distributed Neurons: Open-class properties such as verbs and locations are more distributed across the network than closed-class properties such as conjunctions and determiners.Location is described as very distributed, while some open-class properties are represented by only a few neurons.
- Shared Neurons within and across Properties: Neurons are shared across related properties while others remain specialized, revealing a hierarchy in how linguistic properties are represented.Examples include shared adjective neurons, specialized comparative and superlative neurons, and fine-grained verb neurons aligned with a future-tense neuron.
- Retraining Classifier with the Selected Neurons: Using only 20% of the top neurons allows the classifier to recover much of the performance drop in most cases.The information is distributed across neurons but skewed toward the top-ranked neurons.
- Cross-model Correlation Ranking: Cross-model rankings identify word-position neurons and structural properties such as relations, conjunctions, determiners, and punctuation.The top position neurons overlap with those identified by Linguistic Correlation Analysis.
- Comparing NMT vs. NLM: 12.8% and 25.5% are the top-versus-bottom 10% absolute accuracy gaps for NMT and NLM, respectively, averaged across properties.The authors speculate that NMT distributes information more than NLM, while noting that the difference may reflect bidirectional versus unidirectional architectures.
Conclusion and Future Work
The paper concludes that its two neuron-extraction methods identify salient dimensions for task-specific and model-level analysis. The rankings are supported by ablations and suggest applications in distillation, architecture search, and behavioral manipulation.
- Methods: The two methods extract salient neurons with respect to either an extrinsic task or the model itself.Linguistic Correlation targets dimensions responsible for particular properties, whereas Cross-model Correlation identifies neurons important to the model.
- Validation and Applications: Ablation experiments provide evidence that the neuron rankings identify important dimensions.The paper reports successful experiments manipulating verb-tense neurons to control present- versus past-tense generation.
- Validation and Applications: The methods may support model distillation, neural architecture search, and understanding or manipulating system behavior.The source code is incorporated into the NeuroX toolkit and made publicly available.
Supplementary Material
The supplementary material describes annotated language-property data and reports classifier evaluations for morphological tags using masking and retraining protocols.
- Language Property Data: Annotated data cover French, German, and English tags, with 42k sentences for training and 12k for testing.French POS tags, German POS and morphological tags, and English POS tags were obtained using Tree-Tagger, LoPar, and MXPOST.
- Morphological Tag Results: Morphological-tag experiments evaluate classifier performance when neurons are masked out or when classifiers are retrained using selected neuron subsets.Tables 5 and 6 report masking and retraining results for French and German.