Source-linked AI summary
Identifying and Controlling Important Neurons in Neural Machine Translation
Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, James Glass
TL;DR
The paper asks whether linguistic information in NMT is distributed across representations or localized in individual neurons, and whether those neurons matter for translation. It develops unsupervised cross-model methods to find important neurons, then evaluates their effects, linguistic content, and controllability. The results show that shared neurons affect translation quality, capture linguistic phenomena, and can control tense, number, and gender to varying degrees.
Problem
It is unclear whether NMT’s substantial linguistic information is fully distributed or can be attributed to individual neurons, a question relevant to understanding learned representations.
Method
The paper develops unsupervised methods that compare neurons across NMT models, then uses masking, linguistic analyses, and activation interventions to evaluate and control them.
Results
Shared neurons affect translation quality more than unshared neurons, many identified neurons capture morphological or syntactic properties, and interventions control tense, number, and gender to varying degrees.
Takeaways & Limitations
Individual neurons can carry interpretable grammatical and structural information, and modifying their activations can control translation output according to specified linguistic properties.
Takeaways & Limitations
The analysis is limited to top-layer neurons, and the authors call for extension to other NMT components, architectures, tasks, and more sophisticated control methods.
Abstract
from arXiv · showhide
Neural machine translation (NMT) models learn representations containing substantial linguistic information. However, it is not clear if such information is fully distributed or if some of it can be attributed to individual neurons. We develop unsupervised methods for discovering important neurons in NMT models. Our methods rely on the intuition that different models learn similar properties, and do not require any costly external supervision. We show experimentally that translation quality depends on the discovered neurons, and find that many of them capture common linguistic phenomena. Finally, we show how to control NMT translations in predictable ways, by modifying activations of individual neurons.
1 INTRODUCTION
The paper investigates whether linguistic information in NMT is localized in individual neurons and whether those neurons matter for translation and controllability. It develops unsupervised discovery methods and finds that important neurons affect translation quality, encode linguistic properties, and enable targeted interventions.
- NMT representations contain morphological, syntactic, and semantic information, but prior analyses targeted whole vectors rather than individual dimensions.
- The authors ask whether individual neurons affect translation quality, encode interpretable linguistic information, and permit control of MT output.
- The proposed unsupervised methods rank neurons by comparing corresponding representations across independently trained NMT models using correlation, regression, and SVCCA.
- Highly shared neurons affect translation quality more than unshared neurons when their activations are masked, supporting the hypothesis that shared information matters.
- Individual neurons capture morphological and syntactic phenomena, identified through activation visualization and supervised classification experiments.
- Modifying individual-neuron activations controls translations for tense, number, and gender with varying success, potentially addressing gender bias.
2 RELATED WORK
Prior representation analyses commonly use external supervision to evaluate linguistic information, while work on individual neurons suggests that localized information may also be meaningful. These approaches leave open how individual NMT neurons can be analyzed without task-specific annotations.
- Studies of word, sentence, and NMT representations typically train classifiers or measure correlations against linguistic annotations.
- Computer-vision research found meaningful individual neurons, although later work questioned their importance and used masking to test unit contributions.
3 METHODOLOGY
The methodology ranks potentially important neurons without external supervision by comparing representations across NMT models, then verifies their importance through erasure, supervision, and visualization. SVCCA additionally evaluates distributed directions rather than only individual neurons.
- The analysis uses correlations between neurons from different models instead of external linguistic supervision.
- Models may differ by training epoch, initialization, dataset, or architecture, while the formal setup represents neuron activations across words.
- Unsupervised correlation methods: MaxCorr ranks each neuron by its highest Pearson correlation with any neuron in another model, targeting properties that emerge strongly across models.
- Unsupervised correlation methods: MinCorr ranks neurons by their lowest best-match correlation across other models, seeking neurons correlated with many models rather than only one.
- Unsupervised correlation methods: LinReg predicts a neuron from another model’s full representation and ranks neurons by regression mean squared error, targeting information distributed across models.
- Unsupervised correlation methods: SVCCA applies PCA retaining 99% of variance, then ranks canonically correlated directions; unlike the other methods, it produces directions rather than individual-neuron rankings.
- Verification: Erasure tests importance by zeroing increasingly many top- or bottom-ranked neurons during translation, with top erasure hypothesized to damage performance more.
- Verification: Supervised verification computes conditional activation variance or fits Gaussian mixture models, while visualization examines neuron activation heatmaps for interpretable behavior.
4 EXPERIMENTAL SETUP
Experiments use multilingual and English auto-encoding NMT models trained on the UN parallel corpus. The recurrent systems support analysis of both word-level and sub-word linguistic properties.
- The dataset comprises English-to-Arabic, Chinese, French, Russian, and Spanish translation tasks plus an English-English auto-encoder.
- For each target language, the authors train three models on different 500K-sentence training subsets, yielding 18 models overall.
- The systems are 500-dimensional, two-layer LSTM encoder-decoder models with attention.
- Character-CNN word representations are used in both encoder and decoder to study word and sub-word properties, including morphology.
5 RESULTS
Erasure experiments show that highly ranked neurons and directions have a much larger effect on translation quality than low-ranked ones. The analyses also find that ranking methods reveal different kinds of information, including position, token identity, and interpretable linguistic properties.
- 5.1 ERASURE EXPERIMENTS: 15-20 BLEU points: erasing the top-ranked 10% of MaxCorr, MinCorr, or LinReg neurons degraded performance, versus 2-3 points for the bottom 10%.For SVCCA, erasing the top 1% caused a 15 BLEU point drop, while erasing 10% produced poor performance.
- 5.1 ERASURE EXPERIMENTS: Erasing neurons from the top of the MaxCorr ranking hurt performance more than erasing neurons from the bottom across three language pairs.The same trend was also observed with other language pairs and ranking methods.
- 5.2 WHAT DO IMPORTANT NEURONS CAPTURE?: LinReg and SVCCA tended to identify current-token information, whereas MaxCorr and MinCorr tended to identify sentence-position information.The contrast suggests that current-token information is often distributed across multiple neurons, while position information is more localized.
- 5.3 LINGUISTICALLY INTERPRETABLE NEURONS: Top neurons captured parentheses, with the property often localized in a unique neuron and ranked highly by MaxCorr across models.The most predictive neuron activated positively inside parentheses and negatively outside them.
- 5.3 LINGUISTICALLY INTERPRETABLE NEURONS: A top-scoring neuron predicted tense with 0.56 F1, activating positively on present-tense examples and negatively on past-tense examples.It also detected irregular past-tense verbs, suggesting sensitivity to context beyond sub-word suffixes, while occasionally confusing plural nouns with third-person present tense.
- 5.3 LINGUISTICALLY INTERPRETABLE NEURONS: Other important neurons represented numbers, dates, adjectives, plural nouns, auxiliary verbs, and noun-phrase segmentation, for which neurons achieved 60-80% accuracy in every network.Many were ranked highly by MaxCorr, whereas other methods ranked them less highly.
6 CONTROLLING TRANSLATIONS
The authors control NMT translations by modifying activations of individual neurons associated with linguistic properties. Success varies by property: tense is easiest to control, while number and especially gender are harder and can trade off against BLEU.
- Control protocol: The procedure sets a selected encoder-neuron activation to α before decoding, using property-specific mean activations to modify a source sentence’s translation.The protocol identifies neurons predictive of target-side properties, modifies the top k neurons, and evaluates the resulting translation.
- Results: 67% success was achieved when changing past to present in an English-Spanish model, with only a 2-point BLEU loss at the best success rate.Success rate is the percentage of cases aligned to a target word with the desired property.
- Results: 37% was the best success rate for controlling number, using the five top number neurons.Increasing the number of modified neurons did not improve control for these properties.
- Results: 21% was the gender-control success rate using the five top neurons, making gender the most difficult property to control.The authors conjecture that number and gender are more distributed than tense.
- Qualitative examples: Examples show that increasing α can change number and gender translations, while tense changes from past to present across five target languages.For number, excessive α produced nonsense translations; Spanish tense examples transitioned through past, imperfect, and present.
7 CONCLUSION
The paper concludes that individual neurons can capture interpretable linguistic properties and that their activations can be modified to control translation output. It frames the methods as extensible beyond the analyzed NMT components, architectures, and tasks.
- Contributions: The study combines unsupervised neuron discovery, translation-quality evaluation, linguistic analysis, and a protocol for controlling translations through neuron modification.The analysis includes quantitative prediction tasks and qualitative visualizations of linguistic properties.
- Examples: Table 4 illustrates control examples for number, gender, and tense by varying α on italicized source words.The caption identifies α as the modification value and “–” as no modification.
- Future work: The analysis can be extended to decoder components, other architectures, and other tasks.The authors also call for further study of localized versus distributed information and more sophisticated translation-control methods.
A ADDITIONAL RESULTS AND VISUALIZATIONS
The additional results show that unsupervised neuron rankings identify neurons predictive of noun-phrase structure and linguistic patterns involving dates, numbers, and sentence position. These neurons also support measurable translation control, especially for tense.
- Noun phrase segmentation: 60-80% accuracy was achieved by high-scoring neurons classifying interior, exterior, or beginning of noun phrases across every network.Many were highly ranked by MaxCorr, linking the unsupervised ranking to external noun-phrase annotations.
- Translation control: 67% of past-to-present and 49% of present-to-past tense changes succeeded when modifying only the most predictive neuron.Other cases often erased tense rather than producing the requested tense, which counted as partial success.
- Noun phrase segmentation: A top-scoring noun-phrase neuron activated positively on the first word and negatively on subsequent words within the phrase.The English-Spanish example scored 79%.
- Dates and numbers: Character-aware models produced highly ranked neurons sensitive to month names, years, and other four-digit tokens.The described neurons included a month detector and an approximate year detector.
- List items: One neuron responded strongly to numbers only when they opened a sentence, capturing a conjunction of position and number format.It did not respond strongly to non-number sentence-initial words or to numbers in the middle of sentences.
B EXAMPLES FOR CONTROLLING TRANSLATIONS
Translation-control examples show that changing individual neuron activations can alter number, gender, and tense realizations. The effects can preserve agreement, but excessive modifications may produce nonsense translations or phrasing changes.
- Number: Increasing the modification value changed “The interested parties” between plural and singular Spanish translations, with agreement preserved across adjectives.Very large positive or negative values produced nonsense translations despite correct number.
- Gender: Increasing the modification value changed translations of “parties” and “questions” from masculine to feminine synonyms.The example used a gender neuron from the English-Spanish model.
- Tense: Modifying tense neurons changed “supported” from past to present in all reported examples, sometimes also changing the phrasing.The Arabic example used a relative-clause formulation after the intervention.
C A CATALOG OF TOP RANKED NEURONS
The catalog records the top-ranked neurons or SVCCA directions for an English-Spanish model and annotates their apparent linguistic properties. It also reports variance associated with sentence position and current-token identity.
- Catalog scope: The catalog lists the top 20 neurons or projected SVCCA directions found by each ranking method for an example English-Spanish model.The entries include comments on properties inferred from neuron visualizations.
- Reported measures: For each cataloged neuron, the analysis reports the percentage of activation variance eliminated by conditioning on sentence position or current-token identity.These quantities characterize how strongly the activation relates to those factors.
- Ranking methods: Separate tables provide top-20 rankings for MaxCorr, MinCorr, LinReg, and SVCCA directions.The supplied table captions identify the ranking outputs for each method.