Analyzing Individual Neurons in Pre-trained Language Models

1 Introduction

• The study addresses the limited attention to individual neurons by analyzing linguistic information in pretrained representations and comparing its distribution across network parts and architectures.Source: §1 · p. 1
• A very small neuron subset can predict a linguistic task with the same accuracy as the entire network, although information is distributed across networks.Source: §1 · p. 2
• Morphology prediction requires fewer neurons than syntax prediction, while verbs are distributed across many neurons and interjections are localized in fewer.Source: §1 · p. 2
• Lower layers contain more neurons specialized in word-level properties, while higher layers contain neurons specialized in syntax-level information.Source: §1 · p. 2
• BERT is the most distributed model across properties, whereas XLNet has more focused, disjoint sets of neurons and layers for different properties.Source: §1 · p. 2

2 Methodology

• Linear probing uses elastic-net regularization, balancing focused feature selection with distributed neuron contributions; trained classifier weights serve as a proxy for neuron relevance.Source: §2 · p. 2
• Neurons are ranked by the absolute classifier weights for each task label, then selected iteratively according to their contribution to the task's weight mass.Source: §2 · pp. 2–3
• An automated lambda search favors settings that separate top- from bottom-neuron accuracy while limiting accuracy loss from regularization.The search uses α = β = 0.5 and replaces manual lambda selection.Source: §2 · p. 3
• Minimal neuron sets are built by ranking neurons and retraining with progressively larger top-ranked subsets until accuracy approaches the all-neuron Oracle within threshold δ.Source: §2 · p. 3
• Selectivity compares linguistic-task accuracy with a control task that randomly maps word types to labels sampled to match the task's empirical label distribution.Source: §2 · p. 3

3 Experimental Setup

• Experiments compare ELMo, T-ELMo, BERT-base, and XLNet-base, spanning recurrent and transformer architectures and different training objectives.Source: §3 · p. 3
• The evaluation covers POS tagging, CCG supertagging, syntactic chunking, and semantic tagging using the Penn TreeBank, CCGBank, CoNLL 2000, and Parallel Meaning Bank.Source: §3 · p. 3
• Linear probes use elastic-net regularization and categorical cross-entropy, are optimized with Adam, and use the final subword activation as the word representation for subword models.Source: §3 · p. 4

4.1 Ablation Study

• Top-ranked neurons yielded higher accuracy than random or bottom selections; random neurons sometimes performed well on CCG, consistent with distributed information and redundancy.The ablation retained the selected 20% of activations and zeroed out the rest; the authors describe 20% as an arbitrary choice used to demonstrate ranking efficacy.Source: §4.1 · p. 4, Table 1 · p. 4, §4.2 · p. 4

4.2 Minimal Neuron Set

• In POS and SEM, roughly 400 neurons in BERT and XLNet achieved accuracy similar to using all neurons, while Chunking and CCG required larger sets.Across tasks, selected neurons sometimes achieved better accuracy than using all neurons; the largest cited set was up to 2365 neurons, about one third of T-ELMo.Source: §4.2 · p. 4, Table 2 · p. 5
• Neuron counts varied by task complexity, and model size did not correlate with the number of neurons needed for oracle performance.ELMo generally needed fewer neurons, while T-ELMo needed more; T-ELMo also required more neurons for SEM than the other models.Source: §4.2 · pp. 4–5
• Selectivity was much higher with selected top neurons than with all neurons, indicating less memorization; CCG needed up to 33% of the network and had low selectivity.The authors relate CCG's low selectivity to information spread across several neurons, such that random neurons can also achieve decent performance.Source: §4.2 · p. 5
• A looser threshold of δ = 2 can reduce the minimal neuron set further, potentially improving feature-based transfer learning efficiency.The paper notes that large contextualized feature sets can make classifiers cumbersome to train and inefficient at inference.Source: Table 3 · p. 5, §4.2 · p. 6

5.1 Layer-wise Distribution

• Lexical tasks are captured mainly in lower layers, while CCG supertagging shifts top neurons toward higher layers; BERT remains spread across the network.For BERT, the neurons are still concentrated most in final layers for CCG supertagging.Source: §5.1 · p. 6
• BERT draws top neurons from all layers for POS and semantic tagging, while XLNet and other models tend to rely more on lower layers for lexical tasks.For semantic tagging, BERT’s middle layers contribute most; the first post-embedding layer has the largest share for XLNet, T-ELMo, and ELMo.Source: §5.1 · p. 6
• Chunking shifts BERT’s top-neuron contribution from lower and middle layers toward middle and higher layers compared with semantic tagging.The passage attributes this shift to chunking’s greater syntactic complexity.Source: §5.1 · p. 6

5.2 Localization versus Distributedness

• Some labels use fewer neurons than others: interjections and the TO particle are more localized than proper-noun plural, while equating properties use fewer than organization properties.For chunking, particles inside a chunk use fewer neurons than beginning-of-verb-phrase labels.Source: §5.2 · pp. 7–8
• Neuron localization across layers can be shared across architectures: foreign-word neurons occur in final layers, while adjective and location neurons occur in lower layers.The reported layer examples are BERT 13, XLNet 11, T-ELMo 7, and ELMo 3 for foreign words; for adjectives and locations, BERT 0, XLNet 1, T-ELMo 0, and ELMo 1.Source: §5.2 · p. 8
• XLNet’s top neurons are more localized to individual properties than those in other architectures, despite similar total neuron counts for POS and chunking.Other architectures share neurons across multiple properties, whereas XLNet neurons are more exclusive to specific properties.Source: §5.2 · p. 8, Figure 3 · p. 8

Related approaches

• Visualization methods have illuminated representations and attention, but are often qualitative and anecdotal.Quantitative approaches instead correlate network components with linguistic properties using prediction classifiers.Source: §6 · p. 8

Neuron-level analysis

• The paper extends layer- and representation-level probing to individual neurons in pretrained language models.Dalvi et al. (2019) analyzed neurons in sequence-to-sequence models; this work studies a wider range of deeper architectures and automates regularization-parameter grid search.Source: §6 · pp. 8–9

Probing considerations

• Probing can reflect task learning or lexical memorization, and probe complexity raises a trade-off between performance and interpretability.The authors use linear classifiers with control tasks, and leave selectivity-driven neuron selection for future work because selectivity is not directly tunable.Source: §6 · p. 9

Alternative neuron analyses

• Probing classifiers require annotated supervision, while unsupervised neuron-identification approaches have also been used to manipulate translation output.Other work compared internal representations and attention across contextualized models, finding similar representations but different individual neurons.Source: §6 · p. 9

Conclusion

• A small subset of neurons can carry out a downstream NLP task even though information in neural language models is massively distributed.The conclusion frames this as relevant to efficient feature-based transfer learning, including reducing forward passes and classifier feature sets.Source: §7 · p. 9
• The number of selected features varies with task complexity, and word morphology and lexical semantics are mainly associated with lower layers while syntax is associated with higher layers.BERT is an exception to the layer pattern, with neurons spread across the network.Source: §7 · p. 9
• Closed-class words such as interjections use fewer neurons than polysemous words such as nouns and adjectives, while XLNet’s features are more localized to individual properties than in other architectures.The conclusion presents feature-based transfer learning as a direct application of the analysis.Source: §7 · p. 9

A.1 Data and Representations

• The experiments use POS, semantic, chunking, and CCG supertagging tasks with standard training, development, and test splits.The preprocessing splits are available through the repository released with Liu et al. (2019).Source: §A.1 · p. 13
• The pretrained models were obtained from the paper authors through personal communication.Source: §A.1 · p. 13
• Table 4 covers sentence counts for training, development, and test sets, and the number of tags to predict.Source: Table 4 · p. 13

• Elastic-net regularization balances selecting individual neurons versus neuron groups while maintaining the classifier’s accuracy without regularization.Source: §A.2 · p. 13
• The grid search tests L1 and L2 values from 0 to 1e−7.Source: §A.2 · p. 13
• Table 5 lists the best elastic-net lambda parameters for each task.Source: Table 5 · p. 13

A.4 Ablation Study

• The development-set ablation evaluates classifier accuracy after retaining all, top, random, or bottom 20% of neurons and zeroing out the rest.Results are averaged over three runs.Source: Table 6 · p. 13, §A.4 · p. 13

A.3 Infrastructure and Run Time

• The grid search takes O(MN2), reduced to O(N2) for parameter selection and O(M) for the subsequent neuron search; the overall algorithm is O(M + N2).Average end-to-end runtime is 12 hours, ranging from 3 hours for POS with ELMo to 18 hours for CCG with BERT.Source: §A.3 · p. 13

A.5 Minimal Neuron Set

• The minimal-neuron analysis reports development-set accuracy and selectivity for selected neurons, alongside results on the number needed for oracle accuracy in the main text.Table 7 defines comparisons between all-neuron and selected-neuron accuracy and selectivity.Source: §A.5 · p. 14, Table 7 · p. 14

A.6 Localized versus Distributed Labels

• Some labels are captured by focused neuron sets, while others are distributed across larger sets.Source: §A.6 · p. 14

A.7 XLNet versus Others

• XLNet uses strikingly smaller per-label neuron sets than other architectures, with POS tagging totals of 400 neurons for BERT and XLNet, 320 for ELMo, and 430 for T-ELMo.The authors interpret this as XLNet neurons being mutually exclusive across properties, while neurons in other architectures are shared.Source: §A.7 · p. 14

A.8 Layer-wise Distribution

• Figure 5 maps the layer that predominantly captures each label across architectures.Source: §A.8 · p. 14, Figure 5 · p. 16