Source-linked AI summary
Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals
Yanai Elazar, Shauli Ravfogel, Alon Jacovi, Yoav Goldberg
TL;DR
Probing reveals what information can be extracted from neural representations but cannot establish whether the model uses it for prediction. The paper introduces Amnesic Probing, which removes properties from representations and measures behavioral changes, finding that conventional probing performance is not correlated with task importance.
Problem
Probing performance shows that information is extractable from a representation, but does not establish whether the model uses that information for its task.
Method
Amnesic Probing removes a property from a representation and measures the resulting influence on the model's task behavior.
Results
Across analyses of BERT, conventional probing performance is not correlated with task importance, and linguistic properties can influence masked and non-masked settings differently.
Takeaways & Limitations
Behavioral or causal conclusions should not be inferred from probing results alone; information encoding and task use must be analyzed separately.
Takeaways & Limitations
The intervention removes only linearly present information, so the removed representation should be treated as an approximation of the abstract property and causal interpretations require caution.
Abstract
from arXiv · showhide
A growing body of work makes use of probing to investigate the working of neural models, often considered black boxes. Recently, an ongoing debate emerged surrounding the limitations of the probing paradigm. In this work, we point out the inability to infer behavioral conclusions from probing results and offer an alternative method that focuses on how the information is being used, rather than on what information is encoded. Our method, Amnesic Probing, follows the intuition that the utility of a property for a given task can be assessed by measuring the influence of a causal intervention that removes it from the representation. Equipped with this new analysis tool, we can ask questions that were not possible before, e.g. is part-of-speech information important for word prediction? We perform a series of analyses on BERT to answer these types of questions. Our findings demonstrate that conventional probing performance is not correlated to task importance, and we call for increased scrutiny of claims that draw behavioral or causal conclusions from probing results.
1 Introduction
Probing can show that information is encoded in neural representations, but not whether the model uses it for prediction. Amnesic Probing instead removes a property from representations and measures the resulting behavioral influence.
- Conventional probing performance does not establish that the probed information is used by the model for prediction.Probing trains a simple predictor to recover information, but high accuracy only shows extractability; it provides no evidence for actual model use.
- Amnesic Probing removes a property from a representation and measures how the intervention changes performance on the main task.The method treats a performance decrease after removing property Z as evidence that Z contributes to task T, while little influence suggests no significant contribution.
- The intervention operates on representation layers, making it easier than changing inputs and more efficient than querying combinations of neurons.The method uses Iterative Nullspace Projection to construct counterfactual representations that no longer encode the targeted property linearly.
- The work presents Amnesic Probing as a debugging and analysis tool for determining whether a property is used in prediction.This addresses questions about what would happen to a model's prediction if information were removed from its representation.
- The paper applies Amnesic Probing to BERT and finds that high probing performance is not correlated with task importance.The analyses cover POS and dependency labels, masked and non-masked settings, fine-grained labels, and layer-wise interpretations.
2 Amnesic Probing
Amnesic Probing evaluates the behavioral influence of information by removing it from representations and comparing the model's predictions. INLP performs the removal, while random-direction and selectivity controls help distinguish property-specific effects from generic information loss.
- Amnesic Probing removes property Z from h(x_i) and observes how the classifier's prediction changes on the task of interest.The model is decomposed into an encoder h and classifier c, with the intervention applied to the encoded representation rather than retraining the original model.
- INLP trains sequential linear probes and iteratively removes their property-associated directions, neutralizing linear predictability of Z from representations.The procedure preserves the original model while constructing a counterfactual representation with reduced linear access to the targeted property.
- Amnesic Probing complements conventional probing by using probe-associated directions to derive behavioral outcomes after establishing that the property is decodable.Random probe accuracy indicates that the property cannot be used by the model from that representation in the first place.
- Controls: The intervention can reduce performance for reasons unrelated to the targeted property, including removal of correlated information or arbitrary representation changes.These controls are necessary because INLP removes directions identified by classifiers and may affect features that correlate with the property of interest.
- Control over Information: The random-direction control compares property-directed removal with removing the same number of directions at random.If random removal has lower impact, the targeted property occupies important task directions; similar or greater random impact provides no evidence of property usage.
- Control over Selectivity: Selectivity is tested by restoring the removed property explicitly and fine-tuning subsequent layers to see whether original performance returns.Recovery supports the targeted property as sufficient to account for the damage, whereas incomplete recovery indicates that additional information may have been removed.
3 Studying BERT: Experimental Setup
The experiments use BERT to study which linguistic information supports masked language-model word prediction. They remove six sequence-tagging properties from representations and evaluate changes in prediction distributions and accuracy.
- The study investigates BERT's use of linguistic information for predicting words from context rather than only measuring whether that information is decodable.The model is BERT-BASE-UNCASED, a masked language model used in prior analysis work.
- The six sequence-tagging tasks are coarse- and fine-grained POS, dependency labels, named entities, phrase starts, and phrase ends.The datasets are English UD Treebank for POS and dependencies, and English OntoNotes for named entities and constituency boundaries.
- The experiments train on 100,000 random tokens from the relevant datasets for each task.
- LM accuracy measures word-prediction accuracy, while D_KL measures the change between token distributions before and after intervention.Larger D_KL values indicate a more substantial change in the model's full output distribution.
4 To Probe or Not to Probe?
Amnesic probing compares conventional probe accuracy with the effect of removing linguistic properties from representations during word prediction. The results show that dependency information, POS, and NER matter for prediction, whereas phrase-boundary properties do not, and probing accuracy does not predict task importance.
- After neutralizing each studied property, word-prediction performance drops to majority, while selectivity controls usually restore performance.Dependency performance is not fully restored, suggesting that some nonrelated features were removed alongside dependency features.
- Dependency information, POS, and NER are important for word prediction, with the largest LM-accuracy drops for dependency and fine-grained POS information.The reported decreases are 87.0 and 81.8 accuracy points for dependency and fine-grained POS, and 32.2 and 10.8 points for coarse POS and NER.
- Phrase-start and phrase-end information slightly improves accuracy when removed, suggesting these properties are not used for prediction at this model stage.Accuracy increases by 0.21 and 0.32 points respectively, while their controls perform worse.
- Probe accuracy does not correlate with task importance as measured by amnesic probing.The reported Spearman correlation is 8.5 with p-value 0.871.
- These findings support increased scrutiny of behavioral or causal conclusions drawn from probing performance alone.The results align with prior work questioning probing as an analysis tool [Hewitt and Liang, 2019; Ravichander et al., 2020].
5 What Properties are Important for the Pre-Training Objective?
The masked pre-training setting differs from the unmasked representations commonly used for probing, so the study evaluates property removal directly on masked-token representations. Overall trends remain similar, but the intervention produces a steeper performance decline than random removal and selectivity is mostly recovered.
- The masked analysis evaluates word-prediction accuracy and DKL using Vanilla, Amnesic, random, and 1-Hot controls.Table 2 reports these measures for the inspected properties and uses the controls to assess information loss and selectivity.
- Masked-token experiments largely reproduce the non-masked trends, but baseline LM accuracy is lower because the model must infer each word from context.The experiments use BERT representations of tokens replaced by [MASK], with surrounding tokens left intact.
- Selectivity is recovered for coarse POS, NER, and phrase-marker properties, whereas dependency and fine-grained POS performance does not fully recover.The incomplete recovery for dependency and fine-grained POS may relate to neutralizing most representation dimensions; the study notes that class count may contribute to these gaps.
- Amnesic probing shows a steeper LM-accuracy decline than random direction removal across INLP iterations, indicating that the removed properties correlate with word prediction.Figure 2 compares Vanilla, random-control, and INLP trajectories; the number of removed dimensions per iteration varies with the property’s number of classes.
6 Specific Labels and Word Prediction
Fine-grained amnesic probing shows that linguistic labels differ substantially in their importance for masked word prediction, with function-word categories affected most.
- Removal of Specific Labels: Table 4 reports masked word-prediction accuracy after removing individual fine-grained tag distinctions, with random-control accuracy between 56.05 and 56.49.The caption states that the maximum random-control difference from Vanilla is 0.92 points.
- Fine-Grained Analysis: Removing POS information causes large masked word-prediction drops for conjunctions, particles, and determiners, but much smaller drops for nouns, numbers, and verbs.The reported drops are 73.73, 77.66, and 65.65 for conjunctions, particles, and determiners, versus 8.64, 6.91, and 11.73 for nouns, numbers, and verbs.
- Fine-Grained Analysis: After POS removal, predicted function words can become content words, such as ‘and’ becoming ‘rotate’ and ‘of’ becoming ‘say’.The authors use POS-tag confusion to quantify these changes.
- Removal of Specific Labels: Removing individual POS-versus-rest distinctions again minimally affects nouns and verbs but strongly affects determiners and punctuation.The pattern is consistent with the broader fine-grained POS-removal results.
7 Behavior Across Layers
Across BERT layers, amnesic interventions reveal setting-dependent recoverability and layer importance: masked and non-masked inputs produce substantially different patterns.
- 7.1 Property Recovery After an Amnesic Operation: For non-masked POS representations, removing information from the first seven layers is followed by relatively strong recovery, whereas later-layer removal causes larger probing drops of 16.57–46.39 points.The corresponding early-layer drops range from 5.72 to 12.69 accuracy points.
- 7.1 Property Recovery After an Amnesic Operation: Masked POS information is less recoverable after lower-layer removal, with second-layer removal leaving subsequent probing scores between 32.7 and 42.1.Across properties, masked recoverability has no consistent general trend, unlike the non-masked experiments.
- 7.2 Re-rediscovering the NLP Pipeline: For masked word prediction, POS information is most important around layer 3 and layer 12, NER around layer 4, and phrase markers mainly in middle layers.Phrase-start importance peaks at layers 5 and 7, while phrase-end importance peaks at layer 6.
- 7.2 Re-rediscovering the NLP Pipeline: Amnesic layer-importance conclusions partly differ from the “Pipeline processing” hypothesis because ablation measures task impact, whereas prior work localized where information was extractable.For NER, both approaches emphasize later layers; for POS, amnesic probing instead finds a 40.99-point loss at layer 11.
- 7.1 Property Recovery After an Amnesic Operation: Masked and non-masked results should not be directly compared because confounding factors, including the number of removed dimensions, are uncontrolled.The authors call for clearer separation of the two settings in future work.
8 Related Work
Related work has challenged probing as a direct interpretation method, motivating approaches that distinguish information extractability from information use and causal attribution.
- Probing Critiques: Hewitt and Liang (2019) showed that probes can perform well on linguistically meaningless control tasks, separating decoding ability from learning the intended task.Their control tasks assign consistent but linguistically meaningless labels to tokens.
- Causal Attribution: Ravichander et al. (2020) found that models can encode linguistic properties unnecessary for the task, while this paper removes properties and measures the main-task impact.The approaches address related concerns from opposite directions: encoding versus behavioral influence.
- Information-Theoretic Perspectives: Information-theoretic work proposes alternatives to raw probing accuracy, including ease of extractability and minimum description length.Pimentel et al. (2020) use mutual-information arguments, while Voita and Titov (2020) use MDL to quantify transmission costs.
- Causal Attribution: Feder et al. (2020) concurrently studied causal attribution of concepts to representations using adversarial training guided by causal graphs.Their approach is presented as similar in goal but distinct in method.
9 Discussion
The discussion frames Amnesic Probing as an approximate intervention: INLP targets linearly represented information, but intertwined representations and correlating features limit causal interpretation.
- Methodological Limitations: Complete property neutralization with selective preservation of other information is difficult because neural representations may intertwine multiple properties.The discussion notes that disentangled representations may not exist or may be difficult to detect.
- Methodological Limitations: INLP provides only a first-order approximation because it removes linearly present information, while classifiers may exploit correlated or spurious features.The authors therefore treat the removed signal as an approximation to the abstract property of interest.
- Open Questions: Amnesic probing cannot yet quantify the relative importance of different properties because different erasure amounts make cross-property comparisons difficult.The authors explicitly avoid claims such as dependency information being more important than POS.
10 Conclusions
The paper presents Amnesic Probing as a method for quantifying how specific properties influence task-trained models, arguing that counterfactual intervention gives a richer view than conventional probing.
- 10 Conclusions: Amnesic Probing quantifies the influence of specific properties on a model trained for a task.The paper frames it as an analysis tool for studying how linguistic information is encoded and used.
- 10 Conclusions: Experiments examine linguistic phenomena, masked language modeling, BERT representations, fine-grained properties, and layer-specific property use.The analyses compare unmasked and masked representations and study how different layers use different properties.
- 10 Conclusions: The experiments were logged and tracked using Weights and Biases.
A Appendix
The appendix reports main-task performance during INLP iterations and visualizes language-model accuracy against INLP predictions for masked tokens.
- A Appendix: Figure 5 plots language-model accuracy over INLP predictions for masked tokens alongside main-task performance.
- A Appendix: The figure compares Vanilla word-prediction scores, Amnesic Probing, main-task performance, and vanilla probing performance.
- A Appendix: The number of removed dimensions per iteration differs according to the number of classes in the property.
B Hewitt and Liang’s Control Task
The control-task analysis distinguishes lexical memorization from behavioral effects of removing linguistic properties and compares control performance with Amnesic Probing and random removal.
- B Hewitt and Liang’s Control Task: The control task is intended to distinguish probe extraction of encoded information from lexical memorization.Because the paper measures behavioral outcomes rather than extracting information, it treats this control differently from conventional probing.
- B Hewitt and Liang’s Control Task: The control-performance slope is smaller than Amnesic Probing’s, suggesting weaker behavioral influence for the removed directions.
- B Hewitt and Liang’s Control Task: Control slopes exceed the Rand experiment because label shuffling removes identities of word groups, making this test inadequate for the authors’ purpose.The paper therefore uses Rand and Selectivity as additional controls.