Source-linked AI summary
Understanding Neural Networks through Representation Erasure
Jiwei Li, Will Monroe, Dan Jurafsky
TL;DR
Neural networks achieve strong NLP performance but remain difficult to interpret. This paper analyzes decisions by erasing representations at multiple levels, using methods ranging from metric differences to reinforcement learning. Across NLP tasks, the framework provides interpretable explanations and supports neural-model error analysis.
Problem
Neural network models lack interpretability, making it difficult to determine when and why they make mistakes.
Method
The paper interprets neural decisions by erasing input dimensions, words or phrases, and hidden units, including reinforcement learning to find minimal decision-changing word sets.
Results
Across linguistic feature classification and sentiment-analysis tasks, representation erasure provides interpretable explanations of neural decisions and supports error analysis.
Takeaways & Limitations
Erasure reveals representations that contribute to decisions and representations whose removal improves decisions, supplying a general error-analysis tool across neural architectures and NLP applications.
Takeaways & Limitations
The sentiment-analysis analysis leaves discourse-dependent expressions as a long-term challenge, and the reinforcement-learning policy is trained with the original sentiment model fixed.
Abstract
from arXiv · showhide
While neural networks have been successfully applied to many natural language processing tasks, they come at the cost of interpretability. In this paper, we propose a general methodology to analyze and interpret decisions from a neural model by observing the effects on the model of erasing various parts of the representation, such as input word-vector dimensions, intermediate hidden units, or input words. We present several approaches to analyzing the effects of such erasure, from computing the relative difference in evaluation metrics, to using reinforcement learning to erase the minimum set of input words in order to flip a neural model's decision. In a comprehensive analysis of multiple NLP tasks, including linguistic feature classification, sentence-level sentiment analysis, and document level sentiment aspect prediction, we show that the proposed methodology not only offers clear explanations about neural model decisions, but also provides a way to conduct error analysis on neural models.
1 Introduction
The paper introduces representation erasure as a general way to interpret neural network decisions and conduct error analysis. Erasing representations can reveal both important features and representations that models focus on inappropriately.
- Neural networks are difficult to interpret because vector representations and nonlinear layers obscure how inputs become decisions and errors.This opacity affects word-vector dimensions, hidden units, sentence composition, and output decisions.
- Representation erasure analyzes how removing parts of a representation changes a neural model’s decisions.
- Erasure-related harm identifies representations that significantly contribute to a decision, while beneficial erasure identifies representations the model focuses on inappropriately.The latter cases support error analysis by exposing model behavior that improves when a representation is removed.
- The framework operates across input word-vector dimensions, input words or phrases, and intermediate hidden units, using analyses ranging from log-likelihood differences to reinforcement learning.The reinforcement-learning approach seeks the minimal set of words whose removal changes the model’s decision.
- The methodology offers interpretable explanations across linguistic feature classification and sentiment analysis while providing a general tool for neural-model error analysis.The paper frames this capability as useful for understanding mistakes and correcting them.
2 Related Work
Prior work exposes selected aspects of neural-network behavior through visualization, feature analysis, activation studies, and attention. These approaches are useful but constrained in scope, assumptions, or task applicability.
- Earlier research visualized embeddings and state activations, interpreted semantic dimensions, linked dimensions to lexical properties, and learned sparse interpretable representations.
- Existing methods reveal particular aspects of neural decisions but remain constrained by their applicability.
- Visualization of recurrent models exposed LSTM hidden-cell dynamics but relied on a few manually inspected cases.
- First-order derivative saliency methods examine input-feature importance under the assumption that decision scores are linear combinations of input features.
- Related work also studies unit activations, interpretable text subsets, vector dimensions, and interactive LSTM analysis.
- Attention explains neural models for tasks with input-output alignment, whereas representation erasure can also apply to attention-based models and tasks not well modeled by attention.
3 Linking Word Vector Dimensions to Linguistic Features
The paper studies how neural models encode linguistic features across word-vector dimensions and network layers by erasing representations and measuring decision changes. Results show concentrated importance in selected input dimensions, greater robustness in higher layers, and frequency-related dimensions in GloVe.
- Tasks and Training: The study examines POS, NER, chunking, prefix, suffix, sentiment, word-shape, and word-frequency classification using erasure of input vectors and hidden units.The models use pretrained word2vec and GloVe vectors and four-layer neural architectures.
- Visualization Model: I(d) measures the relative change in correct-label log-likelihood when vector dimension d is erased by setting it to zero.The score is computed over training examples with gold-standard labels.
- Results: Word2vec models concentrate importance in selected dimensions, with POS and chunking sharing dimension 34 and NER, prefix, and suffix sharing dimensions 4 and 31.The heatmap rows represent tasks and columns represent word-vector dimensions.
- Results: GloVe models rely heavily on dimension d31, while removing d31 and retraining shifts dominance to d26; removing both preserves performance and spreads attention across other dimensions.The authors associate these dimensions with word frequency and report sufficient information from redundant dimensions after their removal.
- Results: Higher network layers distribute importance more evenly across hidden units, making final classification decisions more robust to erasing any single dimension.The POS-layer heatmap projects importance values to log space and places layers in rows and units in columns.
4 Finding Important Words in Sentiment Analysis
The paper applies representation erasure to word-level sentiment analysis, measuring how deleting words changes model predictions and using those changes to interpret model behavior. Across RNN, Uni-LSTM, and Bi-LSTM models, erasure highlights sentiment-focused behavior and exposes context-related errors.
- Word-level analysis: The study analyzes word importance on Stanford Sentiment Treebank phrase- and sentence-level classification.Word importance is computed by measuring the relative change in the correct sentiment label’s log-likelihood after erasing a word.
- Models and performance: Bi-LSTM, Uni-LSTM, and standard RNN models achieve accuracies of 0.526, 0.501, and 0.453, respectively.
- Model sensitivity: Bi-LSTM is more sensitive to sentiment-indicative words than Uni-LSTM, which is more sensitive than the standard RNN.The authors associate this ordering with LSTM gates controlling information flow and focusing attention on sentiment-bearing words.
- Model sensitivity: All three models emphasize sentiment words, while LSTM-based models more clearly focus on sentiment and identify intensification tokens that the RNN misses.Examples include “loved,” “entertainment,” “greatest,” and an exclamation mark.
- Error analysis: Negative importance scores show that erasing some words improves predictions, revealing context patterns that contribute to sentiment-analysis errors.Examples include neutral uses, reversed sentiment, irrealis or ironic contexts, and concessive constructions.
- Error analysis: Handling discourse information remains a long-term goal because sentiment-indicator words can change meaning across contexts.Examples include “revelatory” in “a revelatory performance” and “pleasing” in “an intermittently pleasing but mostly routine effort.”
5 Reinforcement Learning for Finding Decision-Changing Phrases
The paper uses representation erasure to find minimal word sets that change neural sentiment decisions, combining discrete erasure with reinforcement learning. The resulting rationales identify aspect-specific sentiment phrases, while comparisons show memory-based models yield more interpretable explanations than LSTMs.
- Problem formulation: Representation erasure seeks the minimum subset of input words whose removal changes the model’s predicted label.The method treats words as discrete units rather than continuously modifying input dimensions.
- Reinforcement-learning method: Because exhaustive enumeration of word combinations is computationally intractable for long inputs, the paper uses reinforcement learning to approximate the optimal erasure.The policy makes binary keep-or-remove decisions for each word and is trained with a reward favoring label changes using fewer removals.
- Reinforcement-learning method: The policy model receives representations from a fixed pre-trained sentiment classifier and outputs a binary removal decision for each word.The original sentiment model remains fixed during reinforcement-learning training.
- Reinforcement-learning method: The reward favors changing the classification label with fewer erased words, and a regularizer encourages similar removal decisions within sentences.This regularization supports leaving out contiguous phrases rather than isolated words.
- Results: On hotel-review aspect prediction, the reinforcement-learning model identifies aspect-specific sentiment phrases that rationalize the classifier’s decisions.The experiments use TripAdvisor reviews and compare vanilla Bi-LSTM and memory-network sentiment models.
- Results: The memory-based model produces more interpretable rationales than the LSTM because its policy erases fewer words and less often removes passages about other aspects or overall sentiment.These conclusions are based on sample results in Table 4.
6 Conclusion
The paper concludes that representation erasure provides a general way to interpret neural network decisions and conduct error analysis. Examining both harmful and beneficial effects of erasure reveals important representations and representations that contribute to mistaken decisions.
- Conclusion: Representation erasure offers interpretable explanations for neural network decisions across models and tasks.The framework analyzes how removing particular representations affects model decisions.
- Conclusion: Negative effects of erasure identify representations that significantly contribute to a model’s decision, while positive effects expose representations associated with erroneous decisions.The latter cases are those where removing a representation improves the model’s decision.
- Conclusion: The framework provides a way to conduct error analysis on neural model decisions.The paper describes this as potentially useful across a wide variety of models and tasks.
7 Appendix
The appendix documents datasets, evaluation measures, model results, and diagnostic visualizations for feature-classification and sentiment experiments. It also catalogs words and model components by erasure-based importance.
- Dataset statistics and evaluation: The feature-classification datasets cover POS, NER, chunking, prefix, suffix, sentiment, word shape, and word frequency, using task-specific corpora and preprocessing.
- Dataset statistics and evaluation: The appendix reports accuracy for classification tasks and mean squared loss for word-frequency regression, with higher accuracy and lower loss preferred.
- Sentiment and aspect-rating experiments: The Stanford Sentiment Treebank provides 215,154 labeled phrases across 11,855 sentences for fine-grained and coarse-grained phrase- and sentence-level classification.
- Sentiment and aspect-rating experiments: Table 6 presents five-class aspect-rating results for two models and other baselines, while Table 7 reports testing accuracy for tagging tasks.
- Erasure-based diagnostics: Figure 6 visualizes log-transformed layer importance by task and dimension, while appendix tables rank words by positive or negative erasure importance.