Source-linked AI summary
Visualizing and Understanding Neural Models in NLP
Jiwei Li, Xinlei Chen, Eduard Hovy, Dan Jurafsky
TL;DR
Neural NLP models are difficult to interpret, particularly in how they compose sentence meaning from words and phrases. The paper introduces visualization and salience methods for examining this process, finding insights into negation, information focus, and LSTM performance. These methods provide an initial step toward understanding meaning composition in neural language models.
Problem
Neural models perform strongly on NLP tasks but provide limited interpretability into how they compose word and phrase meanings into sentence meaning.
Method
The paper combines representation plotting with first-derivative salience analysis, variance-based token comparison, and LSTM-style information-flow visualization.
Results
The visualizations reveal negation asymmetries, local dimensions associated with negation and quantification, and sharper focus on important words in LSTMs than other models.
Takeaways & Limitations
The methods offer general-purpose tools for interpreting meaning composition and other semantic properties of deep neural networks in NLP.
Takeaways & Limitations
The paper describes its analyses as superficial initial insights, with each visualization method having its own pros and cons.
Abstract
from arXiv · showhide
While neural networks have been successfully applied to many NLP tasks the resulting vector-based models are very difficult to interpret. For example it's not clear how they achieve {\em compositionality}, building sentence meaning from the meanings of words and phrases. In this paper we describe four strategies for visualizing compositionality in neural models for NLP, inspired by similar work in computer vision. We first plot unit values to visualize compositionality of negation, intensification, and concessive clauses, allow us to see well-known markedness asymmetries in negation. We then introduce three simple and straightforward methods for visualizing a unit's {\em salience}, the amount it contributes to the final composed meaning: (1) gradient back-propagation, (2) the variance of a token from the average word node, (3) LSTM-style gates that measure information flow. We test our methods on sentiment using simple recurrent nets and LSTMs. Our general-purpose methods may have wide applications for understanding compositionality and other semantic properties of deep networks , and also shed light on why LSTMs outperform simple recurrent nets,
1 Introduction
Neural NLP models achieve strong task performance but remain difficult to interpret, especially in how they compose word and phrase meanings. The paper introduces visualization and salience methods to inspect this composition and reports insights into negation, clause composition, and LSTM behavior.
- Neural models operate on continuous word embeddings and hidden units, making their behavior less interpretable than feature-based classifiers.Traditional systems expose human-interpretable features such as parts of speech, named entities, word shapes, and syntactic parses.
- The paper investigates how neural models compose meanings across words and sentence parts, including negation and intensification.The motivation includes understanding how models filter informational content while building sentence meaning.
- The authors combine representation plotting with simple measures of a neural unit’s salience, or contribution to composed meaning.The salience strategies include first-derivative-based analysis.
- The visualization results indicate that LSTMs focus more sharply on important words than other models and capture compositional asymmetries such as negation.The paper also reports local dimensions associated with negation and quantification and competitive composition across multiple clauses.
- The paper presents its visualization strategies and analytical results across separate sections after reviewing related visualization work and describing datasets and models.
2 A Brief Review of Neural Visualization
Prior neural visualization work largely comes from embedding projections and computer vision methods for reconstructing, back-propagating, or generating inputs. The paper adapts these ideas to NLP while accounting for words, vectors, and structured sequences as the basic units.
- Embedding similarity is commonly visualized by projecting word representations into two dimensions and examining clusters of similar words.
- Other NLP work has analyzed recurrent models statistically, while computer vision has developed several neural visualization strategies.The reviewed NLP and vision work provides inspiration for the paper’s approach.
- Inversion methods train an additional model to reconstruct input images from representations at different neural levels.Reconstructable pixels are treated as content aligned with corresponding image regions.
- Back-propagation and deconvolutional methods project errors or representations from upper layers toward the original image inputs.These methods help identify active regions associated with intermediate representations.
- Generation methods mutate randomly initialized images so that selected neural layers become activated during class-specific image construction.
- NLP requires adapting vision techniques because words and vectors, rather than pixels, are the basic units and sentences have sequential structure.The paper is positioned alongside independent recurrent-model analysis focused on predictions and errors.
3 Datasets and Neural Models
The paper evaluates sequence models on sentiment and sequence-to-sequence reconstruction using datasets ranging from sentiment benchmarks to a large machine-translation corpus. Its models include recurrent, LSTM, bidirectional LSTM, and sequence-to-sequence architectures.
- The experiments use two datasets, one relatively small and one large, for training neural models.
- Stanford Sentiment Treebank: The Stanford Sentiment Treebank contains 215,154 labeled phrases in 11,855 sentences and supports fine-grained and coarse-grained classification.Labels are provided for parse-tree constituents from sentences to individual words.
- Stanford Sentiment Treebank: The sentiment experiments convert parse-tree nodes into token sequences and use standard recurrent, LSTM, and bidirectional LSTM models.Representations are fed to a softmax classifier for phrase and sentence classification.
- Sequence-to-sequence models: Sequence-to-sequence models encode an input sequence with an LSTM and sequentially predict output tokens using a softmax distribution.At each step, prediction combines the current token with previously built embeddings.
- Sequence-to-sequence models: For visualization, the paper uses an autoencoder whose inputs and outputs are identical to examine token effects on sentence representations and predictions.The autoencoder is trained on 4 million English WMT’14 sentences averaging 22.5 words.
4 Representation Plotting
The paper visualizes how neural representations compose modifiers, negations, and clauses, then uses saliency heatmaps to inspect information contributions across sentiment examples.
- Local Composition: Unit-value heatmaps show intensification strengthening a few dimensions, while negation consistently reverses selected dimensions.Modifiers such as “a lot,” “so much,” and “incredibly” strengthen values; negation produces blue-to-red shifts in some dimensions.
- Representation Plotting: t-SNE representations cluster negated positive words with negative words, while “not bad” lies closer to negative than positive words.This pattern reflects the markedness asymmetry of negation and suggests the model does not apply a fixed transform for “not.”
- Concessive Sentences: For sentiment analysis, excessive dimensionality can leave many dimensions non-functional, so opposite-sentiment sentences may differ in only a few dimensions.The passage notes that optimal word dimensionality was reported between 25 and 35 in prior work.
- Concessive Sentences: Concessive representations become closer when final words such as “interesting” and “boring” appear, favoring stronger-clause dominance over collaborative composition.The observed dilution of differences supports the interpretation that the stronger sentiment dominates in these examples.
- Clause Composition: Clause composition preserves negative asymmetry: adding a negative clause makes positive sentences more negative, whereas concessive additions do not make negative sentences positive.Examples include “I like the movie” becoming more negative after negative clauses, while “I hate X but ...” remains very negative.
5 First-Derivative Saliency
The paper uses first-derivative saliency to estimate how input dimensions contribute to neural decisions, then visualizes information flow in sentiment and sequence-to-sequence models. These visualizations highlight sharper LSTM focus, token-to-output alignments, and a limitation of first-order derivatives.
- Gradient back-propagation: First-derivative saliency approximates each input unit’s contribution to a classification score by locally linearizing the model around its embedding.The method uses the derivative of the class score with respect to the embedding; its absolute value measures sensitivity in each dimension.
- Stanford Sentiment Treebank: Saliency visualizations assign high importance to sentiment-bearing words such as “hate” while dampening less relevant tokens, especially in LSTMs and bidirectional LSTMs.The bidirectional LSTM shows the clearest focus, with almost zero emphasis on words other than “hate”; the authors associate this with gated information flow.
- Stanford Sentiment Treebank: For longer sentiment sentences, all models retain the salience of “hate” after 7-8 following convolutional operations, although simple recurrent models filter irrelevant words less effectively.The models still assign the correct sentiment, but recurrent models give too much salience to words unrelated to sentiment.
- Stanford Sentiment Treebank: In concessive sentences, simple recurrent saliency can emphasize the later positive clause even when the model correctly uses the earlier negative clause for classification.This mismatch indicates that low first-clause derivatives do not necessarily represent the clause’s full contribution to the decision.
- Sequence-to-sequence autoencoder: Sequence-to-sequence heatmaps show that input regions associated with a token exert more influence when the corresponding token is predicted, revealing automatically learned alignments.Examples include “hate,” “movie,” “plot,” and “boring.”
- Sequence-to-sequence autoencoder: During decoding, source-input influence gradually diminishes while previously predicted tokens increasingly determine the next output.When predicting “boring,” the model weights earlier generated tokens such as “plot” and “is” more heavily than the corresponding source region.
6 Average and Variance
The paper proposes average-embedding variance as a direct salience measure for learned word embeddings, highlighting words that differ from the sentence-average representation. This method emphasizes sentiment indicators but is limited to learned embeddings and offers unclear local-composition visualization.
- The method computes each word’s salience as its deviation from the average embedding of all words in the sentence.The approach is intended for settings where embeddings are optimized from scratch rather than initialized from pretrained vectors.
- Figure 8 visualizes variance across embedding dimensions for each word in a sentence.Each grid records the deviation value for one word dimension relative to the sentence-average embedding.
- The variance-based measure emphasizes relevant sentiment words in the model’s representations.
- The method is restricted to settings where word embeddings are learned and its ability to visualize local compositionality remains unclear.
7 Conclusion
The paper presents visualization methods for interpreting how neural models compose meaning, including negation asymmetries and aspects of LSTM performance. The authors characterize these methods as an initial, partial step toward understanding neural language models.
- The paper offers several methods for visualizing and interpreting neural models’ composition of meaning.
- The visualizations demonstrate asymmetries of negation and explain some aspects of LSTMs’ strong performance on these tasks.
- The authors describe their analyses as touching only superficial points, with each method having advantages and disadvantages.
- Together, the methods offer initial insights into neural models’ behavior in language tasks and meaning composition.
Appendix
The appendix describes recurrent, multi-layer, bidirectional, and LSTM architectures used to compose sequence representations. These models transform word embeddings and recurrent states through layered or gated computations before classification.
- Recurrent Models: A recurrent network combines each word embedding with the preceding hidden vector to produce the current hidden representation.The resulting representation is passed to the next time step.
- Recurrent Models: For a sequence of length N_s, the final hidden representation h_Ns represents the whole sequence and feeds a softmax classifier.
- Multi-layer Recurrent Models: Multi-layer recurrent models assign a hidden representation h_l,t to every layer and time step, extending the one-layer structure for greater expressivity and flexibility.The original word embedding is the input at layer zero, h_t,0 = e_t.
- Long-short Term Memory: LSTMs use input, control, and memory gates to regulate information and reduce the impact of unrelated inputs.The gate states i_t, f_t, and o_t, together with the cell and hidden states, determine the recurrent output.
- Bidirectional Models: Multi-layer LSTMs extend the same layered composition principle, while bidirectional models compute representations from both forward and backward directions.