Source-linked AI summary
How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings
Kawin Ethayarajh
TL;DR
The paper asks whether contextualized models assign infinitely many context-specific representations or a finite set of word-sense representations. It studies the geometry of ELMo, BERT, and GPT-2 across layers and finds that less than 5% of contextualized variance is explained by a static embedding, while upper layers become more context-specific.
Problem
The paper addresses how contextualized word representations vary by context and whether they reduce to finite word-sense representations.
Method
The authors study layerwise representation geometry in ELMo, BERT, and GPT-2 using contextuality measures adjusted for anisotropy.
Results
Less than 5% of a word’s contextualized variance is explained by a static embedding across all layers of ELMo, BERT, and GPT-2.
Takeaways & Limitations
Static word embeddings would be a poor replacement for contextualized representations even in the best-case scenario.
Takeaways & Limitations
The paper leaves open whether differences in intra-sentence similarity arise from differences in model architecture.
Abstract
from arXiv · showhide
Replacing static word embeddings with contextualized word representations has yielded significant improvements on many NLP tasks. However, just how contextual are the contextualized representations produced by models such as ELMo and BERT? Are there infinitely many context-specific representations for each word, or are words essentially assigned one of a finite number of word-sense representations? For one, we find that the contextualized representations of all words are not isotropic in any layer of the contextualizing model. While representations of the same word in different contexts still have a greater cosine similarity than those of two different words, this self-similarity is much lower in upper layers. This suggests that upper layers of contextualizing models produce more context-specific representations, much like how upper layers of LSTMs produce more task-specific representations. In all layers of ELMo, BERT, and GPT-2, on average, less than 5% of the variance in a word's contextualized representations can be explained by a static embedding for that word, providing some justification for the success of contextualized representations.
1 Introduction
Contextualized representations improve NLP performance but remain poorly understood: the paper asks how many context-specific representations models assign to each word and studies this question geometrically.
- Contextualized representations are sensitive to the contexts in which words appear and have improved diverse NLP tasks.Examples include question-answering and coreference resolution.
- The paper asks whether models assign infinitely many context-specific representations or a finite set of word-sense representations to each word.
- The authors study representation-space geometry across layers of ELMo, BERT, and GPT-2.
- All model layers are anisotropic, with representations occupying a narrow cone rather than being uniformly distributed by direction.GPT-2’s last layer is especially extreme: two random words have almost perfect cosine similarity on average.
- Same-word representations become more dissimilar across contexts in upper layers, suggesting that upper layers produce more context-specific representations.
- Less than 5% of contextualized variance is explained by a word’s first principal component across all layers and models.This argues against finite word-sense representations and against static embeddings replacing contextualized ones.
2 Related Work
Related work contrasts single-vector static embeddings with sentence-dependent contextualized representations and distinguishes this paper’s geometric analysis from prior probing studies.
- SGNS and GloVe generate static embeddings, assigning one vector to each word.Their theoretical connection is to factorizing word-context co-occurrence matrices.
- Static embeddings force all senses of a polysemous word to share one representation.
- ELMo, BERT, and GPT-2 create contextualized representations that are functions of the entire input sentence.
- ELMo uses a bidirectional LSTM, whereas BERT and GPT-2 use bidirectional and unidirectional transformer language models, respectively.
- Prior probing studies predict syntactic and semantic properties but cannot determine how contextual representations are or whether static embeddings can replace them.
- This paper instead studies the geometry of contextualized representation spaces, following work on static embedding geometry.
3 Approach
The approach analyzes layerwise contextuality in pretrained ELMo, BERT, and GPT-2 using sentence data and three similarity- and variance-based measures adjusted for anisotropy.
- The study analyzes pretrained ELMo, BERT, and GPT-2, including each model’s input layer as layer 0.The models have 2, 12, and 12 hidden layers, respectively.
- SemEval Semantic Textual Similarity sentences provide repeated words occurring in different contexts for analysis.The datasets span 2012–2016.
- Contextuality is measured with self-similarity, intra-sentence similarity, and maximum explainable variance.
- Self-similarity averages cosine similarity between a word’s representations across contexts, decreasing as contextualization increases.
- Intra-sentence similarity averages cosine similarity between word representations and their sentence vector, the mean of the sentence’s word vectors.It distinguishes convergence among contextualized words from context-specific representations that remain distinct.
- Maximum explainable variance is the proportion of a word’s contextualized variance explained by its first principal component, bounding static replacement quality.Values closer to 0 indicate poorer static replacement; 1 indicates a perfect replacement.
- Isotropy must be controlled because high average similarity among random words can make raw self-similarity misleading.
- The analysis adjusts each contextuality measure by subtracting a layer-specific baseline estimated from uniformly sampled representations.For similarity measures the baseline is average random-word cosine similarity; for MEV it is random representations’ first-component variance.
4 Findings
Across ELMo, BERT, and GPT-2, contextualized representations become generally more anisotropic and context-specific in higher layers, but context-specificity manifests differently across models. Less than 5% of within-word variance is explained by a static embedding, while lower-layer principal-component embeddings can still outperform traditional static embeddings.
- 4.1 (An)Isotropy: Contextualized representations occupy increasingly narrow cones in higher layers, with GPT-2’s final layer showing almost perfect cosine similarity between random words.ELMo’s input layer is the exception because it produces static character-level embeddings without context; BERT’s penultimate layer also exceeds its final layer in anisotropy.
- 4.2 Context-Specificity: Higher layers have lower self-similarity, indicating that the same word receives more context-specific representations there.ELMo’s input-layer self-similarity is 1.0 because its representations are static character-level embeddings.
- 4.2 Context-Specificity: Context variety, rather than inherent polysemy, appears to drive representational variation, including for highly frequent stopwords.Words such as ‘and’, ‘of’, ‘the’, and ‘to’ have among the lowest self-similarity, arguing against a finite set of word-sense representations.
- 4.2 Context-Specificity: Context-specificity takes distinct geometric forms: ELMo increases within-sentence similarity, whereas BERT decreases it while remaining above the random-word baseline.ELMo makes same-sentence representations more similar in upper layers; BERT makes them more dissimilar, although surrounding context still yields greater similarity than random words.
- 4.2 Context-Specificity: GPT-2’s same-sentence representations are no more similar than random words, showing that high intra-sentence similarity is not inherent to contextualization.Its adjusted intra-sentence similarity is near zero in most layers and highest in the non-contextualizing input layer.
- 4.3 Static vs. Contextualized: Less than 5% of a word’s contextualized variance is explained by a static embedding, while lower-layer principal components can outperform GloVe and FastText on many benchmarks.The best-performing principal-component embeddings come from BERT’s first layer; GPT-2-derived embeddings perform markedly worse than those from ELMo and BERT.
5 Future Work
The paper proposes studying isotropy benefits for contextualized representations and extracting static vectors to ease production deployment of large models.
- Isotropy may benefit contextualized representations, as it benefits static embeddings, despite contextualized models already improving downstream performance.
- Static representations extracted from contextualized models may address BERT’s memory and runtime deployment challenges.
6 Conclusion
Upper layers of ELMo, BERT, and GPT-2 produce more context-specific and anisotropic representations, but this specificity differs across models. After anisotropy adjustment, static embeddings explain less than 5% of contextualized variance on average, making them poor replacements.
- Upper layers of ELMo, BERT, and GPT-2 produce more context-specific representations, accompanied by increased anisotropy.
- Anisotropy-adjusted similarity between words in the same sentence is highest in ELMo but almost non-existent in GPT-2.
- Less than 5% of contextualized variance is explained by a static embedding on average across all layers and models.
- Static word embeddings would therefore be poor replacements for contextualized representations even in the best-case scenario.