Source-linked AI summary
All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality
William Timkey, Marten van Schijndel
TL;DR
The paper asks whether standard similarity measures reliably characterize contextual language-model representations, given counterintuitive results in later layers. It examines the dimensions driving similarity and model behavior, finding that a small rogue subset dominates the measures but not the model’s behavior. Standardization reduces this distortion and supports more faithful similarity analysis, while the origins of rogue dimensions remain an open direction.
Problem
Similarity measures applied to contextual language models produce counterintuitive results, including position sensitivity and weak alignment with human judgments in later layers.
Method
The paper analyzes which embedding dimensions dominate similarity measures and compares their influence with model behavior measured through ablation-induced changes in prediction distributions.
Results
A single dimension contributes more than 99% of expected cosine similarity between randomly sampled tokens in layers 10 and 11 of XLNet, while rogue dimensions do not likewise dominate model behavior.
Takeaways & Limitations
Similarity analysis of transformer language models should account for rogue dimensions, using transformations such as standardization to reveal underlying representational quality.
Takeaways & Limitations
The paper does not explain causally why degenerate embedding spaces arise or target post-processing for task-performance gains.
Abstract
from arXiv · showhide
Similarity measures are a vital tool for understanding how language models represent and process language. Standard representational similarity measures such as cosine similarity and Euclidean distance have been successfully used in static word embedding models to understand how words cluster in semantic space. Recently, these measures have been applied to embeddings from contextualized models such as BERT and GPT-2. In this work, we call into question the informativity of such measures for contextualized language models. We find that a small number of rogue dimensions, often just 1-3, dominate these measures. Moreover, we find a striking mismatch between the dimensions that dominate similarity measures and those which are important to the behavior of the model. We show that simple postprocessing techniques such as standardization are able to correct for rogue dimensions and reveal underlying representational quality. We argue that accounting for rogue dimensions is essential for any similarity-based analysis of contextual language models.
1 Introduction
Similarity measures that worked well for static embeddings produce misleading and counterintuitive analyses in contextualized language models. The paper attributes these problems to a few rogue dimensions that dominate similarity computations without reflecting model behavior.
- Contextual similarity measures can be highly sensitive to word position and yield inconsistent benchmark results.
- Later layers of BERT, RoBERTa, GPT-2, and XLNet show little to no correlation with human semantic similarity or relatedness judgments.
- Rogue dimensions dominate cosine similarity and Euclidean distance across tested contextual language models, often numbering only 1-5 dimensions.
- These dimensions are centered far from the origin and have disproportionately high variance, causing similarity measures to rely on less than 1% of the embedding space.
- Rogue dimensions dominate similarity measures without likewise dominating model behavior, while correlating strongly with absolute position and punctuation.
- Standardization accounts for rogue dimensions, making cosine similarity more closely reflect human judgments and preserving representational quality across layers.
2 Background
Standard similarity measures have been widely applied to contextual representations, but prior work identifies important sensitivities and outlier-related concerns. This paper focuses on characterizing those outliers rather than explaining their causes or optimizing task performance.
- Cosine similarity and Euclidean distance have been used to study semantic similarity, brain activation patterns, syntax, semantic shift, and compositionality.
- Prior work notes that cosine similarity is highly sensitive to outlier dimensions and suggests rank correlations such as Spearman’s ρ as a more robust alternative.
- The study does not aim to provide a causal explanation of degenerate embedding spaces or improve task performance through post-processing.
3.1 Anisotropy
The study decomposes anisotropy in contextual embedding spaces by dimension and finds that cosine similarity is concentrated in a small number of rogue dimensions rather than distributed globally. Across transformer models, these dimensions are especially dominant in anisotropic later layers, unlike static embeddings.
- Setup: The analysis estimates anisotropy as the expected cosine similarity of randomly sampled token pairs and decomposes that similarity by embedding dimension.The experiment samples 500k token pairs from an 85k-token English Wikipedia corpus and normalizes each dimension’s contribution by total expected cosine similarity.
- Results and Discussion: Across transformer models, a small subset of rogue dimensions dominates cosine similarity, especially in the more anisotropic final layers.In layers 10 and 11 of XLNet, a single dimension contributes more than 99% of expected cosine similarity between randomly sampled tokens.
- Results and Discussion: Static Word2Vec and GloVe embeddings are relatively isotropic and are not dominated by any single dimension.This contrast supports the finding that transformer anisotropy reflects sensitivity to outlier dimensions rather than a global property across all dimensions.
- Results and Discussion: The dimensions driving anisotropy are centered far from the origin and can concentrate embeddings along a single axis or low-dimensional subspace.In the final XLNet layer, dimension 667 has mean activation 180.0, compared with 0.77 for the other dimensions.
3.2 Informativity of Similarity Measures
The paper tests whether cosine-similarity variability reflects the full contextual embedding space and finds that later-layer variability is often controlled by only 1–5 dimensions.
- Informativity of Similarity Measures: Removing the top 1–5 dimensions can completely change which token pairs appear similar or dissimilar.The analysis ranks dimensions by their contribution to cosine similarity and compares similarities before and after removal.
- Scope: The study also examines Euclidean distance and reports similar conclusions, while relating the analysis to prior anisotropy findings.The authors state that their conclusions extend beyond cosine similarity to Euclidean distance.
- Results: In XLNet-12 and BERT-11, removing one dimension leaves only r2 = 0.028 and 0.046 of cosine-similarity variance explained, respectively.These values indicate that the remaining d −1 dimensions explain almost none of the original variability.
- Results: Cosine similarity in later contextual layers therefore behaves like a measure over 1–5 rogue dimensions rather than the entire embedding space.The same dimensions drive both high average similarity and variability in the measure.
4 Rogue Dimensions and Model Behavior
The authors compare dimensions that dominate cosine similarity with dimensions that influence language-model predictions, finding a strong mismatch between geometric prominence and behavioral importance.
- Setup: The study measures each dimension’s behavioral influence by clamping it to zero and computing the resulting change in the model’s prediction distribution.Influence is quantified as the mean KL divergence between the original and ablated distributions across inputs.
- Results: Across models, dimensions dominating cosine similarity do not likewise dominate model behavior.The comparison uses normalized cosine-similarity contributions and behavioral influence across the final layers.
- Results: In final XLNet and GPT-2, removing cosine-dominant dimensions produces only vanishingly small changes to the language-modeling distribution.The mismatch is less severe in BERT’s final layer but particularly pronounced in XLNet and GPT-2.
- Behavioral Correlates: Rogue dimensions often show activation patterns tied to token type or position, especially the period token and input position zero.In layers 2–11 of GPT-2 and RoBERTa, mean similarity for position-zero tokens exceeds .99, versus .623 and .564 for other tokens.
5 Postprocessing and Representational Quality
The authors evaluate simple postprocessing methods for reducing rogue-dimension effects and find that corrected embeddings align better with human judgments while preserving representational quality across layers.
- Postprocessing: Standardization subtracts each dimension’s mean and divides by its standard deviation, producing z-score representations with µi = 0 and σi = 1.The method targets dimensions with means far from zero and disproportionately high variance.
- Results: Postprocessing improves alignment with human similarity judgments across almost all layers and models, with standardization performing best overall.Standardization consistently improves over original embeddings except in BERT’s early layers.
- Results: All-but-the-top is generally effective, but final-layer RoBERTa and GPT-2 remain poorly correlated with human judgments.Its performance depends strongly on the number of removed components, and no choice substantially improves final-layer GPT-2.
- Results: Mean subtraction yields substantial gains in most models but fails in final GPT-2 and XLNet layers, where rogue dimensions retain exceptionally high variance.Making the space more isotropic does not by itself reduce dimension-wise variance.
- Results: Rank-based Spearman correlation also improves correlations with human judgments across all model layers, though it is often weaker than standardization or all-but-the-top.The method transforms embedding values into ranks before computing similarity.
- Interpretation: The findings suggest that lexical-semantic quality is preserved in final layers but obscured by a small number of rogue dimensions.After postprocessing, later layers correlate as well as or better than intermediate layers with human judgments.
6 Discussion and Future Work
The discussion argues that rogue dimensions should be prevented or better understood, while noting that their origins and generality across models remain open questions.
- Future work: Future work should design language models that do not develop rogue dimensions during pretraining.The authors identify understanding how these dimensions arise over time as a prerequisite for this goal.
- Open questions: Prior accounts emphasize token frequency, outliers and scaling factors, or positional embeddings, while this work observes correlations with specific tokens and positions.The authors call for these accounts to be unified.
- Open questions: Future studies could test whether rogue dimensions are incidental or required for good performance using MultiBERT checkpoints.Such checkpoints may also clarify how the dimensions emerge during pretraining.
- Future work: The authors favor building models requiring no post-hoc transformations, although standardization provides a simple correction for rogue dimensions.They associate this direction with more interpretable embedding spaces and potentially better performance.
7 Conclusion
The conclusion finds that contextual-model similarity measures can reflect a few rogue dimensions rather than the broader embedding space. It therefore argues that similarity analyses should account for these dimensions, particularly through techniques such as standardization.
- Main conclusion: Similarity measures in contextual language models are largely reflective of a small number of rogue dimensions, not the entire embedding space.Removing a single dimension often completely changed which token pairs were similar.
- Main conclusion: Previously observed anisotropy is essentially an artifact of rogue dimensions rather than a global property of the embedding space.Similarity variability is driven by just 1-5 dimensions.
- Interpretation: Rogue dimensions are not the main drivers of model behavior and appear to handle a small subset of abilities, including punctuation and positional information.This creates a mismatch between similarity dominance and behavioral importance.
- Implication: Similarity measures such as cosine similarity and Euclidean distance should account for rogue dimensions using techniques such as standardization.The authors frame this as a prerequisite for analyses involving representational similarity.
A Removing Dominant Dimensions and Representational Geometry
Removing the top five dimensions makes contextual embedding geometry more isotropic, consistent, and self-similar across models and layers, indicating that rogue dimensions obscure shared representational structure.
- Anisotropy: All tested models had highly anisotropic representations, including XLNet and RoBERTa.XLNet was more anisotropic than GPT-2 in its final two layers, while RoBERTa became anisotropic earlier than XLNet and BERT.
- Anisotropy: After removing 5 dimensions, anisotropy became relatively low, with Â(f_l) never exceeding 0.25 in any model layer.The reduced anisotropy also became more consistent across models and layers.
- Shared geometry: The deviant dimensions driving anisotropy are model- and layer-specific, whereas geometry without rogue dimensions shows similar properties across models and layers.This suggests rogue dimensions obscure common representational geometry.
- Self-similarity: After removing 5 dimensions, words of the same type show consistently high self-similarity across all layers and models.This contrasts with earlier extreme cases where same-type words were no more similar than randomly sampled words.
B Informativity of Euclidean Distance
The appendix evaluates whether dominant dimensions explain Euclidean-distance variability and examines how removing them changes geometric and similarity-based diagnostics across transformer models.
- Euclidean distance: Euclidean-distance variability was analyzed by removing dimensions selected according to their per-dimension variance.The procedure otherwise matched the earlier analysis, and results were reported in Table 3.
- Euclidean distance: In XLNet’s extreme case, removing one dimension explained none of the variability in Euclidean distances.Euclidean distance in that layer was effectively a measure of a single dimension.
- Representational geometry: Removing the top 5 dimensions drastically decreases anisotropy and makes it more consistent across models and layers.Figure 4 compares the full space with the space after removing those dimensions using E[CC_i].
- Representational geometry: Removing the top 5 dimensions makes intra-sentence similarity more consistent and monotonically increasing across layers.Figure 5 reports this comparison using E[CC_i] for the full and reduced spaces.
- Self-similarity: In GPT-2 and XLNet, same-type words are no more similar than random tokens in the full space but become more similar than the random baseline after removing 5 dimensions.Figure 6 compares average self-similarity by layer before and after removal.
- Dimension analysis: Figures 7-9 examine dimension contributions to cosine similarity, behavioral influence, and activation distributions across token types and positions.The activation analyses focus on the highest-variance dimension across early and later layers.
- Quantitative analyses: Tables 3-5 quantify variance explained in Euclidean and cosine similarity and the contribution of the top dimensions to expected cosine similarity.Table 4 also reports anisotropy estimates for each model layer.