Source-linked AI summary
BERT Busters: Outlier Dimensions that Disrupt Transformers
Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, Anna Rumshisky
TL;DR
Prior work found Transformers robust to pruning, leaving open how a very small set of output dimensions affects pretrained models. This paper identifies recurring outlier dimensions and finds that disabling them degrades performance across BERT-family and other Transformer models.
Problem
Prior studies found Transformers robust to pruning, while the effect of modifying a very small number of output dimensions in pretrained models remained an open question.
Source: §1 · p. 1Method
The authors identify recurring outlier dimensions using LayerNorm scaling factors and biases that are jointly at least 3σ from their means in at least half the model layers, then disable them.
Source: §1 · p. 1, §3.1 · p. 3Results
Takeaways & Limitations
The findings indicate that final transformations play a larger role in Transformer layers than usually assumed and warrant further investigation.
Source: §1 · p. 1Whether outlier-distance thresholding can indicate completion of pretraining remains untested; the authors leave that hypothesis for future work.
Source: §5.2 · p. 8
Abstract
from arXiv · showhide
Multiple studies have shown that Transformers are remarkably robust to pruning. Contrary to this received wisdom, we demonstrate that pre-trained Transformer encoders are surprisingly fragile to the removal of a very small number of features in the layer outputs (<0.0001% of model weights). In case of BERT and other pre-trained encoder Transformers, the affected component is the scaling factors and biases in the LayerNorm. The outliers are high-magnitude normalization parameters that emerge early in pre-training and show up consistently in the same dimensional position throughout the model. We show that disabling them significantly degrades both the MLM loss and the downstream task performance. This effect is observed across several BERT-family models and other popular pre-trained Transformer architectures, including BART, XLNet and ELECTRA; we also show a similar effect in GPT-2.
1 Introduction
Although prior studies found Transformers robust to pruning, this work shows that disabling a very small set of recurring output dimensions can sharply degrade performance. The dimensions appear across pretrained encoder layers, with LayerNorm as the focus in most studied models.
1 Introduction
- Very few outlier dimensions recur in the same positions across the pretrained encoder layers of a given Transformer model.The paper contrasts this finding with prior studies reporting robustness to pruning, noting that modifying less than 0.0001% of BERT weights can disrupt performance.Source: §1 · p. 1
- For the encoder models studied, the affected final transformation is generally LayerNorm; GPT-2 instead shows the phenomenon in its decoder output dense transformation.The study focuses on LayerNorm scaling factors and biases in the encoder models.Source: §1 · p. 1
- Disabling outlier dimensions drastically degrades language-modeling and downstream performance, by up to 44 points, across BERT-family and other Transformer models.The reported scope includes six BERT-family models and ELECTRA, BART, and XLNet; GPT-2 shows a similar phenomenon in decoder output dense transformations.Source: §1 · p. 1
- Outlier weights emerge gradually and early in pretraining, producing abnormal spikes at selected output-embedding dimensions.The authors state that it is not clear why these features emerge and call for further investigation of the role of final transformations.Source: §1 · p. 1
- The authors present this as the first work to establish regular outliers in Transformer output representations and their importance for model performance.They argue that final transformations play a larger role in Transformer layers than usually assumed.Source: §1 · p. 1
2 Related work
This section situates Transformer pruning and normalization research, and establishes terminology for the final layer transformation. It contrasts common pruning approaches with prior findings on BERT subnetworks and summarizes work on normalization placement and alternatives.
2 Related work
- Transformer encoder layers combine multi-head self-attention and a dense layer, and their normalization placement distinguishes post-LN from GPT-2’s pre-LN configuration.In post-LN, outputs of self-attention and linear layers undergo LayerNorm; GPT-2 applies LayerNorm before those transformations.Source: §2 · p. 2
- The paper calls outputs of the encoder layer’s final transformation “features” and that transformation’s parameters “weights”; the final transformation is LayerNorm except for GPT-2’s MLP.This terminology frames the subsequent analysis of output dimensions and their parameters.Source: §2 · p. 2
- Prior normalization studies examined training processes, including LayerNorm location effects on gradient flow and warmup, inserted normalization blocks, and alternatives with improved gradient propagation.The passage describes these as a small body of Transformer normalization research.Source: §2 · p. 2
- Transformer compression became a research subfield after reports of BERT redundancy, with overviews by Ganesh et al. (2020) and Rogers et al. (2020).The passage lists multiple compression studies alongside those overviews.Source: §2 · p. 2
- Pruning commonly sets selected weights to zero for compression with minimal performance loss, often choosing weights by magnitude; this paper uses pruning for model analysis.The passage also notes that pruning can serve efficiency goals.Source: §2 · p. 2
- BERT’s largest weights can form subnetworks retrainable to near-full-model performance, while the worst structured subnets remain better than the worst magnitude-based subnets.The cited explanation is that structured subnets presumably retain many high-magnitude weights.Source: §2 · pp. 2–3
3 Outlier weights in BERT models
The section identifies recurring LayerNorm outlier dimensions, disables their scale and bias across layers, and examines their effects on model performance and embeddings. The outlier features show high magnitudes in BERT representations, which the authors associate with embedding-space distortion.
3.1 Identification
- Outliers are dimensions whose LayerNorm bias and scale are each at least 3σ from their means in at least half the model layers.For BERT-large, the frequency threshold was relaxed to one-third of layers; for RoBERTa, the deviation threshold was reduced to 2σ.Source: §3.1 · p. 3
3.2 Disabling
- Disabling output LayerNorm outliers has an unexpectedly large effect, unlike disabling input-embedding or linear-layer weights, which produced no significant change.The authors focus most experiments on output LayerNorm weights.Source: §3.2 · p. 3
3.3 Visualization
- Feature 308 has high-magnitude output values in most BERT layers, while feature 381 is visibly high in layers 7–10.Both LayerNorm scale and bias contribute to the outlier effect, to varying degrees across layers.Source: §3.3 · p. 4
- Up to 95% of WikiText input-token representations have abnormally high magnitudes at identified outlier dimensions, and removing the outliers makes the embedding space relatively uniform.The authors report that the distortion is not attributable to the input embedding layer.Source: §3.3 · p. 4
3.2 Disabling
- RoBERTa’s two outlier dimensions produce incorrect predictions, while disabling two random dimensions leaves its output almost unchanged.The table caption describes this example; the paper reports that fewer than 0.001% of weights are affected.Source: Table 1 · p. 4
4 Effects of Disabling Outlier Weights
Disabling a few outlier dimensions sharply degrades language modeling and downstream performance, while comparable changes to non-outlier dimensions have little effect. The effect also appears in other Transformer models, including GPT-2, with model-specific modifications.
4.1 Masked Language Modeling
- Removing 24 LayerNorm parameters raises RoBERTa’s loss by almost 4×, while fewer than 0.001% of model weights are affected.Figure 4 compares the loss after zeroing a dimension’s scaling factor and bias with the full-model loss.Source: §4.1 · pp. 4–5, Figure 4 · p. 5
4.2 BERT Downstream Tasks
- Disabling both outliers drops STS-B by 44.1 points and CoLA by 32.3; MNLI and SST-2 also lose over 20 points.The effects of individual outliers vary by task: dimension 308 affects STS-B more, while 381 affects CoLA more.Source: §4.2 · p. 6
- Across tasks, disabling random non-outlier dimensions has negligible effect; pairing one non-outlier with one outlier has the effect of disabling one outlier alone.Source: §4.2 · p. 6
- Before fine-tuning, disabling two outliers lowers average performance by 1.1, versus 0.31 when disabling 10 non-outliers.The comparison asks whether fine-tuning can recover information after the parameters are disabled.Source: §4.2 · pp. 5–6
4.3 Outliers in Other Transformers
- BART, ELECTRA, XLNet, and GPT-2 also show especially disruptive dimensions; disabling six in GPT-2 increases perplexity by over 300 times.For GPT-2, the experiment disables dense output-layer weights; the other models’ encoder LayerNorm dimensions are disabled. The models are evaluated on their native language-modeling tasks, except BART, which is evaluated on CNN/Daily Mail summarization.Source: §4.3 · p. 7, Figure 6 · p. 7, Table 4 · p. 7
5 What Makes Outlier Weights Special
The paper tests whether outlier effects are explained by parameter magnitude or by where dimensions recur across layers, and tracks their emergence during pre-training. Its findings point to both magnitude and consistent location, while leaving the cause of their emergence unresolved.
5.1 Magnitude or Location?
- Magnitude-based pruning yields cross-entropy 1.3× smaller than disabling both BERT outliers.The comparison tests magnitude-based selection against the identified outliers; Table 5 reports BERT-base WikiText validation loss.Source: §5.1 · p. 8, Table 5 · p. 8
- Disabling outliers only in the input or final Transformer layer has little to no effect, while disabling them in more earlier layers progressively raises loss.Source: §5.1 · p. 8, Table 6 · p. 8
- The authors conclude that both magnitude and consistent location across layers contribute to the distinct embedding features BERT relies on.Source: §5.1 · p. 8
5.2 How Do Outlier Weights Emerge?
- After approximately 50k pre-training steps, scaling factors and biases begin diverging from initialization as training loss and evaluation perplexity begin to fall.The authors note that whether these changes are causally related remains a question for future work.Source: §5.2 · p. 8
- The scratch-trained BERT-medium had one dimension whose scaling factor and bias both exceeded their respective means by more than three standard deviations; the published model had two.The authors suggest that thresholding outlier distance might serve as a pre-training completion signal, but leave that hypothesis for future work.Source: §5.2 · p. 8
6 Implications and Future Work
The outlier dimensions may enable attacks that degrade Transformer quality while changing very few weights, and consistent pruning of them may be more damaging than random or magnitude-based pruning. The authors suggest checksum verification as a safeguard and identify initialization as a direction for future work.
6 Implications and Future Work
- Outlier dimensions may enable attacks that degrade Transformer quality while modifying very few weights, and checksum verification is suggested to check deployed model files.Because these perturbations do not break the model completely, the attack may be detected late.Source: §6 · pp. 8–9
- In BERT-medium, the 417th dimension’s scaling factor and bias leave the three-sigma range by the end of pretraining, and token embeddings show outlier values at that dimension.Source: Figure 7 · p. 9
- Consistently disabling outliers could cause a larger performance drop than random or magnitude-based pruning.Source: §6 · p. 9
- Manipulating initialization to encourage outlier dimensions may improve performance or convergence speed, but this remains a direction for future work.Source: §6 · p. 9
7 Conclusion
The paper identifies a small set of outlier dimensions whose high-magnitude LayerNorm scaling factors and biases disrupt model performance. The phenomenon emerges early, warps the embedding space, and appears across BERT-family and other Transformer models.
7 Conclusion
- The outlier dimensions disrupt performance while involving less than 0.0001% of model parameters, and the authors attribute the effect to high-magnitude scaling factors and biases at the same positions throughout the model.Source: §7 · p. 9
- The outlier phenomenon emerges early in training and consistently warps the embedding space.Source: §7 · p. 9
- Outliers occur in LayerNorm outputs in the six examined BERT-family models and in four other Transformer models, though disabling them has varying effects.Source: §7 · p. 9
9 Impact statement
The impact statement describes an analysis of BERT-family LayerNorm behavior using standard benchmarks, without presenting new models or collecting data. It also reports estimated energy use and associated emissions.
10 Appendix
The appendix lists candidate outlier weights for BERT-like models, gives detailed BERT-base LayerNorm statistics, and presents sample masked-token predictions after disabling selected weights.
10.1 Candidate outlier dimensions of the models.
- Candidate outlier weights are reported for the tested BERT-like models, with tables covering BERT-small, BERT-medium, and BERT-base.Source: §10.1 · p. 12, Table 7 · p. 12, Table 8 · p. 12
10.2 Scaling factor and bias statistics for BERT-base.
- For BERT-base, LayerNorm statistics include per-layer means, standard deviations, three-sigma counts, and the values and ranks of outlier weights 308 and 381.The outliers rank among the largest or smallest weights throughout the model, but are not necessarily the most extreme values.Source: §10.2 · p. 12, Table 13 · p. 13
10.3 Sample language model outputs after disabling outlier LayerNorm weights.
- Sample masked-token predictions compare full RoBERTa and BERT models with setups disabling two outlier dimensions or two random dimensions across layers.The prediction tables note that 48 weights are modified in each of the Random and Outliers setups.Source: §10.3 · p. 12, Table 14 · p. 14, Table 15 · p. 14