Source-linked AI summary
Revealing the Dark Secrets of BERT
Olga Kovaleva, Alexey Romanov, Anna Rogers, Anna Rumshisky
TL;DR
BERT performs strongly across NLP tasks, but the mechanisms behind its success remain unclear. The paper analyzes individual self-attention heads using handcrafted linguistic features and selected GLUE tasks, finding repeated attention patterns, overparametrization, and cases where disabling heads improves performance.
Problem
The exact mechanisms contributing to BERT’s outstanding performance remain unclear, particularly the linguistic information encoded in its self-attention weights.
Method
The paper qualitatively and quantitatively analyzes per-head self-attention maps from pretrained and fine-tuned BERT models using handcrafted linguistic features and a subset of GLUE tasks.
Results
Repeated attention patterns across heads indicate overparametrization, while disabling selected heads is not detrimental and can improve performance by up to 3.2%.
Takeaways & Limitations
The findings suggest model pruning and reduced sub-architectures as directions for interpreting and simplifying BERT.
Takeaways & Limitations
The authors identify cross-lingual analysis as future work, noting that English may exhibit relatively low attention-pattern variety because subject-predicate and token-order patterns can coincide.
Abstract
from arXiv · showhide
BERT-based architectures currently give state-of-the-art performance on many NLP tasks, but little is known about the exact mechanisms that contribute to its success. In the current work, we focus on the interpretation of self-attention, which is one of the fundamental underlying components of BERT. Using a subset of GLUE tasks and a set of handcrafted features-of-interest, we propose the methodology and carry out a qualitative and quantitative analysis of the information encoded by the individual BERT's heads. Our findings suggest that there is a limited set of attention patterns that are repeated across different heads, indicating the overall model overparametrization. While different heads consistently use the same attention patterns, they have varying impact on performance across different tasks. We show that manually disabling attention in certain heads leads to a performance improvement over the regular fine-tuned BERT models.
1 Introduction
Transformer-based models, including BERT, have become state-of-the-art across many NLP tasks, yet the mechanisms behind BERT’s performance remain unclear. This paper analyzes linguistic information in self-attention and reports evidence of overparametrization alongside performance gains from disabling selected heads.
- Transformer-based models have become the de-facto standard for state-of-the-art performance on many NLP tasks.
- BERT uses a bi-directional Transformer encoder pretrained on language modeling and is fine-tuned for specific tasks.
- The exact mechanisms contributing to BERT’s outstanding performance remain unclear, motivating experiments on selected linguistic features.
- The paper proposes a detailed analysis of linguistic information encoded in BERT’s self-attention weights.
- Absolute performance gains of up to 3.2% result from a simple approach that improves BERT by disabling selected attention heads.
2 Related work
Prior work examined linguistic knowledge in BERT representations and compared self-attention with recurrent and convolutional alternatives. This paper instead focuses directly on self-attention patterns and connects that analysis to neural-network pruning and overparametrization.
- Earlier studies found evidence that BERT captures structural and linguistic properties across masked-language-modeling, probing, and intermediate-layer analyses.
- One comparison reported that LSTMs generalize better to longer sequences and resist agreement distractors more robustly than Transformers.
- Related findings suggest Transformers can extract semantic features and that their middle layers transfer effectively, while higher layers may be less task-specific than RNN layers.
- Unlike layer-focused representation studies, this work examines the self-attention mechanism itself as a key Transformer component.
- The study relates its analysis to pruning research showing that overparameterized architectures can be reduced without performance loss.
3 Methodology
The methodology combines research questions about attention patterns, linguistic information, and head importance with fine-tuning on a subset of GLUE tasks. Researchers extract per-head attention maps and inspect their links to linguistic features and task behavior.
- The study asks how attention patterns change during fine-tuning, what linguistic knowledge they encode, and how head differences affect task performance.
- Fine-tuning uses seven GLUE tasks: MRPC, STS-B, SST-2, QQP, RTE, QNLI, and MNLI matched.
- CoLA and the Winograd Schema Challenge are excluded, respectively because of reported methodological problems and the latter dataset’s small size.
- For each input, the researchers extract self-attention weights from every head and layer, producing an L×L self-attention map for sequence length L.
- The maps are analyzed for attention to parts of speech, syntactic roles, semantic relations, and negation tokens.
4 Experiments
Experiments identify recurring self-attention patterns, probe their linguistic content and fine-tuning changes, and test whether disabling heads affects GLUE performance. The results suggest repeated patterns and task-dependent head importance, with selected head or layer disabling sometimes improving performance.
- 4.1 BERT’s self-attention patterns: Five recurring attention classes—Vertical, Diagonal, Vertical+Diagonal, Block, and Heterogeneous—appear across pre-trained and fine-tuned BERT heads.Heterogeneous maps are variable and make the class list exhaustive.
- 4.1 BERT’s self-attention patterns: The estimated upper bound of potentially informative Heterogeneous heads ranges from 32% on MRPC to 61% on QQP.This estimate captures heads that might encode structural information beyond adjacency and separator tokens, not confirmed informative heads.
- 4.2 Relation-specific heads in BERT: Two of 144 pre-trained heads tended to attend to FrameNet core elements, with average maximum attention weights of 0.201 and 0.209.These values exceeded the 99th percentile of the distribution across heads, providing limited evidence for relation-sensitive attention.
- 4.3 Change in self-attention patterns after fine-tuning: For all selected tasks except QQP, the last two layers changed most from pre-trained BERT during fine-tuning, while fine-tuned models gained 35.9 absolute points on average.The authors attribute task-specific features to later layers and more fundamental information to earlier layers.
- 4.4 Attention to linguistic features: Vertical attention primarily targeted [CLS] and [SEP], with earlier layers favoring [CLS] and later layers favoring [SEP] across tasks.Attention to nouns, direct objects, and negation tokens was detected in some tasks but was negligible relative to attention to special tokens.
- 4.4 Attention to linguistic features: Potential heads for noun-pronoun and verb-subject links coincided with diagonal maps, making these relations difficult to distinguish from adjacent-token attention.The authors link this difficulty to dependent elements frequently appearing close together in English syntax.
- 4.6 Disabling self-attention heads: Disabling selected heads increased performance across all tasks and datasets, while other heads harmed performance when disabled.A single-head gain ranged from 0.1% on STS-B to 1.2% on MRPC.
- 4.6 Disabling self-attention heads: Disabling the first layer on RTE produced a 3.2% absolute gain, whereas layer disabling reduced QNLI and MNLI performance by up to -0.2%.The effects of disabling an entire layer therefore varied across tasks.
5 Discussion
Different BERT heads can repeat the same self-attention patterns while differing in task importance, supporting overparameterization. Disabling selected heads can preserve or improve performance, but frame-semantic attention patterns were not shown to improve GLUE results.
- Repeated self-attention patterns across heads, together with non-detrimental head disabling, support significant overparameterization in the smaller base BERT model.The model sometimes improves when single or multiple heads are disabled.
- 2 out of 144 heads associated with core frame-semantic relations were not important in any GLUE task, since disabling either did not reduce accuracy.The authors found no evidence that these attention patterns improve BERT’s performance.
- STS-B and RTE relied on the same pair of heads, which assigned high weights to words appearing in both input sentences.This pattern suggests word-by-word sentence comparison, although comparable interpretations were not found for heads important to other tasks.
6 Conclusion
The authors analyze BERT self-attention and find repeated, redundant patterns across heads, while disabling attention can improve performance. They identify multilingual self-attention analysis as a direction for future work.
- The authors compare self-attention patterns in pre-trained and fine-tuned BERT using a proposed analysis methodology.
- Repeated attention patterns and beneficial attention disabling suggest model pruning and smaller BERT sub-architectures as future research directions.
- Future work should examine self-attention patterns in other languages, especially verb-final and free-word-order languages.
A Examples of full self-attention maps
This section illustrates different self-attention head patterns through complete maps from fine-tuned models. Each map uses a randomly sampled example from its corresponding dataset.
- The section presents examples of full self-attention maps to illustrate different head patterns.
- The maps come from a set of fine-tuned models.
- Each map uses a randomly sampled example from its corresponding dataset.