Source-linked AI summary
What Does BERT Look At? An Analysis of BERT's Attention
Kevin Clark, Urvashi Khandelwal, Omer Levy, Christopher D. Manning
TL;DR
Although BERT performs well after pre-training, the linguistic features learned from unlabeled data remain insufficiently understood. This paper analyzes BERT’s attention mechanisms and finds recurring head behaviors, head-level correspondences to syntax and coreference, and substantial syntactic information in attention maps.
Problem
BERT’s strong supervised-task performance leaves open which linguistic features its pre-training learns, while prior analyses mainly examine outputs or internal vector representations.
Method
The paper analyzes BERT’s attention maps and probes individual heads and combinations of heads for surface patterns, syntactic relations, coreference, and dependency parsing.
Results
Particular heads identify direct objects, determiners, prepositional objects, possessive-pronoun objects, and coreferent mentions with high accuracy, while an attention-based classifier achieves 77 UAS on dependency parsing.
Takeaways & Limitations
The findings show that substantial linguistic knowledge, including syntax, is captured in BERT’s attention maps and that attention analysis complements probing outputs and hidden states.
Takeaways & Limitations
Individual heads perform well on selected dependency relations but do not capture dependency structure as a whole.
Abstract
from arXiv · showhide
Large pre-trained neural networks such as BERT have had great recent success in NLP, motivating a growing body of research investigating what aspects of language they are able to learn from unlabeled data. Most recent analysis has focused on model outputs (e.g., language model surprisal) or internal vector representations (e.g., probing classifiers). Complementary to these works, we propose methods for analyzing the attention mechanisms of pre-trained models and apply them to BERT. BERT's attention heads exhibit patterns such as attending to delimiter tokens, specific positional offsets, or broadly attending over the whole sentence, with heads in the same layer often exhibiting similar behaviors. We further show that certain attention heads correspond well to linguistic notions of syntax and coreference. For example, we find heads that attend to the direct objects of verbs, determiners of nouns, objects of prepositions, and coreferent mentions with remarkably high accuracy. Lastly, we propose an attention-based probing classifier and use it to further demonstrate that substantial syntactic information is captured in BERT's attention.
1 Introduction
The paper analyzes BERT’s attention heads to understand what linguistic structure emerges from pre-training, complementing prior analyses of outputs and vector representations. It finds recurring attention patterns, heads specialized for particular syntactic and coreference relations, and substantial syntactic information in attention maps.
- The study examines BERT’s 144 attention heads as an interpretable complement to analyses of model outputs and hidden representations.Attention weights indicate how strongly each word contributes to the current word’s next representation.
- BERT’s heads exhibit recurring behaviors, including fixed positional offsets, broad sentence-wide attention, and strong attention to the [SEP] delimiter.Heads within the same layer also tend to behave similarly.
- >75% accuracy is achieved by particular heads identifying direct objects of verbs, determiners of nouns, objects of prepositions, and objects of possessive pronouns.A separate head also performs quite well for coreference, despite receiving no explicit syntax or coreference supervision.
- 77 UAS is achieved by an attention-based probing classifier for dependency parsing, indicating substantial syntactic information in BERT’s attention maps.The classifier uses attention maps to measure syntax beyond the specialization of individual heads.
2 Background: Transformers and BERT
Transformers compute attention by comparing token representations and combining value vectors, while BERT is a pretrained Transformer that uses special delimiter tokens in its input. These design choices determine how token-to-token information is weighted and aggregated.
- Each Transformer attention head transforms token vectors into queries, keys, and values, then computes pairwise softmax-normalized attention weights.The head output is a weighted sum of the value vectors.
- Attention weights govern how important every other token is when producing the next representation for the current token.
- BERT adds [CLS] at the beginning and [SEP] at the end of input, using additional [SEP] tokens to separate multiple texts.The paper reports that these special tokens play an important role in BERT’s attention.
3 Surface-Level Patterns in Attention
BERT’s attention heads exhibit recurring surface-level patterns involving relative position, special tokens, and attention breadth. Heads within the same layer often behave similarly, while some heads concentrate on delimiters or nearby tokens and others broadly aggregate sentence information.
- Relative Position: Four heads put >50% of their attention on the previous token, while five heads put >50% on the next token.These positional-offset heads occur especially in earlier layers.
- Attending to Separator Tokens: Over half of BERT’s attention in layers 6-10 focuses on [SEP], far above the roughly 1/64 attention expected for a token occurring twice in 128-token segments.Early heads often attend to [CLS], middle heads to [SEP], and deep heads to periods and commas.
- Attending to Separator Tokens: Heads processing [SEP] attend more than 90% of the time to themselves and the other [SEP] token rather than broadly over the segment.This pattern, together with non-nouns attending to [SEP] when a head’s specific function is inapplicable, motivates interpreting special-token attention as a possible no-op.
- Attending to Separator Tokens: Gradient magnitudes for attention to [SEP] become very small starting in layer 5, where attention to [SEP] also becomes high.The gradients measure how changing attention to a token affects BERT’s masked-language-modeling outputs.
- Focused vs Broad Attention: Some lower-layer heads have high-entropy attention distributions, assigning at most 10% of their attention mass to any single word.Their outputs therefore resemble a bag-of-vectors representation of the sentence.
- Focused vs Broad Attention: The last layer’s [CLS] representation has attention entropy of 3.89 nats, indicating very broad attention across the input.The [CLS] representation is used for next-sentence prediction during pre-training.
4 Probing Individual Attention Heads
The paper probes BERT’s attention heads on dependency syntax and coreference, finding that individual heads often specialize in particular linguistic relations rather than capturing syntax uniformly.
- 4 Probing Individual Attention Heads: Attention maps are converted from token-level to word-level representations by summing attention to split words and averaging attention from split words.This addresses BERT’s byte-pair tokenization before evaluating word-level linguistic relations.
- 4.2 Dependency Syntax: Dependency parsing evaluation compares both attention directions against fixed-offset baselines on Stanford Dependencies from the WSJ Penn Treebank.The setup tests whether heads predict heads or dependents and includes simple positional comparisons.
- 4.2 Dependency Syntax: 34.5 UAS is the best individual-head syntax score, only modestly above the 26.3-UAS right-branching baseline.No single attention head performs well across syntax overall.
- 4.2 Dependency Syntax: Certain heads specialize in specific dependency relations, sometimes achieving high accuracy and substantially outperforming fixed-offset baselines.For most listed relations, dependents attend to their single head, while heads may have multiple dependents.
- 4.2 Dependency Syntax: Figure 5 illustrates attention heads corresponding to linguistic phenomena, with line darkness encoding attention strength and red highlighting selected words.The figure’s examples include heads associated with syntactic relations and omit Head 9-6’s attention to [SEP] for clarity.
- 4.2 Dependency Syntax: The authors caution that individual heads should not be taken to capture dependency structure as a whole because many relations only slightly improve over simple baselines.They identify cross-language analysis as future work.
- 4.3 Coreference Resolution: Coreference evaluation measures antecedent-selection accuracy and compares attention heads with nearest-mention, string-matching, rule-based, and neural baselines.One attention head improves by over 10 accuracy points over the string-matching baseline and performs close to the rule-based system, especially for nominal mentions.
5 Probing Attention Head Combinations
The paper combines attention heads, optionally with word embeddings, to probe dependency syntax and coreference. The attention-and-words probe achieves 77 UAS, while individual heads specialize in particular relations rather than encoding dependency structure uniformly.
- Coreference: One BERT attention head improves coreference antecedent selection by over 10 accuracy points over string matching and performs close to the rule-based system.Its performance is particularly strong for nominal mentions.
- Dependency parsing: The probing classifiers treat BERT attention outputs as fixed and predict a probability distribution over candidate syntactic heads for each input word.They train only a small number of parameters rather than back-propagating into BERT.
- Dependency parsing: The attention-only probe learns a supervised linear combination of attention weights in both directions between candidate heads and dependents.The model uses attention weights from all heads and trains weight vectors on the training set.
- Dependency parsing: The attention-and-words probe conditions attention-head weights on GloVe embeddings, allowing word-sensitive selection of specialized heads.For example, the classifier can favor a determiner-specialized head for a dependent–head pair such as “the” and “cat.”
- Dependency parsing: 77 UAS: the Attn + GloVe probing classifier substantially outperforms the baselines on Penn Treebank dependency parsing.The comparison includes right-branching, a GloVe-and-distance network, and a randomly initialized BERT-attention baseline.
6 Clustering Attention Heads
The authors cluster BERT attention heads by Jensen–Shannon divergence between their attention distributions. The resulting structure shows behavioral clusters and frequent similarity among heads within the same layer.
- Clustering method: The distance between two attention heads is based on the Jensen–Shannon divergence between their attention distributions.Multidimensional scaling embeds the heads in two dimensions so Euclidean distances approximate these divergences.
- Results: The analysis finds several clear clusters of attention heads with similar behavior, often corresponding to behaviors discussed elsewhere in the paper.The clusters therefore organize heads by their observed attention distributions.
- Visualization: Figure 6 displays BERT attention heads in two dimensions, with distances approximately matching average Jensen–Shannon divergences between corresponding heads.The plotted head behaviors come from the paper’s preceding attention analyses.
- Results: Heads within the same layer are often close together, indicating similar attention distributions within layers.The authors note this apparent redundancy may be related to attention dropout, which zeroes some weights during training.
7 Related Work
Related work analyzes what language models learn through outputs, internal representations, and attention. This paper extends attention analysis with quantitative studies of linguistic behavior, while distinguishing its goal from explaining model predictions.
- Model outputs: One research line evaluates language-model outputs on carefully chosen sentences, including subject–verb agreement over long distances.These evaluations use model performance as evidence about learned linguistic behavior.
- Internal representations: Another line probes internal vector representations with supervised classifiers for tasks such as part-of-speech tagging.High probing accuracy suggests that representations reflect the corresponding linguistic aspect.
- Attention analysis: Prior attention research includes visualization, quantitative analysis of memory-network attention, and studies correlating attention with syntax or anaphora.The paper positions its BERT analysis as a quantitative extension of these efforts.
- Attention analysis: Concurrent work identifies syntactic, positional, and rare-word-sensitive heads in machine translation and reports that many heads can be pruned without substantially reducing performance.Related work also shows that important remaining heads tend to have identifiable behaviors, including in BERT.
- Interpretation: Jain and Wallace argue that attention weights often do not explain predictions, whereas this paper uses attention to study information learned by models.The paper’s motivation is therefore representational analysis rather than prediction explanation.
8 Conclusion
The paper presents methods for analyzing attention mechanisms and applies them to BERT, showing that attention maps contain substantial linguistic knowledge. It argues that probing attention complements analyses of hidden states and model outputs.
- The study develops methods for analyzing neural-network attention mechanisms and applies them to BERT.
- BERT’s attention maps contain substantial linguistic knowledge, alongside information found in hidden states.
- Probing attention maps complements analyses of model outputs and vector representations.