Source-linked AI summary
Analyzing the Structure of Attention in a Transformer Language Model
Jesse Vig, Yonatan Belinkov
TL;DR
The paper asks how attention in a Transformer language model relates to syntax and token relationships. It visualizes GPT-2 attention at multiple levels and analyzes it over a large corpus, finding depth-specific syntactic and distance patterns. These analyses also produce interpretable examples, while their scope is limited by the sentence style and length of the analyzed text.
Problem
The paper investigates how attention patterns relate to dependency relations, part-of-speech tags, and long- versus short-distance relationships in a Transformer language model.
Method
The authors visualize GPT-2 attention at attention-head, model, and neuron levels and analyze attention in aggregate over a large corpus.
Results
Attention targets different part-of-speech tags at different layer depths, aligns most strongly with dependency relations in middle layers, and spans the greatest distance in deepest layers.
Takeaways & Limitations
Attention analysis yields human-interpretable descriptions of GPT-2 behavior and exemplar sentences showing specific patterns targeted by attention heads.
Takeaways & Limitations
The analysis uses Wikipedia sentences with similar encyclopedic style and does not establish how these attention patterns manifest in other content types or much longer text.
Abstract
from arXiv · showhide
The Transformer is a fully attention-based alternative to recurrent networks that has achieved state-of-the-art results across a range of NLP tasks. In this paper, we analyze the structure of attention in a Transformer language model, the GPT-2 small pretrained model. We visualize attention for individual instances and analyze the interaction between attention and syntax over a large corpus. We find that attention targets different parts of speech at different layer depths within the model, and that attention aligns with dependency relations most strongly in the middle layers. We also find that the deepest layers of the model capture the most distant relationships. Finally, we extract exemplar sentences that reveal highly specific patterns targeted by particular attention heads.
1 Introduction
The paper analyzes GPT-2 attention through visualizations and corpus-level comparisons with syntax, asking how attention relates to dependencies, part-of-speech tags, and relationship distance.
- The authors extend attention visualization to attention-head, model, and neuron levels, adapting it to decoder-only GPT-2 and encoder-only BERT.
- The corpus analysis asks whether attention aligns with dependency relations, which heads target particular part-of-speech tags, and how attention captures long- versus short-distance relationships.
- The study applies its analysis to the GPT-2 small pretrained model and examines attention over a large corpus.
- Attention aligns most strongly with dependency relations in the middle layers, targets different parts of speech at different depths, and spans the greatest distance in the deepest layers.
- The method also extracts exemplar sentences that reveal intuitive, highly specific patterns targeted by particular attention heads.
2 Related Work
Prior work shows that Transformers and recurrent models encode syntactic information, while this paper directly compares attention patterns with syntactic constructs rather than relying on probing tasks alone.
- Previous studies suggest that Transformers encode dependency trees, anaphora, and subject-verb pairings, while deeper layers capture increasingly high-level constructs.
- This paper directly compares attention patterns with dependency relations and part-of-speech tags, contrasting with linguistic probing approaches.
- Earlier work also evaluated dependency trees induced from attention weights, but in encoder-decoder translation models rather than this GPT-2 language-model setting.
3 Transformer Architecture
GPT-2 is a 12-layer, 12-head stacked decoder Transformer whose self-attention assigns normalized, causal weights between token positions and combines value vectors.
- GPT-2 small uses 12 decoder layers and 12 attention heads, with embeddings followed by repeated self-attention, feedforward, normalization, and residual components.
- For each token xi, self-attention assigns positive weights over input tokens, with weights summing to one and GPT-2 restricting attention to positions j ≤ i.
- The weights are computed from scaled query-key dot products followed by softmax, then used to form a weighted sum of value vectors.
- In multi-head attention, queries, keys, and values are linearly projected h times, processed in parallel, and concatenated.
4 Visualizing Individual Inputs
The paper visualizes Transformer attention at head, model, and neuron levels, enabling inspection of individual heads, cross-layer patterns, and neuron contributions.
- Attention-head View: The attention-head view connects attending tokens to attended tokens with colored lines whose weights reflect attention strength.
- Attention-head View: In the example, layer 4 head 11 focuses attention primarily on the previous token position.
- Model View: The model view arranges heads in rows by layer and columns by head, using thumbnails to show coarse attention-pattern shapes.
- Model View: Horizontal stripes indicate attention to the current position, triangular patterns indicate attention to the first token, and defined lines indicate specific token relationships.
- Neuron View: The neuron view displays token queries and keys and shows how element-wise products of neuron values influence the attention dot product.
5 Analyzing Attention in Aggregate
The paper examines aggregate attention properties across a corpus, focusing on syntax, distance, and head-specific patterns.
- The analysis examines how attention interacts with syntax across an entire corpus.
- The analysis compares long-distance and short-distance attention relationships.
- The authors extract exemplar sentences that reveal patterns targeted by each attention head.
5.1 Methods
The methods quantify attention's relationships with part-of-speech tags, dependency relations, positional variability, and token distance, while using visualizations to inspect attention patterns.
- 5.1.1 Part-of-Speech Tags: The authors measure each head's proportion of attention directed to particular part-of-speech tags over a corpus.They also compute the share of attention directed from each part of speech.
- 5.1.2 Dependency Relations: Dependency alignment measures the proportion of attention connecting tokens that share a dependency relation.The analysis tests three parent-direction formulations: attending token, receiving token, or either token as parent.
- Visualizations: The neuron view traces attention computation from selected tokens to attended tokens and identifies neurons contributing to the pattern.The model view summarizes attention across layers and heads, while Figure 4 shows example heads and corpus-level metrics.
- 5.1.2 Dependency Relations: Attention variability measures how a head's attention changes across inputs, distinguishing content-dependent from position-based patterns.High variability suggests content dependence, whereas low variability indicates content independence.
- 5.1.3 Attention Distance: The authors measure each head's mean attention distance as the attention-weighted average distance between token pairs.They also investigate whether more dispersed attention patterns capture more distant relationships.
5.2 Experimental Setup
The analysis uses GPT-2 small on English Wikipedia, combining attention weights with syntactic features across sampled sentences while filtering uninformative null attention.
- Filtering Null Attention: Entropy calculations removed first-token attention, renormalized remaining weights, and excluded tokens directing over 90% of attention to that token.These exclusions prevented disproportionate influence from residual attention.
- Processing: Attention weights came from a pretrained GPT-2 small implementation, while spaCy supplied syntactic features mapped onto GPT-2 tokenizer pieces.When words split into multiple pieces, features were assigned to all corresponding word pieces.
- Filtering Null Attention: The analysis excluded attention to each sentence’s first token because it was treated as uninformative null attention.On average, 57% of attention targeted this token, and some heads exceeded 97%.
- Visualizations: Figures 6 and 7 summarize attention proportions by part-of-speech tag, layer, and head, distinguishing targeted attention from originating attention.Scales vary by tag, and results for all tags appear in the appendix.
5.3 Results
Attention heads specialize by part of speech and layer depth, align with dependency relations most strongly in middle layers, and span the greatest distances in the deepest layers. These patterns vary substantially across heads and include position-based and highly specific attention behaviors.
- Part-of-Speech Tags: For 13 of 16 part-of-speech tags, some head received more than twice the tag’s mean attention share.For example, nouns received 43% of attention in layer 9, head 0, versus a 21% mean across heads.
- Part-of-Speech Tags: Heads targeting particular tags clustered by layer depth: proper-noun specialists concentrated in the last three layers, while determiner specialists occupied the first four.The authors relate this pattern to deeper layers focusing on higher-level properties.
- Part-of-Speech Tags: Beyond the initial layers, heads also specialized in the part-of-speech tags from which they drew attention, although less strongly than for targeted tags.For 7 of 16 tags, a head exceeded twice the tag-average originating frequency; punctuation specialists clustered in the last six layers.
- Dependency Relations: Dependency alignment was strongest in the middle layers, while lower alignment in initial layers may reflect position-focused heads.Layer-0 examples had dependency alignment scores of 0.04 and 0.10.
- Dependency Relations: Layer 4, head 11 combined the model’s highest dependency alignment, DepAlα = 0.42, with its strongest position focus, Variabilityα = 0.004.It attended to the previous token, which had a 42% chance of sharing a dependency relation with the adjacent token in the corpus.
- Dependency Relations: Subjects were targeted more in deeper layers, whereas auxiliaries, conjunctions, determiners, expletives, and negations were targeted more in lower layers.Interpretations for some other dependency types were less clear.
- Attention Distance: Attention distance was greatest in the deepest layers and varied widely across heads, with a standard deviation of 3.6.This result supports longer-distance relationships in deeper layers despite substantial within-layer variation.
- Attention Distance: Attention distance and entropy showed a moderate correlation, r = 0.61, p < 0.001.Many heads in layers 0 and 1 had high entropy, potentially contributing to their higher attention distance than layers 2–4.
5.4 Qualitative Analysis
The qualitative analysis extracts exemplar sentences to expose lexical patterns that strongly activate individual attention heads.
- Exemplar Sentences: Sentences were ranked by each sentence’s maximum token-to-token attention weight within a head.Examples revealed heads detecting entities, passive verbs, acronyms, nicknames, paired punctuation, and other syntactic or semantic properties.
6 Conclusion
The analysis finds structured, layer- and head-specific attention in GPT-2, with part-of-speech specialization, strongest dependency alignment in middle layers, and longest-distance relationships in deepest layers. Qualitative examples connect these patterns to GPT-2’s left-to-right language-modeling objective, while attention analysis complements linguistic probing.
- Many attention heads specialize in particular part-of-speech tags, with different tags targeted at different layer depths.
- Attention captures the most distant relationships in the deepest layers and aligns most strongly with dependency relations in middle layers.
- GPT-2 attention often focuses on words most relevant to predicting the next token, reflecting its left-to-right language-modeling objective.
- Exemplar sentences reveal specific head patterns, including acronym-to-phrase, comma-to-place-name, and noun-phrase-end-to-head-noun attention.
- The analysis is limited because its Wikipedia sentences share an encyclopedic format, leaving other content types and much longer texts for further study.
- Attention-based interpretation is complementary to linguistic probing, offering simpler human-interpretable descriptions while probing quantifies encoded information.
A Appendix
The appendix extends the part-of-speech attention visualizations to the full tag set. Its heatmaps organize attention by layer and head, showing either where attention is directed or which part of speech originates it.
- Figures A.1 and A.2 provide the full-set results corresponding to Figures 6 and 7 for all part-of-speech tags.
- Figure A.1 heatmaps show the proportion of total attention directed to each part of speech across layers and heads.
- Figure A.2 heatmaps show the proportion of attention originating from each part of speech across layers and heads.