Source-linked AI summary
A Primer in BERTology: What we know about how BERT works
Anna Rogers, Olga Kovaleva, Anna Rumshisky
TL;DR
BERT’s strong performance is not well understood, limiting explanation and hypothesis-driven improvement. This survey synthesizes over 150 studies of BERT’s learned knowledge, representations, training and architectural modifications, compression, and analysis. It concludes by identifying unresolved questions and directions for future research.
Problem
BERT works remarkably well, but it remains unclear why, while model size limits experimentation and ablation studies.
Method
The paper surveys evidence about BERT’s knowledge and representations, technical modifications, overparameterization, compression, pruning, and unresolved research questions.
Results
The survey synthesizes a rapidly growing literature on BERT and identifies major unresolved questions for future research.
Takeaways & Limitations
Future work should develop comprehensive linguistic stress tests and examine which information BERT actually uses at inference time.
Takeaways & Limitations
Probing evidence is limited because observed information does not establish how BERT uses it, and different methods can produce contradictory conclusions.
Abstract
from arXiv · showhide
Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue and approaches to compression. We then outline directions for future research.
1 Introduction
BERT achieves strong NLP benchmark performance, but the reasons for its success remain unclear. This survey reviews evidence about what BERT learns and proposals for improving, compressing, and analyzing it.
- BERT became a state-of-the-art baseline across numerous NLP benchmarks.
- BERT’s success is difficult to explain, limiting hypothesis-driven architectural improvement and ablation studies.
- The survey reviews evidence about BERT’s linguistic and world knowledge and how that knowledge is represented.
- It also surveys proposed changes to BERT’s architecture, pre-training, fine-tuning, compression, and pruning-based analysis.
2 Overview of BERT architecture
BERT is a Transformer encoder built from stacked self-attention layers, with token representations formed from wordpieces and multiple embeddings. Its conventional training uses masked language modeling and next sentence prediction before downstream fine-tuning.
- BERT consists of stacked Transformer encoder layers containing multiple self-attention heads, combined with feed-forward layers, skip connections, and layer normalization.
- Pre-training uses masked language modeling and next sentence prediction as self-supervised tasks.
- Fine-tuning typically adds one or more fully connected layers above the final encoder layer for downstream applications.
- Each input word is tokenized into wordpieces, then token, position, and segment embeddings are combined into a fixed-length vector.
- BERT variants such as base and large differ in their numbers of heads and layers and in hidden-state size.
3 What knowledge does BERT have?
BERT contains recoverable syntactic, semantic, and world knowledge, but evidence is mixed about how directly that knowledge is represented and whether it is used during inference. Probing results depend substantially on the method and formalism.
- Syntactic knowledge: Syntactic structure is not directly encoded in self-attention weights, although syntactic dependencies can be recovered from token representations.
- Syntactic knowledge: A parameter-free MLM probe found that BERT naturally learns some syntactic information, though it does not closely match annotated linguistic resources.
- Syntactic and semantic knowledge: BERT’s MLM predictions reflect subject-predicate agreement and some semantic-role information, including preferences among semantically related fillers.
- Semantic knowledge: Probing classifiers detect entity types, relations, semantic roles, and proto-roles in BERT representations.
- Semantic and commonsense knowledge: BERT struggles with numerical representations, abstract and perceptual properties, pragmatic inference, and generic named-entity representations.
- World knowledge: BERT can retrieve some world knowledge through MLM templates, but it cannot reliably reason over relationships or multiple inference steps.
- Limitations: Probing conclusions are method-dependent: detected information does not establish that BERT uses it, and different probes can yield contradictory results.
4 Localizing linguistic knowledge
BERT’s linguistic knowledge is distributed across layers and represented in both token embeddings and attention patterns, but its localization and interpretability remain incomplete. Syntactic information is most prominent in middle layers, while later layers become more task-specific and semantic information is spread more broadly.
- BERT embeddings: Distilled contextualized embeddings often encode lexical semantic information better for traditional word-level tasks, while later BERT layers become more context-specific and anisotropic.Identical-word representations become more context-dependent in later layers, and embeddings increasingly occupy a narrow cone in vector space.
- BERT embeddings: BERT’s contextualized embeddings form distinct clusters for different word senses, supporting successful word sense disambiguation.However, representations of the same word also depend on sentence position, likely due to the next sentence prediction objective.
- Attention heads: Some attention heads specialize in syntactic relations, especially object positions, but evidence for other relations varies across studies.Complex dependencies may be encoded by combinations of heads rather than individual heads, although that evidence is qualitative.
- Attention heads: Attention weights are weak indicators of subject-verb agreement and reflexive anaphora, often remaining close to a uniform baseline despite distractor sensitivity.Individual-head patterns do not provide a complete picture in a multilayer model with nonlinear transformations.
- Attention heads: Most self-attention heads do not directly encode non-trivial linguistic information, with fewer than 50% showing heterogeneous patterns after GLUE fine-tuning.Many heads instead attend vertically to special tokens, and norm-based analyses still find that most influential heads are not clearly interpretable.
- Attention heads: Inter-word attention analyses can miss information carried by special tokens, whose functions remain poorly understood and may include sentence-level aggregation or no-op signaling.Attention to [CLS], [SEP], and punctuation varies across layers, so excluding special tokens can make syntactic conclusions unwarranted.
- BERT layers: Syntactic information is most prominent in middle BERT layers, whereas final layers are more task-specific and middle layers are more transferable.Tree-depth reconstruction succeeds around layers 6-9 in base BERT and 14-19 in BERT-large; restoring lower-layer weights after fine-tuning does not dramatically hurt performance.
- BERT layers: Evidence about layerwise localization is mixed: basic syntax may appear earlier, semantic features higher, and semantic information may be distributed across the entire model.Different probing studies disagree about syntactic chunks and semantic peaks, while BERT-base and BERT-large show the same cumulative pattern with different spread.
5 Training BERT
This section surveys modifications to BERT’s architecture, training objectives, pretraining data, and fine-tuning, while emphasizing expensive ablations, variable gains, and reproducibility concerns.
- Model architecture choices: The number of layers mattered more than the number of attention heads, while larger hidden representations consistently improved results but with setting-dependent gains.
- Model architecture choices: More diverse attention patterns could increase the information represented with the same number of weights, whereas naturally repeated patterns help explain successful pruning.
- Training optimization: 25% faster training was achieved by recursively warm-starting deeper models from shallower trained versions without sacrificing performance.
- Pretraining objectives and data: Alternative objectives modify masking, sentence-order prediction, pretraining data, or injected linguistic and structured knowledge to improve BERT.
- Pretraining and fine-tuning: Pretraining generally helps, but its exact contribution remains uncertain because some tasks favor randomly initialized fine-tuning and systematic evaluation is costly.
- Fine-tuning BERT: During GLUE fine-tuning, most changes occurred in the final two layers and shifted attention toward [SEP] rather than linguistically interpretable patterns.
- Evaluation and scope: Reported improvements can vary with initialization and data order, while the survey does not cover several fine-tuning alternatives.
6 How big should BERT be?
This section examines BERT’s overparameterization, compression methods, and pruning-based analysis. Evidence indicates substantial redundancy, but also task dependence and unresolved questions about diffuse representations.
- Overparameterization: Most Transformer heads can be pruned without significant performance loss, and BERT’s repeated within-layer attention patterns help explain this redundancy.
- Overparameterization: 30–40% of weights can be pruned without impact on downstream tasks, while some heads or layers can even harm task performance.
- Model size: Larger BERT models generally perform better, but BERT-base outperformed BERT-large on subject-verb agreement and sentence subject detection.
- Compression techniques: Knowledge distillation, quantization, and pruning are the main compression approaches, with quantization reducing memory through lower-precision weights but often requiring compatible hardware.
- Compression techniques: Pruning removes model components or weights to reduce computation, including structured removal of architecture blocks and searches for useful pretrained subnetworks.
- Pruning and model analysis: Pruning can also analyze models by identifying retained components, but syntactic and positional heads versus linguistically non-trivial heads produce inconclusive evidence across studies.
- Pruning and model analysis: A key limitation of head and layer ablations is their assumption that knowledge is localized, despite evidence for representations distributed across the network.
7 Directions for further research
Future research should test BERT’s competence more rigorously, teach it to perform reasoning, and identify which knowledge it actually uses at inference time.
- Benchmarks that require verbal reasoning: BERT’s apparent language skills can rely on shallow heuristics across several NLP tasks, motivating benchmarks that require verbal reasoning.The survey cites evidence from natural language inference, reading comprehension, argument reasoning comprehension, and text classification.
- Benchmarks for the full range of linguistic competence: More comprehensive stress tests are needed to measure the full range of BERT’s linguistic competence, including both errors and sensitivity.The Checklist behavioral testing is presented as one step toward this goal.
- Developing methods to teach reasoning: Because pretrained models often fail when reasoning must be performed over possessed facts, researchers are developing methods to teach BERT quantification, conditionals, comparatives, and boolean coordination.Richardson et al. propose a method targeting these reasoning capabilities.
- Learning what happens at inference time: Future analysis should determine which knowledge BERT uses during inference rather than only probing what information it contains.Suggested approaches include amnesic probing, identifying prediction-important features, and pruning unimportant components.
8 Conclusion
BERT quickly became a ubiquitous NLP baseline and prompted many studies of its behavior and possible improvements. The survey aims to help researchers focus on the field’s biggest unresolved questions as this work accelerates.
- In a little over a year, BERT became a ubiquitous baseline in NLP experiments and inspired studies analyzing it and proposing improvements.