Source-linked AI summary
How Does BERT Answer Questions? A Layer-Wise Analysis of Transformer Representations
Betty van Aken, Benjamin Winter, Alexander Löser, Felix A. Gers
TL;DR
Transformer models are difficult to interpret internally, motivating a layer-wise study of BERT’s hidden states on Question Answering. The paper combines visualizations with general and QA-specific probing across layers and finds recurring transformation phases, limited fine-tuning effects on semantic abilities, and early-layer signals of prediction errors.
Problem
The paper addresses insufficient understanding of Transformer models’ internal functioning and the need for more transparent analysis of their predictions.
Method
The authors analyze fine-tuned BERT hidden states layer by layer using token-vector visualizations and general plus QA-specific probing tasks across QA datasets.
Results
BERT’s transformations pass through similar phases across fine-tuned tasks, while fine-tuning has little impact on semantic abilities and errors can appear in early representations.
Takeaways & Limitations
Hidden-state analysis provides information for identifying misclassified examples, model weaknesses, and context parts considered important for answering questions.
Takeaways & Limitations
The analysis focuses on BERT and uses preliminary tests on the small GPT-2 model, while HotpotQA probing includes a majority-class behavior in its first five layers.
Abstract
from arXiv · showhide
Bidirectional Encoder Representations from Transformers (BERT) reach state-of-the-art results in a variety of Natural Language Processing tasks. However, understanding of their internal functioning is still insufficient and unsatisfactory. In order to better understand BERT and other Transformer-based models, we present a layer-wise analysis of BERT's hidden states. Unlike previous research, which mainly focuses on explaining Transformer models by their attention weights, we argue that hidden states contain equally valuable information. Specifically, our analysis focuses on models fine-tuned on the task of Question Answering (QA) as an example of a complex downstream task. We inspect how QA models transform token vectors in order to find the correct answer. To this end, we apply a set of general and QA-specific probing tasks that reveal the information stored in each representation layer. Our qualitative analysis of hidden state visualizations provides additional insights into BERT's reasoning process. Our results show that the transformations within BERT go through phases that are related to traditional pipeline tasks. The system can therefore implicitly incorporate task-specific information into its token representations. Furthermore, our analysis reveals that fine-tuning has little impact on the models' semantic abilities and that prediction errors can be recognized in the vector representations of even early layers.
1 INTRODUCTION
The paper addresses the limited transparency of Transformer models by analyzing BERT’s hidden states layer by layer on complex QA tasks. It combines visualization and probing to examine internal abilities, fine-tuning effects, and recurring transformation phases.
- The paper targets the lack of transparency, reliability, and prediction guarantees in deep-learning models used for real-world tasks.
- Rather than relying on attention values, the analysis examines hidden states between encoder layers directly.
- The study asks whether Transformers decompose question answering, whether layers solve different tasks, how fine-tuning changes internal states, and whether layers reveal prediction failures.
- Question Answering is used as a complex downstream task because answering can require multiple simpler NLP abilities.
- The authors visualize token representations layer by layer and apply general and QA-specific probing tasks to analyze encoded abilities and fine-tuning effects.
- BERT’s transformations pass through similar phases across differently fine-tuned tasks, with general language properties encoded earlier and used later for downstream solving.
2 RELATED WORK
Prior work studies Transformer interpretability through probing, visualization, and attention, but this paper extends layer-wise analysis to fine-tuned BERT models and task-specific transformation phases.
- Earlier research applied probing tasks and post-hoc diagnostic methods to inspect information encoded in trained neural models.
- Some BERT analyses examined attention values or layer representations only in pre-trained models, without studying downstream fine-tuning.
- The closest related work focused on general transferability and did not analyze the specific phases BERT undergoes.
- Because attention may not always provide adequate explanations, this paper instead evaluates hidden states and token representations.
3 BERT UNDER THE MICROSCOPE
The analysis follows token representations through BERT’s layers using qualitative visualization and standardized probing tasks. It combines dimensionality reduction, clustering, and QA-relevant probes to track stored information.
- The study analyzes fine-tuned BERT by examining token-vector positions qualitatively and probing language abilities quantitatively across layers.
- For qualitative analysis, hidden states from correctly and falsely predicted test samples are collected at every layer after padding removal.
- Distances between token vectors are treated as indications of semantic relations because vector-space positions lack fixed semantic references.
- t-SNE, PCA, and ICA reduce 512- or 1024-dimensional vectors to two dimensions, while k-means checks whether observed clusters reflect high-dimensional distributions.
- The probing setup uses every BERT layer and Edge Probing to classify labeled token edges after pooling them into fixed-length representations.
- The probes cover Named Entity Labeling, Coreference Resolution, Relation Classification, Question Type Classification, and Supporting Fact Identification.
- Supporting Fact probing tests whether token representations encode which context sentences are relevant to a given question.
4 DATASETS AND MODELS
The experiments compare BERT and GPT-2 across diverse QA datasets designed to vary in context, reasoning, and task structure. Models are fine-tuned under dataset-specific settings and analyzed through probing results.
- The study uses SQuAD, bAbI, and HotpotQA to diversify its analysis across distinct Question Answering settings.
- SQuAD focuses on span prediction over natural questions, whereas bAbI uses artificial multi-sentence tasks involving positional reasoning, argument relations, and coreference.
- HotpotQA tests multihop reasoning by combining supporting and distracting facts in contexts averaging 900 words, with inputs reduced to fit BERT’s 512-token limit.
- The models include BERT and GPT-2 Transformer variants, with sequence classification used for bAbI and span prediction used for the other datasets.
- Training uses publicly available pre-trained models that are fine-tuned separately on each dataset.
- BERT-base probing results compare SQuAD-, bAbI-, and non-fine-tuned models, while BERT-large results compare HotpotQA fine-tuning with no fine-tuning.
- The dataset-specific settings distinguish HotpotQA Support Only from the distractor task and use different bAbI training and prediction modalities.
5 RESULTS AND DISCUSSION
BERT’s hidden states evolve through recurring phases: early layers encode general language information, middle layers connect entities and relations, and later layers match supporting facts and extract answers. These representations also expose task-specific behavior, fine-tuning effects, and likely causes of incorrect predictions.
- 5.1 Phases of BERT’s Transformations: BERT’s token representations pass through multiple phases across diverse QA tasks, supported by PCA visualizations and probing results.The observed phases recur in SQuAD, HotpotQA, and bAbI models.
- 5.1 Phases of BERT’s Transformations: Early layers group tokens into topical clusters with little task-specific information, resembling general-purpose embedding spaces.These layers show low accuracy on semantic probing tasks.
- 5.1 Phases of BERT’s Transformations: Middle layers connect entities, mentions, and attributes through contextual relations while filtering toward question-relevant information.Examples include clusters linking entities such as Emily, wolf, and Wolves in bAbI, and contextual relations in SQuAD.
- 5.1 Phases of BERT’s Transformations: Named entity recognition improves before coreference resolution and relation recognition, whose probing performance peaks in higher layers.The same pattern appears in both BERT-base and BERT-large models.
- 5.1 Phases of BERT’s Transformations: Higher layers match question tokens to relevant context and supporting facts, but HotpotQA’s weak supporting-fact discrimination helps explain its poorer performance.Supporting-fact extraction is strongest in higher layers for SQuAD and bAbI, while fine-tuned HotpotQA does not reach high accuracy.
- 5.1 Phases of BERT’s Transformations: In the last layers, BERT separates answer tokens from the remaining context, producing task-specific representations and reducing general probing performance.This loss of general ability is especially visible in the fine-tuned BERT-large HotpotQA model.
- 5.3 Additional Findings: Hidden-state patterns reveal wrong-answer mechanisms, including confident selection of an incorrect supporting fact or low-confidence persistence of homogeneous semantic clusters.The representations can expose likely errors even in early layers.
- 5.3 Additional Findings: Fine-tuning has little effect on core NLP abilities overall, while positional information remains influential in late layers and question-type abilities change by task.SQuAD improves question-type resolution, bAbI loses some of it, and HotpotQA shows no fine-tuning advantage on this probing task.
6 CONCLUSION AND FUTURE WORK
The paper argues that hidden-state analysis reveals interpretable information and distinct problem-solving phases within Transformer networks, with implications for transfer learning, modularity, and future model design.
- Interpretability: Hidden-state visualizations reveal interpretable information that can identify misclassified examples, model weaknesses, and context considered important for answering questions.The authors connect this information to decision legitimisation but leave methods for further processing to future work.
- Transferability: Lower layers may suit some problems better than later layers, so transfer-learning systems should choose layer depth according to the task.The paper also suggests studying skip connections between non-adjacent layers that solve different tasks.
- Modularity: Distinct transformation phases and task-specific layers support a possible modularity that could be exploited during training.The authors suggest fitting parts of the network to specific tasks during pre-training instead of relying solely on an end-to-end language-model objective.
- Future work: The work directs future research toward understanding how state-of-the-art Transformer models solve downstream tasks in order to improve them.This recommendation follows the paper’s broader aim of revealing internal processes within Transformer-based models.