Source-linked AI summary
Open Sesame: Getting Inside BERT's Linguistic Knowledge
Yongjie Lin, Yi Chern Tan, Robert Frank
TL;DR
The paper asks how much BERT represents hierarchical syntax rather than merely linear position. Using layerwise diagnostic classifiers and self-attention analyses, it finds that BERT shifts from positional encoding in lower layers toward increasingly hierarchical representations in higher layers, though structural sensitivity remains incomplete.
Problem
The paper investigates how and where BERT representations express sentence-level hierarchical organization rather than only linear positional information.
Method
The authors combine layer-specific diagnostic classifiers with self-attention analyses of agreement and anaphor-antecedent dependencies.
Results
BERT shifts from positional information in lower layers toward increasingly hierarchical representations in higher layers, including syntactic information relevant to agreement and reflexive dependencies.
Takeaways & Limitations
BERT composes increasingly abstract linguistic structure through self-attention, but its attention is only incompletely modulated by structural and grammatical features.
Takeaways & Limitations
BERT performs poorly on possessive generalization cases, possibly because the ambiguous ’s token encourages reliance on local cues rather than unified sentence structure.
Abstract
from arXiv · showhide
How and to what extent does BERT encode syntactically-sensitive hierarchical information or positionally-sensitive linear information? Recent work has shown that contextual representations like BERT perform well on tasks that require sensitivity to linguistic structure. We present here two studies which aim to provide a better understanding of the nature of BERT's representations. The first of these focuses on the identification of structurally-defined elements using diagnostic classifiers, while the second explores BERT's representation of subject-verb agreement and anaphor-antecedent dependencies through a quantitative assessment of self-attention vectors. In both cases, we find that BERT encodes positional information about word tokens well on its lower layers, but switches to a hierarchically-oriented encoding on higher layers. We conclude then that BERT's representations do indeed model linguistically relevant aspects of hierarchical structure, though they do not appear to show the sharp sensitivity to hierarchical structure that is found in human processing of reflexive anaphora.
1 Introduction
The paper investigates how and where pretrained BERT representations encode sentence-level hierarchical structure, using diagnostic classifiers and self-attention analyses. The findings indicate that BERT favors syntactically relevant structural information over linear information as its layers build increasingly abstract linguistic representations.
- Research aim: The paper examines how and where pretrained BERT representations express the hierarchical organization of sentences.This addresses how BERT captures linguistic structure beyond its strong downstream task performance.
- Diagnostic classifiers: Diagnostic classifiers probe whether BERT word representations contain hierarchical or linear properties using a poverty-of-the-stimulus paradigm.The enriched generalization set distinguishes solutions based on linear versus hierarchical information.
- Self-attention analysis: Self-attention analyses test whether BERT attends preferentially to syntactically relevant noun phrases in agreement and anaphora.The approach provides evidence about structural information through attention weightings.
- Findings: BERT’s attention favors the correct noun phrase for agreement and anaphora, but syntactic structure lacks the strong categorical effects sometimes found in natural language.Thus, BERT represents syntactically relevant hierarchical information without exhibiting fully categorical structural sensitivity.
- Overall contribution: BERT’s self-attention layers compose increasingly abstract linguistic representations without explicit word-order information, expressly favoring structural over linear information.This structural bias helps explain BERT’s performance on downstream tasks requiring complex modeling of structural relationships.
2 Diagnostic Classification
Diagnostic classifiers test whether BERT embeddings encode sentence roles and token positions beyond linear heuristics. Results show strong hierarchical behavior for main auxiliaries and some subject-noun cases, while lower layers retain positional information and higher-layer subject-noun performance varies by construction and model.
- 2 Diagnostic Classification: Diagnostic classifiers recover main auxiliaries, subject nouns, and nth-token positions from layer-specific BERT embeddings, with pre-embeddings serving as input-information baselines.The classifiers are trained and evaluated separately at every BERT layer, including the pre-embeddings as a zero-th layer.
- 2 Diagnostic Classification: Training on sentences compatible with both rules tests whether classifiers generalize from linear patterns to structurally defined main auxiliaries and subject nouns.The generalization sets contain more complex sentences where hierarchical and sequential rules make different predictions.
- Main auxiliary: Over 85% accuracy past layer 5 shows that both models identify main auxiliaries on the generalization set, despite training examples always placing them first.bbu reaches near-perfect accuracy from layers 4 to 11, while both models achieve near-perfect development-set accuracy across layers.
- Subject noun: Subject-noun results are model- and construction-dependent: bbu learns to exclude compound-noun dependents, whereas blu performs poorly and both models struggle with possessives.The possessive difficulty may reflect ambiguity in ’s, which can mark possession or contract an auxiliary verb.
- Nth-token: Layers 1–3 achieve near-perfect accuracy for arbitrary nth-token identification, but accuracy drops sharply with increasing n from layer 4 onward.The contrast between pE and pE – pos indicates that positional information is strongest in lower-layer representations and diminishes in higher layers.
3 Diagnostic Attention
BERT’s self-attention was evaluated with a confusion score on subject-verb agreement and reflexive-anaphora dependencies. The results show sensitivity to syntactic structure and grammatical features, with confusion generally decreasing in deeper layers but not reaching ideal hierarchical behavior.
- Diagnostic metric: The confusion score quantifies how preferentially self-attention selects the true dependency trigger over competing candidate triggers.It is computed from normalized attention distributions across candidate triggers for each dependency target.
- Dataset design: Synthetic datasets varied clause type, distractor features, and antecedent accessibility while controlling semantic effects across 10000 examples per condition.Subject-verb datasets manipulated PP versus RC embedding and distractor number; reflexive datasets manipulated RC presence, object presence, and gender matching.
- Subject-verb agreement: Subject-verb agreement confusion was higher for PP distractors than RC distractors, and matching distractors significantly increased confusion.Although all agreement conditions were unambiguous and ideally should have zero confusion, observed scores were closer to the uniform probability baseline of 1.
- Reflexive anaphora: Reflexive-anaphora confusion increased with the number of distractor noun phrases, while grammatically accessible and feature-matching distractors produced stronger effects than inaccessible distractors.This indicates sensitivity to both syntactic structure and grammatical features, despite added confusion from grammatically inaccessible distractors.
- Layerwise diagnosis: Confusion was negatively correlated with layer depth for both reflexive anaphora and subject-verb agreement, supporting increasingly abstract hierarchical representations.Confusion increased at layer 4, which was also identified as the level where linear information was prominent.
4 Conclusion
BERT shifts from encoding positional information in lower layers to increasingly abstract hierarchical representations in higher layers. Diagnostic attention also shows syntactically relevant information for agreement and reflexive dependencies, motivating further study of layer-wise changes and representations.
- 4 Conclusion: BERT encodes positional information from the pre-embedding level through lower layers, while higher layers become less positional and more hierarchical.Higher layers also encode increasingly complex representations of sentence units.
- 4 Conclusion: BERT’s self-attention encodes syntactic structure relevant to subject-verb agreement and reflexive dependencies, with more accurate representations on higher layers.The study uses diagnostic attention to observe distractor intrusion effects on self-attention.
- 4 Conclusion: Further work should examine layer-on-layer changes in structural and positional information and retrieve BERT’s increasingly hierarchical representations across layers.The proposed focus includes the transition around layers 3 and 4, when positional information is largely phased out.
A Appendix · A.1 Context-free grammars for dataset generation
This appendix specifies context-free grammars used to generate the main auxiliary, subject noun, subject-verb agreement, and reflexive anaphora datasets. The grammars define noun phrases, relative clauses, and singular/plural agreement variants through explicit production rules.
- A.1 Context-free grammars for dataset generation: The main auxiliary grammar defines NPM and NPO noun phrases with determiner-noun structures, prepositional modifiers, or relative clauses.Its relative-clause productions combine relational elements with auxiliary verbs, intransitives, transitive verbs, and determiner-noun complements.
- A.1 Context-free grammars for dataset generation: NPM expands as Det N, Det N Prep Det Nom, or Det N RC, while NPO expands as Det Nom, Det Nom Prep Det Nom, or Det Nom RC.These productions are identified as part of the main auxiliary dataset grammar.
- A.1 Context-free grammars for dataset generation: RC expands as Rel Aux VI, Rel Det Nom Aux VT, or Rel Aux VT Det Nom in the main auxiliary dataset.The grammar therefore includes both intransitive and transitive relative-clause patterns, with the latter appearing before or after a determiner-noun complement.
- A.1 Context-free grammars for dataset generation: The subject noun grammar replaces the NPM head with MNom while retaining prepositional and relative-clause alternatives.Its NPM productions are Det MNom, Det MNom Prep Det Nom, and Det MNom RC; NPO retains the three corresponding Det Nom patterns.
- A.1 Context-free grammars for dataset generation: The subject-verb agreement grammar distinguishes singular and plural noun phrases through NPsg Agr and NPpl Agr productions.NPsg Agr uses Det Nsg, Det Nsg Prep Det N, or Det Nsg Prep RCsg, whereas NPpl Agr uses Det Npl, Det Npl Prep Det N, or Det Npl Prep RCpl.
- A.1 Context-free grammars for dataset generation: The agreement grammar defines RCsg with singular and plural auxiliary configurations alongside intransitive and transitive continuations.RCsg expands as Rel Auxsg VI, Rel Auxsg VT Det N, Rel Det Nsg Auxsg VT, or Rel Det Npl Auxpl VT.
- A.1 Context-free grammars for dataset generation: Figure 9 presents a context-free grammar for generating the reflexive anaphora dataset.The supplied appendix passage identifies the figure but does not provide its individual production rules.