Source-linked AI summary
Assessing BERT's Syntactic Abilities
Yoav Goldberg
TL;DR
The paper asks whether BERT captures English syntax-sensitive structures. It adapts natural, nonce, and manually controlled syntactic evaluations to BERT’s bidirectional masked-word setup and finds remarkably strong performance across all test cases. Direct comparison with prior work is limited by differences in protocol, stimuli, and training data.
Problem
The paper asks to what extent BERT, despite its attention-based architecture, captures syntax-sensitive English structures.
Method
The study adapts prior agreement and reflexive-anaphora stimuli to BERT by masking a focus verb in the complete sentence and comparing singular versus plural predictions.
Results
BERT performs remarkably well on all syntactic test cases, including natural, nonce, and manually controlled stimuli.
Takeaways & Limitations
The results suggest that purely attention-based BERT models capture syntactic regularities and perform real syntactic generalization on these tests.
Takeaways & Limitations
BERT results are not directly comparable with prior work because the protocol, discarded stimuli, and training corpus differ.
Abstract
from arXiv · showhide
I assess the extent to which the recently introduced BERT model captures English syntactic phenomena, using (1) naturally-occurring subject-verb agreement stimuli; (2) "coloreless green ideas" subject-verb agreement stimuli, in which content words in natural sentences are randomly replaced with words sharing the same part-of-speech and inflection; and (3) manually crafted stimuli for subject-verb agreement and reflexive anaphora phenomena. The BERT model performs remarkably well on all cases.
1 Introduction
The paper asks whether BERT captures syntax-sensitive structures despite relying on attention rather than recurrent word-order tracking. It evaluates BERT using established agreement and anaphora tests and finds strong performance without task-specific fine-tuning.
- The study assesses whether BERT captures English syntax-sensitive structures beyond its strong benchmark performance.
- The evaluation uses naturally occurring agreement sentences, nonce sentences preserving part-of-speech and inflection, and manually controlled syntactic stimuli.
- BERT’s attention-based architecture lacks an explicit word-order mechanism beyond absolute-position embeddings, unlike recurrent models that track sentence states.
- Out-of-the-box BERT-BASE and BERT-LARGE perform very well across the syntactic tasks without task-specific fine-tuning.
2 Methodology
The methodology adapts prior unidirectional syntactic evaluations to BERT’s bidirectional masked-word prediction setup. It discards stimuli that cannot be fairly or directly evaluated under this protocol.
- Protocol: The protocol adapts stimuli from Linzen et al., Gulordava et al., and Marvin and Linzen, but its results are not strictly comparable to prior numbers.The changes reflect BERT’s bidirectionality and stimulus filtering.
- Prior stimuli: Prior evaluations include natural agreement sentences with attractors, nonce sentences replacing content words by matching part-of-speech and inflection, and controlled grammaticality pairs.
- Adaptation to BERT: The study feeds complete sentences to BERT while masking the single focus verb, then compares predictions for its singular and plural forms.This conditions the verb on both preceding and following context.
- Models: The experiments use pretrained bert-large-uncased and bert-base-uncased models supplied through the PyTorch implementation.
- Discarded material: The evaluation discards two-word-changing NPI stimuli, some is/are copular cases, and focus verbs or inflections absent as single BERT wordpieces.
- Limitations: The bidirectional protocol and discarded stimuli prevent direct comparison with earlier results, while BERT also uses a different, larger corpus containing Wikipedia and books.
3 Results
BERT achieves high scores across the syntactic test cases, generally exceeding earlier reported results, though the comparisons are not direct. Larger BERT is not consistently better than BERT-BASE.
- All syntactic cases receive high BERT scores, usually substantially higher than previously reported results.The author cautions that the datasets, context, training data, and evaluation protocols differ.
- BERT-BASE outperforms BERT-LARGE on many syntactic conditions, showing that larger model size is not necessarily better here.
- The only reported exceptions are the “In a sentential complement” and “Short VP coordination” conditions in Table 3.
4 Discussion
BERT performs remarkably well across all syntactic test cases, including settings designed to reduce reliance on selectional preferences or memorized training examples. The results suggest syntactic generalization and motivate further study of attention-based models.
- BERT performs remarkably well on natural, nonce, and manually controlled syntactic test cases.
- The nonce and manually controlled conditions reduce the possibility that performance relies on selectional-preference cues or memorized Wikipedia examples.
- The findings suggest that purely attention-based BERT models capture hierarchy-sensitive and syntactic dependencies.