Source-linked AI summary
Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIs
Alex Warstadt, Yu Cao, Ioana Grosu, Wei Peng, Hagen Blix, Yining Nie, Anna Alsop, Shikha Bordia, Haokun Liu, Alicia Parrish, Sheng-Fu Wang, Jason Phang, Anhad Mohananey, Phu Mon Htut, Paloma Jeretič, Samuel R. Bowman
TL;DR
It is unclear whether different methods for evaluating models' grammatical knowledge yield similar conclusions. This paper compares five methods on English NPI licensing and finds that BERT shows systematic knowledge, but the methods reveal meaningfully different aspects of that knowledge.
Problem
Although sentence representation models perform language-understanding tasks, substantial direct comparisons of methods for evaluating their grammatical knowledge are lacking.
Source: §1 · p. 1Method
The paper compares five methods for testing BERT's knowledge of English NPI licensing, including acceptability classification, minimal pairs, cloze testing, and probing.
Source: §1 · p. 1Results
BERT shows systematic knowledge of NPI licensing, but the five methods yield meaningfully different results and reveal unequal knowledge across domains.
Source: §1 · p. 2Takeaways & Limitations
In this English NPI domain, using multiple evaluation methods reveals different aspects of a model's grammatical knowledge.
Source: §8 · p. 9Human validation found over-acceptance of any, with 78.3% agreement for any sentences versus 87.3% for ever sentences.
Source: §5 · p. 6
Abstract
from arXiv · showhide
Though state-of-the-art sentence representation models can perform tasks requiring significant knowledge of grammar, it is an open question how best to evaluate their grammatical knowledge. We explore five experimental methods inspired by prior work evaluating pretrained sentence representation models. We use a single linguistic phenomenon, negative polarity item (NPI) licensing in English, as a case study for our experiments. NPIs like "any" are grammatical only if they appear in a licensing environment like negation ("Sue doesn't have any cats" vs. "Sue has any cats"). This phenomenon is challenging because of the variety of NPI licensing environments that exist. We introduce an artificially generated dataset that manipulates key features of NPI licensing for the experiments. We find that BERT has significant knowledge of these features, but its success varies widely across different experimental methods. We conclude that a variety of methods is necessary to reveal all relevant aspects of a model's grammatical knowledge in a given domain.
1 Introduction
Prior work uses diverse methods to evaluate grammatical knowledge, but direct comparisons have not established whether they yield similar conclusions. This paper compares five methods on English NPI licensing and finds that BERT's apparent knowledge varies by method.
1 Introduction
- Researchers have used probing, minimal-pair probability comparisons, and Boolean acceptability judgments, but their conclusions about a model's grammatical knowledge have not been directly compared substantially.It remains unclear whether these methods tend to yield similar conclusions about what a model knows.Source: §1 · p. 1
- NPI acceptability requires both a licensor and the NPI's presence within that licensor's syntactic scope.The paper illustrates the contrast with any: acceptable under negation, unacceptable in positive sentences or outside the negation's scope.Source: §1 · p. 1
- The study compares five methods for evaluating BERT's NPI knowledge: acceptability classification, absolute and gradient minimal pairs, cloze testing, and probing.The methods assess sentence judgments, pairwise acceptability contrasts, masked-token preferences, and grammatical features encoded in representations.Source: §1 · pp. 1–2
- BERT shows systematic knowledge of NPI licensing, but the five methods reveal meaningfully different strengths and weaknesses across licensing environments and relevant features.Gradient minimal pairs, and to a lesser extent acceptability classification and cloze tests, suggest systematic knowledge across environments; absolute minimal pairs and probing reveal unequal knowledge.Source: §1 · p. 2
2 Related Work
Prior work evaluates grammatical knowledge through Boolean acceptability judgments, minimal pairs, and probing, while theories of NPI licensing seek to unify diverse licensing environments. Both evaluation methods and licensing theories leave open questions about what models capture and how the environments relate.
2 Related Work
- Boolean acceptability classification predicts whether a single sentence is acceptable, while minimal-pair methods test sensitivity to one grammatical contrast and probing predicts specific linguistic features.These approaches have been applied to agreement, NPI and reflexive licensing, syntactic and surface features, and other grammatical properties.Source: §2 · p. 2
- A popular account proposes that NPIs are licensed if and only if they occur in downward-entailing environments, where inferences from sets to subsets are licensed.Under negation, an inference from having been to France to having been to Paris is valid; without negation, it is not.Source: §2 · pp. 2–3
- No theory is accepted as unifying all NPI licensing environments, and the downward-entailing account does not capture cases such as questions or the scope of only.Source: §2 · pp. 2–3
- NPI research in computational linguistics reports that LSTM language models do not systematically prefer licensed NPIs, while other work examines licensing context, scope, and structural supervision.Structurally supervised models outperform state-of-the-art sequential LSTMs on non-local dependencies in Wilcox et al. (2019).Source: §2 · p. 3
- CoLA provides over 10,000 syntactically diverse sentences with Boolean acceptability labels and is used to train supervised acceptability classifiers.The corpus draws examples from linguistics literature and spans syntactic, semantic, and morphological phenomena.Source: §2 · p. 3
3 Methods
The experiments evaluate BERT's NPI knowledge using supervised acceptability judgments, minimal-pair evaluations, masked-token preferences, and feature probing. These methods differ in whether they require correct sentence labels, compare relative preferences, or predict linguistic features.
3 Methods
- Boolean acceptability fine-tunes sentence representations to classify sentences as acceptable or unacceptable, measuring performance with Matthews Correlation Coefficient.BERT uses a classifier on the last-layer [CLS] embedding; the BoW model uses max pooling and an MLP.Source: §3 · p. 3
- Absolute minimal-pair evaluation counts a pair correct only when the classifier correctly labels both minimally different sentences.The paired sentences differ in one NPI-related Boolean feature and have different acceptability.Source: §3 · p. 3
- Gradient minimal-pair evaluation counts a pair correct when the acceptable sentence receives a higher acceptable-class output, even if the classifier does not label it correctly.This is a more lenient criterion than absolute minimal-pair evaluation.Source: §3 · p. 3
- The cloze test uses BERT's masked-language-model head to compare acceptable and unacceptable pairs that match in length and differ in one masked token.The pair is correct when the model assigns a higher probability to the acceptable sentence's token; the test uses BERT's unsupervised MLM functionality.Source: §3 · pp. 3–4
- Feature probing trains lightweight classifiers on frozen sentence representations to predict licensor presence, NPI presence, and whether the NPI falls within the licensor's scope.Each metadata label is tested separately and does not by itself determine acceptability.Source: §3 · p. 4
4 Data
The paper constructs a controlled, generated dataset to test NPI acceptability and licensing features across environments. It also validates a sample with native English speakers and evaluates classifiers on a separate set of handcrafted sentences.
4 Data
- The controlled dataset contains 136,000 English sentences spanning nine NPI-licensing environments and two NPIs, any and ever; all but one licensor–NPI pair follows a 2×2×2 paradigm.The paradigm manipulates licensor presence, NPI presence, and whether the NPI occurs within the licensor’s scope.Source: §4 · p. 4
- The dataset substitutes non-licensing words with replacements of similar structural distribution and tracks whether the NPI or replacement lies within the licensor's scope.The scope of a licensor depends on its type; Table 2 illustrates the manipulated features and five minimal pairs in the Questions environment.Source: §4 · p. 4, Table 2 · p. 5
- Simple Questions use a reduced 2×2 paradigm with two minimal pairs and no scope manipulation because the question takes scope over the whole sentence.Source: §4 · p. 4
- Templates and a vocabulary annotated with 30 syntactic, morphological, and semantic features generate sentences while avoiding blatantly implausible combinations.The vocabulary contains over 1,000 lexical items, and each paradigm's sentences are kept within the same data split.Source: §4 · pp. 4–5
- A separate test set contains 104 handcrafted sentences from Wilcox et al. (2019), including a condition where the NPI follows its licensor linearly but is outside its scope.The authors sampled 500 generated sentences for validation; 20 self-identified native English speakers in the US rated each on a 1–6 scale.Source: §4 · pp. 5–6
- 82.8% overall agreement was observed between dataset labels and MTurk validators' acceptability ratings across NPI-licensing environments.Agreement was 81.3% for sentences labelled acceptable and 85.2% for sentences labelled unacceptable; acceptable sentences were rated more acceptable in every environment and for each NPI (all p-values < 0.001).Source: §4 · p. 6
5 Experimental Settings
The experiments compare a GloVe bag-of-words baseline with BERT and two BERT variants fine-tuned on related tasks. They evaluate transfer across NPI environments using several training-evaluation configurations.
Implementation
- Experiments used the jiant 0.9 toolkit, AllenNLP, and HuggingFace’s BERT implementation.Source: §5 · p. 6
Human Ratings
- 82.8% overall agreement was observed between MTurk ratings and dataset labels; agreement was 87.3% for “ever” sentences and 78.3% for “any” sentences.The authors report that “any” sentences tend to be over-accepted.Source: §5 · p. 6
Models
- The model set includes GloVe BoW, cased BERT-large, BERT fine-tuned on MNLI, and BERT fine-tuned on CCG.The paper chose cased BERT-large because it performed best for the tasks in pilot experiments.Source: §5 · p. 6
Training-Evaluation Configurations
- Five configurations test transfer: CoLA training, training and testing on one NPI environment, training on other environments, all-but-one training, and training on all NPI environments.The configurations assess whether models generalize to previously unseen licensing environments.Source: §5 · p. 6
6 Results
Across the experiments, BERT’s performance depended on the evaluation method and the NPI feature being tested. Minimal-pair, acceptability, cloze, and probing results each revealed different strengths and limitations.
Acceptability Judgments
- BERT outperformed BoW across test data and fine-tuning settings, reaching MCC 1.0 on all test data in the 1 NPI setting; All-but-1 training lowered performance, including to 0.58 on simple questions.In the All-but-1 setting, conditionals and sentential negation remained above 0.9 MCC.Source: §6 · pp. 6–7
Absolute and Gradient Minimal Pairs
- NPI-presence minimal pairs yielded (nearly) perfect accuracy for BoW and plain BERT, while licensor and scope pairs were harder on absolute than gradient measures.BERT remained near perfect on gradient licensor and scope measures; absolute judgments were more challenging for scope and for licensor contrasts involving more lexical items and syntactic configurations.Source: Figure 2 · p. 7, §6 · pp. 7–8
Model Variants
- Under Avg Other NPI, plain BERT outperformed BERT→MNLI and BERT→CCG in 7 of 9 NPI environments; in the remaining two, it performed about as well as both variants.Source: §6 · p. 7
Cloze Test
- Without NPI-data supervision, BERT’s cloze accuracy for NPI presence ranged from 0.76 to 0.93, while licensor detection varied by environment.Licensor detection was especially strong for conditionals, sentential negation, and only, and low for quantifier and superlative environments.Source: Figure 3 · p. 8, §6 · p. 8
Feature Probing
- BERT outperformed BoW on scope probing, while BoW was nearly perfect at detecting NPI and licensor presence; scope detection was not lower than licensor detection in minimal-pair experiments.Source: §6 · p. 8
Acceptability Judgments
- CoLA fine-tuning especially improved BERT’s NPI-presence detection, while NPI-data fine-tuning improved scope detection; held-out superlative, quantifier, and adverb environments remained difficult.Source: §6 · p. 8
7 Discussion
BERT shows systematic knowledge of NPI licensing, but its apparent success depends on the evaluation method and feature being tested. Gradient minimal pairs can conceal environment-level differences that other methods reveal.
7 Discussion
- BERT performs nearly perfectly on gradient minimal pairs across configurations and nearly all licensing environments, while other methods reveal a more nuanced picture.Gradient evaluation has ceiling effects that fail to expose differences between licensing environments visible in absolute, cloze, and probing tests.Source: §7 · pp. 8–9
- BERT represents NPI and licensor identity more strongly than licensor scope, where detecting scope requires word-order information rather than co-occurrence alone.BERT without acceptability fine-tuning is near ceiling at detecting licensors but inconsistent at scope detection; BoW also reaches ceiling on licensor presence.Source: §7 · p. 8
- BERT’s systematic acceptability knowledge varies across licensing environments and is not categorical, while human validation also shows evidence of gradience.The discussion contrasts this pattern with generative models’ categorical account of human language knowledge.Source: §7 · p. 9
- Additional CCG and MNLI pretraining does not improve BERT’s performance and lowers it in some cases.The authors note this contrasts with Wilcox et al. (2019), attributing the difference as likely due to their different model and training procedure.Source: §7 · p. 9
- Future evaluations should use varied methods within a domain because generally good performance may hide differences, while poor performance may miss systematic knowledge.The discussion suggests an easier task may reveal systematicity when a model performs poorly on another task.Source: §7 · p. 9
8 Conclusion
The paper concludes that evaluation tasks reveal different aspects of sentence encoders’ knowledge within a defined domain of English grammar. Across several methods, BERT shows systematic NPI licensing knowledge that is unequal across relevant features and does not reflect their Boolean effect on acceptability.
8 Conclusion
- Across evaluation methods, BERT shows systematic NPI licensing knowledge that is unequal across relevant features and does not reflect their Boolean effect on acceptability.Different tasks reveal different aspects of sentence encoders’ knowledge within a well-defined domain of English grammar.Source: §8 · p. 9
Appendix
The appendix captions describe a reduced Simple Questions paradigm and validation results, alongside figures covering acceptability, minimal-pair, and probing evaluations.
Appendix
- Table 3 presents a reduced Simple Questions paradigm with two minimal pairs and no scope manipulation.The caption identifies licensor substitutions in bold, NPI substitutions in italics, and arrows from unacceptable to acceptable sentences.Source: Table 3 · p. 12
- Table 4 reports MTurk validation by licensing environment and intended acceptability label, with percentage accepted and a difference score.Diff is the percentage of acceptable sentences rated acceptable minus the percentage of unacceptable sentences rated acceptable.Source: Table 4 · p. 12
- Figure 5 organizes acceptability-judgment results by evaluation test in columns and fine-tuning setting in rows.Source: Figure 5 · p. 13
- Figure 6 reports minimal-pair results for the NPI-presence contrast across training-evaluation configurations and licensing environments.Smaller sector diagrams show BoW and BERT variants under two minimal-pair evaluation methods.Source: Figure 6 · p. 13
- Figure 7 reports minimal-pair results for the licensor-presence contrast across training-evaluation configurations and licensing environments.Smaller sector diagrams show BoW and BERT variants under two minimal-pair evaluation methods.Source: Figure 7 · p. 14
- Figure 8 reports minimal-pair results for the scope contrast across training-evaluation configurations and licensing environments.Smaller sector diagrams show BoW and BERT variants under two minimal-pair evaluation methods.Source: Figure 8 · p. 14
- Figure 9 reports probing classification results in MCC for NPI presence, licensor presence, and scope detection across nine NPI environments.Letters A–I identify ADV, COND, D-NEG, S-NEG, ONLY, QNT, QUES, SMP-Q, and SUP.Source: Figure 9 · p. 15