Source-linked AI summary
BLiMP: The Benchmark of Linguistic Minimal Pairs for English
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, Samuel R. Bowman
TL;DR
BLiMP addresses limited and fragmented evidence about how language models represent grammatical phenomena across English. It introduces a broad minimal-pair benchmark and finds strong performance on basic agreement but persistent difficulties with islands, quantifiers, NPIs, and argument structure.
Problem
Existing LM evaluations use different metrics and small sets of linguistic paradigms, limiting broad conclusions about models’ knowledge across English phenomena.
Method
BLiMP contains 67 linguistically motivated datasets of 1,000 automatically generated minimal pairs each, and evaluates language models including GPT-2 on acceptability contrasts.
Results
GPT-2 achieves the highest overall BLiMP accuracy, while models robustly encode basic agreement but struggle with islands, quantifier and NPI licensing, and argument structure.
Takeaways & Limitations
BLiMP provides a large-scale tool for tracking what language models learn about grammar and identifying domains where they remain below human performance.
Takeaways & Limitations
The forced-choice BLiMP task cannot support strong conclusions about models’ wh-dependency representations, and factorial designs are better suited to fully assessing syntactic islands.
Abstract
from arXiv · showhide
We introduce The Benchmark of Linguistic Minimal Pairs (shortened to BLiMP), a challenge set for evaluating what language models (LMs) know about major grammatical phenomena in English. BLiMP consists of 67 sub-datasets, each containing 1000 minimal pairs isolating specific contrasts in syntax, morphology, or semantics. The data is automatically generated according to expert-crafted grammars, and aggregate human agreement with the labels is 96.4%. We use it to evaluate n-gram, LSTM, and Transformer (GPT-2 and Transformer-XL) LMs. We find that state-of-the-art models identify morphological contrasts reliably, but they struggle with semantic restrictions on the distribution of quantifiers and negative polarity items and subtle syntactic phenomena such as extraction islands.
1 Introduction
BLiMP addresses the limited and fragmented evaluation of linguistic knowledge in language models by providing a broad benchmark of controlled English minimal pairs. The paper uses BLiMP to compare models and finds uneven learning across grammatical phenomena.
- Benchmark and motivation: Prior studies used different metrics and covered few linguistic paradigms, limiting broad conclusions about models’ English grammar knowledge.BLiMP is designed to support more systematic, fine-grained comparisons.
- Benchmark and motivation: BLiMP contains 67 automatically generated datasets, each with 1,000 minimal pairs organized into 12 linguistic categories.The pairs target acceptability contrasts across previously studied and novel English phenomena.
- Evaluation: The evaluation tests whether n-gram, LSTM, and Transformer models assign higher probability to the acceptable sentence in each pair.This probability comparison provides indirect evidence about linguistic knowledge and enables fine-grained model comparisons.
- Evaluation: Training-size experiments show steady overall improvement, but models learn phenomenon-specific distinctions at different rates.Alternative evaluation metrics produce results that do not differ drastically from full-sentence probability comparisons.
- Findings: The results indicate that model knowledge is concentrated in specific grammar domains, leaving several phenomena where models lack human-like knowledge.BLiMP is intended to focus future evaluation on these weaker areas.
2 Background & Related Work
The paper situates BLiMP between traditional language-model metrics, representation probing, and supervised acceptability classification. It motivates unsupervised minimal-pair evaluation as a way to test phenomenon-specific grammatical sensitivity without labeled training data.
- Evaluation gap: Perplexity on large language-model benchmarks does not provide detailed insight into models’ grammatical knowledge.Language modeling has shifted from local n-gram statistics to neural and Transformer sequence models, while perplexity remains a primary metric.
- Evaluation approaches: Representation-probing studies train classifiers to predict grammatical properties from neural representations, whereas this paper uses acceptability judgments.The approach avoids requiring grammatical-concept labels for evaluation.
- Prior minimal-pair work: Earlier minimal-pair studies examined specific contrasts but covered a limited set of phenomena, excluding areas such as control, raising, ellipsis, and quantification.BLiMP expands coverage while retaining controlled grammatical contrasts.
- Supervised acceptability evaluation: CoLA offers broad acceptability classification, but its evaluation requires supervised training that confounds encoder knowledge with knowledge learned during acceptability training.This limits strong conclusions about the sentence encoder itself.
- Minimal-pair evaluation: Minimal-pair evaluation avoids supervised classification, provided that factors such as sentence length and lexical content are controlled.BLiMP is designed around this controlled-comparison requirement.
3 Data
BLiMP is a controlled, template-generated benchmark of English minimal pairs spanning 67 paradigms and 12 phenomena. Its design emphasizes lexical and structural variation, human validation, and explicit coverage boundaries.
- Dataset scope: BLiMP contains 67 minimal-pair paradigms with 1,000 sentence pairs each, grouped into 12 English phenomena.The benchmark uses paradigms for minimal-pair types and phenomena for broader categories.
- Data generation: Linguist-crafted grammar templates automatically generate controlled pairs by sampling lexical items from a vocabulary of over 3,000 items.The procedure holds the intended contrast constant while allowing varied lexical and syntactic realizations.
- Generation limits: Generated examples can occasionally be implausible, but paired sentences share that world-knowledge implausibility so probability differences remain attributable to the intended contrast.This is an author-supported limitation of automatic generation.
- Design constraints: The paradigms target morphology, syntax, and semantics and require equal-length pairs differing in at most one vocabulary item.Some implementation choices make the operational coverage narrower than the corresponding linguistic definitions.
- Coverage limits: The benchmark has broad but non-exhaustive coverage because not every English grammatical phenomenon can be included or represented by these templates.The implementation also relies on some dialect-specific contrasts and stereotyped gender-name pairings.
- Lexical and syntactic variation: BLiMP includes verbs with 11 subcategorization frames and more lexical variation than related generated datasets.Its vocabulary and syntactic variety are intended to broaden empirical coverage while preserving control.
- Human validation: Human validation uses 6,700 judgments from US-based self-reported native English speakers rating five pairs from each paradigm.The validation process includes attention checks and hidden field questions.
4 Models
The study evaluates BLiMP with pretrained Transformer, LSTM, and n-gram language models. The main Transformer model is GPT-2-large, while additional models vary architecture and training data.
- GPT-2: GPT-2-large uses a 36-layer Transformer architecture with 774M parameters and is pretrained on approximately 8B WebText tokens.The paper evaluates GPT-2 through the jiant codebase.
- Transformer-XL: Transformer-XL Large has 18 Transformer-decoder layers and 16 attention heads per layer, trained on 103M WikiText-103 tokens.Its BLiMP evaluation is also implemented in jiant.
- LSTM: The pretrained LSTM language model is trained on an 83M-token English Wikipedia corpus.Additional LSTM and Transformer-XL models are retrained at smaller data sizes for training-size analyses.
- 5-gram baseline: The 5-gram language model is trained on the 3.1B-token English Gigaword corpus.It provides a non-neural baseline for BLiMP evaluation.
5 Results & Discussion
BLiMP reveals that language models’ grammatical knowledge is uneven: they handle basic agreement and morphology relatively well but struggle with islands, semantic licensing, and argument structure. Performance differences are difficult to attribute cleanly because pretrained models vary in architecture and training data.
- Overall performance: GPT-2 achieves the highest BLiMP accuracy, while the 5-gram model performs worst overall and across every category.The 5-gram results suggest BLiMP is unlikely to be solvable from local co-occurrence statistics alone.
- Performance factors: Because models differ in architecture and training data, the authors can only speculate about performance drivers; training-data access may be the main factor for the neural models evaluated.Transformer-XL and the LSTM perform similarly despite architectural differences, while GPT-2 is trained on roughly two orders of magnitude more data.
- Agreement and morphology: Models generally perform best on morphological phenomena, with GPT-2 within 5 points of humans on anaphor agreement.
- Syntactic phenomena: Islands are hardest for most models: GPT-2 performs above chance but remains 12 points below humans, while models otherwise represent long-distance wh-dependencies.
- Agreement and morphology: Current state-of-the-art models robustly encode basic English agreement, although longer dependencies and agreement attractors reduce performance.
- Semantic phenomena: Models struggle with negative polarity items and quantifiers, whose violations often involve semantic composition or pragmatic reasoning.
- Semantic and lexical phenomena: Weak argument-structure performance may reflect difficult event structure, verb coercion, and alternate verb meanings, despite humans detecting the intended contrast 90% of the time.
- Syntactic phenomena: All models succeed more often when an unacceptable sentence lacks a necessary gap than when it contains an illicit gap, so single paradigms provide incomplete evidence about grammatical knowledge.
6 Analysis
BLiMP analysis shows that model performance depends on agreement locality, training data, model scale, and evaluation method, with substantial variation across grammatical phenomena.
- Agreement locality: The n-gram model loses 35 points on determiner-noun agreement when an adjective intervenes, while neural models lose 3–5 points.The n-gram effect is attributed to the greater frequency of determiner-noun bigrams than determiner-adjective-noun trigrams.
- Agreement locality: GPT-2 and LSTM performance drops 22 and 20 points with agreement attractors, compared with a 5-point reduction for Transformer-XL.All neural models remain above chance when an attractor is present.
- Regular vs. irregular agreement: GPT-2 and Transformer-XL perform less than 1.3 points worse on irregular than regular agreement paradigms.Their high overall performance suggests number features are encoded without relying on segmental cues.
- Training size and BLiMP performance: Different phenomena have distinct learning curves despite equivalent full-model accuracy, and harder phenomena improve more slowly with training.The shallowest reported slopes are for NPIs and islands, while agreement phenomena have steeper slopes.
- Training size and BLiMP performance: GPT-2 models from 117M to 1558M parameters all achieve 0.84±.01% overall BLiMP accuracy, suggesting model size is unlikely to improve performance.The authors conclude that training-data quantity has the largest impact among the factors examined.
- Evaluation methods: Full-sentence and prefix evaluation methods generally yield comparable accuracies, although some model-phenomenon combinations differ across methods.The authors retain the full-sentence method because it supports paradigms incompatible with prefix methods.
7 Conclusion & Future Work
BLiMP provides both overall and phenomenon-specific measures of language-model sensitivity to grammatical contrasts. Its results identify concentrated knowledge and persistent gaps that motivate more targeted evaluation of language models.
- Benchmark utility: BLiMP combines an overall score with a breakdown by linguistic phenomenon, enabling fine-grained conclusions about learned grammatical features.The phenomenon-level analysis complements common metrics such as perplexity.
- Scientific significance: BLiMP provides indirect evidence relevant to debates about whether self-supervised learners acquire human-like, phenomenon-specific grammatical acuity.The benchmark evaluates whether language models acquire human-like grammatical knowledge in particular domains.
- Future work: Future work should investigate domains where models acquire some relevant knowledge but remain below human performance, including argument structure.The authors also identify filler-gap dependencies, negative polarity items, and binding as areas warranting further study.
Appendix
The appendix documents BLiMP’s examples, caveats, evaluation conventions, and complete paradigm table. It clarifies that the benchmark targets specific English contrasts and reports both model and human accuracy.
- Appendix contents: BLiMP’s appendix provides examples from all 67 paradigms and a table listing model accuracy and estimated human agreement.The paradigms cover established contrasts in English morphology, syntax, and semantics.
- Caveats: Some anaphor-agreement and binding paradigms rely on stereotyped gender associations attached to names.For example, the benchmark marks gender-incongruent reflexive sentences as unacceptable and excludes singular they variants.
- Caveats: BLiMP uses mainstream US and UK English judgments, which may penalize models conforming to other English dialects.The appendix gives dialect-sensitive judgments such as rejecting “Suzy don’t lie.”
- Evaluation conventions: Model and human performance are reported as percent accuracy, while prefix-method columns indicate whether each pair supports one- or two-prefix evaluation.All sentences support the simple language-model method.