Source-linked AI summary
Targeted Syntactic Evaluation of Language Models
Rebecca Marvin, Tal Linzen
TL;DR
Perplexity conflates multiple sources of language-model success and is especially difficult for measuring syntactic predictions. The paper introduces targeted minimal-pair evaluation and finds that syntactic multitask training helps recurrent models, but substantial weaknesses remain on complex dependencies.
Problem
Perplexity conflates collocational, semantic, pragmatic, and syntactic sources of success, making syntactic predictions difficult to measure.
Method
The paper supplements perplexity with minimally different grammatical and ungrammatical sentence pairs, testing whether models assign higher probability to the grammatical sentence.
Results
Multi-task training with CCG supertagging improved recurrent-model performance on some non-local dependencies, but performance still degraded markedly and remained weaker than humans.
Takeaways & Limitations
Targeted minimal-pair evaluation is a viable strategy for exposing syntactic weaknesses that standard language-model evaluation can obscure.
Takeaways & Limitations
NPI contrasts differed in syntactic structure and low-level n-gram probabilities, making performance on that contrast harder to interpret.
Abstract
from arXiv · showhide
We present a dataset for evaluating the grammaticality of the predictions of a language model. We automatically construct a large number of minimally different pairs of English sentences, each consisting of a grammatical and an ungrammatical sentence. The sentence pairs represent different variations of structure-sensitive phenomena: subject-verb agreement, reflexive anaphora and negative polarity items. We expect a language model to assign a higher probability to the grammatical sentence than the ungrammatical one. In an experiment using this data set, an LSTM language model performed poorly on many of the constructions. Multi-task training with a syntactic objective (CCG supertagging) improved the LSTM's accuracy, but a large gap remained between its performance and the accuracy of human participants recruited online. This suggests that there is considerable room for improvement over LSTMs in capturing syntax in a language model.
1 Introduction
The paper argues that perplexity poorly isolates syntactic competence and introduces targeted minimal-pair evaluation to test grammatical predictions directly. Using this dataset, LSTMs struggled with complex constructions, while syntactic multitask training helped but did not reach human performance.
- Perplexity conflates collocational, semantic, pragmatic, and syntactic sources of next-word prediction success.
- The proposed metric tests whether a language model assigns higher probability to the grammatical member of a minimally different sentence pair.
- The dataset uses automatically constructed examples spanning subject-verb agreement, reflexive anaphora, and negative polarity items.
- The evaluation compares an n-gram baseline, two RNN language models, and human judgments using the same materials.
- Multitask syntactic training improved RNN performance, but the models remained weaker than humans and struggled on complex constructions.
2 Overview of the approach
The approach evaluates complete-sentence grammaticality through controlled minimal pairs, using templates to generate a large balanced dataset across hierarchical phenomena. This general setting avoids limitations of single-word prediction and naturally occurring corpus examples.
- Single-word prediction requires the ungrammaticality locus to be one word and the relevant information to occur in its left context.
- Negative polarity items demonstrate why complete-sentence comparison is necessary: an earlier quantifier’s acceptability depends on a later negative-polarity item.
- The evaluation compares probabilities of complete grammatical and ungrammatical sentences when grammaticality cannot be localized to one word.
- Template construction provides experimental control, balanced phenomenon coverage, and fewer corpus-based confounds than naturally occurring examples.
- Templates automatically generate approximately 350,000 English sentence pairs covering subject-verb agreement, reflexive anaphora, and negative polarity items.
- The dataset aims to isolate syntactic capability by minimizing semantic and collocational cues while avoiding very implausible word combinations.
3 Related work
Prior work evaluated syntax through challenging corpus sentences, acceptability judgments, and syntax-integrated language models, but lacked a broad targeted dataset for language-model evaluation.
- No existing dataset targeted a range of syntactic constructions specifically for language model evaluation.
- Earlier studies also evaluated graded human acceptability judgments and used forced-choice methods in acceptability experiments.
- Researchers have proposed injecting explicit syntax into language models because recurrent models can still benefit from syntactic information for long-distance dependencies.
4 Data set composition
The dataset covers controlled minimal pairs testing agreement, reflexive binding, and negative-polarity licensing across increasingly hierarchical sentence structures. Its conditions manipulate attractors, clause structure, relativizers, animacy, and lexical number.
- Subject-verb agreement: Agreement conditions test whether verbs match the correct subject rather than a linearly nearby or structurally embedded noun.
- Subject-verb agreement: The dataset includes agreement across prepositional phrases, subject relative clauses, sentential complements, and object relative clauses.
- Subject-verb agreement: The materials vary noun-number configurations, omit relativizers, manipulate animacy, and avoid introducing two agreement errors simultaneously.
- Subject-verb agreement: Coordination conditions test whether both verbs agree with the subject, including longer dependencies that reduce reliance on local trigrams.
- Subject-verb agreement: Object-relative conditions require distinguishing an embedded subject from the main subject when predicting either the main or embedded verb.
- Reflexive anaphora: Reflexive conditions test number and gender agreement plus structural binding constraints within clauses and across embedded relative clauses.
- Negative polarity items: Negative polarity items are tested under structural negation, where a negative noun phrase must c-command the item.
5 Experimental setup
The experiments compare language models with increasing levels of syntactic sophistication, including models trained with and without CCG annotations, and evaluate them against human judgments.
- Three language models were trained with increasing syntactic sophistication on a 90-million-word Wikipedia corpus.The models were an n-gram model, an unannotated LSTM language model, and an LSTM additionally trained with CCG supertags.
- The n-gram baseline was trained as a 5-gram model using Kneser-Ney smoothing.
- The single-task RNN used a two-layer LSTM architecture with 650 units per layer and was trained for 40 epochs.
- The multi-task RNN combined language modeling and CCG supertagging by summing their objective functions with equal weights.Supertagging was used as a scaffold to induce better syntactic representations rather than as a competitive standalone task.
- Human participants judged which sentence in each minimal pair was more acceptable, using the same materials as the language models.The experiment was conducted on Amazon Mechanical Turk and was designed to mirror the LM task.
6 Results
The models handled simpler and local dependencies relatively well but struggled with non-local agreement, reflexive anaphora, and negative polarity items. Performance varied substantially by construction and lexical item, with syntactic and frequency-related factors complicating interpretation.
- Agreement: 79% n-gram accuracy on simple agreement and sentential-complement agreement reflected local-context performance rather than robust syntactic generalization.Unseen subject–verb combinations caused the n-gram model to fall back to unigram probabilities.
- Agreement: 94% single-task and 100% multi-task RNN accuracy on simple agreement exceeded the n-gram model, while both reached 90% on short VP coordination.Sentential-complement agreement was also high: 99% for the single-task RNN and 93% for the multi-task RNN.
- Agreement: Non-local agreement reduced RNN performance, although multitask learning raised long-VP-coordination accuracy from 61% to 81%; human accuracies ranged from 82% to 88%.Both RNNs remained poor on agreement across an object relative clause, especially without the overt relativizer that.
- Reflexive anaphora: 83% RNN accuracy on simple reflexives was below the 94% simple-agreement result, with multi-task accuracy at 100% for himself and themselves but 49% for herself.Accuracy across a relative clause was 55%, compared with 87% for humans.
- Negative polarity items: Around 40% single-task RNN accuracy on NPIs improved to 48% on simple NPIs and 73% across a relative clause with multi-task training.Word-level probabilities suggested that non-grammatical collocational factors could outweigh the model’s NPI-sensitive signal.
- Perplexity and lexical variation: Perplexity was 157.5 for the n-gram model, 78.65 for the single-task RNN, and 61.10 for the multi-task RNN, tracking syntactic-data accuracy rather than dissociating from it.Lexical variation was substantial: single-task RNN agreement accuracy ranged from 100% for is and are to 60% for swims, despite 94% average accuracy.
7 Case study: agreement and object relative clauses
The finer-grained analysis shows that agreement across object relative clauses is especially difficult for RNN language models, while agreement within object relative clauses is somewhat better but still reveals model-specific heuristics.
- Agreement across an object relative clause: Both RNNs performed poorly on agreement across an object relative clause, often agreeing with the irrelevant embedded subject instead of the correct main subject.The multitask model was more symmetric on matching-number cases, whereas the single-task model often preferred a singular verb even when both subjects were plural.
- Agreement across an object relative clause: The single-task RNN performed poorly even when both subjects were plural, often preferring a singular verb.This default singular preference matches behavior reported for younger children.
- Agreement within an object relative clause: Agreement within an object relative clause was better, but the single-task model made the most errors when both subjects were singular.The single-task model may have treated multiple subjects as a cue for plural verbs, while the multitask model showed a general bias toward singular subjects.
- Word-level probabilities: Despite poor object-relative-clause performance, the single-task RNN typically assigned higher probability to the grammatical word in minimal pairs.This indicates that aggregate accuracy and word-level probability behavior can diverge in these cases.
8 Discussion
The discussion presents targeted syntactic evaluation as a useful test of language-model grammar, while showing that RNN performance is strong locally but degrades on non-local dependencies and remains below human performance.
- Evaluation strategy: The dataset evaluates grammatical knowledge by pairing sentences that differ only in grammaticality and testing whether the model prefers the grammatical sentence.This provides the paper’s operational criterion for capturing relevant aspects of grammar.
- Local dependencies: An RNN language model achieved 94% accuracy on local subject-verb agreement, significantly outperforming an n-gram baseline.The result supports the viability of the targeted evaluation strategy for distinguishing syntactic behavior.
- Non-local dependencies: RNN performance degraded markedly on non-local dependencies, approaching chance levels on agreement across an object relative clause.Multi-task training with CCG supertagging mitigated this drop for some, but not all, tested dependencies.
- Comparison with prior work: The authors contrast their results with Gulordava et al.’s 81% accuracy on corpus sentences and 74% on constructed sentences.They suggest their sentences were likely more syntactically challenging than examples available in the smaller manually annotated treebank.
- Limitations: The NPI contrasts were harder to interpret because their sentence pairs differed in both syntactic structure and low-level n-gram probabilities.This is identified as a limitation of the approach’s minimal grammaticality contrasts.
- Conclusion: LSTM language models remained far from matching naive annotators’ performance, so the paper does not advocate LSTMs as effective syntax-modeling architectures.The dataset is intended to support measuring progress and understanding the syntactic generalizations captured by language models.
3. Other quantifiers: most, many, no, few
The dataset uses controlled sentence templates to test hierarchical dependencies through subject-verb agreement, reflexive anaphora, and negative polarity items across varied structural configurations.
- Lexical and number variation: The materials vary main and embedded subjects across animate and inanimate noun classes, including singular and plural configurations.Examples include authors, movies, security guards, and chefs.
- Subject-verb agreement: Subject-verb agreement examples cover local contexts, sentential complements, prepositional phrases, subject relative clauses, and object relative clauses.The object-relative conditions distinguish agreement across the clause from agreement within the clause, with and without “that.”
- Subject-verb agreement: The agreement materials manipulate whether main and embedded subjects are singular or plural and whether they are animate or inanimate.These combinations are represented in examples across prepositional phrases and relative clauses.
- Reflexive anaphora: Reflexive anaphora examples test simple dependencies, sentential complements, and dependencies across relative clauses.The contrasts include singular and plural antecedent configurations with himself, themselves, and related forms.
- Negative polarity items: Negative polarity item examples use quantifiers such as no and most with ever in simple, future-tense, and relative-clause constructions.The relative-clause examples include grammatical and intrusive configurations.