Source-linked AI summary
Neural Network Acceptability Judgments
Alex Warstadt, Amanpreet Singh, Samuel R. Bowman
TL;DR
The paper asks whether neural networks that perform well on NLU tasks also acquire human-like grammatical competence. It introduces CoLA and trains neural acceptability classifiers, finding improved performance over unsupervised baselines but substantial gaps from humans and limited generalization across phenomena and domains.
Problem
The paper investigates whether neural networks’ success on applied NLU tasks entails knowledge of grammar, as tested through acceptability judgments.
Method
The paper introduces CoLA, a 10,657-sentence expert-labeled corpus, and evaluates recurrent neural network acceptability classifiers against unsupervised baselines.
Results
Neural sequence models outperform Lau et al.’s unsupervised baselines, while LSTM models use word order and the best model performs strongly in-domain and out-of-domain.
Takeaways & Limitations
The models acquire substantial grammatical knowledge, including basic word-order generalizations, but do not yet rival human linguistic competence.
Takeaways & Limitations
Supervised models suffer substantial out-of-domain performance drops, suggesting specialization to training-set phenomena rather than fully general English grammar.
Abstract
from arXiv · showhide
This paper investigates the ability of artificial neural networks to judge the grammatical acceptability of a sentence, with the goal of testing their linguistic competence. We introduce the Corpus of Linguistic Acceptability (CoLA), a set of 10,657 English sentences labeled as grammatical or ungrammatical from published linguistics literature. As baselines, we train several recurrent neural network models on acceptability classification, and find that our models outperform unsupervised models by Lau et al (2016) on CoLA. Error-analysis on specific grammatical phenomena reveals that both Lau et al.'s models and ours learn systematic generalizations like subject-verb-object order. However, all models we test perform far below human level on a wide range of grammatical constructions.
1 Introduction
The paper tests whether neural networks acquire grammatical knowledge by training them on acceptability judgments, introducing CoLA as a large expert-labeled benchmark. Models outperform unsupervised baselines but remain far below humans and show uneven knowledge across grammatical phenomena.
- Motivation: Acceptability judgments provide a behavioral measure of humans’ grammatical knowledge, but strong NLU performance does not by itself establish grammatical competence.The paper therefore evaluates neural networks on distinctions central to linguistic competence rather than only applied NLU tasks.
- Contributions: The paper introduces CoLA, a corpus of over 10k linguistics-literature sentences with expert acceptability labels.CoLA is presented as the largest resource of its kind and is accompanied by baseline code and evaluation resources.
- Contributions: Neural sequence models trained for acceptability classification outperform Lau et al.’s unsupervised models but remain far below human performance.The comparison is designed to test acquisition of grammatical concepts identified as central to human linguistic competence.
- Contributions: The experiments vary training domains and data quantities and test models on specific linguistic phenomena to characterize their acquired grammatical knowledge.The paper reports systematic knowledge of basic subject-verb-object order but no evidence for some non-local dependencies involving agreement and questions.
2 Acceptability Judgments
The paper defines acceptability classification as Boolean sentence judgment grounded in linguistic examples, while emphasizing that the task spans morphological, syntactic, and semantic contrasts. CoLA uses carefully selected literature examples but excludes cases requiring non-grammatical knowledge or unavailable lexical items.
- Task definition: Acceptability classification asks models to distinguish acceptable from unacceptable sentences using Boolean judgments comparable to generative-linguistic practice.The paper notes that other work models gradient judgments, but adopts single-sentence Boolean classification here.
- Data rationale: CoLA consists of linguistics-literature examples whose author-provided labels and minimal contrasting content isolate identifiable grammatical constructions.This construction strategy makes unacceptable examples tend to differ from acceptable ones for a single identifiable reason.
- Exclusions: The dataset excludes pragmatic anomalies, unavailable meanings, prescriptive-rule violations, and nonce words from acceptability classification.Nonce words are excluded because typical word-level models lack them in their vocabularies; other exclusions avoid judgments requiring real-world knowledge or specific interpretations.
- Task definition: The task covers morphological, syntactic, and semantic anomalies, including inflection mismatches, extraction-island violations, and animacy violations.These contrasts are intended to probe challenging grammatical distinctions rather than superficial sentence properties.
3 CoLA
CoLA is assembled from diverse linguistics sources, edited into sentence-level examples, and split into in-domain and out-of-domain data. Its annotations cover many phenomena, though the development set is unevenly distributed and human judgments show substantial but imperfect agreement with the corpus labels.
- Sources and preparation: CoLA draws usable examples from linguistics publications spanning broad topics, time periods, and audiences.Illustrative sources include a syntax textbook, a verb-lexicon reference, and a dissertation on wh-movement and extraction islands.
- Sources and preparation: The corpus preserves original author judgments where possible, filters non-Boolean labels, edits vocabulary and fragments, and expands optional variants into separate examples.The preparation process restricts vocabulary to the 100k most frequent British National Corpus words and converts fragments into sentences.
- Data splits: CoLA separates 8551 in-domain training examples, 527 development examples, and 530 test examples from an additional out-of-domain split.The out-of-domain set uses sources absent from training and is designed to include varied source sizes, specificity, and time periods.
- Phenomenon coverage: Argument alternations appear in over 40% of development examples, while most other annotated constructions occur in about 10–20%, making CoLA fairly balanced but not uniform.The authors caution that alternative annotation schemes could reveal additional dataset biases and that balancing phenomena involves contested design choices.
- Human performance: Human majority judgments agree with CoLA labels on 87% of sampled sentences, with MCC 0.713; 13% of labels contradict the observed majority judgment.The authors attribute disagreement to PDF character-recognition errors, missing context, and dialect variation.
4 Experiments
The experiments compare neural acceptability classifiers using language-model representations, real/fake pretraining, CoLA supervision, and established baselines. The main architecture encodes sentences with bidirectional LSTMs and max-pooling before classification.
- Evaluation: The comparisons include human judgments, but the authors limit grammatical annotation and constrain encoder pretraining data to make model–human comparisons more meaningful.The paper notes that GloVe is trained on substantially more text than humans encounter during language acquisition, making those comparisons less directly comparable.
- Representations: The study evaluates BNC, ELMo-style, and GloVe word representations, with ELMo-style embeddings combining hidden states from an LSTM language model.The LSTM language model is trained on the 100-million-token British National Corpus, while GloVe provides pretrained 300-dimensional embeddings.
- Auxiliary Training: The real/fake auxiliary task distinguishes BNC sentences from language-model samples and word-permuted sentences without explicit grammatical knowledge.The task supplies unlimited automatically generated fake examples and is intended to learn features relevant to downstream acceptability classification.
- Pooling Classifier: CoLA classifiers use a pooling architecture that converts variable-length token sequences into fixed-length sentence embeddings before predicting acceptability.A deep bidirectional LSTM produces forward and backward hidden states, which are concatenated and max-pooled; a sigmoid layer outputs the positive-class probability.
- Experimental Models: The experiments compare CBOW, n-gram and LSTM language-model baselines, real/fake classifiers, CoLA-trained classifiers, and transferred encoders.The pooling-classifier variants differ in whether they use real/fake training, direct CoLA training, or frozen sentence encoders transferred from the auxiliary task.
5 Results and Discussion
CoLA-trained neural models outperform the unsupervised baselines and exploit word order, but their generalization and absolute performance remain substantially below human judgments. Unsupervised pretraining helps, while supervised models lose performance out of domain.
- Overall Results: The best real/fake model with ELMo-style embeddings achieves the highest MCC and accuracy in both domains by a large margin.It also outperforms models using GloVe embeddings.
- Domain Generalization: All real/fake encoders trained on CoLA outperform Lau et al.’s unsupervised models in-domain, while Lau et al.’s baselines rank second out-of-domain.The authors’ models perform worse out-of-domain than in-domain, with MCC falling by as much as 50% in one case.
- Learned Structure: Sequence models consistently outperform the word-order-independent CBOW baseline, indicating that LSTMs use word order non-trivially for acceptability classification.N-gram language-model baselines perform worse than LSTM language models, consistent with their more limited feature windows.
- Human Comparison: LSTMs are the most effective low-bias learners tested, but their absolute performance remains far below human performance.The authors conclude that these networks acquire substantial grammatical knowledge without rivaling human linguistic competence.
- Pretraining: Unsupervised pretraining improves models, although unsupervised systems remain less effective than models trained directly on CoLA.Even so, unsupervised models outperform random guessing and the CBOW baseline, indicating significant acquired grammatical knowledge.
- Limitations: Supervised models show substantial in-domain-to-out-of-domain performance drops, suggesting specialization to training-set phenomena rather than fully general English grammar.The authors identify regularization and improved pretraining strategies as likely ways to address this overfitting.
6 CoLA Design Experiments
The experiments examine how CoLA training-set size and source-based domain splits affect acceptability-classifier performance. More data generally helps, but performance varies substantially across domains and much can be learned from relatively few examples.
- Training Set Size: Increasing CoLA training data from 100 to 8551 sentences produces approximately log-linear performance improvements.The experiment uses random downsampling and 20 restarts with the best-performing ELMo-style real/fake encoder.
- Training Set Size: 300 training examples suffice for models to learn much of what they learn from CoLA.The authors interpret this as evidence that training uses existing grammatical knowledge in the sentence encoders rather than teaching many specific acceptability facts.
- Splitting CoLA: The original source-based split yields an average in-domain MCC 0.125 higher than out-of-domain MCC, near the maximum observed difference of 0.162.The comparison averages performance over 20 restarts.
- Splitting CoLA: Domain differences are generally meaningful for these models, although one source split produces better average out-of-domain performance.The authors conclude that the particular sources in each domain strongly affect what models learn.
7 Phenomenon-Specific Analysis
Phenomenon-specific analyses show that models handle some frequent or coarse grammatical patterns but struggle with several harder constructions. Targeted tests likewise reveal systematic word-order judgments alongside weak performance on most other contrasts.
- CoLA Performance by Phenomenon: Syntactically simple sentences and sentences with adjuncts are easier than average, while binding and violation sentences are among the hardest.Complement clauses, embedded VPs, and argument alternations are approximately as difficult as the average CoLA sentence; question-like syntax is also difficult, likely because of long-distance dependencies.
- Targeted Test Sets: The targeted evaluations use five auxiliary datasets generated with simple rewrite grammars to isolate specific grammatical contrasts.The evaluations are designed to probe which grammatical features models acquire easily.
- Targeted Test Sets: Some models reliably judge gross word order, with the GloVe pooling classifier achieving near-perfect correlation on SVO.The targeted test sets include controlled permutations of subject, verb, and object.
- Targeted Test Sets: The remaining targeted tests yield much poorer performance, although the models consistently outperform Lau et al.’s baselines on the causative-inchoative alternation.This alternation depends on the lexical semantics of particular verbs, such as whether popped but not blew permits the alternation.
- Targeted Test Sets: Lau et al.’s LSTM language-model baseline is the only model that reliably identifies wh-word–gap dependencies, while all models struggle on reflexive examples.The baseline also performs relatively better on agreement judgments.
- Targeted Test Sets: The models’ poor agreement performance may reflect underrepresentation of relevant examples in CoLA, where morphological violations comprise about 6% of examples.The authors contrast this with prior findings that LSTMs can identify agreement errors without sub-word information.
8 Motivation & Related Work
Acceptability classification is motivated as a fine-grained way to evaluate grammatical knowledge in neural networks and to connect computational modeling with questions about linguistic competence. It complements probing tasks by supporting multiple contrasts within one task and direct comparison with human judgments, while retaining important limitations.
- Motivation: Acceptability classification may provide finer-grained evaluation tools for sentence-understanding models and inform questions about whether grammar is learned or innate.These are the paper’s two chief motivations for studying the task.
- Related Work: Probing tasks target individual constructions, whereas acceptability classification can evaluate many unrelated grammatical contrasts with a single classifier.Acceptability judgments can therefore be used to probe constructions such as number agreement and negation scope.
- Related Work: Acceptability classification also enables direct ANN–human comparisons because native speakers can readily perform the same judgment task.This distinguishes it from many probing tasks.
- The Poverty of the Stimulus: Human-level performance by linguistically uninformed models would limit the scope of phenomena requiring the poverty-of-the-stimulus account.The APS predicts failure at human-level acceptability judgments under child-like data and no prior syntactic knowledge.
- Limitations: CoLA’s reading-independent judgments cannot test whether models distinguish role reversals in sentences such as “Bo is easy to please” and “Bo is eager to please.”The limitation concerns semantic interpretation distinctions that depend on who performs or receives an action.
- Limitations: Supervised acceptability-classifier success cannot falsify the APS because unacceptable examples do not apparently participate in child language acquisition.The authors argue that unsupervised acceptability classification would be needed for that stronger test.
9 Conclusion
CoLA provides a large-scale resource for training and evaluating neural networks on acceptability judgments, while baseline systems remain far from human performance. The task retains substantial research potential for testing grammatical knowledge and broader models.
- CoLA enables modern neural networks to be trained and evaluated on acceptability judgments at large scale.The corpus is described as the first large-scale corpus of acceptability judgments.
- The acceptability-classification task remains an underused tool for fine-grained evaluation and testing the Poverty of the Stimulus Argument.
- Future work should evaluate a broader range of effective low-bias models and investigate which grammatical principles they learn.