PaperlayerView exact revision on arXiv

Source-linked AI summary

A Systematic Assessment of Syntactic Generalization in Neural Language Models

Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, Roger P. Levy

arXiv:2005.03692v2cs.CL

TL;DR

The paper asks whether perplexity and model properties predict human-like syntactic generalization. It systematically evaluates neural language models on targeted syntactic tests and finds that architecture matters more than training-data scale, while perplexity and syntactic generalization dissociate.

  • Problem

    The paper examines whether improving perplexity predicts more human-like syntactic generalization and how architecture and training-data size affect it.

  • Method

    The authors systematically assess neural language models on 34 targeted syntactic test suites, reporting Syntactic Generalization scores across syntactic circuits.

  • Results

    Model architecture has a larger effect than training-data scale on syntactic generalization, and perplexity substantially dissociates from SG score.

  • Takeaways & Limitations

    Perplexity and syntactic generalization capture complementary features of language-model knowledge, with different architectures showing different syntactic strengths.

  • Takeaways & Limitations

    The evaluation remains an initial step requiring broader syntactic test suites, more varied models and training conditions, and further architectural innovation.

Abstract

from arXiv · show

While state-of-the-art neural network models continue to achieve lower perplexity scores on language modeling benchmarks, it remains unknown whether optimizing for broad-coverage predictive performance leads to human-like syntactic knowledge. Furthermore, existing work has not provided a clear picture about the model properties required to produce proper syntactic generalizations. We present a systematic evaluation of the syntactic knowledge of neural language models, testing 20 combinations of model types and data sizes on a set of 34 English-language syntactic test suites. We find substantial differences in syntactic generalization performance by model architecture, with sequential models underperforming other architectures. Factorially manipulating model architecture and training dataset size (1M--40M words), we find that variability in syntactic generalization performance is substantially greater by architecture than by dataset size for the corpora tested in our experiments. Our results also reveal a dissociation between perplexity and syntactic generalization performance.

1 Introduction

The paper scales targeted syntactic evaluation to compare neural language models across architectures and training-data sizes, asking how syntactic generalization relates to perplexity. It finds architecture matters more than dataset size and that perplexity and syntactic generalization substantially dissociate.

  • Research questions: The study directly examines whether lower perplexity predicts more human-like syntactic generalization.It also asks how architecture and training-data size jointly affect syntactic performance when evaluation spans dozens of controlled tests.
  • Evaluation framework: 34 targeted test suites assess syntactic generalization across a wide range of syntactic phenomena.The suites include 33 adapted from prior work and one novel suite, with performance reported as Syntactic Generalization scores.
  • Experimental design: 20 combinations of model types and data sizes are compared through controlled training on newswire corpora.The controlled datasets contain 1, 5, 14, and 42 million tokens, enabling apples-to-apples comparisons across architectures and sizes.
  • Main findings: Model architecture has a larger effect on Syntactic Generalization scores than training-data size for the tested models and corpora.The results also identify substantial differences in syntactic performance across architectures.
  • Main findings: Perplexity and Syntactic Generalization scores substantially dissociate across the tested architectures and dataset sizes.Thus, broad-coverage predictive performance does not by itself track targeted syntactic performance in these experiments.

2 Background

The paper motivates targeted syntactic evaluation as a complement to perplexity and uses controlled sentence manipulations to test human-like generalizations. These tests can require multi-condition interaction patterns rather than simple probability comparisons.

  • Perplexity: Perplexity measures how well a model predicts words on average, but it may not adequately assess human-like syntactic knowledge.A grammatical sentence can still receive very low probability, motivating finer-grained linguistic tests.
  • Targeted evaluation: Targeted syntactic evaluations minimally vary grammaticality-determining features while holding most lexical and syntactic properties constant.This design adapts methods from psycholinguistic experiments to probe specific syntactic phenomena.
  • Targeted evaluation: Some evaluations test garden-pathing through a 2 × 2 interaction over surprisal across controlled sentence conditions.Successful generalization requires three probability relations, including a smaller C–D than A–B surprisal difference.
  • Targeted evaluation: The controlled tests are used to describe and assess human-like syntactic knowledge in language models.The evaluation framework is intended to cover more complex behavior than simple conditional-probability comparisons.
  • Relation to prior work: This paradigm compares critical sentence regions and uses strict multi-fold success criteria rather than full-sentence probabilities.The design aims to factor out confounds such as individual-token frequency and low-level n-gram statistics.

3 Methods

The study uses controlled syntactic test suites to compare model architecture and training-data size against perplexity and syntactic generalization. It covers multiple syntactic circuits and includes a novel pseudo-cleft evaluation.

  • The evaluation manipulates model class and dataset size while measuring perplexity and syntactic generalization capacity.The design enables controlled comparisons across architectures and training-data scales.
  • Each test suite varies minimally controlled conditions and specifies surprisal inequalities that encode a syntactic prediction.Accuracy is the proportion of items whose model behavior conforms to the prediction.
  • Syntactic coverage: The test collection targets 16 of 47 textbook phenomena, spanning 11 of 15 chapters, or approximately 34% and 73%, respectively.The authors use this inspection to assess syntactic coverage.
  • Five paired modifier suites test whether models remain stable when syntactically irrelevant but semantically plausible content intervenes before the critical region.Inserted material includes prepositional phrases and relative clauses.
  • Syntactic circuits: The suites cover six syntactic circuits, including agreement, licensing, garden paths, gross syntactic expectation, center embedding, and long-distance dependencies.The circuits are grouped by the type of algorithm required to process each construction.
  • Novel test suite: Cleft: The novel Cleft suite tests pseudo-cleft processing, requiring models to distinguish verb-driven VP expectations from lexicalized-verb NP expectations.DO verbs should favor VPs, whereas lexicalized verbs should favor NPs in the critical region.
  • Corpora: Training uses nested BLLIP newswire corpora containing 1M, 5M, 14M, and 42M tokens.The corpora are BLLIP-XS, BLLIP-SM, BLLIP-MD, and BLLIP-LG, respectively.
  • Perplexity is evaluated on a shared held-out test set, with a shared validation set used for tuning and early stopping.This standardizes perplexity evaluation across datasets.

4 Results

Across the tested models, syntactic generalization varies more with architecture than training-data scale, and it is substantially dissociated from perplexity. Circuit-level analyses reveal both broad architectural differences and phenomenon-specific exceptions.

  • Syntactic generalization and perplexity: SG scores are substantially dissociated from perplexity across the tested models and training-set sizes.Targeted syntactic evaluation can therefore reveal information orthogonal to perplexity.
  • Inductive bias and data scale: Model class clearly influences SG score, whereas increasing training data produces only a much more minor mean increase.Figure 3 compares SG-score deltas by model class and training dataset size.
  • Inductive bias and data scale: Both architecture and log training-dataset size significantly predict suite-level SG performance, but architecture improves regression fit more after ablation (AIC=–581 versus AIC=–574).The comparison uses a linear mixed-effects regression model.
  • Inductive bias and data scale: Controlled GPT-2 models achieve roughly the same SG score as pretrained GPT-2 despite using less than 1% of its training data.The result suggests diminishing returns to training-data scale for syntactic generalization.
  • Circuit-level effects on SG score: Circuit-level SG differences vary more by architecture than dataset size, although dataset size matters for Agreement on the smallest corpus.Architecture–circuit interactions are stronger than size–circuit interactions (AIC=–654 and AIC=–623 when size and architecture are ablated).
  • Circuit-level effects on SG score: Architectural inductive biases separate strongly on some circuits but have little effect on Licensing, which may tap a distinct syntactic process.The proposed explanation involves tracking two co-varying tokens when the downstream token is optional.
  • Robustness to intervening content: Model class influences robustness to syntactically irrelevant intervening content, with RNNG more stable than ON-LSTM and GPT-2-XL unaffected near ceiling.The robustness analysis uses five minimally edited test suites.

5 Discussion

The paper finds that perplexity and syntactic generalization capture complementary aspects of language-model knowledge, with architecture more influential than training-data scale in controlled tests. Its broad evaluation remains an initial step toward understanding contemporary models’ syntactic capabilities.

  • Architecture plays a larger role than training-data scale in producing correct syntactic generalizations.
  • Perplexity and syntactic generalization are dissociated, suggesting that they capture complementary features of model knowledge.
  • Circuit-level analysis finds consistent failure on Licensing but inconsistent behavior on other circuits.
  • The paper’s results bear on how much syntactic knowledge can be acquired from string input alone.
  • Targeted evaluations are only first steps toward comprehensive understanding of syntactic capabilities.Further progress requires broader test suites, more varied models, and new architectural innovations.

A Syntactic coverage of test suites

The test suites cover a substantial but incomplete portion of theory-neutral, observable syntactic phenomena. They use controlled condition manipulations and probability-based success criteria to measure model accuracy.

  • Coverage included theory-neutral phenomena referring to observable linguistic data, excluding entries that presuppose a particular syntactic analysis.
  • 16 of 47 textbook phenomena were covered, spanning 11 of 15 surveyed chapters.This corresponds to approximately 34% of phenomena and 73% of chapters; the final two chapters were not assessed.
  • A coverage checkmark indicates that a phenomenon was implicated in model success, not necessarily that the suite explicitly targeted it.
  • Each test-suite item appears in multiple conditions differing only through a controlled manipulation targeting a grammatical feature.
  • A suite’s success criterion specifies inequalities among conditional probabilities of sentence substrings.
  • Suite accuracy is the percentage of items satisfying the criterion, with chance accuracy based on random ordering of condition probabilities.

B.2 Center embedding

The center-embedding suites test whether models track nested subject–verb dependencies and predict verbs in the appropriate order. A modifier version increases the linear distance between dependencies, potentially making the test harder.

  • Center embedding requires models to represent preceding word order and predict upcoming verbs in the appropriate order.
  • The basic suite contrasts correct and incorrect center embeddings using verb transitivity and subject–verb plausibility.
  • The with-modifier version increases the linear distance over which nested dependencies must be tracked.The added relative clause may make this version harder.
  • The criterion requires the correct verb sequence to receive higher probability than the incorrect sequence.For the example item, P(painted deteriorated|The painting that the artist) > P(deteriorated painted|The painting that the artist).
  • Chance performance on these center-embedding suites is 50%.

B.3 Pseudo-clefting

The pseudo-cleft suite tests whether models use a cleft’s syntactic category to anticipate the category of the postcopular predicate. Related filler–gap suites test whether extracted constituents create expectations for gaps in subordinate clauses.

  • B.3 Pseudo-clefting: Pseudo-clefting links an extracted constituent with a matching constituent through the copula, targeting noun phrases and verb phrases.
  • B.3 Pseudo-clefting: The criterion requires category-mismatching postcopular predicates to be more surprising than matching predicates, averaged across noun-phrase and verb-phrase predicates.
  • B.3 Pseudo-clefting: A stricter criterion was rejected because semantic coercion can make a noun-phrase predicate acceptable after a do cleft.
  • B.3 Pseudo-clefting: Filler–gap dependencies arise when constituents such as subjects, objects, or prepositional-phrase elements are extracted from subordinate clauses.
  • B.3 Pseudo-clefting: The extraction suites compare conditional expectations for gaps after that versus wh-complementizers.For obligatorily transitive embedded verbs, omitting a direct object without a filler should be highly surprising.

B.4.4 Tests for unboundedness

These tests examine whether models handle filler–gap dependencies across increasing levels of clausal embedding. Success is evaluated with four-condition suites using the same criterion as the basic object-extraction test.

  • Filler–gap dependencies are unbounded because the filler can be extracted across any number of clausal levels.The harder suites therefore use three or four levels of clausal embedding.
  • Three- and four-level embedding conditions instantiate the harder versions of the object-extracted test suites.The passages provide example sentences for each embedding depth.
  • The longer embedded sentences serve as base sentences for four-condition test suites.These suites retain the manipulations and success criterion of the basic object-extraction test.

B.5 Main-verb/reduced-relative garden-path disambiguation

This task evaluates whether models use reduced-relative and main-clause cues to resolve garden-path ambiguities. Its four conditions manipulate relative-clause reduction and participial-verb ambiguity, with success defined by predicted surprisal effects.

  • The task targets a well-studied syntactic garden-path involving ambiguity between a reduced relative and a main-verb analysis.The item set contrasts sentences such as “The child kicked in the chaos” and relative-clause variants.
  • The four conditions vary whether the relative clause is reduced and whether the participial verb is ambiguous with a simple past-tense form.The critical continuation is the main verb V∗, whose surprisal is compared across conditions.
  • Success requires V∗ to be less surprising after an unreduced relative clause and after an unambiguous participial verb.The interaction between these effects should be smaller for unambiguous participles.
  • Chance performance is below 25%.

B.8 Subject–verb number agreement

These suites test whether models track the subject’s number and reflexive licensing constraints despite intervening distractor noun phrases. They vary distractor placement across prepositional phrases and relative clauses and require condition-sensitive predictions.

  • Subject–verb number agreement: The agreement task tests whether finite present-tense verbs match the subject’s number rather than a distractor noun phrase.The suites use singular and plural subjects with distractors in several postmodifier configurations.
  • Subject–verb number agreement: Distractors occur in prepositional-phrase, subject-extracted relative-clause, and object-extracted relative-clause postmodifiers.The examples contrast grammatical and ungrammatical singular/plural verb forms across these structures.
  • Subject–verb number agreement: Success requires predicting the preferred verbform for both singular- and plural-subject versions of each item.This criterion focuses on discriminating the upcoming verbform of the given lemma.
  • Subject–verb number agreement: Chance performance is 25%, while a context-insensitive baseline that assigns different probabilities to Vsg and Vpl scores 50%.
  • Reflexive licensing: Reflexive suites test whether the model prefers singular or plural reflexives in the licensing context matching the subject.The task also varies distractor placement and the gendered singular reflexive form.

B.10 Subordination

The subordination suites test whether initial subordinators prompt models to expect a following main clause. More difficult versions add increasingly complex postmodifiers to the initial clause.

  • Subordination: Initial subordinators such as As, When, Before, After, and Because signal that the following clause is subordinate rather than the main clause.
  • Subordination: The acceptable continuation places a second clause after the initial subordinate clause.The examples use “After the minister praised the building, it started to rain.”
  • Subordination: Three harder versions lengthen and complicate the initial clause by postmodifying its subject and object.The versions use prepositional phrases, subject-extracted relative clauses, and object-extracted relative clauses.
  • Subordination: The most complex version uses object-extracted relative clauses as postmodifiers of both the subject and object.
  • Subordination: The criterion requires a subordinator to reduce the probability of ending without a main clause and increase the probability of a second clause.The examples contrast incomplete, unexpected, and acceptable continuations.
Loading 2005.03692v2…