Source-linked AI summary
Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languages
Shauli Ravfogel, Yoav Goldberg, Tal Linzen
TL;DR
The paper asks how typological properties affect RNNs’ acquisition of syntax while avoiding confounded comparisons across natural languages. It generates synthetic English variants from a parsed corpus and evaluates agreement prediction across controlled changes in word order, agreement, and case. RNNs benefit from joint subject-object prediction and overt case marking, while their word-order performance reveals a recency bias.
Problem
Cross-linguistic evidence about RNN syntactic learning is difficult to interpret because languages differ in corpora, constructions, and multiple typological properties.
Method
The paper generates synthetic versions of English from a parsed treebank, varying agreement patterns, core-element order, and morphological case, then tests RNN agreement-feature prediction.
Results
Joint subject-object prediction improves agreement performance, SVO is easier than SOV in the reported comparison, and overt case marking improves prediction even when highly syncretic.
Takeaways & Limitations
RNN syntactic performance reflects architecture-compatible biases, including recency, while morphological cues and cross-task supervision make agreement learning easier.
Takeaways & Limitations
The word-order manipulation changes only core-element order, not ordering inside noun phrases, adpositional phrases, or other phrase types.
Abstract
from arXiv · showhide
How do typological properties such as word order and morphological case marking affect the ability of neural sequence models to acquire the syntax of a language? Cross-linguistic comparisons of RNNs' syntactic performance (e.g., on subject-verb agreement prediction) are complicated by the fact that any two languages differ in multiple typological properties, as well as by differences in training corpus. We propose a paradigm that addresses these issues: we create synthetic versions of English, which differ from English in one or more typological parameters, and generate corpora for those languages based on a parsed English corpus. We report a series of experiments in which RNNs were trained to predict agreement features for verbs in each of those synthetic languages. Among other findings, (1) performance was higher in subject-verb-object order (as in English) than in subject-object-verb order (as in Japanese), suggesting that RNNs have a recency bias; (2) predicting agreement with both subject and object (polypersonal agreement) improves over predicting each separately, suggesting that underlying syntactic knowledge transfers across the two tasks; and (3) overt morphological case makes agreement prediction significantly easier, regardless of word order.
1 Introduction
The paper asks whether typological properties affect RNNs’ acquisition of syntax, addressing confounded cross-linguistic comparisons with controlled synthetic variations of English.
- Motivation: Cross-linguistic comparisons are difficult because languages differ in training corpora, tested constructions, and multiple typological dimensions simultaneously.These dimensions include morphological richness, word order, and explicit case marking.
- Approach: The proposed paradigm generates parametrically varied synthetic languages from a parsed English corpus, isolating the effects of selected typological properties.The experiments manipulate agreement patterns, core-element order, and morphological case.
- Approach: The study evaluates whether RNNs can predict subject and object plurality from sentences with an omitted verb.The task focuses on agreement features rather than syntactic annotations.
- Experiments: The paper examines polypersonal agreement, alternative subject-object-verb orders, and overt case marking as controlled case studies.Figure 1 illustrates the resulting synthetic-language sentence patterns.
2 Setup
The setup converts an expert-annotated English treebank into controlled synthetic corpora and trains models to predict argument plurality for omitted verbs.
- Synthetic language generation: The authors transform the English Penn Treebank into parametrically modified corpora varying case systems, agreement patterns, and core-element order.The corpus is converted to Universal Dependencies before tree representations are manipulated.
- Argument collection: Agreement examples are collected from parsed verb-argument relations, recording plurality for dependents linked by subject and object relations.Clausal-complement verbs without subjects are excluded from collection.
- Prediction task: Models predict subject plurality, object plurality, or both jointly, using SINGULAR, PLURAL, and, for intransitive objects, NONE categories.Joint prediction therefore treats subject and object agreement as a multitask output.
- Model: The model is a bidirectional LSTM whose contextual representation feeds separate MLP predictors for subject and object plurality.Words combine word embeddings with character n-gram embeddings.
3 Polypersonal agreement
The polypersonal-agreement experiments show that jointly predicting subject and object plurality improves performance, while object prediction is harder in the basic setting.
- Motivation: The task is motivated by the possibility that identifying one argument’s grammatical role disambiguates the other’s role.A constituent serving as a verb’s subject cannot simultaneously be its object.
- Corpus creation: The experiments use suffixes encoding argument features on verbs, with the same suffix inventory marking plurality or case on nouns and agreement on verbs.The paper also notes that indirect-object agreement was omitted because indirect-object recall was very low in preliminary experiments.
- Single task results: In single-task training, subject plurality was predicted more accurately than object plurality, partly because only about one-third of sentences contained direct objects.On transitive-only data, object prediction reversed this pattern, consistent with object heads often being closer to the verb.
- Multitask training: Subject accuracy rose from 94.7% to 95.7% with joint training, while object precision increased from 88.9% to 90.0% and recall from 81.8% to 85.4%.The authors interpret these gains as evidence that supervision transfers across subject and object prediction.
- Multitask training: Joint polypersonal agreement prediction was easier for the model than predicting subject or object agreement separately.The proposed explanation is that both tasks encourage more robust abstract syntactic representations.
4 Order of core elements
The study tests RNN agreement prediction across synthetic languages with all six core-element orders, showing that intervening arguments and missing disambiguating cues shape performance. Results support a recency-based inductive bias, while flexible order creates a generally harder learning problem.
- 4.2 Results: Subject prediction was worst in VOS and SOV, where objects intervene before the verb, consistent with attraction effects and a bias toward recent dependencies.The authors link this pattern to an inductive bias favoring dependencies with recent elements.
- 4.2 Results: SVO produced the highest accuracy among the synthetic languages for object prediction, whereas OSV produced the lowest, and subjects were easier than objects overall.Only 35% of training verbs had an object, compared with all verbs having a subject, likely contributing to the subject–object gap.
- 4.2 Results: Flexible word order yielded 88.6% subject accuracy, 74.1% object accuracy, and 60.2% object recall, because the model lacked consistent word-order or case cues.Some sentences remain genuinely ambiguous without overt case or reliable order, especially when both arguments are animate.
- 4.3 Withholding direct objects in training: 90.6% accuracy in full SOV training fell to 60.0% when transitive sentences were withheld, while VOS fell from 89.5% to 48.3%, strongly supporting an RNN recency bias.The models generalized as if subjects directly preceded verbs, despite training data equally supporting the subject-as-first-core-argument analysis.
- 4.4 Discussion: The manipulation changed only subject, object, and verb order, and mostly fixed that order, making the setting simpler than natural languages with broader phrase-order variation and flexibility.The authors propose testing consistent reordering across additional phrase categories in future work.
- 4.4 Discussion: Difficulty did not clearly track typological frequency: the model performed best on rare OVS order, while SOV was harder than more frequent SVO.The authors interpret this as weak support for functional explanations, without ruling out human learning biases.
5 Overt morphological case systems
The experiments test whether overt case marking improves agreement prediction across flexible and rigid word orders. Unambiguous marking produced the strongest results, while even highly syncretic marking substantially helped.
- Setup: The study compares unambiguous, partially syncretic, and fully syncretic fused suffix systems across flexible, OVS, and VOS orders.
- Results and analysis: Accuracy exceeded 98% for all three word orders with unambiguous case marking, substantially improving subject and object plurality prediction.
- Results and analysis: Flexible word order without case yielded 88.6% subject and 60.2% object accuracy, whereas fully syncretic marking reached 96.0% and 86.1%.
- Results and analysis: Partial syncretism reduced performance relative to unambiguous marking, especially for object prediction, except under flexible word order.
- Results and analysis: Fully syncretic marking remained helpful on rigid orders, indicating that benefits mainly came from marking the heads of all arguments.
6 Related work
The paper builds on work using synthetic or cross-linguistic data to study neural inductive biases. Its approach modifies parsed natural-language corpora parametrically to isolate selected typological properties, trading generality for experimental control.
- Related work: The methodology is inspired by Wang and Eisner’s synthetic-language work but manually modifies parsed corpora to control selected parameters.
- Related work: Simpler synthetic languages have been used to study inductive biases, while Cotterell et al. (2018) examined character-level modeling across languages in a parallel corpus.
7 Conclusions
The paper introduces a controlled methodology for testing RNN grammatical inductive biases through parametric variations of existing languages. Its experiments implicate recency bias, cross-argument syntactic transfer, and overt case marking as major influences on agreement prediction, while some synthetic settings may challenge learners generally.
- Conclusions: The proposed methodology generates parametric language variations from existing languages and evaluates RNN syntactic feature prediction.
- Conclusions: Multitask polypersonal agreement training improved performance, suggesting syntactic representations generalized across subject and object arguments.
- Conclusions: Performance varied across word orders in relation to attractor frequency rather than worldwide word-order frequency, demonstrating a recency bias.
- Conclusions: Overt case marking on argument heads dramatically improved plurality prediction, even when the case system was highly syncretic.
- Limitations: Some synthetic languages with flexible word order and no case may make agreement prediction difficult for both RNNs and humans.