Source-linked AI summary
Colorless green recurrent networks dream hierarchically
Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, Marco Baroni
TL;DR
The paper asks whether RNNs trained only for generic language modeling learn abstract hierarchical syntax, rather than exploiting lexical or semantic cues. It evaluates long-distance number agreement across four languages and in meaningless nonce sentences, comparing Italian models with humans. The models perform reliably, exceed strong baselines, and are not far from human performance, supporting the possibility that they form abstract grammatical representations.
Problem
It is unclear whether RNN agreement performance reflects abstract hierarchical syntax or semantic, collocational, and frequency-based cues.
Method
The study evaluates generic language-model-trained RNNs on long-distance number agreement in original and morphologically matched nonce sentences across Italian, English, Hebrew, and Russian, with Italian human judgments.
Results
RNNs perform reliably across four languages and constructions, exceed strong baselines on nonsense sentences, and are not far from human performance.
Takeaways & Limitations
The findings tentatively support the conclusion that language-model-trained RNNs can construct abstract grammatical representations from input without an explicit syntactic bias.
Takeaways & Limitations
The largest original-to-nonce accuracy drop occurred in Hebrew, where alternative construction readings may license incorrect number; this requires more systematic analysis.
Abstract
from arXiv · showhide
Recurrent neural networks (RNNs) have achieved impressive results in a variety of linguistic processing tasks, suggesting that they can induce non-trivial properties of language. We investigate here to what extent RNNs learn to track abstract hierarchical syntactic structure. We test whether RNNs trained with a generic language modeling objective in four languages (Italian, English, Hebrew, Russian) can predict long-distance number agreement in various constructions. We include in our evaluation nonsensical sentences where RNNs cannot rely on semantic or lexical cues ("The colorless green ideas I ate with the chair sleep furiously"), and, for Italian, we compare model performance to human intuitions. Our language-model-trained RNNs make reliable predictions about long-distance agreement, and do not lag much behind human performance. We thus bring support to the hypothesis that RNNs are not just shallow-pattern extractors, but they also acquire deeper grammatical competence.
1 Introduction
The paper asks whether generic language-model training enables RNNs to learn hierarchical syntactic structure rather than relying on shallow lexical or semantic patterns. It tests long-distance agreement across languages and meaningless sentences, finding reliable predictions that approach human performance in Italian.
- RNNs may learn grammatical structure despite not explicitly encoding linguistic hierarchy.Their success in language modeling and syntactic parsing motivates testing whether they track abstract syntax.
- Earlier agreement tests did not exclude semantic, collocational, or frequency-based cues.Lexical representations and typical associations could help models predict agreement without abstract structural knowledge.
- The paper tests grammatical agreement in nonce sentences that are grammatical but meaningless, alongside corpus-extracted examples.Replacing content words with morphologically matched random words reduces potential lexical and semantic confounds.
- The evaluation spans English, Italian, Hebrew, and Russian, multiple agreement constructions, and Italian human judgments.It broadens prior work beyond English subject-verb agreement and adds a human comparison point.
- Generic language-model-trained RNNs solve long-distance agreement reliably, including nonce sentences, with Italian performance not far from humans.The results are consistent across languages, and language-model perplexity predicts agreement accuracy.
- The best RNN outperformed the earlier models replicated from Linzen et al., highlighting the importance of architecture and hyperparameter search.The comparison suggests that strong language-modeling systems can extract syntactic generalizations.
2 Constructing a long-distance agreement benchmark
The benchmark extracts dependency-linked number-agreement constructions and tests whether language models maintain the relation across intervening context. It contrasts original sentences with morphologically matched nonce variants while documenting construction-coverage and representation limits.
- Original treebank sentences are converted into grammatical but nonsensical nonce sentences by replacing content words with random words matching POS and morphology.The procedure generates nine nonce variants per original sentence.
- Long-distance agreement tests whether a model uses a structural cue when arbitrary words intervene before the target.Adjacent agreement can be predicted from local morphosyntactic co-occurrences, so the benchmark requires at least three intervening tokens.
- The benchmark extracts cue-target pairs connected by dependency arcs and represents intervening context with POS-tag sequences.Candidate pairs and their contexts define constructions such as NOUN VERB ADV VERB.
- The construction inventory does not necessarily align with standard syntactic structures.One construction can combine different clause contexts, while a standard structure can be split across constructions.
- The evaluation retains constructions whose cue and target consistently agree in number across treebank instances.This excludes patterns such as verb-object constructions when either number is possible.
- The benchmark contains between two English and 21 Russian constructions per language, with VP-conjunction as the only construction found in all four languages.Subject-verb constructions were unavailable for Russian because flexible word order makes the dependency relation ambiguous.
- Models compare probabilities for singular and plural target forms that match in every other morphological feature.Accuracy is based on whether the model assigns higher probability to the correct number variant given the preceding prefix.
3 Experimental setup
The experiments train RNN language models on large Wikipedia corpora and evaluate them against local and frequency-based baselines on treebank-derived agreement tests. Italian model predictions are additionally compared with native-speaker judgments.
- Training uses Italian, English, and Russian Wikipedia text plus a preprocessed Hebrew Wikipedia corpus, with 90M-token subsets for each language.The corpora are tokenized and filtered for unknown-word rates before model training.
- The study compares simple RNNs and LSTMs trained with two hidden-layer sizes and varied batch sizes, learning rates, and dropout rates.Because LSTMs performed better, the discussion focuses on them and uses LSTM and RNN interchangeably.
- Nonce-sentence evaluation can produce plausible frequent-word probabilities even when neither agreement candidate is the model’s most probable word overall.This is a property of the evaluation setting rather than a comparison of unrestricted sentence probabilities.
- Baselines include a unigram model, a Kneser-Ney-smoothed 5-gram model, and a five-token-window LSTM.These baselines test performance against frequency-based and limited-context alternatives.
- The Italian human experiment presents 119 original and 1071 nonce sentences to native Italian speakers, alongside filler sentences.Participants choose between singular and plural target forms, and responses are filtered for proficiency.
- Results are averaged across the five models with the lowest validation perplexity, with standard deviations reported across those models.Original and nonce rows in Table 1 report percentage accuracy.
4 Results
Across languages and constructions, LSTM language models predicted long-distance agreement reliably, including nonce sentences, with performance often close to humans. Accuracy varied with morphology, ambiguity, attractors, and model configuration.
- Overall performance: LSTM accuracy remained strong on long-distance agreement, with only modest degradation from original to nonce sentences in the evaluated languages.The nonce condition removes many semantic and lexical cues, making syntactic generalization more relevant.
- Model comparison: Agreement accuracy correlated with validation perplexity across 68 LSTM variants per language, with Pearson r ranging from −0.55 in Hebrew to −0.78 in English.All reported correlations were significant at p < 0.001.
- Differences across languages: English was the hardest language, while richer morphology and lower part-of-speech ambiguity were associated with better LSTM accuracy and smaller original–nonce gaps in Italian and Russian.The English V NP conj V construction reached 67.5% accuracy, and English had fewer identifiable morphological agreement patterns.
- Human comparison: 92.5±2.1% versus 92.3%: LSTM accuracy on nonce subject-verb agreement with an embedded clause was comparable to human accuracy.The comparison concerns one of the harder Italian constructions.
- Attractors: Accuracy decreased as the number of attractors increased, with a sharper drop for nonce sentences and an overall model pattern comparable to humans.The analysis included at most two attractors because the corpus contained only two sentences with three or more.
- Model comparison: The authors’ LM-trained LSTM greatly outperformed Linzen et al.’s best LM-trained model and approached their explicitly supervised model.The comparison used Linzen et al.’s attractor test set.
5 Related work
Earlier work showed that RNNs can process hierarchical patterns in artificial and natural-language settings, while later studies probed the linguistic information encoded in their representations. This paper builds on that literature by evaluating syntactic abilities with meaningless nonce sentences.
- Early experiments found that RNNs can handle data generated by context-free and even context-sensitive grammars.These studies used small, controlled artificial languages in which complex hierarchical phenomena were often overrepresented.
- Linzen et al. tested whether RNNs could predict English subject-verb agreement in corpus-extracted sentences requiring hierarchical structure.Their experiments confirmed that RNNs can, in principle, handle such constructions.
- Other research used classifiers to decode morphological and grammatical properties from hidden states of RNN-based translation systems and sentence embeddings.
- The paper's naturally occurring-data approach is most closely related to Linzen et al. and Bernardy and Lappin, alongside related work on RNN linguistic representations.
- Psycholinguistic studies used jabberwocky sentences to probe how humans process syntactic information, while earlier language-model work estimated the probability of “colorless green ideas.”The paper notes that it was not aware of prior evaluations using nonce sentences made of real words to test models or human subjects.
6 Conclusion
The paper evaluates generic language-model-trained RNNs on long-distance agreement across four languages and multiple constructions, including challenging nonsense sentences. Results were consistent across languages, exceeded strong baselines, and approached human performance, supporting abstract grammatical representations while leaving their internal encoding for future work.
- The study analyzed generic language-model-trained RNNs predicting long-distance number agreement across four languages and multiple constructions.
- RNN results were consistent across languages, exceeded strong baselines on nonsense sentences, and were not far from human performance.
- The evaluated constructions were infrequent, reaching an average sentence frequency of only 0.8% even in Hebrew, their most common language.
- The authors tentatively conclude that LM-trained RNNs can construct abstract grammatical representations from raw input without an explicit syntactic bias.
- Future work will investigate information flow across RNN states, other long-distance phenomena, and constructed evaluation sentences.