Source-linked AI summary
What do RNN Language Models Learn about Filler-Gap Dependencies?
Ethan Wilcox, Roger Levy, Takashi Morita, Richard Futrell
TL;DR
The paper asks what syntactic generalizations RNN language models learn, focusing on whether they represent long-distance filler–gap dependencies and constraints. Using experimentally controlled sentences and two pre-existing LSTMs, it finds licensing effects across multiple argument positions and evidence for several island constraints, although both models fail on some conditions.
Problem
The representations and syntactic generalizations learned by RNN language models are not properly understood, including their ability to learn filler–gap dependencies involving absent material.
Method
The study evaluates two pre-existing LSTM language models on controlled sentences, using wh-licensing interactions to test filler–gap dependencies and their sensitivity to syntactic islands.
Results
The models represent filler–gap dependencies in subject, object, and indirect-object positions and learn constraints involving wh-islands, adjunct islands, and complex NP islands, but fail on some island conditions.
Takeaways & Limitations
Generic sequence models can learn these dependencies and some of their constraints without an obvious inductive bias for hierarchical structures.
Takeaways & Limitations
The method tests probabilistic dependencies rather than grammaticality judgments, whose interpretation would require auxiliary assumptions.
Abstract
from arXiv · showhide
RNN language models have achieved state-of-the-art perplexity results and have proven useful in a suite of NLP tasks, but it is as yet unclear what syntactic generalizations they learn. Here we investigate whether state-of-the-art RNN language models represent long-distance filler-gap dependencies and constraints on them. Examining RNN behavior on experimentally controlled sentences designed to expose filler-gap dependencies, we show that RNNs can represent the relationship in multiple syntactic positions and over large spans of text. Furthermore, we show that RNNs learn a subset of the known restrictions on filler-gap dependencies, known as island constraints: RNNs show evidence for wh-islands, adjunct islands, and complex NP islands. These studies demonstrates that state-of-the-art RNN models are able to learn and generalize about empty syntactic positions.
1 Introduction
The paper asks what syntactic generalizations LSTM language models learn, focusing on filler–gap dependencies and their constraints. It extends controlled-sentence analyses to an unexplored dependency involving empty syntactic positions.
- Motivation: LSTM language models achieve strong results across NLP tasks, but the representations they learn remain insufficiently understood.The authors frame this as an engineering, accountability, and theoretical concern as these models become more widely deployed.
- Research focus: The study uses specially controlled sentences to investigate whether models represent syntactic dependencies.Earlier work found learned subject–verb number agreement and English auxiliary inversion; this paper examines filler–gap dependencies.
- Research focus: A filler–gap dependency links a wh-complementizer such as “what” or “who” to an empty syntactic position licensed by that filler.For example, “what” licenses the missing object after “devoured”; without the filler, the gap is ungrammatical.
- Scope: The paper treats filler–gap dependencies as licensing relationships rather than semantic relationships between fillers and gaps.The authors explicitly study whether language models represent licensing, while setting aside the semantic direct-object relation.
- Motivation: Filler–gap dependencies are challenging because they involve absent material and are subject to numerous complex island constraints.Their complexity has also featured in arguments about whether children require substantial innate knowledge to learn language.
- Research focus: The experiments examine basic dependencies across multiple syntactic positions and test sensitivity to wh-, adjunct, complex NP, and subject islands.The authors report sensitivity to some, but not all, of these constraints.
2 Methods
The authors evaluate two pre-existing LSTM language models with controlled 2 × 2 sentences and surprisal-based interactions. They test whether wh-licensors reduce gap surprisal locally and across embedded clauses, without interpreting model probabilities as grammaticality judgments.
- Language models: The study analyzes two pre-existing LSTMs trained on English language-modeling objectives, with an n-gram model as baseline.The Google model uses the One Billion Word Benchmark and character-CNN inputs; the Gulordava model uses 90 million English Wikipedia tokens.
- Dependent variable: Surprisal is the log inverse probability assigned by the RNN to each current word or character.The probability comes from the softmax activation, and base-2 logarithms measure surprisal in bits.
- Experimental design: The experiment detects filler–gap learning through a 2 × 2 interaction between wh-licensor presence and gap presence.This wh-licensing interaction measures how much a wh-licensor reduces the surprisal associated with a gap.
- Experimental design: Items place gaps in obligatory argument positions inside complement clauses and manipulate wh-licensor and gap presence across four conditions.The paradigm avoids do-support and main-verb tense manipulation, increasing similarity across conditions.
- Dependent variable: Surprisal is measured immediately after filled gaps and across the embedded clause to capture local and global licensing effects.The local measure targets gap-related expectations, whereas the whole-region measure captures changed expectations about overall sentence well-formedness.
- Predictions: The predicted licensing pattern is higher surprisal for a wh-licensor without a gap and for a gap without a wh-licensor.Together, these contrasts motivate a positive interaction when the filler and gap jointly reduce surprisal superadditively.
- Statistical analysis: The analysis uses mixed-effects linear regression with item random intercepts, and figures report 95% confidence intervals for condition contrasts.Random slopes are omitted because items do not have repeated observations within conditions.
- Scope: The method tests probabilistic links between fillers and gaps, not whether model outputs classify sentences as grammatical.Interpreting model probability patterns as grammaticality judgments would require auxiliary assumptions the authors do not pursue.
3 Representation of filler–gap dependencies
RNNs learn flexible, long-distance filler–gap dependencies across syntactic positions and maintain them despite intervening material. They also learn the one-to-one pairing between a wh-phrase and its licensed gap.
- 3.1 Flexibility of Wh-Licensing: RNNs show wh-licensing interactions for gaps in subject, object, and indirect-object or PP positions.The experiments compare three syntactic positions using controlled sentence variants.
- 3.1 Flexibility of Wh-Licensing: The licensing interaction spikes immediately after the gap and returns near zero across the remainder of the sentence.The spike height corresponds to the interaction in the position-based analysis, while its area corresponds to the summed interaction.
- 3.2 Robustness of Wh-Licensing to Intervening Material: Intervening material does not consistently attenuate licensing interactions: the Google model showed no significant length effect, while the Gulordava model showed only tiny mixed effects.For Gulordava, significant correlations appeared only in post-gap material for object and goal positions, with β = 0.0289 and β = 0.0047.
- 3.2 Robustness of Wh-Licensing to Intervening Material: Across eight experiments, 86.4% of licensing-interaction measurements were positive, whereas the n-gram baseline produced exactly zero interactions.Positive interactions indicate that filler and gap together reduced surprisal superadditively.
- 3.3 Multiple Gaps: When an upstream subject gap was added, both LSTMs showed significantly lower licensing effects, supporting a one-to-one expectation for typical object gaps.The reduction was significant across all conditions, with p < 0.001.
4 Syntactic islands
The experiments test whether LSTM language models learn island constraints by comparing wh-licensing interactions inside and outside syntactic islands. The models reduced licensing interactions for wh-islands, adjunct islands, and most complex NP islands, but did not clearly learn subject islands.
- 4.1 Wh-Island Constraint: Wh-island extractions produced significantly lower wh-licensing interactions than null-complementizer extractions in both LSTMs.The Google model also differed significantly from that-headed clauses, while the Gulordava model showed a gradient pattern treating that-headed clauses as semi-islands.
- 4.2 Adjunct Island Constraint: Adjunct-island extractions significantly reduced wh-licensing interactions relative to object extractions in both post-gap and whole-clause measurements.The Google model showed significant reductions across both adjunct positions, and the Gulordava model showed the same pattern.
- 4.3 Complex NP and Subject Islands: Licensing interactions shrank inside complex NPs in both subject and object configurations, with seven of eight comparisons showing significant reductions.Both LSTMs generally did not expect extraction from within complex NPs, although the Google model lacked a significant effect for that-headed subject relatives.
- 4.3 Complex NP and Subject Islands: Neither LSTM clearly learned the subject constraint for prepositional-phrase extractions.The Google model treated PPs as licit extraction domains, whereas the Gulordava model treated them like islands.
5 Conclusion
The study finds that state-of-the-art LSTMs represent filler–gap dependencies and learn several, but not all, constraints on them. Despite major differences in model scale and training data, the larger Google model was not qualitatively more human-like than the Gulordava model.
- 5 Conclusion: LSTMs learned filler–gap dependencies in subject, object, and indirect-object positions, maintaining one-to-one expectations despite irrelevant interveners.Licensing effects were bidirectional because wh-licensing interaction measures both filler effects on gaps and gap effects on fillers.
- 5 Conclusion: Licensing effects shrank in wh-islands, adjunct islands, and most complex NP islands, but subject-island learning was unclear.Some residual trace-licensing interaction remained in island conditions.
- 5 Conclusion: Tenfold larger training data and hidden-state size did not make the Google model qualitatively more human-like than the Gulordava model.The models failed on different complex-NP and wh-island conditions, while both failed on subject islands.
- 5 Conclusion: Generic sequence models without an obvious hierarchical inductive bias can learn filler–gap dependencies and some of their constraints.The authors qualify this conclusion because the models were trained on much more data than a typical child learner receives.