Source-linked AI summary
Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
Tal Linzen, Emmanuel Dupoux, Yoav Goldberg
TL;DR
The paper asks whether LSTMs without explicit hierarchical representations can learn syntax-sensitive dependencies. It tests English subject-verb agreement under direct, indirect, and language-modeling objectives, finding that explicit supervision enables strong performance while language modeling alone does not reliably capture the dependency.
Problem
The study asks whether sequence models without explicit syntactic structure can learn syntax-sensitive dependencies from natural-language data.
Method
The authors evaluate LSTMs on English subject-verb agreement using number prediction, grammaticality judgments, and next-word language modeling.
Results
With explicit supervision, LSTMs learned most agreement cases but made more errors on difficult sentences; language modeling alone was insufficient for structure-sensitive dependencies.
Takeaways & Limitations
Syntax-sensitive tasks may benefit from training objectives that directly supervise the desired dependency rather than relying only on language modeling.
Takeaways & Limitations
The extent to which these results generalize to other grammatical constructions and languages remains an empirical question.
Abstract
from arXiv · showhide
The success of long short-term memory (LSTM) neural networks in language processing is typically attributed to their ability to capture long-distance statistical regularities. Linguistic regularities are often sensitive to syntactic structure; can such dependencies be captured by LSTMs, which do not have explicit structural representations? We begin addressing this question using number agreement in English subject-verb dependencies. We probe the architecture's grammatical competence both using training objectives with an explicit grammatical target (number prediction, grammaticality judgments) and using language models. In the strongly supervised settings, the LSTM achieved very high overall accuracy (less than 1% errors), but errors increased when sequential and structural information conflicted. The frequency of such errors rose sharply in the language-modeling setting. We conclude that LSTMs can capture a non-trivial amount of grammatical structure given targeted supervision, but stronger architectures may be required to further reduce errors; furthermore, the language modeling signal is insufficient for capturing syntax-sensitive dependencies, and should be supplemented with more direct supervision if such dependencies need to be captured.
1 Introduction
The paper asks whether sequence-based LSTMs can learn syntax-sensitive dependencies without explicit structural representations. It tests this question on English subject-verb agreement using increasingly indirect supervision and finds that targeted supervision supports strong performance, whereas language modeling alone is insufficient.
- Research question: The study probes whether RNNs can learn natural-language hierarchical structure from corpora without syntactic annotations.It focuses on English subject-verb agreement as a dependency commonly regarded as evidence for hierarchical structure.
- Experimental approach: The experiments compare direct verb-number prediction, sentence-level grammaticality judgments, and unsupervised next-word language modeling.The supervision becomes less direct across these settings, from identifying the crucial verb to receiving only sentence-level or next-word signals.
- Main findings: 0.83% errors were observed overall in the supervised number prediction model, although errors rose on cases requiring structural information.Even with four opposite-number nouns intervening, the model made 17% errors, while baselines performed far below chance.
- Main findings: Mistakes were especially common in noun-noun compounds and reduced relative clauses, where overt syntactic cues such as function words were unavailable.The pattern indicates that the model captured substantial syntactic knowledge but relied heavily on function words.
- Main findings: Language modeling performed worse than chance on harder agreement cases, and a large-scale language model made more than five times as many mistakes as the number prediction model.The results suggest that explicit supervision is necessary for learning this agreement dependency with the tested architecture.
2 Background: Subject-Verb Agreement as Evidence for Syntactic Structure
English subject-verb agreement depends on the syntactic subject’s head rather than on the nearest intervening noun. Its potentially unbounded distance and susceptibility to agreement attraction make it a test of whether sequence models can recover hierarchical structure.
- Agreement dependency: The form of a third-person present-tense verb depends on whether the syntactic subject’s head is singular or plural.The dependency is illustrated by contrasting singular and plural subjects with matching or mismatching verb forms.
- Structural challenge: Intervening nouns can separate the subject from the verb, and their number may either match the subject or conflict with it.The subject and verb can be separated by sentential material, including nouns inside modifiers and other constructions.
- Structural challenge: Correct agreement requires identifying the subject head corresponding to a verb and ignoring structurally irrelevant intervening nouns.A syntactic parse makes this identification straightforward through the direct nsubj relation.
- Structural challenge: Because subject-verb distance can contain an arbitrary number of words, fixed-order n-gram models cannot capture the dependency in general.RNNs can in principle represent dependencies of unbounded length, but whether they learn them from natural corpora is empirical.
- Structural challenge: An agreement attractor is an intervening noun whose number differs from the subject’s number.Such attractors create a conflict between a structurally relevant subject and a misleading sequential cue.
3 The Number Prediction Task
The number prediction task directly tests whether an LSTM can infer a following verb’s number from its preceding context. It uses corpus-derived examples and compares the full sequence input with noun-only baselines to isolate syntactic information.
- Task definition: The model reads a sentence prefix ending immediately before a present-tense verb and predicts PLURAL or SINGULAR.The task requires encoding syntactic number and identifying the correct subject for the following verb.
- Task demands: The task jointly tests whether the model can identify the relevant subject and assign the correct grammatical number.Identifying the corresponding subject often requires sensitivity to hierarchical syntax.
- Data: The task can generate practically unlimited examples by querying a corpus for sentences containing present-tense verbs and recording their number.The study generated approximately 1.35 million Wikipedia-based number prediction problems.
- Model: The LSTM receives word embeddings sequentially, and its final state is fed to a logistic regression classifier trained end to end.Words are represented as one-hot vectors embedded in a 50-dimensional space, with 50 hidden units in the LSTM.
- Baselines: Noun-only baselines preserve the original order of nouns while removing other words to isolate the contribution of syntactic information.One baseline uses common nouns only; another also includes pronouns and proper nouns.
4 Number Prediction Results
The LSTM achieved very high overall accuracy, but performance declined on dependencies where intervening nouns or syntactic structure conflicted with sequential cues. Its analyses nonetheless show sensitivity to distance, attractors, relative clauses, and structural modifiers, while longer embedded constructions exposed memory limitations.
- 0.83% of dependencies contained incorrect number predictions, compared with 4.2% and 4.5% errors for noun-only baselines.The low baseline error rates indicate that many agreement dependencies can be resolved from preceding noun sequences alone.
- Distance: Performance did not degrade considerably as subject-verb distance increased to 15 words when no nouns intervened.The model generalized from common distances of 0 and 1 to rarer distances of 10 or more.
- Agreement attractors: 6.5% and 5.4% errors occurred when the last intervening noun was an agreement attractor, versus 0.7% and 1.4% when it matched the subject’s number.Noun-only LSTM baselines reached 46.4% and 40% errors in attraction cases.
- Agreement attractors: 17.6% was the error rate with four attractors, while noun-only baselines reached up to 84% errors.Errors increased gradually as more attractors intervened, confirming the importance of syntactic cues in harder cases.
- Relative clauses: 9.9% errors occurred with an overt relative clause, compared with 3.2% without one and 25% without an overt relativizer.The analysis considered dependencies with one attractor and no other intervening nouns because of data sparsity.
- Visualizing activations: The model correctly predicted all 40 PP sentences but made three errors in RC sentences, where function words changed which noun determined agreement.In longer modifiers, RC predictions initially diverged correctly but later reconverged incorrectly as the network’s embedding-status representation weakened.
5 Alternative Training Objectives
The paper compares increasingly indirect training objectives for learning subject-verb agreement, from explicit verb-number supervision to grammaticality judgments and language modeling. Explicitly supervised models perform strongly, whereas language models struggle especially when agreement attractors create syntactic difficulty.
- Training objectives: The number prediction task trains the network to predict an upcoming verb’s number from preceding words, while verb inflection additionally supplies the verb’s singular form.Verb inflection therefore asks the model to choose between the singular and plural forms of a particular verb.
- Training objectives: Grammaticality judgments provide weaker supervision by asking whether a complete sentence is grammatical after flipping verb number in half the examples.Unlike the explicitly targeted objectives, the network reads the entire sentence and receives feedback only at the end.
- Training objectives: Language modeling predicts the next word from unlabeled sentences without specifically directing the network toward grammatical information.The model’s output is a distribution over the vocabulary rather than a binary agreement decision.
- Results: 0.8% versus 0.83% errors: verb inflection performed slightly but significantly better than number prediction, while grammaticality judgments reached 2.5% errors.Across all agreement dependencies, all models had error rates below 7%.
- Results: 6.78% compared to 0.83% errors: the language model made eight times as many errors as the original number prediction network.On dependencies with agreement attractors, the language model performed worse than chance, while the other models degraded more gracefully as attractors increased.
- Results: A much larger Google language model still performed poorly compared with the supervised model despite substantially more parameters and training data.The comparison used a random sample of 500 sentences for each attractor count rather than the entire test set.
6 Additional Experiments
Additional experiments test whether performance depends on recurrent cell type or on emphasizing difficult dependencies during training. LSTMs outperform SRNs broadly, while targeted training reduces errors on dependencies containing an attractor.
- Recurrent cell types: The SRN made about twice as many errors as the LSTM across attractor counts, without showing a qualitative difference in how error rates changed.The comparison used the same number of hidden units for both cell types.
- Targeted training: Training only on dependencies with at least one intervening noun reduced errors by 27% on cases with exactly one attractor.The error rate fell from 4.1% to 3.0%, despite using a smaller total training corpus.
7 Error Analysis
Error analysis identifies asymmetries and structural failure modes beyond overall accuracy. Models sometimes confuse attractors, misidentify compound-noun heads, or misanalyze verbs that resemble plural nouns, while the task itself conflates subject identification with number assignment.
- Singular versus plural subjects: LSTM attraction errors were somewhat higher with plural than singular attractors, but the asymmetry was much smaller than in humans.The SRN showed a much larger asymmetry, reaching 60% error with four plural attractors.
- Qualitative analysis: The qualitative analysis examined 200 cases in which most of 20 number-prediction runs made the wrong prediction.These cases represented about 0.6% of the dependencies.
- Structural errors: The networks often misidentified the heads of noun-noun compounds, indicating incomplete mastery of compound structure.In the example, the models treated conservation as determining agreement instead of the head refugees.
- Lexical errors: Verbs ambiguous with plural nouns sometimes acted as attractors even in sentences containing no plural nouns.The reported examples involve drives and lands.
- Task limitation: The number prediction task jointly evaluates identifying the subject and assigning the correct number to noun phrases.This coupling limits what can be inferred when an error reflects number assignment rather than subject identification.
8 Related Work
Related work includes artificial-language studies of recurrent networks, analyses of neural representations, grammaticality-judgment modeling, and proposals for strategically sampled dependency evaluations. The paper positions its contribution as isolating syntactic capabilities that aggregate NLP tasks can obscure.
- Motivation: Standard NLP evaluations average across syntactic constructions, making it difficult to isolate a network’s syntactic capabilities.This motivates evaluating specific dependency types rather than relying only on broad task performance.
- Artificial languages: Artificial-language studies found that LSTMs can learn the context-free language anbn and generalize to n values as high as 1000 from training on n ∈ {1, . . . , 10}.Simple recurrent networks struggled with this language in the cited studies.
- Artificial languages: Studies of miniature languages examined recurrent-network learning of relative clauses and anaphora, with mixed findings and later challenges to some conclusions.The cited work includes both SRN and LSTM investigations.
- Neural analyses: Other work proposed visualization methods for RNNs trained on nonsyntactic tasks such as image captioning and sentiment analysis.These methods target character-level and word-level recurrent representations.
- Grammaticality and evaluation: Prior research connected grammaticality judgments with probabilities assigned by language models and developed neural approaches to grammaticality judgments.The cited literature spans early neural models and more recent language-model analyses.
9 Discussion and Future Work
The paper proposes evaluating models on grammatically challenging sentences and training them on difficult dependencies to expose and improve structure-sensitive behavior. It also suggests multitask supervision, while emphasizing that generalization beyond the studied constructions and language remains empirical.
- Evaluation: Random-sentence evaluation can hide failures on difficult grammatical cases because simple heuristics achieve high overall accuracy.The authors recommend sampling by grammatical complexity for more nuanced language-model evaluation.
- Training: Training on dependencies with intervening nouns improved performance, but the improvement was quantitative rather than qualitative.The authors present this as an initial step toward oversampling grammatically challenging sentences.
- Future work: Multitask learning could develop sentence representations that benefit other structure-sensitive dependencies, including pronoun coreference, quantifier scope, and negative polarity items.The proposed setup trains the model concurrently on the task of interest and an agreement-related task.
- Limitations: The extent to which these results generalize to other constructions and languages remains a matter for empirical research.The paper notes that negative polarity items involve semantic factors and unbounded dependencies may require richer syntactic representations.
- Relation to human behavior: Human agreement-attraction errors and structurally irrelevant nearest-constituent effects provide relevant parallels for interpreting model behavior.Such effects have been reported in human production, comprehension, and acceptability judgments, as well as in Slovenian.
10 Conclusion
LSTMs lack built-in hierarchical representations but can approximate structure-sensitive subject-verb agreement when given explicit supervision. Language modeling alone is insufficient for these dependencies, motivating more expressive architectures or supplemental direct supervision.
- Conclusion: Explicitly supervised LSTMs learned verb-number agreement in most cases, though errors increased on particularly difficult sentences.The task probes a dependency that crucially depends on hierarchical syntactic structure.
- Conclusion: Language modeling alone was insufficient for learning structure-sensitive dependencies, while more expressive architectures may be needed to eliminate errors altogether.The authors suggest supplementing language models with joint training objectives when syntax-sensitive dependencies matter.