Source-linked AI summary
The emergence of number and syntax units in LSTM language models
Yair Lakretz, German Kruszewski, Theo Desbordes, Dieuwke Hupkes, Stanislas Dehaene, Marco Baroni
TL;DR
The paper asks how LSTMs achieve long-distance number agreement and whether their success relies on genuine hierarchical structure rather than surface heuristics. By analyzing internal cell and gate dynamics, it identifies specialized number units and syntactic units that regulate them. The findings provide evidence that LSTMs can implement structure-dependent grammatical processing, while leaving some unit interactions and gate-control mechanisms unresolved.
Problem
The study addresses the lack of mechanistic understanding of how LSTMs track number across long-distance dependencies and whether this ability uses hierarchical structure.
Method
The authors analyze a pretrained LSTM’s number-agreement behavior through single-unit ablation, cell-state and gate dynamics, and identification of units tracking syntactic structure.
Results
Long-distance number information is carried by specialized singular and plural units, whose gate control is linked to a unit encoding subject-verb syntactic structure.
Takeaways & Limitations
The relationship between syntax and number units suggests that LSTM agreement cannot be entirely explained by superficial heuristics and involves structure-based syntactic representations.
Takeaways & Limitations
The analysis excludes relative-clause agreement and leaves the interplay between long- and short-range number units for future work.
Abstract
from arXiv · showhide
Recent work has shown that LSTMs trained on a generic language modeling objective capture syntax-sensitive generalizations such as long-distance number agreement. We have however no mechanistic understanding of how they accomplish this remarkable feat. Some have conjectured it depends on heuristics that do not truly take hierarchical structure into account. We present here a detailed study of the inner mechanics of number tracking in LSTMs at the single neuron level. We discover that long-distance number information is largely managed by two `number units'. Importantly, the behaviour of these units is partially controlled by other units independently shown to track syntactic structure. We conclude that LSTMs are, to some extent, implementing genuinely syntactic processing mechanisms, paving the way to a more general understanding of grammatical encoding in LSTMs.
1 Introduction
The study moves beyond behavioral evidence to examine how LSTMs mechanistically track number during language modeling. It finds specialized number units whose long-distance behavior is regulated by syntactic-structure units, providing direct evidence of structure-dependent processing.
- 1 Introduction: The study investigates an LSTM’s inner dynamics during number agreement rather than inferring linguistic capacities solely from output behavior.The analysis targets a mechanistic account of how the model performs the task.
- 1 Introduction: Two specialized “grandmother” cells carry subject-number features across intervening material in long-distance dependencies.One unit stores singular information and another encodes plurality.
- 1 Introduction: Number prediction is more distributed when subject and verb are close, while grandmother number cells become crucial in harder long-distance cases.The two mechanisms have distinct operating conditions and dynamics.
- 1 Introduction: A cell tracking an embedded phrase has strong efferent connections to the long-distance number cells, suggesting syntactic regulation of agreement-feature percolation.The syntactic cell encodes structure separating the main subject-verb dependency.
- 1 Introduction: The findings provide direct evidence that LSTMs trained on unannotated corpora can learn structure-dependent linguistic operations despite lacking significant linguistic priors.The authors suggest raw linguistic input and generic memory mechanisms may suffice to induce non-trivial grammatical rules.
2 Related work
Earlier work established strong behavioral agreement performance but left the network’s internal mechanics largely unexplained. This study is distinguished by systematically characterizing linguistic processing through individual recurrent-cell dynamics.
- 2 Related work: Long-distance number agreement evaluates whether a model predicts the correct verb form when subject and verb are separated by intervening nouns.Accuracy is based on how often the model selects the correct inflection.
- 2 Related work: LSTM language models achieved near-human agreement performance, while studies disagreed about whether their success reflected genuine syntax or surface heuristics.The evidence motivating this debate was primarily behavioral.
- 2 Related work: Most subsequent research tested network outputs rather than explaining how linguistic behavior follows from internal representations.Earlier qualitative work had identified cells tracking grammatical number in a model trained directly on agreement.
- 2 Related work: Diagnostic classifiers reveal what information is recoverable from activations but provide only indirect evidence about the mechanics of linguistic processing.Their value is strongest for identifying information encoded across layers or time points.
- 2 Related work: The present approach extends cell-level visualization work by systematically characterizing a linguistic phenomenon through recurrent-cell dynamics.The authors identify no comparable prior systematic analysis at this level.
3 Setup
The study analyzes a pretrained, Wikipedia-trained LSTM on naturalistic and synthetic number-agreement tasks. The setup samples sentence positions and syntactic depths to separate these correlated factors, while excluding relative-clause agreement.
- 3 Setup: The analyzed model has 650-dimensional embeddings, two 650-dimensional hidden layers, and a 50,000-word output vocabulary.It was pretrained on Wikipedia without number-agreement fine-tuning.
- 3 Setup: The evaluation combines a corpus-derived Linzen agreement test set with synthetically generated number-agreement datasets.The task suite probes agreement under varied intervening structures.
- 3 Setup: Key findings were replicated across different initialization seeds and hyperparameter variations.This replication used the same model family with altered training settings.
- 3 Setup: The synthetic task examples vary intervening material, including adverbs, conjunctions, names, and noun phrases.Representative conditions include Simple, Adv, 2Adv, CoAdv, NamePP, and NounPP variants.
- 3 Setup: Agreement across a relative clause was excluded because it introduces an additional agreement process inside the relative clause.This defines a scope boundary for the setup.
- 3 Setup: The syntactic-depth dataset contains 4,033 positions from 1,303 sentences sampled across lengths of 2–25 words and depths of 3–8.Sampling uniformly covered position-depth combinations within the 7–12 position and 3–8 depth ranges.
4 Experiments
Experiments identify two long-range number units whose activity supports subject-number storage across intervening nouns, while a syntax-tracking unit regulates their behavior during dependencies.
- 4.1 Long-range number units: Ablating units 776 and 988 reduced performance by more than 10% across conditions and brought difficult incongruent cases near chance.The effects were consistently strongest among the tested units in namePP, nounPP, and nounPPAdv tasks.
- 4.1 Long-range number units: Ablation effects were number-specific: unit 776 primarily supported plural subjects, whereas unit 988 primarily supported singular subjects.The authors therefore designate them the plural and singular long-range number units, respectively.
- 4.1 Long-range number units: The units use gate and cell-state dynamics to encode subject number, preserve it across an interfering noun, and update it near the verb.The singular unit opens its input gate for the subject, closes it during the dependency, and maintains its stored state through forget-gate control.
- 4.1 Long-range number units: Singular and plural number-unit outputs are segregated toward the corresponding verb forms, with weight signs matching their negative number encoding.These output-layer connections provide complementary support for identifying the two units as number-specific.
- 4.3 Syntax units: The syntax unit 1150 tracks the subject–verb dependency across varied structures, including intervening verbs and exceptionally long dependencies.Its activity rises across the dependency and drops abruptly after it, indicating structured syntactic tracking.
- 4.4 Syntax-number units connections: Strong connections from syntax unit 1150 drive the number units toward remembering during the dependency and updating afterward, while reciprocal connections mutually inhibit them.The reported gate weights implement remember and update flags and help produce an unambiguous number signal.
5 Summary and discussion
The study identifies a local LSTM circuit for long-distance number agreement, combining specialized number units with syntax-tracking units that regulate their behavior. This provides direct evidence that generic language-model training can produce structure-dependent linguistic operations, while leaving variation across architectures, languages, domains, and training time for future study.
- Mechanism: Three highly interactive units form the central part of a neural circuit performing long-distance number agreement.The circuit emerged from simply training an LSTM language model on raw corpus data.
- Number tracking: Two number units encode and carry singular or plural grammatical information across long-range dependencies.The singular unit stores subject number across the dependency, while another unit similarly encodes plurality.
- Syntax interaction: Syntax-tracking units regulate number storage and release, with one unit encoding embedded structure and strongly connecting to the long-distance number units.Its activity correlates with syntactic complexity and is naturally interpreted as propagating syntax-based remember and update flags.
- Complementary mechanisms: Number is also redundantly encoded in a distributed mechanism, but that mechanism cannot carry information across embedded syntactic structures.The network therefore contains two types of units encoding similar information with distinct properties and dynamics.
- Conclusion: The syntax–number relationship suggests that agreement cannot be entirely explained by superficial heuristics and that the network learned structure-based syntactic representations.The authors qualify this conclusion as applying only to some extent.
- Open questions: Future work will test how the encoding varies across architectures, hyperparameters, languages, domains, and training time.The study does not yet characterize these sources of variation.