Source-linked AI summary
Neural Language Models Learn the Contextual Distributions of Dependency Structures: a statistical learning theory to compositionality
Wang Bojun, Junjie Chen, Holly Jenkins, Elizabeth Wonnacott
TL;DR
The paper asks how NLMs acquire structural meaning that cannot be derived from lexical semantics alone. It proposes that learned dependency structures become units for contextual-distribution learning and tests this account in a synthetic grammar. Models learn contextual profiles unavailable from token statistics alone, while the experiments establish only a first step toward recursion.
Problem
The paper asks how NLMs acquire structural meaning and compositional semantics beyond what component lexical meanings provide.
Source: §1 · p. 2Method
The paper proposes tracking contextual distributions of learned composite dependency structures and tests the account with a synthetic grammar separating structure-context profiles from token-level statistics.
Source: §1 · pp. 3–4Results
Models distinguished contextual categories and generalized to unseen structure-context combinations, with external learning beginning when internal dependency relations were first detected.
Source: §2.1 · p. 5, §2.2 · p. 7Takeaways & Limitations
The findings support treating composite dependency structures as distributional units whose contextual features encode properties not inferable from component tokens.
Source: §3 · p. 13The clean synthetic setting separates token and structure profiles, but natural corpora may have overlapping profiles, and recursion itself was not tested.
Source: §3 · p. 13
Abstract
from arXiv · showhide
It is unclear how Neural Language Models (NLMs) acquire the structural meaning encoded by grammatical structures that is independent of lexical semantics. We propose a statistical learning process in which learned dependency structures themselves become new distributional units for subsequent statistical learning. Under this account, once a dependency structure is acquired, the model tracks its contextual distributions. These contextual features reflect the semantic properties of a composite structure. To test this hypothesis, we design a synthetic grammar in which each grammatical structure has distinct contextual distributions that cannot be recovered from the distributional statistics of their component tokens alone. We train a series of BERT-style masked language models on this grammar and examine their developmental trajectory. The results show that models can successfully learn the contextual distributions of composite dependency structures even though they cannot be inferred from token statistics alone. Developmental analysis further reveals a clear developmental trajectory. The learning of the dependency relations that define a grammatical structure consistently precedes the learning of its contextual features. These findings suggest that statistical learning in NLMs is not merely the accumulation of token co-occurrence statistics, but a process in which learned dependency structures become new units of distributional learning. We argue that this process provides a statistical-learning account of how NLMs solve the compositionality problem in language. Finally, we discuss the possibility that this statistical learning process provides an explanatory theory on how language cognition could emerge from pure distributional statistics.
1 Introduction
The introduction frames compositionality as a problem of deriving structural meaning beyond lexical semantics and proposes that learned dependency structures become units for tracking contextual distributions. It motivates and outlines a synthetic-grammar test in which those contextual profiles cannot be recovered from token-level statistics alone.
1 Introduction
- Compositionality requires explaining how composite semantics is derived from lexical meanings and how syntactic elements map to semantic roles.The introduction notes that lexical meanings alone may not identify who is agent or patient in an utterance.Source: §1 · p. 2
- Because NLMs acquire semantic information by tracking contextual distributions rather than attending to semantic environments, their statistical learning merits study.The introduction presents statistical learning as an under-explored aspect of NLM cognition.Source: §1 · pp. 1–3
- Learned composite dependency structures are proposed as distributional units whose external contextual profiles reflect properties that internal dependencies alone cannot capture.The account treats relations among a structure and other units in its context as its external distributional profile.Source: §1 · p. 3
- The synthetic grammar defines BAA and ABA structures through A–B dependencies and gives them distinct M and N contexts despite identical token-level distributions.Both structures contain two A tokens and one B token, so their contextual difference emerges only when the composite structures are treated as units.Source: §1 · p. 3
- The dataset combines exhaustive structure-context combinations with compensation sequences so M and N have no overall dependency while structure-context relations remain learnable.The resulting dataset sets P(M)=P(N)=0.4 and P(M,N)=0.16; its only potentially learnable dependencies are the internal structure relations and external structure-context relations.Source: §1 · p. 4
- A control dataset uses ABA for both contexts, and a 90/10 training-validation split withholds combinations so models must generalize distributional propensities.The control removes the contextual distributional difference between M and N.Source: §1 · p. 4
2 Results
Models generalized to contextual combinations absent from training, while developmental probes showed internal dependencies generally emerged before external profiles. The findings replicated across the tested model sizes.
2.1 Vector analysis
- A/B vectors separated by 4,000 iterations, while M/N separated only after 39,200, placing contextual categorization later in development.Source: §2.1 · p. 5, Figure 1 · p. 6
2.2 Probability mass in sequence generating
- Target-category probability mass approached one on all four probes, including unseen BAA-M and ABA-N combinations, indicating productive external-dependency categories.Those combinations had never been observed in training.Source: Figure 3 · p. 7, §2.2 · p. 7
- External-dependency learning began when internal relations were first detected rather than waiting for their full acquisition, although internal dependencies were generally learned earlier.Source: §2.2 · p. 7
2.3 Scaling the model
- Vector analyses of experimental-language models and architecture-matched control-language models in 4L-4H-16D and 8L-8H-64D suggest 8L-8H-32D findings could replicate at other sizes, not that they did.Source: §2.3 · p. 9
3 Discussion
The discussion argues that learned composite dependency structures become distributional units whose contextual features capture properties unavailable from component tokens alone. It frames recursive nesting as a possible extension, while noting that the synthetic setting and experiments leave important complexities untested.
3 Discussion
- Models treat learned composite dependency structures as distributional units, tracking contextual features to learn relational properties unavailable from component-token properties alone.Source: §3 · p. 13
- The synthetic grammar disentangles structural contextual features from token-level distributions, unlike more complex natural corpora where these profiles may overlap.The authors call for further investigation in such complex environments.Source: §3 · p. 13
- The grammar omits dependencies between composite structures, which may attract or repel one another and add complexity beyond token dependencies.Source: §3 · p. 13
- The study demonstrates one nesting level; learned external dependencies could become internal to larger structures, but experiments do not test this recursion directly.Source: §3 · p. 13
- The proposed recursion links sentence-level event structures into a global event structure, a possible statistical-learning account of emergence that this study does not examine.Source: §3 · p. 13
4 Materials and Methods
The methods implement a synthetic grammar and train Transformer variants, then analyze token representations and masked-position category probability mass across checkpoints. They also describe held-out probes and controls for architecture comparisons.
4.1 Synthetic grammar implementation
- The generator balanced M and N categories (P(M)=P(N)=0.40; P(M,N)=P(M)P(N)=0.16), while the control replaced BAA with ABA so both occurred in ABA.Category labels were withheld during training, and ordinary-token identifiers were shuffled before model input.Source: §4.1 · p. 14
4.2 Architecture and training for vector analysis models
- Vector-analysis models used an 8-layer, 8-head, 32-dimensional main architecture and 4L/4H/16D and 8L/8H/64D scaling models, with checkpoints through 100,000 updates.Models used dynamic exact-two-position masking, AdamW, a minibatch size of 64, a learning rate of 1 × 10−4, and weight decay of 0.01.Source: §4.2 · p. 14
4.3 Architecture and training for Probability-mass analysis models
- Probability-mass analyses used three architectures trained for 200,000 updates, with dynamic exact-two-position masking, AdamW, and a minibatch size of 512.The architectures were 4L/4H/16D, 8L/8H/32D, and 8L/8H/64D.Source: §4.3 · p. 14
4.4 Developmental Value-vector analysis
- First-layer Value projections provided static context-free token vectors, which separate autoencoders reduced to two dimensions for visualization.The 2D coordinates, orientations, scales, and absolute distances should not be compared across model panels.Source: §4.4 · pp. 14–15
4.5 Probability-mass evaluation
- Held-out probes used softmax probabilities to measure total probability assigned to a target lexical category, averaged over 512 probes per family and 100 models.Confidence intervals treated the model, rather than each probe string, as the independent unit.Source: §4.5 · p. 15
4.6 Scaling analysis and reproducibility
- Scaling analyses varied layer count, head count, and hidden dimension while holding the other procedures constant and aggregating multiple seeds identically.Source: §4.6 · p. 15