PaperlayerView exact revision on arXiv

Source-linked AI summary

A Data-free Universal Prior over Syntactic Structures

Fermín Moscoso del Prado Martín

arXiv:2609.16854v3cs.CLcond-mat.dis-nn

TL;DR

The paper asks whether syntactic-structure probabilities arise solely from language-specific experience. It derives a data-free prior from language production and finds that it favors attested trees over uniform-tree expectations while correlating positively with language-specific PCFG probabilities.

Abstract

from arXiv · show

The probabilities of syntactic structures in human languages are assumed to emerge fully from language-specific experience. Here, I show that a universal prior over syntactic structures emerges from a model of human language production, in which words are progressively integrated into syntactic structure. Without fitting any parameters to specific language data, the resulting prior assigns higher probabilities to attested than to random dependency trees in all 138 typologically diverse languages examined. These prior probabilities correlate positively with those estimated from corpora in 33 of 34 languages. The results indicate that part of the probability structure of syntax can arise independently of language-specific learning. This identifies human language production as a possible cognitive source of universal statistical structure in language, while providing a data-independent structural bias for probabilistic models, including large language models.

A universal prior over syntactic structures

The paper derives a prior over dependency trees from incremental language production, modeling attachment through the structure built so far. It accounts for compatible insertion orders and averages over the attachment parameter rather than fitting it to language data.

A universal prior over syntactic structures

  • Words enter the developing dependency tree as new nodes, each attaching to a word already present.The process is framed as a growing network in which dependencies are edges.Source: A universal prior over syntactic… · pp. 2–3
  • A word’s existing direct dependents provide an observable proxy for its latent propensity to attract further dependents, motivating preferential attachment.The model gives higher attachment probability to nodes with more existing connections.Source: A universal prior over syntactic… · p. 3
  • Because production is only approximately incremental, simultaneous planning and structural priming weaken independent decisions and motivate sublinear preferential attachment.The model’s attachment probability increases sublinearly with a node’s existing dependents.Source: A universal prior over syntactic… · p. 3
  • A tree’s probability sums its generation probabilities across insertion orders compatible with that tree; the model treats all n! orders as equally likely a priori.Compatible orders place each node after its ancestors and are the tree’s increasing labellings, or linear extensions.Source: A universal prior over syntactic… · p. 4
  • The resulting data-independent prior integrates the attachment parameter over its full sublinear range using an uninformative prior.This marginalizes the parameter instead of fixing it.Source: A universal prior over syntactic… · p. 4

The universal prior distinguishes language from randomness

The sPA prior assigns attested dependency trees higher probabilities than a uniform prior, a pattern reported across 138 typologically diverse languages. The study also compares the prior’s behavior on trees generated by sPA and by a uniform process.

The universal prior distinguishes language from randomness

  • The comparison used up to 500 attested trees per language, considering sentences of 50 words or fewer.Languages with fewer than 500 suitable trees contributed all available trees.Source: The universal prior distinguishes… · p. 5
  • As controls, the study compared the attested trees with same-size trees generated by sPA and with trees sampled uniformly from rooted labelled trees.Both controls were evaluated under the same two priors.Source: The universal prior distinguishes… · p. 5
  • Attested dependency trees are systematically favored by the sPA prior over the uniform prior across the 138 languages studied.The pattern also holds across sentence lengths and typologically diverse languages.Source: The universal prior distinguishes… · p. 5

The universal prior predicts language-specific probabilities

The paper tests whether the data-free sPA prior predicts language-specific syntactic probabilities, approximated with PCFGs. The probabilities are positively related across sentence lengths and in 33 of 34 languages, though the correlations are moderate.

The universal prior predicts language-specific probabilities

  • The study approximates language-specific probabilities with PCFGs induced for 34 languages and compares them with sPA probabilities for sampled sentences.The evaluated languages had at least 10,000 non-empty trees; sampled sentences contained 4–30 words.Source: The universal prior predicts… · p. 6
  • sPA and language-specific PCFG probabilities correlate positively at every sentence length and in 33 of 34 languages.Italian is the exception, with a non-significant, slightly negative average correlation.Source: The universal prior predicts… · p. 6
  • The Italian exception may reflect limitations of the PCFG approximation rather than a failure of the universal prior.Italian dependency trees, like those of other languages, were better predicted by sPA than by the uniform prior.Source: The universal prior predicts… · p. 6
  • The correlations between language-specific PCFG probabilities predicted using the sPA prior and the uniform prior are only moderate, but substantial considering the sPA prior contains no language-specific information.Length-standardised sPA probability positively predicted language-specific length-standardised PCFG probability (β = .171, 95% CI [.142, .199], P < .0001).Source: The universal prior predicts… · p. 7

The origins of syntactic structure probabilities

The paper argues that incremental language production can give rise to a universal, non-uniform prior over syntactic structures before language-specific statistics are learned. It proposes that experience updates this structured prior and that production may be a cognitive source of statistical regularities in language.

The origins of syntactic structure probabilities

  • The results suggest syntactic probabilities do not arise solely from language-specific experience: production processes themselves may help shape them.The paper describes this as the reverse of the established influence of syntactic probabilities on production.Source: The origins of syntactic structure… · p. 7
  • Incremental production can yield a non-uniform universal prior before language-specific statistics, leaving experience to update a structured distribution rather than induce probabilities from scratch.The authors suggest this production-based bias could also inform computational models, including large language models.Source: The origins of syntactic structure… · pp. 7–8
  • The proposed data-independent structural bias could be used in probabilistic computational models, including large language models that predominantly learn from linguistic data.Source: The origins of syntactic structure… · p. 8
  • The account echoes the broader principle that explaining why forms arise requires considering the processes by which they are built, associated with D’Arcy Thompson’s study of biological form.The passage cites this principle as 25.Source: The origins of syntactic structure… · p. 8
  • The authors connect incremental construction of syntactic structures to cognitive pressures that may constrain grammatical variation and enduring statistical universals.They frame this connection as a possible pressure, not as a demonstrated cause of every universal.Source: The origins of syntactic structure… · p. 8

Methods

The methods construct a data-free prior from incremental dependency-tree growth, compare it with random trees, and test its relationship to language-specific PCFG probabilities. The final mixed-effects model finds a positive association, with a slightly weaker effect for longer sentences.

Corpora and preprocessing

  • The corpus analysis used Universal Dependencies treebanks, concatenating treebanks by language, retaining lexical vertices, and removing punctuation, range vertices, and relation labels.Trees were limited to at most 30 words and to at least four vertices for shape comparisons.Source: Corpora and preprocessing · pp. 11–12

Computing the universal prior

  • The prior calculation combines the number of compatible node orderings, computed exactly, with the tree-generation probability averaged across those orderings, estimated by Monte Carlo sampling.Compatible orderings are increasing labellings, counted with the hook length formula; the sampling integrates over orderings and attachment exponent α.Source: Computing the universal prior · p. 12
  • For each sampled ordering and α, the tree probability is computed by replaying sublinear preferential attachment and multiplying stepwise attachment probabilities while updating dependent counts.The analysis uses 1,000 joint samples per tree; the estimator is unbiased and converges to the exact expectation as sample size increases.Source: Computing the universal prior · pp. 12–14

Generation of random tree controls

  • Random-tree controls used uniformly sampled Prüfer sequences27 with uniformly selected roots for edge orientation, while sequential sPA trees began at one root, added one node per step, and used α = .42.The fixed α = .42 was previously found optimal for human language.Source: Computing the universal prior · p. 14

Language-specific probability estimation

  • For language-specific probability estimation, dependency trees were converted to phrase-structure trees with parts of speech as leaves; a separate maximum-likelihood PCFG was induced for each of 34 languages with at least 10,000 non-empty dependency trees.Source: Computing the universal prior · p. 14

Statistical Analyses

  • The sPA–PCFG relationship was tested with a linear mixed-effects model using log PCFG probability as the outcome and log sPA probability, sentence length, and their interaction as fixed effects.Probabilities were standardized within sentence length, and sentence length was centered.Source: Statistical Analyses · p. 15
  • Adding a language-specific random slope for sPA probability improved model fit, and the final model was refitted with a random intercept and slope.The likelihood-ratio test was χ2(2) = 403.23, P < .0001.Source: Statistical Analyses · p. 15
  • β = .171 (95% CI [.142, .199]) for length-standardised sPA probability, and β = −.004 (95% CI [−.006, −.002]) for its interaction with sentence length in predicting PCFG probabilities.The sPA effect was positive; the interaction indicates a slightly weaker effect for longer sentences.Source: Statistical Analyses · p. 15

Methods-Only References

The methods-only references identify sources for random-tree generation, grammatical-diversity measurement, and statistical modeling software.

Methods-Only References

Notes

The note reports that the results are qualitatively robust to alternative priors and to fixing the attachment exponent at its previously estimated optimal value.

Notes

  • The analyses produced qualitatively equivalent results with alternative priors.Source: Notes · p. 16
  • The note says the findings do not depend on marginalizing over α.Source: Notes · p. 16
  • Fixing the attachment exponent at its previously estimated optimal value also produced qualitatively equivalent results.Source: Notes · p. 16

Related papers

Loading 2609.16854v3…