PaperlayerView exact revision on arXiv

Source-linked AI summary

Left-Branching Transformers Excel at Right-Branching Languages: Data Shapes Word Order Preferences in Language Models

Varvara Arzt, Allan Hanbury, Terra Blevins

arXiv:2608.15129v1cs.CLcs.AI

TL;DR

Do decoder-only language models develop word-order preferences because of their architecture or their training data? Across artificial and natural languages at multiple scales, the paper finds opposite architectural rankings and a growing, data-driven preference for SVO in natural languages.

  • Problem

    It remains unclear whether language models’ word-order preferences are architectural or data-driven, a pressing question amid concerns about LLM-mediated language homogenisation.

  • Method

    The study trains decoder-only models on 192 artificial languages and evaluates artificial and typologically diverse natural languages across training-data scales.

  • Results

    Models prefer left-branching configurations in artificial languages, but increasingly prefer SVO over SOV in natural languages as data grows, reflecting resource level and data quality.

  • Takeaways & Limitations

    Because highly resourced languages are overwhelmingly SVO, imbalanced training data creates a precondition for gradual reduction of word-order diversity.

  • Takeaways & Limitations

    The evaluation uses written, relatively formal translated corpora, so results may not generalise to spoken or informal language and may reflect translationese biases.

Abstract

from arXiv · show

We systematically compare word order preferences in decoder-only language models across 192 artificial languages and typologically diverse natural languages. On artificial languages, models exhibit a left-branching preference that aligns with neither natural language universals nor human word order learning biases. On natural languages, monolingual models show no clear base word order bias at small scales, but as data grows, a preference for right-branching subject-verb-object (SVO) languages emerges while SOV falls behind despite being the most frequent order cross-linguistically. This SVO advantage extends to multilingual models and correlates with language resource level and data quality rather than word order. Thus, the same architecture exhibits opposite preferences on artificial and natural languages, establishing that word order biases observed in practice are data-driven. Since highly-resourced languages are overwhelmingly SVO, these biases risk gradually reducing word order diversity, particularly in languages that productively use multiple word orders, with the widespread adoption of LLMs.

1 Introduction

Across controlled artificial- and natural-language evaluations, decoder-only models show opposite word-order preferences: left-branching for artificial languages but increasingly SVO for natural languages as data grows. These findings implicate training data, especially the dominance of highly resourced SVO languages, as a source of practical word-order bias.

  • Study design: The study trains models on 192 artificial languages spanning six base word orders and 32 internal constituent orderings, alongside monolingual and multilingual models on typologically diverse natural languages.Artificial and natural evaluations are compared across growing and matched training-data scales to disentangle architectural and data-driven biases.
  • Artificial languages: On artificial languages, models prefer left-branching configurations, aligning with neither cross-linguistic universals nor human learners’ preference for consistent directionality.Constituent surprisal suggests the preference may partly reflect the entropy structure of a semantics-free grammar, and rankings can shift with data volume.
  • Natural languages: At 5 MB, monolingual models show no visible natural-language word-order preference, whereas at 1000 MB they find SVO easiest and SOV hardest.The result implicates data quality rather than quantity alone.
  • Natural languages: The SVO advantage extends to multilingual models and disappears only when very-low-resource languages dominate the word order, as in BLOOM.This links the preference to language resource level and training-data composition rather than word order itself.
  • Implications: Because highly resourced languages are overwhelmingly SVO, imbalanced training data creates a precondition for gradual word-order homogenisation and reduced typological diversity.The introduction frames this risk as especially important because LLMs are increasingly deployed across typologically diverse languages.

2 Related Work

Prior studies have examined language-model word-order preferences mainly with artificial languages or synthetic variants of natural languages. This work combines controlled artificial and natural-language experiments to analyze word-order preferences, which prior work had not done.

  • Prior research has mainly studied LM word-order preferences using artificial languages and synthetic variants of natural languages.
  • White and Cotterell (2021) found a preference for left-branching configurations in PCFG-generated languages, with best performance on both SOV and OVS.Their findings suggested no alignment with typological frequency, and the present artificial-language experiments build upon this approach.
  • Synthetic variants of attested languages have been used to study recency bias in agreement prediction and the difficulty of translating flexible-order languages in low-resource settings.Ravfogel et al. (2019) altered English word order and case marking for RNN agreement prediction, while Bisazza et al. (2021) examined flexible-order languages in neural machine translation.
  • No prior work had combined controlled artificial- and natural-language experiments to analyze word-order preferences.The present study addresses this gap.

3 Methodology

The methodology separates architectural effects from training-data effects by comparing controlled artificial languages with typologically diverse natural languages. It standardizes word-order representations, evaluates multiple datasets and model settings, and measures both clause-level and constituent-order preferences.

  • Artificial languages: 192 artificial languages combine six S/V/O base orders with 25 head-direction switch configurations to isolate word-order effects.The switches control complement clauses, complementisers, adpositions, adjectives, and relative clauses, selecting left-branching or right-branching variants.
  • Natural-language encoding: Natural languages are mapped to the same encoding scheme using WALS, Grambank, and APiCS labels, supplemented by manual verification.The classifications generally reflect grammar descriptions rather than corpus frequencies, and a single categorical label may obscure gradient variation.
  • Data: Evaluation uses FLORES-200 for broad typological coverage and PUD treebanks to verify findings and provide gold-standard annotations for token-level surprisal analysis.Automatic Stanza parsing complements the syntactic annotations.
  • Models: Each artificial language receives a separate GPT-2-style decoder-only model, testing whether word-order preferences arise from the transformer architecture itself.Natural-language experiments primarily use monolingual Goldfish GPT-2-based models trained at four data volumes on one of up to 350 languages.
  • Measures: Word-order preferences are measured at clause-level S/V/O linearisation and internal constituent order, with artificial-language models additionally evaluated by test perplexity and PPL-curve AUC.The internal constituent orders are aggregated into a directionality scale; training dynamics are tracked every 50 steps, but this is unavailable for Goldfish models.

4 Experimental Setup

The experiments train GPT-2-style decoder-only models at 5 MB and 10 MB data volumes, using disjoint splits and averaged test perplexity. They evaluate Goldfish and multilingual models on FLORES-200 and PUD, with syntactic annotations from Stanza and gold UD.

  • Model training: Models use GPT-2-style decoder-only architectures trained primarily on 5 MB and 10 MB datasets, after finding 0.5 MB results comparable to 5 MB.The setup includes approximately 0.5 MB and 10K sentences in the original condition, plus 5 MB and 10 MB settings.
  • Model training: The architecture matches the Goldfish configuration with 4 layers and 8 attention heads, using a 15k-vocabulary BPE word-boundary SentencePiece tokenizer.This differs from the whitespace tokenizer used by White and Cotterell (2021).
  • Evaluation protocol: Each model is trained on 10 fully disjoint train/dev/test splits, and results are reported as averaged test perplexity for robustness.Implementation details, hyperparameters, and compute costs are reported in Appendix A.1.
  • Evaluation protocol: Goldfish models are evaluated at four training-data sizes, while multilingual models are evaluated on FLORES-200 and PUD within their training-language overlap.Surprisal analyses use Stanza syntactic annotations on FLORES-200 and gold UD annotations on PUD.

5 Results

Artificial-language models consistently favor left-branching structures, while natural-language models develop a right-branching SVO advantage as training data scales. This divergence reflects training-data composition and resource level rather than a stable architecture-wide word-order preference.

  • 5.1 Artificial languages: Models consistently prefer left-branching configurations in artificial languages, conflicting with cross-linguistic universals and human word-order learning biases.Mean PPL increases with the number of right-branching switches across all six base orders, while top-ranked configurations lack close natural-language counterparts.
  • 5.1 Artificial languages: Relative-clause direction dominates the artificial-language effect: all sixteen prenominal configurations are easier, with ∆PPL of 4–7 points on 5 MB, despite postnominal relatives outnumbering prenominal ones approximately 4:1.Other switches show smaller but consistent effects, including a preference for prenominal adjectives; all five paired t-tests remain significant after Holm–Bonferroni correction.
  • 5.1 Artificial languages: Artificial-language base-order rankings are unstable: at 5 MB SOV ranks first and VSO last, whereas at 10 MB the three VS orders rank 1–3 and the three SV orders rank 4–6.At 10 MB, SVO is the hardest order, opposite to natural-language results at scale.
  • 5.2 Natural languages: Natural-language models show no clear preference at 5 MB, but by 1000 MB right-branching SVO languages become easiest to model.For monolingual Goldfish models, SVO, SOV, and NoDominant all center around BPEC 2.13 at 5 MB; at 1 GB, SVO–SOV Cliff’s δ=−0.76 and SOV incurs ∆BPEC ≈0.13 higher compression cost.
  • 5.2 Natural languages: The SVO and right-branching advantages extend to multilingual models and correlate with resource level and data quality rather than word-order type.In BLOOM, the median SVO/SOV ratio is 11×, reduced to 5× in XGLM after upsampling; highly resourced languages benefit more from cleaner data as scale grows.

6 Conclusion

The study disentangles architectural and data-driven influences on word order preferences, finding opposite patterns in artificial versus natural languages. Its conclusion highlights typological risks from SVO advantages and identifies directions for testing their broader implications.

  • Conclusion: Across artificial and natural languages, decoder LMs reveal opposite word order preferences, separating architectural biases from data-driven effects.The analysis spans multiple training-data scales and language types.
  • Conclusion: Artificial-language models prefer left-branching configurations, while natural-language models develop an SVO preference as data grows and SOV falls behind.At small scales, natural-language models show no clear preference; at scale, SVO languages become favored while SOV languages lag.
  • Conclusion: 1 GB of training data suffices for the SVO preference to emerge even in monolingual models, raising typological concerns because highly resourced languages are overwhelmingly SVO.The authors expect these effects to be more pronounced in frontier models with more training data.
  • Conclusion: Future work should test larger models, spoken-language data, finetuning, downstream tasks, and mechanistic explanations of how word order preferences are encoded.These directions target the scope, causal modulation, and internal representation of the observed preferences.

Limitations

The study’s conclusions are limited by restricted experimental coverage, written translated evaluation data, coarse data-quality proxies, and simplified categorical word-order labels. These choices may constrain generalization and obscure variation in language use and the mechanisms behind observed effects.

  • Experimental scope: The artificial-language experiments cover only a limited subset of typological possibilities, and the study evaluates the GPT-2 architecture.The PCFG rules cannot fully represent strategies such as the range of relative-clause placements attested across languages.
  • Evaluation data: Results from FLORES-200 and PUD may not generalize to spoken or informal language because both are written, relatively formal, translated corpora.Spoken language has greater word-order flexibility, register and genre affect constituent ordering, and translated text may introduce translationese biases.
  • Data quality: The OS-CAR web-crawl share is a coarse proxy for data quality, so the analysis cannot identify which finer quality dimensions drive the SVO advantage.Direct annotations of register, source, and corpus cleanliness would be needed to isolate the effect.
  • Word-order labels: Categorical word-order labels from WALS and Grambank simplify often variable ordering patterns into feature classifications.Such labels may omit variation across clause types and the pragmatic conditions under which ordering alternations arise.

A Appendix · A.1 Artificial Language Details

The artificial-language experiments use controlled pseudoword grammars, shared tokenization, and matched small training corpora to isolate word-order effects. The appendix details the model, training, evaluation, and computational setup across 192 grammars and two data sizes.

  • A.1 Artificial Language Details: The vocabulary contains ∼50k pseudoword forms across five open-class categories, with nouns comprising 50%, adjectives 10%, and verb classes 40% combined.Closed-class items include 4 prepositions and 7 pronouns.
  • A.1 Artificial Language Details: Each language variant generates 5 MB and 10 MB of training text by sampling sentences from the PCFG.These corpus sizes are compared with the 10,000 sentences used by White and Cotterell (2021).
  • A.1 Artificial Language Details: The PCFG permits recursive expansion up to 400 expansions per sentence, producing generated sentences averaging 12 tokens.The recursion bound follows White and Cotterell (2021).
  • A.1 Artificial Language Details: A shared SentencePiece BPE tokeniser uses a 15k vocabulary in word-boundary mode, preventing merges across word boundaries.Because the 192 languages share vocabulary and word frequencies, separately trained tokenisers yield identical 15,000-piece vocabularies.
  • A.1 Artificial Language Details: The study reproduces White and Cotterell (2021) before adopting the Goldfish architecture to match natural-language model size and enable direct comparison.Pilot experiments found train–validation loss crossover near epochs ∼10 and ∼16 for 5 MB and 10 MB corpora, respectively.
  • A.1 Artificial Language Details: Training is limited to 20 epochs, with evaluation at the step level and checkpoint selection based on the lowest validation loss.The protocol is motivated by validation-loss behavior and prior findings of diminishing gains beyond 16 epochs of data repetition.
  • A.1 Artificial Language Details: The experiment trains 1,920 models across 192 grammars and 10 splits at each of 5 MB and 10 MB data sizes.The computation uses NVIDIA A100 GPUs and totals approximately 252 A100 GPUhours over approximately 2 days with parallelised Slurm array jobs.

A.2 Natural Language Details

The study analyzes natural languages using typological word-order labels and a controlled set of evaluation languages. Goldfish comparisons keep the language sample fixed across training sizes to isolate data-volume effects.

  • Word Order Labeling: Artificial-grammar switches are mapped to attested typological features from WALS and Grambank, using lang2vec aggregates of WALS, SSWL, and Ethnologue.The mapped features include base word order, complement-clause order, complementiser position, adposition order, adjective–noun order, and relative-clause–noun order.
  • Evaluation Languages: 103 FLORES-200 languages with base-word-order labels form the evaluation basis, including 68 Goldfish, 46 BLOOM, 30 XGLM, and 56 mGPT training languages.20 of the 103 languages also have a PUD treebank.
  • Goldfish Model Coverage on FLORES-200: Goldfish models are evaluated on 183 FLORES-200 languages under strict one-to-one matching between FLORES codes and dedicated models.Cross-variety substitutions are excluded because differences in word order could confound the analysis.
  • Evaluation Languages: The same languages are compared across all four training scales, enabling attribution of observed changes to data volume rather than changing language samples.Larger sets available at individual sizes, such as 130 languages at 5 MB, are not used in §5.

A.3 Evaluation

The evaluation uses raw perplexity for artificial languages and normalized bits per English character for natural languages. It also tests four Greenbergian implicational universals using syntactic constructions available in the artificial languages.

  • Evaluation metrics: Raw perplexity is directly comparable across artificial languages because all 192 variants share vocabulary, tokenization, and derivation probabilities.For natural languages, per-token perplexity is not comparable across scripts and tokenizers, motivating BPEC normalization.
  • Evaluation metrics: BPEC normalizes corpus negative log-likelihood by the English character count of parallel corpora for natural-language evaluation.The normalization follows Cotterell et al. (2018) and uses FLORES-200/PUD parallel corpora.
  • Evaluation metrics: Evaluation prepends the model’s start token as conditioning context but excludes it from scored tokens.Goldfish uses [CLS], whereas multilingual models use [BOS].
  • Greenbergian Universals: Four testable Greenbergian implicational universals cover base subject–verb–object order, adpositions, adjective–noun order, and relative clauses.The artificial PCFG generates only declarative sentences, restricting evaluation to these construction types.

A.4 Language Resource Level and Word Order Distribution … A.7 Artefact Use and Licensing

Across natural-language analyses, SVO languages are more highly resourced and increasingly outperform SOV languages as training data grows, with the interaction surviving controls. Additional results replicate the reported artificial- and natural-language patterns, while the artefacts are used for research under specified licences.

  • A.4 Language Resource Level and Word Order Distribution: SVO languages are more highly resourced than SOV languages, averaging resource classes 3.35 versus 2.24.The difference is significant (Mann–Whitney U=472, p=0.003, r=0.72).
  • A.4 Language Resource Level and Word Order Distribution: Among 19 highly-resourced Class 4–5 languages with SVO or SOV order, 14 are SVO (74%).
  • A.5 Mixed-Effects Analysis of the SVO–SOV Gap: The SOV×size interaction remains significant across M2–M9 and maximal random-effects models, shrinking only ∼20% from +0.049 to +0.038.This interaction represents growth of the SVO–SOV performance gap with training-data scale.
  • A.5 Mixed-Effects Analysis of the SVO–SOV Gap: Resourcedness predicts the interaction, with resource_lvl×size β=−0.012 and p=.037, while covariates jointly explain the average gap but not its growth.
  • A.6 Additional Results: Additional results are reported in Figures 6, 7, and 11 for artificial languages and Figures 8–10, 12, and Table 7 for natural languages.
  • A.7 Artefact Use and Licensing: The artefacts are used for research consistent with their intended purpose and carry licences including CC-BY-SA 4.0, CC-BY 4.0, MIT, Apache 2.0, GPL, and BigScience RAIL v1.0.Goldfish models are publicly available but unlicensed.
  • A.7 Artefact Use and Licensing: At 10 MB, all five paired t-tests remain significant after Holm–Bonferroni correction, with p < .001.
Loading 2608.15129v1…