Source-linked AI summary
Structural priors for data-efficient language learning
Yana Veitsman, Jonas Mayer Martins, Jonathan Lautenschlager, Lisa Beinborn
TL;DR
The paper examines whether non-language data can provide structural priors that reduce the data and compute demands of multilingual language learning. It pretrains models on symbolic data, then evaluates language-modeling efficiency, weight shifts, and linguistic benchmarks; symbolic data improves next-token prediction but not reliably broader linguistic performance. Overall, structural transfer is a partial substitute for language data on next-token prediction, yet remains less efficient than additional natural-language training.
Problem
The paper asks how language models can learn efficiently when data and computational resources are limited, particularly in multilingual settings.
Method
The study pretrains GPT-2-style models on symbolic data, then continues multilingual training and evaluates loss, parameter shifts, and downstream linguistic benchmarks.
Results
Structural data, especially music, probabilistic grammars, and cellular automata, improves next-token-prediction efficiency, while gains do not consistently transfer to linguistic benchmarks.
Takeaways & Limitations
Non-language data can partially substitute for language data for next-token prediction, but does not reliably support broader linguistic generalization.
Takeaways & Limitations
The study does not exhaustively cover structural data types or data-mixing effects, and its English-only Wikipedia baseline cannot assess transfer across natural languages.
Abstract
from arXiv · showhide
Efficient language learning requires methods to reduce the reliance on large data and computational resources. We investigate structural transfer: First training models on non-language data to induce useful priors for natural language. This approach is a form of weight initialization for multilingual language modeling. We evaluate transfer via next-token-prediction loss, weight shifts in the model, and downstream linguistic benchmarks. Several symbolic data types - notably music, probabilistic grammars, and cellular automata - yield lower language-modeling loss than random initialization. These gains coincide with smaller weight shifts during subsequent language training, suggesting that structural transfer positions models in a more favorable region of the parameter space. However, a lower loss does not translate consistently into better downstream linguistic performance, and transfer from non-language data is less efficient than additional language data. We conclude that non-language data can serve as a partial substitute for language data for the training objective of next-token prediction but does not reliably support broader linguistic generalization.
1 Introduction
The paper asks whether non-language data can induce structural priors that make multilingual language learning more data-efficient. It evaluates this transfer through language-modeling efficiency, parameter changes, and downstream linguistic performance.
- Motivation: The study targets data efficiency because language-model performance depends strongly on scarce data and compute, especially for low-resource and cognitively plausible learning.
- Motivation: Pre-pretraining is framed as structural transfer: models learn to predict structured symbolic signals before reusing their parameters for language training.
- Motivation: Multilingual training tests whether structural-transfer benefits persist across diverse languages and whether the induced priors are language-specific.
- Approach and contributions: Structural transfer pretrains models on symbolic data before multilingual language training to test whether initialization can accelerate learning.The structural sources include probabilistic grammars, cellular automata, piano music, and protein sequences.
- Approach and contributions: The analysis measures natural-language next-token prediction, model-internal weight shifts, and downstream BabyLM linguistic benchmarks.
2 Related work
This work frames structural transfer as using prior exposure to structured data to initialize models for more efficient language learning. Related approaches include interaction, curricula, multilingual learning, and transfer across tasks or languages, but prior findings on downstream benefits are mixed.
- Structural transfer: Structural transfer treats prior training on structured data as a parameter initialization that can move models toward more efficient language learning.The framing connects structural biases to the model’s starting parameter region rather than only to architecture.
- Related approaches: Prior BabyLM work improves sample efficiency through interaction, curriculum learning, and multilingual learning when data or compute cannot scale.
- Multilingual modeling: Multilingual transfer raises whether performance differences reflect model or data biases, linguistic properties, or the informational complexity of the training signal.
- Study scope: The study examines structural transfer using several symbolic signals, including probabilistic grammars, cellular automata, music, and protein sequences.
- Empirical findings: Existing structural-transfer findings are mixed: abstract grammars can induce hierarchical representations, but downstream gains are not consistently comparable or substantial.
3 Data
The study pretrains language models on four structural data types or random sequences before multilingual language training. These datasets span grammatical, dynamical, musical, biological, and unstructured signals, with English, Dutch, and Mandarin used in the target corpus.
- Data pipeline: The data pipeline has two stages: structural-data pretraining from scratch followed by training on the target natural-language corpus.
- Structural data: Stage I compares synthetic grammars, cellular automata, music, and protein sequences against uniformly random integer sequences.
- Synthetic grammars and cellular automata: Probabilistic context-free grammars sample rules modeled on the Penn Treebank, while cellular automata generate trajectories from local update rules.
- Music: Music is represented as integer sequences of note events, timing, and pedal states, capturing long-range temporal and hierarchical structure without linguistic semantics.
- Protein sequences: Protein data consists of Swiss-Prot amino-acid sequences normalized to the 20 canonical amino acids plus an unknown symbol.
- Target language data: Stage II uses cognitively plausible multilingual data sampled equally from English, Dutch, and Mandarin Chinese.
- Baselines: The primary baseline skips structural pretraining, while a secondary baseline first trains on English Wikipedia to test whether structural transfer beats additional natural-language data.
4 Experimental setup
The experiments use GPT-2 models trained with controlled data splits and early stopping, then evaluate structural transfer on multilingual next-token prediction and downstream linguistic tasks. The setup also defines loss-based metrics comparing transfer conditions with random initialization.
- Training protocol: Models use a GPT-2 architecture, 80/20 data splits, and early stopping to select structural and natural-language training amounts.
- Evaluation: Structural-transfer gains are evaluated using held-out multilingual validation loss plus zero-shot and fine-tuned linguistic benchmarks.
- Tokenization: A trilingual BPE tokenizer has a fixed 16,897-token vocabulary, while 512 stage-I-only tokens are excluded from stage-II lexical transfer by reinitializing embeddings.
- Loss metrics: Next-token prediction is measured with cross-entropy loss over training tokens.
- Loss ratio: The loss ratio ρ compares areas under condition and baseline loss curves through N tokens, using a 2 M-token minimum cutoff.
- Token efficiency: Token efficiency τ measures the tokens required with structural and language training to reach the final baseline loss.
5 Results
Structural transfer improves multilingual next-token prediction and can reduce subsequent language-training weight shifts, especially for music, grammars, and CA16. These gains do not generally extend to linguistic benchmarks, while additional language data remains more effective.
- 5.1 Transfer on next-token prediction: Structural data generally lower multilingual next-token-prediction loss than random initialization, with the clearest gains from MUSIC, PCFGUNI, PCFGZIPF, and CA16.PCFGUNI and PCFGZIPF overlap, while CA256 converges to the baseline by 220 M tokens.
- 5.1 Transfer on next-token prediction: Additional English Wikipedia data outperforms every symbolic-data condition by a wide margin on language-modeling loss.The best structural-transfer conditions require approximately 60% of the baseline token budget to reach equivalent language-modeling performance.
- 5.2 Model-internal representations: Synthetic grammars, MUSIC, and CA16 produce smaller language-training weight shifts than the no-structure baseline, averaging approximately ¯δ = 0.34 across layers.PROTEINS and RANDOM do not reduce language-training shifts, and CA256 requires shifts as large as the random baseline.
- 5.2 Model-internal representations: Language-training weight shift strongly correlates with loss improvement (r = 0.96, p < 0.01), whereas the structural-data correlation is weaker (r = −0.60, p = 0.11).The results associate smaller language-training shifts with larger language-loss improvements.
- 5.3 Linguistic benchmarks: Loss reductions from structural transfer do not generally improve average zero-shot or fine-tuning benchmark performance across English, Dutch, and Chinese.The only robust stage-I improvement for English and Dutch comes from training on English texts; other structural conditions do not significantly change average results.
- 5.3 Linguistic benchmarks: Bayesian estimates place all structural-data effects close to zero with HDIs including zero, while Wikipedia training improves zero-shot performance by ∆ = +1.2 pp.Wikipedia also yields ∆ = +1.3 pp for fine-tuning, with a 95% HDI of [−0.1, 2.7].
6 Discussion
Structural patterns speed gains on next-token prediction, but these gains do not reliably transfer to linguistic benchmarks or match the efficiency of additional natural-language data.
- Structural patterns enable faster gains on next-token prediction, but none of the tested conditions matches the efficiency of additional natural-language data.The results suggest that synthetic structural data can partially substitute for language data for the training objective, but not as efficiently.
- Lower next-token-prediction loss does not consistently produce better linguistic-benchmark performance because the evaluations measure language competence through different metrics.
- PCFG variants perform equally, suggesting that a Zipfian distribution alone has little effect on transfer because unigram structure is learned quickly from natural language.
- Protein structure is not readily accessible to the transformer under naive one-token-per-amino-acid tokenization, limiting the observed transfer from protein sequences.
- Structural data may initialize models in functionally useful parameter regions, potentially connecting transfer to learned attention-head mechanisms such as copying or retrieval.
- The better-performing structural data are partly language-like, but they lack language semantics, leaving the contribution of semantics to transfer unresolved.
7 Conclusion
Structural transfer improves next-token prediction efficiency and is associated with smaller parameter changes during multilingual training, but its broader benefits remain limited.
- Structural data before natural language improves next-token prediction efficiency, with the strongest-benefiting models requiring smaller parameter changes during multilingual training.The authors interpret smaller weight shifts as evidence that prior structural exposure places models in a more favorable parameter region.
- Improvements on the language-modeling objective do not reliably translate into stronger downstream linguistic benchmark performance.
- Additional natural-language data remains more efficient than structural data when it is available.
- Structural transfer may provide a useful initialization head start when compute and model capacity are limited.
Limitations
The study’s conclusions are bounded by its selected structural datasets, GPT-2-small architecture, limited data-mixing analysis, and coarse weight-shift evaluation.
- The selected structural data types are not exhaustive, leaving possible transfer effects from other data types unexamined.
- The experiments use GPT-2-small, so varying model size could clarify the limits of observed next-token-prediction improvements.
- Data-mixing effects are not explored exhaustively, and the English-only Wikipedia baseline prevents assessing transfer across different natural languages.
- Weight-shift magnitude is only a coarse representation-level analysis, motivating future mechanistic-interpretability work.
Ethical considerations
The supplied passages describe synthetic grammar and cellular-automaton data generation, but do not state ethical considerations.
- The supplied material describes technical data-generation procedures rather than ethical considerations.
B.1 Technical implementation
The technical implementation generates deterministic neural cellular-automaton trajectories and integer-encoded piano sequences for structural pretraining.
- Neural cellular automata update each cell from a three-cell neighborhood using embedding, convolution, GELU, unembedding, and argmax steps.
- The resulting cellular-automaton rules are deterministic given fixed parameters and an initial condition, producing trajectories that are flattened into language-model sequences.
- The CA datasets vary rules, grid sizes, trajectory lengths, initial conditions, and state or architecture settings to provide structural-data variety.
- Piano pieces are converted into integer sequences encoding note onsets, offsets, pedal changes, pitches, velocities, and time shifts.
D BabyLM natural-language data
The BabyLM data comprise multilingual Chinese, Dutch, and English corpora, with language-specific domain mixtures and substantial token totals.
- The BabyBabelLM corpora contain 138 M Chinese, 110 M Dutch, and 99 M English tokens.
- Within each language, the corpus combines multiple domains whose percentages sum to 100%.
E Training regime
Training uses fixed settings and a multilingual BabyLM evaluation pipeline, while the challenge submission mixes three structural data types across training stages. Loss ratio is more robust than token efficiency for comparing training outcomes.
- Experiments keep hyperparameters fixed across structural data types and training stages and use a single NVIDIA A100 80GB GPU for approximately 250 GPU hours.
- Evaluation follows the BabyLM 2026 multilingual pipeline for zero-shot and fine-tuning tasks across English, Dutch, and Chinese.The evaluation overview reports accuracies for all metrics except MECO, which is excluded from two figures.
- Loss ratio ρ is more robust to outliers than token efficiency τ because token efficiency depends strongly on the training-step cutoff.Integrating over the loss curve provides a more robust interpretation by reducing cutoff-point sensitivity.
- The challenge submission mixes MUSIC, CA16, and PCFGZIPF in stage I before training on English, Dutch, and Chinese in stage II.The model is selected by validation loss after hyperparameter search, with five seeds and ten epochs per stage.
- Piano pieces in the ARIA-MIDI dataset are represented as integer-coded events containing pitch, volume, pedal state, and tape-head shifts.