PaperlayerView exact revision on arXiv

Source-linked AI summary

Training Language Models via Neural Cellular Automata

Dan Lee, Seungwook Han, Akarsh Kumar, Pulkit Agrawal

arXiv:2603.10055v1cs.LGcs.AIcs.CL

TL;DR

Natural-language pre-training is costly, finite, biased, and entangles knowledge with reasoning, motivating alternatives for acquiring useful representations. The paper pre-pre-trains transformers on structured NCA trajectories before natural-language training and finds improved, faster language modeling with transfer to reasoning benchmarks. It also shows that the useful NCA complexity depends on the downstream domain.

  • Problem

    Natural-language pre-training has finite high-quality data, human biases, and entangles knowledge with reasoning, raising whether language is necessary for useful representations.

  • Method

    The paper uses neural cellular automata to generate structured, non-linguistic trajectories for transformer pre-pre-training before standard natural-language pre-training.

  • Results

    NCA pre-pre-training improves language modeling by up to 6%, accelerates convergence by up to 1.6×, and transfers gains to GSM8K, HumanEval, and BigBench-Lite.

  • Takeaways & Limitations

    Synthetic pre-training can provide meaningful signal without linguistic content, and its distribution can be tuned to target domains through complexity choices.

  • Takeaways & Limitations

    For larger alphabet sizes, downstream improvement follows a reverse U-shaped trend and plateaus beyond an intermediate token budget.

Abstract

from arXiv · show

Pre-training is crucial for large language models (LLMs), as it is when most representations and capabilities are acquired. However, natural language pre-training has problems: high-quality text is finite, it contains human biases, and it entangles knowledge with reasoning. This raises a fundamental question: is natural language the only path to intelligence? We propose using neural cellular automata (NCA) to generate synthetic, non-linguistic data for pre-pre-training LLMs--training on synthetic-then-natural language. NCA data exhibits rich spatiotemporal structure and statistics resembling natural language while being controllable and cheap to generate at scale. We find that pre-pre-training on only 164M NCA tokens improves downstream language modeling by up to 6% and accelerates convergence by up to 1.6x. Surprisingly, this even outperforms pre-pre-training on 1.6B tokens of natural language from Common Crawl with more compute. These gains also transfer to reasoning benchmarks, including GSM8K, HumanEval, and BigBench-Lite. Investigating what drives transfer, we find that attention layers are the most transferable, and that optimal NCA complexity varies by domain: code benefits from simpler dynamics, while math and web text favor more complex ones. These results enable systematic tuning of the synthetic distribution to target domains. More broadly, our work opens a path toward more efficient models with fully synthetic pre-training.

1. Introduction

The paper asks whether structured, non-linguistic data can provide useful pre-training signals without natural-language semantics. It proposes NCA pre-pre-training and reports improved language modeling, reasoning transfer, and domain-targeted complexity choices.

  • Natural-language pre-training faces finite high-quality data, human biases, and entanglement between knowledge and reasoning.
  • The paper hypothesizes that models can acquire useful computational structure from data rather than language semantics.
  • NCA generates diverse non-linguistic sequences with local rules, long-range spatiotemporal patterns, and heavy-tailed token distributions.
  • NCA pre-pre-training transfers perplexity gains to GSM8K, HumanEval, and BigBench-Lite, outperforming natural-language pre-pre-training with more data and compute.
  • Optimal NCA complexity varies by domain, with simpler dynamics benefiting code and more complex dynamics benefiting math and web text.
  • Attention layers account for most transfer gains, while MLP transfer depends on alignment between synthetic and target domains.

2. Related Works

Prior work shows that algorithmically generated data can transfer representations across domains, but language applications remain less developed. This paper builds on that literature by studying structured cellular-automaton data for language-model pre-training.

  • Synthetic data spans LLM-generated examples and simple algorithmic procedures; this paper focuses on the latter and uses non-linguistic data.
  • Algorithm-Based Synthetic Data: Algorithmic visual data lacking semantic content has produced representations that transfer effectively to real images.
  • Algorithm-Based Synthetic Data: Language-domain synthetic pre-training has explored algorithmic procedures and context-free grammars, but remains comparatively uncommon.
  • Algorithm-Based Synthetic Data: Earlier cellular-automata language studies did not evaluate whether learned representations transfer to language.
  • Shared Underlying Computation: Cross-domain transfer studies motivate the possibility that neural networks learn general computations shared across modalities.
  • Emergent Complexity: Research on emergent complexity examines how simple procedures can generate richly structured data resembling real-world complexity.

3. Method

The method generates neural cellular-automaton trajectories with controllable structural complexity, tokenizes them as grid patches, and trains a transformer by autoregressive next-token prediction.

  • NCA generation: NCA generalizes classical cellular automata by parameterizing update rules with neural networks, enabling diverse dynamics sampling.
  • NCA generation: The experiments use 2D discrete NCA on a 12×12 periodic grid with a 10-state alphabet and locally computed transitions.
  • Complexity sampling: Random transition rules and initial conditions produce trajectories ranging from predictable fixed points or short cycles to chaotic, unpredictable dynamics.
  • Complexity sampling: Gzip compression ratios provide a practical complexity measure for selecting NCA trajectories with controlled structure.
  • Tokenization: Each grid is tokenized into nonoverlapping 2×2 patches, producing a fixed vocabulary of 104 patch tokens and sequences up to 1024 tokens.
  • Training objective: Autoregressive cross-entropy training requires inferring each trajectory’s latent transition rule from context and applying it consistently.

4. Experimental Setup

The experiments test a three-stage synthetic-to-natural training pipeline against scratch and Dyck baselines across web text, math, code, and reasoning benchmarks. Transfer is measured through language-model perplexity, convergence speed, and task accuracy.

  • Training pipeline: The training paradigm consists of NCA pre-pre-training, natural-language pre-training, and task-specific fine-tuning.
  • Training pipeline: The study evaluates whether computational structure learned from synthetic data improves language pre-training and downstream reasoning.
  • Pre-pre-training: A 1.6B-parameter Llama-based transformer receives 164M NCA tokens sampled from the 50%+ gzip-compressibility band before natural-language training.
  • Pre-pre-training: Embedding layers are reinitialized for the natural-language vocabulary, while all parameters are updated during subsequent pre-training.
  • Baselines and controls: Baselines include scratch training and K-Dyck pre-pre-training, with hyperparameters selected by grid search.
  • Evaluation: Language pre-training uses OpenWebText, OpenWebMath, and CodeParrot, while reasoning evaluations use matched BigBench-Lite, GSM8K, and HumanEval domains.
  • Evaluation: Transfer is assessed mainly by validation perplexity and tokens required to reach scratch-model final perplexity, with pass accuracy or pass@k for reasoning.

5. Results

NCA pre-pre-training consistently improves language-model training across scales and domains, transfers to reasoning, and reveals attention and data complexity as key determinants of transfer.

  • 5.1. NCA Pre-Pre-Training Improves Language Modeling: NCA pre-pre-trained models consistently outperform scratch, Dyck, and C4 baselines across 400M, 600M, and 1.6B parameter scales.On OpenWebText, improvements over scratch reach 8.6% for 400M models and 5.7% for 1.6B models, consistently across random seeds.
  • 5.1. NCA Pre-Pre-Training Improves Language Modeling: NCA pre-pre-training outperforms C4 despite using 160M tokens versus 1.6B, improving perplexity by 5% and converging 1.4× faster.The comparison remains favorable with and without transferring pre-trained embedding layers.
  • 5.1. NCA Pre-Pre-Training Improves Language Modeling: NCA improves validation perplexity by 4–5% on OpenWebMath and CodeParrot and achieves up to 1.6× faster convergence, with advantages persisting throughout training.The gains generalize across web text, math, and code rather than reflecting only an initialization effect.
  • 5.2. Language Modeling Gains Translate to Downstream Reasoning: NCA pre-pre-training transfers to reasoning across math, logic, and code benchmarks, including GSM8K, HumanEval, and BigBench-Lite.GSM8K accuracy rises from 3.8% to 4.4% at pass@1, while BigBench-Lite reaches 36.5% at pass@4 versus 29.7% for C4 and 25.9% for scratch.
  • 5.3. What Drives Transfer?: Re-initializing attention causes the largest transfer degradation, while MLP and LayerNorm effects depend on the downstream domain.Attention therefore carries the most transferable computational primitives, whereas other components can encode domain-specific or interfering statistics.
  • 5.3.2. DATA COMPLEXITY MODULATES TRANSFER AND THE OPTIMUM IS DOMAIN-DEPENDENT.: The optimal NCA complexity is domain-dependent: OpenWebText favors 50%+ gzip complexity, whereas CodeParrot favors intermediate 30–40% complexity.NCA alphabet size also matters: the smallest alphabet, n = 2, continues improving where larger alphabets plateau.

6. Discussion

The discussion argues that structured synthetic data can transfer computational primitives without linguistic content, but effective transfer depends on matching generator complexity to target domains. It also identifies attention as the main carrier of transferable structure and highlights unresolved questions about scaling and principled complexity matching.

  • Mechanisms of transfer: Attention layers account for most transfer gains, while MLP transfer depends on alignment between synthetic and target domains.Attention is associated with long-range dependency tracking and in-context learning; MLPs encode more domain-specific knowledge.
  • Scaling and structural choices: The smallest alphabet, n = 2, scales most favorably, while larger alphabets show diminishing returns and plateauing improvements.For n = 10 and 15, downstream improvement is strongest at an intermediate token budget before the gap narrows.
  • Scaling and structural choices: NCA pre-pre-training can outperform natural-language pre-pre-training despite using fewer tokens, but the comparison occurs while 1.6B-token language training remains early-regime.The discussion hypothesizes that limited C4 training emphasizes surface regularities, whereas NCA exposes models to long-range dependencies and in-context learning.
  • Mechanisms of transfer: NCA sequences may provide a purer signal for in-context rule inference because each sequence is generated by one latent transition rule.The model must infer the rule from context and apply it consistently, making next-token prediction nearly deterministic once the rule is identified.
  • Open problems: A central open problem is developing principled methods to sample synthetic structures whose relevant complexity axes match particular target domains.Gzip compressibility and alphabet size capture only some aspects of complexity; other axes include NCA network size, grid size, and epiplexity.

A. Analysis on Natural and Synthetic Data Distributions

The paper compares natural-language and NCA-generated data using token-frequency distributions and gzip compressibility as high-order structural heuristics.

  • Analysis framework: Figure 8 compares relative token-frequency distributions across natural-language corpora and NCA data while using gzip compressibility to characterize complexity.The natural-language corpora span OpenWebText, OpenWebMath, and CodeParrot.
  • Analysis framework: Token-frequency distribution and gzip compressibility are the two primary heuristics used to compare natural-language and NCA-generated data.The analysis treats these as distribution-level measures rather than semantic comparisons.

A.1. NCA data exhibits similar token distributions to natural language

NCA-generated data has a heavy-tailed, Zipfian token distribution structured similarly to natural language, although natural-language distributions vary by domain.

  • Token distributions: NCA-generated data follows a heavy-tailed, Zipfian token distribution that is structured similarly to natural language.The comparison uses relative token frequencies from sampled and tokenized sequences.

A.2. Increasing the vocabulary size n leads to more complex generated trajectories

Increasing the NCA alphabet size makes generated trajectories less compressible and therefore more complex by expanding the space of possible rules.

  • Trajectory complexity: As alphabet size n increases, the distribution of gzip complexity shifts toward less compressible, more complex trajectories.The reported alphabet sizes are n = 2, 10, and 15.

B. Detailed Pre-pre-training and Pre-training Setup

The experiments use controlled sweeps, multiple seeds, and domain-matched downstream evaluations to assess NCA pre-pre-training and its effects across synthetic-data settings.

  • Experimental reproducibility: The pipeline uses four randomness seeds for each main pipeline and at least two seeds for each ablation run.The main pipelines are NCA Pre-pre-training, Scratch, and C4 Pre-pre-training.
  • Hyperparameter selection: The study sweeps batch sizes from 32 to 512, learning rates from 1 × 10−3 to 1 × 10−5, and weight decays from 1 × 10−4 to 1 × 10−6.
  • NCA complexity: Figure 9 compares NCA complexity distributions for alphabet sizes n = 2, 10, and 15.Increasing alphabet size inherently increases data complexity.
  • Downstream evaluation: BigBench-Lite evaluation samples at most 300 tasks per major English-language problem category when at least 100 training examples are available.The sampling addresses the benchmark’s quantity and task imbalance.
  • Downstream evaluation: GSM8K and BigBench-Lite use domain-specific fine-tuning, whereas HumanEval is evaluated without fine-tuning as a code-completion task.GSM8K uses 10 epochs and Chain-of-Thought traces; BigBench-Lite uses one epoch.
  • Metrics and decoding: Pass@k is evaluated for k = 1, 8, 16, and 32, using 64 total decodings per run and temperature 0.4 with top-p 0.95.For BigBench, evaluation demonstrates up to four passes because some tasks are multiple-choice.

D. NCA Pre-Pre-Training is more token efficient than natural language

The paper evaluates convergence using token efficiency relative to scratch training and finds that NCA pre-pre-training reaches equivalent performance with substantially fewer tokens across web text, math, and code.

  • Metric: Token Efficiency Gain is defined as 1 − T_NCA / T_PT, comparing NCA pre-pre-training and pre-training tokens required to reach the scratch model’s final loss.The no-pre-pre-training baseline sets T_PPT = 0.
  • Results: 31% token efficiency gain is achieved on OpenWebText, 27% on OpenWebMath, and 49% on CodeParrot.These are average gains required to reach equivalent performance to the scratch baseline.
  • Results: The largest average token efficiency gain occurs on CodeParrot, followed by OpenWebText and OpenWebMath.
Loading 2603.10055v1…