PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Pre-pretraining

Alexander Stern, Tim Rocktäschel, Acyr Locatelli, Laura Ruis

arXiv:2610.40051v1cs.CLcs.AI

TL;DR

This paper asks whether synthetic pre-pretraining can provide useful priors when natural-language data is scarce and whether its benefits hold across training regimes. It compares synthetic curricula with natural-language training under additive and compute-matched budgets, finding no compute-matched advantage for PPT and initialization-sensitive additive gains that disappear at larger language budgets.

  • Problem

    Synthetic data has been proposed to supply useful linguistic and computational priors when high-quality natural language is scarce, but the robustness of PPT gains across training configurations and regimes remains unclear.

    Source: §1 · pp. 1–2

  • Method

    The paper compares synthetic curricula with natural-language baselines in additive-compute and compute-matched experiments spanning several training regimes.

    Source: §1 · p. 2, §3 · p. 5

  • Results

    Across compute-matched regimes, allocating compute to PPT never outperforms natural-language training; repeated epochs over 100M unique tokens also beat every synthetic curriculum.

    Source: §1 · p. 3, §8 · p. 10

  • Takeaways & Limitations

    The findings motivate rethinking explicit PPT phases and investigating interleaving, while leaving open whether interleaved synthetic data is a better use of compute than natural language.

    Source: §1 · p. 3, §8 · p. 11

    Above 9 × 10^18 FLOPs, the study could not run a full hyperparameter search and fixed data-unconstrained PPT shares at 1% and 5%.

    Source: §7 · p. 10

Abstract

from arXiv · show

We report that allocating compute to synthetic pre-pretraining (PPT) never outperforms pure natural-language training in our compute-matched experiments. This holds across four PPT curricula (k-Dyck, neural cellular automata (NCA), declarative reference corpora, and code) and three training regimes: data-constrained, data-unconstrained, and a high token-to-parameter-ratio regime (881M parameters trained on 50B tokens). When natural language is scarce, repeated epochs over a 100M-token corpus beat every synthetic curriculum on natural-language validation loss, with no consistent downstream accuracy advantage from PPT. We find that PPT gains reported in prior work are sensitive to the weight initialization scheme. In an NCA reproduction, we recover a 5.4% perplexity improvement from PPT under the original initialization scheme. An alternative scheme improves both methods, but the no-PPT baseline more, closing the gap between them. However, when PPT receives additional compute rather than a share of the natural-language budget, gains persist at small natural-language budgets even with improved initialization, but disappear by 1B tokens. Finally, models forget what they learn during PPT in subsequent natural-language training; interleaving the same data throughout training prevents forgetting, but does not significantly improve natural-language performance. Our results suggest limited benefits of synthetic pre-pretraining as a distinct training stage and motivate rethinking how such data are used in language model training.

1 INTRODUCTION

Synthetic pre-pretraining (PPT) is proposed as a way to acquire useful computational priors when natural-language data is scarce, but its robustness across training regimes remains unclear. Across compute-matched regimes, natural-language training performs as well as or better than PPT; additive-compute gains are limited to smaller language budgets, and PPT-learned gains are lost across the phase boundary.

1 INTRODUCTION

  • Prior work uses synthetic distributions to target computational priors and reports language-modeling and context-recall gains, including up to 8.6% lower NCA perplexity.The introduction identifies robustness across training configurations and regimes as an open question motivating this study.Source: §1 · p. 1
  • PPT gains depend on initialization: an NCA reproduction recovers a 5.4% perplexity improvement, but changing embedding initialization closes the gap with the no-PPT baseline.The alternative initialization improves the baseline itself by 9.0%.Source: §1 · p. 2
  • With additional compute, k-Dyck and NCA improve over the best no-PPT baseline at 50M and 100M natural-language tokens, but the advantage disappears by 1B tokens.At the larger budget, benefits appear only against baselines without spectral-based initialization.Source: §1 · p. 2
  • Natural-language training performs as well as or better than PPT across compute-matched regimes, including when repeated epochs use only 100M unique natural-language tokens.The pattern holds on most DCLM CORE and BLiMP evaluations, with exceptions at the smallest compute budget.Source: §1 · p. 3
  • PPT gains on its own distribution disappear after natural-language training begins, while interleaving preserves synthetic-distribution performance without significantly improving natural-language validation loss.The authors suggest front-loading synthetic data is a poor use and point to gradual integration as a direction to investigate.Source: §1 · p. 3

2 BACKGROUND & RELATED WORK

Prior pre-pretraining work uses synthetic data to instill language-modeling priors, but reported gains vary across configurations. This paper revisits selected synthetic curricula and compares how compute should be allocated under different data and compute constraints.

2.1 PRE-PRETRAINING

  • The paper focuses on k-Dyck and NCA, two synthetic distributions for which prior work reports the largest natural-language perplexity improvements.Prior PPT corpora also include algorithmic tasks and data sampled from randomly initialized neural networks.Source: §2.1 · p. 3
  • k-Dyck requires each opening bracket to be closed, while an NCA grid evolves through a shared local update rule represented by a randomly initialized convolutional neural network.The paper’s k-Dyck curriculum is the k-Shuffle Dyck variant reported to perform best at k = 128.Source: §2.1 · p. 3
  • NCA corpora are generated by simulating random initial grids under random convolutional update rules, filtering rollouts by gzip compression ratio, and flattening retained grids into token sequences.The retained grids are concatenated in time-step order.Source: §2.1 · p. 3
  • In the common protocol, a synthetic phase precedes natural-language training, and its tokens replace an equal number of natural-language tokens in the control at equal total tokens and model size.PPT token count is varied to optimize downstream natural-language loss.Source: §2.1 · pp. 3–4
  • Prior work reports up to 5.7% lower perplexity for NCA and around 0.6% lower cross-entropy loss for k-Dyck, alongside downstream benchmark gains.A reproduction found k-Dyck gains did not consistently persist across model sizes, tokenizers, and languages; concurrent work reports downstream gains but not validation-loss improvements.Source: §2.1 · p. 4

2.2 TRAINING REGIMES

  • The study compares data-constrained, data-unconstrained, and over-Chinchilla regimes, where the scarce resource or token-to-parameter ratio differs.Model size is searched separately by method and budget in the data-constrained and data-unconstrained regimes.Source: §2.2 · p. 4

3 EXPERIMENTAL SETUP

The experimental setup tests whether PPT improves natural-language performance when compute is added or when it must compete with natural-language training. It varies training budgets and curricula while holding evaluation protocols and, in compute-matched experiments, total compute fixed.

3 EXPERIMENTAL SETUP

  • The study’s central question is whether PPT improves over a natural-language baseline in additive and compute-matched settings.The shared setup covers architecture, datasets, hyperparameter optimization, and evaluation.Source: §3 · p. 4
  • In additive-compute experiments, a 336M-parameter model uses natural-language budgets of 50M, 100M, or 1B tokens while PPT token counts and initialization schemes are varied.The tested PPT token counts range from 5M to 1B.Source: §3 · p. 4
  • Compute-matched runs allocate a fraction of total FLOPs to PPT followed by natural-language training, with α ∈ {0, 0.01, 0.05, 0.10}.The 10% share is used only in the data-constrained regime; α denotes compute share, not token share.Source: §3 · p. 5
  • The compute-matched configurations fix 100M natural-language tokens at 1 × 10^18 FLOPs, sweep 1, 3, and 9 × 10^18 FLOPs with abundant language, or train 881M parameters on 50B tokens.The over-Chinchilla configuration tests NCA and k-Dyck at 1% PPT; compute-optimal model size is searched at each data-unconstrained budget.Source: §3 · p. 5
  • Synthetic corpora include k-Dyck, NCA, and DeclRef, while code and C4 serve as additional pre-pretraining corpora or controls depending on the regime.Synthetic-language phases expand the vocabulary at the boundary and reinitialize the embedding table and language-modeling head; only the transformer body carries across.Source: §3 · p. 6
  • FineWeb held-out BPB is the primary metric, with C4, GitHub code, DCLM CORE, and BLiMP also evaluated; metrics are means over three seeds.DCLM CORE aggregates 22 benchmarks across knowledge, commonsense, reading comprehension, and symbolic reasoning.Source: §3 · p. 6

4 ADDITIVE COMPUTE

The additive-compute experiments reproduce an NCA gain under the original initialization, but show that an alternative initialization improves the baseline enough to remove the observed separation. At small natural-language budgets, PPT helps even against a well-initialized baseline; the result at 1B tokens depends on initialization.

4 ADDITIVE COMPUTE

  • 5.4%: NCA PPT improves perplexity over the no-PPT baseline with N(0, 1) embeddings, but the methods become indistinguishable when both use N(0, 0.02) embeddings.The alternative embedding initialization improves the baseline by 9.0%; the comparison uses a 400M-parameter Llama model trained on about 9B OpenWebText tokens.Source: §4 · p. 6, Figure 3 · p. 7
  • At 50M and 100M natural-language tokens, both PPT methods improve over the best no-PPT baseline, but at 1B tokens they fall within its seed spread under spectral-based initialization.Against alternative baseline initializations, both curricula still appear to improve at 1B tokens; the experiment does not retune PPT token count for the alternative initialization in the reproduced setup.Source: §4 · p. 7

5 COMPUTE-MATCHED

The compute-matched experiments ask whether synthetic PPT merits compute that could instead extend natural-language training; across tested regimes, it does not outperform that alternative. The section also reports limited additive-compute gains at small natural-language budgets and forgetting of PPT abilities after training shifts to natural language.

5 COMPUTE-MATCHED

  • Across tested compute-matched regimes, allocating compute to synthetic PPT never beats spending it on natural-language training.In the data-constrained setting, additional epochs over the same natural-language data are a better use of compute; neither NCA nor k-Dyck improves over baseline in the over-Chinchilla regime.Source: §5 · p. 7
  • With additional compute, NCA and k-Dyck improve over the best no-PPT baseline at 50M and 100M natural-language tokens, but the gain disappears by 1B under spectral-based initialization.Spectral-based initialization yields the lowest loss for every method at 1B tokens and is used in the compute-matched experiments.Source: Figure 4 · p. 8
  • At the lowest compute budget, only DeclRef and code show aggregate gains, C4 reaches a similar DCLM CORE range, and no PPT recipe scales better than baseline.The reported power-law fits show no PPT method with a lower multiplicative constant or significantly larger scaling exponent.Source: §5 · p. 8, Table 1 · p. 8
  • Some PPT methods improve individual benchmark tasks, often at the cost of others, leaving most aggregate performance on par.BLiMP phenomena can improve under PPT, consistent with previously reported k-Dyck gains.Source: §5 · p. 9
  • PPT methods have no consistent ranking across configurations or scales; at large token and parameter budgets their validation losses become very similar to baseline.DeclRef is marginally better than NCA and k-Dyck in the data-constrained regime, but ordering varies in the data-unconstrained regime.Source: §5 · p. 9
  • Front-loaded PPT models forget their synthetic-distribution ability at the PPT-to-PT boundary, while interleaving 5% PPT data throughout training prevents that forgetting.Interleaved runs continue improving on the PPT distribution; their FineWeb BPB is in a similar range to PPT counterparts, but runs were not hyperparameter-tuned.Source: Figure 5 · p. 9

7 LIMITATIONS

The reported comparisons have limits from reduced hyperparameter search at larger scales and from differences between this setup and prior work.

7 LIMITATIONS

  • Above 9 × 10^18 FLOPs, the authors could not run a full hyperparameter search; over-Chinchilla runs swept only peak learning rates.PPT shares were fixed at 1% and 5% in the data-unconstrained setting, leaving intermediate and budget-varying shares untested.Source: §7 · p. 10
  • BPB differences are not directly comparable to prior perplexity differences, and NanoChat also differs from prior work in tokenizer, architecture, and natural-language dataset.The authors note that the setup is held fixed across their baseline and PPT methods.Source: §7 · p. 10

8 DISCUSSION

Across compute-matched regimes, allocating compute to synthetic PPT does not outperform natural-language training, though additive-compute gains persist at small language budgets. The discussion also highlights initialization sensitivity, forgetting across the PPT-to-PT transition, and limits of the study’s evaluation scope.

8 DISCUSSION

  • Across three synthetic corpora and multiple regimes, compute-matched PPT never outperforms spending the same compute on natural language, including repeated training on 100M unique tokens.On most downstream DCLM CORE and BLiMP evaluations, results also match the baseline; exceptions are confined to the smallest compute budget.Source: §8 · p. 10
  • Models progressively lose PPT-distribution ability after natural-language training begins, including on code despite overlap between code and natural-language tokens.The authors recommend tracking synthetic-distribution ability throughout training as a proxy for retained inductive biases.Source: §8 · p. 10
  • At 881M parameters trained on 50B tokens, neither NCA nor k-Dyck improves on the baseline, and fitted scaling laws show no PPT method scaling better.The authors conclude these recipes are unlikely to overtake natural-language training at larger scales.Source: §8 · p. 10
  • Aggregate BLiMP and DCLM CORE scores mostly match the baseline, while individual BLiMP subtasks shift in both directions without a clear pattern.The study’s conclusions center on natural-language validation loss and do not examine modalities beyond language modeling.Source: §8 · p. 11
  • Small-budget gains persist when PPT receives additional compute, but the authors leave open whether better interleaving schedules make synthetic data more compute-efficient than natural language.Interleaving can avoid forgetting across a distinct PPT phase, but its comparative compute value remains unresolved.Source: §8 · p. 11

REPRODUCIBILITY STATEMENT

The released repositories provide the materials needed to inspect and reproduce the reported experiments.

REPRODUCIBILITY STATEMENT

AI USE STATEMENT

The authors used generative AI tools for coding, literature search, and editing, with human review and responsibility for the final work.

AI USE STATEMENT

  • Generative AI assisted with repository code, related-work discovery, and polishing the paper’s writing.Source: AI USE STATEMENT · p. 11
  • The authors reviewed and edited all AI-assisted output under close supervision.Source: AI USE STATEMENT · p. 11
  • LLMTuna uses a language model to propose hyperparameter configurations within a purpose-built search loop.The authors say it is not simply a coding agent and accept responsibility for the final content and artifacts.Source: AI USE STATEMENT · p. 11

A DATA

The data appendix describes three synthetic corpora: k-Dyck, neural cellular automata, and declarative-reference data. Their constructions target structured sequences, evolving grid dynamics, and context-dependent retrieval, respectively.

A.1 k-DYCK

  • k-Dyck uses the k-Shuffle Dyck generator with 128 bracket types, maximum nesting depth 16, and 2048-token sequences.The resulting vocabulary has 256 tokens.Source: §A.1 · p. 14

A.2 NEURAL CELLULAR AUTOMATA

  • NCA sequences come from randomly initialized convolutional update rules applied across 12 × 12 grids of ten colors, retaining rollouts with gzip compression ratios above 50%.Grids are serialized as 2 × 2 patches; training at 2048 tokens shows twice as much automaton evolution as the 1024-token setting described for prior work.Source: §A.2 · p. 14

A.3 DECLARATIVE-REFERENCE DATA

  • Declarative-reference documents use declare and reference statements within nested scopes, where names remain usable while in scope and enclosing-scope names persist after inner scopes close.The corpus targets context recall with little semantics and higher entropy than pure copying.Source: §A.3 · p. 15
  • Declarative-reference values are only partly predictable at declaration, while document-specific token frequencies and scope determine which references are possible.The design makes retrieval from the current document more important than memorization.Source: §A.3 · p. 16

B NCA REPRODUCTION EXPERIMENT

The NCA reproduction recovers a perplexity gain under the original initialization, but changing initialization narrows the gap to the no-PPT baseline. The section also describes FLOP-based compute matching and LLMTuna’s search process.

B NCA REPRODUCTION EXPERIMENT

  • 5.4% perplexity improvement from PPT was recovered under the original NCA initialization, while changing token-embedding initialization to N(0, 0.02) closed the gap with the no-PPT baseline.The reproduction used a 400M-parameter Llama model, 164M NCA tokens, and about 9B OpenWebText tokens.Source: B NCA REPRODUCTION EXPERIMENT · p. 16, Figure 6 · p. 17
  • Each run uses its tested initialization during PPT; the body carries over at the transition while the token embeddings and language-modeling head are redrawn under that initialization.The three schemes include the original setup, N(0, 0.02) embeddings and head, and NanoChat’s spectral-based initialization; the latter does not transfer exactly to Llama.Source: B NCA REPRODUCTION EXPERIMENT · p. 16

C FLOP MEASUREMENT

  • Compute matching converts a FLOP budget into steps and tokens using measured forward-and-backward FLOPs per optimization step, allowing PPT and baseline runs to be compared at equal compute.Because PPT vocabularies are smaller, PPT receives more steps per FLOP budget; this difference shrinks as models grow.Source: C FLOP MEASUREMENT · pp. 16–17

D.1 HOW LLMTUNA WORKS

  • LLMTuna proposes configurations from an ordered history of prior configurations and validation scores, using a language model that can also receive code, notes, and constraints as context.The search space specifies Float, Int, and Choice parameters with descriptions and optional bounds or initial values; suggest returns a structured proposal and observe records its score.Source: §D.1 · p. 17

D.2 BENCHMARK AGAINST OPTUNA

This section compares LLMTuna with TPE and describes how searches, tuned parameters, and the 881M-parameter over-Chinchilla runs were configured. The search uses bounded trials and validation BPB to guide proposals.

D.2 BENCHMARK AGAINST OPTUNA

  • LLMTuna reaches FineWeb validation BPB 0.9780 at trial 10, while TPE reaches 0.9808 at trial 22 and does not improve in its remaining eight trials.The plotted curves show best-so-far BPB by trial index, with faint dots for individual trials.Source: §D.2 · pp. 17–18, Figure 7 · p. 18

D.3 SEARCH CONFIGURATION

  • Searches are run separately for each setting’s budget and PPT configuration, with at most 30 trials and early stopping after five proposals fail to improve best validation BPB by more than 10^-3.Data-unconstrained searches carry history forward and warm-start from the previous budget’s best configuration.Source: §D.3 · p. 18

D.4 PROPOSER PROMPT

  • The proposer is instructed to balance exploration and exploitation, prioritizing depth, matrix learning rate, and PPT learning rate because their optima shift with scale.The prompt specifies validation BPB as the objective and notes that each of the three parameters needs broad enough search to detect a poor setting.Source: §D.4 · p. 19

D.5 HYPERPARAMETERS FOUND

  • During natural-language pretraining, Muon updates matrix parameters while embeddings and unembedding tables use AdamW; during PPT, all three groups share one learning rate.The reported configuration tables list only parameters that differ across cells; the other settings are shared across runs.Source: §D.5 · p. 19, Table 2 · p. 19

D.6 OVER-CHINCHILLA CONFIGURATION

  • The 881M-parameter over-Chinchilla run uses 50B tokens, about 57 tokens per parameter, with 0.5B tokens on PPT and 49.5B on FineWeb.It evaluates NCA and k-Dyck at 1% of total compute; peak learning rates are swept without an LLMTuna search.Source: §D.6 · pp. 19–20

E FULL RESULTS

Across the reported settings, the full results compare synthetic PPT with natural-language training on validation loss and downstream benchmarks. They cover additive compute, data-unconstrained and data-constrained budgets, an over-Chinchilla run, and benchmark definitions.

E FULL RESULTS

  • The results appendix defines evaluation protocols and reports configurations for additive, over-Chinchilla, retention, and benchmark runs.For additive BLiMP, the table caption says all methods perform similarly to the same-initialization baseline; the DCLM CORE caption defines significance relative to that baseline.Source: E FULL RESULTS · p. 20, Table 5 · p. 21, Table 6 · p. 21

E.1 ADDITIVE SETTING

  • In the additive setting, k-Dyck and NCA are reported to improve over the best no-PPT baseline at small natural-language budgets, while the advantage disappears by 1B tokens.Figure 8 compares FineWeb bits-per-byte across natural-language budgets, initialization schemes, and PPT amounts; its caption says PPT benefits remain with alternative initializations, while AdamW and Muon are indistinguishable at 1B tokens under spectral initialization.Source: Figure 8 · p. 22

E.2 DATA-UNCONSTRAINED SETTING

  • No PPT method beats the baseline at any tested data-unconstrained compute budget, and no method has a significantly larger scaling exponent than the baseline.The baseline has the lowest fitted A; increasing PPT from 1% to 5% raises A for every method, while exponent effects are unclear.Source: §E.2 · p. 24, Table 10 · p. 24, Figure 9 · p. 25

E.3 DATA-CONSTRAINED SETTING

  • In data-constrained experiments, neither PPT-phase optimizer reaches the no-PPT baseline; AdamW is lower for NCA and DeclRef at every PPT percentage, whereas Muon is lower for k-Dyck.Table 15 concerns BLiMP and DCLM CORE in the data-constrained regime, using 100M unique natural-language tokens at a fixed 1×10^18-FLOP budget.Source: Figure 10 · p. 27, Table 15 · p. 27

E.4 OVER-CHINCHILLA SETTING

  • In the 881M-parameter, 50B-token over-Chinchilla setting, no method separates from the baseline on BLiMP or DCLM CORE.The benchmark summary covers NCA and k-Dyck at 1%.Source: Table 16 · p. 28, §E.4 · p. 28

E.5 DOWNSTREAM BENCHMARKS

  • The downstream evaluation includes DCLM CORE, a 22-task suite spanning reasoning, language, and other benchmarks.The appendix states that BLiMP scores minimal pairs using total sentence log-probability, counting ties as errors and leaving the first token unscored.Source: E FULL RESULTS · p. 20, §E.5 · p. 28

Related papers

Loading 2610.40051v1…