PaperlayerView exact revision on arXiv

Source-linked AI summary

Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior

Atsuki Yamaguchi, Tatsuro Inaba, Joel Niklaus, Michal Štefánik, Aline Villavicencio, Nikolaos Aletras

arXiv:2609.39827v1cs.CLcs.AIcs.LG

TL;DR

Prior PPT evidence left open whether synthetic warm-up gains persist at larger scales, longer PT horizons, and mixed-domain data. This study evaluates five PPT tasks across four scales and mixtures, finding persistent gains without a consistent grammatical-prior explanation; long-range retrieval and web-text presence delimit the reported pattern.

  • Problem

    Prior PPT studies used models at or below 1B parameters, short PT horizons, and predominantly web-text data, leaving its effectiveness at larger scales and with diverse mixtures uncertain.

    Source: §1 · pp. 1–2

  • Method

    The study evaluates five PPT tasks across four model scales, four PT data mixtures, and PT budgets up to 100B tokens.

    Source: §1 · p. 2

  • Results

    PPT gains persist under parameter and PT-budget scaling, lack a consistent grammatical-prior explanation, arise with tasks requiring long-range retrieval, and diminish only when web text is absent.

    Source: §7 · pp. 9–10

  • Takeaways & Limitations

    Future PPT tasks may benefit more from targeting long-range retrieval than from mimicking natural-language grammar's hierarchical structure.

    Source: §7 · p. 10

    Compute constraints limit 7B evaluations to the Marin mixture, up to 75.5B PT tokens, and k-Shuffle Dyck.

    Source: §4.2 · p. 5

Abstract

from arXiv · show

Pre-pretraining (PPT) on synthetic non-natural language data improves token efficiency during language model pre-training (PT). Prior work attributes this gain to a grammatical prior, i.e., a structural inductive bias learned during PPT that transfers to natural language grammar. However, PPT has only been tested on models of at most 1B parameters and PT budgets below 2B tokens on predominantly web text. It is unknown whether PPT is effective at larger scales and under more realistic PT data mixtures that combine diverse sources (e.g., code and math). We therefore present a comprehensive study on PPT spanning five PPT tasks, four PT data mixtures, four parameter scales (500M to 7B), and PT budgets of up to 100B tokens. Our results demonstrate that the downstream performance and token efficiency gains of PPT persist at scale, e.g., saving at least 21B PT tokens at the 3B scale. However, in contrast to prior work, we find no consistent evidence that these gains stem from a grammatical prior. Downstream performance does not consistently align with grammatical acceptability across model sizes. Instead, we find that downstream gains arise from PPT tasks that improve long-range retrieval. Finally, PPT performance gains are robust to how PT data mixtures are composed and diminish only when web text is absent. Overall, PPT is a low-cost addition to PT, and future PPT task design should target long-range retrieval rather than natural language grammar.

1 INTRODUCTION

Prior PPT studies left open whether synthetic warm-up gains persist at larger scales, longer training horizons, and more diverse data mixtures. This study finds that gains persist, but they do not consistently track grammatical acceptability; instead, they are associated with long-range retrieval and depend on web text being present.

1 INTRODUCTION

  • Prior PPT evidence was limited to models at or below 1B parameters, short PT runs, and predominantly web-text corpora, leaving its behavior under larger-scale, mixed-domain training unresolved.Code and mathematics may supply structural signals similar to PPT, potentially making it redundant.Source: §1 · pp. 1–2
  • The study systematically evaluates five PPT tasks, four PT data mixtures, four model scales from 500M to 7B, and PT budgets up to 100B tokens.Evaluations include grammatical acceptability, verbatim retrieval, and ten general-capability benchmarks.Source: §1 · p. 2
  • PPT's downstream-performance and token-efficiency gains persist through PT budgets up to 100B tokens, saving at least 21B PT tokens at 3B and remaining positive at 7B.The study spans 500M to 7B parameters.Source: §1 · p. 2
  • PPT gains do not consistently align with grammatical acceptability; downstream gains arise only from PPT tasks that improve long-range retrieval.The paper therefore finds no consistent evidence for a grammatical-prior explanation.Source: §1 · p. 2
  • A 13× increase in mathematical data leaves PPT's downstream gain unchanged, while removing web text collapses it.This distinguishes the presence of web text from the proportion of code and mathematics.Source: §1 · p. 2

2 RELATED WORK

Prior work uses synthetic pre-pretraining to transfer structure from artificial sequences, but varies in the structures and tasks it studies. Whether its findings generalize remains uncertain given limited scales, training regimes, and evidence on mixed-domain data.

2.1 SYNTHETIC PRE-PRETRAINING

  • Modern decoder-only PPT warms up a randomly initialized model on a generated synthetic corpus, then uses its parameters to initialize natural-language pre-training.The embedding matrix and LM head may be reinitialized for the natural-language tokenizer.Source: §2.1 · pp. 2–3
  • PPT work has expanded beyond formal grammars to algorithmic, logical-derivation, cellular-automaton, and recurrent-structure tasks (Cheng et al., 2026; Lee et al., 2026; Guo et al., 2026).Earlier approaches also include artificial formal grammars (Papadimitriou & Jurafsky, 2020; Ri & Tsuruoka, 2022; Papadimitriou & Jurafsky, 2023).Source: §2.1 · p. 3

2.2 INTERPLAY BETWEEN PRE-PRETRAINING AND PRE-TRAINING

  • Prior PPT explanations invoke grammatical transfer, while code- and math-containing PT mixtures may themselves induce structural abstractions that could make PPT unnecessary (Kim et al., 2024).The interaction between PPT and such diverse mixtures had not been examined.Source: §2.2 · p. 3

2.3 SCALE, OPTIMIZATION DYNAMICS, AND WARM-UP RETENTION

  • Earlier evaluations' ≤1B scales, small batches, and ≤2B-token PT horizons leave open whether model capacity, gradient noise, or extended optimization changes PPT's apparent benefits (Guo et al., 2026).Larger models may internalize hierarchical abstractions, while extended optimization may dilute initialization biases.Source: §2.3 · p. 3

3 A SYSTEMATIC FRAMEWORK FOR EVALUATING PRE-PRETRAINING

The framework treats PPT as a small-corpus warm-up whose learned weights initialize PT, then tests generalization across PPT tasks, PT mixtures, model scales, and training budgets.

3.1 PROBLEM SETTING

  • PPT first minimizes loss on a much smaller corpus than PT, then uses the resulting weights to initialize PT; the evaluation varies task, mixture, scale, and budget.The paper compares this with standard PT from random initialization.Source: §3.1 · p. 3

3.2 PRE-PRETRAINING DATA

  • Five PPT approaches span formal languages, structured synthetic tasks without formal grammars, and a natural-language control.The set includes k-Shuffle Dyck and MP-Struct Core, Set and neural cellular automata, plus text sampled from PT data.Source: §3.2 · pp. 3–4

3.3 PRE-TRAINING DATA MIXTURE

  • Four PT mixtures range from C4 web text to curated mixtures with code and mathematics, including the STEM-weighted OLMo3 and filtered-web SmolLM3.The runs follow the mixtures’ reported data ratios.Source: §3.3 · p. 4, Table 1 · p. 4

3.4 MODEL SCALES

  • The study tests 500M, 1B, 3B, and 7B models, extending prior PPT evaluations beyond 1B parameters.The 3B setting tests whether capacity alone makes PPT redundant; 7B tests whether PPT helps further as capacity grows.Source: §3.4 · pp. 4–5

3.5 PRE-TRAINING BUDGET

  • The PT budgets address whether PPT gains persist beyond the roughly 2B-token runs typical of prior studies, using a standard 21B-token budget and an extended 100B-token budget.The standard setup uses 512 sequences per batch and a 4,096-token context window; the extended budget is tested at 3B.Source: §3.5 · p. 5

4 EXPERIMENTAL SETUP

The experiments keep model architecture and most training settings aligned across scales, comparing PPT against matched baselines. Evaluation covers downstream capabilities and linguistic competence, with results averaged over late-training checkpoints.

4.1 MODEL TRAINING

  • Models share the SmolLM3 architecture and tokenizer, with width, depth, and head counts varied; at 7B, the learning rate is lowered for stability.The optimizer, learning-rate schedule, batch size, context window, and precision otherwise remain identical.Source: §4.1 · p. 5
  • PPT models begin PT from the 500-step PPT checkpoint with optimizer moments and scheduler reset, while PT-Only starts from random initialization and Control uses held-out PT text for the preliminary phase.The PPT and PT-Only runs therefore differ only in initialization state.Source: §4.1 · p. 5, §4.2 · p. 5

4.2 EVALUATION

  • Evaluation spans ten downstream benchmarks in four categories and, for linguistic competence, BLiMP and verbatim retrieval; reported run results give mean and standard deviation over checkpoints from 5K to 10K.Figure 2 bars show mean paired downstream-score differences from PT-Only, with whiskers denoting ±1 SD.Source: §4.2 · pp. 5–6, Figure 2 · p. 6

5 RESULTS

Across most model-scale and data-mixture pairs, k-Shuffle Dyck PPT improves downstream performance, with gains also appearing across task types. These gains do not consistently align with grammatical acceptability; verbatim-retrieval results instead point to long-range retrieval as a more reliable capability.

5.1 GENERAL CAPABILITY

  • In 9 of 12 scale-mixture pairs, k-Shuffle Dyck improves downstream average by at least 0.6 points, with a mean gain of 1.6 among them.Mean gains are similar across scales: 0.8 at 500M, 1.4 at 1B, and 1.3 at 3B.Source: §5.1 · p. 6
  • Control stays close to PT-Only, while k-Shuffle Dyck outperforms Control in 10 of 12 pairs by mean margins of 1.0, 1.0, and 1.1 points across scales.This comparison supports the reported distinction between gains from synthetic PPT data and gains from additional optimization steps.Source: §5.1 · p. 6
  • Each task category improves in at least 9 of 12 scale-mixture pairs, and language modeling gains the most at all three scales.ReCoRD and HellaSwag improve in all 12 pairs, and LAMBADA in 10; all three require information from the preceding passage.Source: §5.1 · pp. 6–7

5.2 LINGUISTIC COMPETENCE

  • Only 1 of 12 BLiMP gains is stable, versus 10 of 12 downstream-average gains, so grammatical-acceptability results do not consistently track downstream gains.Neither morphology nor syntax shows a stable gain.Source: Figure 3 · p. 7, §5.2 · p. 7
  • k-Shuffle Dyck lowers verbatim-retrieval NLL in all 12 pairs, with 7 stable reductions, versus 1 stable BLiMP gain.Control improves verbatim-retrieval NLL in 8 pairs, with 3 stable reductions.Source: Table 2 · p. 7, §5.2 · p. 7
  • The authors attribute this contrast to retrieval demands: verbatim retrieval and k-Shuffle Dyck require locating earlier sequence content, unlike sentence-level BLiMP comparisons.They suggest this long-range retrieval capability explains why gains are largest on LAMBADA, ReCoRD, and HellaSwag, which depend on preceding context.Source: §5.2 · pp. 7–8

6 ANALYSIS

PPT benefits persist under extended PT budgets and larger models, and composition tests find that more mathematics does not diminish gains whereas removing web text sharply reduces them. Across synthetic tasks, gains align with retrieving specific earlier positions; the weak OLMo3 result remains unexplained.

PT Data Type

  • Raising mathematics from 1.3% to 17.0% leaves the average gain at 2.0 points versus 1.9 for Marin, while removing web text reduces it to 0.2 points.DCLM-only retains 1.5 points of average gain and FineWeb-Edu retains 1.0; gains diminish only in the web-free variant described.Source: §6 · p. 8
  • Neither mathematics share nor web share explains OLMo3's weak PPT gains; the authors leave a direct ablation and the responsible property for future work.The 17%-Math variant gains 2.0 points, while SmolLM3 gains 1.8 at 3B against 0.0 for OLMo3.Source: §6 · p. 8

Further Model and Data Scaling

  • Further runs were beyond the authors' computing capacity after the reported 3B experiments had consumed approximately 1.07T PT tokens.Source: §6 · p. 8

PPT Tasks

  • At 3B, k-Shuffle Dyck, MP-Struct Core, and NCA gain 1.3, 0.9, and 1.2 points on average, while Set degrades performance by 7.2 points.The effective tasks retrieve a specific earlier position; Set only tracks whether tokens have appeared before.Source: §6 · p. 9
  • The task results suggest long-range retrieval, rather than formal grammar, is the property associated with effective PPT tasks.This account converges with prior work on the relevant task property, while differing on what the model acquires.Source: §6 · p. 9

Further Model and Data Scaling

  • Across 14 extended-budget points, downstream average improves; at 3B on Marin, PPT reaches 62.3 after 63B tokens versus PT-Only's 62.0 after 84B, saving at least 21B PT tokens.The benefit also remains positive at every tested budget for the 7B Marin runs, though narrower than at 3B.Source: Table 4 · p. 9, §6 · p. 9

7 CONCLUSION

This study evaluates PPT across five tasks, four PT mixtures, four model scales, and budgets up to 100B tokens, finding that downstream benefits persist under scaling. It finds no consistent grammatical-prior account; gains instead align with long-range retrieval and diminish when web text is absent.

7 CONCLUSION

  • The systematic evaluation spans five PPT tasks, four PT data mixtures, four model scales from 500M to 7B, and PT budgets up to 100B tokens.Source: §7 · p. 9
  • Downstream gains persist under parameter and PT-budget scaling, but the attribution of those gains to a grammatical prior does not hold consistently.Source: §7 · pp. 9–10
  • PPT is effective when tasks require retrieving earlier sequence positions, and its gains diminish only when web text is absent.The authors suggest future PPT tasks may benefit more from directly targeting long-range retrieval than from mimicking natural-language grammar's hierarchical structure.Source: §7 · p. 10

AI USE STATEMENT

The statement reports how AI tools were used, and the appendix directory lists supplementary materials. Preprocessing standardizes synthetic corpora while documenting task-specific generation choices.

AI USE STATEMENT

  • Gemini and Claude tools assisted with PPT data-generation scripts, debugging, literature review, and draft editing.Antigravity with Gemini Flash extended DataTrove preprocessing and debugged Nanotron scripts; Gemini Pro assisted with the literature survey and, with Claude Opus and Gemini Flash, draft clarity.Source: AI USE STATEMENT · p. 10

Appendix Directory

  • The appendix directory groups supplementary material into experimental setup, results, and analysis.The listed setup topics are preprocessing, training, and evaluation; results include checkpoint sensitivity and random-seed sensitivity.Source: Appendix Directory · p. 18

A.1 PRE-PROCESSING DETAILS

  • Synthetic corpora use space-separated integers tokenized and packed so each document occupies one 4,096-token sequence.Symbol identifiers are limited to three digits so each integer maps to a fixed number of BPE pieces; Control uses the same tokenization and packing as DPT.Source: §A.1 · p. 18
  • The k-Shuffle Dyck and MP-Struct Core generators follow prior released configurations, with MP-Struct documents lengthened to 2,048 integers.k-Shuffle Dyck bracket depth is constrained to [1, 8]; MP-Struct Core retains its default structural and dependency settings.Source: §A.1 · p. 18
  • Set documents concatenate an input, a separator, and unique values in first-appearance order; NCA retains trajectories with GZIP compression ratio at least 0.50.NCA documents average 4,085.6 tokens (SD 206.7) because five-digit patch identifiers introduce subword-tokenization length variation.Source: §A.1 · p. 18

A.2 TRAINING DETAILS

Training spans four model scales and uses a common software environment across multiple GPU clusters. At 3B, the 500-step PPT stage accounts for 4.74% of total wall-clock execution time.

A.2 TRAINING DETAILS

  • The study evaluates 500M, 1B, 3B, and 7B models, with architectural and optimization hyperparameters summarized across scales.Table 6 provides the specifications and training hyperparameters for the four scales.Source: §A.2 · p. 18, Table 6 · p. 19
  • Runs use Nanotron, FlashAttention-2, DataTrove, datasets, and flash_adamw across NVIDIA GPU clusters in a uniform containerized environment.The clusters include A100, H100, H200, or GH200 devices; the environment uses the NVIDIA vLLM container, PyTorch 2.10.0, and CUDA 13.1.Source: §A.2 · p. 19

Compute Cost

  • 4.74% of total wall-clock execution time is spent on the 500-step PPT stage for the 3B model.On 16 A100 40GB GPUs, PPT takes 3 hours 32 minutes (about 56.5 GPU hours), compared with 2 days 23 hours (about 1,136 GPU hours) for 21B-token pretraining.Source: §A.2 · p. 19

A.3 EVALUATION DETAILS

The evaluation combines downstream task measures with BLiMP and verbatim retrieval, using distinct procedures for retrieval and other evaluations. Table 7 reports task-level downstream changes relative to PT-Only.

A.3 EVALUATION DETAILS

  • Evaluations other than verbatim retrieval use lm-evaluation-harness with default settings, automated batch sizing, and bfloat16 precision.BLiMP uses its twelve paradigm groups assigned to Semantics, Morphology, and Syntax.Source: §A.3 · p. 19
  • Table 7 reports task-level downstream performance changes from PPT relative to PT-Only.The table presents mean paired differences over checkpoints from 5K–10K steps, with standard deviations; green marks improvement.Source: Table 7 · p. 20
  • Verbatim retrieval scores five repeat-condition stimulus sets with mean NLL over each full stimulus, averaged across stimuli.Each context presents a noun list twice, with word-level markers distinguishing the first occurrence from the repeat; scoring uses a single teacher-forced forward pass.Source: §A.3 · p. 20

B SUPPLEMENTARY RESULTS

The supplementary results examine BLiMP task breakdowns, checkpoint and averaging-window sensitivity, and random-seed variation. Downstream effects are more directionally stable than BLiMP effects across these checks.

B SUPPLEMENTARY RESULTS

  • In BLiMP Morphology and Syntax, k-Shuffle Dyck changes are at most 1.0 point in 19 of 24 cases and unstable in 20; movement and variance concentrate in Semantics.Average checkpoint SD is 5.7 in Semantics, versus 1.5 in Morphology and 0.9 in Syntax.Source: B SUPPLEMENTARY RESULTS · p. 20, Table 8 · p. 21

B.1 CHECKPOINT SENSITIVITY OF SINGLE-SNAPSHOT ESTIMATES

  • Downstream single- and multi-checkpoint estimates never reverse sign and differ by 0.54 points on average, while BLiMP reverses sign in half of 12 pairs and differs by 1.50 points.A single BLiMP checkpoint can diverge from the multi-checkpoint estimate; Table 9 compares the 10K-step delta with the 5K–10K average.Source: §B.1 · p. 21, Table 9 · p. 22
  • Across averaging-window starts from 2K to 7K, downstream performance improves in 11 of 12 pairs and verbatim retrieval in all 12, while BLiMP reverses sign in five or six pairs.Mean downstream gains shift by at most 0.3 points, preserving the ranking across model scales.Source: §B.1 · p. 21, Table 10 · p. 22
  • Excluding the final cooling phase leaves all three scale means positive, with 10 of 12 downstream pairs improving and the 500M mean falling from +0.83 to +0.64.The ordering of scale means and the BLiMP and verbatim counts remain unchanged.Source: §B.1 · pp. 21–22, Table 10 · p. 22

B.2 SENSITIVITY TO RANDOM SEEDS

  • Across three seeds at 3B on Marin, the k-Shuffle Dyck downstream gain varies only from +1.7 to +1.9 points, and every downstream task category keeps its sign.The additional seeds resample initialization and data order while holding architecture and optimization hyperparameters fixed; the gain variation is within the reported 0.5 checkpoint SD.Source: §B.2 · pp. 23–24, Table 11 · p. 23
  • Across those seeds, BLiMP shifts from −0.3 to +0.9, while verbatim retrieval improves in two of three seeds and averages −0.033 NLL.The remaining retrieval seed is +0.014; the authors base the retrieval account on stable gains across task-mixture pairs rather than this single configuration.Source: Table 12 · p. 23, Table 13 · p. 23, §B.2 · p. 24

C SUPPLEMENTARY ANALYSIS

The supplementary analysis provides per-mixture and extended-budget breakdowns of PPT results. It also identifies PPT improvements in the k-Shuffle Dyck results.

C SUPPLEMENTARY ANALYSIS

Related papers

Loading 2609.39827v1…