PaperlayerView exact revision on arXiv

Source-linked AI summary

Synthetic bootstrapped pretraining

Zitong Yang, Aonan Zhang, Hong Liu, Tatsunori Hashimoto, Emmanuel Candès, Chong Wang, Ruoming Pang

arXiv:2509.15248v3cs.CLcs.AI

TL;DR

Standard pretraining does not efficiently model inter-document correlations, a potential signal for better performance as high-quality text is depleted. SBP learns relations between documents in the pretraining corpus and synthesizes data for joint training; it consistently beats a repetition baseline and closes up to 60% of the gap to an oracle with 20x more unique data.

  • Problem

    Standard pretraining models token correlations within documents but is not designed to efficiently model potentially useful inter-document correlations, as high-quality text data is rapidly depleted.

    Source: §1 · p. 1

  • Method

    SBP identifies semantically similar document pairs, trains a conditional synthesizer from them, and applies it to the pretraining corpus to create data for joint training.

    Source: §1 · p. 1

  • Results

    SBP consistently surpasses the strong repetition baseline across pretraining scales and closes up to 60% of the performance gap to an oracle with 20x additional unique data access.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Qualitative analysis suggests SBP abstracts latent concepts to generate documents beyond paraphrases, with a Bayesian interpretation as latent-concept inference.

    Source: §1 · p. 2

    SBP is inherently a large-scale algorithm that cannot be scaled down.

    Source: §6.1 · p. 12

Abstract

from arXiv · show

We introduce Synthetic Bootstrapped Pretraining (SBP), a language model (LM) pretraining procedure that first learns a model of relations between documents from the pretraining dataset and then leverages it to synthesize a vast new corpus for joint training. While the standard pretraining teaches LMs to learn causal correlations among tokens within a single document, it is not designed to efficiently model the rich, learnable inter-document correlations that can potentially lead to better performance. We validate SBP by designing a compute-matched pretraining setup and pretrain a 3B-parameter and a 6B-parameter model on up to 1T tokens from scratch. We find SBP consistently improves upon a strong repetition baseline and delivers up to 60% of performance improvement attainable by an oracle upper bound with access to 20x more unique data. Qualitative analysis reveals that the synthesized documents go beyond mere paraphrases -- SBP first abstracts a core concept from the seed material and then crafts a new narration on top of it. Besides strong empirical performance, SBP admits a natural Bayesian interpretation: the synthesizer implicitly learns to abstract the latent concepts shared between related documents.

1 INTRODUCTION

SBP addresses the depletion of high-quality text by modeling inter-document correlations in a fixed pretraining corpus and using them to synthesize training data. It consistently surpasses a repetition baseline and closes up to 60% of the performance gap to an oracle with 20x additional unique data access.

1 INTRODUCTION

  • Standard pretraining models within-document token correlations, while the pretraining corpus also contains inter-document correlations that may provide an additional learning signal.Source: §1 · p. 1
  • SBP identifies semantically similar document pairs, learns their conditional relation, and applies the synthesizer to the pretraining corpus to create data for joint training.Source: §1 · pp. 1–2
  • The compute-matched evaluation trains 3B- and 6B-parameter models from scratch on up to 1T tokens.Source: §1 · pp. 1–2
  • SBP consistently surpasses the strong repetition baseline across pretraining scales and closes up to 60% of the performance gap to an oracle with 20x additional unique data access.Source: §1 · p. 2
  • Qualitative analysis finds synthesized documents go beyond paraphrases, while a Bayesian interpretation describes the synthesizer as abstracting latent concepts from documents.Source: §1 · p. 2

1.1 RELATED WORK

  • SBP’s self-boosting setup uses a fixed collection of pretraining documents as its knowledge source and requires minimal human intervention.Source: §1.1 · p. 3

2 OUR METHOD

Standard pretraining treats documents as independent samples, overlooking shared structure. SBP pairs related documents, trains a conditional synthesizer, and jointly trains on original and synthesized corpora.

2.1 DATA-CONTRAINED PRETRAINING SETUP

  • Standard pretraining treats each document as an independent sample and learns its marginal distribution, overlooking structural similarities shared across texts.Source: §2.1 · pp. 3–4

2.2 SYNTHETIC BOOTSTRAPPED PRETRAINING

  • SBP uses approximate nearest-neighbor search and inner-product similarity to select document pairs whose scores exceed threshold α.Source: §2.2 · p. 4
  • SBP maximizes the conditional probability of d2 given d1, initializing from pretrained weights; multiple d2 targets per d1 encourage diverse, high-entropy synthesis.Source: §2.2 · p. 4
  • Sampling a random seed d1 from Dpretrain and then d2 from pθST(·|d1) creates Spretrain, with variation from both seed diversity and conditional entropy.Source: §2.2 · p. 4

3 EXPERIMENT SETUP

The experiments use a cleaned document corpus, 3B- and 6B-parameter models, and nine benchmarks to evaluate SBP under compute-matched comparisons with repetition and oracle baselines.

3.1 DATA, MODEL, AND EVALUATION

  • The study curates 582M high-quality documents totaling 482B tokens, after deduplication, cleaning, and filtering documents longer than 4,096 tokens.The synthesizer context window is 8,192 tokens, so filtering allows both paired documents to fit within it.Source: §3.1 · p. 5
  • The authors evaluate 3B- and 6B-parameter Llama 3-based models on nine benchmarks covering test loss, world knowledge, and commonsense reasoning.The 3B model uses QK-norm, and the evaluation includes perplexity and QA accuracy.Source: §3.1 · p. 5, §3 · p. 5

3.2 COMPUTE-MATCHED COMPARSION

  • SBP is validated at 200B tokens with 10B-token data access and at 1T tokens with 50B-token access; its 1T oracle uses all 482B available unique tokens as a surrogate.Source: §3.2 · p. 6
  • The repetition baseline reuses the pretraining dataset 20 times, while the oracle upper bound supplies as many unique documents as the data pool allows.Training compute is controlled by total tokens seen, which is proportional to training FLOPs at fixed batch size and context window.Source: §3.2 · p. 6

4 EXPERIMENT RESULTS

Across three compute scales, SBP improves on the repetition baseline, while synthesized documents show new framing and generally preserve diversity and seed relevance. The results also show better factuality at larger scales.

4.1 MAIN BENCHMARK PERFORMANCE

  • SBP initially trails the baseline and oracle, then continues scaling after the baseline plateaus, suggesting synthetic data provides a signal absent from the real dataset alone.The training-dynamics comparison is at 200B scale.Source: §4.1 · p. 7, Figure 2 · p. 7
  • SBP improves over the repetition baseline at all three scales, recovering 42% of oracle QA-accuracy improvement at 200B, 48% for the 3B model at 1T, and 58% for the 6B model at 1T.The oracle has access to 20x more unique data; the 6B model also has a higher optimal synthetic-data ratio.Source: §4.1 · pp. 7–8, §4 · p. 7

4.2 ANALYSIS OF SYNTHETIC DATA

  • Synthesized examples move beyond paraphrase: one reframes a coffee-shop guide as espresso exposition, while another adopts a promotional comparison with New York.The examples stay on topic while varying in framing, depth, and fidelity to the seed.Source: §4.2 · p. 8, Figure 3 · p. 8
  • Synthesized data matches real data on repetition and duplication checks, has a 6.5% non-factual rate for the 6B model, and 1T synthesis is more relevant than 200B synthesis while avoiding simple copying.The text cautions that diversity checks alone do not establish high-quality or educational content.Source: §4.2 · p. 9

5 STATISTICAL FOUNDATIONS OF SBP

The section interprets SBP through a hierarchical model in which documents share latent concepts, then explains how synthesizer training can learn concept inference without explicitly modeling that hierarchy.

5.1 A HIERARCHICAL CONCEPT MODEL FOR NATURAL LANGUAGE

  • The idealized model samples abstract concepts from a semantic space, then generates documents conditional on each concept.The transformer-paper and code example share the concept of a transformer neural network.Source: §5.1 · p. 10
  • Same-concept documents are probabilistically dependent, while standard pretraining learns only the marginal document distribution.The hierarchical model represents documents as generated conditionally on shared concepts.Source: §5.1 · p. 10
  • The synthesizer must infer a latent concept from seed d1, then generate d2; standard pretraining omits this posterior-inference signal.The section presents this as one explanation for SBP’s improvement.Source: §5.1 · p. 10
  • Table 3 examples show synthesized documents share a concept with their seed, despite not being simple paraphrases or summaries.The table places real-document summaries, inferred concepts, and synthesized-document summaries side by side.Source: §5.1 · p. 10, Table 3 · p. 11
  • Synthetic documents distill the synthesizer’s inferred structure into a student trained on both real and synthetic data.The synthesizer is cast as a more knowledgeable teacher that infers latent structure.Source: §5.1 · p. 11

5.2 FROM IDEALIZED MODELS TO LANGUAGE MODEL REALITY

  • For real language, transformers learn pθ(d2|d1) directly from paired data, without explicitly modeling the true hierarchical distribution.The section describes this as bypassing the need to model the unknown data-generating process.Source: §5.2 · p. 11

6 DISCUSSION

The discussion frames SBP as a way to exploit inter-document correlations while retaining transformers’ scalability, and notes both implementation and scaling-law questions for future work.

6 DISCUSSION

  • SBP offers another form of self-supervision as internet data is depleted, using inter-document correlations beyond next-token prediction.The discussion says other underused forms of supervision remain to be explored.Source: §6 · p. 11
  • SBP addresses transformers’ poor inductive bias for learning relations while preserving their parallelizability and scalability.The proposed approach encodes missed correlations through synthetic data.Source: §6 · p. 12

6.1 FUTURE DIRECTIONS

  • The implementation uses Qwen3-0.6B-Embedding for document embeddings; using the target model’s own activations is left as future work.The authors say that alternative would avoid relying on an external embedding model, but their larger-scale setup uses optimized vLLM infrastructure.Source: §6.1 · p. 12
  • SBP shows larger relative improvements with the 6B model across three evaluated (N, D) points, but a full scaling law remains unfit.Scaling down risks incoherent synthesis, and changing model or data size requires rerunning synthesizer tuning, data synthesis, and sometimes neighbor matching.Source: §6.1 · p. 12

6.2 CONCLUSION

  • SBP leverages inter-document correlations missed by standard pretraining and implicitly infers latent concepts, with synthesized text extending beyond simple paraphrases.The conclusion reports experiments on 3B- and 6B-parameter models trained from scratch for up to 1T tokens.Source: §6.2 · p. 12

A.1 SBP IMPLEMENTATION DETAILS

The implementation pairs documents through distributed ScaNN search, filters pairs and tunes the synthesizer, then generates and filters synthetic text at scale.

Nearest neighbor pairing

  • Documents are embedded as 1,024-dimensional vectors, then searched with ScaNN using 8-bit quantization and asymmetric sharding.The setup uses Qwen3-Embedding-0.6B for 10B- and 50B-document pretraining datasets.Source: §A.1 · p. 24
  • Salting key shards across searcher copies supports top-200 search over 60M documents within 155M CPU hours.Each value vector gets a ScaNN search tree; a key shard is assigned to each salted searcher copy.Source: §A.1 · p. 24, Figure 4 · p. 24
  • Pairs above similarity 0.75 are retained, then near-duplicates are removed using normalized text and 13-token shingle overlap.The threshold was selected to keep the synthesizer-tuning dataset tractable; higher similarity scores yield more relevant pairs but also more duplicates.Source: §A.1 · p. 24

Synthesizer-tuning

  • Synthesizer tuning uses a constant 5e-6 learning rate and 16M-token steps; longer tuning improves Pair-novelty.The authors found lower generated-text quality with a larger learning rate and cosine decay.Source: §A.1 · p. 25

Synthesis at scale

  • Generation uses temperature 1.0 and top-p 0.9, filters repeated 13-token shingles, and reaches 8.3K tokens per B200 second.Synthesis takes 2.5K, 4.2K, and 8.4K B200 hours at the 200B, 1T (3B), and 1T (6B) scales, respectively.Source: §A.1 · p. 25

A.2 ABLATION ON DATA MIXTURE RATIO

The best-performing models use mixtures of real and synthetic data, with the optimal synthetic share increasing from 3B to 6B. Mixture-ratio sweeps show that the preferred amount varies by model scale and benchmark.

A.2 ABLATION ON DATA MIXTURE RATIO

  • At 200B-scale, OpenWebText2 and LAMBADA perplexity decrease monotonically as synthetic tokens increase, while most QA benchmarks peak around 75B.The sweep evaluates 0B, 25B, 50B, 75B, and 100B synthetic tokens.Source: §A.2 · p. 25
  • At 1T-scale, 125B synthetic tokens performs best across 3B benchmarks except LAMBADA perplexity, while the 6B model’s optimum is around 250B.The 3B sweep evaluates 0B, 125B, and 250B; the 6B sweep evaluates the same amounts.Source: §A.2 · p. 25
  • The best-performing model uses a real-and-synthetic mixture, with the optimal share rising from 12.5% for 3B to 25% for 6B.The discussion attributes the mixture’s value to synthetic data providing a signal that real data alone cannot capture as repetition yields diminishing returns.Source: §A.2 · p. 25

A.3 RANDOM PAIRS AND EMBEDDING ANALYSIS

The random-pairs ablation tests whether SBP depends on specific inter-document correlations. SBP-generated documents remain more semantically relevant to their seed than outputs from a synthesizer trained on random pairs.

A.3 RANDOM PAIRS AND EMBEDDING ANALYSIS

  • The ablation trains the synthesizer on random document pairs and compares seed-to-output semantic similarity with SBP.Similarity is measured using Qwen3-Embedding-0.6B.Source: §A.3 · p. 25
  • SBP outputs score 0.66 in seed-document relevance, versus 0.32 for outputs from the random-pairs model.The paired training documents score 0.79 in semantic similarity, while randomly paired documents score 0.15.Source: §A.3 · p. 26, Table 4 · p. 26

B.1 ANALYZE CONCEPTS IN DOCUMENTS

The analysis characterizes concepts inferred from real documents along dimensions of subject domain and abstract role, then examines variation between real and synthesized documents. Together, the taxonomies describe topical breadth, conceptual modes, and document relations.

B.1 ANALYZE CONCEPTS IN DOCUMENTS

  • The analysis classifies inferred concepts by domain and by abstract type, separating subject area from the concept’s role or nature.Examples of domains include science, psychology, health, and culture; types include theory, method, comparison, and symbol.Source: §B.1 · pp. 26–27, Table 5 · p. 27, Table 6 · p. 28
  • Concept domains range from sociocultural phenomena such as Culture to individual health and wellbeing, while types include methods, events, comparisons, and analyses.Examples include immigration policy, the Mediterranean migrant crisis, and multidimensional poverty measurement.Source: §B.1 · pp. 27–28, Table 5 · p. 27, Table 6 · p. 28
  • Relations between real and synthesized documents include scope distinctions, causal connections, and contrasts.Examples include specific versus general, corruption leading to reform, and Constitution articles versus Articles of Confederation.Source: §B.1 · pp. 28–29, Table 7 · p. 29

B.2 FACTUALITY ANALYSIS

The factuality analysis finds more factual errors in synthetic than real data, with synthetic-data factuality improving as seed data increases. The appendix also describes a pre-run quality evaluation for large synthesis runs.

B.2 FACTUALITY ANALYSIS

  • SBP uses document-level correlations without human filtering for non-factual content, so generated outputs are expected to contain factual errors.The paper defines factuality as undefined for primarily subjective or opinion-driven documents, or those concerning personal, obscure, or unverifiable entities; other documents are treated as verifiable.Source: §B.2 · pp. 30–31
  • Synthetic data contains more factual errors than real data, but its factuality improves with more seed data and approaches that of real data.The analysis samples 10,000 documents from real data and synthetic datasets initialized with 10B or 50B seed tokens, then classifies them with LM-as-a-judge.Source: §B.2 · p. 30, Table 8 · p. 30
  • Reported synthetic-text errors include false football transfer and timeline claims, plus incorrect institutional, company-location, and certification details.The paper highlights historical events and certification standards as areas for rigorous fact-checking.Source: §B.2 · pp. 30–31

B.3 MIDEVAL PROMPTS

  • Before billion-token-scale synthesis runs, mideval tests a small data subset to maximize pair relevance while monitoring pair novelty and non-repetition.The quality of both synthesizer-tuning pairs and synthesized documents influences final-model performance.Source: §B.3 · p. 31

B.4 SYNTHESIZED DOCUMENTS FROM THE 1T-SCALE EXPERIMENT

The 1T-scale examples complement the earlier 200B-scale example, showing synthesized text as subjective commentary and as continuations of real documents.

B.4 SYNTHESIZED DOCUMENTS FROM THE 1T-SCALE EXPERIMENT

  • The 1T-scale examples supplement the 200B-scale example in Section 4.2.The section presents additional synthesized documents at the larger scale.Source: §B.4 · p. 34
  • The first-row synthesized texts contrast with factual production and release information by offering subjective commentary, opinions, and behind-the-scenes anecdotes about both films.In the second row, synthesized documents continue the real document.Source: Figure 9 · p. 34

C ADDITIONAL PRETRAINING RESULTS

The additional experiments validate a repeated-token proxy for unique data and compare scaling model size with SBP, finding their benefits appear orthogonal.

C.1 TWO EPOCHS VALIDATION

  • At 400B scale, 200B tokens repeated twice yield nearly identical performance to 400B unique tokens, validating the repeated-token proxy used for 1T-scale oracle experiments.The 1T-scale oracle setup uses 482B tokens repeated twice because the DCLM-baseline dataset contains 80% duplicates.Source: §C.1 · p. 34, Table 11 · p. 35

C.2 MODEL SCALING

  • The model-scaling experiment fixes the training-token budget and replaces the 3B-parameter model with a 6B-parameter model.Table 12 gives the 6B model setup; Table 13 reports the 200B-scale comparison, with 10B unique tokens for the 6B model.Source: §C.2 · p. 34, Table 12 · p. 35, Table 13 · p. 35
  • The 6B model consistently outperforms the baseline, while SBP and model scaling each perform better on some benchmarks, suggesting their benefits are orthogonal.The authors note the potential to combine both approaches for an even better model.Source: §C.2 · p. 35, Table 13 · p. 35
Loading 2509.15248v3…