PaperlayerView exact revision on arXiv

Source-linked AI summary

BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining

DatologyAI, :, Pratyush Maini, Vineeth Dorna, Parth Doshi, Aldo Carranza, Fan Pan, Jack Urbanek, Paul Burstein, Alex Fang, Alvin Deng, Amro Abbas, Brett Larsen, Cody Blakeney, Charvi Bannur, Christina Baek, Darren Teh, David Schwab, Haakon Mongstad, Haoli Yin, Josh Wills, Kaleigh Mentzer, Luke Merrick, Ricardo Monti, Rishabh Adiga, Siddharth Joshi, Spandan Das, Zhengping Wang, Bogdan Gaza, Ari Morcos, Matthew Leavitt

arXiv:2508.10975v2cs.LGcs.CL

TL;DR

High-quality web text is increasingly scarce at modern pretraining scales, while the factors governing synthetic-data quality remain poorly understood. BeyondWeb studies rephrasing grounded in web documents and diversified across styles and formats. It reports improvements over web and synthetic-data baselines, while concluding that strong outcomes require jointly optimizing multiple factors.

  • Problem

    As pretraining budgets grow, high-quality information-dense web text becomes scarce, and the factors determining when and how synthetic pretraining data works remain poorly understood.

    Source: §1 · p. 2, §6 · p. 20

  • Method

    BeyondWeb rephrases real web documents and diversifies generated content across styles, formats, and topics to fill gaps in web-scale corpora.

    Source: §3 · p. 5, §6 · p. 20

  • Results

    +7.1pp over RedPajama and +2.6pp over Nemotron-Synth were reported for 8B models trained for 180B tokens on BeyondWeb.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Effective synthetic pretraining requires jointly optimizing source quality, deployment-style alignment, and generation diversity; no single factor suffices.

    Source: §6 · pp. 20–21

    High-quality synthetic data is challenging and potentially costly to generate, with no silver-bullet approach; naive methods can yield little benefit or detriment.

    Source: §1 · p. 3

Abstract

from arXiv · show

Recent advances in large language model (LLM) pretraining have shown that simply scaling data quantity eventually leads to diminishing returns, hitting a data wall. In response, the use of synthetic data for pretraining has emerged as a promising paradigm for pushing the frontier of performance. Despite this, the factors affecting synthetic data quality remain poorly understood. In this work, we introduce BeyondWeb, a synthetic data generation framework that produces high-quality synthetic data for pretraining. BeyondWeb significantly extends the capabilities of traditional web-scale datasets, outperforming state-of-the-art synthetic pretraining datasets such as Cosmopedia and Nemotron-CC's high-quality synthetic subset (Nemotron-Synth) by up to 5.1 percentage points (pp) and 2.6pp, respectively, when averaged across a suite of 14 benchmark evaluations. It delivers up to 7.7x faster training than open web data and 2.7x faster than Nemotron-Synth. Remarkably, a 3B model trained for 180B tokens on BeyondWeb outperforms an 8B model trained for the same token budget on Cosmopedia. We also present several insights from BeyondWeb on synthetic data for pretraining: what drives its benefits, which data to rephrase and how, and the impact of model size and family on data quality. Overall, our work shows that there's no silver bullet for generating high-quality synthetic pretraining data. The best outcomes require jointly optimizing many factors, a challenging task that requires rigorous science and practical expertise. Naive approaches can yield modest improvements, potentially at great cost, while well-executed methods can yield transformative improvements, as exemplified by BeyondWeb.

1 Introduction

BeyondWeb addresses the data wall and the limited scientific understanding of synthetic pretraining data by systematically studying rephrasing across model scales. Its findings favor targeted, diverse rephrasing, while showing that no single factor guarantees high-quality data.

1 Introduction

  • The paper studies when and how synthetic pretraining data works, with experiments spanning models up to 8B parameters.Despite growing use of synthetic data, the factors determining its quality remain poorly understood.Source: §1 · p. 2
  • In data-constrained settings, simple continuations offer limited gains over repeating data, while data filling web-distribution gaps offers larger benefits.The paper also finds that simple summarization can match generator-driven approaches such as Cosmopedia.Source: §1 · p. 2
  • Rephrasing high-quality sources helps, but high-quality inputs alone do not produce the highest-quality synthetic data.Source: §1 · p. 2
  • Web data contains 2.7% conversational content, and matching conversational style improves downstream performance, but gains saturate quickly as conversational data increases.Source: §1 · pp. 2–3
  • Diverse generation strategies are critical for sustaining synthetic-data benefits at trillion-token training budgets.Source: §1 · p. 3
  • Synthetic-data benefits are broadly consistent across rephraser families, rephraser quality does not predict data quality, and 8B rephrasers offer only marginal gains over 3B.The reported size trend suggests effective generation need not require massive computational resources given the right recipe.Source: §1 · p. 3
  • +7.1pp over RedPajama and +2.6pp over Nemotron-Synth, with 7.7x and 2.7x training speedups, respectively, were achieved by 8B models trained for 180B tokens on BeyondWeb.A 3B BeyondWeb model also surpasses strong 8B Cosmopedia baselines; the work reports that BeyondWeb was part of the curation platform for AFM4.5B’s 7T-token pretraining dataset.Source: §1 · p. 3
  • High-quality synthetic data has no silver-bullet recipe: strong results require jointly considering selection, generation, diversity, and quality control.Naive generation can yield little benefit or detriment while incurring substantial computational cost.Source: §1 · p. 3

2 A Tale of Two Approaches for Synthetic Pretraining Data

The section contrasts generating training data from models with rephrasing existing web documents, then explains why the paper focuses on pretraining. It situates that focus against synthetic-data uses in fine-tuning and evaluation.

2.1 The Generator-Driven Paradigm: Creating Knowledge from Models de Novo

2.2 The Source Rephrasing Paradigm: Enhancing Existing Knowledge

  • Source rephrasing transforms existing web documents into structured or targeted formats with smaller models, reducing dependence on expensive generators while retaining natural, diverse, topical language.WRAP reported pretraining speedups above 3× (Maini et al., 2024); the paradigm also includes Nemotron-CC (Su et al., 2024), multilingual augmentation (Pieler et al., 2024), and Phi-4 (Abdin et al., 2024b).Source: §2.2 · p. 4

2.3 Synthetic Data Beyond Pretraining

  • Synthetic data also serves fine-tuning and evaluation, including task-oriented training, behavior alignment, toxicity reduction, and downstream generalization.Source: §2.3 · p. 4
  • The paper focuses exclusively on pretraining because research on synthetic data for pretraining is relatively limited and successful pretraining improvements can have substantial impact.Source: §2.3 · pp. 4–5

3 Introducing BeyondWeb

BeyondWeb uses targeted, diverse rephrasing to create synthetic pretraining data that combines web coverage with content designed to fill gaps. Across model sizes, it improves benchmark performance over the reported baselines and can reduce the tokens needed to match their performance.

BeyondWeb approach

  • BeyondWeb combines web-scale coverage with targeted format transformation, style modification, and content restructuring to create diverse, information-dense data.Examples include converting web content into question–answer pairs and modifying style to enhance pedagogical tone.Source: §3 · p. 5

Performance improvements across scales

  • At 8B parameters, BeyondWeb achieves 63.7% average accuracy, 7.1 percentage points above RPJ and 2.6 percentage points above Nemotron-Synth, its strongest synthetic pretraining data baseline.Source: §3 · p. 6

Training efficiency

  • For an 8B model, BeyondWeb matches or exceeds RedPajama’s 180B-token performance in 23.2B tokens, a 7.7× speedup, and Nemotron-Synth’s in 66.2B tokens, a 2.7× speedup.The paper describes the resulting savings in training cost, energy, and iteration time.Source: §3 · p. 6

Performance and model scale

  • At 180B tokens, BeyondWeb’s 3B model achieves 60.8% accuracy and outperforms all but the strongest 8B baselines.The comparison is presented as a new speed–accuracy Pareto frontier.Source: §3 · p. 6

Consistent improvements across tasks

  • At 1B, 3B, and 8B, BeyondWeb beats baselines on 13, 12, and 13 of 14 tasks, respectively, and has the highest average accuracy at each scale.The table reports averages across tasks and 0-shot and 5-shot settings.Source: Table 1 · p. 7, §3 · p. 7

4 Systematically Evaluating Synthetic Data

The systematic evaluation examines synthetic data’s benefits, rephrasing choices, and generator-model properties against challenges in web-only pretraining. Its results show that outcomes depend on how synthetic data is designed and on the quality of the material being rephrased.

4 Systematically Evaluating Synthetic Data

  • The evaluation addresses web-data redundancy, style mismatch, underrepresented domains, and the value of maximizing per-token knowledge utility.The paper frames its systematic evaluation around synthetic-data capabilities, rephrasing techniques, and generator-model properties.Source: §4 · p. 7

4.1 Experiment Setup

  • The baseline uses RedPajama high-quality web data; unless an ablation specifies otherwise, experiments train a Llama-3.2-1B model for 20B tokens with a 50:50 real–synthetic split.Synthetic data is generally rephrased from the same 10B tokens of high-quality web data included in the mixture.Source: §4.1 · p. 8

4.2 Research Question (RQ) 1: Are Generator-Driven Approaches Approximated by Summarization?

  • 46.7% versus 47.1%: simple summarization nearly matches Cosmopedia, while BeyondWeb reaches 50.4% and exceeds summarization by +3.7pp.The authors identify increased per-token information density as one possible mechanism, not the only one.Source: §4.2 · p. 9

4.3 RQ2: Can Synthetic Data Break the Data Wall?

  • 50.4% accuracy: BeyondWeb exceeds the full-data upper bound of 46.2% by +4.2pp, while simple continuation reaches 46.2%.The comparison uses controlled 20B-token training corpora; the authors note that the continuation model had seen more than the 20B tokens available in the full-data experiment.Source: Figure 4 · p. 12, §4.3 · p. 12

4.4 RQ3: How Important is the Quality of the Seed Data for Rephrasing?

  • 49.2% for HQ Synth + HQ Web exceeds 48.6% for LQ Synth + HQ Web, but BeyondWeb’s 50.4% shows that high-quality seed data alone does not yield the highest-performing synthetic data.The experiment rephrases the same high-quality web data that is also included in the training mixture.Source: §4.4 · p. 13

4.5 RQ4: Is Distributional Style Matching Important?

Conversational-style matching improves performance modestly before saturating. Diverse generation sustains gains during extended training, while synthetic-data quality varies little across generator families and shows diminishing returns beyond moderate rephraser sizes.

4.5 RQ4: Is Distributional Style Matching Important?

  • Accuracy rose from 43.2% at the 3.67% RPJ baseline to 44.1% with 50% conversational content, with gains saturating beyond 20% under a 20B-token training budget.Source: §4.5 · p. 15, Figure 6 · p. 15

4.6 RQ5: How Important is Diversity at Scale?

  • At 8B scale, BeyondWeb’s improvements over RedPajama continued to increase, while fixed-style strategies tended to saturate; at 1B, it maintained positive trends around 50× beyond Chinchilla-optimal compute.Other baselines began to overfit and degrade in the 1B regime.Source: §4.6 · pp. 16–17, Figure 7 · p. 16

4.7 RQ6: How Important is the Synthetic Rephraser Model Family?

  • Synthetic datasets from four generator families improved over the 45.5% RPJ baseline, but their quality differed by less than 1 percentage point and rephraser benchmark quality did not positively correlate with data quality.OLMo-2-7B produced the highest-quality data at 49.9%, despite the lowest general benchmark accuracy among the tested rephrasers.Source: §4.7 · p. 17

4.8 RQ7: Does Rephraser Size Matter?

  • A 3B generator yielded 48.8% versus 47.3% for 1B, while an 8B generator yielded 49.2%, only 0.4pp above 3B.The reported 3B-to-8B gain suggests diminishing returns in this setup.Source: Figure 9 · p. 19, §4.8 · p. 19

5 Future Directions

The paper identifies future work on synthetic-data scaling and repetition, smaller rephrasers, alignment, and applications beyond web text.

Scaling Laws and Inherent Repetition

  • Synthetic data may contain repetition shaped by generator architecture and prompting, motivating new scaling laws and metrics to quantify its diversity and value.The authors identify possible fundamental limits or opportunities that may differ from real-data scaling laws.Source: §5 · p. 20

Democratizing Synthetic Data Generation

  • The authors propose investigating the minimum rephraser size that preserves quality, since rephrasing may not require extensive world knowledge and smaller models could reduce generation costs.Source: §5 · p. 20

Alignment with Human Values

  • Controlled synthetic-data generation could support pretraining methods that explicitly promote human-value alignment as an alternative to post-hoc techniques.Source: §5 · p. 20

Applicability Across Data Domains

  • Source rephrasing could extend to domain-specific or proprietary data and other modalities without requiring the rephraser to have domain expertise.The authors identify these settings as directions for future exploration.Source: §5 · p. 20

6 Conclusion

The conclusion presents BeyondWeb as diverse, web-grounded source rephrasing for pretraining and identifies multiple factors behind synthetic-data quality. It argues that strong outcomes require jointly optimizing those factors through rigorous source rephrasing.

6 Conclusion

  • BeyondWeb addresses the data wall with a rephrasing-driven approach grounded in real web documents and diversified across styles and formats.Source: §6 · p. 20
  • The paper recommends high-quality sources, style alignment with deployment, and diverse generation, while finding that no single factor is sufficient and each factor’s isolated contribution is often modest.Source: §6 · pp. 20–21
  • High-quality synthetic pretraining data is challenging and potentially costly to generate, but the authors report that rigorous source rephrasing can yield strong outcomes without a silver-bullet solution.Source: §6 · p. 21

7 Contributions and Acknowledgements

This section credits the research, infrastructure, and leadership teams, names contributors, and acknowledges additional support.

7 Contributions and Acknowledgements

  • Core Research credits Pratyush Maini and Vineeth Dorna.Source: §7 · p. 21
  • Core Infra credits Parth Doshi, Aldo Carranza, Fan Pan, Jack Urbanek, and Paul Burstein.Source: §7 · p. 21
  • The acknowledgements name leadership, contributors, and colleagues who supported the work with food, coordination, logo design, and draft feedback.Source: §7 · p. 21

A.1 Additional Training Details

Training uses a shared AdamW setup, with model-size-specific warmup lengths.

A.1 Additional Training Details

  • All trainings use AdamW with β1=0.9, β2=0.95, a learning rate of 5e−4, and weight decay of 1e−7.A linear warmup schedule uses 4K steps for the 1B model and 16K steps for the 3B and 8B models.Source: §A.1 · p. 25
  • Training uses fully sharded data parallelism (FSDP) with a batch size of 512 and a context length of 2048.Source: §A.1 · p. 25
  • The linear warmup lasts 4K steps for the 1B model and 16K steps for the 3B and 8B models.Source: §A.1 · p. 25

A.2 Evaluation Tasks

The evaluation uses 14 datasets selected to provide established measures of model performance across varied tasks.

A.2 Evaluation Tasks

  • The authors evaluate with lighteval on 14 datasets, including the eight used by FineWeb and additional established datasets.The additional datasets demonstrate above-chance performance and monotonic improvement during training.Source: §A.2 · p. 25
  • ARC-Challenge and ARC-Easy assess multiple-choice science question answering, with 2,590 and 7,787 questions, respectively.BoolQ adds naturally occurring yes/no questions with 15,942 examples.Source: §A.2 · p. 25
  • The broader task set includes commonsense reasoning, subject knowledge, reading comprehension, physical and social reasoning, and science questions.Examples include CommonsenseQA, COPA, HellaSwag, MMLU, OpenbookQA, PIQA, RACE, SciQ, SIQA, and WinoGrande.Source: §A.2 · p. 26

A.3 Quantifying Conversational Content in Web-Scale Corpora

The section estimates how much web text is conversational by applying GPT-4 classification to a sample from C4.

A.3 Quantifying Conversational Content in Web-Scale Corpora

  • The classification distinguishes interactive exchanges and question-answer pairs from one-way, non-interactive formal or encyclopedic text.The task labels texts as conversational (1) or non-conversational (0).Source: §A.3 · p. 26
  • Examples classify explanatory prose as non-conversational and dialogue or question-answer exchanges as conversational.The examples include explanations, instructions, and factual descriptions as well as coding help and everyday exchanges.Source: §A.3 · p. 27
  • GPT-4 classified 10,000 randomly sampled C4 texts, finding approximately 2.7% conversational.The authors describe this proportion as reflective of conversational-text prevalence in a large-scale web corpus.Source: §A.3 · pp. 26–27

B Additional Results

Across model scales, BeyondWeb leads through training and improves performance on most evaluated tasks; its 3B model also exceeds most 8B baselines at the same token budget.

B Additional Results

  • BeyondWeb outperforms other methods on most benchmarks at each scale across 14 tasks, while Figure 10 shows it leading throughout training.Source: B Additional Results · pp. 27–28, Figure 10 · p. 28, Table 3 · p. 28
  • 57.4%, 60.8%, and 63.7% are BeyondWeb's final average accuracies for the 1B-1TT, 3B-180BT, and 8B-180BT configurations, respectively.Average accuracy is the mean across 14 standard benchmarks and both 0-shot and 5-shot settings.Source: Figure 10 · p. 28
  • The 1B BeyondWeb model outperforms all but one 3B baseline, and the 3B model outperforms all but one 8B baseline at the same 180B-token budget.Source: Figure 11 · p. 29
Loading 2508.10975v2…