PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Laws of Synthetic Data for Language Models

Zeyu Qin, Qingxiu Dong, Xingxing Zhang, Li Dong, Xiaolong Huang, Ziyi Yang, Mahmoud Khademi, Dongdong Zhang, Hany Hassan Awadalla, Yi R. Fung, Weizhu Chen, Minhao Cheng, Furu Wei

arXiv:2503.19551v3cs.CLcs.AI

TL;DR

With high-quality pre-training data diminishing, it is unclear whether synthetic data scales predictably like natural data. The paper introduces SYNTHLLM, which generates synthetic datasets from pre-training corpora, and finds that these datasets follow a rectified scaling law across model sizes, with diminishing gains beyond approximately 300B tokens.

  • Problem

    As high-quality web pre-training data diminishes, it remains unclear whether synthetic data follows predictable scaling laws comparable to natural pre-training data.

    Source: §1 · p. 1

  • Method

    SYNTHLLM transforms pre-training corpora into synthetic datasets by generating questions from reference documents and recombining concepts across documents with a graph algorithm.

    Source: §1 · p. 2, §3.2 · pp. 5–6

  • Results

    SYNTHLLM synthetic data follows a rectified scaling law across model sizes, with gains diminishing beyond approximately 300B tokens; an 8B model peaks at 1T tokens, compared with 4T for a 3B model.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings support synthetic data generated by SYNTHLLM as a scalable alternative to reduce reliance on human-generated data.

    Source: §1 · p. 2, §5 · p. 10

Abstract

from arXiv · show

Large language models (LLMs) achieve strong performance across diverse tasks, largely driven by high-quality web data used in pre-training. However, recent studies indicate this data source is rapidly depleting. Synthetic data emerges as a promising alternative, but it remains unclear whether synthetic datasets exhibit predictable scalability comparable to raw pre-training data. In this work, we systematically investigate the scaling laws of synthetic data by introducing SynthLLM, a scalable framework that transforms pre-training corpora into diverse, high-quality synthetic datasets. Our approach achieves this by automatically extracting and recombining high-level concepts across multiple documents using a graph algorithm. Key findings from our extensive mathematical experiments on SynthLLM include: (1) SynthLLM generates synthetic data that reliably adheres to the rectified scaling law across various model sizes; (2) Performance improvements plateau near 300B tokens; and (3) Larger models approach optimal performance with fewer training tokens. For instance, an 8B model peaks at 1T tokens, while a 3B model requires 4T. Moreover, comparisons with existing synthetic data generation and augmentation methods demonstrate that SynthLLM achieves superior performance and scalability. Our findings highlight synthetic data as a scalable and reliable alternative to organic pre-training corpora, offering a viable path toward continued improvement in model performance.

1 Introduction

As high-quality pre-training data dwindles, it remains unclear whether synthetic data can scale predictably like natural data. SYNTHLLM transforms pre-training corpora into synthetic datasets and shows predictable gains across model sizes.

Motivation

  • The central question is whether synthetic data follows predictable scaling laws like natural pre-training data, as high-quality web data is rapidly diminishing.The paper frames this as a test of synthetic data’s potential as a long-term response to data scarcity.Source: §1 · p. 1

Approach

  • SYNTHLLM filters domain-relevant web documents, generates questions or prompts through three complementary methods, and uses open-source LLMs to produce answers.It also extracts and randomly recombines high-level concepts across multiple documents using a graph algorithm, grounding generated questions in diverse references.Source: §1 · p. 2

Findings

  • SYNTHLLM data follows a rectified scaling law across model sizes, with gains diminishing beyond approximately 300B tokens; an 8B model peaks at 1T tokens versus 4T for a 3B model.The framework’s comparisons with existing synthetic generation and augmentation methods also show superior performance and scalability.Source: §1 · p. 2

2 Scaling Law of Synthetic Data

The paper situates synthetic-data scaling against established organic-data laws, then tests whether synthetic datasets follow a rectified law. In mathematical reasoning experiments, fitted curves predict performance, with diminishing returns beyond approximately 300B synthetic tokens.

2 Scaling Law of Synthetic Data

  • The section introduces synthetic-data scaling as its subject before presenting the SYNTHLLM generation approach in Section 3.The paper’s scaling analysis follows established work on organic-data scaling laws.Source: §2 · p. 2, §2.1 · p. 2

2.1 Scaling Laws for Organic Data

  • Organic-data scaling models predictive loss as a function of model parameters N and training tokens D, with fitted coefficients and exponents controlling convergence.The rectified law adds pre-learned data size Dl to capture task-relevant knowledge acquired during pre-training; the paper says this refinement better depicts pre-training’s influence on downstream performance.Source: §2.1 · pp. 2–3

2.2 Scaling Laws for Synthetic Data

  • The study focuses on mathematical reasoning, training 1B, 3B, and 8B Llama models on progressively larger synthetic-data subsets and evaluating MATH error rates.The paper identifies synthetic-data scaling as a first systematic empirical exploration and validation, and also reports preliminary coding experiments.Source: §2.2 · p. 3
  • Synthetic data generated by SYNTHLLM follows the rectified scaling law, while the vanilla form fails to model the results accurately; gains diminish beyond approximately 300B tokens.The authors hypothesize that this mismatch arises because the base models already have mathematical problem-solving capabilities from pre-training.Source: §2.2 · pp. 3–4
  • The 1B model has a smaller decay exponent β (0.34) than larger models, while the 3B and 7B models share identical β values.The paper attributes differences between the 3B and 7B models primarily to initial knowledge and model capacity, rather than synthetic-data scaling efficacy.Source: §2.2 · p. 4
  • At 4.5B synthetic tokens, fitted curves forecast MATH error rates of 40.0% for 3B and 28.7% for 8B models, corresponding to accuracies of 60.0% and 71.3%.The authors estimate that a 3B model needs around 4T tokens to approach its performance ceiling, while an 8B model needs about 1T.Source: §2.2 · p. 4

3 SYNTHLLM: Web-Scale Synthetic Data Generation

SYNTHLLM filters domain-specific reference documents, generates questions through three increasingly diverse document-grounded methods, and uses open-source LLMs to generate answers.

3.1 Reference Documents Filtering

  • Synthetic syllabus-based positives and randomly sampled Fineweb-Edu negatives train a cold-start classifier; iterative fine-grained classification expands references, retaining documents scoring above 6.5, with two iterations sufficient.Both classifiers use random forests trained on Fineweb-Edu document embeddings generated by StableLM-2-1.6B.Source: §3.1 · p. 5

3.2 Document-Grounded Question Generation

  • Level 1 extracts or rephrases existing questions and adds document-inspired questions, while Level 2 recombines extracted concepts to broaden generation beyond existing questions.Level 2 conditions question generation on the reference document and a randomly selected subset of its extracted topics and key concepts.Source: §3.2 · pp. 5–6
  • Level 3 samples related topics and concepts from a global co-occurrence graph, then grounds questions in the two reference documents most aligned with the sampled concepts.The graph includes topic-topic, topic-concept, and concept-concept edges; multi-document grounding is intended to broaden question diversity beyond a single document.Source: §3.2 · pp. 6–7

3.3 Answer Generation

  • The framework generates answers to its questions with open-source LLMs, such as Qwen2.5-Math-72B-Instruct.It does not currently add an answer-verification process; the authors cite satisfactory preliminary quality and leave validation improvements for future work.Source: §3.3 · p. 7

4 Experiments

SYNTHLLM outperforms other synthetic datasets across mathematical benchmarks, while its diversity-enhancing methods improve on direct extraction and augmentation baselines. Preliminary coding experiments also show gains over Level-1 and performance comparable to a code-specific model.

4.1 Experimental Settings

  • The experiments train models on the full 7.4M-sample SYNTHLLM dataset or a uniformly sampled 3.2M-example subset and evaluate across six math benchmarks.The benchmarks include MATH, GSM8K, OlympiadBench-Math, CollegeMath, Minerva Math, and Gaokao 2023 En.Source: §4.2 · p. 8, §4.1 · p. 8

4.2 Main Experimental Results

  • SYNTHLLM consistently surpasses other synthetic datasets across all benchmarks, and SYNTHLLM-8B (3.2M) significantly outperforms OpenMathInstruct-2 on out-of-distribution benchmarks despite its smaller dataset.On in-distribution MATH and GSM8K, the smaller SYNTHLLM dataset achieves comparable performance; the models also surpass similarly sized Llama Instruct models and match or outperform much larger models.Source: §4.2 · p. 8

4.3 Ablation Studies about Our SYNTHLLM Framework

  • On the 3B model, Level-2 and Level-3 outperform Level-1 across benchmarks; on the 1B model, Level-3 improves on most benchmarks except College and Gaokao.Level-1 yields 2.3M samples versus Level-2's 2.6M because direct extraction produces redundant questions that are later removed, limiting its scalability.Source: §4.3 · p. 9
  • Level-2 generates more diverse within-document questions than direct extraction, as shown by lower cosine similarity between generated question pairs.The comparison samples 2,000 documents and generates 50 questions per document for each method.Source: §4.3 · p. 9
  • Level-2 and Level-3 outperform rephrase and persona augmentation, continuing to gain as generation increases to 150 questions per document while those augmentation methods saturate.The comparison fixes 2,000 reference documents and evaluates datasets of 50K to 300K questions.Source: §4.3 · p. 9

4.4 Preliminary Experiments in Coding Domain

  • In coding, Level-2 and Level-3 improve over Level-1, and combined Level-123 outperforms the instruction-tuned model while performing comparably to the code-specific model.The reported comparison uses the Llama-3.1-8B base model.Source: §4.4 · p. 10

5 Conclusions and Future Work

The conclusion presents SYNTHLLM as a framework for studying synthetic-data scaling and reports that its data follows a rectified scaling law. It also identifies future work on data utilization and answer verification.

  • SYNTHLLM-generated data follows a rectified scaling law, enabling accurate performance prediction when dataset size doubles.The conclusion reports superior performance and scalability compared with existing synthetic-data generation and augmentation methods.Source: §5 · p. 10
  • The framework could extend beyond the studied setting to domains such as physics, law, and finance.Source: §5 · p. 10
  • Future work includes improving pre-training-data utilization and answer verification to improve accuracy.Source: §5 · p. 10

A Related Work

Prior synthetic-data methods commonly rely on scarce annotated seeds or direct extraction, while pre-training documents remain an underused resource. This work relates its scaling analysis to the rectified scaling law for fine-tuning.

Synthetic Data Generation

  • Conventional synthetic-data generation often uses a limited set of human-annotated seeds, constraining dataset diversity and scalability.Other approaches generate data from scratch using LLM uncertainty or predefined knowledge taxonomies.Source: A Related Work · p. 15
  • Pre-training data remains underused for scalable post-training synthesis, and SYNTHLLM is presented as a more efficient way to leverage web documents for diverse data.Prior work includes direct extraction from web documents and document backtranslation.Source: A Related Work · p. 15

Scaling Law of LLMs

  • This work extends the rectified scaling law for fine-tuning to show that it applies robustly when LLMs are fine-tuned with synthetic data.The cited research distinguishes this setting from scaling laws studied for organic data.Source: A Related Work · p. 15

B Details of Training Model and Our Dataset

The appendix details supervised fine-tuning, dataset construction, comparison datasets, augmentation baselines, evaluation benchmarks, and computational costs.

Training Settings

  • Training uses supervised fine-tuning with AdamW, a 1 × 10−5 learning rate, batch size 512, a linear scheduler, and three epochs.The largest model-training task used 32 AMD MI300 GPUs and took around two days; dataset generation used 32 Nvidia H100 GPUs and took around three days.Source: B Details of Training Model and Our… · pp. 15–16

Our Dataset

  • The math dataset contains about 520K filtered documents, yielding around 2.3M Level-1, 2.6M Level-2, and 2.6M Level-3 samples.Level-3 uses a concept graph of about 32,000 topics and 200,000 concepts, with five random-walk epochs; sampling was limited by training cost.Source: B Details of Training Model and Our… · p. 15

Statistics of Synthetic Questions and Answers

Other Baselines

  • The comparison datasets include MAmmoTH2, NaturalReasoning, JiuZhang3.0, OpenMathInstruct-2, and NuminaMath, with differing origins and construction.OpenMathInstruct-2 has 14M samples and uses GSM8K and MATH as seed datasets.Source: B Details of Training Model and Our… · p. 16
  • Rephrase and persona augmentation generate new questions from Level-1 extracted questions, using two-shot prompts and released personas, respectively.Source: B Details of Training Model and Our… · p. 16

Evaluation

C More Results

This section presents additional experimental results about Section 4.

C.1 Ablation Study about Our Three Distinct Methods

Level-2 and Level-3 generally outperform direct extraction, with the comparison varying by model size and benchmark.

  • On the 3B model, Level-2 and Level-3 outperform Level-1 and its instruct-tuned version across all benchmarks.Source: §C.1 · p. 16
  • On the 1B model, Level-3 improves over Level-1 on most benchmarks, except College and Gaokao.Source: §C.1 · p. 16
  • Level-1 yields 2.3M examples versus Level-2's 2.6M because duplicate questions from direct extraction are removed.Each document contributes five sampled questions; the authors identify high redundancy in direct extraction as a scalability limitation.Source: §C.1 · p. 16

C.2 Ablation Study on Scaling Law Form

On Llama-3.2-3B, the rectified scaling form fits and predicts synthetic-data scaling more accurately than the marginal form.

  • The rectified form captures the scaling curve and predicts performance when scaling up, while the marginal form fails at both.Source: §C.2 · pp. 16–17, Figure 6 · p. 17
  • Figure 6 plots MATH error rates against training-token counts, with green fit points and red prediction-test points.Source: Figure 6 · p. 17
  • The comparison underscores the importance of including the pre-learned size Dl in a fine-tuning scaling law.Source: §C.2 · p. 17

C.3 Ablation Study on Answer Verification Process

The answer-verification ablation finds only marginal performance gains from filtering noisy answers; the appendix also shows prompts used at three synthesis levels.

C.3 Ablation Study on Answer Verification Process

  • The preliminary experiment randomly selected 2,000 reference documents, generated 50 Level-2 questions per document and 8 candidate answers per question, then compared answer retention with and without filtering.Llama-3.1-70B-Instruct scored each question’s 8 candidates; answers judged “potentially incorrect” were discarded, and one remaining answer was randomly selected, versus one selected from all 8 without filtering.Source: §C.3 · p. 17
  • Filtering potentially incorrect answers yields only marginal gains on MATH and average performance across other benchmarks.Source: §C.3 · p. 18
  • Prior studies and these experiments suggest SFT performance is relatively robust to some noise in answer correctness.Source: §C.3 · p. 18

D Used Prompts

  • The prompts progress from Level-1 using a reference document to Level-2 adding extracted concepts and Level-3 adding graph-sampled metadata.The topic-and-concept extraction prompt takes a reference document as input.Source: Figure 7 · p. 19, Figure 8 · p. 20, Figure 9 · p. 21

E Examples about Extracted Topics and Key Concepts from Reference Document

The section illustrates topic and key-concept extraction from reference documents using Mistral-Large-Instruct-2407 and reports that the outputs are well structured.

Loading 2503.19551v3…