PaperlayerView exact revision on arXiv

Source-linked AI summary

A Tale of Tails: Model Collapse as a Change of Scaling Laws

Elvis Dohmatob, Yunzhen Feng, Pu Yang, Francois Charton, Julia Kempe

arXiv:2402.07043v2cs.LGcs.AIcs.CL

TL;DR

As synthetic content increasingly enters training corpora, the paper asks how it changes neural scaling laws and whether learning continues to improve. It develops a scaling-law framework and reports that altered synthetic-data tails can erode scaling, while clean-data mixtures can mitigate collapse through grokking.

  • Problem

    As synthetic content increasingly enters training corpora, the paper asks how it changes neural scaling laws and whether learning continues to improve.

    Source: §1 · pp. 1–2

  • Method

    The authors derive scaling-law theory for synthetic-data tail changes and mixtures, then test predictions with Llama2-7B text fine-tuning and GCD transformer experiments.

    Source: §1 · pp. 2–3

  • Results

    Scaling laws cease to persist under altered, less heavy-tailed synthetic-data distributions, while mixing even a small amount of clean data mitigates collapse through grokking.

    Source: §1 · p. 3, §7 · p. 9

  • Takeaways & Limitations

    Careful mixing and data curation matter: learning plateaus and changed curves can otherwise prompt unintended early stopping.

    Source: §7 · p. 9

    The pure grokking result requires deterministic ground-truth labels; probabilistic text-like settings retain only an S-shaped remnant.

    Source: §4.3 · p. 8

Abstract

from arXiv · show

As AI model size grows, neural scaling laws have become a crucial tool to predict the improvements of large models when increasing capacity and the size of original (human or natural) training data. Yet, the widespread use of popular models means that the ecosystem of online data and text will co-evolve to progressively contain increased amounts of synthesized data. In this paper we ask: How will the scaling laws change in the inevitable regime where synthetic data makes its way into the training corpus? Will future models, still improve, or be doomed to degenerate up to total (model) collapse? We develop a theoretical framework of model collapse through the lens of scaling laws. We discover a wide range of decay phenomena, analyzing loss of scaling, shifted scaling with number of generations, the ''un-learning" of skills, and grokking when mixing human and synthesized data. Our theory is validated by large-scale experiments with a transformer on an arithmetic task and text generation using the large language model Llama2.

1. Introduction

The paper asks how synthetic data alters neural scaling and argues that AI generation can truncate or narrow heavy tails, degrading scaling and skills. It develops scaling-law theory for these effects and examines generation depth and clean/synthetic mixtures, with experiments on Llama2 and GCD transformers.

  • Synthetic regeneration can cut off or narrow heavy tails through finite-sample bias and generation choices such as top-p sampling or temperature scaling, undermining scaling and skills.The paper presents tail cutting and tail narrowing as distinct ways regenerated data changes the distribution.Source: §1 · p. 2
  • The theoretical predictions are empirically checked by fine-tuning Llama2-7B on about 2 million Wikitext-103 samples and training transformers on GCD prediction.Source: §1 · p. 2
  • The double-scaling analysis finds that finite tail cutoffs impose a cutoff-dependent error floor, while tail narrowing changes downstream scaling.The theory covers simplified non-autoregressive language models and toy bigram LLMs.Source: §1 · p. 3
  • For memory-limited associative models, the triplet scaling law relates sample size, embedding dimension, and frequency cutoff, with an exponent depending on the data tail and update algorithm.Source: §1 · p. 3
  • Across recursive generations, cutoff-related degradation scales multiplicatively with generation count; the authors suggest that long-run pollution could make learning impossible.Source: §1 · p. 3
  • Even a small clean-data fraction can halt the grokking plateau and resume error decrease; its duration grows as k^β/π, while π = 0 leaves it indefinite.The grokking result is stated for deterministic ground-truth labels.Source: §1 · p. 3

2. A Deterministic Infinite Memory Model

The section models deterministic, infinite-memory learning over Zipf-distributed data and analyzes how tail truncation and repeated AI-data generation alter test-error scaling. It also considers how the tail behavior of synthetic data changes the resulting scaling law.

2. A Deterministic Infinite Memory Model

  • The core model assumes infinite memory, deterministic ground-truth labels, and a Zipf input distribution whose β > 1 controls tail length.The paper says later sections lift the first two restrictions.Source: §2 · p. 4

2.1. What Causes Model Collapse ?

  • Finite training samples effectively cut off the data tail, while deliberate truncation can also cause tail cutting and lower-temperature generation narrows the tail.The section also notes that token-wise tail cutting can narrow sequence-level tails for autoregressive models.Source: §2.1 · p. 4

2.2. A New Scaling Law in the Hutter LLM

  • Etest ≍ T^−(β−1)/β + k^−(β−1): with AI training data cut off at rank k, test error plateaus once T ≳ k^β.The error is measured on the original distribution; increasing the AI-generated sample size beyond that threshold no longer improves performance.Source: §2.2 · p. 4
  • When synthetic data has a shallower tail exponent β′ < β, test error decays as Etest ≍ T^−(β−1)/β′ rather than plateauing.The finite-initial-sample setting likewise produces loss of scaling through sampling-induced tail cutting.Source: §2.2 · pp. 4–5

2.3. Collapse Over Multiple Generations of AI Data

  • Model collapse occurs iff n ≫ (T0/T)^c, with test-error loss scaling linearly in the number of generations.When T0 ≫ T, collapse need not occur for constant n; when T0 ≲ T, it eventually occurs.Source: §2.3 · p. 5

3. Mitigating Model Collapse via Data Mixing

The section studies whether curated tail data or a mixture of clean and AI-generated data can mitigate scaling collapse. It finds that tail data must be placed near the missing cutoff, while even a small clean-data share yields eventual renewed error decline.

3.1. Acquiring Missing Tail

  • Curating data at the chopped-tail boundary restores Etest ≍ T^−c, but sampling from a tail whose starting rank is farther out is worthless.The supported boundary is N/k → 1; for C > 1 the error retains the k^−α term.Source: §3.1 · p. 6

3.2. A Grokking Phenomenon

  • For any fixed clean-data fraction π > 0, test error eventually resumes declining after a plateau, with the plateau length of order k^β/π.The renewed decline follows the clean-data scaling law, up to a multiplicative factor depending on π.Source: §3.2 · p. 6
  • AI data can improve performance when combined with real data, but its benefit plateaus; for Treal ≪ k^β, the stated error scales with total sample size plus a k-dependent term.The section identifies regimes in which adding AI data can help, including the case of a stronger generating model.Source: §3.2 · p. 6

4. A Tailed Bigram Model

The section extends the Hutter model to probabilistic bigrams, establishes a clean-data scaling law, and analyzes how truncation, repeated generations, and real–AI mixtures affect test error.

4.1. The Hutter++ Algorithm

  • Hutter++ estimates each conditional next-token distribution from context-wise pair counts and measures test error by expected total-variation distance.The passage notes this metric may also inform autoregressive models evaluated with perplexity loss.Source: §4.1 · p. 7

4.2. A Scaling Law for Hutter++

  • With clean data, Hutter++ retains the classical scaling law: Etest ≲ T^-c, and for β in (1, 2) with permutation mappings, Etest ≍ T^-c.Here c = min(1 − 1/β, 1/2); the upper bound assumes the theorem’s stated finite-sum condition.Source: §4.2 · p. 7

4.3. Model Collapse in Probabilistic Setting

  • With top-k-truncated probabilistic data, test error is bounded by T^-c + k^-βc, where c = min(1 − 1/β, 1/2).The learner’s data are drawn from the conditional distribution restricted to its k most frequent outputs.Source: §4.3 · p. 7
  • The scaling result is also reported for autoregressive bigrams, and the paper states that its multiple-generation relation continues to hold for probabilistic distributions.The autoregressive bigram result is empirically confirmed in Figure 11; the multiple-generation statement refers to Equation (13).Source: §4.3 · pp. 7–8
  • The pure grokking result does not hold for probabilistic mixtures tested on clean data, where distribution distance imposes an error floor; an S-shaped scaling pattern remains.The pure form is limited to deterministic ground-truth labels.Source: §4.3 · p. 8

5. Capacity-Limited Memory Models: A Triplet Scaling Law

The section extends the Hutter model to finite-memory associative models, treating embedding dimension as capacity and deriving scaling across sample size, capacity, and frequency cutoff.

5. Capacity-Limited Memory Models: A Triplet Scaling Law

  • Embedding dimension d plays the role of model capacity in this finite-memory extension of the infinite-memory Hutter model.The model uses embeddings and a memory matrix to retain input/output pairs seen in training.Source: §5 · p. 8
  • The triplet scaling law is Etest ≍ T^-(1−1/β) + d^-cq + k^-(β−1), combining sample size, embedding dimension, and frequency cutoff.The exponent cq depends on β and the learner algorithm q.Source: §5 · p. 8
  • The triplet law is empirically confirmed, and the paper reports tapered-off synthetic-data scaling plus the same cross-generation scaling loss and grokking phenomena.The authors state that the latter phenomena carry over to this model as well.Source: §5 · p. 8

6. Experiments

Experiments span Hutter++, autoregressive bigrams, GCD-predicting transformers, and Llama2-7B; they also examine mixtures of clean and synthetic data and use top-p or temperature mechanisms beyond the theory’s tail assumptions.

6. Experiments

  • Hutter++ shows Gen 1 data-scalability degradation and rapid performance decline across generations, even when each later dataset comes from the preceding generation’s strongest model.That selection is intended to eliminate inadequate sampling as an explanation; the figure uses top-pinf = 0.95 and β = 3/2.Source: §6 · p. 8, Figure 6 · p. 9
  • Autoregressive bigrams evaluated with averaged test-set perplexity show scaling loss and progressive collapse across generations, consistent with the earlier Hutter++ findings.The experiments vary top-pinf inference, temperature, limited real data, and AI dataset size.Source: §6 · p. 9
  • On the GCD task, the transformer exhibits the predicted single-generation scaling law, skill unlearning with exclusively generated data, and grokking with mixtures.The generator is the model after 300M samples; learned GCD groups and their products are treated as skills.Source: §6 · p. 9
  • Across Llama2 generations, scaling decays; the second generation’s synthetic data has markedly diminished utility, and collapse is more pronounced than in the other studied settings.The model is fine-tuned with LoRA on Wikidata-103; evaluation uses the same final 32 tokens withheld from training.Source: §6 · p. 9
  • Mixing 2% original data with Llama-generated data significantly mitigates collapse and produces the predicted grokking curve.The blend is applied across all fine-tuning phases.Source: Figure 7 · p. 9, §6 · p. 9

7. Conclusion

The conclusion argues that synthetic data changes scaling behavior and calls for data curation that accounts for those changes.

7. Conclusion

  • With AI-generated data in training corpora, scaling laws cease to persist and test error tapers off as data distributions become less heavy-tailed.The authors report observing predicted phenomena in relevant large models across two settings.Source: §7 · p. 9
  • Mixing clean and synthesized data can introduce learning plateaus and altered curves that may lead to unintended early stopping.The conclusion describes these effects as part of the interplay between clean and synthesized data.Source: §7 · p. 9
  • The authors call for collapse-aware synthetic-data proliferation and better watermarking, arguing that original data will become more valuable.They frame this as moving beyond scale alone.Source: §7 · p. 9

Impact Statement

The Impact Statement frames synthetic data as a challenge to sustainable model performance and argues for preserving clean, high-quality data. The prior-work review covers scaling laws, self-consuming collapse, and mixed-data findings, while noting limited rigorous analysis of scaling decay in self-consuming loops.

Impact Statement

  • The authors present model collapse as a risk to sustainable performance and call for strategic integration of clean data and preservation of high-quality training datasets.They emphasize possible clean-data scarcity and implications for data-driven industries.Source: Impact Statement · p. 10

A. Prior Work

  • Shumailov et al. (2023) attribute collapse to finite-sample tail cutoffs and function-approximation error, and observe increasingly peaked distributions under iterative resampling.Their language-model experiment uses five-way beam search, which approximately generates low-perplexity data.Source: A. Prior Work · p. 14
  • Mixed-data studies report that synthetic-only retraining can degenerate, while sufficient clean data can preserve learning under technical conditions.The review also notes that prior work finds an initial performance boost from modest synthetic-data mixtures before later degradation.Source: A. Prior Work · p. 14
  • Prior empirical work reports collapse across image and text settings, including degradation with iterative retraining and declining linguistic-diversity metrics.The review cites image-generation studies and text experiments using nanoGPT on logical expressions.Source: A. Prior Work · p. 14
  • Synthetic-data benefits are reported especially when generation uses a different task or a stronger model, but the impact of generative augmentation remains unresolved.Burg et al. (2023) find that diffusion-generated data helps classification, while the diffusion model's pretraining data alone performs better.Source: A. Prior Work · p. 15
  • The review situates the work amid broad empirical and theoretical scaling-law research linking power-law error changes to data, model size, and compute.It also connects scaling with emerging skills and notes prior evidence for inverse scaling under flawed objectives or data.Source: A. Prior Work · p. 15
  • Few prior papers rigorously analyze scaling decay in self-consuming loops, and Mobahi et al. (2020) study a distinct, carefully optimized kernel distillation setting.The review distinguishes that setting from synthesized data in the wild, where such optimization cannot be assumed.Source: A. Prior Work · pp. 15–16

B. Complimentary Figures for Sections 2, 3 and 4

The supplementary figures illustrate scaling predictions and collapse behavior in Hutter-style models, including bigram variants and successive generations. The accompanying proof sections establish scaling regimes for tail-cut data and mixtures of clean and synthesized data.

B. Complimentary Figures for Sections 2, 3 and 4

  • The figures report a theorem-matching Hutter rate across training sizes, a cutoff-dependent asymptote, and grokking under the stated Zipf setting.Figure 8 uses β = 3/2; Figure 9 describes asymptotes for different initial sample sizes, and Figure 10 shows grokking for β = 2.Source: Figure 8 · p. 16, Figure 9 · p. 16, Figure 10 · p. 17

C.1. Proof of Theorem 2.1

  • For T around or above k^β, the cutoff error floor dominates, whereas below that scale the proof gives a constant-order term; the figure panels test the predicted cutoff scaling.The cutoff-dependent asymptote is k^−(β−1), compared with the uncut rate T^−(β−1)/β.Source: Figure 9 · p. 16, Figure 8 · p. 16, §C.1 · p. 19

B. Complimentary Figures for Sections 2, 3 and 4

C.2. Proof of Corollary 2.3

  • For Corollary 2.3, the proof concludes that test error scales as T^−c.Source: §C.2 · p. 20

C.3. Proof of Theorem 3.2 and Corollary 3.3

  • For a clean-data proportion π, the proof gives a clean-sample scaling regime when πT exceeds k^β and a cutoff-controlled regime otherwise.The corresponding orders are (πT)^−c and k^−βc, respectively.Source: §C.3 · pp. 20–21

C.4. Grokking for Tail Narrowing

  • With tail narrowing, error first follows the synthesized-data rate and later switches to the clean-data rate, recovering ordinary sample-size scaling.The transition occurs at the threshold defined from π and the tail exponents; clean data decreases test error.Source: §C.4 · p. 21

C.5. An interesting detour: Grokking for Fixed-size AI Dataset.

With a fixed-size AI dataset, scaling can reappear as clean data grows: the model initially benefits from AI data, then grokks and forgets it. The remaining proof sections derive mixed-distribution and classical Hutter error expressions and bound contextwise terms.

C.5. An interesting detour: Grokking for Fixed-size AI Dataset.

  • As clean data grows alongside a fixed AI dataset, the model can grokk and then forget the AI data, recovering the normal scaling law in clean-data size.The figure describes an interval of grokking after more real data becomes available.Source: §C.5 · p. 22, Figure 20 · p. 22

C.6. Proof of Theorem 3.1

  • The proof for Theorem 3.1 decomposes mixed-distribution test error into contributions from separate rank ranges and sample-size terms.The derivation identifies three summed contributions, including terms involving T^−c and k^−α − N^−α.Source: §C.6 · p. 23

D.1. Warm-up: Revisiting the Classical Hutter Setup

  • In the classical Hutter setup, expected error is determined by whether each rank was observed in training, recovering the classical Hutter result.The proof relates the probability of an unseen rank to (1 − p_i)^T.Source: §D.1 · p. 23

D.2. Proof of Theorem 4.1

  • Theorem 4.1's proof bounds contextwise terms using sample counts, with Jensen's inequality and binomial concentration controlling the resulting expectation.Source: §D.2 · p. 24

D.3. Proof of Theorem 4.2

The proof of Theorem 4.2 handles cutoff effects by capping context counts. The extensions allow general outcome-frequency distributions, while the capacity-limited memory result gives a triplet scaling law empirically illustrated in Figure 21.

D.3. Proof of Theorem 4.2

  • The Theorem 4.2 proof replaces context counts by counts capped at k^β, yielding a cutoff contribution of order k^−βc.Source: §D.3 · p. 24

D.4. Extensions

  • The extension allows any distribution ρ on positive integers, with the resulting scaling law depending explicitly on ρ.Choosing ρ(z) proportional to z^−β recovers the setup considered earlier.Source: §D.4 · p. 24

E. Proof and Illustration of Triplet Scaling Law (Theorem 5.1)

  • For capacity-limited memory, the Triplet Scaling Law depends on the tail exponent and the embedding-update algorithm, and Figure 21 is presented as its empirical confirmation.Tail-cutting at rank k changes the test-error expression, and Figure 21 illustrates the result.Source: E. Proof and Illustration of Triplet… · p. 25, Figure 21 · p. 25

F. Details and Results from the Autoregressive Bigram model with Perplexity

The appendix tests autoregressive bigram perplexity under top-p and temperature variations, targeting paired-bigram and generational settings. All described sequences have length 100.

F. Details and Results from the Autoregressive Bigram model with Perplexity

G. Details and Results on Transformer Arithmetic Experiments

In GCD learning, synthetic-data models stop improving while clean-data models continue to learn new GCDs. Mixing clean data with synthetic data yields delayed learning, with longer delays at higher synthetic-data fractions.

Transformer arithmetic experiments

  • The arithmetic task uses sequence-to-sequence transformers whose GCD predictions are deterministic, and whose correct outputs depend on the encoding base.For a GCD the model has not learned, it predicts the largest learned GCD that divides the true value.Source: G. Details and Results on Transformer… · p. 28
  • The experiments create dirty datasets from transformer-generated pseudo-GCD labels and compare training on them with training on correctly computed GCD labels.The correct and generated datasets use sampled integer pairs; the generated dataset can be made arbitrarily large.Source: G. Details and Results on Transformer… · p. 28
  • With fewer than 100M examples, models trained on AI data achieve better accuracy than models trained on correct data.The authors attribute this early advantage to generated data smoothing away hard cases and presenting a cleaner signal initially.Source: G. Details and Results on Transformer… · p. 28, Table 2 · p. 28
  • For all tested bases, models trained on correct GCD data learn new GCDs as training proceeds, whereas models trained on generated data do not exceed their original performance.The comparison covers models trained on 300M and 1 billion examples.Source: G. Details and Results on Transformer… · pp. 28–29, Table 3 · p. 29
  • After about 300M examples, models trained on AI-generated GCD data stop learning while clean-data models continue and learn skills unavailable to the synthetic-data models.The figures show test-loss curves for clean and generated data and the emergence of new groups of GCD skills.Source: G. Details and Results on Transformer… · p. 29, Figure 28 · p. 29, Table 3 · p. 29
  • The synthetic-data generators were trained on about 300M examples and correctly predict 22, 16, and 17 GCDs below 100 for bases 1000, 2023, and 4913, respectively.Clean-data models later exceed these skill counts in the reported experiments.Source: G. Details and Results on Transformer… · p. 31
  • Mixtures of clean and AI-generated data show grokking-like delayed learning, and the delay grows as the AI-generated fraction increases.The experiments compare mixtures with 9%, 27%, 50%, and 73% AI-generated data; Table 4 reports samples to learn a GCD unavailable to AI-trained models and the delay since the previous GCD.Source: G. Details and Results on Transformer… · pp. 29, 31, Table 4 · p. 31

H. Details of Experiments with Llama2

The Llama-2 experiments fine-tune successive models on generated Wikitext-103 continuations, and the reported mixed-data curve aligns with the predicted grokking pattern. A separate study finds that next-token tail cutting can narrow tails under perplexity.

H. Details of Experiments with Llama2

  • The Llama-2 7B study fine-tunes successive models on Wikitext-103 chunks, generating each new dataset from the preceding model’s completions.The task supplies the first 96 tokens of each 128-token chunk and trains on its final 32 tokens; the dataset contains about 2.2 million chunks.Source: H. Details of Experiments with Llama2 · p. 32
  • Training and evaluation are restricted to loss on the final 32 tokens to prevent information leakage across chunks.The experiments use LoRA with learning rate 5e−5, Adam, and dropout 0.1.Source: H. Details of Experiments with Llama2 · p. 32
  • The curve for mixing Llama-generated data with original data aligns with the grokking phenomenon predicted by the theory.Figure 34 uses a 90-to-10 generated-to-original data mixture beginning at generation 1.Source: H. Details of Experiments with Llama2 · p. 32, Figure 34 · p. 32
  • The Llama experiments use fine-tuning rather than training an LLM from scratch, which the authors did not run because the language experiments took weeks.The authors describe from-scratch training as a possible way to demonstrate scaling-law decay.Source: H. Details of Experiments with Llama2 · p. 32

I. More Studies on Tail Cutting and Tail Narrowing Effects

  • In the autoregressive bigram study, next-token tail cutting can narrow tails for sequence-level metrics such as perplexity.The authors suggest combinatorial factors in additive or multiplicative sequence measures as a likely explanation; Figure 35 uses top-p infinity 0.95.Source: §I · p. 33, Figure 35 · p. 33
Loading 2402.07043v2…