Source-linked AI summary
The Curse of Recursion: Training on Generated Data Makes Models Forget
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, Ross Anderson
TL;DR
The paper asks what happens when models train on data generated by earlier models, a question made urgent by model-generated content entering web-scale corpora. It formalizes and examines this process across generative model types, finding that Model Collapse progressively removes information from the original distribution. The paper concludes that sustained learning requires preserving access to genuine human-generated data.
Problem
The paper investigates what happens when text produced by earlier GPT versions forms most of later models’ web-scraped training data.
Method
The paper analyzes generational training theoretically and demonstrates the process in Gaussian Mixture Models, Variational Autoencoders, and Large Language Models.
Results
Model-generated training data cause Model Collapse: models progressively forget the true distribution, first losing its tails and later converging toward low-variance outputs.
Takeaways & Limitations
Sustained learning requires preserving access to the original distribution and genuine human-produced data, especially when distribution tails matter.
Takeaways & Limitations
The theoretical setup assumes generational datasets of i.i.d. samples and models the original distribution through a specified functional approximation and data mixture.
Abstract
from arXiv · showhide
Stable Diffusion revolutionised image creation from descriptive text. GPT-2, GPT-3(.5) and GPT-4 demonstrated astonishing performance across a variety of language tasks. ChatGPT introduced such language models to the general public. It is now clear that large language models (LLMs) are here to stay, and will bring about drastic change in the whole ecosystem of online text and images. In this paper we consider what the future might hold. What will happen to GPT-{n} once LLMs contribute much of the language found online? We find that use of model-generated content in training causes irreversible defects in the resulting models, where tails of the original content distribution disappear. We refer to this effect as Model Collapse and show that it can occur in Variational Autoencoders, Gaussian Mixture Models and LLMs. We build theoretical intuition behind the phenomenon and portray its ubiquity amongst all learned generative models. We demonstrate that it has to be taken seriously if we are to sustain the benefits of training from large-scale data scraped from the web. Indeed, the value of data collected about genuine human interactions with systems will be increasingly valuable in the presence of content generated by LLMs in data crawled from the Internet.
1 Introduction
The paper asks what happens when future models train on model-generated online content. It identifies Model Collapse as an inevitable degenerative process and argues that genuine human-generated data remain essential.
- The paper investigates the consequences of predecessors’ generated text becoming most of the training data for later GPT versions.
- Model Collapse appears across Gaussian Mixture Models, Variational Autoencoders, and Large Language Models, demonstrating its breadth across model types.
- Over generations, models first lose information in distribution tails, then converge toward low-variance point estimates unlike the true distribution.
- Access to genuine human-generated content is essential when the tails of the underlying distribution matter.
- Model Collapse is a degenerative learning process in which models forget improbable events as they become trained on their own outputs.
2 Related work
The paper distinguishes Model Collapse from established ideas such as catastrophic forgetting and data poisoning. Its setting involves multiple model generations whose outputs alter the training distribution.
- Continual learning and catastrophic forgetting: Continual learning studies sequentially supplied, changing data distributions, including task-free settings resembling the paper’s gradually shifting distributions.
- Examples of degraded OPT-125m text outputs illustrate successive generations trained on data produced by their predecessors.
- Continual learning and catastrophic forgetting: Model Collapse differs from catastrophic forgetting because later models misinterpret reality through reinforced beliefs rather than forgetting previously learned data.
- Continual learning and catastrophic forgetting: Catastrophic forgetting describes forgetting previous samples while learning new information, often addressed with regularisation or retained data.
- Data poisoning: Traditional data poisoning inserts malicious training examples to degrade deployed performance or trigger unintended behaviours, unlike generational self-contamination.
3 What is Model Collapse?
Model Collapse occurs when generated data pollute later training sets, causing models to mis-perceive the original distribution. Statistical and functional approximation errors compound this process across generations.
- Model Collapse means generated data pollute the next generation’s training set, causing early tail loss and later mode entanglement with little resemblance to the original distribution.
- The feedback process trains each generation on an ensemble combining prior data and newly generated samples, allowing model outputs to enter subsequent training.
- Causes of model collapse: Statistical approximation error is primary: finite resampling creates a non-zero probability of losing information at every step.
- Causes of model collapse: Functional approximation error is secondary and arises when the model family is insufficiently expressive or assigns likelihood outside the original support.
- Causes of model collapse: Combined inaccuracies commonly cascade, while overfitting can assign high density to unsupported low-density regions that are then sampled arbitrarily.
- Causes of model collapse: Floating-point representation introduces a smaller computational error that is fixable with more precise hardware and therefore less influential on Model Collapse.
4 Theoretical intuition
The paper develops theoretical intuition for model collapse under recursive training on generational data. Finite sampling and functional approximation errors compound across generations, causing increasing divergence from the original distribution and eventual loss of information.
- Learning with Generational Data: The generational-data process alternates functional approximation with resampling, producing the next generation from an approximated distribution and prior data.At generation i, the model approximates p_i with p_θi+1, then forms p_i+1 through a mixture involving the approximation and earlier distribution terms.
- Discrete distributions with exact approximation: Finite sampling first removes low-probability tail events and can ultimately drive a discrete distribution toward a delta function.States with probability q ≤ 1/M have expected sample counts below one, making their information likely to disappear during resampling.
- Single dimensional Gaussian: Sampling errors make successive Gaussian parameter estimates behave like a random walk, with variance diverging linearly when sample size remains constant.Theoretical scaling and Figure 4 both show drift in the estimated mean and standard deviation across generations.
- Single dimensional Gaussian: Maintaining finite distance from the original distribution requires sample sizes to increase superlinearly across generations, while zero expected distance requires infinite sampling at every step.Even superlinear sampling leaves a non-zero expected distance after finitely many steps.
- Noisy approximation model: Functional approximation error adds noise beyond sampling error, and the resulting lower bound retains the same superlinear sample-scaling requirement in more generic models.The paper separates statistical approximation error, caused by finite samples, from secondary functional approximation error caused by insufficiently expressive or overly expressive approximators.
- Noisy approximation model: Under the stated assumptions, the average distance from the original distribution grows across generations and can become infinite because errors compound over time.The analysis assumes unbiased mean updates and data-dependent error terms whose expected effects average out.
5 Evaluation
Across GMMs, VAEs, and language models, repeated training on generated data degrades models’ representation of the original distribution. The experiments show distributional drift, tail errors, and partial mitigation when original data are preserved.
- Gaussian Mixture Models: Within 50 GMM resampling iterations, the model’s perception of two underlying Gaussians changes substantially, eventually converging toward very low variance.The process begins with a good fit to the original data, but later generations lose the original distribution’s structure.
- Variational Autoencoders: Over VAE generations, distinct modes of the original distribution become entangled, and generated reconstructions increasingly appear unimodal.The original encoder places descendant-generated data progressively closer to the mean representation.
- Language Models: Language models trained on generated data can still learn the task, but performance degrades by 20 to 28 perplexity points without original data.This setting uses five training epochs and excludes the original training data after generation.
- Language Models: Preserving 10% of the original training data leads to better fine-tuning and only minor performance degradation across generations.The comparison uses ten training epochs with a random 10% of original data retained for each new generation.
- Language Models: Later language-model generations produce more sequences likely under the original model while also developing longer tails containing increasingly improbable errors.The original model evaluates individual sequence perplexities across generations.
- Language Models: Fine-tuning does not prevent Model Collapse: generations increasingly reproduce probable original sequences while introducing their own improbable sequences.Additional experiments with a repeating penalty show that repetition is not sufficient to explain the collapse effect.
6 Discussion and Conclusion
The discussion argues that automated LLM-generated content can scale data poisoning and erase low-probability events from future training data. Sustaining learning therefore depends on preserving access to original human-produced data and its provenance.
- Implications: Automated LLM-generated content differs from earlier poisoning attacks because it can operate at much greater scale.The paper connects this scale to the need to distinguish generated from other Internet content.
- Implications: Preserving low-probability events matters because they are often relevant to marginalised groups and help explain complex systems.The discussion frames these events as important targets of model preservation.
- Conclusion: The evaluation suggests a first-mover advantage because training on another generative model’s samples can shift the distribution and eventually cause Model Collapse.The paper recommends maintaining access to original data and additional data not generated by LLMs.
- Conclusion: Tracking the provenance of Internet content at scale remains unclear, raising the possibility that future models may lack pre-adoption human-generated data.The paper identifies community-wide coordination as one possible response.
A.1 Absorbing Markov Chain
The repeated sampling-and-fitting process is modeled as an absorbing Markov chain. As transient-state probabilities vanish, the process converges to absorbing states, while the time to collapse depends on the fitting procedure.
- Markov-chain formulation: The repeated sampling-and-fitting process is represented as a Markov chain whose transition probabilities correspond to the functional approximation scheme.The analysis studies the limiting behavior of repeated transition-matrix application.
- Transition structure: Q records transitions among transient states, while R records transitions from transient states to absorbing states.The block matrix also contains an all-zero transient-from-absorbing block and an identity absorbing-state block.
- Limit behavior: With probability one, the chain converges to an absorbing state because transition probabilities back to transient states vanish in the limit.For discrete distributions, absorbing states correspond to delta functions centered at non-zero-probability points of the original distribution.
- Limit behavior: The expected number of steps before absorption may be large, but it cannot be calculated without knowing the fitting procedure and therefore the matrix Q.The limiting argument establishes convergence, not a universal collapse time.
A.2 Alternative assumption for noisy approximations
Replacing orthogonality with bounded noise of sufficient size yields a similar model-collapse bound. The analysis and figures show that collapse persists under this alternative assumption and requires superlinear sample scaling to mitigate.
- Empirical illustration: Over generations, the logarithmic L2 distance between fitted GMM distributions begins to grow and can become large.Jumps at large sample sizes are attributed to fixed iterations and expectation-maximization precision.
- Alternative assumption: Bounded noise larger than the sample-mean deviation replaces the orthogonality assumption while preserving a similar theoretical result.The alternative assumption uses a generation-independent constant K to bound the extra noise term.
- Resulting bound: The resulting bound remains similar to earlier model-collapse bounds when the relevant sample-size term becomes large.The final two terms contribute at most order 3/2 under the stated conditions.
- Scaling requirement: Superlinear scaling is required to minimise the model-collapse lower bound even for more generic approximation models.This remains true when the next-step mean includes an additional bounded term of order at most 1/M_i.
- Empirical illustration: Later generations produce samples increasingly likely under the original model while also developing longer tails of samples it would never produce.The perplexity histograms illustrate both concentration toward high-likelihood samples and later-generation misperception.