PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning by Surprise: Adaptive Mitigation of Model Collapse in Large Language Models

Daniele Gambetta, Gizem Gezici, Fosca Giannotti, Dino Pedreschi, Alistair Knott, Luca Pappalardo

arXiv:2410.12341v4cs.CLcs.AI

TL;DR

As AI-generated content increasingly enters web training data, recursive fine-tuning raises concerns about model collapse, while existing understanding and mitigation strategies have limitations. The paper studies collapse with model-intrinsic measures and perplexity-based filtering; across datasets and LLM families, the filter mitigates collapse without requiring human-versus-AI labels.

  • Problem

    The paper addresses limited understanding of model collapse and mitigation strategies that rely on distinguishing human-authored from AI-generated content.

    Source: §1 · p. 2

  • Method

    The authors study collapse through next-token probability concentration and perplexity, then filter fine-tuning documents to prioritize high-surprise examples.

    Source: §1 · pp. 2–3

  • Results

    Across datasets and LLM families, perplexity-based filtering consistently mitigates collapse with performance comparable to, and sometimes better than, human-authored-data fine-tuning.

    Source: §1 · p. 3, §9 · p. 22

  • Takeaways & Limitations

    The results support a model-centric filtering approach that does not require labels identifying documents as human-authored or AI-generated.

    Source: §1 · p. 3, §9 · p. 22

    The findings are based on simulations, and human-authored prompts may mask collapse effects in fully synthetic generation loops.

    Source: §9 · p. 22

Abstract

from arXiv · show

As AI-generated content increasingly populates the web, generative AI models are at growing risk of being trained on their own outputs, a process known as AI autophagy. This feedback loop has been shown to induce model collapse, typically characterized by a loss of diversity in generated content. However, existing work offers a limited understanding of this phenomenon and relies on mitigation strategies that assume access to human-authored data. In this paper, we conduct extensive simulations across multiple datasets and LLMs to address key gaps in the study of model collapse. First, we introduce model-intrinsic measures based on next-token probability distributions, showing that model collapse corresponds to an increasing concentration of probability mass on a small set of tokens. Second, we demonstrate that model collapse is also associated with a loss of common sense, as measured by a decline in commonsense inference accuracy. Third, we identify perplexity (a measure of model "surprise") as a key driver of collapse: fine-tuning on the least "surprising" documents leads to more severe degeneration. Building on this insight, we propose a perplexity-based filtering strategy that prioritizes high-surprise documents during fine-tuning. Unlike existing approaches, our method does not require distinguishing between human-authored and AI-generated content. Across datasets and LLM families, this strategy consistently mitigates model collapse, achieving performance comparable to, and in some cases better than, human-data baselines, while substantially reducing the concentration of next-token probabilities. Overall, our results provide a unified, model-centric understanding of model collapse and suggest practical, scalable strategies for training generative AI systems in increasingly synthetic environments.

1 Introduction

As AI-generated content enters web training corpora, recursive fine-tuning on model outputs—AI autophagy—raises concerns about model collapse, while existing understanding and mitigation remain limited. The paper introduces model-intrinsic measures, examines commonsense and data-property links, and proposes high-perplexity filtering that mitigates collapse without source labels.

1 Introduction

  • The study investigates four gaps: intrinsic characterization, commonsense effects, data properties that contribute to collapse, and mitigation without human-versus-AI labels.The introduction situates these questions amid growing risk that AI-generated content will be reused for training.Source: §1 · p. 2
  • Model-intrinsic measures characterize collapse through concentrated next-token probabilities, quantified with the Gini coefficient and a near-one maximum-probability indicator.The paper defines collapse as assigning disproportionate probability mass to a small subset of tokens.Source: §1 · p. 2
  • Model collapse is associated with declining commonsense-continuation probability as AI autophagy unfolds.The paper introduces a measure of the probability that an LLM generates a commonsensical continuation.Source: §1 · p. 2
  • Perplexity links document properties such as diversity, novelty, and redundancy to changes in an LLM’s next-token distributions.The paper describes these distributional changes as driving and revealing collapse.Source: §1 · p. 2
  • The perplexity-based filter consistently mitigates collapse across datasets and LLM families, matching or sometimes exceeding human-authored-data performance without source labels.The approach is presented as extending detector-based and accumulation methods to unlabeled web corpora.Source: §1 · p. 3

2 Related Work

Prior work examines collapse across model types, data mixtures, simulation settings, and mitigation strategies. This paper positions its question around document properties that may trigger collapse, including under repeated fine-tuning on human-authored data.

2 Related Work

  • Earlier studies report collapse across language, image, and other generative models, with findings on diversity loss and the effects of human–AI data mixtures [1] [5] [6] [10] [11] [20] [22] [30] [31].Reported patterns include declining image quality, deteriorating performance as synthetic-image proportions rise, and eventual collapse after initial gains from mixed training.Source: §2 · pp. 3–4
  • Prior work links collapse dynamics to task creativity, learning rate, and original word frequency [20] [23] [42].The cited studies examine diversity changes across use cases and the timing of collapse or word forgetting.Source: §2 · p. 3
  • Mitigation studies use accumulation strategies, verification, data-mixture control, semi-synthetic edits, or AI-text detection [6] [12] [14] [17] [26] [49].These approaches include preserving or incorporating human-authored data and controlling synthetic-data accumulation.Source: §2 · p. 4
  • The paper asks which document properties make fine-tuning more likely to trigger collapse, noting that repeated fine-tuning on the same human-authored data can also induce collapse-like effects.The authors frame this question against approaches that assume human- and AI-generated text can be reliably distinguished [1] [6] [12] [30].Source: §2 · p. 4

3 The Model Collapse problem

The section formalizes AI autophagy as iterative generation followed by fine-tuning on generated data, and defines collapse as worsening deviation from the true data distribution over iterations along a quality dimension.

3 The Model Collapse problem

  • AI autophagy iterates generation by a model and fine-tuning that model on the documents it generated.The formal sequence begins with a pretrained LLM and a human-authored dataset.Source: §3 · p. 4
  • Model collapse along a quality dimension occurs when deviation from the true data distribution increases over multiple autophagy iterations.The deviation function depends on the selected metric; existing studies diagnose collapse when it increases monotonically or persistently.Source: §3 · pp. 4–5
  • Collapse can be measured along different dimensions, such as declining linguistic entropy or rising repetition, with repetitive text illustrating reduced lexical diversity.The example connects declining entropy to growing deviation along the linguistic-entropy dimension.Source: §3 · pp. 4–5

4 Autophagy Simulation Framework

The simulation alternates prompt-conditioned generation and fine-tuning, using a fixed set of human-authored prompts. Fine-tuning data can be AI-generated, human-authored, or mixed and selected by a ranking function.

4 Autophagy Simulation Framework

  • The framework alternates model generation and fine-tuning, producing successive models and datasets over simulation steps.The initial model generates continuations from a fixed prompt set, and the resulting prompt–continuation pairs are used for fine-tuning.Source: §4 · pp. 5–6
  • Prompts are sampled from human-authored documents, truncated to k tokens, and held fixed across iterations to support controlled analysis.Each continuation is constrained so the combined prompt and continuation do not exceed L tokens.Source: §4 · p. 6
  • The fine-tuning scenarios use AI-generated documents, randomly sampled human-authored documents, or a mixed pool of human and accumulated AI-generated documents.In the mixed scenario, documents are sampled from the pool according to a ranking function.Source: §4 · p. 6
  • The mixed-data ranking function can encode properties such as linguistic diversity or alignment with a target distribution, with the top-z documents selected.The framework uses prompt-conditioned generation rather than masking the human-authored context; this can reduce recursive self-training effects and partially mask collapse.Source: §4 · p. 6

5 Measuring model collapse: a standard measure, and three new ones

The section combines linguistic entropy with three measures that assess output diversity, next-token concentration, and commonsense inference. These measures track different aspects of model collapse, including increasingly concentrated predictions and declining commonsense performance.

Linguistic entropy

  • Linguistic entropy tracks diversity in generated documents; a decline over simulation steps indicates increasingly repetitive or predictable text.The authors choose entropy as a parsimonious measure of information content.Source: §5 · pp. 6–7

Next-token probability distribution

  • The Gini coefficient measures inequality in next-token probabilities, with higher values indicating concentration on fewer tokens.It is computed over the model's top 100 candidate tokens for each prompt.Source: §5 · pp. 7–8
  • A prediction is flagged as collapsed when any next-token probability exceeds 0.99, capturing near-deterministic behavior.The reported percentage is the share of prompts flagged as collapsed.Source: §5 · p. 8

Commonsensical inference accuracy

  • Commonsense Inference Accuracy measures the share of HellaSwag prompts for which the model selects the coherent continuation.The task presents four candidate endings, only one of which is commonsensical.Source: §5 · p. 8

6 Experimental settings

The experiments simulate AI autophagy over ten iterative steps across four open-weight LLMs, three source datasets and their combination, and AI-only, human-only, and mixed fine-tuning scenarios. Each configuration is run three times, with results reported as means and standard errors.

Experimental settings

  • Prompts are 64 tokens and documents are capped at 128 tokens, yielding a 50% human-prompt and AI-continuation split.Appendix A.1 varies the prompt length to 32 or 96 tokens while keeping document length fixed.Source: §6 · p. 9

Experimental procedure

  • Each configuration is run three times, and metrics are reported as means and standard errors.For next-token distributions, the study uses 1,000 randomly selected unseen-test prompts truncated to 64 tokens.Source: §6 · p. 9

Large Language Models

  • Four open-weight models are tested: Llama2-7B, Llama3-8B, Mistral-7B, and Qwen3-0.6B.Llama2-7B results appear in the main text; results for the other models are deferred to the Appendix.Source: §6 · p. 9

Human-authored datasets

  • The study uses Wikitext, English XL-Sum, SciAbs, and a combined dataset, drawing the source datasets from Guo et al. [20].The datasets span encyclopedic text, news articles, and computational linguistics and NLP abstracts.Source: §6 · p. 10

Fine-tuning scenarios

  • AI-only, human-only, and mixed fine-tuning scenarios vary whether training documents come from AI outputs, human authors, or both.The human-only scenario serves as a baseline without AI-generated training data.Source: §6 · p. 10

Simulation horizon

  • Ten iterative steps of document generation and fine-tuning are used for every model, dataset, and training scenario.The fixed horizon is aligned with prior work.Source: §6 · p. 10

7 Initial experiments with model collapse – and new observations

AI autophagy is accompanied by declining linguistic diversity, increasingly concentrated next-token probabilities, and reduced commonsense inference accuracy. The severity varies with dataset and the fraction of AI-generated training tokens.

Initial experiments

  • At step 10, linguistic entropy falls approximately 7.5% on the combined dataset and 4.7%–8.7% on individual datasets; human-only fine-tuning shows no decline.Similar entropy declines are reported for Llama3-8B, Mistral-7B, and Qwen3-0.6B.Source: §7 · p. 10
  • Across 1,000 prompts, top-token probabilities shift upward, while G(q) and the percentage of collapsed predictions increase across fine-tuning scenarios.For one SciAbs prompt, “paper” comes to dominate the next-token distribution by step 10; similar probability shifts are reported for Wikitext and XL-Sum.Source: §7 · p. 11, Figure 2a · p. 11, Figure 2 · p. 12
  • At step 10, commonsensical inference accuracy falls approximately 23% on the combined dataset and 1.7%–16.3% on individual datasets.The decline is less pronounced with human-authored training data and is also reported for Llama3-8B, Mistral-7B, and Qwen3-0.6B.Source: §7 · pp. 11–12, Figure 4 · p. 13
  • Generated continuations progress from fluent and meaningful at step 0 to redundant and mildly incoherent at step 5, then repetitive or meaningless by step 10.Table 3 shows examples from prompts drawn from three human-authored datasets.Source: §7 · p. 12, Table 3 · p. 14
  • As k decreases and the fraction of AI-generated tokens increases, collapse becomes more pronounced across measures, intensifying diversity loss and overestimation of high-probability tokens.The reported result is consistent with Shumailov et al..Source: §7 · p. 13

8 Learning by surprise: a data filtering strategy for mitigating model collapse

The section introduces adaptive perplexity filtering to select surprising documents rather than treating all human-authored text as equally useful. Higher-perplexity training documents are associated with milder collapse, and the filtering strategy mitigates collapse across several measures while requiring repeated model inference.

8 Learning by surprise: a data filtering strategy for mitigating model collapse

  • At each simulation step, the strategy scores documents with the evolving model and selects the 1,000 highest-perplexity items regardless of origin.This makes selection adaptive to the model-document relationship rather than based on a fixed document-quality score.Source: §8 · pp. 16, 18
  • Across entropy, commonsense accuracy, Gini coefficient, and collapsed predictions, perplexity filtering matches human-data entropy, slightly exceeds its commonsense accuracy, and yields the lowest Gini and collapse rates among baselines.The pattern also holds for Llama3-8B, Mistral-7B, and Qwen3-0.6B.Source: §8 · p. 16, Figure 7 · p. 17
  • The selected set is predominantly human-authored, but its AI-generated share rises slightly over later steps, and selected AI documents usually come from earlier rather than the immediately preceding step.SciAbs and XL-Sum alternate in contribution; Wikitext contributes fewer than 100 selected documents per step.Source: §8 · p. 17, Figure 8 · p. 19
  • High-perplexity AI-generated examples selected at step 10 are predominantly from earlier simulation steps and are described as informative and well-formed rather than noise or hallucinations.Table 4 reports the five highest-perplexity AI-generated documents with respect to M10.Source: §8 · p. 18, Table 4 · p. 20

8.1 Relation between perplexity and model collapse

  • Higher-perplexity documents are associated with milder collapse for both human-authored and AI-generated training data, although AI-generated data exhibits greater collapse overall.The comparison uses next-token Gini coefficients and counts of collapsed predictions across perplexity deciles.Source: §8.1 · p. 18, Figure 9 · p. 19

8.2 Computational complexity of the perplexity-based filtering strategy

  • Perplexity filtering repeatedly runs model inference over candidate documents, making it computationally heavier than static entropy scoring and potentially limiting scaling to massive datasets or long schedules.The paper suggests subsampling or incremental scoring to reduce overhead, with approximations that may still capture model drift.Source: §8.2 · pp. 20–22

9 Discussion and Future Works

The discussion connects surprise-based filtering to learning from unexpected inputs and outlines limits on the study’s scope and the strategy’s broader applicability. It also identifies future directions across content domains and human–AI systems.

9 Discussion and Future Works

  • The filtering strategy is an engineering analogue of findings that surprise attracts attention and supports learning, prioritizing documents likely to update model beliefs.The authors relate this to infants learning from unexpected events and Bayesian surprise.Source: §9 · p. 22
  • The study’s scope is limited to textual LLMs, and extending the approach to image, audio, video, or multimodal generation requires validation.The authors also note that fixed fine-tuning document counts and learning rates were chosen for computational efficiency and to reduce confounding.Source: §9 · p. 22
  • Prompting simulations with human-authored text may mask exposure bias and underestimate collapse in fully synthetic input-output loops.The authors caution that simulated findings should be tested on actual generative AI platforms, where interacting factors make systems more complex.Source: §9 · p. 22
  • Open-weight models’ undocumented training data means evaluation sets may overlap pretraining, while fully controlled held-out evaluation remains infeasible for most state-of-the-art models.The authors identify this as a limitation shared by existing AI-autophagy and model-collapse studies.Source: §9 · p. 22
  • The filtering strategy’s effectiveness may depend on task, and its downstream evaluation covers only commonsense inference, leaving other task trade-offs unresolved.In the reported setting, the authors find no performance degradation and possible slight improvements.Source: §9 · p. 23
  • Perplexity filtering costs more computation than linearly scaling static entropy and may hinder larger datasets or longer training, though scoring approximations could reduce overhead.Suggested options include subsampling, incremental scoring, and score caching.Source: §9 · p. 23
  • Each model family was tested at only one scale, so whether collapse dynamics and filtering effectiveness generalize across model sizes remains open.The evaluated sizes include 7B for Llama2, 8B for Llama3, 7B for Mistral, and 0.6B for Qwen3.Source: §9 · p. 23
  • Future work could examine collapse in code and mathematical writing, where low baseline surprise may accelerate collapse but reduced diversity could reinforce useful conventions.The study covered Wikitext, SciAbs, and XL-Sum; Wikitext had the lowest perplexity and fastest collapse.Source: §9 · p. 23

Authors’ contributions.

The authors divide simulation implementation, data analysis, figure design, plotting, writing, and research supervision among the named contributors.

Authors’ contributions.

  • DG and GG implemented simulations, DG handled data analysis, DG and LP designed figures, and DG made the plots.DG, LP, GG, and AK wrote the paper; LP and AK directed and supervised the research, with all authors contributing to discussion and approving the paper.Source: Authors’ contributions · p. 24

A.1 Different percentage of synthetic tokens

Across the appendix analyses, greater synthetic-token shares and successive autophagy steps are associated with stronger collapse, while perplexity filtering’s mitigation patterns extend across prompt lengths and model families.

A.1 Different percentage of synthetic tokens

  • Across measures, collapse becomes more pronounced as prompt length falls from 96 to 32 tokens, corresponding to synthetic-token shares rising from 25% to 75%.The measures are linguistic entropy, commonsense inference accuracy, average Gini coefficient, and collapsed-prediction percentage.Source: §A.1 · p. 27, Figure 10 · p. 27

A.2 Top-ranked next-token probabilities for all datasets

  • By step 5, more than half of Wikitext top tokens exceed probability 0.9, while fewer than half of SciAbs top tokens exceed 0.9 even at step 10.All three datasets show a shift toward higher top-token probabilities over simulation steps.Source: §A.2 · p. 28, Figure 11 · p. 28

SciAbs

  • SciAbs exhibits the weakest top-token probability collapse among the datasets, with fewer than half of top tokens above 0.9 at step 10.Figure 11 includes distributions for Wikitext, XL-Sum, and SciAbs across steps 0, 5, and 10.Source: §A.2 · p. 28, Figure 11 · p. 28

A.3 Evaluation of Gini coefficient and collapsed predictions with prompts of 64 tokens

  • With 64-token prompts, perplexity-based filtering again achieves the lowest model-collapse degree across all reported metrics.This replicates the main-manuscript analysis using 32-token prompts.Source: §A.3 · p. 29, Figure 12 · p. 29

A.4 Mitigation of model collapse using different models

  • The qualitative mitigation patterns observed for Llama2-7B generalize to Llama3-8B, Mistral-7B, and Qwen3-0.6B.The additional models were evaluated under the same fine-tuning scenarios and model-collapse metrics.Source: §A.4 · p. 29, Figure 13 · p. 30, Figure 14 · p. 31

Related papers

Loading 2410.12341v4…