PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Consuming Generative Models Go MAD

Sina Alemohammad, Josue Casco-Rodriguez, Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaei, Daniel LeJeune, Ali Siahkoohi, Richard G. Baraniuk

arXiv:2307.01850v1cs.LGcs.AIcs.CV

TL;DR

The paper asks how repeatedly training generative models on synthetic data affects later generations as such data increasingly enters training corpora. Through theoretical analysis and empirical studies of three autophagous image-model loops, it finds that insufficient fresh real data causes precision or recall to progressively degrade, especially under quality-biased sampling.

  • Problem

    Synthetic data is increasingly used to train generative models, but the consequences of repeating this process across generations are poorly understood.

  • Method

    The paper combines theoretical analysis with experiments on three self-consuming training-loop families and biased sampling settings using generative image models.

  • Results

    Without enough fresh real data each generation, future models go MAD, with precision or recall progressively degrading and artifacts amplified.

  • Takeaways & Limitations

    Fresh real data is required across generations to prevent progressive loss of generative-model quality or diversity.

  • Takeaways & Limitations

    The analysis focuses on imagery and distribution-manifold distance, leaving its manifestation in downstream tasks such as classification as an open question.

Abstract

from arXiv · show

Seismic advances in generative AI algorithms for imagery, text, and other data types has led to the temptation to use synthetic data to train next-generation models. Repeating this process creates an autophagous (self-consuming) loop whose properties are poorly understood. We conduct a thorough analytical and empirical analysis using state-of-the-art generative image models of three families of autophagous loops that differ in how fixed or fresh real training data is available through the generations of training and in whether the samples from previous generation models have been biased to trade off data quality versus diversity. Our primary conclusion across all scenarios is that without enough fresh real data in each generation of an autophagous loop, future generative models are doomed to have their quality (precision) or diversity (recall) progressively decrease. We term this condition Model Autophagy Disorder (MAD), making analogy to mad cow disease.

1 Introduction

Synthetic data is increasingly entering generative-model training, creating self-consuming loops whose effects are poorly understood. The paper studies three loop families and finds that insufficient fresh real data causes quality or diversity to decline.

  • Internet-sourced datasets increasingly contain AI-generated data, including synthetic images in LAION-5B used to train models such as Stable Diffusion.
  • Repeatedly training on synthetic outputs forms autophagous loops that may amplify generator artifacts and degrade model properties.
  • The paper models fully synthetic, synthetic augmentation, and fresh data loops, distinguished by how real data enters successive training sets.
  • In fully synthetic loops, either precision or recall decreases across generations, while fixed real data in augmentation loops only delays degradation.
  • With enough fresh real data, quality and diversity do not degrade over generations; without enough, all three loop models go MAD.
  • Sampling bias preserves quality at the expense of faster diversity loss, whereas unbiased autophagy can rapidly reduce both quality and diversity.
  • The conclusions are supported across Gaussian and multiple image-model families, including DDPM, StyleGAN-2, WGAN, and Normalizing Flows.

2 Self-consuming generative models

The paper formalizes self-consuming training as sequences of generative distributions trained with real and synthetic data, then analyzes how data composition and sampling bias affect deterioration. It measures drift and quality-diversity changes using distributional distance, precision, and recall.

  • 2.1 Autophagous processes: An autophagous process trains each generation on data containing samples from previous generative models, while the first generation uses purely real data.
  • 2.1 Autophagous processes: A MAD process is defined as a sequence whose expected distance E[dist(Gt, Pr)] from the reference distribution increases with generation t.
  • 2.2 Realistic models for autophagous loops: The balance of real versus synthetic data and the sampling procedure are the two critical factors that can drive an autophagous process MAD.
  • 2.2 Realistic models for autophagous loops: The three loop mechanisms differ in whether training uses only previous synthetic data, fixed real data plus synthetic data, or fresh real data plus synthetic data.
  • 2.3 Biased sampling in autophagous loops: Synthetic samples are biased with a universal parameter λ, where λ = 1 is unbiased and λ = 0 samples from zero-variance modes, increasing quality while reducing diversity as λ decreases.
  • Evaluation measures: The analysis uses Wasserstein distance approximated by FID to measure distributional drift, with precision representing quality and recall representing diversity.

3 The fully synthetic loop: Training exclusively on synthetic data leads to MADness

The fully synthetic loop repeatedly trains each generative model on samples from its predecessor, causing estimation errors to accumulate across generations. Without sampling bias, quality and diversity both decline; with bias, improved quality comes at the cost of faster diversity loss.

  • Analytical setup: Repeatedly fitting models to samples from the preceding generation makes the fully synthetic loop a MAD generative process.The analysis focuses on inter-generational propagation of estimation errors and sampling biases.
  • Gaussian analysis: The Gaussian analysis attributes quality loss to mean drift and diversity loss to covariance collapse, even when sampling is unbiased (λ = 1).The mean forms a Gaussian random walk, while the covariance converges to zero.
  • Without sampling bias: Finite-sample estimation drives a random walk away from the real distribution, reducing precision and recall while increasing FID over generations.These trends eventually saturate, and precision and recall can drop to zero despite non-zero variance.
  • Without sampling bias: Without sampling bias, MNIST modes progressively drift from real modes, merge together, and become almost illegible by Generation 10.The merging contributes to declining precision and recall.
  • With sampling bias: Sampling bias increases precision but accelerates recall loss, preserving legibility while causing modes to collapse around individual high-quality images.This quality–diversity trade-off still produces a MAD generative process.
  • Empirical scope: The MAD pattern also appears in experiments with Gaussian mixtures, WGANs, and normalizing flows, alongside DDPM and StyleGAN-2 results.Experiments span MNIST and FFHQ image datasets.

4 The synthetic augmentation loop: Fixed real training data may delay but not prevent MADness

The synthetic augmentation loop retains a fixed real dataset while progressively adding synthetic data. Fixed real data slows degradation, but quality or diversity still declines over generations, and sampling bias trades faster diversity loss for improved quality.

  • Loop design: The synthetic augmentation loop progressively augments a fixed real dataset with synthetic data from previous generations.The loop is motivated by the use of generative models for dataset augmentation.
  • Main conclusion: Fixed real training data only delays the eventual degradation of generative-model quality or diversity over generations.The paper presents this as the main conclusion for the synthetic augmentation loop.
  • Experimental setup: The experiments retain original real data at every generation while adding synthetic samples from prior models, with the synthetic pool growing over time for StyleGAN.For the StyleGAN setup, the synthetic pool grows linearly as n_t^s = (t − 1)70k.
  • Without sampling bias: Without sampling bias, retaining the full FFHQ dataset still increases FID and decreases both precision and recall, although more slowly than in the fully synthetic loop.Cross-hatching artifacts also appear, but with less prevalence.
  • With sampling bias: Sampling bias still increases FID more slowly, while higher precision is obtained only at the cost of accelerated recall loss.The authors identify λ = 0.8 as an example that better mitigates FID and precision losses.

5 The fresh data loop: Fresh real data can prevent MADness

Fresh real data can prevent a purely random-walk-like MAD process, but the long-run outcome depends on the amounts and bias of real and synthetic data. Modest synthetic augmentation may help, whereas excessive or biased synthetic data harms the effective distributional estimate.

  • Fresh data mitigates MADness: Fresh samples keep the generative distribution somewhat close to the reference distribution instead of undergoing a purely random walk.The paper reports that fresh data mitigates development of a MAD generative process.
  • Synthetic data can help: Limited synthetic data can improve the fresh-data distributional estimate by transferring previously used real data and increasing effective dataset size.The effective sample size ne is admissible when ne/nr ≥1.
  • Initial models are forgotten: Later-generation performance converges to a point determined by the amounts of real and synthetic data, not the initial model.For Gaussian and MNIST DDPM models, Wasserstein distance and FID converge to limiting values independent of the initial real-sample count nini.
  • Synthetic data can hurt: Excessive synthetic data causes a critical-threshold failure in which the distributional estimate deteriorates dramatically.When ne/nr <1, synthetic data effectively reduces the number of real samples.
  • Admissible sample sizes: As nr grows, the admissible synthetic-sample threshold tends to become constant rather than maintaining a fixed nr-to-ns ratio.Large ns can be useful when nr is small, but the threshold stabilizes as nr increases.
  • Sampling bias: Greater sampling bias, represented by smaller λ, reduces the number of synthetic samples usable without harming performance.The unbiased limit λ →1 appears to ensure that the effective sample count increases.

6 Discussion

The paper concludes that self-consuming training loops can progressively degrade generative quality or diversity without sufficient fresh real data. It discusses practical safeguards, broader data modalities, and open questions about mixed loops and downstream task effects.

  • Main conclusion: Without enough fresh real data per generation, quality or diversity progressively degrades and artifacts can be amplified, producing Model Autophagy Disorder.The conclusion is based on analytical and experimental studies of three autophagous-loop families.
  • Dataset safeguards: Synthetic-data detection and filtering are suggested for maintaining datasets containing exclusively real data.The paper notes that watermarking may introduce artifacts that autophagy could amplify.
  • Practical implications: Synthetic data can be useful in fresh-data loops at modest amounts but harms performance when its amount becomes large.This defines a practical good region for balancing fresh real and synthetic data.
  • Open questions: The study leaves open how combinations of fixed real, fresh real, and synthetic data affect autophagous loops.It also identifies downstream classification performance as an unresolved question.
  • Broader scope: The conclusions are presented as extending beyond imagery to text and other data types, including text produced by large language models.The paper states that precision and recall translate into properties of text after generations of autophagy.

A Proof of synthetic Gaussian martingale variance collapse

The proof models the trace of the Gaussian covariance through a martingale-like process and shows that its limiting value must be zero. This establishes almost-sure variance collapse in the fully synthetic process, including the effect of λ ≤ 1.

  • The trace of the covariance matrix forms a lower-bounded submartingale.
  • Doob’s martingale convergence theorem gives an almost-sure limit W for tr[Σt].
  • The proof shows that the limiting value must satisfy W = 0.
  • For λ = 1, persistent fluctuations in the generalized χ2 factors force the covariance trace limit to zero.
  • For general λ ≤ 1, multiplying the λ = 1 process by λ^(t−1) also drives tr[Σt] to zero almost surely.

B Additional experiments for the fully synthetic loop

Additional experiments examine fully synthetic loops with GANs and Gaussian mixture models. Across these settings, quality and diversity deteriorate, modes disappear, and recall best captures multimodal diversity loss.

  • FID monotonically increases while precision and recall monotonically decrease across 100 fully synthetic WGAN-GP generations on MNIST.
  • A fully synthetic loop with 2D GMMs gradually reduces the number of covered modes.
  • Recall best reflects diversity loss in the multimodal GMM experiment.
  • The compared diversity measures are overall variance, average modal variance, and recall.

B.3 Additional MNIST DDPM fully synthetic loop results

The section reports additional MNIST DDPM results and a controlled Rosenbrock normalizing-flow experiment. The latter visualizes how fully synthetic training behaves with and without sampling bias across generations.

  • MNIST DDPM loops are evaluated using FID, precision, and recall across more generations and different sampling biases λ.
  • The Rosenbrock experiment uses normalizing flows to visualize a fully synthetic loop in a controlled two-dimensional setting.
  • Normalizing flows permit exact likelihood evaluation through invertibility, simplifying the controlled training procedure.
  • The experiment begins with 10^4 samples from the 2D Rosenbrock distribution and trains subsequent generations for 16 generations with and without sampling bias.
  • Sampling with or without bias loses distribution tails, while λ < 1 accelerates diversity loss.

C FFHQ fully synthetic loop images with λ = 1

This section presents additional randomly selected images from an unbiased fully synthetic StyleGAN loop on FFHQ. The images are shown for generations t = 1, 3, 5, 7, and 9.

  • The section shows additional randomly chosen synthetic samples from the unbiased fully synthetic StyleGAN FFHQ loop.
  • The displayed generations are t = 1, t = 3, t = 5, t = 7, and t = 9.

D FFHQ fully synthetic loop images with λ = 0.7

The biased fully synthetic loop is examined across generations and sampling settings, with λ = 0.7 shown at generations 1, 3, and 5. Comparisons with unbiased sampling and synthetic augmentation highlight different failure patterns.

  • λ = 0.7 is shown for fully synthetic-loop generations t = 1, t = 3, and t = 5.
  • Without sampling bias, synthetic MNIST modes drift from real modes and merge together across generations.
  • With sampling bias, synthetic MNIST modes drift and collapse around a single high-quality image before merging.
  • A synthetic augmentation loop with λ = 1 is shown at generations t = 3 and t = 6.

G Additional results for the fresh data loop

Additional fresh-data experiments examine memory across previous generations, the real-data fraction, dataset size, and sampling bias. They show that synthetic data slows progress, while sampling bias can make progress eventually stop as dataset size grows.

  • Previous-generation memory: Increasing memory K when sampling from previous generations can boost performance, but improvement becomes slower as K increases.
  • Experimental setup: The experiments vary the real-data fraction p, total dataset size n, and sampling bias λ when evaluating Wasserstein distance.
  • Fresh-data fraction and dataset size: With synthetic data present, increasing dataset size n reduces the rate at which Wasserstein distance decreases and increases the overall distance each generation.
  • Sampling bias: With sampling bias λ < 1, Wasserstein distance follows a sub-linear trend, so progress eventually effectively stops despite realistically increasing total dataset size.
Loading 2307.01850v1…