PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Stability of Iterative Retraining of Generative Models on their own Data

Quentin Bertrand, Avishek Joey Bose, Alexandre Duplessis, Marco Jiralerspong, Gauthier Gidel

arXiv:2310.00429v5cs.LGstat.ML

TL;DR

As web datasets increasingly contain generated content, future generative models may train on mixtures of real and synthetic data. The paper develops a framework for iterative retraining and proves stability under specified conditions, then validates the theory on image datasets.

  • Problem

    Because web-scale datasets may already contain generated content, the paper asks whether training on finite mixtures of real and self-generated data alters performance.

    Source: §1 · p. 2

  • Method

    The paper extends a maximum-likelihood framework to iterative retraining on mixed datasets and analyzes when the updates remain stable.

    Source: §1 · p. 2, §2.1 · p. 3, §3 · p. 4

  • Results

    Stability holds when the initial model approximates the data distribution sufficiently well and each iteration retains a sufficiently large proportion of real data, with empirical validation on CIFAR10 and FFHQ.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported stability result applies to mixed-data retraining when the initial model is sufficiently accurate and real data is reused in sufficient proportion.

    Source: §1 · p. 2

    Theorem 1 assumes infinite samples from the data distribution and gives only a sufficient stability condition.

    Source: §3.1 · p. 6

Abstract

from arXiv · show

Deep generative models have made tremendous progress in modeling complex data, often exhibiting generation quality that surpasses a typical human's ability to discern the authenticity of samples. Undeniably, a key driver of this success is enabled by the massive amounts of web-scale data consumed by these models. Due to these models' striking performance and ease of availability, the web will inevitably be increasingly populated with synthetic content. Such a fact directly implies that future iterations of generative models will be trained on both clean and artificially generated data from past models. In this paper, we develop a framework to rigorously study the impact of training generative models on mixed datasets -- from classical training on real data to self-consuming generative models trained on purely synthetic data. We first prove the stability of iterative training under the condition that the initial generative models approximate the data distribution well enough and the proportion of clean training data (w.r.t. synthetic data) is large enough. We empirically validate our theory on both synthetic and natural images by iteratively training normalizing flows and state-of-the-art diffusion models on CIFAR10 and FFHQ.

1 INTRODUCTION

As web-scale datasets increasingly contain synthetic content, future generative models face training data mixed from real and generated sources. This paper develops a framework for studying that setting and proves stability under stated conditions, with empirical validation on image datasets.

1 INTRODUCTION

  • The framework extends maximum-likelihood training to iterative retraining on mixtures of clean real data and self-generated data, covering VAEs, normalizing flows, and diffusion models.Source: §1 · p. 2
  • Stability follows when the initial model is sufficiently well trained and each iteration retains a sufficiently high proportion of original clean data.Source: §1 · p. 2
  • The authors validate the theory through iterative retraining on CIFAR10 and FFHQ using OTCFM, DDPM, and EDM diffusion models.Source: §1 · p. 2

2 ITERATIVE RETRAINING OF GENERATIVE MODELS

The paper formalizes iterative retraining as updating a generative model using both real data and samples from its preceding version. Its Gaussian warm-up shows that retraining solely on generated samples collapses covariance.

2.1 PRELIMINARIES

  • Iterative retraining samples synthetic data from the current model and updates it on the union of those samples and the real dataset, with λ controlling synthetic-data volume.Source: §2 · p. 3, §2.1 · p. 3
  • For likelihood-based models, training maximizes the likelihood of an empirical sample from the data distribution; subsequent iterations resume from the previous parameters, amounting in practice to finetuning.Source: §2.1 · p. 3

2.2 WARM-UP WITH MULTIVARIATE GAUSSIAN

  • In the Gaussian warm-up, the mean and covariance updates have closed-form formulas, and the collapse result generalizes earlier Gaussian-collapse results while giving a convergence rate.Source: §2.2 · p. 4
  • With finite-sample retraining of a single multivariate Gaussian solely on its generated data, model collapse occurs and covariance vanishes linearly with the number of retraining steps.Source: §2.2 · p. 4

3 STABILITY OF ITERATIVE RETRAINING

The analysis separates idealized retraining without statistical error from finite-sample retraining. It establishes local stability when the initial model is sufficiently close to the optimum and enough real data is retained, then bounds finite-sample error under additional assumptions.

3 STABILITY OF ITERATIVE RETRAINING

  • The analysis distinguishes finite-sample statistical error from model-class approximation error and first studies an idealized setting with full access to pdata.The authors frame practical retraining as having nonzero statistical error and then analyze that case separately.Source: §3 · p. 4, §3.1 · p. 4

3.1 ITERATIVE RETRAINING UNDER NO STATISTICAL ERROR

  • In the no-statistical-error setting, θ⋆ is a fixed point of infinite-sample retraining, and a nearby local maximizer is unique when εL < α.Here θ⋆ is the optimal parametric model under the data distribution.Source: §3.1 · pp. 5–6
  • When λ(1 + Lε/α) < 1/2, the fixed-point operator has Jacobian norm below one and nearby infinite-sample iterates converge to θ⋆.The theorem also gives a convergence-rate bound; stability is local around θ⋆.Source: §3.1 · p. 6
  • Theorem 1 gives a sufficient stability condition, not a necessary one, and assumes infinitely many samples from pdata.The paper also notes that pure-synthetic retraining can collapse a multivariate Gaussian model to vanishing variance.Source: §3.1 · p. 6

3.2 ITERATIVE RETRAINING UNDER STATISTICAL APPROXIMATION ERROR

  • With finite sampling, retraining remains within a neighborhood of θ⋆ with high probability, and its error separates optimization, statistical, and iterative-retraining terms.The statistical term decreases with sample size, while the iterative term depends on the initial distance to θ⋆.Source: §3.2 · p. 7

4 RELATED WORK

Prior work reports model collapse under iterative retraining from scratch or fine-tuning, and degradation when web-scale data contain synthetic samples. This paper studies mixed real-and-synthetic retraining with theoretical stability guarantees, unlike the cited prior studies.

Generative Models trained on Synthetic Data

  • Alemohammad et al. (2023) and Shumailov et al. (2023) report model collapse when models are retrained from scratch and fine-tuned, respectively.Alemohammad et al. also advocate injecting real data, including fresh data, but do not provide theoretical stability guarantees.Source: §4 · p. 7
  • Studies of web-scale datasets contaminated with synthetic samples report degradation in generated-sample quality.The cited works are Martínez et al. (2023a;b) and Hataya et al. (2023).Source: §4 · p. 7

Performative Predicition

  • The paper connects its fixed-point analysis to performative prediction, where model outputs affect sampling distributions and thereby influence objectives.The cited comparison notes that performative-prediction work concerns supervised learning and requires stronger assumptions for global convergence than this paper's assumptions around an optimal point.Source: §4 · p. 7

5 EXPERIMENTS

Experiments examine how the amount of generated data and repeated retraining affect generative-model performance, using synthetic and image datasets. Synthetic-only retraining diverges, while stability with mixed data varies by model and dataset.

Datasets and Models

  • The image experiments use CIFAR-10 and FFHQ-64 with OTCFM and diffusion models, including DDPM and EDM; synthetic toy data are also tested.The setup evaluates performance as a function of λ and retraining iterations while holding the real-dataset size constant.Source: §5 · p. 8

Results on Synthetic Data

  • In toy-data experiments, fully retraining on synthetic samples collapses, whereas retraining on equal amounts of real and generated data does not diverge.The examples use DDPM on an 8-Gaussians dataset and conditional flow matching on two-moon data.Source: §5 · p. 8

Experimental Setup

  • Training uses a constant number of images rather than epochs because mixed-dataset size varies with λ.After each iteration, models and optimizers are checkpointed so the next retraining step resumes from that checkpoint.Source: §5 · p. 8

Analysis of Results

  • Synthetic-only retraining diverges across models and datasets; EDM remains stable on CIFAR-10 for λ from 0 to 1, and on FFHQ-64 when λ < 0.5.For λ = 1, generated and real data are used in equal amounts; for EDM on FFHQ-64, stability requires a smaller generated-data proportion.Source: §5 · pp. 8–9, Figure 2 · p. 9
  • EDM precision decreases and recall increases with retraining across tested λ values, while OTCFM precision on CIFAR-10 shows the inverse trend.The reported metrics are FID, precision, and recall; λ = 0 is the no-synthetic-data baseline and λ = 1 uses equal amounts of generated and real data.Source: §5 · pp. 8–9, Figure 2 · p. 9

6 CONCLUSION AND OPEN QUESTIONS

The paper concludes that iterative retraining is stable when the initial model is sufficiently good and each training round retains enough real data. It identifies a gap between sufficient and necessary stability conditions and flags finite-sampling effects for future work.

6 CONCLUSION AND OPEN QUESTIONS

  • Stability is shown for the retraining procedure when the initial model is good enough and each iteration mixes synthetic data with real data.The experiments validate the theoretical findings on CIFAR-10 and FFHQ-64 using OTCFM, DDPM, and EDM.Source: §6 · p. 9
  • For iterative retraining on a mixture of synthetic and real data, stability is shown when the initial model is good enough; under Assumptions 1 and 2, the sufficient condition may not be necessary, leaving a gap from necessity λ < +∞.Source: §6 · p. 9
  • The impact of finite sampling on iterative-retraining stability remains an avenue for future work.The finite-sampling aspect is described in Section 3.2.Source: §6 · p. 9

A PROOF OF PROPOSITION 1

The proof establishes Gaussian collapse under retraining solely on generated data, treating the one-dimensional case before extending the argument to multiple dimensions. A separate appendix proof invokes a Gaussian log-determinant gradient identity.

A PROOF OF PROPOSITION 1

  • For every initial mean and covariance, repeatedly learning a single multivariate Gaussian solely from its generated data yields model collapse.The proof presents the one-dimensional case first and then treats the multivariate case.Source: A PROOF OF PROPOSITION 1 · p. 14

A.1 1D-GAUSSIAN CASE.

  • The one-dimensional proof tracks variance recursively, then bounds the expected standard deviation rather than the variance.Its final step bounds the expected square root of a chi-squared variable; the resulting α is below 1, depends on sample size, and tends to 1 as sample size grows.Source: §A.1 · pp. 14–15

A.2 MULTI-DIMENSIONAL CASE.

  • The multivariate proof follows the same scheme, using an upper bound on the expected square root of the covariance matrix.It invokes strict Jensen inequality for the matrix square-root operator.Source: §A.2 · pp. 16–17

B PROOF OF PROPOSITION 2

C PROOF OF PROPOSITION 3

The proof establishes that, under Assumptions 1 and 2 and if εL < α, the local maximum-likelihood solution is unique. It uses the implicit function theorem to continue a stationary point near θ⋆.

C PROOF OF PROPOSITION 3

  • If εL < α, a solution θ⋆ of Equation (7) satisfying Assumptions 1 and 2 has a unique local maximizer in a sufficiently small neighborhood.The proof constructs a continuous stationary-point function near θ⋆ using the implicit function theorem, then shows it gives a local maximizer.Source: C PROOF OF PROPOSITION 3 · p. 18

D PROOF OF PROPOSITION 4

Proposition 4 states that the optimal parametric generative model is a fixed point of the retraining operator. The statement applies to any θ⋆ solving Equation (7) and all λ > 0.

D PROOF OF PROPOSITION 4

E PROOF OF THEOREM 1

Theorem 1 establishes local stability of iterative retraining by bounding the fixed-point operator’s Jacobian. Its guarantee depends on the stated regularity and curvature assumptions and a sufficiently small λ.

E PROOF OF THEOREM 1

  • Under Assumptions 1 and 2, the expected log-likelihood is locally twice differentiable and has negative curvature at θ⋆, bounded by −αId.Source: E PROOF OF THEOREM 1 · p. 20
  • If λ(1 + Lε/α) < 1/2, the fixed-point operator’s Jacobian has operator norm strictly below 1 at θ⋆.The bound is λ(α + εL)/(α − λ(α + εL)) < 1.Source: E PROOF OF THEOREM 1 · p. 20
  • A sufficiently close starting point θ0 remains in the theorem’s local basin, where successive retraining iterates converge toward θ⋆.Source: E PROOF OF THEOREM 1 · p. 20
  • The proof derives a Jacobian formula using the implicit function theorem and Schwarz theorem, then bounds it using Kantorovich-Rubenstein duality.The resulting condition ensures the Jacobian norm is below one.Source: E PROOF OF THEOREM 1 · p. 20

F PROOF OF THEOREM 2

The proof presents approximate stability under the theorem’s assumptions, while the experiments contrast collapse under fully synthetic retraining with stable mixed-data regimes. The appendices also define the evaluation metrics and show sample comparisons.

F PROOF OF THEOREM 2

  • Theorem 2 gives approximate stability with probability 1 − δ under Theorem 1’s assumptions, Assumption 3, and a sufficiently close initial model.Its proof invokes the contraction factor ρ < 1 and conditional independence of successive samplings.Source: F PROOF OF THEOREM 2 · p. 23

G.1 EIGHT GAUSSIANS – DDPM

  • On the eight-Gaussians task, synthetic-only retraining collapses after 100 iterations, whereas retraining on mixed real and generated data does not collapse.Source: Figure 3 · p. 24

G.2 TWO MOONS – FLOW MATCHING

  • Figure 4 depicts stability versus collapse during iterative retraining of generative models on their own data, plotting each model’s density against retraining steps; its first two columns show true density and a diffusion model trained on true data.Source: Figure 4 · p. 24

G.3 CIFAR-10 – DDPM

  • Training only on synthetic data yields divergence; Figure 5 reports FID, precision, and recall across retraining counts for multiple generated-data fractions λ, with D = Dreal ∪ {x̃i}⌊λ·n⌋.Source: §G.3 · p. 25, Figure 5 · p. 25

G.4 FULL GRAPHS FOR FULLY SYNTHETIC CIFAR10-OTCFM, CIFAR10-EDM, AND FFHQ-EDM

  • The EDM models on CIFAR-10 remain stable for generated-data proportions λ from 0 to 1, while FFHQ-64 is stable only when λ < 0.5.Across all models and datasets, training only on synthetic data yields divergence.Source: §G.4 · p. 26, Figure 6 · p. 26

G.5 DETAILS ON THE METRICS: PRECISION AND RECALL

  • Precision measures generated samples within the training distribution’s support, while recall measures training samples within the generated distribution’s support.The empirical support is approximated with unions of nearest-neighbor balls; FID compares 50,000 generated and training samples, and precision and recall use k = 4.Source: §G.5 · pp. 26–27

G.6 SAMPLE VISUALIZATION

  • Figure 7 shows EDM samples from iterative retraining on synthetic data alone in the top row and on mixed synthetic and real data in the bottom row.Source: Figure 7 · p. 27

G.7 QUANTITATIVE SYNTHETIC EXPERIMENTS

Quantitative experiments show that mixed real-and-synthetic retraining stabilizes distributional distance, while fully synthetic retraining diverges. Parameter changes are also larger under fully synthetic retraining than under mixed retraining.

G.7 QUANTITATIVE SYNTHETIC EXPERIMENTS

  • Wasserstein-1 and 2 distances diverge under fully synthetic retraining but stabilize when retraining uses a mixture of real and synthetic data.Figure 8 averages distances over 50 runs; lines show means and shaded areas show standard deviations.Source: Figure 8 · p. 28
  • The Figure 8 experiment uses 1,000 two-moons samples learned with OT-CFM, a continuous normalizing flow with exact likelihood.Hyperparameters use the implementation defaults.Source: §G.7 · p. 28

G.8 PARAMETER CONVERGENCE

  • Fully synthetic retraining moves parameters farther from their initial values than mixed real-and-synthetic retraining.Figure 9 rescales the Euclidean parameter difference by the number of model parameters; columns show different dataset and algorithm combinations.Source: Figure 9 · p. 29
  • Figure 9 finetunes EDM and OTCFM models for 20 epochs and 20 retraining rounds on CIFAR-10 and FFHQ.Source: §G.8 · p. 29

Related papers

Loading 2310.00429v5…