PaperlayerView exact revision on arXiv

Source-linked AI summary

Feature Selective Model Collapse in Diffusion Models: Total Replacement versus Fixed-Budget Training

Hanna Malet, Gabriel Turinici

arXiv:2610.01318v1cs.LGcs.AImath.STstat.TH

TL;DR

Recursive training can produce model collapse, but prior findings differ across data-building protocols. This paper analyzes diffusion models trained under total replacement and fixed-budget accumulation, combining experiments with a linear-response theory. Fixed-budget degradation is partial and feature-selective, while the theory distinguishes fragile from robust features.

  • Problem

    Prior studies report differing collapse outcomes under replacement and accumulation, leaving the consequences of finite-budget recursive training for diffusion models at issue.

    Source: §1 · pp. 1–2, §7 · p. 6

  • Method

    The paper recursively trains diffusion models under total replacement and fixed-budget accumulation, and analyzes multi-generational parameter dynamics with a linear-response framework.

    Source: §1 · p. 2, §8 · p. 7

  • Results

    Fixed-budget training partially degrades specific features on a 2D spiral and image benchmarks, unlike the rapid deterioration under total replacement.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The analysis identifies fragile features that decay early and robust features that can persist for practically unbounded horizons under fixed-budget training.

    Source: §5 · p. 5, §7 · p. 6

    The theory is a first-order, NTK-style linear response based on assumptions only approximately met by real recursive training, and semantic interpretations of M’s eigenvectors are not consequences of the model.

    Source: §7 · p. 6

Abstract

from arXiv · show

Model collapse arises when generative models are trained on synthetic data produced by earlier models. The phenomenon has attracted considerable attention because of its societal and technical implications. However, previous studies have reached seemingly contradictory conclusions: replacing real data with synthetic data causes collapse (Shumailov et al.), yet accumulating real data alongside synthetic data can prevent it. For diffusion models, we study an intermediate regime typical of finite-budget pipelines: all past datasets and the real data are kept, but each new model is trained on a fixed-size sample from this growing pool, so the real fraction vanishes without any data being removed. Experiments on a 2D spiral dataset as well as the image benchmarks (MNIST, Fashion-MNIST, and CIFAR-10) show that replacement protocol degrades dataset rapidly as in the literature, whereas the fixed budget degrades only partially, sparing some features. A linear-response model of the multi-generational parameter dynamics, analyzed by stochastic recursion, confirms that the two protocols differ: some features will be fragile and lost within a few generations for both protocols, while some will be robust and preserved over practically unbounded horizons under the fixed budget protocol.

1 Introduction

The paper studies recursive training of diffusion models under a fixed training budget, an intermediate regime between replacing real data and accumulating it without a fixed-size constraint. Experiments and a per-feature theory show that fixed-budget degradation is partial, unlike the rapid deterioration under total replacement.

1 Introduction

  • Prior findings differ across recursive-training protocols: replacement can erase distribution tails [2024], while accumulation and other settings yield different collapse outcomes.The authors locate the distinction in how each step builds its training set; fixed training-set size removes the vanishing-noise-share weighting behind accumulation’s protective effect.Source: §1 · pp. 1–2
  • The fixed-budget, accumulate-subsample regime keeps prior data but draws a constant-size sample from the growing pool, so the real-data fraction vanishes.The paper complements prior empirical work on this regime with a per-feature theoretical analysis.Source: §1 · p. 2
  • Fixed-budget training partially degrades specific features on a 2D spiral and image benchmarks, whereas total replacement rapidly deteriorates the data.The fixed-budget behavior differs both from complete collapse and from the bounded-error regime predicted under unbounded accumulation.Source: §1 · p. 2
  • The per-feature analysis models multi-generational parameter dynamics and explains rapid replacement degradation alongside heterogeneous feature evolution under fixed-budget accumulation.It derives per-feature halving times to characterize when fragile features are lost.Source: §1 · p. 2

2 Methodology

The methodology recursively trains diffusion models and generates a synthetic dataset at each generation. It compares training on the latest synthetic data alone with training on a fixed-size sample from the accumulated pool.

2 Methodology

  • Each trained model is a generation, and its parameters are used to sample a synthetic dataset for the next stage.The experiments use DDPM training [Ho et al., 2020] and DDIM sampling [Song et al., 2021].Source: §2 · p. 2
  • The last-generation protocol trains on the previous synthetic dataset alone, while fixed-budget training samples a constant-size set from the growing pool without discarding data.Consequently, the probability that a sampled training example is real decreases over generations.Source: §2 · p. 2
  • Experiments use a 2D spiral and MNIST, Fashion-MNIST, and CIFAR-10, with image evaluation using FID, pixel-FID, and pixel-level W2.For the spiral, W2 is computed by discrete optimal transport; Fréchet distance compares Gaussian approximations of the point clouds.Source: §2 · p. 2

3 Modeling the multi-generational dynamics and Diffusion

The modeling framework approximates diffusion training and data-mixture effects through first-order parameter dynamics. It uses these approximations to describe how recursive datasets evolve across generations.

3 Modeling the multi-generational dynamics and Diffusion

  • The framework treats training and sampling as two mappings using neural-network parameters, while fixing the architecture and sampling prior.It assumes each dataset has a unique parameter representation, while acknowledging that this hypothesis is generally too strong and depends on sufficient model capacity for negligible error.Source: §3 · p. 3

3.1 The simplest Diffusion training equations

  • The training map is approximated by a first-order linear response in the initialization, with an additive centered fluctuation for diffusion noise and linearization error.The matrix M is treated as a symmetric positive-definite local sensitivity operator, with eigenvalues between zero and one.Source: §3.1 · pp. 3–4

3.2 The simplest model for dataset evolution: Linearity with Respect to Data Mixtures

  • For fixed-budget evolution, the model assumes that a mixture of datasets is represented approximately by the corresponding convex mixture of their parameter vectors.The last-generation protocol instead sets the next dataset’s representative to the previous generation’s parameter.Source: §3.2 · p. 4

4 Theoretical results : convergence and large perturbation analysis

The theoretical analysis specifies stochastic recursions for successive model parameters under the two data protocols. It finds different convergence behavior and feature-decay dynamics for fixed-budget accumulation and total replacement.

4 Theoretical results : convergence and large perturbation analysis

  • The two protocols are analyzed using a shared stochastic training equation, but fixed-budget target evolution combines the previous target with the newly trained model.The recursion includes centered, independent training noise with finite variance.Source: §4 · p. 4
  • Under total replacement, target and model parameters converge to a stationary distribution with zero mean but nonzero limiting variance.The stated result assumes independently initialized generations with centered initialization and finite variance.Source: §4 · p. 4
  • Under fixed-budget accumulation, target parameters converge pointwise to zero, while features with small eigenvalues retain initial-data memory far longer than features with large eigenvalues.The analysis associates large eigenvalues with faster feature decay and small eigenvalues with greater resilience.Source: §4 · p. 5

5 Interpretation of the theoretical results

The protocols differ in whether feature memory persists: last-generation dynamics approach a stationary distribution, while fixed-budget training can preserve low-eigenvalue features for extremely long times. Both protocols lose high-eigenvalue features early.

5 Interpretation of the theoretical results

  • Last-generation parameters converge in distribution to a stationary limit, not pointwise, and the limit variance remains positive.When the initialization terms are centered, the average decays exponentially toward zero.Source: §5 · p. 5
  • About 69 generations erase a feature with λ = 0.01 under the last-generation protocol, while a feature with λ = 0.99 takes about one generation.Source: §5 · p. 5
  • Fixed-budget training preserves some low-eigenvalue features for extremely long times, while high-eigenvalue features are lost early.The paper describes these as robust features (eigenvalues near 0) and fragile features (eigenvalues near 1).Source: §5 · p. 5
  • The protocols differ sharply for low-eigenvalue features: last-generation training erases their memory within a finite number of generations, while fixed-budget training can preserve it for practically unbounded horizons.Source: §5 · p. 5

6 Empirical results

The experiments show rapid deterioration under the last-generation protocol and partial deterioration under fixed-budget training. The latter retains some dataset features, while the last-generation protocol reaches higher FID values.

6 Empirical results

  • All features are lost under the last-generation protocol, producing almost-white-noise samples, while fixed-budget samples retain some features.Retained examples include digit structure in MNIST, clothes in Fashion-MNIST, and scene texture in CIFAR-10.Source: Figure 1 · p. 6, §6 · p. 6
  • Fixed-budget deterioration is partial and affects only some features, as shown by the dataset comparison and metric evolution.Source: §6 · p. 6, Figure 1 · p. 6
  • The last-generation protocol reaches much higher FID values, indicating greater departure from the original dataset.Source: §6 · p. 6

7 Discussion and limitations

The analysis frames the protocols' difference in per-feature dynamics, reconciling apparently divergent prior conclusions for diffusion models. The authors connect fragile features to rare or under-represented cases, while noting theoretical and empirical scope limitations.

7 Discussion and limitations

  • The per-feature decay analysis reconciles apparently divergent conclusions about recursive training in diffusion-model settings.The paper cites Shumailov et al. [2024], Gerstgrasser et al. [2024], Seddik et al. [2024], and Barzilai and Shamir [2026].Source: §7 · p. 6
  • Fragile features are associated with tails and low-density regions, including rare and under-represented cases.The authors suggest provenance tracking, synthetic-content disclosure, or refreshing data with real examples as ways to strengthen fragile features.Source: §7 · p. 6
  • The theory is first-order and NTK-style, its assumptions only approximate recursive training, and its eigenvector-to-semantic-attribute mapping is interpretive.Empirically, the study retrains small models from scratch with unconditional DDIM sampling, without sweeping budget size or curating the pool; its metrics measure distributional distance only.Source: §7 · pp. 6–7

8 Conclusion

The conclusion finds rapid, general deterioration under total replacement but partial, feature-selective degradation under a fixed budget, with theory distinguishing fragile from robust features. Its literature positioning emphasizes that recursive-training outcomes depend on how each generation’s training set is built.

8 Conclusion

  • Fixed-budget diffusion training degrades only partially and selectively, unlike the rapid collapse under total replacement, across the spiral and three image benchmarks.The analysis distinguishes fragile features, lost within a few generations, from robust features preserved over practically unbounded horizons under the fixed budget.Source: §8 · p. 7
  • The conclusion identifies semantic feature analysis, larger pretrained and conditional models, pool curation, and fairness-linked halving times as next steps.It frames anticipating the loss of tails and under-represented fragile modes as a potential use of its findings.Source: §8 · p. 7

A Related literature and positioning

  • Prior studies’ conclusions differ in part because their training-set construction differs: growing accumulated data can make each generation’s noise share vanish, while fixed-size sampling removes that weighting.The fixed-budget protocol samples from the accumulated pool without increasing the training-set size.Source: A Related literature and positioning · pp. 8–9
  • Kazdan et al. [2025] introduced accumulate-subsample; this paper adds a per-feature account to its intermediate test-loss and plateau behavior.The cited account associates the plateau with robust features and distinguishes them from fragile features lost as quickly as under replacement.Source: A Related literature and positioning · p. 9
  • The paper’s framework gives per-feature halving times, complementing the stability condition of Bertrand et al. [2024].The fixed-budget real fraction vanishes as 1/(n + 1), and random re-initialization contributes a noise term.Source: A Related literature and positioning · p. 9
  • Zhu et al. [2025] study fixed-proportion synthetic text without long recursive chains, while this paper examines recursive diffusion-model training.Their token-level editing proposal addresses narrowed synthetic-data coverage, described as analogous to loss of low-density features here.Source: A Related literature and positioning · pp. 9–10
  • Shi et al. [2025] study accumulate-subsample empirically over fewer than ten iterations, whereas this work follows the protocol for one hundred generations and analyzes which features survive.Their study measures sample novelty; this paper provides a theoretical account of feature survival.Source: A Related literature and positioning · p. 10

B Classical stochastic recursion convergence results

The appendix invokes stochastic-recursion results to establish convergence and stationary behavior, then applies them to the last-generation protocol. Its feature-wise mean decays geometrically, yielding the stated halving-time criterion.

B Classical stochastic recursion convergence results

  • For the stated stochastic-approximation setup, the associated ODE has a globally asymptotically stable equilibrium y*=M^-1 μ̄.The proof cites a convergence theorem and boundedness result under its listed conditions.Source: B Classical stochastic recursion… · p. 10
  • A stable linear stochastic recursion has a unique stationary distribution, with its series representation converging in L2.The result applies to a symmetric positive definite matrix whose spectrum lies in (0, 1).Source: B Classical stochastic recursion… · pp. 10–11
  • The stationary covariance Σ∞ is the unique solution of the discrete Lyapunov equation determined by A and ΣU.Here ΣU is the covariance of U0, and Σ∞ is the covariance of the stationary limit.Source: B Classical stochastic recursion… · p. 11

C.1 Proof of Theorem 4.1

  • Under the last-generation protocol, the target recursion is T_n+1 = A T_n + U_n, with A := I − M and U_n := MZ_n + ε_n.The stationary-recursion lemma is then used to obtain the convergence result.Source: §C.1 · p. 11
  • The last-generation feature mean decays as m_n = (1 − λ)^n m_0, so its halving time is the smallest n satisfying (1 − λ)^n ≤ 1/2.The passage identifies this criterion with equation (11).Source: §C.1 · p. 11

C.2 Proof of Theorem 4.2

The fixed-budget proof rewrites the parameter update as a stochastic recursion with a generation-dependent step size. It establishes pointwise convergence of the target parameter and analyzes the feature mean’s slower decay.

C.2 Proof of Theorem 4.2

  • The fixed-budget update is T_n+1 = T_n − M T_n/(n + 1) + U_n, with U_n := MZ_n + ε_n.The proof obtains this recursion by substituting the fixed-budget protocol into the parameter update.Source: §C.2 · p. 11
  • The target parameter T_n converges pointwise to zero under the fixed-budget recursion.The proof applies Lemma B.1 to establish this convergence.Source: §C.2 · p. 11
  • The fixed-budget feature mean decays asymptotically as n^-λ, giving a halving time that grows exponentially in 1/λ.The passage gives the asymptotic factor 1/Γ(1 − λ) for the mean and an approximate halving-time expression.Source: §C.2 · p. 11

D Metrics results

The metric plots show rapid distance increases under the last-generation protocol and more tempered increases under fixed-budget training. They cover the spiral and three image datasets.

D Metrics results

  • Distance rapidly increases under the last-generation protocol across the plotted datasets and metrics.Figure 2 plots FID, pixel-FID, and Wasserstein-2 for the 2D spiral, MNIST, Fashion-MNIST, and CIFAR-10.Source: Figure 2 · p. 12
  • Both figures track FID, pixel-FID, and Wasserstein-2 for the 2D spiral, MNIST, Fashion-MNIST, and CIFAR-10.The dataset panels run from the spiral at the top to CIFAR-10 at the bottom.Source: Figure 2 · p. 12, Figure 3 · p. 13
  • Under the fixed-budget protocol, distance increases in all plotted cases, but more slowly than under the last-generation protocol.Figure 3 plots the same metrics and datasets as Figure 2.Source: Figure 3 · p. 13

E Detailed limitations

This section describes the study’s methodological scope, including limitations on interpreting and generalizing its theory and experiments, and the Attention U-Net decoder and training and inference settings.

E Detailed limitations

  • The theory assumes exact dataset parameter representatives, linear parameter interpolation, a fixed-spectrum positive-definite contraction, and independent centered noise and initialization, unlike real recursive training.The authors frame it as a simple relevant linear model rather than a precise account; only additive representation error is absorbed after Theorem 4.2.Source: E Detailed limitations · p. 14
  • The model defines features as eigenvectors of M; identifying them with semantic attributes or distribution tails is an experimental interpretation, not a theoretical result.The interpretation is supported by experiments and analogy with Seddik et al. [2024].Source: E Detailed limitations · p. 14
  • The experiments use unconditional DDIM sampling, so the authors’ positive result does not transfer to guided or conditional generation, where guidance scale drives degradation.They note that the maximum-likelihood-based result of Barzilai and Shamir [2026] does not apply; Yoon et al. [2025] show guidance scale drives degradation.Source: E Detailed limitations · p. 14
  • The study retrains small models from scratch with nd samples, leaving behavior in large pretrained models fine-tuned for few iterations untested.The untested setting is discussed in Bohacek and Farid, 2025 and Yoon et al., 2025.Source: E Detailed limitations · p. 14
  • The study does not vary budget size or pool-mixing ratio, or curate and filter the pool, although curation can change the picture.Bohacek and Farid [2025] report that model curation can change the picture.Source: E Detailed limitations · p. 14
  • FID, pixel-FID, and W2 measure distributional distance rather than downstream utility or explicit fairness, leaving the link between halving times and those outcomes open.Source: E Detailed limitations · p. 14

I.4 Attention U-Net Architecture

  • The decoder upsamples through 8 × 8, 16 × 16, and 32 × 32 resolutions, concatenates skip features, and uses residual blocks before a 1 × 1 convolution predicts three-channel noise.The listed residual-block widths are (4D + 4D) → 4D → 4D, (4D + 2D) → 2D → 2D, and (2D + D) → D → D.Source: §I.4 · p. 21

I.5 Main hyperparameters

  • Training uses 10,000 images for 1000 epochs, with batch size 256, T = 1000 diffusion steps, and 200 DDIM inference steps.Other settings are learning rate 5 × 10−4, EMA decay 0.8, base channel width D = 8, time embedding dimension 64, projected time dimension 32, and 4 attention heads.Source: §I.5 · p. 21
Loading 2610.01318v1…