Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferences

1 Introduction

• LAION-5B and other datasets already contain synthetic content, including generated images in a large-scale web-crawled dataset used to train future models.The passage also identifies JourneyDB and SAC as datasets composed of synthetic data.Source: §1 · p. 1
• Human curation is common in generated-image datasets: LAION-Aesthetics uses reward-model curation, while JourneyDB includes user-picked, upscaled Midjourney images.Popular text-to-image interfaces also offer several alternatives for a prompt for users to choose among.Source: §1 · pp. 1–2
• Prior iterative-retraining studies did not account for curated generated data, despite reported benefits for downstream tasks and generative models and concerns about bias amplification.The cited prior work reports both benefits and potential issues; the passage does not resolve their balance.Source: §1 · p. 2
• The paper aims to characterize how synthetic-data curation relates to its underlying reward model and the distribution learned through retraining.Its contributions include theoretical analyses of reward and stability, plus experiments illustrating bias amplification from classifier-confidence curation.Source: §1 · p. 2

2 Iterative retraining with curated synthetic data

• Users select one of K samples according to a reward-based discrete choice model, and the next model trains on curated samples or a mixture with reference samples.The reference may be real data or the initial generative model.Source: §2 · p. 3

2.1 Iterative retraining only on the curated synthetic samples

• In the K→∞ limit, t rounds of curated retraining are equivalent to RLHF from the initial model with β=1/t.The corresponding regularization parameter is inversely proportional to the number of retraining steps.Source: §2.1 · p. 4
• Under Assumption 2.1, each retraining iteration increases expected reward in proportion to reward variance; expected reward converges toward reward maximizers, and the variance vanishes.A fixed point must put mass on a single level set of the reward function.Source: §2.1 · pp. 4–5
• The KL-convergence result for purely curated retraining additionally assumes positive initial mass on the maximum-reward level set; without it, support can vanish toward the maximizer.Under this additional assumption, the loop converges to the initial density restricted to that level set and renormalized.Source: §2.1 · p. 5
• The purely synthetic loop converges to the initial distribution restricted to its highest-reward level set, so probability on lower-reward regions vanishes and diversity can collapse.The stated consequence applies in the limit of many retraining steps.Source: §2.1 · p. 5

2.2.1 Iterative retraining without curation

• When Lε < α and λ < α 2Lε, for any initial parameters θ0 in a neighborhood of θ*, pθt converges to pθ* exponentially fast, with the result expressed in distribution.Source: §2.2.1 · p. 6

2.2.2 Iterative retraining on a mixture of real and curated samples

• With real and curated samples, expected reward increases relative to initialization, while a bound on KL divergence keeps the retrained distribution close to its reference.Using real data as reference bounds deviation from the data distribution; reusing the initial model as reference stabilizes retraining around initialization, and small K acts as a regularizer.Source: §2.2.2 · pp. 6–7

3 Related work

• Prior work reports degradation in fully synthetic retraining and stability when real and synthetic data are mixed, while other studies examine performance bounds and bias amplification.The cited literature also includes analyses of large language models and large-scale datasets.Source: §3 · p. 7
• RLHF fits a reward model from pairwise human comparisons, then maximizes expected reward with KL regularization; DPO and other variants offer alternative preference-optimization approaches.The passage names IPO and KTO alongside DPO.Source: §3 · p. 7

4.1 Natural images on CIFAR10

• On CIFAR-10, filtering by airplane-class probability makes the airplane share dominate as average reward rises across retraining.The authors use r(x) = γ · q0(x), with γ = 5.Source: Figure 2 · p. 8, §4.1 · p. 8

4 Experiments

• On the two synthetic datasets, curated-only retraining collapses to one Gaussian mode or one moon, while real-data mixing stabilizes training and raises density in high-reward regions.The Gaussian experiment clips the distance-based reward to avoid collapse to a single point.Source: §4 · pp. 7–8

4.1 Natural images on CIFAR10

• For CIFAR-10, the experiments use a pretrained normalizing flow, retain 2.5 · 10^3 of 5 · 10^4 generated samples per iteration, and score them with a pretrained VGG11 classifier.The classifier has 92.39% test accuracy; the reported figures show one run each, with three-run curves in the appendix.Source: §4.1 · p. 8
• With classifier confidence as reward, average reward rises while class proportions become increasingly heterogeneous during synthetic-only retraining.The figure contrasts this synthetic-only setting with retraining on a real-and-curated-synthetic mixture.Source: §4.1 · p. 8, Figure 3 · p. 9
• Mixing real and curated synthetic samples keeps CIFAR-10 class proportions approximately uniform as average reward rises and then stabilizes.The reported mixture uses hyperparameter λ = 1/2.Source: §4.1 · pp. 8–9, Figure 3 · p. 9

• The paper reports that curated-sample retraining increases expected reward, collapses its variance, and converges, with stability guarantees when real data is reused.The authors connect the real-data results to RLHF and preference optimization.Source: §5 · p. 9
• The authors do not propose an algorithm to address emerging bias amplification, describing it as beyond the paper’s scope.They call it a research priority and suggest studying how the filtering reward function affects bias amplification.Source: §5 · p. 9
• The conclusion characterizes curated synthetic data on the web as a mechanism that automatically optimizes preferences for future trained large models.Source: §5 · p. 9

• The effects of synthetic-dataset curation depend on the curator and the underlying reward function.Source: §6 · p. 9
• The Simulacra Aesthetic Captions dataset documentation describes its human curators as a group lacking diversity and mostly WEIRD individuals.The paper says similar bias is likely in JourneyDB and synthetic data appearing on the web.Source: §6 · p. 9
• The authors frame their theoretical analysis as raising awareness of implicit alignment and potential bias amplification in self-consuming generative models.They state that they believe this awareness outweighs potential unforeseen negative consequences of the work.Source: §6 · pp. 9–10

• The proof samples K candidates independently from the current distribution and selects an index using a generalized Bradley-Terry model.Source: §A.1.1 · p. 14
• Independence and symmetry make the K candidate-specific terms in the resulting density equal, allowing the proof to rewrite it using one term.Source: §A.1.1 · p. 14
• For K tending to infinity, the proof applies the law of large numbers to exp(r(x)) under the assumption that its expectation is finite.Source: §A.1.1 · p. 14

• Expected reward increases at each step even without assuming that reward is bounded.Source: §A.1.2 · p. 15
• K-wise filtering increases the expected reward at each retraining step.Source: §A.1.2 · p. 15
• The proof uses Jensen’s inequality with a convex function to establish the increase.It identifies x ↦ a/(b+x) as convex for a, b > 0.Source: §A.1.2 · p. 15

• At each retraining iteration, expected reward increases proportionally to its variance under Assumption 2.1.Source: §A.1.3 · p. 15
• Expected reward converges to the maximum reward, and reward variance vanishes.Source: §A.1.3 · pp. 15, 17
• For every ε > 0, the proof maintains probability at least δ on rewards within ε of r*, then uses this bound to establish convergence.The proof uses the increasing expected reward to show it eventually reaches an ε-dependent threshold.Source: §A.1.3 · p. 17
• The variance converges to zero because it is positive and has a finite sum over retraining steps.Source: §A.1.3 · p. 17

A.1.4 Fixed points of the retraining loop with filtering

• A fixed point of the filtering loop must place all probability mass on a single reward level set, and any such distribution is a fixed point.The proof rules out a fixed point when reward values with disjoint intervals both have positive probability.Source: §A.1.4 · p. 17

A.1.5 Proof of Theorem 2.1

• Theorem 2.1 states that the self-consuming loop on curated samples pt converges to p*, while the cited argument concludes that Pt(r(x) = r*) tends to 1.Source: §A.1.5 · pp. 17, 19

A.1.8 Additional lemma: retraining on a convex combination of previous iterations

• Retraining on a fixed-ratio convex combination of previous iterations becomes constant after the predefined initial iterations and therefore need not converge to the data distribution.The discussion says convergence requires decreasing the relative share of previous iterations or using only the current iteration's distribution.Source: §A.1.8 · pp. 20–21

A.1.10 Proof of Theorem 2.2

• With Lε < α and λ < α/(2Lε), the mixed-data process converges exponentially fast from initial parameters near θ∗.The convergence is stated for a neighborhood of the optimal distribution parameters.Source: §A.1.9 · p. 21, §A.1.10 · p. 22

A.2.1 MoG and two moons datasets - DDPM

• Fully curated synthetic retraining collapses to one high-reward Gaussian mode or one moon, whereas mixing real data with curated samples provides stability.The experiments use a DDPM and retrain for five iterations.Source: §A.2.1 · p. 23, Figure 4 · p. 24, Figure 5 · p. 24

A.2.2 FID, precision, recall

• FID increases during retraining in the first two CIFAR-10 settings, precision rises in all three, and reusing real data produces much smaller FID variations than confidence-based filtering alone.Confidence-based filtering yields a larger precision increase than the other two settings.Source: §A.2.2 · p. 24, Figure 8 · p. 25, Figure 7 · p. 25

A.2.3 Compute Cost

• CIFAR-10 retraining for 20 iterations with 50,000 generated samples took about 22 hours on an A100 GPU with four workers.The synthetic-dataset experiments ran on a single GPU in a few minutes.Source: §A.2.3 · p. 26

A.3 Examples of sets of four generated images on MidJourney and Stable Diffusion 2.1

• The examples show users choosing among four generated images, with Midjourney allowing the selected image to be upscaled.The upscaled Midjourney images are incorporated into JourneyDB.Source: §A.3 · p. 26, Figure 10 · p. 27