PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Consuming Generative Models with Co-Evolving Human Preferences

Xiukun Wei, Tian Xie, Ding Zhu, Xueru Zhang

arXiv:2610.09415v1cs.LG

TL;DR

Prior work largely treats user preferences as fixed, although model exposure can reshape those preferences and create coupled model-user dynamics. The paper formulates and analyzes these dynamics, then studies how reference-data mixing can steer their equilibria. Purely synthetic curation can produce singleton equilibria dominated by an initially advantaged instance, whereas sufficient reference mixing yields a unique globally attracting equilibrium.

  • Problem

    Prior work largely treats preferences as fixed, despite evidence that model outputs can reshape user preferences and thereby couple user and model dynamics.

    Source: §1 · p. 1

  • Method

    The paper models discrete-time co-evolution of model distributions and user preferences, analyzes equilibrium behavior, and formulates reference-data steering as a constrained optimization problem.

    Source: §1 · p. 2

  • Results

    Purely synthetic curation can drive the system toward singleton equilibria dominated by an initially advantaged instance, while sufficient reference mixing yields a unique globally attracting equilibrium.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Jointly choosing the reference distribution and its mixing weight can steer the equilibrium to preserve desired attributes while minimizing data-collection cost.

    Source: §1 · p. 2

    The main analysis uses a finite instance space of distinguishable categories; extension to continuous spaces is deferred to the appendix.

    Source: §2 · p. 3

Abstract

from arXiv · show

Generative models are increasingly trained in self-consuming iterative loops, where users curate preferred samples from model-generated candidates and the curated samples are used to train future generations of the model. Prior work has largely assumed fixed user preferences, but in practice exposure to model outputs gradually reshapes what users perceive as desirable, creating a feedback loop in which model distributions and user preferences co-evolve. We take a first step toward understanding the long-term behavior of such coupled dynamics. We show that when training relies entirely on user-curated synthetic data, iterative curation amplifies initial biases and drives the system toward one of multiple singleton equilibria in which the instance holding an initial advantage eventually dominates. In contrast, injecting reference data into training at a sufficiently large rate fundamentally changes the dynamics and yields a unique globally attracting equilibrium. Building on this insight, we study how reference-data injection can be used to control long-term outcomes, and propose an efficient algorithm that jointly selects a reference distribution and its mixing weight to steer the coupled system toward equilibria that preserve desired attributes while minimizing data collection costs.

1 Introduction

The paper models a closed loop in which human preferences and generative-model outputs co-evolve, addressing a gap in work that treats preferences as fixed. It analyzes how reference data changes long-run outcomes and how reference data can steer them.

1 Introduction

  • Unlike prior work that largely assumes fixed preferences, this paper studies settings where model exposure reshapes what users find desirable.Prior work describes human curation of synthetic outputs as steering models toward user preferences.Source: §1 · p. 1
  • The proposed model couples preference-guided curation and model retraining with preference drift toward the updated model’s feature direction.The model generates candidates, users curate one, and the curated set may be mixed with reference data before retraining.Source: §1 · p. 2, Figure 1 · p. 2
  • With purely synthetic curation, any item with a sufficient initial reward advantage eventually dominates; sufficient reference-data mixing instead yields a unique globally attracting equilibrium.Under the stated mixing condition, convergence is geometric and independent of initialization.Source: §1 · p. 2
  • The paper formulates equilibrium steering as a constrained optimization problem and reduces a worst-case sufficient condition to linear programs for jointly choosing the reference distribution and mixing weight.The objective is to preserve specified diversity at equilibrium while minimizing data-collection cost.Source: §1 · p. 2
  • Experiments on image generation with CFM and DDPM and text generation with GPT-2 verify the theoretical results and the proposed algorithm’s effectiveness.The introduction identifies CIFAR-10 and AG News as the evaluation settings.Source: §1 · p. 2

2 Problem Formulation

The formulation represents a finite set of output categories, preference-guided selection among model-generated candidates, and retraining on curated data optionally mixed with reference data. User preferences drift toward the updated model’s mean feature vector, and the paper studies when the coupled system converges and how reference data can steer its equilibrium.

2 Problem Formulation

  • The main analysis uses a finite instance space of distinguishable output categories; extending the formulation to continuous spaces is deferred to the appendix.The model distribution at round t is denoted p_t.Source: §2 · p. 3
  • Users select one of K generated candidates according to feature alignment with their preference vector, represented by a fixed unit-norm feature map and reward.The K-way Luce rule governs this preference-based selection.Source: §2 · p. 3
  • The next model is trained on user-curated synthetic data, optionally mixed with a fixed reference distribution at a rate controlled by η.The retraining objective admits a multiplicative reweighting of the current model distribution.Source: §2 · p. 3
  • After retraining, preferences adapt toward the updated model’s mean feature vector; together, these updates define a joint dynamical system over model and preference states.The preference adaptation rate is β, with projection onto the allowed preference set.Source: §2 · p. 4
  • The analysis asks how candidate-set size, synthetic-data weight, and feature-map geometry affect convergence, equilibria, and collapse versus stabilization.A related objective is to choose the reference distribution and mixing weight to preserve desired attributes while limiting intervention cost.Source: §2 · p. 4

3 Long-Term Behavior of User–Model Dynamics

The section characterizes how model distributions and user preferences evolve under purely synthetic retraining versus reference-data mixing. Pure curation can lock in an initially advantaged instance, while sufficient reference mixing yields a unique equilibrium attracting every trajectory.

3.1 Retraining with purely synthetic user-curated data

  • Pure-synthetic fixed points require all instances in the model distribution’s support to share the same reward, because curation shifts mass toward higher-reward instances.Repeated odds contraction compounds the advantage of higher-reward instances and suppresses lower-reward ones.Source: §3.1 · pp. 4–5
  • Under purely synthetic retraining, there are n singleton equilibria, and with separated feature directions, a sufficient initial advantage lets an instance lock in as the long-run outcome.The mass on competing instances decays geometrically; duplicate feature directions prevent local stability of the corresponding singleton equilibrium.Source: §3.1 · p. 5
  • Larger choice sets K accelerate convergence toward a singleton equilibrium because they make curation more selective.As K approaches infinity, the update approaches exponential tilting.Source: §3.1 · p. 5

3.2 Retraining on mixed synthetic and reference data

  • With preferences held fixed, reference mixing yields a unique model-distribution equilibrium when ηLp < 1; as η varies, the equilibrium balances reference mass against preference-driven concentration.As η approaches 0 the equilibrium recovers the reference distribution; as η approaches 1, the tilt dominates and mass concentrates on the highest-reward instance.Source: §3.2 · p. 6
  • Sufficient reference mixing makes the coupled update contractive, yielding a unique equilibrium to which every trajectory converges geometrically, regardless of initialization.The contraction condition is η(Lp + 2Lw) < 1; preference drift adds the Lw stability cost beyond the frozen-preference condition.Source: §3.2 · pp. 6–7

4 Steering Equilibria via Reference Data Mixing

The section formulates reference-data design as minimizing collection cost while preserving specified features at equilibrium. A worst-case sufficient condition replaces direct equilibrium computation with linear programs and an outer search over the mixing weight.

4 Steering Equilibria via Reference Data Mixing

  • The design objective minimizes per-round reference-data collection cost while ensuring the equilibrium represents each specified feature above its threshold.The reference distribution allocates collected samples across instances, whose per-sample costs can differ.Source: §4 · p. 7
  • A worst-case relaxation replaces the intractable equilibrium constraint with a sufficient condition that reduces the design problem to linear programs indexed by η.The original constraint depends on a nonlinear coupled-map fixed point with no closed form.Source: §4 · pp. 7–8
  • The offline procedure searches over η and solves an inner linear program for the reference distribution; the inner problem has n variables and L + 1 constraints.A larger η reduces direct reference-sample cost but tightens the preservation constraint, so the outer search balances these effects.Source: §4 · p. 8
  • For fixed η, the optimal reference distribution satisfying the preservation condition can concentrate on at most L + 1 instances.This follows from the linear program’s structure; the outer search selects η to minimize per-round collection cost.Source: §4 · p. 8

5 Experiments

The experiments illustrate initialization-dependent lock-in under pure synthetic training and stabilization under reference mixing, then test a designed reference distribution for preserving minority classes. The designed reference maintains protected-class proportions above the target while requiring less reference data than uniform baselines.

Experimental setup

  • The empirical study models finite samples and imperfect training over 20 rounds, using 10,000 generated candidates and selecting 5,000 per round for OT-CFM on CIFAR-10.Each setting is reported as a single run in the main experiments; the appendix reports three independent runs under the same setting.Source: §5 · pp. 8–9

Initialization and mixing

  • Under pure synthetic training, airplane rose from about 10% to over 95%, while a cat-preferring initialization instead drove the model toward cat.The experiments used CIFAR-10 and AG News, with OT-CFM, DDPM, and GPT-2 architectures; the main text reports OT-CFM results.Source: §5 · pp. 8–9, Figure 2 · p. 9
  • With η=0.1, class proportions stayed close to uniform across 20 rounds for both airplane- and cat-preferring initializations.The two runs converged to a similar equilibrium, consistent with the uniqueness guarantee for sufficiently small η.Source: §5 · p. 9, Figure 2 · p. 9

Reference-data design

  • At round 20, every protected class exceeded θℓ=0.08 under the designed reference, which used η=0.67 versus either uniform baseline.The designed reference therefore used a higher synthetic ratio, reducing the amount of reference data required.Source: §5 · pp. 9–10, Figure 3 · p. 10, Table 2 · p. 10

6 Discussion

The analysis focuses on a basic co-evolutionary setting with one aggregate preference state interacting with a self-consuming generative model through preference-guided curation.

Discussion

  • The discussion scopes the analysis to a single aggregate preference state coupled to one self-consuming generative model through preference-guided curation.The paper identifies nonlinear preference responses and heterogeneous user populations as extensions, with stability analysis becoming response-dependent.Source: §6 · p. 10
  • Stochastic individual adaptation offers a micro-level interpretation of the deterministic preference update studied in the paper.Heterogeneous user populations would instead involve group-specific preferences coupled through a shared generative distribution.Source: §6 · p. 10
  • Nonlinear gap-response updates can include saturating and state-dependent gated responses, while leaving the model-update branch unchanged.The stability analysis for these extensions becomes response-dependent.Source: §6 · p. 10

7 Conclusion

The conclusion summarizes a coupled model-preference dynamic, its distinct pure-synthetic and reference-mixed behaviors, and a linear-program approach to equilibrium steering. The supplied sections also relate the framework to prior work, illustrate it across discrete and continuous examples, and prove properties of the curation update.

7 Conclusion

  • When η=1, an instance with sufficient initial advantage can lock in at a singleton equilibrium; if η(Lp + 2Lw) < 1, the coupled update is a contraction with a unique globally attracting equilibrium.The paper formulates equilibrium steering as a bilevel problem and reduces it to linear programs indexed by η.Source: §7 · p. 10

A Related Work

  • Prior work studies self-consuming loops, fixed-signal human curation, performative prediction, and evolving preferences, while this paper couples preference-guided retraining with endogenous preference change.The related-work discussion contrasts the framework with recommendation settings that use a fixed catalog rather than generative retraining on curated outputs.Source: A Related Work · p. 16

B Complete Instantiation Examples

B.1 Discrete Examples

  • In the discrete examples, curation can make casual text dominate while poetic and technical writing vanish, or drive airplane content upward while frog and horse disappear.The text example proposes protecting poetic and technical writing; the image example describes airplane reward of about 0.84 and near-zero rewards for cat and frog.Source: §B.1 · pp. 17–18

B.2 Continuous Examples

  • In the continuous CLIP example, curation suppresses monochrome images as the model distribution and preference vector move toward vibrant imagery.The example proposes collecting 500 monochrome reference images and preserving a minimum projection onto the monochrome direction.Source: §B.2 · pp. 18–19

C.1 Proof of Explicit Retraining Update

  • The explicit retraining-update proof establishes total-variation convergence of the curated distribution to its large-K exponential-reweighting limit.The argument invokes the strong law of large numbers and dominated convergence.Source: §C.1 · p. 20

C.2 Proof of Propostion 3.1

The curation operator preserves reward ordering and contracts the odds between instances under the stated bound. Its definition follows the probability of retaining a sample under K-way preference-weighted selection.

C.2 Proof of Proposition 3.1

  • The curation operator is strictly increasing in reward, so an instance with higher reward receives higher curated probability.The operator also reflects strict reward order.Source: §C.2 · pp. 20–21
  • The retained-sample distribution is obtained by averaging the K-way Luce selection probabilities over the candidate samples.The selection probability weights each candidate by its exponentiated reward.Source: §C.2 · p. 21
  • The odds between two instances contract by the bound λK(rΔ) given for the curation operator.The proof derives the bound by substituting the stated expression into the odds ratio.Source: §C.2 · p. 22

C.3 Proof of Theorem 3.2

Under purely synthetic retraining, a distribution is fixed exactly when all instances in its support have equal reward under the current preference. Every instance also yields a singleton equilibrium when preferences align with its features.

C.3 Proof of Theorem 3.2

  • A distribution is a fixed point of retraining under preference w if and only if its support contains only instances with the same reward.The condition allows multiple support points when their rewards match.Source: §C.3 · p. 23
  • For every instance xi, the point mass δxi paired with preference φ(xi) is an equilibrium of the pure coupled dynamics.At this equilibrium the model is concentrated on xi and preferences align with its features.Source: §C.3 · p. 23
  • The fixed-point criterion follows because unequal rewards among supported instances contradict the data-update fixed-point condition.The proof uses strict monotonicity of the curation probabilities in reward.Source: §C.3 · p. 23

C.4 Proof of Theorem 3.3

The appendix establishes local convergence to singleton equilibria under a positive reward gap, while duplicate feature representations undermine local stability. It also develops sensitivity bounds and characterizes equilibria when preferences are frozen.

C.4 Proof of Theorem 3.3

  • With distinct feature vectors and a positive reward gap, trajectories from the theorem’s specified initial region converge toward a singleton equilibrium as off-winner mass decays geometrically.The proof establishes forward invariance of the region and geometric decay of off-winner mass.Source: §C.4 · pp. 24–25

C.5 Proof of Corollary 3.5

  • If another instance shares xi’s feature vector, the singleton equilibrium at xi is not locally asymptotically stable.Nearby equilibria with mass on both instances approach the singleton as their secondary mass vanishes.Source: §C.5 · p. 26

C.6 Proof of Proposition 3.6

  • The sensitivity lemma establishes bounds for the retraining step across distributions and preferences, including Lipschitz continuity in each input.It also establishes Lipschitz continuity of the mean-feature operator.Source: §C.6 · pp. 26, 28, 29

C.7 Proofs of Lemma 3.7 and Proposition 3.8

  • For frozen preferences, reference mixing yields a unique model equilibrium when ηLp < 1, and every trajectory converges to it geometrically.The appendix also states a closed-form equilibrium in the limit K → ∞, but the supplied passage does not give its formula.Source: §C.7 · p. 29

C.8 Proof of Theorem 3.9

The appendix proves global convergence of the reinjected coupled system under a contraction condition and gives a sufficient condition for preserving specified attributes. It extends the analysis to compact instance spaces, where pure-curation equilibria and local convergence require additional characterization and assumptions.

C.8 Proof of Theorem 3.9

  • When η(Lp + 2Lw) < 1, the coupled reinjected update is a strict contraction with a unique equilibrium to which every trajectory converges geometrically.The proof applies a weighted distance to the joint model-preference state.Source: §C.8 · pp. 30–31

C.9 Proof of Proposition 4.1

  • Proposition 4.1 gives a sufficient condition under which the unique equilibrium satisfies the preservation constraint.The supplied passages do not specify the condition’s full inequality.Source: §C.9 · p. 32

D Extension to Continuous Instance Spaces

D.1 Retraining with Purely Synthetic User-Curated Data

  • On compact spaces under pure curation, a distribution is a retraining fixed point exactly when reward is constant on its support; every point mass paired with its feature vector is an equilibrium.The fixed-point criterion is stated for η = 1.Source: §2 · p. 34

2. Normalization. R

  • Local convergence to a singleton on compact spaces requires the multiscale reward-gap assumption; identical feature vectors at distinct points instead rule out local asymptotic stability.The multiscale conditions address competitors arbitrarily close to the target, whose reward disadvantage can be arbitrarily small.Source: §2 · pp. 34–35

D.2 Retraining on Mixed Synthetic and Reference Data

The appendix extends the mixed-training convergence analysis and tests how mixing, curation settings, and reference-data design affect long-run class proportions. Across CIFAR-10 image models, reference mixing stabilizes diversity, while pure synthetic training produces initialization-dependent concentration.

D.2 Retraining on Mixed Synthetic and Reference Data

  • When η(Lp + 2Lw) < 1, the coupled update is a strict contraction with a unique equilibrium and geometric convergence.For fixed preferences, the model update is likewise a strict contraction when ηLp < 1, yielding a unique equilibrium.Source: §D.2 · pp. 36–38

E.1.1 Effect of Key Parameters

  • At η = 1.0, airplane exceeds 95% by round 20; at η = 0.2, it stays below 50% and other classes remain above θ = 0.08.Larger β weakens and slows concentration; τ = 1.0 keeps class proportions near balanced, while K = ∞ and K = 10 yield nearly identical concentration trajectories.Source: §E.1.1 · pp. 39–40, Figure 4 · p. 39

E.1.2 Reference data design

  • With heterogeneous class thresholds, the feasibility boundary shifts to η∗ = 0.61, and the LP allocates more reference mass to classes with higher thresholds.The uniform-threshold setting has a feasibility boundary of η∗ = 0.67.Source: §E.1.2 · p. 40, Table 4 · p. 40

Designed reference

  • The designed reference strategy is evaluated against uniform reference using total cost and each protected class’s round-20 proportion.Table 5 defines J as total cost, (1 − η) · N · C∗.Source: Table 5 · p. 41

E.1.3 Generated Samples

  • The designed reference distribution keeps all four protected classes above θℓ = 0.08 by round 20.The visualized samples illustrate output evolution under different reference strategies; under pure synthetic training, samples collapse to airplane images, while uniform reference mixing increasingly preserves diversity.Source: §E.1.3 · p. 40, Figure 9 · p. 41

E.2 DDPM Experiments on CIFAR-10

  • Under pure synthetic training, DDPM shows initialization-dependent lock-in; with η = 0.1, both tested initializations converge to a similar near-uniform equilibrium.DDPM concentration is slower than CFM, reaching 60% by round 10; the authors attribute this to image-quality degradation and note that finite capacity and training error violate the idealized training-mixture assumption.Source: Uniform reference · p. 42, Figure 11 · p. 42

Uniform reference

  • Across three DDPM seeds with η = 1.0, airplane proportion reaches 58.29%–59.02% by round 10.The runs show consistent trajectories and similar preference-vector paths.Source: Uniform reference · p. 42, Figure 12 · p. 43
  • DDPM samples become blurrier over successive rounds, especially under η = 1.0, while pure synthetic training still concentrates on one class and reference mixing preserves diversity.The authors describe the increasing blur and loss of fine detail as accumulated training error across generations.Source: Uniform reference · p. 42, Figure 13 · p. 43

E.3 GPT-2 Experiments on AG News

The GPT-2 experiments test whether the image-model co-evolution patterns also appear in text generation. They report initialization-dependent lock-in under pure synthetic training and stabilization with reference mixing, while noting that GPT-2 departs from the theory’s idealized assumptions.

E.3 GPT-2 Experiments on AG News

  • With pure synthetic training, the Sports proportion exceeds 90% by round 10, and SciTech likewise locks in; smaller η stabilizes both initializations.The experiments use GPT-2-small on AG News, with a pretrained BERT classifier providing class features.Source: §E.3 · p. 43, Figure 14 · p. 44, Figure 15 · p. 44
  • GPT-2’s generated class proportions are not uniform even with uniformly distributed reference data, and proportions fluctuate across rounds rather than converging smoothly.The authors attribute these deviations to imperfect learning and possible fine-tuning instability on small datasets, while reporting that qualitative lock-in and stabilization patterns remain.Source: §E.3 · p. 43

F Discussion and Extensions

  • The discussion frames the analysis as a basic setting with one aggregate preference state interacting with a self-consuming model.It points to richer preference dynamics and population structures as extensions.Source: F Discussion and Extensions · p. 44

F.1 Nonlinear Preference Responses

The discussion considers nonlinear and stochastic preference updates and multiple user groups as extensions of the aggregate-preference model. It identifies which analysis components remain applicable and which open questions require further work.

F.1 Nonlinear Preference Responses

  • Nonlinear gap responses preserve the curation and retraining results, but contraction conditions and convergence rates depend on the response, and explicit bounds for particular responses remain future work.The discussion considers saturating and state-dependent gated responses; singleton equilibria remain fixed when the response vanishes at zero gap.Source: §F.1 · p. 45

F.2 Stochastic Preference Adaptation

  • Stochastic individual-level adaptation recovers the deterministic preference update as conditional-mean dynamics, but concentration around those dynamics and stochastic lock-in behavior remain open questions.The discussion also raises whether reference mixing yields stochastic stability or a concentrated stationary distribution.Source: §F.2 · p. 45

F.3 Heterogeneous User Populations

  • With multiple user groups, groups affect one another indirectly through the shared evolving model; characterizing equilibrium and stability and balancing group-specific preservation goals remain open.Source: §F.3 · p. 46

G Broader impacts

  • The steering mechanism is value-neutral about which attributes are preserved and could be used to filter out specific content, making transparent attribute selection important in deployment.Source: G Broader impacts · p. 46

Related papers

Loading 2610.09415v1…