Source-linked AI summary
Mitigating Model Collapse through Dreaming Learning
Matteo Benati, Alessandro Londei, Denise Lanzieri, Raffaello Mastromarino, Vittorio Loreto
TL;DR
Synthetic-data retraining can reduce generative-model diversity, and the paper studies this degeneration through the spectrum of a learned transition operator. It analyzes Dreaming Learning, which trains on the model’s own higher-temperature samples alongside real data; the measured results show mitigation across Markov and language-model settings, though GPT-2 drift is slowed rather than arrested.
Problem
Retraining on synthetic data lowers diversity and performance, while existing accumulation analyses commonly fix sampling temperature at Ts = 1 and lack consensus on real-to-synthetic ratios.
Source: §1 · pp. 1–2Method
The paper tracks learned transition-operator spectra and evaluates Dreaming Learning, which adds a loss on the model’s own higher-temperature samples to regular training.
Source: §1 · p. 2, §4.1 · p. 3Results
Dreaming keeps subdominant eigenvalues approximately constant over 5000 Markov-chain generations and yields flat LSTM validation loss, but slows rather than arrests GPT-2 drift.
Source: §4.1 · p. 3, §4.2 · pp. 3–4Takeaways & Limitations
The spectrum reveals structural change beyond scalar loss, but Dreaming Learning’s stable spectrum can remain biased, with the learned chain mixing faster than the target.
Source: §6 · p. 5At GPT-2 scale, empirical tuning is the main scaling obstacle, and dreaming roughly doubles per-step generation cost.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Training generative models on synthetic data causes loss of diversity and a decrease in performance, a phenomenon known as model collapse. We study this degeneration spectrally in a controlled Markov setting. The eigenvalue spectrum of the learned transition operator offers a direct view of the process, since the subdominant eigenvalues set how fast the chain mixes and therefore how much of the state space it continues to explore. We find that sampling at temperature $T_s < 1$ acts on the operator like repeated sharpening, pushing it toward a deterministic system: the subdominant eigenvalues approach the unit circle and the spectral gap narrows. When $T_s < 1$, even data accumulation does not appear to arrest this drift. We then analyze Dreaming Learning, a regulariser that, in parallel with regular training, adds a loss term on the model's own samples drawn at a raised temperature $T_d > 1$. Because high-temperature samples flatten the predicted distribution, training on them keeps the entries of the learned operator from becoming vanishingly small, which in turn keeps the eigenvalues bounded away from the unit circle. Measurements of the spectra over 5000 generations are consistent with this picture, and the mitigation effect persists in more complex settings such as LSTM language models and a 124M-parameter GPT-2.
1 Introduction
The introduction frames synthetic-data retraining as a diversity and entropy problem, then motivates tracking structural change through the learned transition operator’s spectrum.
1 Introduction
- Retraining on model-generated content lowers diversity and drives learned distributions toward low entropy.The paper describes this effect across GANs, diffusion models, and language models.Source: §1 · p. 1
- Accumulation is the leading mitigation, but analyses commonly fix sampling at Ts = 1, real-to-synthetic ratios lack consensus, and dataset provenance may be unclear.These constraints can make accumulation suboptimal.Source: §1 · p. 2
- The paper proposes a spectral view of collapse and Dreaming Learning, and measures spectra across Markov chains, LSTMs, and a 124M-parameter GPT-2.Its proposed mechanism is that training on Ts < 1 samples sharpens the operator, while a loss on Td > 1 samples keeps its entries from approaching zero.Source: §1 · p. 2
2 Collapse and the spectral gap
The section links low-temperature self-training to repeated transition-matrix sharpening and a narrowing spectral gap, while high-temperature sampling acts in the opposite direction.
2 Collapse and the spectral gap
- The second-largest eigenvalue modulus, |λ2|, sets the chain’s asymptotic convergence rate to stationarity and thus reflects how quickly it forgets its current state.Finite samples can also under-represent conditional tails even with unbiased sampling.Source: §2 · p. 2
- At Ts < 1, repeated row-wise sharpening concentrates transition rows on their largest entries and drives the matrix toward determinism.Deterministic matrices with cycles have subdominant eigenvalues approaching the unit circle, narrowing the spectral gap.Source: §2 · p. 2
- At Td > 1, flattening rows keeps transition entries away from zero; a positive entry floor bounds non-Perron eigenvalues away from 1.Dreaming Learning supplies this floor through high-temperature samples that spread mass across states.Source: §2 · p. 2
3 Dreaming Learning
Dreaming Learning adds training on the model’s own recursively generated, higher-temperature sequences to ordinary real-data training, using the additional loss as a regularizer.
3 Dreaming Learning
- Dreaming Learning adds an auxiliary phase in which a probabilistic model generates synthetic sequences and learns from them.The model’s output must be mappable back into its input space.Source: §3 · p. 2
- The dreaming batch consists of recursively sampled sequences from the model’s own Softmax outputs tempered at Td, and its loss is cross-entropy.The combined objective adds this term to real-data next-token cross-entropy, weighted by α.Source: §3 · p. 3
- Unlike the original formulation’s regime-shift adaptation, this epoch-based implementation uses dreaming to preserve statistical features across repeated retraining.The authors describe the combined losses as fitting real data while exploring the model’s own Adjacent Possible.Source: §3 · p. 3
4 Experiments
Across controlled Markov chains and language models, the experiments test recursive training on self-generated data and Dreaming Learning. Dreaming stabilizes the Markov spectrum and slows degradation in language models, though it does not arrest GPT-2 drift.
4.1 Markov chains
- The Markov experiment accumulates 5000 symbols sampled at Ts = 0.7 per generation and retrains a feed-forward next-symbol model on the growing dataset.The model’s outputs provide a transition matrix at each generation; results cover 10 seeds.Source: §4.1 · p. 3
- At Td = 1.5 and α = 0.8, the 5000-generation stationary-distribution KL divergence improves by > 2 orders of magnitude.The passage relates this improvement to strong dreaming keeping transition probabilities away from zero.Source: §4.1 · p. 3
4.2 Language models
- At Ts = 0.7, LSTM validation-loss degradation has slope 8.7 · 10^-3 per generation, versus 9.6 · 10^-5 at Ts = 1; dreaming is flat at both temperatures.The dreaming model also converges to a lower validation loss from generation 0.Source: §4.2 · p. 3
4.1 Markov chains
- Across 5000 generations, vanilla Markov-chain eigenvalues rise toward the unit circle, while dreaming keeps them approximately constant but below the true spectrum.The dreaming spectrum indicates faster mixing than the target; the offset depends on Td and α.Source: §4.1 · p. 3, Figure 1 · p. 4
4.2 Language models
- At generation 15, GPT-2 validation perplexity is roughly 1960 for vanilla training versus roughly 980 for dreaming (Td = 1.1, α = 0.5).The two runs separate from about generation 5 onward, with dreaming slowing rather than arresting drift.Source: Figure 2 · p. 4, §4.2 · p. 4
5 Limitations
The experiments leave limits on accuracy, temperature coverage, operator recoverability, and scaling cost.
5 Limitations
- A stable dreaming spectrum can still be biased: the learned chain mixes faster than the target.The size of the offset depends on Td and α.Source: §5 · p. 4
- Temperature dependence is observed only at Ts ∈ {0.7, 0.8, 1.0}; behavior near Ts = 1 is suggested by the sharpening picture, not established by a fine-grained sweep.Source: §5 · p. 4
- The Section 2 argument assumes exact operator recovery, which is not necessarily available in practice.Source: §5 · p. 4
- Td and α require empirical tuning, and at GPT-2 scale tuning is the main scaling obstacle while dreaming roughly doubles per-step generation cost.Useful ranges are Td ∈ [1, 3] and α ∈ [0.3, 1], though optimal pairs vary by setting.Source: §5 · p. 4
6 Conclusion
The conclusion presents the learned transition spectrum as a structural view of model collapse and reports that Dreaming Learning stabilizes it under self-training at Ts < 1. It also identifies faster mixing than the target as a residual bias and leaves its reduction and prediction open.
6 Conclusion
- The learned transition spectrum reveals a view of model collapse that scalar loss does not provide.Source: §6 · p. 5
- Under self-training at Ts < 1, subdominant eigenvalues drift toward the unit circle and the spectral gap narrows; Dreaming Learning leaves the spectrum flat.The proposed mechanism is that samples drawn at Td > 1 keep operator entries away from zero.Source: §6 · p. 5
- Dreaming leaves a faster-mixing bias, and whether annealing Td can reduce it or the floor can predict the optimal (Td, α) remains open.Source: §6 · p. 5
A Markov model implementation
The Markov experiments use a controlled 10-state chain to assess dreaming across temperature and loss-weight configurations. Figures 3 and 4 report stationary-distribution divergence after 5000 generations.
A Markov model implementation
- The baseline is a 10-state Markov chain with normalized stationary entropy 0.8, and results are unchanged for other entropies and state counts.Each generation adds 5000 symbols sampled at Ts = 0.7; the dreaming batch also contains 5000 symbols, and results cover 10 seeds.Source: A Markov model implementation · p. 6
- Figure 3 compares stationary-distribution KL divergence after 5000 generations while varying Td at α = 0.8 or α at Td = 1.5.Shaded regions span the 25th to 75th percentile across 10 seeds; VANILLA denotes no dreaming term.Source: Figure 3 · p. 6
- Figure 4 reports stationary-distribution KL divergence after 5000 generations across the (Td, α) grid.The reconstruction is a two-hidden-layer feed-forward network; α is a loss coefficient and is not restricted to [0, 1].Source: Figure 4 · p. 6, A Markov model implementation · p. 6
B Language model implementation
The language-model experiments use public-domain literary texts and evaluate WikiText-2 validation perplexity across a limited set of dreaming configurations. The GPT-2 sweep was narrowed because exhaustive evaluation was costly.
B Language model implementation
- The LSTM corpus uses three public-domain novels for training and three for validation, with 43,138 word types and 597,035 training tokens.The training texts are by Christie and Trollope; validation texts are by Hillenbrand, Collins, and Grey.Source: B Language model implementation · p. 7
- Figure 6 plots WikiText-2 validation perplexity at the best primary-validation epoch for each tried (Td, α) configuration.Only vanilla and Td = 1.1, α = 0.5 are averaged over seeds; remaining curves are single runs, and some end at generation 10.Source: Figure 6 · p. 7
- The two-line GPT-2 sweep required approximately 36 GPU-hours, versus 88 trajectories for the exhaustive grid, and was sufficient to find a configuration better than vanilla.The sweep varied Td at fixed α = 0.5 and α at fixed Td values.Source: Meta-parameter selection for GPT-2 · pp. 7–8