A Fragility Spectrum for Recursive Language-Model Training

Yangze Liu; Zhongyi Han

Source-linked AI summary

A Fragility Spectrum for Recursive Language-Model Training

Yangze Liu, Zhongyi Han

arXiv:2609.11149v1Sep 10, 2026cs.CLcs.AIcs.LG

TL;DR

This paper examines why recursive training on model-generated text affects checkpoints differently. Across a shared five-generation ecosystem, checkpoints show a stable fragility spectrum, which can be screened cheaply with self-loops and moderated by controlling the sampling tail.

Problem
Prior work establishes diversity collapse from repeated training on model-generated data, but has largely examined protocols and mixtures rather than how different checkpoints vary under the same process.
Method
The study places 13 publicly released checkpoints in a shared-corpus recursive ecosystem for five generations, then tests self-loop screening and interventions on generation and data filtering.
Results
Five-generation outcomes differ by roughly five-fold across checkpoints, with stable ordering across ecosystem changes and seeds; scale and static indicators do not explain fragility, while self-loops and sampling-tail control are informative.
Takeaways & Limitations
Collapse fragility is a stable checkpoint property that should be reported alongside recursive-training protocols and can be screened before selecting models for further study.
Takeaways & Limitations
The conclusions are bounded by one reset-from-base protocol family, 13 checkpoints at 1–4B parameters, small pools, and untuned common training and decoding settings.

Abstract

from arXiv · show

Model-generated text is finding its way back into training corpora, and there is plenty of evidence that training on such data over and over collapses output diversity. Prior work has studied the phenomenon itself: which protocols and which data mixtures cause collapse. But different models behave very differently under the same process. We fix one recursive contamination protocol and let 13 publicly released checkpoints form an ecosystem that shares a common corpus for five generations. The unique 4-gram outcome after five generations ranges from 0.187 to 0.940 across checkpoints, a roughly five-fold spread: some models are barely touched, others degenerate into repetitive fragments. Changing the composition of the shared pool or mixing in human text keeps the Spearman correlation of the ordering at 0.91--0.97, and changing the random seed keeps it at 0.93--0.98. Whether a model collapses easily under recursive training is, then, a property of the checkpoint itself, and one that has gone largely unexamined. Parameter scale alone does not explain it, since a three-size ladder within one family is not monotonic in size, and none of the static indicators we tested predicts it either. What does work is cheap: let a model iterate on its own output for two or three generations, and its fragility in the larger ecosystem can be inferred from that alone. Collapse speed also responds to intervention. Tightening top-p, which cuts the low-probability tail at generation time, nearly stops collapse within three generations and stabilizes six checkpoints spanning the whole spectrum together, while data-side filtering slows collapse without stopping it.

1 INTRODUCTION

Recursive training on synthetic text narrows output diversity, but checkpoints differ sharply in how quickly they collapse under the same contamination process. The resulting fragility ordering is stable across ecosystem changes, while scale and static indicators do not explain it.

1 INTRODUCTION: The shared recursive protocol compares checkpoints under identical generation, pooling, and training conditions, isolating differences in their starting checkpoints.Each model contributes to and trains on a common pool across five generations.
1 INTRODUCTION: Five-generation diversity outcomes span roughly a five-fold range across checkpoints, and the collapse ordering remains stable when ecosystem composition, human-text mixing, or random seeds change.This motivates treating collapse fragility as a checkpoint property rather than an artifact of one ecosystem configuration.
1 INTRODUCTION: A same-family size ladder is non-monotonic in collapse speed, and calibration, initial-output statistics, and one fine-tune signal fail to predict fragility before recursion.The tested pre-recursion indicators did not separate robust from fragile checkpoints.
1 INTRODUCTION: A two- or three-generation self-loop can cheaply infer a checkpoint’s relative fragility, while tightening the generation-time low-probability token tail nearly stops collapse within three generations.Data-side filtering slows collapse but does not stop it.

2 RELATED WORK

Prior work established recursive synthetic-data collapse, studied ecosystem interactions and scale, and explored early warnings and mitigation. This paper’s context spans protocol effects, model-network composition, scale regimes, and interventions such as real-data mixing and filtering.

2 RELATED WORK: Repeated training on model-generated outputs removes distributional tails first and reduces diversity or quality, with outcomes depending on how synthetic data enters later generations.Replacing old data with synthetic data invites collapse, whereas retaining or accumulating real data can alter the outcome.
2 RELATED WORK: Multi-model studies examine how shared-data ecosystems converge as a function of interaction structure, model diversity, and data sources.These studies treat models as nodes and the ecosystem configuration as the object of analysis.
2 RELATED WORK: Prior scale results are regime-dependent: larger models can be more robust in some settings, while theory allows scale effects to reverse across statistical regimes.The paper therefore tests scale with a same-family, same-recipe ladder rather than claiming a general scaling law.
2 RELATED WORK: Calibration shifts, real-data mixing, data accumulation, and sample filtering have been proposed as warnings or mitigations, but biased filtering can itself accelerate tail loss.These approaches motivate the paper’s evaluation of early signals and data-side interventions.

3 SETUP

The study runs a reset-from-base recursive protocol in which 13 checkpoints repeatedly generate text for and train on a shared pool. It measures collapse primarily through unique 4-gram diversity across ecosystem and self-loop chains.

3 SETUP: Each generation has 13 models generate text independently, mix outputs into a shared pool, and fine-tune from each model’s clean pretrained weights through generation five.The protocol carries forward only the text distribution, not model weights.
3 SETUP: The experiment compares 13 publicly released 1–4B base checkpoints from 10 families, where each checkpoint bundles architecture, tokenizer, pretraining history, and learned parameters.The experimental unit is the released checkpoint, not one isolated model ingredient.
3 SETUP: Ecosystem chains share a pool across 13 models, while alternative environments vary dominant members, model shares, and human-text doses to test composition sensitivity.Self-loop chains provide a complementary single-model recursive system for predicting ecosystem outcomes.
3 SETUP: The primary metric is u4, the fraction of unique 4-grams in equal-sized outputs, with lower values indicating more concentrated vocabulary and phrasing.Encoder distances, perplexity, and word-frequency tail statistics are auxiliary readings.
3 SETUP: Figure 1 displays five-generation u4 trajectories for all 13 checkpoints and verbatim generation-five excerpts from the spectrum’s two ends.The top panel shows trajectories and endpoints; the bottom panel shows representative outputs.

4 RESULTS

Under identical recursive contamination, checkpoints occupy a stable fragility spectrum: five-generation diversity varies widely, and the ordering persists across ecosystem variants and seeds. Fragility is not explained by parameter scale or static pre-recursion readings, but short self-loops reveal it and sampling-tail interventions strongly alter collapse speed.

4.1 A STABLE FRAGILITY SPECTRUM UNDER A FIXED PROTOCOL: 0.187 to 0.940 gen-5 u4 across 13 checkpoints shows a roughly five-fold diversity spread under the same recursive protocol.The median is 0.509, and the fragile and robust ends are not organized by model family.
4.1 A STABLE FRAGILITY SPECTRUM UNDER A FIXED PROTOCOL: Spearman correlations of 0.91–0.97 across alternative ecosystems and 0.929–0.984 across seeds preserve the checkpoint ordering.The alternatives vary dominant-member identity, mixing shares, and human-text content.
4.2 NEITHER PARAMETER SCALE NOR PRE-RECURSION INDICATORS PREDICT THE SPECTRUM: 0.390 mean g3 u4 for Qwen3 1.7B exceeds 0.135 for 0.6B and 0.124 for 4B, so collapse is non-monotonic in scale.The middle size is the most stable in the same-family, same-recipe ladder.
4.2 NEITHER PARAMETER SCALE NOR PRE-RECURSION INDICATORS PREDICT THE SPECTRUM: 0.192 correlation for static calibration error and below 0.4 for other static readings show that pre-recursion indicators failed to predict the five-generation outcome.The strongest early calibration-shift signal was −0.643 but fell to −0.482 after removing two extreme models.
4.3 A TWO-TO-THREE-GENERATION SELF-LOOP TRACKS THE ORDERING: 0.676 self-loop-to-ecosystem ordering correlation at g3 shows that two to three generations provide a cheap fragility screen.The correlation rises to 0.797 at five generations, but early screening can miss late starters whose collapse begins after the window.
4.4 THE SAMPLING TAIL SETS THE SPEED, DATA FILTERING ONLY SLOWS: 0.978 g3 u4 with top-p 0.8 versus 0.111 at top-p 0.95 and 0.009 at top-p 1.0 shows that truncating the low-probability tail nearly stops collapse.Across six checkpoints, protective sampling lifts all g3 u4 values to at least 0.675, whereas data filtering only postpones collapse.
4.4 THE SAMPLING TAIL SETS THE SPEED, DATA FILTERING ONLY SLOWS: All 800 M06 generation-3 samples contain the same four-word string, whereas M02 declines from 0.93 to 0.77 to 0.18 across generations one to three.These staged outputs illustrate repetitive collapse and a delayed-collapse trajectory.

5 DISCUSSION AND CONCLUSION

Collapse fragility is a stable checkpoint property that remains sensitive to the starting model, while its origin is unresolved and conclusions are bounded by scale and protocol limitations.

Changing only the checkpoint moved five-generation outcomes by a factor of five, motivating checkpoint-specific reporting and additional runs.The paper recommends stating the exact released checkpoint and, where resources allow, attaching a second checkpoint under the same recipe.
A self-loop can rank candidate checkpoints before recursive training, with two generations providing an initial ordering and a third sharpening it.
The producer-side sampling tail affects collapse speed, but the receiver-side source of checkpoint fragility remains unresolved across architecture, tokenizer, data, and parameters.The design cannot separate these possible sources.
The conclusions are limited to 13 checkpoints at 1–4B and one reset-from-base protocol using small pools, shared recipes, and nominal decoding settings.Frontier scale, larger pools, weight inheritance, other pooling rules, and per-checkpoint tuned recipes require separate experiments.
Overall, fragility is dormant before recursion, surfaces through a two-to-three-generation self-loop, and can be steered by controlling the sampling tail.

A EXPERIMENTAL SETUP DETAILS

The main experiment compares 13 publicly released base checkpoints, excluding two technical failures before cross-model measurement and spanning ten model families.

The 13 base checkpoints are ordered from fragile to robust using the ecosystem’s five-generation endpoint, with median u4 equal to 0.509 for M05.
Two candidates were excluded before comparison because one produced empty outputs after fine-tuning and another crashed the generation engine deterministically.The retained checkpoints come from 10 model families, whose same-family pairs do not occupy consistent positions.

A.2 Generation and training parameters.

All runs use the same decoding settings, shared-pool budget, and fixed 800-sample diversity evaluation, with self-loops and ladders generating their own pools.

Generation uses temperature 1.0, top-p 0.95, repetition penalty 1.15, frequency penalty 0.3, and at most 128 new tokens per sample.
Each generation uses a 2,100-text pool, while diversity is measured on a fixed 800-sample slice.In the ecosystem, each member contributes 800 samples and preset shares determine the pool; self-loop and ladder models fill their pools independently.

A.3 Experiment matrix.

The appendix evaluates ecosystems, self-loops, scale ladders, sampling interventions, filtering rules, and static probes across specified checkpoints, seeds, and generations.

The experiment matrix spans six ecosystems, self-loops for all 13 models, a three-size Qwen ladder, six-checkpoint sampling tests, and equal-budget filtering interventions.
Static probes include calibration, output-stream statistics, reference-model perplexity, word-frequency tails, human-centroid distance, and one human-text fine-tune change.Calibration uses ten equal-width bins and CommonsenseQA, with ARC-Easy and OpenBookQA as robustness checks.
Filtering keeps half of each pool at equal budget and compares near-duplicate selection, inverted selection, random retention, and a perplexity gate.

B THE 7–8B SUPPLEMENT

Across four 7–8B base models, self-loop recursion again separates checkpoints into fast- and slow-collapsing groups, with slower collapse than in the 1–4B models.

B THE 7–8B SUPPLEMENT: Four 7–8B base models again split into fast- and slow-collapsing ends under the g0–g3 self-loop.OLMo-3-1025-7B was excluded before measurement because the generation engine did not recognize its architecture.
B THE 7–8B SUPPLEMENT: The slow-collapsing side shows the same late-onset pattern observed among smaller models.
B THE 7–8B SUPPLEMENT: The 7–8B models collapse more slowly than the 1–4B models on the reported rates.

C THE FIRST-GENERATION LENGTH SIGNAL (EXPLORATORY)

Static starting length provides no useful signal, whereas first-generation length non-expansion strongly rank-correlates with later ecosystem collapse.

C THE FIRST-GENERATION LENGTH SIGNAL (EXPLORATORY): +0.780 Spearman correlation links first-generation length non-expansion with ecosystem g5 u4, with p = 0.002.After partialing out g0 u4 and g0 length, the correlation rises to +0.820.
C THE FIRST-GENERATION LENGTH SIGNAL (EXPLORATORY): −0.121 correlation between g0 mean output length and the ecosystem endpoint indicates no static starting-length signal.
C THE FIRST-GENERATION LENGTH SIGNAL (EXPLORATORY): u4 is the word-level fraction of distinct 4-grams, while perplexity uses GPT-2-large and Self-BLEU uses fixed 4-gram subsamples.The u4 denominator is equalized at 20,000 4-grams per model, and Self-BLEU uses 80 texts from each 800-sample slice.

E SAMPLING GRID × SIX CHECKPOINTS, FULL TABLE

Across six checkpoints, sampling conditions preserve the fragile-to-robust ordering when collapse occurs, while protective settings compress the spectrum and raise diversity.

E SAMPLING GRID × SIX CHECKPOINTS, FULL TABLE: 0.558, 0.453, and 0.145 separate the robust and fragile halves under full distribution, top-p 1.0, and temperature 1.3, respectively.The halves never trade places, and top-p 1.0 versus full distribution has Spearman correlation 0.943.
E SAMPLING GRID × SIX CHECKPOINTS, FULL TABLE: Protective sampling conditions raise every g3 reading to 0.675–0.986 and compress the fragility spectrum.
E SAMPLING GRID × SIX CHECKPOINTS, FULL TABLE: Temperature 1.3 starts at 0.452–0.779, unlike the other four conditions whose 24 chains start at 0.900–0.993.The temperature-1.3 g3 values should therefore be read together with their lower starting points.
E SAMPLING GRID × SIX CHECKPOINTS, FULL TABLE: Same-protocol reruns scatter by about 0.011, while absolute values should be compared only within this grid.

F GRID CHECKPOINTS RE-EVALUATED UNDER ONE DECODING

Re-evaluating grid checkpoints with a shared decoder shows that damage from collapsing conditions persists, whereas protective conditions largely remain near their base anchors.

F GRID CHECKPOINTS RE-EVALUATED UNDER ONE DECODING: 0.665–0.979 below base anchors, the fragile four remain damaged under the shared decoder after three collapsing conditions.
F GRID CHECKPOINTS RE-EVALUATED UNDER ONE DECODING: Five of six checkpoints from each protective condition land within 0.13 of their base anchors under the shared decoder.M10 is the exception, at 0.234 and 0.235 below its base anchor.
F GRID CHECKPOINTS RE-EVALUATED UNDER ONE DECODING: Equal-gram accounting gives the same pattern: every other protective-condition gap is below 0.09, while M10’s gaps are 0.205 and 0.166.
F GRID CHECKPOINTS RE-EVALUATED UNDER ONE DECODING: The M06 ladder falls from 0.9811 to 0.0432 across four stages, while M02 falls from 0.9256 to 0.1824.The corresponding 4-gram count for M02 is zero at every stage.