What a Cross-Model Fixed-Point Census Can and Cannot Arbitrate About Repetition

1.1 Two accounts, and why they have not been arbitrated

• One account attributes degeneration to repeated training text, including Fu et al. [2021]’s high-inflow explanation and Li et al. [2023]’s training intervention; another locates it in copying circuits.Hernandez et al. [2022] and Aoyama and Wilcox [2026] connect repeated data to induction-head effects.Source: §1.1 · pp. 1–2
• The missing evidence is a broad cohort of independently pretrained models measured on one readout, not a lack of connection between the accounts.The causal work trains its own few models under a single recipe; Li et al. [2023] also identify architecture and size as unevaluated factors.Source: §1.1 · p. 2

1.2 What this paper is, stated before the results

• The cross-model measurement is observational and cannot refute Li et al. [2023]’s controlled training-data intervention.Its narrower claim is that corpus and scale alone do not determine the outcome on a task-free structural readout.Source: §1.2 · p. 2
• Eight of seventeen models are funnels across seven families and five corpora, but only one available family funnels within The Pile cohort.That subset cannot establish that the cross-family split is corpus-independent.Source: §1.2 · p. 2

2 Setup

• From 96 random two-token starts, deterministic argmax iteration yields ϕ, the proportion terminating at a token that maps to itself.The map was established in earlier work [Vera Zúñiga, 2026b]; this paper adds census scale, a class taxonomy and its stability, and corpus-versus-weights attribution.Source: §2 · p. 2
• The readout is inherently short-window: widening the state to the last W tokens makes raw ϕ fall to 0.000 on four of six models by W = 16.The authors caution that ϕ should not be taken as a general property of the model.Source: §2 · pp. 2–3
• Every census is unprompted, and nine conditioning tokens can move ϕ across most of its range and change the four-way class.An instruction-tuning intervention worth 60.5 IFEval points moves the class by zero; the reported labels apply under the raw-domain condition.Source: §2 · p. 3
• The four classes are funnel, none, fragmented and borderline; modal share measures destination concentration independently of whether an endpoint is a fixed point.Class is a pre-specified deterministic function of ϕ and modal share; funnel and none are separated by an unoccupied band.Source: §2 · p. 3
• The class is stable across census seeds in 17 of 17 models, while the modal endpoint is stable in 15 of 17 and unstable cases are excluded from comparisons.Table 1 covers two size-matched The Pile tiers; both seeds agree on the class for all seven models.Source: §2 · p. 3, Table 1 · p. 4
• The readout samples and generates no text, making it cheap to run across models but distinct from rep-n, to which its results do not automatically transfer.It measures a property of the model’s conditional rather than a generation procedure.Source: §2 · p. 3

3 Related work: an adjacent measurement, and what separates it

• Du and Tanaka-Ishii [2025] measure correlation dimension over continuous log-probability trajectories seeded from real text; this paper measures starts reaching self-mapping tokens in a discrete argmax map.Neither measurement is a special case of the other.Source: §3 · p. 4
• Same-corpus, size-matched architecture comparisons are established; this paper contributes a different readout that provides a counterexample to prior reports of cross-architecture similarity.The cited comparisons include Michaelov et al. [2024] and Wang et al. [2025].Source: §3 · p. 4

4 E3: at fixed corpus and fixed scale, the class is not determined

• Both families hold their class across scale on The Pile: Pythia spans 14× and GPT-Neo 22×, with no model class-unstable across seeds.The split therefore persists beyond the initial matched checkpoints.Source: Table 2 · p. 5, §4 · p. 5
• At matched scale on The Pile, Pythia is a funnel while RWKV, Mamba, and GPT-Neo are not; the class is not determined by corpus and scale.The decisive comparison is gpt-neo-125m versus pythia-160m, 1.28× apart, with ϕ = 0.052 against 0.432.Source: §4 · pp. 4–5
• The comparison establishes non-determination only in the raw, unprompted domain; whether the same models separate under a prefix remains untested.Nine tokens of conditioning can change the class in the companion measurement.Source: §4 · p. 5
• Six of seven models reach the newline, yet GPT-Neo can concentrate on it more strongly than Pythia while having ϕ of 0.000 or 0.036 versus Pythia’s above 0.36.At matched size, RWKV and Pythia also share the newline endpoint at modal shares 0.474 and 0.432, with ϕ of 0.000 and 0.432.Source: §4 · p. 5
• For every Pythia, ϕ and modal share agree within four of 96 starts, while for every GPT-Neo modal share exceeds ϕ by forty to fifty-four starts.The proposed factorisation treats destination concentration as comparable and self-continuation as near 1 for one family and near 0 for the other.Source: §4 · pp. 5–6

5 E1: an off-the-shelf corpus manipulation that does not change the class

• Both Pythia models remain funnel: ϕ = 0.458 for pythia-410m and 0.427 for pythia-410m-deduped, with the same modal endpoint token.This is the nearest available data-side manipulation without training new models.Source: §5 · p. 6
• The pair differs in more than deduplication: the approximately 207B-token deduplicated Pile was trained for approximately 300B tokens, or about 1.45 epochs.The extra exposure reintroduces repetition, so the authors do not treat this as a clean corpus manipulation.Source: §5 · p. 6
• Because repeated-data damage can be strongly non-monotonic, this null is consistent with both accounts if the Pile’s duplication misses the relevant frequency range.The authors report the comparison but do not claim it discriminates between the accounts.Source: §5 · p. 6

• Without frequency control, 12 of 13 endpoints are at or above the 90th inflow percentile, with median 99.87.The authors describe this apparent high-inflow pattern as an artefact of frequency.Source: §6 · p. 6
• Against the 50 nearest tokens in log-frequency, endpoints beat matched peers in 1 of 13 models, with a median matched percentile of 32.0 against 50 by construction.The comparison tests whether endpoints have unusually high inflow beyond what their frequency would predict.Source: §6 · pp. 6–7
• In matched-size own-language checks, endpoint measurements were 6.0 for Spanish, 16.0 for Korean, and 96.0 for Japanese; two endpoints were far below the null.Those two endpoints occur zero times in an English corpus and were therefore unmeasurable there.Source: §6 · p. 7
• The frequency-controlled result is a consistency check, not a refutation: endpoints are common tokens rather than high-inflow ones, as the data-side account predicts.Li et al. [2023] found that merging repetitive high-inflow pairs matched the full high-inflow method, while random high-inflow pairs of equal size did not.Source: §6 · p. 7

• An observational contrast across pretrained models cannot refute Li et al.’s [2023] controlled training-data intervention.The paper measures pretrained models and compares them; Li et al. manipulated training data and measured the consequence.Source: §7 · p. 7
• Within The Pile, the only corpus where training data can be held fixed across families, only Pythia funnels among the available families.A second funnel family in that subset would be needed to show the split is corpus-independent; the public candidate exceeded available memory.Source: §7 · p. 7
• Holding corpus and parameter count fixed does not control training schedule, optimiser, data order, tokenizer, or other differences between independently trained families.The observed difference is not explained by corpus or scale, but the paper does not identify what does explain it.Source: §7 · p. 7
• The readout measures a deterministic two-token map, not repetition in generated text; models without census fixed points may still degenerate under sampling.The relationship between the fixed-point fraction and rep-n on generated text is unestablished, and sampling degeneration was not checked.Source: §7 · p. 7
• No p-values or class–scale rank correlations are reported, and models with unstable modal endpoints or corpus-absent endpoints are excluded rather than assigned values.The cohort has 17 models, and the exhibits rest on two families.Source: §7 · pp. 7–8

• At fixed corpus and parameter count, fixed-point structure is not fixed; the deduplicated Pythia suite does not change the class, though that pair includes a second pass over much of its data.Across two size-matched tiers, two families retain opposite classes across an order of magnitude of scale.Source: §8 · p. 8
• The corpus-side inflow term does not select endpoints once frequency is controlled, across English and three other languages.The paper presents this as observational evidence, not a refutation of a training intervention.Source: §8 · p. 8
• Six of seven models in one ladder reach newline, yet models that concentrate on it most strongly are among those that never stay there.Across families, scales, and corpora, the reported decomposition is agreement about arriving trajectories but disagreement about whether the destination maps to itself.Source: §8 · p. 8