PaperlayerView exact revision on arXiv

Source-linked AI summary

Source Identification Is Not Fitness Testing: Measuring the Limits of Synthetic-Data Attribution

Joss Armstrong

arXiv:2610.00417v1cs.LG

TL;DR

Repeated training on generated data can degrade later models, raising the question of whether provenance helps select examples for reuse. The paper tests source identification after rewriting and compares provenance-based selection with a reference-model score over three recursive rounds; attribution weakens under rewriting, while the planned comparison finds no stable difference in downstream degradation.

  • Problem

    Prior work reports degradation from recursive training on generated data, but it remains unclear what information a curation rule must preserve for downstream use as training changes the learner.

    Source: §I · p. 1, F. The next question · p. 5

  • Method

    The study measures generated-text source identification before and after rewriting, then compares provenance-based and reference-model-perplexity selection across three rounds of generation and retraining.

    Source: §I · p. 1

  • Results

    Generator attribution weakened substantially after rewriting, while the planned comparison of two selection rules detected no stable difference in downstream degradation.

    Source: §XI · p. 6

  • Takeaways & Limitations

    Source identity and training usefulness are separate questions, and neither tested selection signal is established as sufficient for future recursive behavior.

    Source: §XI · p. 6

    Five lineages per primary rule provide limited power for small differences; the simulation reached about 80% power only near a 1.48-point slope difference.

    Source: §X · p. 6

Abstract

from arXiv · show

Repeated training on model-generated data can degrade later models. One possible response is to use provenance when deciding which generated examples to reuse. We test both how reliably that provenance can be recovered and whether it helps identify better training data. Using financial-risk text, we first identify the source of generated passages and then repeat the test after rewriting them. Generator attribution is 98.7% accurate on the original passages but falls to 53.1% after paraphrasing and 29.0% after style rewriting. Generated-versus-human detection remains close to perfect against the tested human comparison set. We then compare two ways of selecting generated examples over three rounds of generation and retraining. One uses source information. The other uses a score from a separate reference model. The two rules select different examples, but the planned comparison does not detect a stable difference in the degradation of the resulting models. The results show that identifying where data came from and identifying which data are useful for training are separate problems. The experiment therefore separates source identity, criterion-facing selection, and recursive training outcome: neither the provenance score nor the tested criterion-facing proxy is established as sufficient for future recursive behaviour.

I. INTRODUCTION

Repeated training on generated data can carry distortions forward, motivating tests of whether provenance can guide example reuse. The paper separates source identification from selection for training and downstream recursive outcomes, finding that different selection rules do not yield a stable detected difference in degradation.

I. INTRODUCTION

  • The paper tests generated-versus-human detection, known-generator attribution, rewriting robustness, and whether source information helps select data for recursive training.The setting uses financial-risk text and compares source-based selection with a reference-model scoring rule.Source: §I · p. 1
  • The paper distinguishes source identification, selection criteria, and recursive outcomes because source accuracy alone does not establish which examples will be useful for later training.It compares a source-based rule with a separate reference-model score over three rounds of generation and retraining.Source: §I · p. 1
  • The two selection rules choose different examples, but the planned comparison detects no stable difference in downstream degradation; this does not establish equivalence or provenance superiority.The paper frames this as a boundary on what provenance or a criterion-facing proxy has been shown to establish.Source: §I · pp. 1–2

A. Generated-text detection and source identification

  • Prior work reports strong controlled-condition source attribution,, while broader benchmarks document sensitivity to domains, decoding choices, and rewriting,.These findings motivate measuring generated-versus-human detection, known-source attribution, and unseen-source recognition separately.Source: A. Generated-text detection and source… · p. 2

B. Recursive training with generated data

  • Prior work shows recursive degradation can depend on the mixture of real and generated data and on training procedure –, but does not establish provenance as the right basis for retaining examples.Research on self-correcting loops also considers preserved real data or correction functions.Source: B. Recursive training with generated… · p. 2

C. Rejecting uncertain source assignments

The paper separates source questions, selection scores, and downstream outcomes, while describing the corpora and classifiers used to study source detection. It also uses conformal prediction to distinguish abstentions from wrong singleton source assignments.

C. Rejecting uncertain source assignments

  • Split conformal prediction creates source-label sets; empty or multi-label sets count as abstentions, while a singleton is an assignment.The study reports coverage, singleton rate, and wrong singleton rate; its finite-sample coverage guarantee is tied to exchangeability, not distribution shift.Source: C. Rejecting uncertain source… · p. 2

D. Three questions kept separate

  • The paper separates where text came from, the score used to retain generated examples, and the outcome after training on selected data.The recursive study evaluates descendants on a separate frozen set of human-written financial-risk paragraphs.Source: D. Three questions kept separate · p. 2

A. Corpus and source-model sets

  • The corpus includes generated financial-risk text from ten models, while the later robustness study uses seven models and is a separate classification task.The seven-source set has 16,384 records per source.Source: A. Corpus and source-model sets · p. 2

B. Human comparison corpus

  • The human comparison uses pre-2020 10-K risk paragraphs that match the financial-risk register but not the exact generation task.For detector training, a cap of 20 paragraphs per ticker retained 879 paragraphs.Source: B. Human comparison corpus · p. 2

C. Source-identification models

  • The source-identification tasks use separate classifiers, including word- and character-TF-IDF models and a fine-tuned DistilRoBERTa classifier.The linear word model uses unigram and bigram TF-IDF features with sublinear scaling and logistic regression.Source: C. Source-identification models · p. 2

D. Rewriting conditions

The study tests source recognition after four rewrites, unseen-source recognition, and conformal abstention, then compares fixed selection rules through three recursive training transitions. The rewriting threat model and recursive setup constrain what the experiments can establish.

D. Rewriting conditions

  • The four rewrites are roundtrip translation, cross-family paraphrasing, human-register style transfer, and generic evasion intended to hinder stylometric detection.The grey-box rewriting model knows stylometric source identification is used but has no detector parameters, training data, or query access.Source: D. Rewriting conditions · p. 2
  • The held-out rewrite tests cover transformations from the same broad rewriting setup, not an entirely new class of transformation.For each test condition, the rewriting model is excluded from detector training.Source: D. Rewriting conditions · pp. 2–3

E. Sources absent from detector training

  • Unseen-source recognition is tested by withholding one of seven reference models or by testing five additional generators kept outside detector development.The two settings train on six versus seven known generators, so their AUROCs are not directly comparable.Source: E. Sources absent from detector training · p. 3

F. Abstention under uncertainty

  • The split-conformal predictor returns an eight-way set at α = 0.05; empty or multi-element sets count as abstentions, and singleton sets as assignments.Coverage, singleton rate, and wrong singleton rate are reported alongside ordinary source-identification accuracy.Source: F. Abstention under uncertainty · p. 3

G. Recursive training experiment

  • Each recursive run begins from Qwen2.5-3B-Instruct and makes three generate-select-train transitions; every candidate is generated by the current model.The human corpus is used only for downstream evaluation, and each rule retains 80 of 160 generated items.Source: G. Recursive training experiment · p. 3
  • The provenance rule retains the 80 items least confidently tied to one known source signature, while the perplexity rule retains 80 items with lowest score under a frozen reference model.The reference-model score is not the downstream outcome used to evaluate retrained descendants; the provenance score is applied to a related but unseen Qwen model.Source: G. Recursive training experiment · p. 3
  • The outcome is corpus perplexity on 200 pre-2020 human financial-risk paragraphs, with five seeds for provenance and perplexity rules.The source classifier was fixed and did not include Qwen2.5-3B-Instruct among its seven training sources.Source: G. Recursive training experiment · p. 3
  • The primary comparison uses lineage-level held-out perplexity slopes and an exact two-sided permutation test; per-generation comparisons are secondary.The study uses an inverted-test 95% interval, a Welch-interval robustness check, and a simulation of detectable slope differences.Source: G. Recursive training experiment · p. 3

IV. SOURCE IDENTIFICATION BEFORE AND AFTER REWRITING

Source classifiers perform strongly before rewriting, but exact generator identification weakens sharply under some rewrites while generated-versus-human detection remains strong against the tested human corpus. The human comparison is specific to that corpus, and style-transfer attribution loss also appears with a transformer classifier.

IV. SOURCE IDENTIFICATION BEFORE AND AFTER REWRITING

  • In the initial ten-generator experiment, ten-way attribution accuracy was 98.15%, while the later seven-generator rewriting study used a separate word-TFIDF classifier.The initial experiment also had mean pairwise AUROC 0.9995, with a minimum of 0.9811.Source: §IV · p. 3
  • 98.7% generator-identification accuracy fell to 96.7% after roundtrip translation, 53.1% after paraphrasing, 29.0% after style transfer, and 46.9% after generic evasion.These are mean accuracies across seven source models for the word-TFIDF classifier.Source: §IV · p. 4
  • DistilRoBERTa generator-identification accuracy fell from 96.0% before rewriting to 9.4% after style transfer, so the loss was not specific to word-TFIDF.Source: §IV · p. 4
  • Generated-versus-human AUROC remained above 0.9999 under every rewrite against the tested human corpus.The source passage cautions that this baseline matches the financial-risk register but not the exact generation task.Source: §IV · p. 4

V. TRAINING ON KNOWN REWRITES AND REJECTING UNCERTAIN CASES

Detector performance depends on the source-identification task and rewriting condition, while the recursive comparison tests whether different selection rules lead to different training outcomes. The reported results do not establish a stable recursive slope difference or a guarantee for arbitrary unseen sources or rewrites.

V. TRAINING ON KNOWN REWRITES AND REJECTING UNCERTAIN CASES

  • With tested rewrite methods represented in training, character-TFIDF accuracy was 99.3% to 99.8%; holding out one rewrite type gave 95.7% mean accuracy and 83.4% on style transfer.These are diagnostics within the tested rewrite family, not evidence of robustness to a new transformation class.Source: §V · p. 4
  • Under style transfer, the mean singleton rate was 0.5467 and the wrong singleton rate was 0.4083, reaching 0.8567 for the worst source model.Paraphrasing and generic evasion usually produced non-singleton prediction sets, with wrong singleton rates of 0.0027 and 0.0032.Source: §V · p. 4
  • Performance on tested rewriting methods does not establish performance on new forms, and rejection helps only when shifted cases are rejected rather than assigned one wrong source label.The unseen-source results likewise do not guarantee recognition of arbitrary future generators.Source: §V · p. 4, §VI · p. 4

VI. RECOGNISING SOURCES ABSENT FROM DETECTOR TRAINING

  • For five generators kept outside detector development, unseen-source AUROC ranged from 0.9714 to 0.9929, with a mean of 0.9822.The separate seven-generator holdout had mean AUROC 0.9261 and ranged from 0.7700 to 0.9923; the result depended substantially on the held-out source.Source: §VI · p. 4

VII. RECURSIVE TRAINING WITH DIFFERENT SELECTION RULES

  • At G1, the perplexity rule had lower held-out perplexity in all five shared-seed comparisons, but that direction did not persist in later generations.The recursive procedure retained 80 of 160 generated candidates per step and evaluated the resulting checkpoints on the same held-out human corpus.Source: §VII · p. 4
  • 7.543 versus 7.367 perplexity points per generation were the mean slopes under the perplexity and provenance rules; the +0.176 difference was not significant (p = 0.746).The planned comparison did not detect a stable separation in recursive degradation.Source: §VII · p. 4
  • The two rules selected different data: mean overlap was 36.0, 37.1, and 36.2 of 80 retained examples at G1, G2, and G3, respectively.Fewer than half of the selected examples were shared at each generation.Source: §VII · p. 4

VIII. WHAT THE RECURSIVE RESULT CAN AND CANNOT SHOW

The primary comparison used an exact two-sided permutation test of lineage slopes, and its interval leaves meaningful differences possible. The evidence therefore does not support treating the rules as equivalent.

A. Exact permutation test

  • The exact permutation test found a +0.176 perplexity-point-per-generation difference and p = 0.746 for perplexity minus provenance.The test compared ten observed primary-arm slopes in two groups of five and assumes exchangeability under the null.Source: A. Exact permutation test · p. 4

B. Interval for the slope difference

  • The 95% interval for the slope difference was [−0.794, +1.267] perplexity points per generation, allowing differences of roughly one point in either direction.The Welch interval was [−0.959, +1.311], also not supporting equivalence.Source: B. Interval for the slope difference · p. 5

C. What differences could this design detect?

The paper distinguishes source-identification tasks, robustness to tested rewrites, and downstream recursive training outcomes. The recursive rules selected substantially different data, but the planned comparison did not detect different mean slopes, with uncertainty that does not establish equivalence.

C. What differences could this design detect?

  • The design had little power for small slope differences: estimated power was 0.157 for 0.5 points and reached 0.799 only at 1.48 points per generation.This sensitivity analysis is not evidence in favour of the null.Source: C. What differences could this design… · p. 5

A. What source identification establishes

  • Generated-versus-human detection was close to perfect on the tested human comparison set, while exact generator attribution fell sharply after paraphrasing and style transfer.Success on one source-identification task does not establish success on the others.Source: A. What source identification… · p. 5

B. Robustness depends on what was tested

  • Detectors performed well on tested rewriting methods and transferred strongly within the small tested family, but behavior on a different rewrite class remains unknown.Under style transfer, many rewritten passages received one wrong source label rather than an abstention.Source: B. Robustness depends on what was tested · p. 5

C. Different selections did not yield a stable recursive separation

  • The provenance and perplexity rules selected substantially different training data, yet their planned lineage-slope comparison did not detect a stable difference.The perplexity rule was lower in all five shared-seed comparisons at G1, but that direction did not persist across G0 to G3.Source: C. Different selections did not yield… · p. 5

D. What the recursive result establishes

  • The recursive comparison’s uncertainty interval still allows differences that could matter, and neither confidence score was shown to contain all information needed to predict training effects.These conclusions apply to the two rules and experimental design tested, not to a complete measure of training value.Source: D. What the recursive result establishes · p. 5

E. Relevance to managed machine-learning systems

Provenance can support audit, traceability, and policy enforcement in managed machine-learning pipelines, but it does not by itself establish whether retaining a generated example improves a later model.

E. Relevance to managed machine-learning systems

  • Provenance can support audit, traceability, and policy enforcement, but this financial-risk experiment does not test network data or a deployed telecommunications system.It does not establish whether retaining or removing a particular generated example improves a later model.Source: E. Relevance to managed… · p. 5

F. The next question

The next problem is to determine what information a curation rule must preserve for a named downstream use, including after training changes the learner.

F. The next question

  • A curation rule must preserve information relevant to its declared downstream use, and whether that requirement survives learner changes remains unresolved.The study identifies this boundary but does not characterize the required recursive information interface.Source: F. The next question · p. 5
  • A criterion-facing signal has not been shown sufficient for future recursive behaviour.Replacing provenance with another fixed curation score alone does not resolve which information must be preserved.Source: F. The next question · p. 5
  • A separate theoretical treatment frames the issue as task-relative information contracts judged by what they preserve for a named downstream criterion.Source: F. The next question · p. 5

X. LIMITATIONS •

The evidence is bounded by one model scale, one domain, three recursive updates, a small lineage count, four rewrite types, a mismatched human comparison, and a detector roster that omitted the recursive source model.

X. LIMITATIONS •

  • The recursive experiment used only Qwen2.5-3B-Instruct, so its result may not transfer to larger or smaller models.Source: §X · p. 5
  • The study used financial-risk text, and its communications relevance concerns managed-training selection rather than performance on telecommunications data.Source: §X · p. 5
  • Three updates from G0 to G3 may not capture longer-horizon effects, while five primary-rule lineages limit power for small differences.The simulation reached about 80% power only near a 1.48-point slope difference.Source: §X · p. 6
  • The four tested transformations are all text rewrites, so holding one out does not represent the full range of future transformations.Source: §X · p. 6
  • The human comparison matches financial-risk register but not generation task, and the detector roster omitted Qwen2.5-3B-Instruct.Confidence on that recursive source model should not be read as a calibrated probability of true source membership.Source: §X · p. 6

XI. CONCLUSION

The conclusion separates source-identification tasks from recursive training outcomes: rewriting weakens exact generator attribution, while the planned comparison finds no stable difference between selection rules and does not establish equivalence.

XI. CONCLUSION

  • The slopes were 7.543 held-out-perplexity points per generation for the perplexity rule and 7.367 for provenance, a +0.176 difference; the test gave p = 0.746.The interval [−0.794, +1.267] does not establish equivalence.Source: Figure 2 · p. 7
  • Generated-versus-human detection, exact generator attribution, and unseen-source recognition are separate tasks, not one combined accuracy measure.Source: §XI · p. 7
  • The caption identifies Table II as comparing generator-attribution accuracy before and after rewriting.Source: Table II · p. 7
  • Generated-versus-human TF-IDF AUROC remained above 0.9999 under every rewrite against a register-matched, not task-matched, human comparison corpus.Source: §XI · p. 7
  • The caption identifies Table VI as reporting per-lineage held-out-perplexity slopes and the primary comparison.Source: Table VI · p. 8
Loading 2610.00417v1…