PaperlayerView exact revision on arXiv

Source-linked AI summary

How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Text

Jenna Russell, Ben Glickenhaus, Katherine Thai, John Wieting, Mohit Iyyer, Max Spero, Bradley Emi

arXiv:2609.40295v1cs.CLcs.LG

TL;DR

The paper asks how naturally occurring, unlabeled AI-generated web text affects pretraining, and tests that question across 800 models while fitting a scaling law. AI tokens help human-text loss only for data-starved models, and the proposed law predicts larger models with lower error than existing laws. The authors recommend prioritizing human text when the target is human text, while retaining AI text when the target is AI text.

  • Problem

    Wild AI text is increasingly mixed into web pretraining data, but its effect on pretraining had not been directly measured.

    Source: §1 · pp. 1–2

  • Method

    The authors pretrain 800 models with varying AI-to-human token ratios, evaluate held-out losses, and fit a scaling law with separate benefit and harm terms.

    Source: §1 · p. 2

  • Results

    0.83 × 10−3 paired error on C4, versus 1.41 × 10−3 for the best existing law, when predicting models 3.6× larger; AI tokens help human-text loss only at low human-data budgets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    For human-text targets, repeat human text before adding AI text and report validation losses on human and AI text separately; AI text remains valuable for AI-text targets.

    Source: §1 · p. 3, §6 · p. 11

    The law is fit on models up to 268M parameters and tested up to 973M; the authors note all models are too small to have meaningful performance.

    Source: §6 · p. 11

Abstract

from arXiv · show

Web text makes up the majority of pretraining data and is increasingly AI-generated. After applying FineWeb quality filtering, we find that 27.5% of tokens from June 2026 web data are labeled as AI-generated by Pangram, rising to 31.1% by August. Unlike synthetic data or model-collapse setups, this *wild* AI text comes from many models, is written for human readers, and arrives unlabeled in pretraining corpora. How does AI text in the wild affect language model pretraining? To answer this question, we pretrain 800 language models, varying the ratio of added AI tokens to human tokens, and fit scaling laws to held-out losses on both human and AI-generated text. For data-starved models, adding AI tokens to pretraining data initially lowers loss on human text, but the benefit saturates as more are added and quickly *reverses* into harm. For models trained on high budgets of human text, AI tokens raise loss almost immediately, while the same number of fresh human tokens keeps lowering it. Scaling laws such as Hoffman et al. (2022) fail to predict this behavior. We propose a new scaling law with separate benefit and harm terms that allows the value of an AI token to change sign while also reducing to Chinchilla in the absence of AI text. When fit on smaller models, our scaling law predicts the effect of AI text on held-out human-text loss for models up to 3.6x larger with 41% lower error than the best existing law over all AI ratios. We recommend filtering AI text when the target is human text, repeating human text before expanding the training dataset with AI-generated web text, and reporting validation loss on human and AI text separately AI text remains valuable when the target is AI text. We release WildAI, an 83B-token corpus with AI, topic, and format labels, all 800 models and code at https://github.com/pangramlabs/WildAI.

1 Introduction

The paper studies naturally occurring AI-generated web text and finds that its effect on human-text loss depends on the human-data budget. It proposes a scaling law for this changing value and draws training recommendations from its predictions.

1 Introduction

  • AI tokens lower human-text loss only below about 10 human tokens per parameter; at 20, their benefit disappears and larger additions raise loss.The same number of fresh human tokens continues to lower loss.Source: §1 · p. 2
  • 0.83 × 10−3 paired error on C4, versus 1.41 × 10−3 for the best existing law, when the new law is fit through 268M and predicts models 3.6× larger.The law pairs saturating benefit with a logarithmically growing harm penalty and reduces to Chinchilla without AI text.Source: §1 · p. 2
  • At an AI share of 31.1%, unfiltered web training needs 1.6× the compute of training on its human subset, rising to 3.0× at the forecast 2028 share.The authors recommend repeating human text before adding AI text and reporting validation losses on human and AI text separately.Source: §1 · p. 3

2 Measuring AI-Generated Web Data

This section measures AI-generated web text, builds the WildAI corpus, and examines how it differs from human text and passes quality filters.

2 Measuring AI-Generated Web Data

  • WildAI contains 96.04M documents and 83.31B tokens, collected by extending FineWeb filtering to Common Crawl data from July 2025 to June 2026.EditLens preselected likely AI documents, so AI text accounts for 42% of WildAI tokens; the corpus is a training-mixture pool, not a natural web sample.Source: §2 · p. 3
  • 27.5% of June 2026 tokens passing FineWeb filters were AI-generated, up from 10.1% in June 2024 and reaching 31.1% in August 2026.The share continued rising across the sampled months.Source: §2 · p. 3
  • AI and human web text differ in format representation: tutorials account for 18.5% of AI-labeled tokens versus 6.2% of human-labeled tokens, while personal blogs account for 1.0% versus 8.6%.Figure 2 compares format token-share ratios and AI- versus human-document survival through FineWeb stages.Source: §2 · p. 4, Figure 2 · p. 4
  • FineWeb and DCLM retained AI-generated documents more often than human-written ones: 2.3× as often for FineWeb and 9.8× for DCLM.The measured pass rates were 29.3% vs. 12.8% for FineWeb and 14.5% vs. 1.5% for DCLM.Source: §2 · p. 4

3 Background and Related Work

The section reviews scaling laws for training compute and data quantity, then situates repetition and data-mixture laws relative to wild AI text.

3 Background and Related Work

  • Chinchilla relates held-out loss to compute, model size, and training-token count, but its data term treats repeated and unique tokens alike.Training compute is commonly approximated as C ≈ 6ND.Source: §3 · p. 4, Scaling laws for repetition · p. 4

Scaling laws for repetition.

  • Repetition laws let repeated tokens have lower effective value than new tokens, but do not allow that value to fall below zero.Other proposals make token value depend on model size or model repetition with an overfitting penalty.Source: Scaling laws for repetition · pp. 4–5

Scaling laws for data mixtures.

  • Wild AI differs from curated synthetic data and recursive model-collapse settings; prior mixture work also finds that synthetic-data effects depend on budget.The surrogate-data law of Jain et al. (2024) predicts held-out C4 runs worse than Chinchilla: 8.00 vs. 4.32, while this paper reports 0.83, all ×10−3.Source: Scaling laws for data mixtures · p. 5

4 Scaling Laws for Human-Written and AI-Generated Text

Added AI text helps human-text loss only when human training data is scarce and harms it at larger budgets; the paper proposes a scaling law that models both effects.

4.1 Empirical Observations

  • At 5 TPPh, AI text lowers loss even at large AI additions, but at 20 or more TPPh it raises loss immediately and increasingly as AI additions grow.For fresh human additions, Chinchilla predicts the held-out C4 paired error more accurately than it does for AI additions.Source: §4.1 · p. 6, Figure 3 · p. 6

4.2 Criteria

  • The proposed law requires AI-token value to vary with model size, allow both benefit and harm, and give the first AI token a finite value.Source: §4.2 · pp. 6–7
  • It also separates benefit from harm and must recover Chinchilla scaling laws when no AI data is added.Source: §4.2 · p. 7

4.3 Scaling Laws for AI-Generated Text

  • The law combines a saturating credit for AI tokens with a harm penalty that grows logarithmically, preserving the Chinchilla backbone when AI text is absent.The benefit uses an exponential credit function with a free first-token parameter; the harm’s marginal effect peaks when AI and human tokens are equal.Source: §4.3 · pp. 7–8
  • 0.83 ×10−3 paired RMSE on C4: the proposed law has the lowest error across four human sources, versus 1.41 ×10−3 for the best existing law.Fit on models up to 268M parameters, it predicts held-out runs up to 973M parameters; Shukor et al.’s law performs best when the target is AI-generated text.Source: Table 1 · p. 8, §4.3 · p. 8

5 The Effects of Training on AI Text and What to Do About It

The section evaluates when wild AI text helps, what it costs to retain in human-targeted training, and how its value depends on the evaluation target. It recommends filtering AI text for human next-token prediction, repeating human data before adding AI text, and reporting human- and AI-text losses separately.

5.1 The Cost of Not Filtering

  • At 2026’s natural AI share, models above about 10 to 15 TPPh had lower C4 loss after filtering AI text in 18 of 19 pairs.The authors recommend removing AI-generated web text for human next-token prediction.Source: §5.1 · p. 9
  • At August 2026’s 31.1% AI share, unfiltered training needs 1.6× the compute, rising to 3.0× in 2028, to match filtered training on C4.The forecast AI share is based on extrapolating the observed rise in AI tokens.Source: §5.1 · pp. 9–10
  • On C4, repeating human tokens lowered loss 2.3% more than adding fresh AI tokens after one added epoch, widening to 6.3–6.4% after eight.The comparison used 19.9M and 35.8M models at 20 TPPh.Source: §5.1 · p. 10

5.2 When Is AI Text Worth Training On?

  • For human-text evaluation, the loss-minimizing AI share on C4 falls from 37% at 5 TPPh to 1.0% at 20 TPPh, while AI-text targets favor shares above 90%.FW22 and Paloma show a similar pattern to C4 for human text.Source: §5.2 · p. 10
  • At 268M parameters, AI-typical phrases rose from 0.16 per 1,000 words without AI text to 0.36 at r = 1 and 0.54 at r = 4.At r = 2, Pangram labeled 37.2% of stories AI, versus 18.6% without AI text.Source: §5.2 · p. 10
  • All 553 models with added AI text lowered loss on AI-labeled FW26 text, while 44% raised loss on human-labeled text; mixed validation reported 95.5% of harmful runs as improvements.The authors recommend reporting loss on human and AI text separately.Source: §5.2 · p. 10

6 Conclusion

The conclusion presents wild AI text as a growing share of filtered web crawls and reports that its effects depend on training and evaluation targets. It summarizes the proposed scaling law, recommendations, release, and study boundaries.

6 Conclusion

  • Across 800 models, AI text helped only data-starved models, while increasing loss on Chinchilla-optimal models; the new law predicts models 3.6× larger with lower human-text error than all eleven tested laws.The law combines a saturating benefit with logarithmically growing harm and reduces to Chinchilla without AI text.Source: §6 · p. 10
  • Figure 6’s optimal AI share is near zero for human text and above 90% for AI text; mixed validation can report harmful additions as improvements.For the 243 additions that raised loss on human-labeled FW26, the reported sign flips at 5.1% AI in the validation set.Source: Figure 6 · p. 11
  • The authors recommend filtering AI text for human targets, repeating human text before adding AI text, and reporting human- and AI-text validation losses separately.WildAI, its labels, code, and all 800 models are released.Source: §6 · p. 11
  • The law is fit on models up to 268M parameters and tested up to 973M, and the study measures next-token loss rather than meaningful performance at these sizes.The authors note that larger models might respond differently to model-generated data.Source: §6 · p. 11
  • The study covers pretraining on English web text labeled by Pangram; the authors leave other domains and training settings for future work.They identify measuring the effects of AI text already present in current models as future work.Source: §6 · p. 11

AI Use Statement

The statement discloses AI assistance across research formulation, experiments, implementation, data processing, and analysis, and says the authors reviewed and validated the assisted work. The reproducibility statement describes released data, models, law specifications, and code.

AI Use Statement

  • Coding agents helped formulate scaling-law terms, propose hypotheses, and suggest experiments, including the repetition experiment.Agents also provided methodological feedback and helped design and run training of the additive models.Source: AI Use Statement · p. 12
  • Coding agents implemented experimental code, ran the FineWeb filtration process on recent crawls, and helped create tables and figures for analysis.The disclosure also lists figure creation, software editing, brainstorming, literature searching, and readability editing among AI uses.Source: AI Use Statement · p. 12
  • The authors state that they reviewed and validated AI-assisted work and take responsibility for the final content and artifacts.Source: AI Use Statement · p. 12

Reproducibility Statement

  • The release includes WildAI’s 83B tokens with document-level labels, every model checkpoint and evaluation loss, scaling-law specifications, and code for the pipeline and analyses.The materials are described as sufficient to regenerate fits, tables, and figures without retraining.Source: Reproducibility Statement · p. 12

A Data

The appendix describes WildAI’s collection and labeling, documents patterns in AI-labeled topics and formats, audits quality filters, and forecasts the web’s AI-token share.

A Data

  • WildAI combines released FineWeb dumps with a FineWeb-pipeline extension through June 2026, totaling 96.04M documents and 83.31B tokens.The extension processes the first 2.6% of each of twelve Common Crawl dumps from July 2025 to June 2026.Source: A Data · p. 18
  • EditLens selects human and AI candidate documents, then Pangram labels them Human or AI only when all sampled windows agree; otherwise documents are Mixed.The final training pool contains 42.19B human tokens and 35.32B AI tokens, with Mixed documents excluded.Source: A Data · p. 18
  • In 2026, 51% of tutorial tokens and 49% of knowledge-article tokens were AI-labeled, compared with 9% of news and 5% of personal-blog tokens.The training-pool formats with the highest AI-labeled shares included knowledge articles, tutorials, and listicles.Source: A Data · p. 19, Figure 7 · p. 19

A.1 Effects of Quality Filtering

  • FineWeb retained AI-labeled documents 2.3× as often as human-labeled ones, while DCLM retained them 9.8× as often.The DCLM gap opens at its learned classifier, and AI documents also survive more often than human documents within the same format.Source: §A.1 · p. 21, Figure 9 · p. 21
  • DCLM’s learned classifier removed 111 of 117 news articles and 106 of 117 personal blogs among human prose reaching it.Many other removed human documents failed English filters or were extraction debris.Source: §A.1 · p. 22

A.2 Forecasting the AI Share of the Web

  • A random walk with drift forecasts 50.7% of web text will be AI by the end of 2028; Figure 10 plots monthly shares and prediction intervals.The forecast uses crawl-month samples and includes 80% and 95% prediction intervals.Source: §A.2 · p. 22, Figure 10 · p. 22, Table 2 · p. 23

B Training Configurations

The 800-model cohort uses a shared nanochat training recipe and matched human-only controls for AI and fresh-human additions. The section also frames AI additions through scaling laws for repeated data, where effective tokens discount repetition.

B Training Configurations

  • All 800 models share a nanochat recipe, and every AI or fresh-human addition uses its control’s human documents in the same order and with the same seed.Runs span 19.9M to 973M parameters; 703 use one seed, while 97 are replicates.Source: B Training Configurations · p. 23
  • The model-size count includes input embeddings, and the aligned human-data budgets are reported in TPPh, rounded to the nearest five in figures.Table 6 supplies exact values; some high-size budget cells are absent.Source: B Training Configurations · pp. 23–24, Table 6 · p. 25

C.1 Scaling Laws for Repetition

  • For repetition, unique tokens are UD and additional epochs beyond the first are RD, giving total tokens D = UD(1 + RD).The effective-data formulation treats repeated tokens as discounted credit, with a fitted saturation scale.Source: §C.1 · pp. 24–25

B Training Configurations

  • The training grid pairs each human-only control with AI-addition and fresh-human-addition runs, reserving the largest marked sizes for evaluation rather than fitting.The 800-model table also distinguishes other seeds and filtering or repetition models, which enter no fit.Source: Table 4 · p. 24, Table 5 · p. 25

C.1 Scaling Laws for Repetition

  • In the mixture benchmark, human tokens serve as unique data and added AI tokens as repeated data, with RD = DA/DH.Related repetition laws also vary effective token value with model size, fresh tokens per parameter, and repetition.Source: §C.1 · p. 25

C.2 Scaling Laws for Data Mixtures

The paper benchmarks mixture and other published scaling-law forms, fitting them to paired changes in loss against human-only controls. It selects a saturating benefit and decelerating harm, with AI’s effect shifting from early benefit at low human budgets to harm at higher budgets.

C.2 Scaling Laws for Data Mixtures

  • Shukor et al.’s mixture law represents human and AI shares in an added term, with a joint form that rescales model and data amplitudes.The joint form is benchmarked as the strongest published comparator.Source: §C.2 · pp. 25–26

C.3 Other Benchmarked Scaling Laws

  • Other published functional forms are specialized to two sources and rewritten in shared coordinates before their coefficients are fit on the experiments.The benchmark includes Hamidieh et al.’s form.Source: §C.3 · p. 26

D Scaling-Law Fitting and Model Selection

  • All laws are fitted on the same 726 development models using residuals from additions against their own human-only controls, with 74 larger models held out.Coefficients are fit by bounded least squares minimizing Huber loss, initialized from a Chinchilla fit.Source: D Scaling-Law Fitting and Model… · p. 26

D.1 Choosing the Scaling Law

  • The selected law combines an exponential saturating benefit with a logarithmic harm penalty, and the benefit window narrows as human budgets grow.On held-out sizes, the exponential benefit scores 0.83 ×10−3, matching a tanh window; alternative harm placements score 0.80 versus 0.83 ×10−3 on C4, within bootstrap noise.Source: §D.1 · pp. 27–28, Figure 12 · p. 27, Table 8 · p. 28

D.2 Law Fits Across Model Sizes and Budgets

  • At low human-token budgets, added AI text first lowers C4 loss and then raises it; from about 20 TPPh, it raises loss from the first tokens, with harm growing roughly as log r.The fitted law also compares its AI-token value with measured values from paired AI and fresh-human runs.Source: §D.2 · p. 28

E Benchmarking Other Scaling Laws

The paper benchmarks 11 existing scaling laws against the effects of added AI text, using held-out paired errors. Its law generally performs best on human-text targets, while the strongest published joint law has strengths on mixed and AI text but unstable extrapolations.

E Benchmarking Other Scaling Laws

  • Five comparators predict essentially no AI-text effect on Paloma because their fitted data terms become negligible, so they incur the error of predicting zero change.Their errors are 9.15, or 7.75 for r < 1.Source: E Benchmarking Other Scaling Laws · p. 29
  • For absolute loss on C4, CD and ATLAS outperform the proposed law, but paired error is the main metric because it isolates AI-text effects from baseline loss.Their C4 errors are 12.51 and 13.65 ×10−3, versus 14.09 ×10−3 for the proposed law.Source: E Benchmarking Other Scaling Laws · p. 30

E.1 Statistical Significance

  • Across all AI ratios, the proposed law has lower paired error than the joint Shukor law in 99.99% of C4 resamples and every FW22 and FW26-H resample.The C4 error gap is 0.58 ×10−3, with a 95% interval of 0.32 to 0.84.Source: §E.1 · p. 30
  • On Paloma, the proposed law beats Shukor’s joint law in 98.7% of resamples but is best among all twelve laws in only 89.4%.The resampling holds each fit fixed while drawing held-out AI runs by control group.Source: §E.1 · p. 30

E.2 Comparison with the Joint Law of Shukor et al. (2025)

  • Shukor’s joint law is best on mixed and AI-text targets, but its 8B projections reverse the predicted AI-text effect depending on fit size.No 8B model was trained, so these are projections for both laws.Source: §E.2 · p. 31

F.1 Compute-Equivalent Gain (CEG)

The section compares AI additions with repeated human data and evaluates AI’s value across human, AI, and mixed targets. Repetition favors human-text evaluation, while AI additions favor AI-text evaluation; mixed validation can conceal harm to human text.

F.1 Compute-Equivalent Gain (CEG)

  • Compute-equivalent gain compares tokens needed by a reference recipe to reach the same loss with the evaluated recipe’s token count.At fixed model size, compute is proportional to training tokens; values above one indicate gain over reference tokens.Source: §F.1 · p. 31

F.2 Optimal AI Share

  • The optimal AI share is obtained by minimizing predicted loss over the AI-to-human token ratio while holding model size and human corpus fixed.Setting the derivative of the data term to zero balances marginal AI-token credit against marginal harm.Source: §F.2 · pp. 31–32

F.3 Repeating Human Tokens vs. Adding AI Tokens

  • On the three human-text sets, repetition beats matched AI additions in all 18 comparisons, lowering loss by 1.7–3.8% while AI text raises it by 0.2–3.4%.On the two AI-text sets, AI additions lower loss by 12–19%, compared with 1.6–4.9% for repetition; FW26’s winner depends on r.Source: §F.3 · p. 32, Figure 20 · p. 34
  • The compute-efficient AI share differs by target: less AI is more efficient for C4, while more AI is more efficient for Cosmopedia.The comparison uses 268M models at 20 TPPh against the 2026 crawl mix of 22.3% AI.Source: Figure 19 · p. 34

F.4 Loss on AI-Generated and Mixed Text

  • A mixed validation set can report apparent improvements for additions that raise loss on human-labeled text, with that share reaching 95.5% at 22.3% AI text.AI-labeled text has lower loss than human-labeled text, and models trained with AI text move further below the equal-loss line.Source: Figure 22 · p. 35, §F.4 · p. 35

G Downstream Metrics

Added AI tokens raise CORE similarly to fresh human tokens, yet can worsen fit to human text; increasing AI exposure also raises AI-typical phrasing in generations.

G Downstream Metrics

  • CORE performance increases similarly with additional AI or human tokens; MMLU is omitted because all models remain at chance accuracy.The authors hypothesize that differences may be greater on tasks requiring diversity or creativity.Source: G Downstream Metrics · p. 35, Figure 23 · p. 36
  • At 20 TPPh, adding at least as many tokens as the human corpus raises CORE by 1.6 points with AI text and 1.9 with fresh human text, while C4 loss rises 2.3% versus falling 3.0%.The comparison covers 23 AI-text runs and 6 fresh-human-text runs; models answer benchmark questions about as well but model human text less closely.Source: G Downstream Metrics · p. 36
  • For story continuations, the rate of 26 phrases typical of AI-generated text roughly doubles by r = 0.5, rising from 0.14–0.17 to 0.34–0.36 per 1,000 words at both budgets.Source: G Downstream Metrics · p. 36
  • At 20 TPPh, AI-typical phrases rise from 0.16 per 1,000 words without AI text to 0.36 at r = 1 and 0.54 at r = 4; at 5 TPPh, the rate reaches 0.72 at r = 4.Pangram labels 18.6% of control stories AI, 28.4% at r = 1, 37.2% at r = 2, and 28.9% at r = 4.Source: G Downstream Metrics · p. 36
  • Our law leads the joint law of Shukor et al. (2025) across AI ratios on human text, except at r < 1 on C4 and FW22.The comparison uses paired RMSE on held-out runs.Source: Table 12 · p. 39
Loading 2609.40295v1…