PaperlayerView exact revision on arXiv

Source-linked AI summary

RePro: Training Language Models to Faithfully Recycle the Web for Pretraining

Zichun Yu, Chenyan Xiong

arXiv:2510.10681v2cs.CLcs.LG

TL;DR

High-quality pretraining data is scarce, while large-model prompting for web recycling is costly and may not preserve organic semantics and structure. RePro trains a small rephraser with reinforcement learning and quality and faithfulness rewards; it improves downstream accuracy over organic-only baselines and doubles ReWire's gains.

  • Problem

    High-quality pretraining data is scarce as organic web text is exhausted and curation discards much of the web; prompting large rephrasers is costly and may not preserve semantics or structure.

    Source: §1 · p. 1

  • Method

    RePro uses reinforcement learning to train a small LM rephraser with a quality reward and three faithfulness rewards for semantics, structure, and length.

    Source: §1 · p. 1

  • Results

    Across 22 downstream tasks, RePro yields 3.7%–14.5% relative accuracy gains over organic-only baselines and doubles ReWire's gains.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Small models with designed rewards can recycle web data more effectively and faithfully; preserving organic characteristics maintains distribution and enhances pretraining performance with recycled data.

    Source: §6 · p. 9

    Faithfulness rewards mitigate, but may not fully eliminate, biases introduced by the generating model in recycled data.

    Source: Impact Statement · p. 10

Abstract

from arXiv · show

High-quality data is a cornerstone of large language model (LLM) pretraining, yet its growth has not kept pace with the needs of frontier models. In this paper, we introduce RePro, a novel web recycling method that trains a relatively small LM with reinforcement learning to generate effective and faithful rephrasings of pretraining data. Specifically, we design one quality reward and three faithfulness rewards, optimizing the LM rephraser to convert organic data into high-quality rephrasings while maintaining its core semantics and structure. In our experiment, we train a rephraser as small as 1B parameters to recycle 72B tokens sampled from DCLM-RefinedWeb. Pretraining results on 400M, 1.4B, and 2.8B models demonstrate that RePro delivers 3.7%-14.5% relative accuracy gains over organic-only baseline on 22 downstream tasks, doubling the performance gains achieved by the state-of-the-art web recycling method that prompts a 70B rephraser. Experiments with different amounts of recycled data highlight that RePro improves organic data efficiency by 2-3x. Individual and distributional analyses validate that RePro preserves more critical information and faithfully reflects the characteristics of organic data compared to prompting-based methods. Together, these results show that RePro provides an efficient and controllable path to effectively recycle organic data for pretraining. Our code is available at https://github.com/cxcscmu/RePro.

1. Introduction

The introduction presents web recycling as a response to scarce high-quality pretraining data and proposes training a small rephraser to improve quality while preserving organic data. It reports gains over organic-only and prompting-based approaches, alongside evidence of greater data efficiency and faithfulness.

1. Introduction

  • Scarce high-quality web data motivates recycling, but large-model prompting is costly and may fail to preserve organic semantics and structure.Standard pipelines discard much web content as low quality, contributing to the pretraining data shortage.Source: §1 · p. 1
  • Across 22 downstream tasks, REPRO yields +3.7% to +14.5% average-accuracy gains over the organic baseline and outperforms ReWire and a 4× enlarged organic pool.The comparison uses models pretrained on high-quality organic and recycled data.Source: §1 · p. 2
  • REPRO boosts organic data efficiency by 2–3×, measured as high-quality data in the final pretraining set relative to high-quality organic data.The experiments increase the amount of recycled data in final mixtures.Source: §1 · p. 2
  • REPRO yields 95% supported key points and reduces omitted key points by up to 92% versus prompting-based WRAP and ReWire.The analysis measures supported, omitted, and contradicted key points in recycled versions of organic data.Source: §1 · p. 2
  • REPRO trains a small language-model rephraser with quality and faithfulness rewards to produce high-quality rephrasings while preserving organic semantics, structure, and length.The introduction identifies DataMan as the quality reward and BERTScore, structure alignment, and length preservation as faithfulness rewards.Source: §1 · pp. 1–2

2. Related Work

Related work describes synthetic data as a potential complement to scarce organic text, while noting challenges with fidelity, diversity, and distributional collapse in pretraining.

2. Related Work

  • Organic text is increasingly scarce, and standard curation discards much collected web data; synthetic data is presented as a scalable complement across training stages.Prior pretraining approaches include rephrasing, guided rewriting, and topic-seeded generation.Source: §2 · p. 2
  • Synthetic pretraining data remains less understood than mid- and post-training generation, with limited diversity and reduced fidelity to real-world distributions identified as challenges.Source: §2 · p. 2
  • Indiscriminate synthetic-data use can trigger model collapse and harm generalization; ProX and RefineX instead restrict refining to conservative operations such as deletion and normalization.Source: §2 · pp. 2–3

3. Methods

The methods construct a pretraining mixture from quality-filtered organic and recycled data, then use reinforcement learning to train a rephraser with quality and faithfulness objectives.

3.1. Recycling Pretraining Data

  • When high-quality organic tokens fall short of the pretraining budget, the setup rephrases organic samples, filters the recycled pool by quality, and combines the high-quality organic and recycled subsets.The recycled high-quality subset is sized to fill the token-budget gap.Source: §3.1 · p. 3

3.2. Training a Faithful Rephraser with RL

  • The RL objective rewards outputs that improve on the original's DataMan quality score and preserve semantic similarity measured with BERTScore.The quality reward compares DataMan scores for the recycled and organic samples.Source: §3.2 · p. 3
  • Additional faithfulness rewards assess textual structure and constrain recycled-text length relative to the original.A structure-comparison language model evaluates structural faithfulness.Source: §3.2 · pp. 3–4
  • GRPO normalizes advantage estimates across a group of generated outputs and updates the rephraser with a clipped surrogate objective; the optimized model then generates the recycled pool.Source: §3.2 · p. 4

4. Experimental Setup

The experiments use DCLM-RefinedWeb organic data, high-quality organic and rephrased tokens in a 1:1 mixture, and decoder-only models at three parameter scales. REPRO is compared with organic-only training and several web-recycling baselines on 22 downstream tasks.

Pretraining Data and Model

  • The main DCLM-RefinedWeb experiments sample 72B/144B organic tokens, set the high-quality organic and rephrased budgets equally, and train on a 1:1 mixture.The organic pool uses rule-based filtering and global deduplication, but not model-based filtering; the models are trained from scratch.Source: §4 · p. 4

Evaluation

  • Pretrained models are evaluated on 22 downstream tasks in zero-shot or few-shot settings, with average centered accuracy across all tasks serving as the primary metric, called the Core score.Tasks assess commonsense reasoning, language understanding, reading comprehension, symbolic problem solving, and world knowledge. CommonsenseQA is excluded from the original DCLM-Core due to high instability and limited informativeness.Source: §4 · p. 4

Baselines

  • REPRO is compared with organic-only training, WRAP, ProX, and ReWire, representing prompting, program-based editing, and chain-of-thought rewriting.ReWire’s released data may give it a larger combined organic pool; RefineX is left for future comparison because its data and code are unavailable.Source: §4 · pp. 4–5

Implementation Details

  • The rephraser is initialized from OLMo2-1B-Instruct or Qwen3-4B, and the RL dataset contains 41k organic examples with DataMan scores below 5.An optional GPT-4o-generated SFT warm-up is used in some experiments; unless specified otherwise, REPRO denotes RL without SFT.Source: §4 · p. 5

5. Evaluation Results

Across model setups, REPRO improves pretraining results over organic-only data and recycling baselines, while analyses examine data budgets, recycling patterns, and reward behavior. The experiments also report that its reward functions jointly improve quality while preserving properties of the organic data.

5.1. Main Results

  • 3.7%–14.5% relative Core-score gains over organic-only training across three setups exceed two standard deviations of the baseline, while REPRO doubles baseline gains.The comparison includes ReWire, which uses a prompted 70B rephraser; with 28.8B additional tokens, the 3B-setup gap grows from +3.7% to +4.7%.Source: §5.1 · p. 6
  • At unique-token budgets of 14.4B, 21.6B, and 28.8B, REPRO beats all compared baselines; the results suggest 2–3× greater organic-data efficiency.The best REPRO result occurs at 14.4B, 21.6B is comparable and yields the largest improvement over baselines, and performance drops at 28.8B; ReWire is excluded from Figure 3 because its full recycled pool is unavailable.Source: §5.1 · p. 6, Figure 3 · p. 6

5.2. Ablation Studies

  • Prompting and SFT alone provide little overall benefit over the organic baseline, and adding reward descriptions to prompts does not close the gap with REPRO.Removing faithfulness rewards substantially reduces performance; the comparison is reported for the 400M ablation setup.Source: Table 3 · p. 6, §5.2 · p. 7

5.2. Ablation Studies / 5.3. Recycling of Non-High-Quality Data

  • Excluding high-quality organic inputs from recycling recovers 77.2% of REPRO’s total gains, while its high-quality recycled pool includes 63.5% of examples sourced from low-quality data.The corresponding low-quality-source proportions are 31.0% for Prompting, 42.5% for WRAP, and 39.0% for ProX; combining recycled and organic high-quality data remains best.Source: §5.2 · p. 7, §5.3 · p. 8

5.2. Ablation Studies

  • Nemotron-CC Diverse QA helps reading comprehension and world knowledge but trails REPRO on overall Core despite using a stronger generator.The authors suggest its gains may arise from distilling parametric knowledge rather than improving the organic data distribution.Source: §5.2 · pp. 7–8

5.3. Recycling of Non-High-Quality Data

  • REPRO flexibly applies paraphrasing, removing, clarification, reorganization, and summarization; paraphrasing is most common and removing is second.Removing can eliminate irrelevant material such as advertisements and metadata.Source: Figure 5 · p. 7, §5.3 · p. 8

5.4. Effectiveness of Adopted Reward Functions

  • REPRO shifts DataMan score-5 data from 20% in organic data to 60%, and its mean BERTScore is 0.75 versus 0.69 for direct prompting and 0.56 without the BERTScore reward.Its structural and length distributions align most closely with organic data, whereas direct prompting tends toward plain text and shorter outputs, and omitting the structure or length reward distorts those distributions.Source: Figure 6 · p. 8, §5.4 · p. 9
  • With all rewards enabled, DataMan, BERTScore, and structure rewards improve during RL training while the length reward remains high; without faithfulness rewards, the other three drop as DataMan quickly converges.The authors interpret this contrast as evidence of overfitting without faithfulness constraints.Source: §5.4 · p. 9, Figure 7 · p. 9

6. Conclusion

REPRO recycles web data with a small language model and improves pretraining accuracy over organic-only baselines. The conclusion attributes its results to preserving key characteristics of organic data.

6. Conclusion

  • 3.7%-14.5% relative accuracy improvements over organic-only baselines; REPRO also doubles ReWire’s gains and surpasses an organic baseline enlarged by 4×.The paper reports these results for pretraining with recycled data.Source: §6 · p. 9
  • Analyses indicate that REPRO maintains organic data’s semantic meaning, structural diversity, and length balance.The conclusion says preserving essential characteristics helps maintain the original distribution and enhances pretraining performance.Source: §6 · p. 9
  • The authors identify more diverse, verifiable reward signals and evaluation on other web corpora as future work.They name checklist-style rewards and FineWeb and RedPajama as examples.Source: §6 · p. 9

Impact Statement

The impact statement considers recycling as a response to constrained supplies of high-quality organic data. It presents a small rephraser as a lower-cost approach while noting that generated data may still reflect model biases.

Impact Statement

  • High-quality organic data may be exhausted before it can scale with growing model capacity, motivating more efficient recycling under data constraints.The statement frames this as an emerging challenge in LLM pretraining.Source: Impact Statement · p. 10
  • A relatively small rephraser reduces computational and financial costs, potentially widening access to web recycling.The statement connects this approach to reliable, scalable data utilization under data-constrained conditions.Source: Impact Statement · p. 10
  • Recycled data may inherit biases from its generating model; faithfulness objectives are intended to mitigate, but may not fully eliminate, associated risks.The stated aim is to preserve original semantics and structure and maintain consistency between recycled and original distributions.Source: Impact Statement · p. 10

A. Per-Category Discussion of Table 2

The per-category discussion relates REPRO’s task performance to the content and style of its rephrasings, while noting a reading-comprehension trade-off.

A. Per-Category Discussion of Table 2

  • For commonsense reasoning and symbolic problem solving, REPRO’s flexible use of operations performs better than ProX’s removal of irrelevant content.The discussion highlights coherent, consistent logical flow as preferred for these tasks.Source: A. Per-Category Discussion of Table 2 · p. 13
  • For language understanding, REPRO is the only discussed web recycling method to surpass the organic-only baseline.The authors hypothesize that its faithfulness better aligns the rephrased data’s content and writing style with human-text interpretation tasks.Source: A. Per-Category Discussion of Table 2 · p. 13
  • For reading comprehension, contextual explanations may help, but repeated patterns may reduce training-data diversity and limit generalization.ReWire is cited as often adding phrases that explain context.Source: A. Per-Category Discussion of Table 2 · p. 13
  • For world knowledge, REPRO outperforms the other baselines except ReWire, which is slightly ahead.The authors suggest ReWire’s much larger rephrasing model can add factual knowledge, with a risk of reduced faithfulness; REPRO removes irrelevant content and adds clarifications.Source: A. Per-Category Discussion of Table 2 · p. 13

B.1. Efficiency Comparison

REPRO is reported to reduce rephrasing time relative to other approaches. A compute-equalized comparison also tests its results against additional organic-data training.

B.1. Efficiency Comparison

  • The rephrasing-hours comparison excludes ProX because it is an operation-based method.Source: Table 4 · p. 14
  • Up to 2.9× speedup over WRAP and 87.2× over ReWire are reported for REPRO’s efficiency comparison.The comparison concerns training and inference hours required for rephrasing.Source: §B.1 · p. 14, Table 4 · p. 14
  • With total rephrasing-plus-pretraining compute equalized, REPROOLMo2-1B outperforms the organic baseline across all five task categories.The baseline uses the top 28.8B high-quality tokens from the 72B organic pool and trains for 115.2B tokens.Source: §B.1 · p. 14

B.2. Training on High-Quality Data Only

REPRO’s recycled data complements organic data, and controlled comparisons and variance analysis support the robustness of its gains over ReWire and the organic baseline.

B.2. Training on High-Quality Data Only

  • Merging REPRO’s recycled and organic data gives the best result; recycled data alone beats the organic-only baseline by +6.5% on Core.ReWire’s recycled data alone slightly underperforms the organic-only baseline.Source: §B.2 · p. 14

B.3. Controlled Comparison with ReWire

  • REPRO beats ReWire by 9.0% on Core with a matched 72B pool and outperforms it across all categories when each method uses a separate 72B pool.These controlled comparisons indicate that the gains are not an artifact of differing organic-data sources.Source: §B.3 · pp. 14–15, Table 7 · p. 15

B.4. Significance Analysis

  • REPRO’s gains over the organic baseline exceed twice the organic baseline’s standard deviation in every category and in aggregate Core.The organic-side variance across three seeds serves as a conservative noise floor because compute and time constraints prevented rerunning the full recycling pipeline across multiple seeds.Source: §B.4 · p. 15, Table 8 · p. 15

B.5. Improvements on All DataMan Criteria

REPRO improves organic data across DataMan criteria, with reward settings that balance quality and faithfulness. Experiments also show that a useful selection metric need not make an effective RL reward.

B.5. Improvements on All DataMan Criteria

  • REPRO outperforms organic data and prompting baselines across DataMan criteria, although novelty and creativity scores remain low after rephrasing.The paper attributes this difficulty to the challenge of enhancing these subjective aspects through rephrasing while preserving original content.Source: §B.5 · p. 15, Figure 8 · p. 16

B.6. Hyperparameter Studies

  • DataMan coefficients 1, 3, and 6 yield similar reward-curve patterns; increasing DataMan’s weight speeds its rise but increases faithfulness-reward fluctuations.The final coefficient balances DataMan and faithfulness rewards; top-p=0.96 performs slightly better than two other prompting sampling settings, but overall differences are small and REPRO remains stronger.Source: §B.6 · pp. 15–16, Table 9 · p. 16, Figure 9 · p. 17

B.7. Choice of Quality Reward

  • FastText reward quickly saturates while influence reward falls to zero, and their learned rephrasers respectively exploit classifier tone and struggle to find consistently rewarding strategies.The authors conclude that a data-selection metric may not be suitable as an RL reward; a useful reward should correlate with pretraining performance, resist hacking, and be optimizable.Source: §B.7 · p. 17, Figure 10 · p. 17
  • DataMan selection alone or combined with DCLM-fastText selection does not beat DCLM-fastText selection alone.The authors infer that merging quality selections may not increase the amount of high-quality data, and that a rephraser-training reward is not necessarily the best selection function.Source: §B.7 · p. 17, Table 10 · p. 17

C.2. Evaluation Tasks

The appendix points readers to the evaluation task and few-shot listings, describes baseline implementations, and gives prompt criteria and examples. It also illustrates how REPRO modifies web text while retaining its key characteristics.

C.2. Evaluation Tasks

  • The evaluation task names and few-shot counts are listed in Table 12.The supplied passages do not enumerate the table’s entries.Source: §C.2 · p. 17, Table 12 · p. 18

C.3. Baselines

  • WRAP uses Mistral-7B-Instruct with a Wikipedia-style prompt, ProX learns program-based edits, and ReWire uses guided rewriting with Llama-3.3-70B-Instruct.The authors distinguish ProX’s program-based operations from their generation-based rephrasing.Source: §C.3 · p. 18

C.4. Prompts

  • The rephrasing prompt asks for high-quality output while preserving semantic, structural, and length faithfulness and avoiding unsupported additions.The instructions also specify accuracy, coherence, language consistency, and focused wording as quality criteria.Source: §C.4 · p. 23

D.1. Behavior of REPRO Recycling

  • REPRO can clarify low-quality text, remove irrelevant webpage artifacts, and reorganize higher-quality text while preserving key characteristics such as style, structure, and genre.Manual inspection found no degeneration, repeated formatting patterns, or reduced diversity across the examined cases.Source: §D.1 · p. 24

D.2. Behavior of Different Rephrasing Methods

Compared with other rephrasing methods, REPRO adheres more closely to original content and structure while improving writing quality. Gallipoli examples illustrate how some alternatives add unsupported material or change the source’s form.

D.2. Behavior of Different Rephrasing Methods

  • ReWire and REPRO without faithfulness rewards can add loosely related information and alter markdown structure, while WRAP also adds content absent from the source.The authors warn that these changes may increase quality scores while reducing faithfulness, posing a risk of model collapse; REPRO instead removes irrelevant advertisements and metadata while adhering to original content and structure.Source: §D.2 · pp. 26–27, §5 · p. 27

5 Key Points About Gallipoli

  • In the Gallipoli examples, prompting preserves the source’s five facts, while RL without faithfulness rewards adds extensive historical details beyond the source.The latter includes a broader campaign narrative and claims about strategic objectives and consequences not present in the short source facts.Source: §5 · pp. 28–29
Loading 2510.10681v2…