PaperlayerView exact revision on arXiv

Source-linked AI summary

Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling

Pratyush Maini, Skyler Seto, He Bai, David Grangier, Yizhe Zhang, Navdeep Jaitly

arXiv:2401.16380v1cs.CL

TL;DR

Large language model pre-training depends on noisy web data while scaling laws increase the required compute and data, making efficient curation important. WRAP rephrases web documents into controlled styles and jointly trains on real and synthetic text. On C4, it reports equivalent models with 5x less data or 3x less compute, alongside improvements across Pile domains and zero-shot tasks.

  • Problem

    Noisy web data and scaling laws make pre-training increasingly dependent on scarce high-quality data, compute, and costly data-curation experimentation.

  • Method

    WRAP uses smaller off-the-shelf instruction-tuned models to rephrase web documents into varied styles, then combines synthetic rephrases with real text for pre-training.

  • Results

    5x lesser data or 3x lesser compute is sufficient to train equivalent models with WRAP on C4, with improvements measured across 13 zero-shot tasks and 21 Pile domains.

  • Takeaways & Limitations

    Synthetic rephrases can provide value beyond repeating real web data by adding higher-quality inputs and styles aligned with downstream domains.

  • Takeaways & Limitations

    Enforcing diversity in generated data remains challenging, and the impact of content diversity in paraphrase models remains for future work.

Abstract

from arXiv · show

Large language models are trained on massive scrapes of the web, which are often unstructured, noisy, and poorly phrased. Current scaling laws show that learning from such data requires an abundance of both compute and data, which grows with the size of the model being trained. This is infeasible both because of the large compute costs and duration associated with pre-training, and the impending scarcity of high-quality data on the web. In this work, we propose Web Rephrase Augmented Pre-training ($\textbf{WRAP}$) that uses an off-the-shelf instruction-tuned model prompted to paraphrase documents on the web in specific styles such as "like Wikipedia" or in "question-answer format" to jointly pre-train LLMs on real and synthetic rephrases. First, we show that using WRAP on the C4 dataset, which is naturally noisy, speeds up pre-training by $\sim3x$. At the same pre-training compute budget, it improves perplexity by more than 10% on average across different subsets of the Pile, and improves zero-shot question answer accuracy across 13 tasks by more than 2%. Second, we investigate the impact of the re-phrasing style on the performance of the model, offering insights into how the composition of the training data can impact the performance of LLMs in OOD settings. Our gains are attributed to the fact that re-phrased synthetic data has higher utility than just real data because it (i) incorporates style diversity that closely reflects downstream evaluation style, and (ii) has higher 'quality' than web-scraped data.

1 Introduction

WRAP addresses the compute, data, and curation challenges of pre-training on noisy web text by combining real documents with synthetic rephrases. Across C4 experiments, it enables more efficient training and improves generalization across diverse evaluation settings.

  • Open challenges: Synthetic data generation remains costly and can create knowledge bias, while its benefits are difficult to separate from deliberate topic selection.These concerns motivate rephrasing web documents rather than generating new knowledge-targeted content.
  • Motivation and approach: WRAP uses an off-the-shelf language model to rephrase web documents into different styles for joint pre-training on real and synthetic data.The approach targets ambiguity about what data to use, how to train with limited data, and how to improve computational efficiency.
  • Motivation and approach: Synthetic rephrasing uses smaller open-source models and preserves web information diversity without relying on the generator as a knowledge bank.The authors contrast this with prior synthetic-data approaches requiring GPT-3.5-scale generation and potentially introducing topic or factual biases.
  • Main findings: 5x lesser data or 3x lesser compute is sufficient to train equivalent models with synthetic data on C4 across 13 zero-shot tasks and 21 Pile domains.The experiments also report stronger performance than TinyLlama models trained on 3 trillion tokens across several zero-shot question-answering tasks.
  • Main findings: ∼50% lower perplexity on the Pile is reported for WRAP models, including a 350M model trained on 15% of C4 outperforming a 1.3B model trained on all C4.The paper additionally studies data leakage, synthetic styles, and combinations of synthetic data.

2 Related Work

Prior work studies scaling laws, heuristic dataset selection, and synthetic data, but evidence about effective pre-training data curation remains limited. Existing synthetic-data methods show promise while raising concerns about cost, bias, and generalization.

  • Scaling and dataset selection: Scaling laws link model size and required training data, motivating continued research into efficient data selection and use.The Chinchilla results specifically reported a linear relationship between model size and training data and identified earlier models as undertrained.
  • Scaling and dataset selection: Dataset selection has high practical impact but remains understudied, with prior systems relying on heuristics such as Reddit engagement or Wikipedia similarity.The passage characterizes these strategies as examples of document-quality filtering for pre-training.
  • Synthetic data: Synthetic stories and textbook-quality corpora have enabled smaller models to achieve strong performance on selected language, reasoning, and coding tasks.Related work also applies synthetic data to coding and mathematical reasoning during fine-tuning.
  • Synthetic data: Combining synthetic and real data has produced strong in-distribution and out-of-distribution results in vision and multimodal learning.These studies include synthetic image augmentations for domain generalization.

3 WRAP: Web Rephrase Augmented Pretraining

WRAP rephrases web-crawled documents into controlled styles and combines the resulting synthetic data with real text. The design targets higher-quality and style-diverse training inputs while retaining the information diversity of the web.

  • Method: WRAP addresses the cost and bias of synthetic-data curation by rephrasing web articles with smaller open-source models rather than generating new knowledge.The method uses rephrasing to retain natural web diversity and reduce dependence on a large model’s stored knowledge.
  • Rephrasing styles: Four styles—Easy, Medium, Hard, and Q/A—convert web documents into toddler-level, Wikipedia-like, terse, or question-answering text.An instruction-tuned model is prompted to operationalize these stylistic variations.
  • Synthetic generation: A frozen Mistral-7B instruction-tuned model generates parallel synthetic data from C4, with examples capped at 300 tokens to reduce information loss.The Medium prompt requests paraphrases in high-quality English like sentences on Wikipedia, and was refined using human feedback against GPT-4 outputs.
  • Combining data: Combining real and synthetic data preserves exposure to noisy web text while adding cleaner stylistic variation for language-model pre-training.The authors aim to prevent failures on user-facing text containing typos and linguistic errors.
  • Experimental setup: The experiments train decoder-only transformer models at 128M, 350M, and 1.3B parameter scales.The models use the Megatron-LM repository and are evaluated under the paper’s pre-training setup.
  • Experimental setup: WRAP with C4 and QA-85B outperforms C4-only training against models trained on twice as much real data in Pile perplexity.The comparison uses a 1.3B model trained for 300B tokens.

4 Perplexity Evaluation

Perplexity evaluation measures generalization from C4 and its stylistic rephrases to 21 Pile domains rather than performance on C4 alone. The reported results indicate faster learning and substantially lower perplexity with synthetic rephrases.

  • Evaluation protocol: Models are evaluated on validation subsets from 21 Pile domains to measure out-of-distribution perplexity after training on C4 or a stylistic rephrase.Each domain subset contains the first 10,000 documents from that Pile domain.
  • Evaluation protocol: WRAP optimizes risk over a distribution combining real C4 and synthetic rephrases, so its generalization should be assessed beyond C4-only text.The paper states that directly comparing the two objectives on C4 would be unfair because WRAP does not minimize C4-only risk.
  • Results: Nearly 3x lower perplexity is observed on domains such as ArXiv and HackerNews with synthetic data than with real data alone.The result suggests that additional real data may not offset the advantage of synthetic pre-training in many cases.
  • Results: At 10B tokens, WRAP’s average Pile perplexity is lower than C4 training achieves after 15 checkpoints, indicating a 15x pre-training speed-up.The authors defer comparison with zero-shot tasks for a more meaningful learning-speed analysis.

5 Zero-shot Tasks

The zero-shot evaluation covers general understanding and specialized knowledge across 13 benchmarks, comparing models trained with real, synthetic, and combined data. WRAP improves average general-task performance, while specialized-task results show that larger datasets help but gains can saturate.

  • Evaluation Scope: The evaluation uses 13 zero-shot benchmarks spanning common sense, language and knowledge understanding, mathematical reasoning, and specialized domains.The specialized category includes science, medicine, mathematics, and logic tasks.
  • General Understanding: WRAP models combining synthetic data with C4 average 49.4% across general understanding tasks, versus 47.4% for the 85B-token real-C4 split.These tasks cover general reasoning, language understanding, and common sense.
  • Specialized Knowledge: 44.6%: Pythia trained on 300B tokens outperforms the 43.5% average of the smaller 85B-token C4 dataset.This comparison supports a benefit from exposing models to larger datasets.
  • Specialized Knowledge: 0.2%: RefinedWeb trained on 320B tokens exceeds the 160B-token model by only 0.2%, indicating that improvements can saturate as datasets grow.TinyLlama trained on 1T tokens performs comparably to WRAP using 85B tokens of raw web data.
  • Specific Improvements: 44.0%: the Synthetic (85B) model achieves the highest reported TruthfulQA score, while adding real data lowers it to 40.5%.Other datasets, including HellaSwag and BoolQ, continue to show benefits from combining C4 with synthetic rephrases.

6 Analysis and Ablations

The ablations examine how real data, synthetic styles, rephraser quality, augmentation strategies, and semantic preservation affect WRAP performance. Results generally favor combining real C4 with appropriately styled synthetic data, while style combinations provide limited zero-shot gains.

  • 6.1 Data Combination Analysis: Synthetic Q/A data supports strong QA performance, but removing real C4 substantially worsens perplexity on several Pile domains.The degradation is attributed to synthetic data lacking special characters, tags, and styles present in real web scrapes.
  • 6.1 Data Combination Analysis: Combining multiple rephrasing styles yields only a small average perplexity improvement and no zero-shot benefit over Q/A style alone.Q/A and Wikipedia styles can help specialized domains such as Stackexchange, but the zero-shot results favor Q/A plus C4.
  • 6.2 Method Ablations: WRAP outperforms synonym replacement and random deletion, indicating its gains are not merely those of conventional text augmentation.The comparison trains 350M-parameter models for 15B tokens.
  • 6.2 Method Ablations: Matching synthetic-data style to the evaluation domain improves performance, but no single style is best across all Pile domains.An oracle selecting the best style per domain improves perplexity by 16%, supporting diverse training styles for generalization.
  • 6.2 Method Ablations: Synthetic data from a high-quality rephraser performs better than data from a low-quality rephraser such as fine-tuned T5-base.The comparison includes Qwen-1.8B-chat, Mistral-7B-chat, Vicuna-13B-chat, and T5-base rephrasers.
  • 6.2 Method Ablations: Real-synthetic pairs have higher embedding similarity than several baselines, indicating that rephrasing changes style while largely preserving meaning.The analysis uses cosine similarity from a SimCSE-trained BERT model over 1,000 values after removing outliers.

7 Limitations and Opportunities

The paper considers whether synthetic data is preferable to training longer on real data, including the costs of generation and training. It also identifies content-diversity control as an unresolved limitation of rephrasing-based data generation.

  • 7.1 Cost Analysis: Training longer on real data may have limited improvement potential for 350M- and 1.3B-parameter models compared with jointly training on real and synthetic data.A TinyLlama model trained for 3 trillion tokens underperforms a jointly trained model and is comparable to models trained on 300B real-data tokens.
  • 7.1 Cost Analysis: Generating 85B synthetic tokens costs about 25K GPU hours, while training for 300B tokens costs about 6K GPU hours on 64 A100s.Training a 13B model would cost about 30K GPU hours, making 3–10x training-cost reductions relevant to synthetic-data overhead.
  • 7.1 Cost Analysis: Using Qwen-1.8B for rephrasing increases token throughput 3x and reduces generation cost by 3x relative to the reported Mistral setup.Preliminary results indicate Qwen-generated rephrases perform comparably to Mistral-generated rephrases.
  • 7.1 Cost Analysis: Synthetic-data generation is a one-time, parallelizable investment that can support multiple model scales and run on individual GPUs.The authors contrast this with training, which requires a large cluster with fast inter-node connectivity.
  • 7.2 Diversity of Synthetic Generations: Enforcing diversity in generated data remains an open limitation because instruction-tuned models may reduce content diversity.The paper leaves the presence and impact of content diversity in paraphrase models for future work.

8 Conclusion

The paper presents synthetic rephrases as a way to improve pre-training efficiency and generalization, while highlighting generation cost and diversity as ongoing limitations. It examines how synthetic style and the balance between real and synthetic data affect model training.

  • Synthetic rephrases can provide more value than repeating existing data when high-quality data are scarce.
  • Synthetic data may improve generalization across text domains and support styles underrepresented in the pre-training corpus.
  • The paper studies generator quality, real–synthetic data balance, and diminishing returns from repeated synthetic-data training.
  • Synthetic-data generation remains costly because it requires strong language models, while maintaining diversity is challenging.
  • WRAP uses the web’s natural diversity for rephrasing, limiting new knowledge acquisition while improving training efficiency through higher-quality inputs.
  • The experiments use C4, Refined Web, and 20 Pile subsets to compare synthetic rephrases with web data and curated alternatives.

A.3 Zero-shot Evaluation Dataset

The zero-shot evaluation covers 13 benchmarks spanning specialized knowledge and general understanding. The benchmark descriptions target domain expertise, reasoning, comprehension, and common-sense abilities.

  • The evaluation uses 13 zero-shot benchmarks divided into Specialized Knowledge and General Understanding subsets.
  • Specialized Knowledge: Specialized Knowledge benchmarks assess domain-specific expertise in areas including science, medicine, mathematics, and professional subjects.
  • General Understanding: General Understanding benchmarks test cognitive skills, language comprehension, and common-sense reasoning.
  • The benchmark suite includes scenario completion, truthful answering, broad factual reasoning, and logical reasoning tasks.
  • The datasets are selected to challenge distinct aspects of natural-language processing models.
  • Synthetic-output filtering removes unwanted introductory text when it appears before delimiters in the first sentence.

B.1 Methodology

The methodology evaluates whether synthetic rephrases improve pre-training through data-quality and stylistic properties rather than information leakage. It compares semantic similarity, reading level, diversity, and syntactic complexity across synthetic and real corpora.

  • The filtering procedure splits generated text into sentences, checks the first sentence for delimiters, and removes unwanted preceding segments.
  • The filtering modification leaves fewer than 0.1% of sentences containing unwanted elements based on manual inspection.
  • The analysis compares semantic similarity, syntactic complexity, and diversity between synthetic data, C4, and Pile subsets.
  • Synthetic and real rephrases are compared with BERT-based similarity measures to assess whether performance gains reflect knowledge leakage.
  • Medium-style rephrases raise reading level from C4’s 7–8 range to 10, while QA rephrases reduce it to 6.
  • Medium-style synthetic data has diversity similar to most Pile subsets, whereas QA data has particularly low type-token ratio.
  • Medium-style rephrases increase dependency-tree depth, mean dependency distance, and syntactic complexity.

D Evaluation Metrics

The evaluation uses macro token-level perplexity, computed from cumulative dataset loss and the total number of tokens. An upper limit stabilizes the metric for high-loss cases.

  • Macro token-level perplexity is computed from accumulated loss over a dataset and its total token count.
  • The metric applies the exponential function to the cumulative loss divided by the total number of tokens.
  • The value 20 caps the metric to stabilize evaluation when loss values are high.

E.1 Results for 350M Models Trained for 75B Tokens

At 350M parameters and 75B training tokens, WRAP combines real C4 text with synthetic rephrases to improve perplexity and downstream task performance. QA rephrases provide additional gains, while medium rephrases improve over training on C4 alone.

  • The 350M models were trained for a total of 75B tokens across the evaluation setup.
  • 1.5% improvement on general understanding tasks and roughly 3% on specialized knowledge tasks resulted from adding QA rephrases.These results were reported in Tables 8–9 for the 350M setting.
  • Medium rephrases improved performance over C4-only training, while matching the QA style produced further gains.

F LLM Leaderboard Few-shot Results

The few-shot evaluation tests whether WRAP's gains persist when models receive test examples. WRAP models substantially improve ARC-Challenge, remain consistent with zero-shot TruthfulQA results, and perform comparably across other datasets.

  • Few-shot evaluation examined whether synthetic-rephrase gains persist when models have access to test samples.The study evaluated six tasks from the OpenLLMLeaderboard2.
  • Substantial improvements were observed on ARC-Challenge, with TruthfulQA results consistent with the zero-shot setting.
  • Performance was comparable across the other evaluated datasets.
  • The evaluated models were trained for 300B and 350B tokens, corresponding to roughly 85B and 100B unique C4 tokens.
  • The WRAP models outperformed the publicly released Falcon-1.3B and Pythia-1.4B models in the reported comparisons.
Loading 2401.16380v1…