PaperlayerView exact revision on arXiv

Source-linked AI summary

When Less is More: Investigating Data Pruning for Pretraining LLMs at Scale

Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, Sara Hooker

arXiv:2309.04564v1cs.CLcs.LG

TL;DR

No established best practices measure the quality of individual examples in noisy web-scraped pretraining corpora. The paper evaluates perplexity, EL2N, and memorization for large-scale data pruning, finding that perplexity can outperform the more complex metrics and improve test-set perplexity with less data.

  • Problem

    Rule-based filters remove some noisy examples, but there are no established best practices for measuring individual training-example quality in web-scraped pretraining data.

    Source: §1 · pp. 1–2

  • Method

    The study evaluates perplexity, EL2N, and memorization as model-output-based metrics for ranking and pruning pretraining data at scale.

    Source: §1 · p. 2

  • Results

    Perplexity pruning improves test-set perplexity by 1% using half the dataset and by 1.5% at 1.5B parameters over the stated full-data and same-size no-pruning baselines, respectively.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Perplexity-based pruning can retain or improve performance with substantially less pretraining data, including models trained on half the dataset.

    Source: §6 · p. 14

Abstract

from arXiv · show

Large volumes of text data have contributed significantly to the development of large language models (LLMs) in recent years. This data is typically acquired by scraping the internet, leading to pretraining datasets comprised of noisy web text. To date, efforts to prune these datasets down to a higher quality subset have relied on hand-crafted heuristics encoded as rule-based filters. In this work, we take a wider view and explore scalable estimates of data quality that can be used to systematically measure the quality of pretraining data. We perform a rigorous comparison at scale of the simple data quality estimator of perplexity, as well as more sophisticated and computationally intensive estimates of the Error L2-Norm and memorization. These metrics are used to rank and prune pretraining corpora, and we subsequently compare LLMs trained on these pruned datasets. Surprisingly, we find that the simple technique of perplexity outperforms our more computationally expensive scoring methods. We improve over our no-pruning baseline while training on as little as 30% of the original training dataset. Our work sets the foundation for unexplored strategies in automatically curating high quality corpora and suggests the majority of pretraining data can be removed while retaining performance.

1 Introduction

Web-scale pretraining data contains noisy text, while rule-based filters lack established practices for measuring the quality of individual examples. This work evaluates automatic pruning metrics and finds that perplexity-based pruning can outperform more complex alternatives and improve over full-data training.

1 Introduction

  • Rule-based filters remove some noisy examples, but there are no established best practices for measuring the quality of individual training examples.Examples include removing repetitive or special-character-heavy documents, blocklisted websites, and documents outside length thresholds.Source: §1 · pp. 1–2
  • The paper asks whether removing least-impactful pretraining examples can preserve or improve performance and whether simpler quality estimates outperform more costly ones.Prior pruning work focused mainly on supervised vision, with fewer language studies centered on fine-tuning or hand-picked corpora.Source: §1 · p. 2
  • The study rigorously evaluates perplexity, EL2N, and memorization as automatic pruning metrics that rely on model outputs rather than a preselected high-quality dataset.The evaluation spans models from 124M to 1.5B parameters and different pretrained corpora.Source: §1 · pp. 2–3
  • The framework compares keeping 10%, 30%, 50%, or 70% of data from the bottom, middle, or top of pruning-score distributions.It also examines reference-model and training variations and evaluates downstream generalization on six GLUE tasks.Source: §1 · p. 3

2 Methodology

The paper frames pruning as selecting training instances by a score while seeking to preserve model performance. It compares perplexity, EL2N, and memorization, alongside random pruning, using contrasting subsets of the score distributions.

2 Methodology

  • Pruning selects scored training instances to remove while aiming to preserve performance; reference and trained models share a context length for consistent scoring contexts.The study uses static pruning, applied once before training, and evaluates keeping top, middle, or bottom score percentiles.Source: §2 · pp. 3–4

2.1 Pruning Methods

  • The study benchmarks simple, inexpensive ranking approaches such as perplexity against more computationally expensive methods such as EL2N and memorization.It treats the metrics as alternative ways to determine which instances to prune.Source: §2 · p. 3, §2.1 · p. 4

2.1.1 Selection via Perplexity

  • Perplexity estimates how probable each sequence is under a language model, with lower scores indicating that the model assigns higher probability to the text.The token negative log likelihoods are used to compute the sequence-level metric.Source: §2.1.1 · p. 4

2.1.2 Selection via EL2N

  • EL2N measures a text sequence’s average L2 error between the reference model’s predicted token distribution and the ground-truth one-hot representation.The study tests scores from reference models at 14% and 55% of training, then averages scores from ten initializations.Source: §2.1.2 · pp. 4–5

2.1.3 Memorization Ranking

  • Memorization ranks examples by the fraction of generated tokens that exactly match the original sequence; experiments use M = N = 32.A reference model receives the first M tokens and greedily generates N more; the authors hypothesize this ranking may identify examples needing additional learning.Source: §2.1.3 · p. 5

2.1.4 Random Pruning

  • Randomly pruning samples provides a baseline for testing whether proposed pruning methods outperform a random selection.The paper describes random pruning as a lower bound of expected performance.Source: §2.1.4 · p. 5

3 Experiments

Experiments train GPT-style decoder-only Transformers on prefiltered CommonCrawl and Wikipedia data, varying pruning choices and reference models. Evaluation uses held-out CommonCrawl perplexity and GLUE fine-tuning.

3.2 Data

  • The experiments use prefiltered May 2022 CommonCrawl data downsampled to 7.6B tokens and an English-only Wikipedia dataset of 5.3M tokens.CommonCrawl filters remove repetitive documents, documents with special-character percentages, and documents containing explicit words and toxic text.Source: §3.2 · p. 6

3.3 Ablations

  • Pruning experiments retain 10%, 30%, 50%, or 70% of data from the top, middle, or bottom of score distributions, with separate models trained for each variant.Ablations vary reference-model size, training data, and training steps; the study trains at least nine 124M models per experimental variant.Source: §3.3 · pp. 6–7

3.4 Evaluation

  • Evaluation measures perplexity on a 266M-token CommonCrawl test set and fine-tunes selected models on six GLUE classification tasks.GLUE task data are not pruned; comparisons are made after 8,000 fine-tuning steps, approximately 4.4 pretraining-dataset epochs.Source: §3.4 · p. 7

4 Results and Discussion

Across the tested pruning methods, perplexity-based pruning performs best and can surpass no pruning with less data. Results also show how subset choice and reference-model quality, size, and training stage affect pruning, with benefits extending to larger models and downstream tasks.

4.1 Removing Easy Instances Improves Performance

  • For perplexity, middle subsets perform best, and retaining 30% or 50% can outperform 70%; EL2N also favors the middle, while memorization favors the least-memorized bottom subset but never beats no pruning.The middle EL2N subset at 50% outperforms 70%, but its best run does not beat the baseline.Source: §4.1 · pp. 7–8

4.2 Simple Pruning Metrics Outperform More Sophisticated Approaches

  • Perplexity-based pruning of the middle-scoring subset outperforms the best EL2N and memorization variants across dataset sizes, and beats no pruning at 30% and 50% of the data.The middle 50% improves perplexity by 0.97% over no pruning; the middle 30% improves it by 0.80%.Source: Figure 4 · p. 9, §4.2 · p. 9

4.4 Improved Pruning Signals Result from Reference Models Trained on Cleaner Data

  • For the middle and top subsets, a Wikipedia-trained reference model yields lower validation perplexity than a CommonCrawl-trained one; its best middle-70% variant leads by 0.69%.The authors suggest that reference models trained on higher-quality data provide more effective pruning rankings.Source: §4.4 · p. 11

4.5 Early Reference Model Checkpoints Serve as Effective Scoring Models

  • A reference model trained on 55% of the data performs similarly to a fully trained model, whereas the 14% checkpoint provides inadequate pruning scores.The 55% checkpoint performs best with the middle subset, worst with the bottom, and comparably with the top; averaging EL2N scores across reference models makes the subset pattern more like that of larger, fully trained models.Source: §4.5 · pp. 11–12

4.6 Perplexity-based Pruning Improvements Generalize to Larger Scale Models

  • Perplexity-based pruning with a 52B reference model outperforms random pruning at both 124M and 1.5B model scales.The authors report that the improvement over random pruning follows a consistent pattern across the two scales.Source: Figure 7 · p. 12, §4.6 · p. 12

4.7 Downstream Evaluation on GLUE

  • Pretraining-data pruning improves performance across the evaluated GLUE tasks, but no single pruning strategy is best across all tasks.The least-memorized 30% performs best for SST2 and WNLI; perplexity pruning performs best for QQP at 50% and QNLI at 70%.Source: §4.7 · p. 12, Table 2 · p. 13

Related papers

Loading 2309.04564v1…