PaperlayerView exact revision on arXiv

Source-linked AI summary

A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity

Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, Daphne Ippolito

arXiv:2305.13169v2cs.CLcs.LG

TL;DR

Pretraining data curation is under-documented and often guided by unsupported intuitions. Using 28 1.5B-parameter models, the paper measures how data age, filtering, and domain composition affect downstream behavior, finding significant trade-offs that finetuning cannot eliminate and no universal filtering strategy.

  • Problem

    Pretraining data design is rarely documented with evidence explaining why curation protocols were chosen or what effects they have.

  • Method

    The authors train 28 1.5B-parameter models with varied corpus ages, quality and toxicity filters, and source-domain compositions, then evaluate downstream tasks after finetuning.

  • Results

    Data age, filters, and source domains significantly affect downstream behavior, producing specific trade-offs that finetuning reduces but does not eliminate; filtering effects are not one-size-fits-all.

  • Takeaways & Limitations

    Dataset curation policies should be treated as model-design hyperparameters, with choices guided by their task-specific trade-off profiles rather than generic assumptions.

  • Takeaways & Limitations

    Toxicity evaluation relies on a potentially irreproducible black-box API and a narrow setting, limiting implications for toxic generation in real applications.

Abstract

from arXiv · show

Pretraining is the preliminary and fundamental step in developing capable language models (LM). Despite this, pretraining data design is critically under-documented and often guided by empirically unsupported intuitions. To address this, we pretrain 28 1.5B parameter decoder-only models, training on data curated (1) at different times, (2) with varying toxicity and quality filters, and (3) with different domain compositions. First, we quantify the effect of pretraining data age. A temporal shift between evaluation data and pretraining data leads to performance degradation, which is not overcome by finetuning. Second, we explore the effect of quality and toxicity filters, showing a trade-off between performance on standard benchmarks and risk of toxic generations. Our findings indicate there does not exist a one-size-fits-all solution to filtering training data. We also find that the effects of different types of filtering are not predictable from text domain characteristics. Lastly, we empirically validate that the inclusion of heterogeneous data sources, like books and web, is broadly beneficial and warrants greater prioritization. These findings constitute the largest set of experiments to validate, quantify, and expose many undocumented intuitions about text pretraining, which we hope will help support more informed data-centric decisions in LM development.

1 Introduction

The paper examines under-documented pretraining data decisions across collection time, filtering, and domain composition using systematic model experiments. Its findings show that these choices materially affect performance, toxicity, and model comparisons, motivating more empirically informed curation.

  • Motivation and contribution: The study addresses documentation debt by empirically quantifying how dataset curation protocols affect language-model behavior.Its experiments span 28 models and are presented as a large publicly documented curation study intended to inform future model development.
  • Age of a Dataset: Temporal shifts between evaluation data and pretraining data degrade performance, and substantial finetuning does not overcome the deficit.The effect is exacerbated in larger models and can complicate comparisons between newer and older models depending on evaluation-data age.
  • Quality and Toxicity Filters: Quality filtering increases toxic generation and downstream performance, whereas removing toxic data reduces toxic generations but lowers generalization performance.Inverse toxicity filters provide targeted benefits, while quality-filtering effects are not predictable from text characteristics.
  • Domain Compositions: Books and heterogeneous web data are among the best-performing domains but also contribute most to toxic generation.The authors recommend prioritizing more books and diverse web data while generously including sources less directly relevant to downstream tasks.

2 Methodology

The methodology varies pretraining datasets by age, domain composition, and content filtering, then evaluates the resulting models on downstream tasks. It combines C4 and the Pile with controlled interventions and finetuned evaluations covering general utility, temporal alignment, domain generalization, and toxic behavior.

  • Experimental pipeline: Each experiment filters a pretraining dataset, trains a language model on the curated data, and evaluates it on downstream tasks.This pipeline is used to measure how curation choices affect downstream performance.
  • Pretraining datasets: The study begins with C4 and the Pile, deduplicating both datasets before applying controlled curation interventions.C4 is a 2019 Common Crawl snapshot, while the Pile is an 800GB collection spanning 22 sources.
  • Dataset interventions: The interventions vary dataset age, selectively remove Pile domains, and apply toxicity or quality filters at document level.Multiple Common Crawl years are used for C4, while domain removal is used to control topical composition.
  • Content filtering: Quality and toxicity are operationalized with classifier-based filters rather than human ground truth labels.Quality identifies text resembling known high-quality sources, while toxicity covers profane, explicit, insulting, or threatening content as detected by automated classifiers.
  • Content filtering: Toxicity filtering removes documents above thresholds 0.95, 0.9, 0.7, 0.5, or 0.3, alongside an inverse filter removing the least predicted-toxic documents.The toxicity score ranges from 0 to 1, and threshold choice depends on practitioner goals.
  • Evaluation: Finetuned models are evaluated on toxicity identification, toxic generation, diverse question-answering tasks, and temporally annotated tasks.QA evaluation uses MRQA and UnifiedQA datasets spanning multiple domains, while temporal evaluation tests pretraining–evaluation time alignment.

3 Impact of Data Curation on Data Characteristics

Curation choices substantially alter measurable dataset characteristics, and these effects interact across domains, quality, toxicity, and collection time. Simple quality labels do not cleanly predict other content properties or filtering effects.

  • Domain characteristics: Books combine high predicted quality with unusually long, readable, toxic, profane, and PII-filled documents.Books are described as an outlier domain whose quality and potentially harmful-content indicators are simultaneously high.
  • Curation interactions: The observed feature patterns reveal substantial interactions among curation choices rather than independent effects.The analysis measures toxicity, quality, PII, readability, lexical diversity, and sentiment across datasets and slices.
  • Dataset comparisons: The Pile contains documents that are longer, more readable, higher quality, and richer in non-ASCII characters than C4, but also more PII.Relative to C4, Pile documents are 2.4x longer, 1.9x richer in non-ASCII characters, 1.2x higher quality, and 1.8x more readable.
  • Quality and toxicity: Toxicity and quality are not well aligned: high-toxicity documents can have higher text quality than low-toxicity documents.Quality-based classifications also show little difference in profanity, toxicity, and sexually explicit content.
  • Domain characteristics: Technical domains such as PubMed, Code, and Academic receive low predicted-quality scores despite potentially useful specialized content.This indicates that positively defined web-quality filters may remove specialized text.
  • Collection time: More recent web-scraped text becomes more diverse and less toxic while measured quality declines.Across C4 collection years, non-ASCII characters increase, toxicity decreases slightly, and sentiment increases.

4 Impact of Dataset Age on Pretrained Models

Pretraining data that is temporally misaligned with evaluation data is associated with lower performance, whether it is older or newer than the evaluation data. This degradation persists after substantial finetuning and is stronger for larger models.

  • Asymmetry and evaluation: Temporal misalignment complicates comparisons across models and evaluations produced at different times.Newer evaluations may under-estimate older models, while older evaluations may under-estimate newer models.
  • Temporal degradation: Temporal misalignment between pretraining and evaluation data causes performance degradation that finetuning does not overcome.Models were evaluated across time-split tasks after finetuning on temporally relevant training data.
  • Performance relationship: Mean relative performance increases as temporal misalignment approaches zero across five datasets.Figure 4 summarizes the relationship using medians, means, quartiles, and whiskers, with different evaluation-year ranges by dataset.
  • Temporal degradation: 0.4 average temporal degradation occurs for one year of pretraining misalignment, with Pearson correlation r = 0.61 between misalignment and performance degradation.The five tasks all pass a one-sided Wald test with p < 0.05 for a positive slope.
  • Asymmetry and evaluation: Performance degradation occurs when pretraining data is collected either before or after evaluation data, but is steeper when evaluation follows pretraining.The asymmetry implies that both models and evaluation datasets can become stale.
  • Model scale: Temporal degradation is greater for LM-XL models with 1.5B parameters than for LM-Small models with 20M parameters.The reported effect was not significant for LM-Small models, suggesting greater sensitivity to temporal information in larger models.

5 Impact of Quality & Toxicity Filters on Pretrained Models

Quality and toxicity filters produce opposing and task-dependent effects, so filtering should reflect the intended model behavior rather than a universal rule. Quality filtering generally improves downstream performance despite removing data, whereas toxicity filtering reduces toxic generation at a generalization cost.

  • Practical implication: The intended model behavior should determine the filtering strategy because no single toxicity filter serves all objectives.The paper explicitly characterizes toxicity filtering as a trade-off between toxicity identification and toxic generation.
  • Quality filtering: Quality filtering improves performance across nearly all tested tasks despite reducing training data quantity and variety.It improves toxicity identification by 2% and most QA categories by 1–6%, even after removing 10%+ of training data.
  • Filter trade-offs: Quality filtering increases both toxic identification and toxic generation, while toxicity filtering decreases both abilities.Figure 5 varies score thresholds and removes documents below each threshold.
  • Quality filtering: Quality-filtering gains are not predictable from domain quality labels: academic and biomedical QA benefit most, while Books QA is hurt.Books, Wikipedia, and Web are classified as highest quality, whereas academic and biomedical data rank among the lowest.
  • Filter trade-offs: A one-dimensional quality score does not explain filtering behavior across domains or toxic-generation tendencies.Both quality and inverse-quality filters increase toxic-generation tendencies, while Wikipedia and Web QA respond differently to inverse filtering.
  • Toxicity filtering: Toxicity filtering trades reduced toxic generation against poorer toxicity identification and lower performance on unrelated QA tasks.The observed QA decline may partly reflect the overall decrease in training data.
  • Toxicity filtering: The inverse toxicity filter produces the strongest toxicity-identification performance across every tested dataset.Practitioners optimizing toxicity-identification tasks are advised to remove the least toxic content.

6 Impact of Domain Composition on Pretrained Models

Domain composition substantially affects downstream QA performance: heterogeneous web and book data provide the strongest benefits, while targeted domains help aligned evaluations but are often less beneficial than diverse web data.

  • Overall composition: Data source heterogeneity is more important than data quality or size for downstream performance.Books and OpenWeb are smaller than Common Crawl but provide more heterogeneous and predicted-quality content.
  • Domain-level effects: Common Crawl, OpenWeb, and Books have the strongest positive effects on downstream performance.Removing web-based domains produces the largest average degradation, especially on challenging Common Sense and Contrast Sets tasks.
  • Heterogeneity versus targeting: Domain heterogeneity is often more beneficial than targeted data, even for targeted evaluations.Aligned domains still matter: PubMed supports BioMed QA, Wikipedia supports Wikipedia benchmarks, and web content supports web evaluations.
  • Overall composition: The best performing models use all, or nearly all, pretraining data sources.The main exceptions are removing Code or Academic domains, which may be poorly matched to the QA evaluations.
  • Toxicity trade-offs: Removing Common Crawl, OpenWeb, and Books causes the largest decreases in toxicity identification and generation metrics.The figure caption reports that removing Wikipedia instead strongly increases toxicity generation.

7 Discussion

The discussion argues that pretraining curation choices have durable effects on model behavior and should be treated as important design parameters. It recommends better modeling of data–capability relationships while highlighting trade-offs involving data age, filtering, and source diversity.

  • Curation as a design choice: Pretraining curation choices significantly affect models and cannot be easily erased by subsequent finetuning.The paper therefore frames dataset curation policies as hyperparameters and calls for better tools to model data–capability relationships.
  • Age of the pretraining corpus: Temporal mismatch between pretraining and evaluation data causes performance degradation that persists despite plentiful finetuning data.The effect is worse for larger models and can complicate comparisons across evaluation datasets of different ages.
  • Data source composition: Diverse web and book data yield the largest benefits, but assembling heterogeneous sources is more complex and costly and may expose models to less careful curation.The paper’s experiments nevertheless suggest retaining broad source coverage when generalization across many text-to-text tasks is desired.
  • Filtering for toxicity and quality: Filtering creates a trade-off between generalization ability and toxic generation, with different filters producing opposite behavioral effects.Quality filtering improves downstream performance while increasing toxic generation; removing toxic data reduces toxic generations but also reduces generalization performance.
  • Filtering for toxicity and quality: Quality filtering significantly improves performance across domains despite removing large portions of training data.Observational domain-quality characteristics are insufficient to predict which domains benefit most from filtering.

8 Limitations

The study’s scope is constrained by costly, single-shot pretraining experiments, English-only data, and evaluation focused on finetuned rather than prompted settings.

  • Compute Expense & Single Shot Experiments: 28 1.5B-parameter pretraining ablations were computationally and environmentally costly, limiting the experiment set and preventing repeated rounds of reflection and replication.The authors selected experiments on corpus age, quality filters, toxicity filters, and source domains in advance.
  • Blackbox APIs: The analysis used Perspective’s toxicity API, whose implementation may shift over time and whose narrow evaluation setting may not reflect toxic generation in real applications.The authors describe these API-based measures as the best available proxies for now.
  • English vs Multilingual Data: The analysis was limited to two English datasets, leaving multilingual and non-English training-composition questions unexamined.The authors note that balancing languages and finding sufficiently large high-quality corpora can be challenging.
  • Relevance to Zero- & Few-Shot Prompted Settings: The experiments focus on finetuned settings, so they cannot establish how well the findings translate to zero- or few-shot prompting.The authors suspect the findings are strongly correlated across settings but do not establish this experimentally.

9 Related Work

Related work frames pretraining data curation as a growing but unresolved concern involving data composition, filtering, toxicity, and temporal change. Prior studies report mixed findings, motivating systematic empirical comparison.

  • Data & Domains: Generative-model development has repeatedly mixed noisy web data with smaller perceived high-quality corpora, beginning with GPT-2’s WebText response to noisy Common Crawl.This history provides context for comparing heterogeneous web and curated sources.
  • Pretraining Dataset Analysis: Prior research presents conflicting recommendations about filtering, including warnings that filtering can remove low-quality or underrepresented-community text.Other work recommends more robust filtering to reduce harmful content, illustrating the unresolved trade-off.
  • Data, Toxicity, & Quality: Existing language models commonly use quality and toxicity preprocessing, although the benefits of these practices are often implicit rather than externally reported.Academic findings on detoxification techniques remain mixed, including effects on toxicity and perplexity for underrepresented communities.
  • Data, Toxicity, & Quality: Prior work also reduces unwanted toxic generation through instruction tuning and alignment tuning, beyond dataset-level filtering.These approaches form part of the broader literature on controlling harmful generation.
  • Data & Time: Research on temporal distribution shift shows that models can degrade on newer test sets as language, events, and usage patterns change.This motivates examining the relative age of pretraining, finetuning, and evaluation data.
  • Data & Domains: Public pretraining datasets are shaped largely by licensing constraints, while prior work warns that high-quality web text may be nearing exhaustion.This challenge matters because high-quality and diverse training data support strong generalization.
  • Data & Domains: Prior studies investigate domain rebalancing and importance sampling to improve convergence or downstream performance, complementing comparisons of pretraining compositions.One cited study reports a 2% improvement on downstream tasks from importance sampling subsets of the Pile.
  • Model & Data Scaling: Scaling research has primarily examined model size, data quantity, optimization steps, and compute-optimal relationships, while this work centers data design choices.The related work also connects scale with the emergence of new model abilities.

10 Conclusion

The paper concludes that dataset age, filtering, and source domains materially affect downstream behavior, each with a distinct trade-off profile. It presents a broad empirical basis for more informed curation decisions while documenting the study’s experimental scope and evaluation procedures.

  • 10 Conclusion: Relative document age, content filters, and data sources each significantly affect downstream model behavior, with effects reduced but not eliminated by finetuning.The authors recommend matching model and data choices to application needs because each decision has a quantifiable trade-off profile.
  • 10 Conclusion: The evaluation covers toxicity identification, toxic generation, diverse question-answering tasks, and temporally annotated tasks to compare general utility and data-sensitive behavior.The reported evaluation framework combines finetuned downstream tasks with unfinetuned toxic-generation evaluation.
  • 10 Conclusion: Toxicity identification is evaluated by finetuning on Social Bias Frames, DynaHate, and Toxigen and measuring AUC-ROC on their respective test sets.Toxic generation instead uses unfinetuned models and prompt-based continuation evaluation.

C.4 Time Evaluation Details

The time-evaluation procedure measures temporal degradation across pretraining, finetuning, and evaluation years while controlling comparisons for finetuning and evaluation years. Domain experiments separately construct and ablate thematic Pile partitions to map pretraining sources to downstream QA domains.

  • Time Evaluation: Temporal experiments simulate mismatches between training and test years using datasets segmented by collection year.Each task is finetuned separately before evaluation on its corresponding test set.
  • Time Evaluation: Temporal degradation is defined as the average yearly performance deterioration, with Pearson correlation measuring the relationship between time differences and performance decline.The study also replicates a Wald test whose null hypothesis is a zero slope.
  • Time Evaluation: For pretraining temporal degradation, the modified measure compares models with different pretraining years while holding finetuning and evaluation years fixed.The comparison isolates relative changes associated with the pretraining year.
  • Time Evaluation: When no evaluation year matches a pretraining year, the procedure substitutes the term associated with the closest pretraining and evaluation years.The precise term depends on which finetuning-year version is being calculated.
  • Domain Evaluation: Domain experiments pretrain models on different Pile subsets and measure effects across downstream question-answering domains.This design maps the influence of pretraining domains to evaluation domains.
  • Domain Construction: The Pile’s sources are manually partitioned into thematically similar domains, although the subjective categories cannot perfectly separate overlapping content.Wikipedia, Books, and Common Crawl may all contain academic material despite the intended partitions.
  • Domain Evaluation: Each domain is separately ablated, and Natural Questions is used to teach the general question-answering task before evaluating across QA datasets.The design uses QA because many similarly formatted evaluation datasets are available.

D Impact of Data Curation on Data Composition: Further Analysis

The analysis measures how curation-related text features vary across datasets, temporal snapshots, and filtering thresholds. It examines toxicity, quality, readability, lexical, formatting, length, PII, and temporal characteristics.

  • Feature definitions: The feature analysis covers toxicity, profanity, sexually explicit content, text quality, PII, readability, lexical diversity, sentiment, and document-format characteristics.The measured format characteristics include word length, document length, non-ASCII characters, and all-caps words.
  • Quality analysis: Text quality is estimated with a bag-of-words linear classifier distinguishing text resembling Wikipedia and books from other text.The classifier is also applied to the Pile to examine which datapoints pass different quality thresholds.
  • Feature definitions: PII detection targets names, phone numbers, addresses, and emails, while readability uses Flesch–Kincaid grade levels based on sentence length and syllables per word.These classifiers and readability scores characterize distribution shifts across curation ablations.
  • Temporal analysis: Temporal information is estimated by counting dates from 2000 to 2025, revealing many mentions near collection years and a decline about five years earlier.The authors note that this estimate is limited because article publication and collection dates need not coincide.
  • Dataset composition: The study examines how quality-filtering thresholds alter the Pile’s domain composition, alongside corresponding feature differences between C4 and the Pile.The domain breakdown after multiple quality cutoffs is shown in Figure 11, while Figure 9 compares dataset features and temporal snapshots.

E.1 Temporal Degradation Results

The temporal experiments corroborate prior finetuning-misalignment findings and report full results across several temporally split tasks. The accompanying toxicity and quality analyses show distinct effects of filtering on toxicity identification and generation.

  • Temporal degradation: The finetuning-misalignment replication corroborates Luu et al. (2021), while the paper separately evaluates temporal degradation from pretraining misalignment.Figure 12 concerns finetuning misalignment; Figure 3 concerns pretraining misalignment.
  • Evaluation tasks: The temporal results span PubCLS, NewSum, TwiERC, AIC, and PoliAff, using Accuracy or Rouge-L as specified for each task.The tasks cover news-source classification, summarization, Twitter entity classification, scientific-article classification, and political-affiliation classification.
  • Toxicity filtering: Toxicity filtering decreases LM-XL’s ability to identify toxicity and generate toxic text.The results are reported in Table 14 and visualized in Figures 5 and 13.
  • Quality filtering: Quality filtering decreases LM-XL’s ability to identify toxicity but surprisingly increases toxic-text generation.These results are detailed in Table 15 and visualized in Figure 5.
Loading 2305.13169v2…