PaperlayerView exact revision on arXiv

Source-linked AI summary

The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only

Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, Julien Launay

arXiv:2306.01116v1cs.CLcs.AI

TL;DR

Large language models require trillions of training tokens, but curated corpora are labor-intensive and may not scale to these demands. The paper develops Macrodata Refinement to filter and deduplicate CommonCrawl into the five-trillion-token, web-only RefinedWeb dataset. RefinedWeb produces models competitive with the state of the art, including models that outperform systems trained on curated corpora, while the paper notes unresolved toxicity and repeated-epoch questions.

  • Problem

    Scaling large language models requires trillions of tokens, but the scalability of curated high-quality data is uncertain because curation is labor-intensive and data availability may become limiting.

  • Method

    The paper applies Macrodata Refinement, combining stringent filtering and deduplication of CommonCrawl to construct the five-trillion-token, English-only RefinedWeb dataset.

  • Results

    RefinedWeb produces models competitive with the state of the art and can outperform models trained on curated corpora, including public models trained on The Pile at equivalent compute budgets.

  • Takeaways & Limitations

    Properly filtered and deduplicated web data alone can provide a large-scale alternative to curated pretraining corpora, with a 600-billion-token extract publicly released.

  • Takeaways & Limitations

    The toxicity analysis does not cover social biases or harmfulness, and the authors call for further quantitative study of the public RefinedWeb extract.

Abstract

from arXiv · show

Large language models are commonly trained on a mixture of filtered web data and curated high-quality corpora, such as social media conversations, books, or technical papers. This curation process is believed to be necessary to produce performant models with broad zero-shot generalization abilities. However, as larger models requiring pretraining on trillions of tokens are considered, it is unclear how scalable is curation and whether we will run out of unique high-quality data soon. At variance with previous beliefs, we show that properly filtered and deduplicated web data alone can lead to powerful models; even significantly outperforming models from the state-of-the-art trained on The Pile. Despite extensive filtering, the high-quality data we extract from the web is still plentiful, and we are able to obtain five trillion tokens from CommonCrawl. We publicly release an extract of 600 billion tokens from our RefinedWeb dataset, and 1.3/7.5B parameters language models trained on it.

1. Introduction

Scaling language models requires rapidly increasing both model and dataset sizes, but high-quality text may become a data bottleneck. The paper therefore investigates whether carefully processed web data can reduce reliance on labor-intensive curated corpora.

  • Motivation: 3,500 billion tokens are estimated to be required to optimally train a GPT-3-sized 175B-parameter model.This is twice the largest demonstrated pretraining datasets and ten times the largest publicly available English datasets cited.
  • Motivation: Curated corpora provide diversity and quality but require specialized, labor-intensive processing and may raise legal challenges.Examples include books, technical documents, selected web pages, and social media conversations.
  • Motivation: Web crawls scale to trillions of tokens with limited human intervention but have traditionally been considered inferior to manually curated data.Existing processed web sources such as C4 and OSCAR were regarded as producing less performant models.
  • Research direction: The paper explores whether improved web-data processing can produce models as capable as, or more capable than, models trained on curated corpora.This aims to support growing data needs while streamlining pipelines and reducing human-intensive curation.
  • Contributions: The authors introduce RefinedWeb, a five-trillion-token English web-only dataset, and release a 600-billion-token extract with language models trained on it.They report that web data alone can outperform models trained on both public and private curated corpora in zero-shot benchmarks.

2. Related works

Prior work moved from document-focused corpora toward massive web scrapes, but untreated web data contains substantial low-quality content. RefinedWeb extends existing pipelines with broader filtering and large-scale deduplication while challenging the view that web data is inherently worse than curated corpora.

  • Web-data pipelines: Massive web datasets generally derive from CommonCrawl, whose scale introduces challenges including machine-generated spam and pornographic content.Training on unfiltered web data is described as undesirable because it produces poorly performing models.
  • Web-data pipelines: Modern web-data pipelines combine filtering for undesirable content, language or quality, and deduplication of exact or similar documents.The cited practices include removing low-quality pages, identifying high-quality web documents, and matching repeated content.
  • Deduplication: Deduplication methods range from exact substring matching to fuzzy similarity methods based on MinHash or SimHash.These techniques target repeated extracts, documents, or approximate matches.
  • Prior datasets: The Pile remains an important reference point because recent open models often indirectly use its components through mix-and-match training data.This contextualizes comparisons against models trained on The Pile or related mixtures.
  • RefinedWeb novelty: RefinedWeb combines cross-pipeline preparation practices, line-wise corrections, exact and fuzzy deduplication, and a unique 5,000-billion-token scale.A 600-billion-token extract is publicly available under permissive licensing.

3. Macrodata Refinement and RefinedWeb

Macrodata Refinement (MDR) is a large-scale CommonCrawl pipeline that prepares English web data through extraction, filtering, and stringent deduplication. It produces RefinedWeb by retaining a small fraction of the source while targeting high-quality data at trillion-token scale.

  • Pipeline design: MDR combines state-of-the-art practices and novel approaches to filter and deduplicate CommonCrawl into high-quality web data.The pipeline includes URL scoring, line-wise filtering, and large-scale fuzzy and exact deduplication.
  • Design principles: MDR targets 3-6 trillion English tokens without labor-intensive human curation, using CommonCrawl as its source.Its design principles prioritize scale and strict deduplication for training 40-200B-parameter models.
  • Filtering and deduplication: Nearly 90% of the original CommonCrawl documents are removed across MDR, including approximately 50% for non-English content, 24% for insufficient quality, and 12% for duplicates.Removal rates are measured relative to the preceding stage, while kept rates are measured overall.
  • Document preparation: 48% of the original CommonCrawl documents remain after minimal preprocessing, producing the RW-RAW dataset.Most early removals are attributed to language identification.
  • Filtering: 23% of CommonCrawl documents remain after all filtering heuristics, with roughly half of RW-RAW removed during filtering.Filtering removes non-English, low-quality, repetitive, or otherwise unsuitable documents.
  • Deduplication: MDR applies MinHash fuzzy deduplication at high intensity, using 9,000 hashes over 5-grams divided into 20 buckets.The authors report that less aggressive settings, such as The Pile’s 10 hashes, produced lower deduplication rates and worse model performance.
  • Evaluation: Evaluation uses four zero-shot aggregates spanning 18 tasks, with different aggregates serving ablations or comparisons to public model suites.The main aggregate draws on tasks from GPT-3 and PaLM evaluations, while the EAI Harness is used for the authors’ models.

4. Experiments

The experiments evaluate RefinedWeb through zero-shot task aggregates and controlled comparisons against web and curated datasets. Across these evaluations, filtering and especially deduplication improve performance, while RefinedWeb matches or exceeds prominent curated-data baselines.

  • Evaluation: The evaluation emphasizes zero-shot generalization across many tasks rather than validation loss, using aggregates designed for small-scale ablations and broader model comparisons.The setup uses the EleutherAI evaluation harness and defines small, core, main, and ext aggregates.
  • Small-scale results: RefinedWeb outperforms web datasets and The Pile on the small-aggregate zero-shot evaluation, with both filtering and deduplication contributing significantly.The comparison includes C4, OSCAR variants, and The Pile; OSCAR-22.01 underperforms, possibly because deduplication is optional.
  • Full-scale results: At full scale, RefinedWeb models match GPT-3 performance using web data alone, while open models generally underperform private curated-corpus models.The comparison is reported across main-agg, core-agg, and ext-agg evaluations, with an internal The Pile model performing in line with the BigScience model.
  • Ablations: Deduplication produces a steady zero-shot performance boost across datasets, whereas filtering gains are dataset-dependent and may require source-specific tuning.Combining both stages improves results further, but original extraction and processing flaws are not fully compensated for.
  • Ablations: Applying MDR to C4 can produce subsets that might slightly outperform RefinedWeb, but their rejection rates are unsuitable for a 3-6 trillion-token target.The combination is presented as more relevant to shorter runs seeking extremely high-quality subsets.

5. Limitations

The authors identify unresolved limitations around toxicity and bias analysis, repeated training epochs, and the effects of deduplication on curated corpora.

  • Biases: RefinedWeb is about as toxic as The Pile under the Perspective API definition, which covers rude or disrespectful content but not social biases or harmfulness.The authors encourage further quantitative analysis of the public extract.
  • Multiple epochs: Models with 100B+ parameters may be sensitive to even a few training epochs, leaving tradeoffs between repeated higher-quality data and one epoch of lower-quality data unresolved.The authors identify whether deduplication can sustain more epochs as an open research direction.
  • Other results on deduplication: Deduplicating The Pile had limited impact on zero-shot performance, motivating further study of deduplication on curated corpora and in data-constrained multi-epoch settings.Deduplication can reduce available tokens, potentially requiring multiple epochs to compensate.

6. Conclusion

The authors conclude that stringent filtering and deduplication can produce a five-trillion-token web-only dataset whose models compete with state-of-the-art systems. RefinedWeb is publicly released in a 600GT extract.

  • Conclusion: Five trillion tokens of web-only RefinedWeb can produce models competitive with the state-of-the-art, even outperforming models trained on curated corpora.The dataset uses stringent filtering and deduplication and has supported models such as Falcon-40B.
  • Conclusion: 600GT of RefinedWeb is publicly released as an extract of the larger dataset.The conclusion identifies the release as a resource for future use.
  • Conclusion: RefinedWeb has already been used to train state-of-the-art language models, including Falcon-40B.This is presented as an example of the dataset's subsequent use.

A. RefinedWeb Datasheet

The datasheet describes RefinedWeb as a large CommonCrawl-derived dataset released as a random extract, with preprocessing and cleaning designed to remove undesirable content. It also records limits on retained intermediate data and notes the dataset’s documented toxicity profile.

  • Composition: RefinedWeb uses all CommonCrawl dumps through 2023-06, while the public release is a 600GT random extract of the 5,000GT full dataset.Experiments sampled from the public extract or earlier development versions.
  • Composition: The dataset likely contains offensive, insulting, threatening, or anxiety-inducing content because such material is prevalent on the internet.Its estimated toxic-content prevalence is similar to The Pile.
  • Composition: The data spans CommonCrawl dumps from 2008 to January/February 2023.The collection timeframe is explicitly reported in the datasheet.
  • Preprocessing: Preprocessing removes adult-content URLs, extracts page content, identifies language, and applies additional filtering heuristics and line-wise corrections.The pipeline combines URL filtering, trafilatura extraction, fastText language identification, MassiveWeb heuristics, and custom corrections.
  • Preprocessing: Intermediate pipeline outputs were retained for investigations and ablations for about 5% of RefinedWeb, but not for the final production dataset.Storage and resource constraints prevented retaining final-production intermediates.
  • Documentation: The datasheet includes a RefinedWeb datasheet and a Falcon-RW model card following established documentation frameworks.These are identified as Tables 6 and 7.

C. Dataset analysis

The dataset analysis characterizes RefinedWeb through document-length and domain distributions, alongside a toxicity analysis. Compared with other corpora, RefinedWeb has a relatively similar overall makeup to C4 but a longer tail of short documents.

  • Dataset analysis: RefinedWeb is analyzed using document-length distributions, top-domain breakdowns, and toxic-content distributions.These analyses are presented in Figures 5(a), 5(b), and 4.
  • Document lengths: Following filtering, most short documents are discarded from RW-Filtered, while deduplication reintroduces shorter documents into RefinedWeb.The analysis compares RW-Raw, RW-Filtered, RefinedWeb, OSCAR, C4, and The Pile.
  • Document lengths: RefinedWeb and C4 have relatively similar makeup, but RefinedWeb has a longer tail of short documents.The Pile exhibits a unique makeup in the comparison.

D. Multilingual RefinedWeb

Processed CommonCrawl data spans 176 languages, with English comprising 58.20% of documents. Its language distribution only partially matches worldwide speaker populations, and multilingual processing requires language-specific tuning for some filters and deduplication settings.

  • Multilingual data: 58.20% of processed CommonCrawl documents were identified as English, while the dataset was classified into 176 languages.The top 20 non-English languages are ranked by document count and disk size.
  • Multilingual data: CommonCrawl over-represents Russian and under-represents Mandarin Chinese relative to worldwide speaker distributions, while Hindi is absent from the top 20.Russian ranks second in CommonCrawl but eighth worldwide; Mandarin Chinese ranks sixth or seventh in CommonCrawl but second worldwide.
  • Processing multilingual data: The MDR pipeline supports all languages, but line-wise corrections and deduplication parameters generally require language-specific tuning.Text extraction is language-agnostic, whereas some filtering and deduplication settings are not.
  • Additional results: Appendix results use earlier dataset versions for E.1 and E.3 and Falcon-RW models for E.2, limiting direct comparability with the main text.These setup differences are explicitly reported for the additional development results.

E.1. Small-scale ablations on deduplication approaches

Small-scale ablations favor exact substring deduplication over MinHash alone, while masking duplicated spans performs worse than other exact-substring strategies. Separate epoch experiments suggest deduplication may reduce degradation, but results vary substantially across tasks and remain preliminary.

  • Deduplication approaches: MinHash alone does not match exact substring deduplication, and combining both does not significantly improve performance.The final setup uses MinHash followed by exact substring cutting because MinHash is easier to scale as a pruning phase.
  • Deduplication approaches: Masking duplicated spans systematically underperforms other exact-substring approaches, whereas dropping and cutting spans perform similarly.The ablation compares zero-shot aggregates and perplexity in bits-per-bytes on The Pile.
  • Perplexity evaluation: Models trained on RefinedWeb achieve performance close to The Pile on Wikitext despite excluding Wikipedia content.The authors attribute most of the perplexity difference to unfamiliarity with Wikitext’s article-layout idiosyncrasies.
  • Multiple epochs: Deduplication may reduce performance degradation across multiple epochs, but the evidence comes from 1B models trained on 30GT and shows high task variability.The experiment uses 1, 5, 25, and 100 epochs; RefinedWeb degrades less absolutely than RefinedWeb-Filtered, with a widening gap as epochs increase.
  • Evaluation setup: The evaluation averages zero-shot performance over task aggregates, including small, core, main, and extended benchmark groupings.The core aggregate supports broad model comparisons, while the small aggregate is used for 1B-parameter, 30GT ablations.
  • Evaluation setup: The comparison draws on nearly 50 models across 10 series trained on varied curated corpora, with results sourced from multiple papers.GPT-3 and PaLM results come from their respective papers, while most other results use the EAI Evaluation Harness.
  • Related comparisons: The smaller µ-parametrized Cerebras-GPT series performs close to its main series but is omitted from the main figures because compute-scale matches are unavailable.The comparison is shown in Figure 8.
  • Related comparisons: Under the reported aggregate, deduplicated Pythia may slightly outperform non-deduplicated Pythia, broadly aligning with the paper’s Table 5 findings.Prior work had reported minimal differences between the two Pythia variants.

F.3. Datasets

The paper’s dataset comparisons position RefinedWeb among large web-scrape and language-model datasets, while accompanying comparisons include model architecture, training compute, and filtering assumptions. The reported dataset and model comparisons are drawn from heterogeneous literature sources.

  • Dataset comparisons: The supplementary dataset comparison extends an earlier literature table with filtering and deduplication strategies used across datasets.The paper notes that these details are provided in Table 12.
  • Model comparisons: Table 11 compares full-scale Falcon-RW models with state-of-the-art models, including models trained on The Pile.Training budget is calculated as C = 6ND, where N is parameter count and D is pretraining dataset size.
  • Dataset comparisons: Table 12 compares common massive web-scrape and English language-model datasets across the literature.OSCAR and C4 also have significant multilingual versions.
  • Filtering assumptions: The adult-document filter uses URLs rather than document content to reduce overfiltering risks for minority, legal, and medical content.The design avoids content classifiers and content word blocklists for this filtering decision.
  • Filtering assumptions: URL filtering combines an explicit list of 4.6M banned domains with a URL scoring system based on curated subword matches.The system targets adult content, harmful domains, and likely unstructured or spam-heavy sources.

G.1.1. URL BLOCKLIST

The MDR pipeline applies URL filtering, line-wise cleaning, language identification, and aggressive exact and fuzzy deduplication to CommonCrawl. Its filtering choices use manually curated rules and thresholds, while deduplication balances approximate matching, exact-span removal, and the need to retain five trillion tokens.

  • URL blocklist: The blocklist contains about 4.6M explicitly banned URLs and is regularly updated from a list originally intended for university use.The list is organized into categories such as pornography and gambling.
  • URL blocklist: Manual inspection identified fewer than 100 prolific false-positive domains in a 832M-page blocklist audit, where 6.04M pages matched.The matched-page rate was 0.73%, and URL occurrence counts ranged from 1 to 79k.
  • URL blocklist: The URL blocklist covers adult, malicious, spam, and unstructured-text categories, with severity determined by strict, hard, and soft matching rules.Strict subword matches and hard whole-word matches require one match, whereas soft matching requires at least two.
  • Source exclusions: The pipeline excludes curated online sources at the URL stage so RefinedWeb remains a web-only dataset and can later be combined with curated corpora after further deduplication.Wikipedia is among the examples of sources excluded from RefinedWeb.
  • Line-wise filtering: Line-wise filtering removes uppercase-only, numeric-only, counter, and one-word lines, while editing short lines that match navigation or call-to-action patterns.A document is discarded when flagged lines exceed 5% of its words, and the rules require adaptation across languages.
  • Deduplication: Fuzzy deduplication uses MinHash to identify similar documents, while exact substring deduplication targets duplicate text spans of at least 50 tokens.The exact-span variants cut, mask, or drop documents containing duplicated ranges.
  • Fuzzy deduplication: MinHash detects templated or highly similar pages whose interspersed duplicate sections may be shorter than the 50-token threshold for exact matching.This complements exact substring matching by targeting repeated small sections across otherwise similar documents.
  • Fuzzy deduplication: MinHash LSH splits each document signature into r buckets of b hashes and marks documents as duplicates when one bucket matches exactly.The parameters b and r determine the probability of detecting similar documents.

H. Deduplication samples from RefinedWeb

RefinedWeb deduplication identified large duplicate clusters and repeated content arising from URL variations and exact substring matches.

  • MinHash deduplication: MinHash found eight largest duplicate clusters, each spanning hundreds of thousands of documents.The clusters are listed in Table 15.
  • URL-based duplication: Different URL GET parameters sometimes pointed to substantially similar page content.Table 16 illustrates this source of duplicate document pairs.
  • Exact substring deduplication: Exact substring deduplication also detected matching text across documents.Examples are provided in Table 17.
Loading 2306.01116v1…