The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only

• Jointly scaling model and dataset size raises data-availability concerns: optimally training a GPT-3-sized 175B-parameter model requires at least 3,500B text tokens (Hoffmann et al., 2022).That amount is twice the largest demonstrated pretraining datasets and ten times the largest publicly available English datasets discussed in the introduction (Hoffmann et al., 2022; Touvron et al., 2023; Ortiz Su´arez et al., 2019; Raffel et al., 2020; Gao et al., 2020).Source: §1 · p. 2
• Because curated corpora are labor-intensive to process, limited in volume, and may raise licensing challenges, the authors propose improving web-data processing to reduce reliance on human-intensive curation.Web crawls can scale to trillions of tokens with limited human intervention, but have traditionally been regarded as lower quality than curated sources (Rae et al., 2021; Scao et al., 2022b).Source: §1 · p. 2
• RefinedWeb provides five trillion tokens of web-only English pretraining data, and models trained on web data alone outperform public and private curated corpora on zero-shot benchmarks.The authors present this result as challenging current views about data quality.Source: §1 · p. 2
• The authors publicly release a 600B-token RefinedWeb extract and 1B- and 7B-parameter language models trained on it.Source: §1 · p. 2

Pretraining data for large language models

• Training data evolved from coherent, single-domain corpora to massive web scrapes and then aggregated datasets combining web data with curated sources.Targeted sources are commonly upsampled one to five times, while web data alone has been considered insufficient for powerful LLMs (Liu et al., 2019; Scao et al., 2022b).Source: §2 · p. 3

Pipelines for web data

• Web-data pipelines filter low-quality content through language identification, heuristic rules, ML-based quality filtering, and deduplication, but excessive filtering can introduce bias.Such bias can disproportionately affect minorities (Dodge et al., 2021), motivating approaches such as manually curated allowed URLs (Laurenc¸on et al., 2022).Source: §2 · p. 3

Deduplication

• Deduplication uses exact substring matching and fuzzy methods such as MinHash or SimHash, and has been linked to reduced memorization and improved language-model quality.Repeated data is increasingly harmful as model size grows: a hundred duplicates can harm a 1B-parameter model, while even a few can disproportionately affect a 175B-parameter model (Hernandez et al., 2022).Source: §2 · p. 3

Pretraining data for large language models

• The paper surveys widely adopted English LLM pretraining datasets and notes that recent open models often indirectly use The Pile by mixing and matching its components.The dataset overview appears in Table 1, with additional information in Appendix F.3, Table 12.Source: §2 · p. 3

Novelty

• RefinedWeb combines document-preparation and filtering best practices with line-wise corrections and large-scale exact and fuzzy deduplication.Its dataset totals 5,000 billion tokens, with a 600-billion-token extract available for public use under permissive licensing; the authors also challenge the belief that web data is strictly worse than curated corpora.Source: §2 · p. 3

3. Macrodata Refinement and RefinedWeb

• MDR targets 3–6 trillion English tokens from CommonCrawl, combining strict deduplication with neutral filtering that avoids ML-based filters except for language identification.The design avoids labour-intensive human curation and uses simple rules and heuristics, with URL filtering for adult content.Source: §3 · p. 4

3.2. Filtering: document-wise and line-wise

• 23% of CommonCrawl documents remain after filtering, with about 50% of RW-RAW documents removed by repetition, document-quality, and line-wise correction heuristics.The line-wise filter removes a document if corrections affect more than 5% of it; document-quality heuristics require language-specific adaptation.Source: Figure 2 · p. 4, §3.2 · p. 5

3.3. Deduplication: fuzzy, exact, and across dumps

• MDR combines MinHash fuzzy document deduplication with exact substring removal and URL deduplication across CommonCrawl dumps.It uses 9,000 hashes per document over 5-grams for MinHash, removes exact matches longer than 50 consecutive tokens, and tracks kept URLs across dumps.Source: Table 2 · p. 5, §3.3 · pp. 5–6

3.1. Document preparation: reading data, filtering URLs, extracting text, and language identification

• 48% of original documents remain in RW-RAW, mostly due to language identification; MDR starts from raw WARC and applies URL filtering and document-level language identification.It uses trafilatura to extract page content and removes documents whose fastText top-language score is below 0.65; URL rules target fraudulent or adult sites and exclude common curated sources such as Wikipedia and arXiv.Source: §3.1 · pp. 4–5

4.1. Setting

• Evaluation emphasized zero-shot generalization across tasks rather than validation loss, using EleutherAI harness aggregates for small-scale ablations and model comparisons.The four aggregates were small for ablations and core, main, and ext for comparisons.Source: §4.1 · pp. 6–7

4.2. Can web data alone outperform curated corpora?

• In small-scale tests, RefinedWeb outperformed C4, OSCAR, and The Pile, while filtering and deduplication each improved zero-shot performance.OSCAR-22.01 underperformed other datasets, possibly because deduplication was optional; C4 was a strong baseline.Source: Table 4 · p. 7, §4.1 · p. 7, §4.2 · p. 7
• RefinedWeb models significantly outperformed The Pile, including a direct control, and matched GPT-3 performance on core/ext aggregates.On the main aggregate, RefinedWeb models also matched the GPT-3 series despite excluding common high-quality sources used in The Pile.Source: Figure 3 · p. 8, §4.3 · p. 8, §4.2 · p. 8

4.3. Do other corpora benefit from MDR?

• Across existing corpora, filtering effects varied, but deduplication consistently boosted performance; The Pile and OSCAR-22.01 contained 40–60% duplicates, and combined MDR improved further.Filtering may require source-dependent tuning, and even combined filtering and deduplication could not fully compensate for flaws in original text extraction and processing.Source: §4.3 · pp. 8–9, Table 5 · p. 9
• MDR-processed C4 produced subsets that might slightly outperform RefinedWeb, but rejection rates make them unsuitable for 3–6 trillion-token runs.The authors suggest such high-quality subsets may suit shorter runs.Source: §4.3 · p. 9

Biases

• RefinedWeb is about as toxic as The Pile under the Perspective API definition, which excludes social bias and harmfulness.The authors encourage further quantitative analysis of the public extract.Source: §5 · p. 9

Multiple epochs

• Models with 100B+ parameters may be sensitive to even a few epochs, motivating questions about deduplication and data quality in data-constrained training (Hernandez et al., 2022).Popular models train for up to 2 epochs, while curated datasets commonly upsample corpora 2-5 times.Source: §5 · p. 9

Other results on deduplication

• Biderman et al. (2023) found limited zero-shot impact from deduplicating The Pile, while further work could study curated corpora and data-constrained settings.In data-constrained training, multiple epochs may compensate for tokens lost through deduplication.Source: §5 · p. 9

6. Conclusion

• Five trillion tokens of stringently filtered and deduplicated web data produced a dataset suitable for competitive models that can outperform models trained on curated corpora.The authors argue this scale addresses growing demand for pretraining data beyond publicly available corpora.Source: §6 · p. 9
• The authors publicly released a 600GT RefinedWeb extract, and RefinedWeb has been used to train Falcon-40B.Source: §6 · p. 9

A. RefinedWeb Datasheet — MOTIVATION

• RefinedWeb targets trillions of English tokens by prioritizing scale, strict exact and fuzzy deduplication, and neutral filtering based mainly on simple rules and heuristics.The authors avoid labor-intensive human curation and use URL filtering for adult content rather than ML-based filtering beyond language identification.Source: MOTIVATION · p. 15

A. RefinedWeb Datasheet — COMPOSITION

• The full dataset contains 5,000GT, and its public release is a 600GT random extract used for experiments alongside earlier development versions.It uses all CommonCrawl dumps through the June 2023 dump and could be updated as further dumps appear.Source: COMPOSITION · p. 15
• The authors estimate that RefinedWeb’s toxic-content prevalence is similar to The Pile.The estimate is associated with Figure 4; the supplied passage does not provide its numeric values.Source: COMPOSITION · p. 16

A. RefinedWeb Datasheet — COLLECTION

• The dataset uses CommonCrawl dumps collected from 2008 through January/February 2023.Source: COLLECTION · p. 16

A. RefinedWeb Datasheet — PREPROCESSING

• Preprocessing filters adult-content URLs, extracts page text, identifies language, applies heuristics and line-wise corrections, then removes revisited URLs and performs fuzzy and exact substring deduplication.The datasheet follows the framework introduced by Gebru et al. (2021).Source: PREPROCESSING · p. 16, Table 6 · p. 17
• Intermediary pipeline outputs were retained for investigations and ablations for about 5% of RefinedWeb, but not for the final production dataset because of storage and resource constraints.Source: PREPROCESSING · p. 16

B. Falcon-RW Model Cards

• The Falcon-RW model card follows the framework introduced by Mitchell et al. (2019).Source: Table 7 · p. 18

• The section also refers to a separate analysis of the distribution of toxic content in Figure 4.Source: C. Dataset analysis · p. 19
• Filtering discards most short documents, while deduplication reintroduces shorter documents to RefinedWeb by removing spans.OSCAR datasets and RW-Raw have similar document-length distributions.Source: Figure 5 · p. 19
• RefinedWeb’s make-up is relatively similar to C4’s, but has a longer tail of short documents.Source: Figure 5 · p. 19
• The Pile has a distinct document-length make-up, with long and short document tails.The long-document tail includes books.Source: Figure 5 · p. 19
• Top RefinedWeb domains span content platforms and news sites, and include technical-content domains.Examples include Blogspot, WordPress, Tumblr, CNN, New York Times, BioMed Central, and Springer.Source: Figure 5 · p. 19

D. Multilingual RefinedWeb

• 58.20% of processed CommonCrawl documents were identified as English, while Russian was over-represented and Mandarin Chinese under-represented relative to worldwide speaker rankings.Hindi, the third most spoken language worldwide, does not appear in the top 20 languages in CommonCrawl.Source: D. Multilingual RefinedWeb · p. 19
• The CommonCrawl language distribution only partially aligns with worldwide speaker prevalence: Russian ranks second in CommonCrawl versus eighth worldwide, while Mandarin ranks sixth or seventh versus second.Hindi is absent from the CommonCrawl top 20 despite being the third most spoken language worldwide.Source: D. Multilingual RefinedWeb · p. 19
• Figure 6 presents the top 20 non-English languages in processed CommonCrawl by document count and disk size.The language identification filter classified the processed data into 176 languages.Source: Figure 6 · p. 19, D. Multilingual RefinedWeb · p. 19
• The MDR pipeline can process all languages, since features such as text extraction are language-agnostic, whereas line-wise correction filters typically require language-specific tuning.The passage notes that the MDR pipeline can be used to process all languages.Source: D. Multilingual RefinedWeb · p. 19
• Tuning deduplication parameters for individual languages was also found to be beneficial.Source: D. Multilingual RefinedWeb · p. 19

E. Additional results

• Appendix E.1 and E.3 results use earlier dataset versions and are not directly comparable with the main text, while E.2 results are based on Falcon-RW models.These are additional results from development of the Macrodata Refinement pipeline.Source: E. Additional results · p. 20

E.1. Small-scale ablations on deduplication approaches

• The final setup applies MinHash before exact deduplication for scalability and cuts duplicated spans because dropping them caused more rejections and threatened collecting 5 trillion tokens.Approximate deduplication serves as a pruning phase before exact deduplication.Source: §E.1 · p. 20
• MinHash alone did not match exact deduplication’s zero-shot performance, combining them did not improve it, and masking spans underperformed; cutting and dropping performed similarly.The table reports pile-bpb and aggregate zero-shot performance over LAMBADA, PIQA, and HellaSwag.Source: Table 8 · p. 20, §E.1 · p. 20

E.2. Language modeling evaluation

• RefinedWeb models achieve performance close to The Pile on Wikitext despite excluding all Wikipedia content; the authors suggest unfamiliarity with Wikitext’s layout may explain most of the gap.The authors explicitly filter Wikipedia at the URL level.Source: §E.2 · p. 20, Table 9 · p. 20

E.3. Does deduplication help with multiple epochs?

• In the limited 1B-parameter, 30GT experiment, degradation was smaller for RefinedWeb than RefinedWeb-Filtered and their performance gap widened across 1, 5, 25, and 100 epochs, with high task variability.The authors caution that the small-scale setup would need improvement for definitive results.Source: §E.3 · p. 21, Figure 7 · p. 21

F.1. Task aggregates

• The four task aggregates are small for 1B/30GT ablations, core for broad comparisons, main for GPT-3 and PaLM tasks, and ext for BigScience tasks.The evaluation averages zero-shot performance over diverse task aggregates.Source: §F.1 · p. 21

F.2. Models

• The comparison spans nearly 50 models across 10 series trained on curated corpora; the smaller Cerebras-GPT series was omitted from main figures because it lacked similar-compute models and performed close to its main series.The smaller series uses µ-parametrization; the main Cerebras-GPT series is cited as Dey et al. (2023).Source: §F.2 · p. 21

F.1. Task aggregates

• State-of-the-art results were sourced from multiple papers to maximize task coverage; most use the EAI Evaluation Harness, while PaLM and GPT-3 results come from their papers.The GPT-3 paper’s results remain ahead of API results obtained through the EAI harness.Source: §F.1 · p. 21, Table 10 · p. 22

F.2. Models

• Biderman et al. (2023)’s MinHash-deduplicated Pythia may be slightly ahead under the aggregate; the authors suggest their stronger web-data gains may reflect exact deduplication (45% versus 30% reduction) and single-epoch training.Biderman et al. (2023) used a partial extra epoch to reach 300GT, whereas this work always used one epoch because web data was plentiful.Source: §F.2 · p. 22

F.3. Datasets

• Figure 8 reports that µ-parametrization slightly improves Cerebras-GPT zero-shot performance on the core aggregate (Yang et al., 2021; Dey et al., 2023).Source: Figure 8 · p. 23
• Figure 9 reports a small core-aggregate improvement from deduplication in the Pythia suite (Biderman et al., 2023).Source: Figure 9 · p. 23
• Table 12 compares massive web-scrape and English LLM datasets by size, availability, HTML extraction, language identification, content filtering, and deduplication.Source: Table 12
• RefinedWeb is listed at 5,000GT, with 600GT public and trafilatura used for web HTML extraction.Source: §F.3 · p. 25, Table 12

G.1. URL filtering

• URL-only adult-content filtering avoids document classifiers that can overfilter minorities and NSFW word blocklists that can overfilter legal and medical content (Dodge et al., 2021).Source: §G.1 · p. 26
• The pipeline bans 4.6M domains, then scores URLs using curated subword matches; manual curation was cross-checked against ToxicBERT toxicity outliers (Hanu & Unitary team, 2020).Source: §G.1 · p. 26

G.1.1. URL BLOCKLIST

• An aggregated, regularly updated blocklist of about 4.6M URLs targets adult or malicious content, spam, and unstructured text, with benign false positives removed after review.Applied to 832M pages, the list matched 6.04M pages (0.73%); fewer than 100 prolific benign domains were removed.Source: §G.1.1 · p. 26, Table 13 · p. 26

G.1.2. URL SCORING WITH A WORD-LIST

• URL scoring uses strict subword, hard whole-word, and soft-word matches, with stricter matching thresholds for less ambiguous terms.Strict matches ban any matching substring, hard matches require a whole word, and soft matches require at least two hits; the lists were manually curated to avoid unclear partial matches and protect legitimate content.Source: §G.1.2 · pp. 26–27

G.1.3. EXCLUDED HIGH QUALITY SOURCES

• RefinedWeb excludes common curated-source domains, including sources used in The Pile, to keep the dataset distinct and simplify later combination with curated corpora.The blocked domains are listed in Table 14 and are filtered at the URL stage alongside the adult-content blocklist.Source: §G.1.3 · p. 27, Table 14 · p. 27

G.2. Line-wise filtering

• Line-wise filtering discards or edits irrelevant extracted lines, and documents are dropped when flagged lines exceed 5% of their words.Rules target uppercase-only, numeric-only, counter, and one-word lines, plus short lines matching patterns; the manually derived filters require adaptation across languages.Source: §G.2 · p. 27

G.3. Deduplication

• MinHash-based near deduplication followed by EXACTSUBSTR exact matching forms the two-stage document-deduplication pipeline.MinHash identifies similar documents; EXACTSUBSTR then finds duplicate text spans of at least 50 tokens.Source: §G.3 · p. 28

G.3.1. MINHASH APPROXIMATE MATCHING

• With 5-gram MinHash parameters b = 20 and r = 450, each document uses 9,000 minhashes; pairs at similarity 0.75 and 0.8 have 76% and 99.4% probabilities, respectively, of being marked duplicates.The authors skip additional false-positive filtering because they have a large amount of CommonCrawl data and their main concern is improving recall.Source: §G.3.1 · p. 28

G.3.2. EXACT SUBSTRING DEDUPLICATION

• After MinHash, EXACTSUBSTR uses a suffix array to locate exact duplicate spans, and the authors select EXACTSUBSTR-CUT to remove those spans.The stage reduces the dataset to be processed by nearly 40%; reversible normalization maps detected spans back to original text, and exact deduplication removes all copies of duplicate spans.Source: §G.3.2 · p. 29

G.4. Execution environment

• Most processing ran on clusters of 100–250 AWS c5.18xlarge instances, typically totaling 10,000–20,000 vCPUs.EXACTSUBSTR required loading the dataset into memory, so the authors used AWS x2iedn instances with up to 2 TiB of memory each.Source: §G.4 · p. 29

H.1. MinHash clusters

• The 8 largest MinHash duplicate clusters each span hundreds of thousands of documents.Table 15 lists the top eight clusters and shortens some of the longest samples for readability.Source: §H.1 · p. 30, Table 15 · p. 30
• Different URL GET parameters can yield page content that is not significantly different.Table 16 presents example URLs illustrating this behavior.Source: §H.1 · p. 30, Table 16 · p. 31

H.2. Exact substring matches

• Table 17 shows examples of matches found by exact substring deduplication, marked in italics.Source: §H.2 · p. 31, Table 17 · p. 32