Source-linked AI summary
The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf
TL;DR
Public LLM pretraining datasets reveal little about how their web data are curated, despite curation strongly affecting model performance. This work releases FineWeb and FineWeb-Edu, systematically studies filtering and deduplication choices, and finds stronger benchmark performance, especially from educational data.
Problem
Pretraining datasets for leading open LLMs are not publicly available, leaving limited evidence about how effective high-quality dataset curation should be designed.
Method
The authors construct FineWeb from 96 Common Crawl snapshots, ablate filtering and deduplication choices, and filter FineWeb into FineWeb-Edu using an educational-quality classifier.
Results
FineWeb-trained models outperform models trained on other public web datasets, while FineWeb-Edu outperforms FineWeb and other open web datasets on knowledge- and reasoning-intensive benchmarks.
Takeaways & Limitations
The released datasets, curation code, processing library, and ablation models expand public resources for studying and building LLM pretraining datasets.
Takeaways & Limitations
Both datasets consist entirely of Common Crawl web content, and evaluations are limited to academic benchmarks without instruction tuning or alignment.
Abstract
from arXiv · showhide
The performance of a large language model (LLM) depends heavily on the quality and size of its pretraining dataset. However, the pretraining datasets for state-of-the-art open LLMs like Llama 3 and Mixtral are not publicly available and very little is known about how they were created. In this work, we introduce FineWeb, a 15-trillion token dataset derived from 96 Common Crawl snapshots that produces better-performing LLMs than other open pretraining datasets. To advance the understanding of how best to curate high-quality pretraining datasets, we carefully document and ablate all of the design choices used in FineWeb, including in-depth investigations of deduplication and filtering strategies. In addition, we introduce FineWeb-Edu, a 1.3-trillion token collection of educational text filtered from FineWeb. LLMs pretrained on FineWeb-Edu exhibit dramatically better performance on knowledge- and reasoning-intensive benchmarks like MMLU and ARC. Along with our datasets, we publicly release our data curation codebase and all of the models trained during our ablation experiments.
1 Introduction
FineWeb addresses the limited public knowledge of pretraining-data curation by releasing large-scale datasets and documenting their filtering and deduplication choices. FineWeb-Edu further targets educational content and improves performance on knowledge- and reasoning-intensive benchmarks.
- FineWeb is a 15-trillion-token dataset sourced from 96 Common Crawl snapshots and designed to train performant LLMs.It is large enough to train a Chinchilla-optimal model with more than 500 billion parameters.
- The authors select and tune filters from more than fifty candidates and investigate how deduplication strategies and granularities affect performance.
- Models trained on FineWeb perform better than those trained on other public web-based pretraining datasets.
- FineWeb-Edu contains 1.3 trillion tokens of educational text filtered from FineWeb and improves results on knowledge- and reasoning-intensive benchmarks.The dataset uses a custom educational-quality classifier and is released under the permissive ODC-By License.
- The project releases FineWeb, FineWeb-Edu, the datatrove processing library, and models trained during the ablation experiments.
2 Background
LLM pretraining commonly uses web text, especially Common Crawl, but filtering and deduplication choices strongly shape dataset quality and downstream performance. Public datasets therefore differ substantially in their language, quality, boilerplate, toxicity, and duplicate-removal pipelines.
- Autoregressive Transformer-based LLMs model the distribution of the next token conditioned on preceding tokens.
- Web text is a common pretraining source because Common Crawl provides publicly available, continually updated web data.Running an independent web scraper at scale requires substantial resources and expertise.
- Filtering can remove boilerplate and gibberish that may harm performance, but excessive filtering can make the dataset too small.
- Public datasets use varied language-identification, quality-filtering, text-extraction, and deduplication methods.Examples include fastText, heuristic filters, MinHash, hash-based methods, and language-model perplexity filtering.
- Closed-model reports describe additional curation strategies, including WebText selection from Reddit-linked pages, classifier filtering, heuristic quality rules, and MinHash deduplication.
3 Building FineWeb
FineWeb was built through reproducible, controlled ablations of extraction, filtering, deduplication, and benchmark evaluation. The resulting design used independently deduplicated snapshots and custom filters, producing strong performance across open-dataset comparisons.
- Ablation methodology: The authors compared identical 1.71B-parameter models trained on equal-sized samples from alternative dataset versions and evaluated them on the same downstream benchmarks.Ablations used matched architectures, training steps, token counts, and multiple random subsets to reduce evaluation noise.
- Text extraction: Trafilatura extraction from WARC files outperformed default WET text, which retained too much boilerplate and menu text.The extraction ablation used 28B tokens with no filtering or deduplication beyond fastText English filtering.
- Base filtering: Base filtering removed unwanted content using URL, language, quality, and repetition criteria before further dataset curation.The pipeline retained English text with fastText score >= 0.65 and produced roughly 36T tokens from 96 WARC snapshots.
- Deduplication: Global MinHash deduplication reduced the dataset to 4T tokens but yielded little improvement and performed far below RefinedWeb.On a randomly sampled 350B-token subset, global deduplication removed up to 90% of some older snapshots.
- Deduplication: Individually deduplicating each snapshot produced 20T tokens and matched RefinedWeb’s performance, unlike global deduplication.The contrast motivated separate treatment of each crawl rather than deduplicating all snapshots against one another.
4 FineWeb-Edu
FineWeb-Edu is a 1.3-trillion-token educational subset of FineWeb produced with an educational-quality classifier, and it improves performance on knowledge- and reasoning-intensive benchmarks. Its gains include better benchmark scores and matching Matrix with substantially fewer tokens, alongside distinct domain coverage.
- Dataset and filtering: 1.3 trillion tokens comprise FineWeb-Edu, filtered from FineWeb using a classifier trained with synthetic Llama-3-70B educational annotations.The classifier was trained from 460,000 scored webpages and then scaled to FineWeb with a linear regression model over frozen embeddings.
- Benchmark results: MMLU rises from 33% to 37% and ARC from 46% to 57% when using FineWeb-Edu instead of FineWeb.These correspond to approximate relative improvements of 12% and 24%, respectively.
- Benchmark results: FineWeb-Edu surpasses FineWeb and other open web datasets on educational benchmarks including MMLU, ARC, and OpenBookQA.Figure 10 reports an approximately 2% aggregate-score enhancement over the base FineWeb comparison.
- Benchmark results: On MMLU, FineWeb-Edu reaches 33.6% accuracy with 38 billion tokens, while Matrix reaches similar accuracy at 300 billion tokens.The comparison indicates nearly tenfold fewer tokens for comparable final performance.
- Domain coverage: FineWeb has lower perplexity on broad web sources, whereas FineWeb-Edu provides better coverage of Wikipedia and programming content.The domain comparison used macro-average perplexity across Paloma domains without decontamination.
5 Bias analyses
The bias analysis examines sensitive and protected subgroups in FineWeb and FineWeb-Edu, identifying differences in representation and associations across terms. The supplied passages indicate that these datasets can reflect biases present in web text.
- Scope: FineWeb and FineWeb-Edu are analyzed for biases affecting sensitive or protected subgroups recognized in English-speaking countries.The analysis focuses on subgroups historically subject to discrimination and negative societal norms reflected in consumed text.
- Dataset differences: FineWeb has lower perplexity on broad web sources, while FineWeb-Edu has better coverage of Wikipedia and programming content.This domain difference provides context for interpreting how educational filtering changes dataset composition.
6 Conclusion
The paper releases FineWeb and FineWeb-Edu as large-scale public pretraining datasets, together with the code, processing library, and ablation models used to create and study them. It also identifies important scope limits and directions for further improvement.
- Contributions: FineWeb contains 15 trillion tokens from 96 Common Crawl snapshots, while FineWeb-Edu contains 1.3 trillion tokens of educational content from FineWeb.The authors report empirical studies of text extraction, deduplication, and content-filtering choices underlying FineWeb.
- Contributions: The datasets, processing code and library, and all models trained during ablation experiments are publicly released.The releases are intended to support public development and study of performant LLM pretraining datasets.
- Limitations: FineWeb and FineWeb-Edu are limited by Common Crawl-only data, smaller-scale experiments, and academic evaluations without instruction tuning or alignment.The authors suggest adding other data types, using more realistic training scales, and evaluating setups that better reflect current LLM usage.
A FineWeb Datasheet
The FineWeb datasheet documents the dataset’s purpose, provenance, processing pipeline, release conditions, and known limitations. FineWeb is built from Common Crawl web pages and includes an educationally filtered subset.
- Purpose: FineWeb was released to make large language model training more accessible to the machine learning community.The dataset’s stated rationale is to provide a clean, large-scale resource for pretraining performant language models.
- Release: FineWeb is released under ODC-By v1.0 and remains subject to Common Crawl’s Terms of Use.The datasets are hosted on Hugging Face and are planned for regular updates with new Common Crawl snapshots.
- Data fields: The dataset includes text, identifiers, crawl metadata, language annotations, language scores, and GPT-2 token counts.The fields record source pages and snapshots alongside processing-derived annotations.
- Data processing: FineWeb consists of Common Crawl webpages processed with text extraction, language and quality filters, deduplication, PII reformatting, and educational filtering for FineWeb-Edu.The documented pipeline includes Trafilatura, FastText, MassiveText, C4, custom filters, and MinHash deduplication.
- Biases and risks: The dataset may retain toxic content and web-derived biases despite URL-level filtering, including uneven representation of gender and religion terms.The datasheet reports that harmful biases present on the web may be reproduced in FineWeb.
- Known limitations: Filtering may leave code underrepresented, so users are advised to complement FineWeb with code datasets and specialized curated sources.The datasheet specifically mentions Wikipedia and other sources that may offer better formatting or code coverage.
D.1 Model architecture
The section specifies the model configuration and the MinHash deduplication setup, including its similarity-detection probabilities and compute trade-off relative to RefinedWeb.
- The ablation model has 1.71B parameters, 24 hidden layers, 32 attention heads, 32 key-value heads, tied embeddings, and 0.02 initialization standard deviation.
- MinHash deduplication hashes 5-grams with 112 functions, splits signatures into 14 buckets of 8 hashes, and matches documents sharing one bucket.
- RefinedWeb’s 9000 hashes and 450 buckets of 20 hashes provide a steeper cutoff, while FineWeb’s setup reduces compute and storage requirements.
E.2 Measuring the effect of deduplication
Deduplication effects are difficult to measure on small samples because repeated documents emerge at larger scales. Additional global deduplication methods performed worse than individual-snapshot MinHash deduplication.
- At 1B sampled tokens, nearly all documents appear unique despite each being repeated 100 times in the full 20T-token simulation.
- At 1T tokens, most documents repeat up to 8 times and some up to 16 times, illustrating why deduplication effects are scale-dependent.
- The 350B-token evaluation scale would contain many documents duplicated up to 8 times under the simulation’s worst-case cross-snapshot duplication scenario.
- URL and line-based global deduplication were tested after individual snapshot deduplication, but models trained on them consistently performed worse.
- The final pipeline therefore applied no additional deduplication beyond individual-snapshot MinHash deduplication.
F.2 Additional results
FineWeb-Edu is compared with other open datasets and across educational-filter thresholds and topic distributions. A threshold of 3 produced the best average performance, with educational and historical topics more represented.
- A 1.71B model trained on 350B tokens was used to compare FineWeb-Edu with other open web datasets across 9 benchmarks.
- A filtering threshold of 3 yielded the best Aggregate Accuracy when constructing FineWeb-Edu.
- FineWeb-Edu increases representation of Education, Learning, Teaching and History, Culture, Politics relative to FineWeb.
- FineWeb-Edu down-samples Business, Finance, Law and Entertainment, Film, Theater relative to FineWeb.
F.4 Domain fit
Domain-fit results compare FineWeb and FineWeb-Edu perplexity across web and Wikipedia domains. FineWeb-Edu fits some knowledge-oriented domains better, but results vary substantially by source and topic.
- FineWeb-Edu has lower perplexity than FineWeb for Dolma V1.5 pes2o, with values of 10.242 versus 12.226.
- On Dolma V1.5 reddit, FineWeb-Edu has higher perplexity than FineWeb, 29.864 versus 23.814, while Dolma V1.5 wiki is nearly unchanged, 12.243 versus 12.0.
- FineWeb-Edu has lower perplexity than FineWeb on Dolma V1.5 stack, 7.014 versus 7.65, but higher perplexity on Dolma V1.5 common-crawl, 18.336 versus 14.499.
- Within Wikipedia, FineWeb-Edu improves Health science, Nutrition, Public health, Self care, History and events, and Human activities, while worsening several culture and reference domains.
- FineWeb-Edu lowers perplexity for several Wikipedia domains, including Mathematics and logic, 9.903 versus 12.703, and Natural and physical sciences, 10.529 versus 12.884.
G.1 Distributional Analysis
The analysis examines subgroup-term distributions and TF-IDF-based co-occurrence skews in FineWeb and FineWeb-Edu. FineWeb shows substantial frequency imbalances and associations reflecting gender, religious, age, and online-intimacy skews.
- Distributional Analysis: The analysis uses normalized subgroup terms and approximately 10 billion GPT-2 tokens sampled from FineWeb and FineWeb-Edu.Gender, age, and religion terms are examined using the dataset subsets FineWeb 10BT and FineWeb-Edu 10BT.
- Distributional Analysis: ‘man’ appears much more frequently than ‘woman’ and ‘non-binary’, while ‘christian’ appears more frequently than every other tested religion term.These distributions are shown for gender and religion subgroup terms.
- Association Analysis: TF-IDF quantifies subgroup-specific co-occurrence by comparing each subgroup term’s score with the average score of its other co-occurring words.The procedure builds a vocabulary, selects documents containing each subgroup term, computes co-occurrence TF-IDF, and retains positive skews.
- Gender Associations: In FineWeb, ‘man’ co-occurs with ‘god’, ‘police’, ‘said’, and ‘good’; ‘woman’ with intimacy and family terms; and ‘non-binary’ with gender and LGBTQIA+ terms.The reported associations include ‘dating’, ‘love’, and ‘family’ for ‘woman’, and ‘trans’, ‘transgender’, and ‘queer’ for ‘non-binary’.
- Gender Associations: FineWeb-Edu associates ‘woman’ with ‘pregnancy’, ‘cancer’, ‘mother’, ‘children’, and ‘family’, while ‘man’ is associated with ‘god’, ‘war’, ‘great’, and ‘king’.These associations differ from the FineWeb patterns reported in the same analysis.
- Religion and Age Associations: Religion and age analyses reveal online-intimacy skews and an association between ‘young’ and ‘women’, alongside expected links to ‘children’ and ‘school’.Examples include ‘jewish’ with ‘dating’ and ‘singles’, and several religion terms with ‘women’, ‘sex’, or ‘girl’.