Source-linked AI summary
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, Connor Leahy
TL;DR
Large language models need massive, high-quality data, while reliance on Common Crawl limits source diversity. The paper introduces the 825.18 GiB Pile from 22 diverse datasets and evaluates its properties and training utility. Models trained on the Pile outperform raw Common Crawl and CC-100 across the Pile’s components, while the paper documents potentially concerning data characteristics for prospective users.
Problem
Large-scale language modeling increasingly relies on Common Crawl, motivating broader mixtures of diverse, high-quality datasets for training and evaluation.
Method
The paper constructs the Pile from 22 diverse datasets, including newly introduced sources and a filtered Common Crawl subset with improved extraction quality.
Results
Models trained on the Pile significantly outperform models trained on raw Common Crawl and CC-100 across all Pile components, while improving traditional benchmark performance overall.
Takeaways & Limitations
The Pile provides a broad-coverage resource for large-language-model training and cross-domain evaluation, with construction documentation and code released publicly.
Takeaways & Limitations
The paper notes that training on huge datasets will almost inevitably include undesirable content, leaving how models handle such content as a future research direction.
Abstract
from arXiv · showhide
Recent work has demonstrated that increased training dataset diversity improves general cross-domain knowledge and downstream generalization capability for large-scale language models. With this in mind, we present \textit{the Pile}: an 825 GiB English text corpus targeted at training large-scale language models. The Pile is constructed from 22 diverse high-quality subsets -- both existing and newly constructed -- many of which derive from academic or professional sources. Our evaluation of the untuned performance of GPT-2 and GPT-3 on the Pile shows that these models struggle on many of its components, such as academic writing. Conversely, models trained on the Pile improve significantly over both Raw CC and CC-100 on all components of the Pile, while improving performance on downstream evaluations. Through an in-depth exploratory analysis, we document potentially concerning aspects of the data for prospective users. We make publicly available the code used in its construction.
1 Introduction
The Pile is an 825.18 GiB English dataset that combines 22 diverse, high-quality sources to address the growing demand for large-scale language-model training data. Evaluations report improvements over Common Crawl baselines, alongside extensive documentation intended to inform downstream use.
- 1 Introduction: 825.18 GiB: The Pile combines 22 diverse, high-quality datasets for large-scale language-model training.It also serves as a broad-coverage benchmark for cross-domain knowledge and generalization ability.
- 1 Introduction: The Pile includes existing datasets such as Books3, Project Gutenberg, OpenSubtitles, English Wikipedia, DM Mathematics, EuroParl, and Enron Emails.The Enron Emails corpus adds email communications, a modality typically absent from the other datasets.
- 1 Introduction: Pile-CC is a filtered Common Crawl subset with improved extraction quality.It uses jusText on Web Archive files rather than directly using WET files, yielding higher-quality output than direct WET extraction.
- 1 Introduction: 14 new language-modeling datasets are introduced alongside established and newly constructed sources.New sources include PubMed Central, ArXiv, GitHub, FreeLaw, Stack Exchange, the USPTO, PubMed, Ubuntu IRC, HackerNews, YouTube, PhilPapers, and NIH ExPorter.
- 1 Introduction: Models trained on the Pile significantly improve across many domains compared with CC-100 and raw Common Crawl.The paper identifies these evaluations as a core contribution and reports improvements across many domains for GPT-2-sized models.
- 1 Introduction: The paper documents the Pile’s construction and characteristics and publicly releases preprocessing and construction code.This documentation is intended to help researchers make informed downstream-use decisions and investigate their own data.
2 The Pile Datasets
The Pile combines 22 constituent datasets spanning web, books, academic and professional writing, code, dialogue, mathematics, and email. Its construction uses dataset-specific processing and weighting to form the final corpus.
- Construction: The Pile contains 22 constituent sub-datasets, with higher-quality components receiving increased weights and some datasets repeated for multiple epochs.Wikipedia, for example, may be seen up to three times during a full epoch over the Pile.
- Construction: Pile-CC extracts text from Common Crawl Web Archive files with jusText rather than directly using WET plaintext files, yielding higher-quality output.Common Crawl offers broad domain coverage but requires extraction and filtering because data quality varies.
- Major sources: The corpus includes biomedical full text from PubMed Central, books from Books3 and Project Gutenberg, and recent multilingual web content from OpenWebText2.PMC provides nearly five million open full-text publications; PG-19 contains pre-1919 books, while OWT2 extends OpenWebText with content through 2020 and multiple languages.
- Major sources: The Pile also incorporates legal opinions, Stack Exchange questions and answers, patent backgrounds, Wikipedia, PubMed abstracts, subtitles, mathematics problems, and Ubuntu IRC logs.These sources contribute legal, question-answer, technical, expository, biomedical, dialogic, mathematical, and chat-oriented text.
- Major sources: Additional components include Enron Emails, EuroParl, YouTube Subtitles, PhilPapers, ArXiv, and GitHub repositories, broadening coverage across communication, multilingual, academic, research, and code domains.The sources include both natural-language and non-prose formats, such as parliamentary proceedings, closed captions, LaTeX papers, and open-source code.
3 Benchmarking Language Models with the Pile
The Pile benchmarks GPT-2 and GPT-3 across diverse components, revealing systematic variation in zero-shot performance and a scaling relationship without diminishing returns. A proxy analysis identifies domains where GPT-3 performs poorly and where Pile training may offer the greatest gains, while noting important methodological caveats.
- The Pile serves as a broad-coverage evaluation dataset because its components span multiple disparate domains.
- Larger GPT-2 and GPT-3 models generally attain lower perplexity on the Pile.
- GPT-2 and GPT-3 exhibit a clear zero-shot scaling law on the Pile without diminishing returns despite not being trained on it.
- GPT-3 performs poorly on academic, domain-specific, and non-natural-language datasets, including PubMed, ArXiv, FreeLaw, GitHub, and DM Mathematics.
- The componentwise proxy suggests most Pile components are not redundant with GPT-3’s predominantly web-based training data.
- The proxy metric may be confounded by dataset-specific scaling effects, and the authors leave rigorous analysis of these effects to future work.
4 Evaluation
The evaluation compares equal-sized, decontaminated models trained on the Pile, CC-100, and raw Common Crawl using traditional benchmarks and the Pile itself. Pile training improves cross-domain performance broadly, with gains especially pronounced for academic, programming, multilingual, and mathematical data, while filtering can reduce Common Crawl diversity.
- The study trains architecturally identical 1.3B-parameter models on the Pile, CC-100, and raw Common Crawl, evaluating WikiText, LAMBADA, and the Pile.
- The Pile significantly improves WikiText performance and produces negligible changes on LAMBADA.
- Models trained on the Pile significantly outperform both Raw CC and CC-100 across all Pile components, indicating stronger cross-domain generalization without compromising traditional benchmarks.
- Pile-trained models perform significantly better on academic datasets such as ArXiv, PubMed Central, FreeLaw, and PhilPapers.
- Pile training improves programming, EuroParl, and DM Mathematics performance, indicating gains for software, multilingual, and mathematical data.
- Raw Common Crawl outperforms CC-100 on Pile BPB despite losing significantly on LAMBADA and WikiText, possibly because CC-100 filtering limits data diversity.
5 Structural Statistics
The Pile’s structural statistics examine document lengths and bytes per token across its components. These measures reveal a mostly short-document corpus with a long tail and distinguish ordinary text from non-text or non-English content.
- Document Lengths: Most Pile documents are short, but the corpus has a long tail of very long documents.The highest 1 percentile of document lengths is excluded from the plotted distribution.
- Bytes per Token: Bytes per token from the GPT-2 tokenizer provide a rough indicator of how syntactically different each component is from WebText.The analysis uses these statistics to contextualize ablations.
- Bytes per Token: Components resembling ordinary WebText, including NIH ExPorter, OpenWebText2, and Books3, have greater bytes per token.These datasets are described as having ordinary text in a distribution similar to WebText.
- Bytes per Token: GitHub, ArXiv, Stack Exchange, and DM Mathematics have among the lowest bytes per token because they contain substantial non-text content.EuroParl also has low bytes per token because it includes languages other than English.
- Language Coverage: The Pile is predominantly English, but the authors plan a fully multilingual expansion as future work.The supplied passages report that the dataset is 97.4% English and that a multilingual expansion is planned.
6 Investigating and Documenting the Datasets
The authors document the Pile’s composition, topical coverage, potential harms, and consent status to help researchers make informed decisions about its use.
- The documentation addresses ethical concerns and aims to normalize engagement with AI ethics literature for massive-language-model datasets.
- Datasheets and data statements provide the primary documentation frameworks, supplemented by additional contextual analysis of the datasets.
- Topic analysis finds that Common Crawl-derived content misses programming, logic, physics, and legal knowledge modes present elsewhere in the Pile.
- The Pile as a whole and most components appear less profane than Pile-CC, based on an English-only profanity-classification proxy.
- The analyses find no significant overall sentiment bias toward men or women, while reporting differing sentiment associations across religion and racial categories.
- Consent analysis documents dataset availability and identifies cases where hosts lacked authorization or individuals lacked the ability to consent.
7 Implications and Broader Impacts
The paper discusses copyright, consent, and alignment implications of releasing the Pile, while acknowledging unresolved legal variation and risks from undesirable training content.
- The authors discuss why they believe using copyright data in the Pile complies with US copyright law, while noting that this is not legal advice.
- Copyright law varies by country, and component-level exclusion is only a heuristic because the authors lack metadata identifying exactly which texts are copyrighted.
- The authors acknowledge that the Pile could potentially accelerate AI timelines and hope dataset documentation encourages diligence in downstream alignment research.
- They argue that future models should understand morally problematic content without adopting it, but recognize current technology has not reached that stage.
- Training large models will almost inevitably include undesirable content, and the authors state that more research is needed to address this problem.
8 Related Work
Related work shows a field increasingly reliant on Internet-scale data, especially Common Crawl, while the Pile combines web scrapes with targeted higher-quality datasets.
- Self-supervised language models have been trained on large unlabeled corpora including Wikipedia, BookCorpus, web scrapes, and other established datasets.
- As data requirements grew, large-scale language-model training increasingly turned to Internet scrapes, particularly Common Crawl.
- Prior models combined multiple datasets, and the Pile extends this pattern by mixing large web scrapes with smaller, higher-quality sources.
- CC-100 and C4/mC4 are Common Crawl-derived alternatives, whereas the Pile directly addresses the need to combine web-scale and targeted datasets.
Appendices
The paper reports broad author participation, with contributors taking responsibility for project leadership, implementation, evaluation, analysis, documentation, and data processing.
- All authors contributed to the research design and paper writing, with additional responsibilities distributed across project activities.
- Leo Gao led core implementation, training, evaluation, language analysis, and processing for several major Pile components.
- Stella Biderman led data analysis, broader-impact analysis, documentation, and project coordination.
B Excluded Datasets
The Pile excluded datasets for practical, representational, and content-safety reasons. These decisions removed sources that were too small, difficult to obtain, likely to introduce undesirable biases, or unsuitable for general-purpose language modeling.
- Some datasets were excluded because they were too small or their English component did not merit inclusion independently.
- The US Congressional Record was excluded because it contained substantial racist content judged inappropriate for general-purpose language modeling.
- Fanfiction was excluded on logistical grounds despite offering substantial short-form fiction largely absent from typical language-modeling datasets.
- Literotica was excluded because it would have become the sole short-form-fiction source after fanfiction’s removal, potentially creating undesirable training biases.
C Dataset Details
The Pile’s dataset-specific pipelines combine targeted extraction, language and quality filtering, deduplication, and source-specific collection choices. The section also documents practical trade-offs, including English-only Common Crawl processing and limited-resource repository and document selection.
- The dataset documentation describes acquisition and processing details intended to support future replication, with processing code released under permissive open-source licenses.
- Common Crawl: Common Crawl processing uses jusText extraction, English-language prefiltering with pycld2, and quality classification trained against OpenWebText2.
- Common Crawl: The authors favor jusText because it discards more data than alternatives, avoiding unnecessary boilerplate when abundant Common Crawl data is available.
- Common Crawl: Common Crawl is restricted to English because jusText has lower quality on other languages and cannot handle languages such as Chinese and Japanese.
- OpenWebText2: OpenWebText2 combines Reddit-derived URL filtering, Newspaper extraction, document-level MinHashLSH deduplication, and separate filtered and raw releases.The filtered release contains 65.86 GB across 17,103,059 documents, while the raw release contains 193.89 GB across 69,547,149 documents.
- Academic and code data: The collection includes 1,264,405 arXiv papers converted from TEX sources to Markdown, while GitHub repositories are selected using stars and a file-size limit.
D General Data Processing
The Pile combines datasets through weighted random sampling, shuffles them into output piles, and holds out data for validation and testing while preventing verbatim leakage. Processing choices reflect memory constraints and deliberately avoid full-corpus deduplication and universal downstream-overlap removal.
- Combining datasets: Documents are sampled until the target size is reached, weighted by dataset size and desired epochs so each dataset approximately follows its epoch count.
- Shuffling: The output is interleaved into 30 piles to address shuffling challenges caused by limited memory and computational resources.
- Splits and leakage: Approximately 10 GiB is held out, with 2 GiB used for validation and test splits and verbatim overlaps removed from training data.
- Weighting: High-quality and smaller components receive increased sampling weights, with academic texts upweighted most according to source quality and dataset size.
- Deduplication: Deduplication is limited to OpenWebText2 and Pile-CC, where MinHashLSH at approximate Jaccard similarity 0.5 yields duplicate rates of 28% and 26%, respectively.
- Evaluation leakage: The authors do not remove all downstream-evaluation overlaps and instead recommend that users remove overlaps with additional evaluation validation sets.
E Investigating data
The exploratory analysis finds that Common Crawl contains extensive repeated formatting, boilerplate, and other non-content strings among its most frequent 13-grams. These findings identify data-quality concerns relevant to prospective users of the Pile.
- 40,216,231,078 distinct 13-grams occur in Common Crawl, with the 1000 most common ranging from 11 million to 20k occurrences.
- The most frequent 13-grams include repeated styling characters, malformed HTML escape sequences, and boilerplate from forum software.
- Common Crawl’s top 1000 13-grams contain substantial HTML, CSS, advertising, travel, employment, news, and government boilerplate.
E.2 Benchmark Perplexity Computation
The benchmark computes perplexity by scoring document segments within model-specific context limits and aggregating results across each dataset and the full Pile. It also examines how perplexity varies with sequence position and documents challenges in classifying pejorative content.
- Perplexity computation: Documents are tokenized separately, split into segments up to each model’s maximum sequence length, and scored using preceding tokens as context.The stated limits are 1024 tokens for GPT-2 and 2048 for GPT-3.
- Perplexity computation: Whole-Pile perplexity is weighted by constituent dataset size rather than computed as a simple average of dataset perplexities.
- Perplexity computation: GPT-2 and GPT-3 perplexities are directly comparable because both models share the same tokenizer and vocabulary.
- Position effects: Later input positions have lower perplexities because those tokens are predicted with more context.Figure 11 smooths the plotted curves by grouping four positions per datapoint.
- Pejorative-content analysis: The analysis initially divided pejorative content into four categories, but blacklist-based classification faces misspellings, evolving usage, and context dependence.
F Data Samples
The Pile’s validation samples are presented as two random, noncherrypicked 512-byte excerpts from every constituent dataset. The excerpts span varied academic, technical, news, legal, medical, literary, and software-related text.
- Sampling procedure: Two random, noncherrypicked 512-byte validation samples are shown for each constituent dataset of the Pile.
- Sample diversity: Other excerpts represent news, political reporting, legal writing, literary criticism, and historical or cultural reporting.
- Sample diversity: The collection also contains practical and professional prose, including product patents, curriculum research, and cryptocurrency writing.
- Sample diversity: The samples include technical and scientific material, such as medical research, mathematical analysis, gamma-ray data descriptions, and software code.