Source-linked AI summary
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, Matt Gardner
TL;DR
Large web-crawled corpora used to train language models are often minimally documented, leaving their composition and filtering effects unclear. This paper documents C4 through provenance, content, and exclusion analyses, finding unexpected and machine-generated content, benchmark contamination, and disproportionate removal of minority-associated text. It concludes with recommendations for documenting and constructing web-scale datasets.
Problem
Web-scale unlabeled corpora are difficult to document thoroughly, leaving consumers uncertain about pretraining-data influences and potential downstream biases.
Method
The paper analyzes C4’s source metadata, included text, and filtered-out text, while releasing multiple corpus versions and documentation resources.
Results
The analyses find unexpected sources, machine-generated text, benchmark contamination, social biases, and disproportionate exclusion of minority voices through blocklist filtering.
Takeaways & Limitations
Web-scale datasets should document their provenance, contents, and exclusions, and account for contamination and filtering effects.
Takeaways & Limitations
The paper does not show a causal link between measured sentiment statistics in C4.EN and downstream bias.
Abstract
from arXiv · showhide
Large language models have led to remarkable progress on many NLP tasks, and researchers are turning to ever-larger text corpora to train them. Some of the largest corpora available are made by scraping significant portions of the internet, and are frequently introduced with only minimal documentation. In this work we provide some of the first documentation for the Colossal Clean Crawled Corpus (C4; Raffel et al., 2020), a dataset created by applying a set of filters to a single snapshot of Common Crawl. We begin by investigating where the data came from, and find a significant amount of text from unexpected sources like patents and US military websites. Then we explore the content of the text itself, and find machine-generated text (e.g., from machine translation systems) and evaluation examples from other benchmark NLP datasets. To understand the impact of the filters applied to create this dataset, we evaluate the text that was removed, and show that blocklist filtering disproportionately removes text from and about minority individuals. Finally, we conclude with some recommendations for how to created and document web-scale datasets from a scrape of the internet.
1 Introduction
The paper documents C4 to clarify the provenance, content, and filtering effects of a massive web-crawled corpus. It finds unexpected sources, machine-generated and contaminated text, and disproportionate exclusion of minority-associated language while providing accessible data and documentation recommendations.
- Contribution: The paper provides some of the first documentation for C4, one of the largest language datasets, and makes a downloadable copy available for reproducibility.C4 contains more than 156 billion tokens collected from more than 365 million domains.
- Motivation: Thorough documentation of web-scale corpora is difficult because information commonly reported for structured datasets is often unavailable in web-crawled text.This lack of documentation leaves pretrained-model consumers uncertain about how training data influences systems and can inject subtle downstream biases.
- Approach: The authors recommend examining three views of web-scale corpora: source metadata, the text itself, and the text removed by filtering.The metadata view includes internet domains, while text analysis examines machine-generated content and contamination.
- Findings: The corpus contains significant machine-generated text and evidence that evaluation examples from other datasets occur in C4.Examples include text produced by machine translation systems and benchmark contamination.
- Findings: Blocklist filtering disproportionately removes documents in English dialects associated with minority identities and documents discussing LGBTQ+ identities.The paper therefore argues that documentation should report what web-crawled datasets exclude, not only what they contain.
- Resources: The authors publicly host three differently filtered data versions, an indexed version for searching, and a repository for discussing findings.These resources accompany the paper’s documentation and analysis recommendations.
2 The English Colossal Clean Crawled Corpus (C4)
C4.EN is produced from an April 2019 Common Crawl snapshot by applying filters intended to retain natural English. The authors also provide uncleaned and no-blocklist variants for comparison.
- Construction: C4 begins with the April 2019 Common Crawl snapshot and applies filters intended to remove text that is not natural English.Filters remove short or improperly punctuated lines, documents with too few sentences, Lorem ipsum text, and documents containing blocklisted words.
- Construction: An English-language filter removes documents not classified as English with probability at least 0.99, producing the cleaned corpus C4.EN.C4.EN is the version created by applying all described filters.
- Corpus variants: C4.EN.NOCLEAN preserves the English-identified Common Crawl snapshot without the other filters, while C4.EN.NOBLOCKLIST removes only the blocklist filtering step from C4.EN.The three hosted corpora support comparison of filtering choices.
3 Corpus-level statistics
C4.EN draws text from a wide range of domains and websites, with notable representation from US government, military, patent, news, and other sources. Its temporal and geographic distributions are also uneven, with most URLs recently indexed or written but disproportionate hosting in the United States.
- Internet domains: C4.EN includes substantial text from .gov sites and 33,874,654 tokens from .mil sites across 58,394 unique URLs.An additional 1,224,576 tokens come from .mod.uk URLs.
- Websites: Patents.google.com is the single-most represented website in C4.EN, while patents.com is among its top 10 websites.The corpus also contains well-represented Wikipedia, news, publications, book-publishing, and stock-analysis sites.
- Websites: C4.EN’s website distribution is not necessarily representative of the internet’s most frequently visited websites.The paper notes low overlap with the top 25 websites measured by Alexa.
- Utterance Date: 92% of sampled C4.EN URLs were estimated to have been written between 2011 and 2019, although a non-trivial long tail dates back 10–20 years.The estimate uses the earliest Internet Archive indexing date as a proxy for utterance date.
- Utterance Date: The Internet Archive indexed only approximately 65% of C4.EN URLs, and its indexing can occur many months after webpage creation.These limitations constrain the temporal analysis based on archive indexing dates.
- Geolocation: 51.3% of pages were hosted in the United States, while India, Pakistan, Nigeria, and The Philippines accounted for 3.4%, 0.06%, 0.03%, and 0.1% of URLs, respectively.The paper uses hosting location as a proxy for the location of webpage creators, despite geolocation caveats.
4 What is in the text?
C4.EN contains machine-generated and benchmark-derived text, alongside demographic sentiment differences that vary across source domains. These findings reveal important properties of the corpus beyond its size and language filtering.
- Machine-generated text: Machine-generated text enters C4.EN through OCR and machine translation, especially in patent documents from non-English patent offices.Many patents were scanned with OCR, and some non-English patents were processed through OCR followed by machine translation.
- Benchmark contamination: 4.6% of LAMA T-REx examples and 5.7% of Google-RE examples exist verbatim in C4.EN.A pretrained model can retrieve matching training instances for these evaluation examples.
- Benchmark contamination: GLUE input contamination ranges from less than 2% to over 50% across test-set components.For multi-component datasets, contamination is reported separately for each component.
- Demographic biases: The sentiment spread between “Jewish” and “Arab” is 4.5% in New York Times articles, compared with 7.5% overall in C4.EN, and absent in Al Jazeera articles.Bias measurements differ across internet domains.
5 What is excluded from the corpus?
The study examines text removed by C4.EN’s blocklist and finds that exclusions are not limited to offensive material. Minority identity mentions and English varieties associated with minority groups are disproportionately affected.
- Excluded document topics: Only 16 clusters, comprising 31% of excluded documents, are largely sexual in nature.Other excluded clusters concern science, medicine, health, legal, and political topics.
- Whose English is included?: The dialect terminology follows Blodgett et al. (2016), whose category choices the authors acknowledge are disputed.This caveat concerns the terminology used to describe English varieties.
- Whose English is included?: Blocklist filtering removes African American English and Hispanic-aligned English documents at rates of 42% and 32%, versus 6.2% and 7.2% for WAE and other English.The authors estimate dialect using a dialect-aware topic model and assign each document its most probable dialect.
- Whose English is included?: C4.EN assigns 97.8% of documents to WAE, while AAE and Hispanic-aligned English account for 0.07% and 0.09%.These proportions describe the cleaned corpus after blocklist filtering.
6 Discussion & Recommendations
The discussion identifies domain skew, benchmark contamination, social bias, and blocklist-driven exclusion in C4.EN, then recommends greater transparency and continued scrutiny of web-scale corpora.
- Reporting website metadata: C4.EN is not representative of the English-speaking world, spans a wide range of years, and requires domain metadata for interpretation.The authors report strong representation of patents, news, and Wikipedia domains and substantial data from over a decade ago.
- Examining benchmark contamination: Benchmark contamination is a concern because evaluation examples uploaded to websites can enter passively collected webtext and affect benchmark performance.The authors support dynamically collecting data with human involvement to reduce future contamination.
- Social biases and representational harms: C4.EN contains evidence of negative sentiment bias against Arab identities, but the authors have not established a causal link to downstream bias.Different internet domains show different bias distributions, although selecting subdomains cannot easily control many biases simultaneously.
- Excluded voices and identities: Blocklist filtering disproportionately excludes documents associated with Black and Hispanic authors and documents mentioning sexual orientations, including non-offensive content.The authors emphasize that toxicity depends on social context and cannot be reliably identified by detecting isolated words.
- Limitations and Recommendations: The authors have examined only some possible issues in C4.EN and leave personally identifiable information, copyrighted text, and cross-language generalization for future work.They recommend downloadable data and a location where others can report additional issues.
7 Related Work
Related work situates C4.EN among increasingly large pretrained-text corpora and post-hoc analyses addressing the documentation gap created by web-scale data collection.
- Pretraining corpora: BERT, RoBERTa, GPT-3, and later corpora such as PILE, CCNet, and mC4 illustrate the expansion of pretrained datasets built from curated or web-crawled text.The cited examples include BOOKSCORPUS, Wikipedia, CC-NEWS, OPENWEBTEXT, STORIES, and Common Crawl.
- Corpus documentation: Documentation for massive pretraining corpora is often missing because analyzing datasets of their size is challenging.Researchers responded with systematic post-hoc studies of toxicity, fake news, and multilingual web-crawled corpora.
8 Conclusion
The paper presents early documentation and analysis of C4.EN, arguing that web-scraped datasets should be examined across metadata, included text, and excluded text while remaining broadly accessible.
- 8 Conclusion: The authors present some of the first documentation and analyses of C4.EN, a web-scale unlabeled dataset introduced by Raffel et al. (2020).They frame the work as an examination of datasets created by scraping the web and filtering text.
- 8 Conclusion: They advocate analyzing metadata, included data, and excluded data, and provide three downloadable data versions, an indexed version, and a discussion repository.These resources are intended to support accessibility, searching, and public discussion of findings.
9 Societal and Ethical Implications
The paper links webtext construction choices to harms affecting particular communities and argues that better documentation can help identify and address problems in widely used language-model datasets.
- 9 Societal and Ethical Implications: Blocklist filtering can cause allocational harms by disproportionately removing minority-related content from large webtext corpora.The paper identifies this as a consequence of a specific dataset design choice.
- 9 Societal and Ethical Implications: Passively crawled webtext can cause representational harms through disparate cooccurrences of geographic origins with negative sentiment.The paper presents this as an issue affecting specific demographic identities.
- 9 Societal and Ethical Implications: Better documentation of web-crawled and other massive language-modeling datasets can help find and solve issues in models used in production.The stated concern is especially relevant when such systems impact many people.
A.1 Tokenization
The C4-derived English vocabulary excludes blocklisted tokens, producing unexpected tokenizations for some words.
- The T5 SentencePiece vocabulary contains 32,000 word pieces trained from a 10:1:1:1 English:French:German:Romanian mixture.
- Because the English vocabulary comes from cleaned English C4, blocklisted tokens are absent and “sex” may tokenize as “s” + “ex”.
A.2 Geolocation
The supplied passages describe geographic and source-domain analyses of C4, alongside contamination checks and analyses of filtered text and identity-related bias.
- Classification label contamination: The contamination analysis found no input-and-label contamination of standard NLP classification benchmarks in the unfiltered Common Crawl.
- Filtered Text Clustering and Analysis: Blocklist-filtered documents include pornography and hate speech clusters but also medicine, religion, gaming, and infant-care topics.
- Filtered Text Clustering and Analysis: The filtered-document analysis used k-means clustering with k=50 on 100,000 randomly sampled documents represented with TF-IDF.
- Identity and sentiment analysis: The corpus study reported 73.2% positive sentiment among 3.4M sentiment-bearing tokens for “Jewish” in the SocialSent-derived lexicon.