Source-linked AI summary
What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpus
Alexandra Sasha Luccioni, Joseph D. Viviano
TL;DR
Because the massive corpora used to train language models have received comparatively little analysis, this paper examines a sample of Common Crawl for undesirable content. It finds explicit and abusive content despite filtering, and discusses implications for corpus analysis and collection.
Problem
Comparatively little research has analyzed the massive corpora used to train language models, in part because their size makes meaningful analysis challenging.
Source: §2 · p. 2Method
The authors randomly sampled 1% of Common Crawl—roughly 115 GB or 5,835,339 pages—to analyze hate speech, adult content, and perplexity-based filtering efficacy.
Source: §3 · p. 2Results
The analysis found explicit and abusive content despite filtering, and perplexity filtering removed many sexually explicit sites but missed much hate speech flagged by other methods.
Source: §1 · p. 1, §3.3 · p. 3Takeaways & Limitations
The authors call for more mindful data collection, standardized corpus-filtering pipelines, and methods to evaluate bias in trained models.
Source: §4.2 · p. 5Perplexity filtering is not a general-purpose exclusion rule: lower-quality texts may still be useful for specific applications, motivating more nuanced filtering metrics.
Source: §3.3 · p. 4
Abstract
from arXiv · showhide
Whereas much of the success of the current generation of neural language models has been driven by increasingly large training corpora, relatively little research has been dedicated to analyzing these massive sources of textual data. In this exploratory analysis, we delve deeper into the Common Crawl, a colossal web corpus that is extensively used for training language models. We find that it contains a significant amount of undesirable content, including hate speech and sexually explicit content, even after filtering procedures. We discuss the potential impacts of this content on language models and conclude with future research directions and a more mindful approach to corpus collection and analysis.
1 Introduction
The introduction presents the Common Crawl as a major non-curated training source whose content merits scrutiny. The paper’s initial analysis finds explicit and abusive content even after filtering.
1 Introduction
- Large language models require increasingly large training datasets, often combining curated material with non-curated web content.The introduction notes that this non-curated category is becoming increasingly popular.Source: §1 · p. 1
- Common Crawl releases are multilingual web snapshots containing 200 to 300 TB of textual content each month.The introduction describes it as the current largest dataset used to train neural language models.Source: §1 · p. 1
- Common Crawl is contrasted with roughly 5.6 TB of Wikipedia data and around 6 GB in BookCorpus.The introduction says Common Crawl often represents the majority of data used to train these architectures.Source: §1 · p. 1
- The initial analysis identifies explicit and abusive content in the Common Crawl even after filtering.The authors frame this as relevant to the potential downstream impact of corpus content on language models.Source: §1 · p. 1
2 Related Work
Research has examined bias in language models, but comparatively little work has analyzed the massive corpora used to train them. Prior Common Crawl studies surfaced questionable content, motivating this paper’s broader investigation of corpus content and quality.
2 Related Work
- Bias detection and removal have received substantial attention, although the extent to which bias can be removed remains debated (Gonen and Goldberg, 2019).Prior research also linked language-model biases to training data and hate speech on the Internet (Solaiman et al., 2019; Abid et al., 2021).Source: §2 · pp. 1–2
- The massive size and heterogeneity of web corpora make them difficult and costly to mine, and prior corpus analyses had only scratched the surface.The passage cites a survey describing automatic web-page classification as difficult partly because of its computational cost.Source: §2 · p. 2
- Prior work found over 300,000 documents from unreliable news sites and banned subReddit pages containing hate speech and racism in Common Crawl (Gehman et al., 2020).Other research showed that individual training examples could be extracted by querying language models (Carlini et al., 2020); the authors use these concerns to motivate studying corpus content and quality.Source: §2 · p. 2
3 Analyzing the Common Crawl
The analysis examines a 1% Common Crawl sample for hate speech and sexually explicit content, including whether perplexity filtering detects such material. The detection methods flag these content types, but perplexity filtering misses much of the hate speech they identify.
3 Analyzing the Common Crawl
- The study randomly sampled 1% of Common Crawl files, yielding about 115 GB and 5,835,339 web pages after language filtering, and focused on hate speech and sexually explicit content.The authors frame these as examples of undesirable content, noting that desirable model behaviour is application-specific.Source: §3 · p. 2
3.1 Detecting Hate Speech
- 5.24% of sampled websites were flagged by DELIMIT, 4.02% by HateSonar, and 6.38% by the n-gram approach.Qualitative analysis also found adult content mentioning violent acts toward specific ethnic groups among flagged sites.Source: §3.1 · pp. 2–3
3.2 Sexually Explicit Content
- The n-gram approach flagged 2.36% of pages for at least one listed term, 1.36% for three or more, and 0.73% for ten or more.No statistical or machine-learning approach was available to compare with this count-based detection method; the authors note that even a small share of undesirable inputs cannot guarantee models will avoid similar biased outputs in response to specific prompts.Source: §3.2 · p. 3
3.3 Filtering by Perplexity Score
- Perplexity filtering reduced detected sexual content but did not detect much of the hate speech flagged by other methods, and its scores correlated weakly with all tested detection methods.Table 3 compares hate-speech and sexual-content rates across the entire corpus and high- and low-quality sites.Source: §3.3 · p. 3, Figure 1 · p. 4, Table 3 · p. 4
- The authors note that lower-quality texts may still be useful for specific applications and suggest more nuanced filtering, potentially combining embedding-based and statistical approaches.They report exploring modifications to the existing perplexity-based approach.Source: §3.3 · p. 4
3.4 Behaviour of Different Detection Methods
- The three methods use different features; n-grams agreed more with HateSonar than with DELIMIT, and each identified clear hate speech missed by the others.The authors conclude that they are far from a general-purpose dataset-filtering approach and argue that undesirable training content should be taken seriously.Source: §3.4 · p. 4
4 Discussion
The authors frame the analysis as an incomplete, resource-constrained first look and raise questions about whether Common Crawl’s discourse should shape language models. They call for scalable corpus analysis and further work on filtering, output control, and more transparent data practices.
4.1 Summary of Results
- The exploratory analysis is only an initial step: Common Crawl-scale analysis exceeds most research institutions’ resources and calls for scalable, thorough, nuanced approaches.The authors say a small curated dataset would have allowed a more comprehensive analysis, but methods must scale to the data needs of modern language models.Source: §4.1 · pp. 4–5
- Common Crawl’s representation and discourse raise questions about whether it is the corpus society wants shaping consequential language models.It over-represents younger, English-speaking users in developed countries; anonymous internet communities can amplify toxic discourse and limit broader representativeness.Source: §4.1 · p. 5
4.2 Future Work
- Future work should examine undesirable content’s downstream effects, the effects of filtering it, and ways to regularize outputs regardless of training data.Source: §4.2 · p. 5
- The authors call for scalable detection that generalizes to Common Crawl and propose testing labeled-example control, human-in-the-loop fine-tuning, and task-focused models.They also caution that curation may not prevent undesirable generations assembled from distinct innocuous examples.Source: §4.2 · p. 5
- They advocate mindful data collection, standardized filtering, bias evaluation, and corpus transparency, while rejecting a one-size-fits-all solution.They argue that data quality and content affect models’ caliber and appropriateness across downstream applications.Source: §4.2 · p. 5