Source-linked AI summary
What's In My Big Data?
Yanai Elazar, Akshita Bhagia, Ian Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hanna Hajishirzi, Noah A. Smith, Jesse Dodge
TL;DR
Large language-model corpora are massive but poorly understood across statistics, quality, social factors, and evaluation contamination. WIMBD addresses this gap with scalable counting and search, applying sixteen analyses to ten corpora. The analyses reveal substantial duplication and other undocumented content issues, while providing reusable infrastructure for corpus documentation and inspection.
Problem
Massive language-model training corpora are widely used but insufficiently understood across their statistics, quality, social factors, and evaluation-data contamination.
Method
WIMBD combines scalable counting and search to perform sixteen analyses of ten large text corpora used to train language models.
Results
About 50% of documents in RedPajama and LAION-2B-en are duplicates, alongside findings of low-quality and synthetic content, personally identifiable information, toxic language, and benchmark contamination.
Takeaways & Limitations
WIMBD provides reusable analyses and artifacts intended to support corpus comparison, data documentation, and more transparent inspection of new text-based corpora.
Takeaways & Limitations
When evaluating models trained on an existing corpus, contaminated evaluation sets should be exempted; for new corpus construction, WIMBD can support decontamination.
Abstract
from arXiv · showhide
Large text corpora are the backbone of language models. However, we have a limited understanding of the content of these corpora, including general statistics, quality, social factors, and inclusion of evaluation data (contamination). In this work, we propose What's In My Big Data? (WIMBD), a platform and a set of sixteen analyses that allow us to reveal and compare the contents of large text corpora. WIMBD builds on two basic capabilities -- count and search -- at scale, which allows us to analyze more than 35 terabytes on a standard compute node. We apply WIMBD to ten different corpora used to train popular language models, including C4, The Pile, and RedPajama. Our analysis uncovers several surprising and previously undocumented findings about these corpora, including the high prevalence of duplicate, synthetic, and low-quality content, personally identifiable information, toxic language, and benchmark contamination. For instance, we find that about 50% of the documents in RedPajama and LAION-2B-en are duplicates. In addition, several datasets used for benchmarking models trained on such corpora are contaminated with respect to important benchmarks, including the Winograd Schema Challenge and parts of GLUE and SuperGLUE. We open-source WIMBD's code and artifacts to provide a standard set of evaluations for new text-based corpora and to encourage more analyses and transparency around them.
1 INTRODUCTION
WIMBD addresses the limited understanding of massive language-model training corpora by enabling comparable, large-scale analysis of their contents. Its count-and-search tools support sixteen analyses across ten corpora, revealing anomalies and undocumented data properties.
- Motivation: Massive web-scale corpora are increasingly used to train models, yet their contents—including topics, toxicity, and personal information—are poorly documented.Their scale and limited public availability make systematic analysis difficult.
- Approach: WIMBD combines programmatic search with map-reduce counting to explore and analyze large language datasets at scale.The framework supports document retrieval, frequency analysis, length distributions, duplicates, domains, and personally identifiable information.
- Evaluation: WIMBD applies sixteen analyses across ten English corpora used to train language models, spanning statistics, quality, society-related measures, and cross-data analysis.The analyses include corpus statistics, data quality, community and society measurements, and cross-corpus comparisons.
- Findings: Document-length distributions expose overrepresented lengths associated with near-duplicate templates or arbitrary truncation.Punctuation sequences can also dominate n-gram statistics, such as ten repeated dashes being The Pile’s most common 10-gram.
2 BACKGROUND: ON THE IMPORTANCE OF DATA UNDERSTANDING
Understanding training data matters because language models affect people’s lives and learn only from their pretraining corpora. Existing work and tools provide incomplete, non-comparable coverage of massive datasets, motivating broader programmatic analysis.
- The gap: Web-scale datasets are difficult to understand because their massive size and limited transparency hinder comprehensive content analysis.Even shared training data would be challenging to analyze in its entirety.
- Prior limitations: Prior efforts typically analyze selected dimensions or a single corpus, often without programmatic access or comparable cross-dataset analysis.WIMBD combines search, counting, wrappers, and extendable counting capabilities to address this gap.
- Why data matters: Language-model behavior and capabilities depend on pretraining data, making corpus understanding important as these systems become widespread.Training data can shape factual recall, frequency-sensitive performance, similarity effects, and biases.
3 WIMBD: THE PLATFORM
WIMBD is a scalable platform built around counting and search for analyzing terabyte-scale corpora. It combines exact and compressed counting with Elasticsearch retrieval to support statistics, quality checks, and other corpus analyses.
- Platform overview: WIMBD’s two core capabilities—counting and search—enable fast processing and retrieval for large-corpus analyses.The platform uses standard information-retrieval and data-management methods on a compute node and Elasticsearch cluster.
- Counting: Map-reduce supports counting when language-data sparsity and corpus scale make direct computation challenging.Exact counts handle tractable value sets such as document numbers and lengths.
- Counting: Compressed counts use hashing or Bloom-style compression when possible values or document strings are too numerous to keep in memory.This approach targets large n-gram spaces and duplicate detection, including cases where direct storage would exceed available memory.
- Searching: Elasticsearch inverted-index search retrieves documents and frequencies for words, sequences, and Boolean queries.WIMBD wraps the Elasticsearch API to provide customized searches for its analyses.
4 WIMBD: THE ANALYSES
WIMBD applies sixteen scalable analyses to ten large corpora, exposing differences in corpus composition, quality, duplication, social risks, and benchmark contamination. The analyses reveal substantial repetition, templated or anomalous content, personally identifiable information, toxic language, and evaluation-data overlap.
- Scope: WIMBD extends corpus analysis across ten large datasets by combining statistics, quality checks, and community- and society-relevant measurements.The analyses cover corpora ranging from text-only datasets to image captions and code.
- Summary statistics: Domain distributions vary sharply: C4's most common domain contributes under 0.05%, whereas LAION-2B-en's cdn.shopify.com contributes over 6% and RedPajama's arxiv.org over 12%.The reported percentages concern documents or tokens according to the corresponding domain-counting analysis.
- Data quality: Over 50% of documents are duplicates in The Pile, while RedPajama and LAION-2B-en each contain about 50% duplicate documents.The Pile's duplication is attributed to intentional oversampling; C4 has no exact duplicates after deduplication.
- Data quality: Repeated punctuation, boilerplate, and formatting dominate many common n-grams, with ten repeated question marks appearing 9 million times in C4.Manual inspection classified 68% of sampled occurrences as writing style, 18% as noise, and 14% as formatting.
- Data quality: Document-length outliers expose template-generated near-duplicates, truncated datasets, and short autogenerated snippets across C4, The Pile, and OSCAR.C4 contains template-generated reverse-phone-lookup and JavaScript documents, while nearly 1% of The Pile is exactly 8,194 characters long.
- Community- and society-relevant measurements: The corpora contain substantial social and evaluation risks, including toxic language in 1–16.5% of documents, large quantities of PII, and contamination of important benchmarks.RedPajama is above 50% contamination for eight of 15 examined datasets and is fully contaminated for COPA; 67 of 82 examined datasets were not found.
5 DISCUSSION
The discussion frames data as a critical but poorly understood foundation of machine learning and argues that WIMBD can support better curation, documentation, and grounding of language models.
- Data amount and quality are crucial for successfully training state-of-the-art language models, not merely model size.
- Data Curation: WIMBD is intended to ease the costly decisions involved in crafting high-quality large-scale datasets.
- Data Documentation: The authors argue that large pretraining corpora should be documented alongside existing dataset-documentation practices.
- Grounding Models to their Training Data: Because training data shares language with model outputs, it offers a distinctive basis for measuring and grounding model behavior.
6 CONCLUSION
WIMBD analyzes ten model-training corpora through sixteen analyses spanning statistics, quality, social factors, and cross-data comparisons. It reveals substantial duplication and benchmark contamination while providing an extensible framework for further corpus analysis.
- About 50% of documents in RedPajama and LAION-2B-en are duplicates, while WSC, WIC, and RTE appear in corpora such as RedPajama.
- WIMBD covers ten corpora with sixteen analyses across high-level statistics, data quality, community- and society-relevant measurements, and cross-data analysis.
- The framework is extendable for reproducing analyses, developing new ones, and investigating research questions about training data.
A CORPORA: ELABORATION
The appendix describes ten heterogeneous corpora and documents WIMBD’s corpus coverage, metadata, domain analyses, and URL-related findings. Its analyses span text, image-caption, code, and academic-paper collections, with English-focused treatment for mC4 and LAION.
- WIMBD covers ten corpora spanning text-only data, image captions, and source code, with metadata including text type, URLs, scrape dates, and addition dates.
- The analysis fully covers corpus subsets, but focuses on the English portions of mC4 and LAION; toxic-language analysis specifically relies on English resources.
- C4 and mC4-en are Common Crawl-derived corpora, while OpenWebText reproduces GPT-2 training data and RedPajama reproduces LLaMA data.
- The Pile contains 22 pre-selected domains, including Wikipedia, GitHub, arXiv, and EuroParl, with many domains upsampled to increase quality.
- Across URL-bearing corpora, HTTPS accounts for 62.5% to 87.6% of URLs, with OSCAR highest and C4 lowest.
- The most common URL suffix is com, ranging from 60.1% of OSCAR documents to 77.5% of LAION-2B-en documents, alongside a long tail exceeding 3,000 suffixes.
B.2.1 MOST & LEAST COMMON n-GRAMS
The corpus n-gram analyses examine common and rare textual patterns, revealing formatting artifacts, multilingual or unusual Unicode content, and substantial lexical diversity. Duplicate analyses separately compare text and URL repetition across datasets.
- Stop words and punctuation dominate common unigrams, while OSCAR and The Pile contain frequent uncleaned HTML and Markdown sequences.
- Rare-unigram samples reveal non-standard Unicode fonts and multilingual strings, including twelve non-English languages in OpenWebText.
- OpenWebText contains over 88 million unique unigrams, while LAION-2B-en contains over 554 million, representing 1.1% and 1.8% of their total unigrams respectively.
- Duplicates: URL duplicate ratios are roughly an order of magnitude smaller than text-hash duplicate ratios and are not dominated by only a few clusters.
- Duplicates: Figure 14 compares duplicate-document and duplicate-cluster percentages, with duplicate counts shown above bars for datasets containing duplicates.
B.2.3 DOCUMENT LENGTH DISTRIBUTION
Document-length analysis reveals dataset-specific anomalies, including truncation, template-generated material, and repeated metadata patterns. These distributions expose corpus-construction artifacts beyond ordinary variation in document size.
- B.2.3 DOCUMENT LENGTH DISTRIBUTION: LAION-2B-en has the smallest average document lengths and contains many duplicate product descriptions and location-related alt texts.These patterns indicate that its document-length distribution reflects image-caption and template-based sources.
- B.2.3 DOCUMENT LENGTH DISTRIBUTION: The analysis also identifies potentially sensitive or problematic content, including RedPajama pages linked to individual names and documents associated with pornographic content.The latter collection comprises nearly 0.01% of RedPajama.
- B.2.3 DOCUMENT LENGTH DISTRIBUTION: The Stack contains many 20-token Unity metadata files and numerous 20,000-character documents containing float and bit matrices.These repeated lengths are consistent with structured or automatically generated source files.
- B.2.3 DOCUMENT LENGTH DISTRIBUTION: Template-generated content appears across corpora, including Unity metadata, medical-publication records, rental-boat alt texts, and placeholder pages.The Pile includes short Unity and medical-journal metadata files; LAION-2B-en contains template-generated alt texts; RedPajama contains templated alumni pages.
B.4 CROSS-DATA ANALYSIS
Cross-corpus analyses compare unigram distributions and exact-text overlap to expose shared sources and distinctive corpus composition. They show both broad similarity among web corpora and strong divergence for specialized collections.
- B.4 CROSS-DATA ANALYSIS: Fifty percent of RedPajama’s unique documents originate from C4, and 50% of OpenWebText’s unique documents originate from The Pile.These overlaps reveal substantial shared content despite differences in corpus construction and size.
- B.4 CROSS-DATA ANALYSIS: OpenWebText, C4, mC4-en, OSCAR, The Pile, and RedPajama have similar common-unigram distributions, whereas S2ORC, peS2o, LAION-2B-en, and The Stack differ.The Jensen-Shannon comparison summarizes similarity among the top 1,000 common unigrams.
- B.4 CROSS-DATA ANALYSIS: OSCAR’s unigram distribution is most similar to the other corpora on average, with Jensen-Shannon distance 0.19, while The Stack is most distant.The comparison uses the 10,000 most common unigrams from each corpus.
B.4.2 CORPUS OVERLAP
Exact-text and URL comparisons reveal substantial overlap among some corpora but little overlap as more datasets are considered. Formatting differences can obscure shared source material even between closely related datasets.
- B.4.2 CORPUS OVERLAP: Exact text overlap diminishes quickly toward zero as more datasets are included, limiting the size of broad multi-corpus intersections.The analysis compares both text and URL overlap pairwise.
- B.4.2 CORPUS OVERLAP: S2ORC and peS2o have no exact-text overlap despite peS2o being a cleaned version of S2ORC, likely because parsed-paper formatting differs.The paper notes that related academic papers may still appear in other corpora despite failing exact string matching.
- B.4.2 CORPUS OVERLAP: Half of RedPajama’s unique documents originate from C4, while half of OpenWebText’s unique documents overlap with The Pile.The paper attributes these overlaps to larger documents or duplicated C4 content in RedPajama and shared sourcing in OpenWebText and The Pile.
C LIMITATIONS
WIMBD’s limitations concern the cost and scalability of Elasticsearch indexing and the narrowness of its exact-match search. The reported benchmark timings quantify the computational setting but do not remove these infrastructure constraints.
- C LIMITATIONS: Elasticsearch indexing of massive corpora can take days and is costly to keep running, motivating future work on faster, cheaper indexing tools.The limitation concerns both preprocessing time and ongoing infrastructure expense.
- C LIMITATIONS: WIMBD currently supports exact-match search only, not fuzzy or semantic search.This restricts retrieval to literal string matches rather than approximate or meaning-based matches.
- C LIMITATIONS: The reported analyses ran on a 224-CPU, 881-GB-memory machine, with costs estimated at $9.46 per hour.The timing table provides the computational context for the analyses and contamination experiment.
D BENCHMARKING RUNTIMES
WIMBD’s analyses run on a CPU-rich, 882GB-RAM node, with most individual analyses completing within 12 hours and the full C4 workload costing about $443. The n-gram analyses trade exactness for substantially lower memory use through hash-based counting, with distinct error directions for top-k, bottom-k, and uniqueness estimates.
- Runtime and cost: All analyses took less than 12 hours individually, while 13 of 22 completed in several minutes and the C4 analyses totaled an estimated 46 hours and 51 seconds.Run times were measured with Linux TIME and should be treated as rough estimates because of variance.
- Runtime and cost: $443.1.15 was the estimated total cost for all C4 analyses, using a cloud rate of $9.46 per hour.The estimate excludes Elasticsearch hosting costs.
- Approximate counting: Top-k n-gram counts are upper bounds because collisions inflate some counts and can introduce false positives, with accuracy governed by the memory allocated to the hash table.The method is correct only in the absence of hash collisions.
- Approximate counting: Bottom-k n-grams require two corpus passes and can miss rare items after collisions, while their reported counts are lower bounds on the true counts.The first pass builds counts; the second selects n-grams with sufficiently low counts.
- Approximate counting: The unique n-gram estimate counts occupied hash-table entries and therefore remains a lower bound on the actual number of unique n-grams.Each observed n-gram sets its hashed boolean entry to true before occupied entries are tallied.