What's In My Big Data?

• Large training corpora often lack content documentation, while existing analyses rarely compare multiple datasets across shared dimensions, leaving practitioners without practical tools to assess differences.The paper notes that corpus analysis is also hindered by limited public release and the massive size of these datasets.Source: §1 · p. 1
• WIMBD combines search for queried strings, matching documents, and occurrence counts with map-reduce counting for corpus-wide statistics, duplicates, domains, and personal information.The framework is designed to be extendable to indexing, counting, and analyzing other corpora at scale.Source: §1 · p. 2, Figure 1 · p. 2
• WIMBD conducts sixteen analyses across ten English language-model corpora, covering data statistics, quality, societal measurements, and cross-corpus comparisons.Examples include token and domain distributions, duplicates, benchmark contamination, personally identifiable information, and document overlap.Source: §1 · p. 2
• Document-length spikes can signal template-generated or truncated text, while The Pile’s most common 10-gram is a dash repeated ten times.WIMBD presents these as examples of anomalies and unusual data distributions that can support higher-quality corpus curation.Source: §1 · p. 2

• Despite calls for better data documentation, impactful models are increasingly opaque about their training-data sources, size, and contents.Source: §2 · p. 2
• Web-scale datasets make comprehensive analysis difficult, and existing documentation tools target smaller datasets rather than this scale.The authors describe this as a critical missing component they aim to address.Source: §2 · p. 2
• Existing large-corpus efforts often cover one corpus and lack programmatic access, whereas WIMBD combines search and counting with API wrappers and extendable efficient counting.Source: §2 · p. 3
• Prior studies examine how data affects model behavior using dedicated infrastructure; WIMBD provides shared tooling for a wide range of large-corpus analyses.The cited studies address pretraining composition and performance, term frequency and reasoning, in-context abilities, learned biases, fact tracing, and counterfactual training simulations.Source: §2 · p. 3

3 WIMBD: THE PLATFORM

• WIMBD combines counting and search to support analyses of large text corpora.Table 1 summarizes the platform's capabilities and the analyses they enable.Source: §3 · p. 3, Table 1 · p. 3

3.1 COUNTING

• WIMBD uses MapReduce for exact counts when values fit in memory and compressed counts when the value space is too large.Compression reduces memory use but may sacrifice accuracy through hash collisions; larger hash tables reduce that risk. Finding C4 duplicates directly would require over 800 GB of RAM.Source: §3.1 · p. 3

3.2 SEARCHING

• WIMBD uses an Elasticsearch inverted index for document-frequency queries and more complex Boolean searches.A wrapper over the Elasticsearch API supports tailored searches; other search alternatives are left for future work.Source: §3.2 · pp. 3–4

4.2.1 SUMMARY STATISTICS

• mC4-en is the largest corpus at 14.7TB and 2.7 trillion tokens, while The Stack is 7.8TB and exceeds 1.5 trillion tokens; four corpora contain empty documents.The Pile and RedPajama share the same longest document, an encyclopedia book of over 28 million tokens.Source: §4.2.1 · pp. 4–5, §4.2 · p. 4

4.2.2 INTERNET DOMAIN DISTRIBUTION

• Domain distributions vary sharply: C4’s most common domain, patents.google.com, contributes less than 0.05%, while cdn.shopify.com accounts for over 6% of LAION-2B-en documents and arxiv.org over 12% of RedPajama documents.The domain analysis also counts domains per token and per document.Source: §4.2.2 · p. 5, Figure 2 · p. 5

4.3.2 DUPLICATES

• More than 60% of The Pile’s documents are duplicates, while RedPajama and LAION-2B-en each have about 50% duplicate documents.The Pile has 139M documents in 64.6M duplicate-text clusters; LAION-2B-en has 1.25B documents in 342M clusters. C4 has no exact duplicates.Source: §4.3 · p. 5, §4.3.2 · p. 6

4.3.1 MOST & LEAST COMMON n-GRAMS

• Common n-grams reveal uncleaned markup and boilerplate, while ten repeated question marks, periods, and hyphens occur 9M, 7.2M, and 4.4M times in C4.Among 100 sampled C4 documents with ten consecutive question marks, 68% used them stylistically, 18% reflected noise, and 14% reflected formatting.Source: §4.3.1 · pp. 5–6

4.3.3 DOCUMENT LENGTH DISTRIBUTION

• Document-length outliers expose template-generated near-duplicates in C4 and OSCAR, plus unusual truncated mathematics documents and short generated snippets in The Pile.Nearly 1% of The Pile’s documents are exactly 8,194 characters long, reflecting truncated DeepMind Mathematics data; OSCAR has no documents shorter than 100 characters because they were filtered.Source: §4.3.3 · p. 7

4.4 COMMUNITY- AND SOCIETY-RELEVANT MEASUREMENTS

• Toxicity affects 1–16.5% of documents by automatic classifier and 0.01–16.6% by taxonomy; mC4-en contains an estimated 200M email addresses, 4B phone numbers, and 97M IP addresses.mC4-en has the largest PII volume even when controlling for token count.Source: §4.4 · p. 7, §4.4.2 · p. 8

4.4.1 BENCHMARK CONTAMINATION

• RedPajama is the most contaminated of four corpora: eight of 15 reported rates exceed 50%, and COPA is fully contaminated (Roemmele et al., 2011).Overall, 67 of the 82 examined benchmark datasets were not found in the investigated corpora.Source: §4.4.1 · p. 8

• Despite being comparatively understudied (Sambasivan et al., 2021), data amount and quality are critical to state-of-the-art language-model training, beyond model scaling alone (Kaplan et al., 2020).Source: §5 · p. 9

Data Curation

• WIMBD can ease large-scale dataset curation by surfacing content patterns for filtering, and its insights informed Dolma iterations that found noisy common n-grams and de-duplication bugs (Soldaini et al., 2024).Assessing curation decisions is costly because training large models requires substantial computational resources.Source: §5 · p. 9

Data Documentation

• WIMBD advocates tailored documentation for large-scale pretraining corpora beyond supervised-dataset templates (Gebru et al., 2021; McMillan-Major et al., 2023), extending automatic full-corpus analyses (Dodge et al., 2021; Gao et al., 2020).Source: §5 · p. 9

Grounding Models to their Training Data

• Because training data shares models’ natural-language format, it can ground behavior: factual recall derives from training data (Jiang et al., 2020; Elazar et al., 2021a), while performance favors frequent and similar examples (Razeghi et al., 2022a; McCoy et al., 2023; Longpre et al., 2023).Holistic understanding of model behavior requires infrastructure to access large datasets and suitable abstractions of data attributes.Source: §5 · p. 9

6 CONCLUSION

• About 50% of RedPajama and LAION-2B-en documents are duplicates, and several GLUE and SuperGLUE benchmarks, including WSC, WIC, and RTE, appear in corpora such as RedPajama.The analyses span high-level statistics, data quality, community- and society-relevant measurements, and cross-data analysis.Source: §6 · p. 9
• The most common LAION-2B-en text sources include Pinterest, Shopify, SlidePlayer, Amazon, and eBay.Source: §6 · p. 9
• WIMBD studies ten corpora used to train language models or vision-and-language models and provides an extendable framework for reproducing and developing analyses.The authors release WIMBD’s code and artifacts to encourage analysis of existing and new corpora.Source: §6 · p. 9
• An open question is what the instances in large pretraining corpora represent, such as documents, photos, people, or countries.Source: §6 · p. 9

A CORPORA: ELABORATION

• The study covers ten corpora, including web text, image captions, academic papers, and code, and focuses on English subsets of mC4 and LAION.Most analyses are not language-dependent; toxic-language analysis relies on an English lexicon and classifier, though the authors say it can be repeated with non-English resources.Source: A CORPORA: ELABORATION · pp. 18–19

B ADDITIONAL RESULTS; B.2 DATA QUALITY; 3. Document length distribution (§4.3.3, §B.2.3)

• The appendices provide extended corpus results across data statistics, data quality, social measurements, and cross-data analysis, including quality variations such as token-level length distributions.The supplied passages state the scope of these analyses but do not report the length-distribution results.Source: B ADDITIONAL RESULTS · p. 20, §B.2 · p. 26

B.1.1 INTERNET DOMAIN DISTRIBUTION; B.1 DATA STATISTICS

• The 1% quantile of LAION-2B-en domain token counts includes domains with one or fewer tokens.The analysis counts domains by document and by token across five URL-bearing corpora and reports token-count quantiles.Source: §B.1.1 · p. 20, Table 7 · p. 20

B.1.2 INTERNET DOMAIN SCHEMES

• 87.6% of OSCAR URLs use HTTPS, the highest share reported, while C4 is lowest at 62.5%.All corpora with URL information fall below Google’s reported 95% HTTPS traffic share.Source: §B.1.2 · pp. 20–21, Figure 7 · p. 24

B.1.3 INTERNET DOMAIN SUFFIXES

• The .com suffix accounts for 60.1% of OSCAR documents to 77.5% of LAION-2B-en documents, and corpus suffix distributions span over 3,000 suffixes.LAION-2B-en’s top ten also include Germany (de, 0.7%), Russia (ru, 0.5%), France (fr, 0.4%), and Italy (it, 0.4%).Source: §B.1.3 · p. 21, Figure 8 · p. 24

B.1.4 UTTERANCE DATE STATISTICS

• RedPajama and OSCAR are dominated by documents created in the five years before September 2023, while S2ORC and peS2o contain a non-negligible pre-internet fraction.For corpora without reliable creation timestamps, the study estimates dates from Internet Archive indexing or dataset ingestion, whichever is earlier, using a 10,000-document random sample; these estimates are lower bounds.Source: §B.1.4 · p. 21, Figure 9 · p. 25

B.1.5 GEOLOCATION

• Most pages across the five URL-bearing corpora are hosted in the United States, with most of the remainder in the Anglosphere.The study estimates origin from 100,000 randomly sampled URLs per corpus, while noting that hosting location is only a rough proxy for content-creator location.Source: §B.1.5 · p. 21, Figure 10 · p. 25

B.1.6 LANGUAGE DISTRIBUTION

• The study uses CLD2 labels to aggregate corpus language distributions, but notes that the classifier can misidentify languages.The text points to Table 8 for English-document percentages and Figure 11 for non-English language shares; no values are supplied in these passages.Source: §B.1.6 · p. 22, Table 8 · p. 22, Figure 11 · p. 25

B.2.1 MOST & LEAST COMMON n-GRAMS

• Trigrams diverge across corpora: C4 features common English expressions, LAION-2B-en has marketing phrases, and OSCAR and The Pile include uncleaned HTML or markdown strings.Examples include “one of the” in C4, “for sale in” in LAION-2B-en, and “###” among the uncleaned strings.Source: §B.2.1 · p. 26
• Stop words and punctuation are the most common unigrams across datasets, while “of the” and “in the” recur as bigrams in all corpora.Bigrams differ more across corpora; LAION-2B-en includes marketing mentions such as “for sale” and “- Shirt”.Source: §B.2.1 · p. 26
• Random samples of rare unigrams show non-standard Unicode fonts and diverse non-English strings; OpenWebText’s sample includes strings from 12 languages besides English.The listed languages are Urdu, Arabic, Korean, Sanskrit, Hebrew, Armenian, Bengali, Persian, Japanese, Latvian, Sindhi, and Russian.Source: §B.2.1 · p. 26, Figure 12 · p. 28, Figure 13 · p. 29
• OpenWebText has more than 88 million unique unigrams, about 1.1% of its total, while LAION-2B-en has over 554 million, or 1.8%.The unique-unigram ratio is about an order of magnitude smaller in the other corpora, except LAION-2B-en.Source: §B.2.1 · p. 26, Table 10 · p. 27

B.2.2 DUPLICATES

• Table 11 lists the five most frequent text duplicates for datasets with duplicates and notes that OpenWebText and C4 have no duplicate documents.The table truncates some displayed duplicate text.Source: Table 11 · p. 30
• URL duplicate ratios are roughly an order of magnitude smaller than text-hash ratios, and URL-duplicated documents are not dominated by a few clusters.The comparison covers datasets with URL metadata and non-zero URL duplication.Source: §B.2.2 · p. 30
• LAION’s most frequent URL occurs 33,142 times but is invalid, while its second-most frequent URL occurs 27,162 times and leads to an all-white image.Among LAION-2B-en’s top 25 duplicated URLs, some duplicate images are hosted at different URLs; in OSCAR, the 234 URLs after the top two are each duplicated exactly 100 times.Source: §B.2.2 · p. 30, Table 12 · p. 30

B.2.3 DOCUMENT LENGTH DISTRIBUTION

• LAION-2B-en has the smallest average document lengths and includes template-generated alt texts paired with rental-boat location maps; OpenWebText documents over 100,000 characters are chunked.The only OpenWebText document-length outlier is exactly 100,000 characters, the builders’ chunking length.Source: §B.2.3 · pp. 31–32
• RedPajama, The Stack, and The Pile contain template-generated material, including school-alumni placeholders, Unity metadata, and files corresponding to medical-journal publications.RedPajama also has nearly 0.01% of documents that appear to be usernames or titles associated with pornographic content; The Stack includes 20-token Unity metadata files and 20,000-character float and bit matrices.Source: §B.2.3 · p. 32

B.3 COMMUNITY- AND SOCIETY-RELEVANT MEASUREMENTS

• Community- and society-relevant measurements include additional contamination and PII analyses, alongside toxic-language and demographic sentiment co-occurrence analyses.These analyses extend the results reported in the main paper.Source: §B.3 · p. 34

B.3.1 BENCHMARK CONTAMINATION

• Contamination is an upper-bound exact-match measure: all input fields must occur in one document, but labels and input order are ignored.The study uses PromptSource because it can be processed automatically at scale and distinguishes task inputs from metadata; Table 16 reports percentages for 82 datasets across four corpora, but its values are not provided here.Source: §B.3.1 · p. 34, Table 16 · p. 35

B.3.2 PII

• Regexes for email addresses, phone numbers, and IP addresses identify PII; after token normalization, mC4-en has more personal information than other studied corpora.The method uses postprocessing to reduce false positives, but misses other PII types and only identifies a small proportion of potential personal information; it does not estimate sensitivity or risk.Source: §B.3.2 · p. 36

B.3.3 TOXIC LANGUAGE

• C4 is least toxic by taxonomy (0.01%), while OpenWebText is most toxic by classifier and PeS2o by taxonomy.The analysis uses a trigger lexicon and a sentence-level SVM classifier; C4 has 5.75% toxic documents by classifier.Source: §B.3.3 · p. 39
• C4’s keyword filtering corresponds to the analysis’s “Offensive-*” categories, yet it contains 158K offensive-minority mentions and 47 offensive-not-minority mentions.Other datasets without such filters contain several million offensive-not-minority references.Source: §B.3.3 · p. 39

B.3.4 DEMOGRAPHIC SENTIMENT CO-OCCURRENCES

• Most demographic-term sentences show neutral or weakly positive sentiment, but sentences containing “black” are more negative across corpora; The Stack is excluded.Sentiment is computed at the sentence level, unlike Gao et al. (2020), who analyzed the most frequent co-occurring terms.Source: §B.3.4 · pp. 39–40

B.4 CROSS-DATA ANALYSIS

• 50% of RedPajama’s unique documents originate from C4, while 50% of OpenWebText’s unique documents originate from The Pile.Source: §B.4 · p. 41
• Comparing common unigrams reveals distributional and topical differences between corpora.The analysis compares unigram rankings and Jensen–Shannon distances using vocabulary intersections and unions.Source: §B.4 · p. 41, §B.4.1 · p. 52
• Although mC4-en was described as a superset of C4, C4 documents constitute only 0.04% of mC4-en, which is only 10x larger.Source: §B.4 · p. 41

B.4.1 DISTRIBUTIONAL SIMILARITY

• OSCAR’s unigram distribution is most similar to other corpora on average (Jensen–Shannon distance 0.19), while The Stack is most distant.The top-1,000 unigram distributions group OpenWebText, C4, mC4-en, OSCAR, The Pile, and RedPajama as similar, with S2ORC, peS2o, LAION-2B-en, and The Stack quite different from the others.Source: Figure 29 · p. 52, §B.4.1 · p. 52
• C4, OSCAR, and RedPajama share over 1.6 million documents, and LAION-2B-en shares over 30,000 documents with C4 and RedPajama.Even the image-caption corpus LAION-2B-en overlaps with other corpora.Source: Table 23 · p. 52

B.4.2 CORPUS OVERLAP

• 6,767,877 URLs are shared by C4, mC4-en, and OSCAR, while other multi-corpus URL overlaps share at most one URL.Text overlap also diminishes quickly to zero as more datasets are considered; the two largest multi-dataset text overlaps exceed one million document clusters but represent less than 1% of clusters in each involved dataset.Source: §B.4.2 · p. 52, Figure 31 · p. 53, Figure 30 · p. 53
• S2ORC and peS2o have no exact-text overlapping documents, although academic papers may be shared with other corpora.The paper attributes the S2ORC–peS2o result to formatting differences in parsed paper sections, since peS2o is a cleaned version of S2ORC.Source: §B.4.2 · pp. 52–53
• 50% of RedPajama’s unique documents originate from C4, despite RedPajama having 2.5 times as many documents and 6.6 times as many tokens.The paper attributes this to larger documents or duplicate C4 content; likewise, 50% of OpenWebText’s unique documents overlap with The Pile, which includes OpenWebText as a source.Source: §B.4.2 · p. 53
• 0.04% of mC4-en documents come from C4, despite mC4-en being only 10 times larger and built from a superset of C4’s Common Crawl data.The authors speculate that formatting differences between the collections explain the small overlap.Source: §B.4.2 · p. 53

C LIMITATIONS

• Elasticsearch was not designed to scale with large text corpora; indexing can take days and is costly to maintain.The authors hope to explore faster, more cost-effective indexing tools.Source: C LIMITATIONS · pp. 53–54
• Elasticsearch supports only exact-match search, not fuzzy or semantic search.Source: C LIMITATIONS · p. 54

D BENCHMARKING RUNTIMES

• Benchmarks used C4 as a popular, representative-size corpus on a Google Cloud node with 882 GB RAM and 224 CPUs.Although the machine had substantial RAM, analyses typically used no more than 250 GB; the node was chosen for CPU availability.Source: D BENCHMARKING RUNTIMES · p. 55
• All analyses ran in under 12 hours each; 13 of 22 took several minutes, and C4 analyses totaled an estimated 46 hours and 51 seconds.Times are rough estimates based on Linux TIME measurements; the total excludes repeated runs and contamination analyses on other evaluation datasets.Source: D BENCHMARKING RUNTIMES · p. 55
• The cited passage states that the estimation does not include Elasticsearch hosting costs.Source: D BENCHMARKING RUNTIMES · p. 55

E TECHNICAL DETAILS

• The n-gram algorithms use a Bloom-filter-inspired hash-table trick, so their results are approximate and accuracy depends on memory available for the table.The section covers algorithms for the most common, least common, and total unique n-grams.Source: E TECHNICAL DETAILS · p. 56

E.1 MOST COMMON n-GRAMS

• Top-k n-grams are tracked by incrementing hashed counts and retaining candidates whose counts meet the current minimum, but collisions can inflate counts and produce false positives.Reported top-k counts are upper bounds on true counts; without hash collisions, the results are correct.Source: §E.1 · p. 56

E.2 LEAST COMMON n-GRAMS

• Bottom-k collection requires two corpus passes: the first tallies hashed counts, and the second selects n-grams with sufficiently low counts.Hash collisions can omit rare n-grams through false negatives, and reported bottom-k counts are lower bounds on true counts.Source: §E.2 · p. 56

E.3 UNIQUE n-GRAMS

• The unique n-gram estimate counts true entries in a Boolean hash table after scanning the corpus, making it a lower bound on the actual number of unique n-grams.Source: §E.3 · p. 56