Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research

• Opaque pretraining-data composition limits understanding of language-model capabilities and limitations, despite models’ central role across NLP tasks.This lack of transparency affects scientific progress and the public’s ability to assess models.Source: §1 · p. 1
• Data transparency helps developers and users make informed decisions because training-data similarity can affect task performance and social biases may require additional consideration.Open data also enables research on memorization, deduplication, adversarial attacks, benchmark contamination, and training-data attribution.Source: §1 · pp. 1–2
• Dolma provides a larger pool of tokens at comparable quality while maintaining diverse data composition for language-model pretraining research.The corpus is sampled from approximately 200 TB of raw text and curated into an 11 TB dataset.Source: §1 · p. 2, Table 1 · p. 2
• Dolma releases a diverse, multi-source corpus of 3T tokens across over 4B documents from six publicly accessible data sources.Its sources include web text, scientific research, code, public-domain books, social media, and encyclopedic materials.Source: §1 · p. 2
• The Dolma Toolkit is an open, high-performance, portable system for efficiently curating large datasets and studying or improving curation practices.It enables practitioners to reproduce the Dolma dataset as well as investigate alternative data-curation methods.Source: §1 · p. 2

Closed data curation practices

• Pretraining data and curation details are increasingly undisclosed, while even powerful open models such as Llama 2, Mistral, Yi, and Qwen share neither their data nor sufficient reproduction details.Some efforts provide transparent curation documentation or partial data, but comprehensive openness remains uncommon.Source: §2 · p. 2

Open corpora for language model pretraining

• C4 (175B tokens) and Pile (387B tokens) offer quality and demonstrated use but limited scale, while ROOTS (≈400B tokens) provides only 30% English data for English-only training.The authors frame scale and English-only concentration as research foci rather than universal measures of dataset quality.Source: §2 · p. 3
• Falcon (580B tokens) and RedPajama v2 (30T tokens) meet scale needs but rely entirely on Common Crawl, limiting source diversity; RedPajama v2 is also only lightly curated.Its mostly as-is CCNet output leaves model developers responsible for additional filtering before training.Source: §2 · p. 3
• RedPajama v1 (≈1.2T tokens) inspired Dolma, but Dolma pursues broader source coverage, including larger scientific-paper collections and Reddit, while addressing reported data-quality concerns.Other open corpora released during review include FineWeb, Zyda, and datasets for LLM360 Amber, LLM360 K2, and MAP-Neo.Source: §2 · p. 3

Purpose of the design goals

• The authors present these design goals to disclose Dolma’s strengths and limitations and reinforce transparency practices in dataset-curation research.They connect this motivation to curation rationales in Bender and Friedman (2018) and motivation questions in Gebru et al. (2021).Source: §3 · p. 3

Consistency with prior recipes

• Dolma follows known language-model pretraining sources and methods, scopes the corpus to English, and seeks to maximize the generalizability of scientific research.This consistency supports studying and scrutinizing contemporary language models, including closed models, while leveraging established English curation practices.Source: §3 · p. 3

Evidence-backed decisions

• When prior work offers no clear recipe, Dolma prioritizes curation decisions that improve language-model performance across diverse tasks and datasets.These decisions are evaluated against the diverse suite of tasks and datasets discussed in §4.2.Source: §3 · p. 3

Large-scale data

• Dolma targets 2–3T training tokens to support research on the relationship between model size and dataset size under scaling-law considerations.Hoffmann et al. (2022) proposed compute-optimal training through a fixed model-size-to-token ratio, while later work such as Llama 2 suggests benefits from more tokens.Source: §3 · p. 3

Preserving openness

• Openness requires sharing the data and curation process, even when practical, legal, or ethical considerations require deviating from prior recipes.Dolma avoids Books3 (Gao et al., 2020) amid copyright litigation and filters personally identifiable information to mitigate release risks.Source: §3 · p. 3

4.1 The Dolma Toolkit

• The open-source Dolma Toolkit processes hundreds of terabytes by unifying dataset curation into scalable filtering and mixing operations.Filtering applies configurable scoring and removal policies, while mixing handles cross-file operations such as up/down-sampling, deduplication, and decontamination.Source: §4.1 · pp. 3–4
• Filtering removes non-English, low-quality or unnatural, toxic, and personally identifying text at document and sub-document levels.In internal C4-replication tests, filtering ran at 122 CPU hours per TB; processing Dolma’s 200 TB of raw files on 192 vCPUs would take five days.Source: §4.1 · p. 4
• The toolkit’s Rust mixer supports up/down-sampling, deduplication, and test-set decontamination using a compatible Bloom filter for linear-time probabilistic duplicate detection.Test examples seed the Bloom filter, which flags duplicates encountered while mixing pretraining data.Source: §4.1 · p. 4

4.2 Data Ablations

• Data ablations compare intervention and baseline datasets while controlling model architecture and training to isolate the effects of specific curation decisions on downstream performance.Experiments use a 1.2-billion-parameter decoder-only OLMo-family model trained for 150 billion tokens before early termination because full training is prohibitively expensive.Source: §4.2 · p. 4
• Ablation models are evaluated zero-shot across eight contamination-checked datasets spanning capabilities such as question answering and commonsense reasoning.Tasks are cast as ranked text classification using in-context prompt truncation from Min et al. (2022), PromptSource prompts (Bach et al., 2022), and an in-house harness similar to the Eleuther harness.Source: §4.2 · p. 4

5 € Curating Dolma-Web

• The pipeline starts with 2.28T tokens from 25 Common Crawl snapshots collected between 2020-05 and 2023-06, retaining English pages with FastText scores of at least 0.5.CCNet removed 61.7% of the data by byte size during language filtering and initial deduplication.Source: §5 · p. 4

5.2 Z Quality Filtering

• Quality filtering combines all Gopher heuristics with C4 NoPunc; C4 NoPunc alone outperforms C4 All and Gopher All, while their combination performs best.Gopher All flags 15.23% of UTF-8 characters for removal and C4 NoPunc flags 22.73%.Source: §5.2 · p. 5

5.3 Z Content Filtering

• The permissive High toxicity threshold removes 5.5–7.3% of content and generally underperforms the Low threshold, which removes 29.1–34.9%; High is adopted to preserve token scale.Quality, content, and deduplication filters overlap very little, creating a compounded filtering effect when combined.Source: §5.3 · pp. 5–6

5.4 ç Deduplication

• Deduplication filters 53.2% of documents by exact URL, 14.9% of URL-deduped documents by exact document matching, and 18.7% of paragraphs by exact paragraph matching.URL and document deduplication remove duplicate items early, while paragraph deduplication is performed last to reduce boilerplate without disrupting content analysis.Source: §5.4 · p. 6
• Stacking quality filtering, content filtering, and paragraph-level deduplication produces a positive compounding effect on 1.2B model downstream performance over no filtering.The full web pipeline applies URL and document deduplication, then quality and content filtering, followed by paragraph-level deduplication.Source: §5.4 · p. 6, Figure 3 · p. 6

6 Ð Curating Dolma-Code

• Dolma-Code contains 411B tokens derived from GitHub and is based on the deduplicated, permissively licensed Stack collection.The raw Stack dataset was collected in March 2023, and data-heavy files such as JSON and CSV were filtered.Source: §6 · p. 6

6.2 Z Quality Filtering

• Combining RedPajama v1 and StarCoder heuristics lowered perplexity on code datasets such as HumanEval and improved evaluation-suite performance versus RedPajama v1 rules alone.The filters remove repetitive preambles, excessively long-line or mostly numerical files, mostly data or templated files, low-star repositories, unsuitable comment densities, and low-code HTML.Source: §6.2 · pp. 6–7

6.3 Z Content Filtering

• Content filtering applies the web subset’s PII heuristics and removes documents containing code secrets or software-specific personal information detected by detect-secrets.Documents with a detect-secrets match are removed.Source: §6.3 · p. 7

6.4 ç Deduplication

• Deduplication starts from the Stack’s already-deduplicated version, using MinHash (Broder, 2002) and Locality Sensitive Hashing to identify similar documents.The deduplication pipeline was first introduced by Allal et al. (2023).Source: §6.4 · p. 7

7 Ü Curating Dolma-Social

• Dolma-Social comprises 80B tokens from 378M Reddit submissions and comments collected from December 2005 through March 2023 via Pushshift (Baumgartner et al., 2020).The corpus uses Reddit’s tree-like threads to compare atomic, partial-thread, and full-thread document formats before filtering non-English content.Source: §7 · p. 7
• Atomic Content, which treats submissions and comments as independent documents, performs better than partial- or full-thread linearizations on the evaluation suite.The authors hypothesize that artificial formatting introduced by combining thread elements harms language-model training.Source: §7 · p. 7, Figure 4 · p. 7

7.2 Z Quality Filtering

• Quality filtering removes short or oversized documents, low-vote comments, deleted or moderated content, over-18 posts, and material from 26,123 banned or NSFW subreddits.Comments must have at least 500 characters and three votes, submissions at least 400 characters, and documents no more than 40,000 characters.Source: §7.2 · p. 7

7.3 Z Content Filtering

• Content filtering follows the web pipeline but fully removes Reddit documents containing personally identifiable information rather than masking it.This change reflects the short length of many Reddit documents.Source: §7.3 · p. 7

7.4 ç Deduplication

• Document-level deduplication reduces repeated information, including “copypasta” copied across comments and subreddits.The strategy follows the web pipeline while deduplicating at the document level because Reddit submissions and comments are shorter than web documents.Source: §7.4 · pp. 7–8

C4 for Curated Web Content

• C4 was reprocessed through Dolma’s full web pipeline, excluding URL deduplication, to remove additional low-quality and duplicated texts and mask PII.This supplementation follows data recipes used for LLaMA and Llama 2.Source: §8 · p. 8

Semantic Scholar for Academic Literature

• The peS2o dataset contributes approximately 40 million cleaned, filtered, deduplicated, and pretraining-formatted open-access academic papers derived from S2ORC, and is used as-is.peS2o was created specifically for language modeling.Source: §8 · p. 8

Project Gutenberg for Books

• Project Gutenberg contributes English-language public-domain books from an archive of over 70 thousand titles, filtered as in §5.1 and deduplicated by exact book-title match.The archive was collected in April 2023.Source: §8 · p. 8

Wikipedia and Wikibooks for Encyclopedic Content

• Wikipedia and Wikibooks content comes from March 2023 Wikimedia dumps of the English and Simple editions, processed with WikiExtractor and filtered to remove documents containing 25 or fewer UTF-8-segmented words.The dataset is designed to contain no duplicated documents; short pages were removed because they were often templated pages or XML parsing errors.Source: §8 · p. 8

9 Training a Language Model on Dolma

• The final validation trains, evaluates, and releases OLMo-1B, demonstrating comparable quality to other released language models of similar size.The evaluation uses zero-shot results across a range of downstream tasks.Source: §9 · p. 8

9.1 Evaluating OLMo-1B

• OLMo-1B performs better on average than TinyLlama, outperforming it on 4 of 8 evaluation tasks despite comparable parameter counts.Table 2 compares OLMo-1B with similarly sized models; TinyLlama is the closest training-token comparison, while Pythia used nearly 10 times fewer tokens and StableLM2 used 2 trillion tokens for two epochs.Source: OLMo-1B · p. 8, §9.1 · p. 8, Table 2 · p. 8

9.2 Measuring Domain Fit

• Across diverse domains, Dolma-trained models achieve coverage comparable to Pile and better than models trained on single-source corpora.Pile performs well despite its smaller scale; Dolma and, to a lesser extent, RedPajama v1 show similar coverage, while C4, mC4, and RefinedWeb have higher average perplexity.Source: §9.2 · p. 8, Figure 5 · p. 9
• OLMo-1B nearly matches the Pile model’s Paloma perplexity curve despite containing a much larger fraction of web data.Paloma samples marked domains equally across sources, making it sensitive to fit on heterogeneous data and highlighting the value of diverse non-web sources.Source: §9.2 · pp. 8–9

Conclusion

• Dolma is a released three-trillion-token English corpus spanning web documents, scientific papers, code, books, social media, and encyclopedic materials, with its curation toolkit open-sourced.The corpus and toolkit are released under ODC-By and Apache 2.0, respectively; Dolma v1.7 later improved downstream performance while holding the model constant.Source: Conclusion · p. 9

Limitations

• Dolma is limited by its English focus, incomplete source representativeness, 1B dense-transformer validation setup, restricted evaluation tasks, and infeasible full-corpus inspection.These constraints limit conclusions about non-English performance, larger models, alternative architectures, broader capabilities, data distributions, quality, and potential harms.Source: Limitations · p. 9

Ethical Considerations

• The authors addressed individual and societal risks through legal and ethics consultation, masking some personal information, toxic-content decisions, and data-removal requests.They recognize unresolved copyright and fair-use questions, jurisdictional variation, and the inability to reliably detect copyrighted material at corpus scale.Source: Ethical Considerations · p. 10

B Author Contributions

• Contributors collectively handled data acquisition, processing, infrastructure, filtering, ablations, analyses, licensing, release, documentation, advising, and overall project leadership.Kyle Lo and Luca Soldaini led the overall Dolma project, while many contributors owned specialized components across the pipeline and release process.Source: B Author Contributions · p. 25

C (Lack of) details about pretraining data curation for both open and closed language models

• The review uses major language models to demonstrate the need for clear documentation and transparency around pretraining-data curation.It surveys reported or missing information about corpus size, provenance, filtering, deduplication, and related practices.Source: C (Lack of) details about pretraining… · p. 26

C.1 PaLM 2 (Anil et al., 2023)

• PaLM 2 reports a corpus larger than PaLM’s, using web documents, books, code, mathematics, and conversational data, with PII and quality filtering but limited reproducibility details.Toxicity was identified using Perspective API, but text unit, threshold, and removal details were not provided; language frequencies were reported, but reproduction details were lacking.Source: §C.1 · p. 26

1. Corpus size. N/A

• GPT-4’s pretraining-data curation remains largely undocumented, with only limited indirect evidence from data-archeology studies and post-hoc contamination analyses.Reported contamination testing used exact substring matching after whitespace removal and found some contamination with BIG-Bench (Srivastava et al., 2023).Source: §1 · p. 26

C.3 Claude (Anthropic, 2023)

• Claude’s pretraining data are almost entirely undisclosed.The review states that it knows next to nothing about the data used for Claude.Source: §C.3 · p. 27

C.4 Llama 2 (Touvron et al., 2023b)

• Llama 2 (Touvron et al., 2023b) reports 2T tokens and selected exclusions, but provides limited provenance and incomplete details on PII, toxicity, language identification, and source upsampling.It reports extensive deduplication using a modified n-gram tool, while toxicity appears to have been deferred to a later training stage rather than filtered during pretraining.Source: §C.4 · p. 27

C.5 LLaMA (Touvron et al., 2023a)

• LLaMA (Touvron et al., 2023a) provides the most detailed curation account, using 1.4T tokens from identifiable sources with explicit language, quality, and deduplication procedures.Its mixture was 67% CommonCrawl, 15% C4, 4.5% GitHub, 4.5% Wikipedia, 4.5% Books, 2.5% arXiv, and 2.0% StackExchange, with Wikipedia and Books upsampled for a second epoch.Source: §C.5 · p. 27

C.6 OPT (Zhang, 2022)

• OPT (Zhang, 2022) trained on 180B tokens from known-provenance sources, restricting Pile subsets because of gradient norm spikes at the 1B scale and reducing Reddit data by 66%.It used Min-HashLSH deduplication with a Jaccard similarity threshold of 0.95 and English-only filtering, but did not describe the language-identification method.Source: §C.6 · pp. 27–28
• OPT (Zhang, 2022) does not state whether it performed processing for PII, toxicity, quality, or decontamination.Thus, even its documented corpus sources and deduplication do not establish a complete curation recipe.Source: §C.6 · p. 28

D.1 Ablation Setup

• Data ablations train 1B-parameter decoder-only transformers for up to 150B tokens using the OLMo architecture, 2048-token contexts, GPT NeoX tokenization, LionW optimization, and batch size 1024.Models use 16 layers, 16 attention heads, 2048-dimensional representations, ALiBi, SwiGLU, mixed precision, a 1e-4 peak learning rate, 2000-step warm-up, cosine decay, and 1e-2 weight decay.Source: §D.1 · p. 28

D.2 Perplexity Evaluation Suite

• For data ablations, language-model perplexity is tracked using Paloma, with included datasets covering Gab, ICE, Twitter AAE, Manosphere, and 4chan.Source: §D.2 · p. 28

D.3 Downstream Evaluation Suite

• Data-ablation decisions primarily rely on downstream evaluation across science question answering, reading comprehension, commonsense, physical reasoning, and pronoun-resolution tasks.The suite includes AI2 Reasoning Challenge, BoolQ, HellaSwag, OpenBookQA, PIQA, SciQ, and WinoGrande, with only the easy AI2 subset used online and the challenge subset included offline.Source: §D.3 · p. 29

D.4 Training Setup for OLMo-1B

• OLMo-1B training runs for roughly 3.1T tokens with 739,328 maximum steps, batch size 2048, 256 compute units, and AdamW instead of LionW because of optimizer instabilities.This setup otherwise follows the dataset-ablation configuration.Source: §D.4 · p. 29

E Construction of Conversational Threads in Forums Data

• Reddit data API content consists of submissions and comments forming nested threads, with submissions at the root and comments branching into dialogue trees.Submissions may link to external content or initiate discussions, while comments reply to submissions or other comments.Source: E Construction of Conversational… · p. 29
• The Reddit study compares atomic content, partial threads limited by parent depth, and full threads combining each submission with all child comments using indentation to mark hierarchy.Atomic content treats comments and submissions as independent documents; partial threads assemble comment dialogues while leaving submissions separate.Source: E Construction of Conversational… · p. 29
• Atomic Reddit submissions and comments achieve better downstream performance than partial or full conversational threads, which may introduce short and repeated comments through complex formatting.The study leaves improved forum-content formatting for future work.Source: E Construction of Conversational… · p. 29

F Tokenization Analysis

• GPTNeoX uses 50,057 of 50,280 vocabulary tokens, and token counts correlate with token IDs (r = 0.638, p < 0.001), suggesting a good fit to Dolma.The 223 unused tokens are mostly whitespace combinations whose embeddings would not be updated during training.Source: F Tokenization Analysis · p. 30
• Code has fertility 2.45 versus 1.15–1.28 for other sources, making its processing costs more than twice as high.The higher fertility stems from separate tokens for newline, tab, and other non-space whitespace; adding code-specific tokens such as “\nif” may help.Source: F Tokenization Analysis · p. 30

G Auditing our Language Filter

• The language-ID audit correctly classified every ICE English document as English at the permissive 0.5 English-score threshold across all nine countries.The supplied figure caption additionally reports that a majority remain identified as English even at a 0.90 threshold.Source: G Auditing our Language Filter · p. 30, Figure 7 · p. 31

H Details on Toxicity Filters

• The toxicity pipeline trains separate FastText hate and NSFW detectors, with the NSFW detector targeting obscene content rather than sexual content generally.Across location-based English subreddits, toxicity-labeling rates differ by less than 5% for every threshold, suggesting little to no dialectal bias.Source: H Details on Toxicity Filters · p. 31

I Details on PII Filters

• PII filtering replaces documents with five or fewer detected spans using special tokens, while removing documents with six or more spans.These rules affect 0.02% and 0.001% of documents, respectively, in the 25 Common Crawl snapshots.Source: I Details on PII Filters · p. 31

J Do quality and content filters have similar effects?

• Dolma’s language, toxicity, and content-filter audits indicate that the filters generally affect different documents rather than redundantly selecting the same material.The correlation analysis covered 24M, 20M, and 43M Common Crawl documents across high, medium, and low buckets.Source: J Do quality and content filters have… · p. 31, Figure 9 · p. 32
• PII and hate-speech filters show some positive correlation, whereas Gopher rules correlate negatively with deduplication, especially in the high-perplexity tail.Gopher rules remove random-string documents that deduplication misses, supporting filtering beyond deduplication.Source: J Do quality and content filters have… · p. 32

Web domain and year distributions

• 63.6% of Dolma’s web documents come from .com sites, with patents.google.com the most common domain per token, followed by www.nature.com and www.frontiersin.org.Dolma includes documents from a broad set of internet domains, and the next most common site categories are .org and .co.uk.Source: K Dolma data distribution figures… · p. 32
• Dolma’s web documents are concentrated mostly in 2020, 2022, and 2021.The final data composition is inspected using distributions over web domain, year, and language.Source: K Dolma data distribution figures… · p. 32

Language distribution

• After English-only filtering, the most common remaining language label is unidentified (‘un’) at 0.86% of documents, followed by Chinese at 0.06%.Language identification tools are imperfect, so these labels summarize the residual language distribution rather than definitive language assignments.Source: K Dolma data distribution figures… · p. 32

L Test Set Contamination in Dolma

• Six PromptSource datasets are 100% contaminated in Dolma, while four others have over 90% of test examples present; all were excluded from downstream evaluation.The completely contaminated sets are Winograd Schema Challenge, Sick, AX, SemEval Task 1, COPA, and AXb; the mostly contaminated sets include HumanEval, WIC, ESNLI, and SNLI.Source: L Test Set Contamination in Dolma · p. 32, Figure 11 · p. 33

M Strategies for Subsets Mixing and Upsampling with Dolma

• Adding code to C4 and Stack mixtures improves bAbI and WebNLG performance, while program-aided fine-tuning enables code-trained models to outperform C4-only models on GSM8k.All models failed on GSM8k under in-context learning and ordinary fine-tuning, but code-pretrained models benefited when answers were generated through Python programs.Source: M Strategies for Subsets Mixing and… · p. 33, Table 3 · p. 34
• Mixture choice has little effect on C4 web perplexity but strongly affects specialized domains: Web-only mixtures struggle on code and academic text, whereas modest in-domain data improves fit.Reference+ and Gopher-like mixtures outperform the Naïve mixture on S2ORC, despite using 4.9% and 24.2% academic data respectively, with nearly identical results.Source: M Strategies for Subsets Mixing and… · p. 34, Figure 12 · p. 35

Why was the dataset created?

• Dolma was created for training autoregressive language models and contains transformed English text from web, scientific, code, encyclopedic, and public-domain book sources.The dataset can also support corpus analysis, provenance studies, domain-specific models, and other language-model training uses.Source: Why was the dataset created? · p. 34, What (other) tasks could the dataset… · p. 34

Are there obvious tasks for which it should not be used?

• Dolma is unsuitable for directly consuming original content because extensive transformations alter source materials, and source availability varies, notably because Pushshift no longer distributes Reddit data.Original resources may not persist, although source-specific identifiers and archival snapshots can support reconstruction; the data-collection process used source-specific downloads and repositories.Source: Are there obvious tasks for which it… · p. 34, What data does each instance consist… · pp. 35–36, §N.3 · p. 37

What are the instances? Are there multiple types of instances?

• Dolma instances are plain-text spans or code with source-linked metadata, but they have no labels or comprehensive authorship and demographic information.Metadata can connect documents through URLs, repository names, thread identifiers, manuscript IDs, or book titles; summary statistics are reported in Table 1, without values supplied here.Source: What are the instances? Are there… · pp. 34–35, Are relationships between instances… · p. 35

N.3 Data Collection Process

• Dolma data was collected and postprocessed by full-time Allen Institute for AI employees, and no instances were manually annotated.Metadata associated with each instance was obtained directly from its source.Source: §N.3 · p. 37, Over what time-frame was the data… · p. 37

Over what time-frame was the data collected? Does the collection time-frame match the creation time-frame?

• Common Crawl was not representative of the web; Dolma used snapshots from 2020-05 to 2023-06, selecting 24 most recent snapshots, about one quarter of those available.Source: Over what time-frame was the data… · p. 37

Are there any known errors, sources of noise, or redundancies in the data?

• The authors reported no known substantive errors, noise, or redundancies, although a negligible portion of Common Crawl data could have been lost through S3 network failures despite retry mechanisms.Copies could have failed after exceeding retry limits.Source: Are there any known errors, sources of… · p. 37

N.4 Data Preprocessing

• Dolma preprocessing combines English filtering, source-specific quality and content filters, deduplication, and PII masking across web, Reddit, code, books, encyclopedic, and scientific subsets.All data sources use FastText English identification with a 0.5 threshold; Common Crawl additionally undergoes paragraph and URL deduplication, quality filtering, toxicity filtering, and PII handling.Source: §N.4 · pp. 38–39
• Raw data is retained for all subsets except Common Crawl, whose space-constrained release contains only linearized, language-filtered snapshots.Raw data is not available for download outside the Allen Institute for AI; interested users must contact the authors.Source: §N.4 · p. 39

Is the preprocessing software available?

• Preprocessing software is publicly available through the Dolma GitHub repository and PyPI.Source: Is the preprocessing software available? · p. 39

N.5 Dataset Distribution

• Dolma is distributed through the Hugging Face Hub, including Python-package access, direct download, Git-LFS, and an additional AI2 cloud-storage copy.Source: §N.5 · p. 40

When will the dataset be released/first distributed? (Is there a canonical paper/reference for this dataset?)

• The dataset is already available, and the manuscript serves as its reference publication.Source: When will the dataset be… · p. 40

What license (if any) is it distributed under? Are there any copyrights on the data?

• License information and release restrictions are provided on Dolma’s Hugging Face release page, while the dataset is distributed for free.Source: What license (if any) is it… · p. 40, Are there any fees or access/export… · p. 40

N.6 Dataset Maintenance

• The Allen Institute for AI maintains Dolma, provides issue and discussion-based support, and uploads revisions as needed with version labels and changelogs.Users are instructed to track the dataset version and contact the corresponding authors about contributing improvements to future releases.Source: §N.6 · p. 40

N.7 Legal & Ethical Considerations

• Dolma’s creators did not directly inform authors, and they state that the dataset’s scale and nature make it impossible to determine which ethical obligations apply.This includes web, Reddit, academic, and Project Gutenberg-derived content collected without directly contacting individual authors.Source: §N.7 · p. 40
• No consent was collected from represented individuals, but an internal and external ethics committee reviewed Dolma’s plans and recommendations were incorporated.The project states that individuals can request removal of text from or about them.Source: §N.7 · pp. 40–41
• Dolma contains potentially sensitive public-web text; the project measured and removed some detectable personal information, removed toxic documents, restricted access, and provided a removal-request mechanism.The dataset license also restricts uses that might be deemed discriminatory.Source: §N.7 · pp. 40–41
• Dolma is not representative of its sources and may under- or overrepresent communities, STEM disciplines, public-domain books, and global-north events; the project did not alter social-group distributions.Large-scale interventions to correct societal biases are left to future work.Source: §N.7 · p. 41

O.1 Comparing Dolma With Other Corpora; O.2 Deduping Strategy; O.3 Filtering of Personal Identifiable Information; O.4 Comparing Quality Filters for Web Pipeline; O.5 Full Comparison of Web Pipeline; O.7 Comparing Code Processing Pipeline

• The corpus-comparison, deduplication, personal-information, web-filter, code-processing, mixture, forum-formatting, and toxicity studies report Paloma perplexity, downstream-task, or training cross-entropy evaluations in the supplied figures.The passages identify evaluation subsets and tasks but provide no cell values or reported winners.Source: Figure 13 · p. 41, Figure 21 · p. 44, Figure 28 · p. 46

O.8 Studying Dolma Mixture; O.9 Strategies to Format Conversational Forums Pipeline; O.10 Evaluating Toxicity Filtering in Conversational Forums Pipeline

• The supplied figures also evaluate Dolma-mixture choices, conversational-forum formatting, and conversational-forum toxicity filtering using Paloma perplexity, downstream tasks, or training cross-entropy.The captions name the evaluated Paloma subsets and benchmark tasks but do not state numerical results.Source: Figure 63 · p. 56, Figure 69 · p. 58, Figure 76 · p. 60

O.10 Evaluating Toxicity Filtering in Conversational Forums Pipeline

• O.10 evaluates the conversational-forums toxicity-filtering pipeline with Paloma perplexity, six downstream tasks, and training cross-entropy.The reported perplexity subsets are Twitter AAE, English mC4, and S2ORC; downstream tasks include OpenBookQA, ARC-E, WinoGrande, SciQ, HellaSwag, and PIQA.Source: Figure 79 · p. 60, Figure 80 · p. 61, Figure 81 · p. 61

O.11 Training OLMo-1B

• Figures 83–88 report Paloma perplexity results for subsets spanning 4chan, Dolma Reddit and Papers, ICE, M2D2 Wiki, Twitter AAE, Manosphere, English mC4, S2ORC, C4, Penn Tree Bank, Dolma Wikipedia, Gab, Pile, Dolma Books, and C4 100 dom.Source: Figure 83 · p. 62, Figure 84 · p. 62, Figure 85 · p. 62
• O.11 additionally reports OLMo-1B results on six downstream tasks and training cross-entropy.The downstream evaluations comprise OpenBookQA, ARC-E, WinoGrande, SciQ, HellaSwag, and PIQA.Source: Figure 89 · p. 63, Figure 90 · p. 63, Figure 91 · p. 64