PaperlayerView exact revision on arXiv

Source-linked AI summary

Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research

Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, Kyle Lo

arXiv:2402.00159v2cs.CL

TL;DR

Limited disclosure of pretraining corpora constrains reproducible research on how training data affects language-model capabilities and limitations. The paper releases Dolma, a documented three-trillion-token English corpus spanning diverse sources, together with an open curation toolkit and ablation analyses. Dolma provides open artifacts for pretraining research, while its curation experiments report improved performance for selected filtering combinations and its evaluation setup is limited in model scale and architecture.

  • Problem

    Pretraining corpora and curation recipes are often undisclosed or vaguely described, limiting research on how corpus composition affects model capabilities and limitations.

  • Method

    The authors curate and document Dolma, release a toolkit for large-scale data curation, and use controlled ablations across diverse evaluation tasks to study curation decisions.

  • Results

    C4 NoPunc outperforms C4 All and Gopher All on perplexity and downstream tasks, while combining Gopher All with C4 NoPunc gives the best performance.

  • Takeaways & Limitations

    Dolma’s open corpus, documentation, and toolkit support reproduction and further research on large-scale language-model data curation.

  • Takeaways & Limitations

    Ablation validation uses 1B-parameter dense autoregressive models and does not examine alternative architectures, so some decisions may not apply at larger scales.

Abstract

from arXiv · show

Information about pretraining corpora used to train the current best-performing language models is seldom discussed: commercial models rarely detail their data, and even open models are often released without accompanying training data or recipes to reproduce them. As a result, it is challenging to conduct and advance scientific research on language modeling, such as understanding how training data impacts model capabilities and limitations. To facilitate scientific research on language model pretraining, we curate and release Dolma, a three-trillion-token English corpus, built from a diverse mixture of web content, scientific papers, code, public-domain books, social media, and encyclopedic materials. We extensively document Dolma, including its design principles, details about its construction, and a summary of its contents. We present analyses and experimental results on intermediate states of Dolma to share what we have learned about important data curation practices. Finally, we open-source our data curation toolkit to enable reproduction of our work as well as support further research in large-scale data curation.

1 Introduction

Dolma addresses limited transparency in language-model pretraining data by releasing a large, diverse English corpus and its curation toolkit for reproducible research.

  • Closed or vaguely documented pretraining data makes it difficult to study how corpus composition affects model capabilities and limitations.
  • Data transparency helps developers and users make more informed decisions because pretraining data can relate to task performance and social biases.
  • Dolma is an open, three-trillion-token corpus designed to support language-model pretraining research.
  • 3T tokens span over 4B documents from six publicly accessible sources, including web text, scientific papers, code, books, Reddit, and Wikipedia.
  • The Dolma Toolkit is an open-source, high-performance tool for efficiently curating large datasets for language-model pretraining.

2 Related Work

Prior open pretraining corpora provide useful foundations but leave gaps in scale, English-only concentration, source diversity, curation, or coverage of relevant data sources.

  • Proprietary models disclose little about corpus size, provenance, or data artifacts, limiting reproduction and scientific study of pretraining practices.
  • C4 and Pile are high-quality but smaller, while ROOTS is large yet allocates only 30% to English.
  • Falcon and RedPajama v2 meet scale targets but rely entirely on Common Crawl, limiting source diversity and increasing curation responsibility for developers.
  • RedPajama v1 is a close inspiration, but Dolma pursues broader coverage of scientific papers and Reddit and addresses reported cleanup concerns.

3 Data Design Goals

Dolma’s design goals prioritize reproducibility, openness, sufficient scale, and evidence-backed curation decisions while acknowledging practical constraints on open data release.

  • Dolma follows known pretraining data sources and methods to help researchers study and scrutinize language models, including models developed behind closed doors.
  • When prior work does not establish a clear recipe, Dolma uses ablations to favor decisions associated with performance across diverse tasks and datasets.
  • The corpus targets 2–3T tokens to support studying the relationship between model size and training-token quantity.
  • Openness means sharing the data and documenting curation, while practical, legal, or ethical considerations can require deviations from known recipes.

4 Data Curation Methodology

Dolma combines an open toolkit, scalable filtering and mixing operations, and controlled ablations to curate and evaluate large-scale pretraining data.

  • The toolkit processes hundreds of terabytes by unifying common curation steps into filtering and mixing operations.
  • Filtering configures text units, scoring methods, and removal policies to parallelize removal of undesirable text at massive scale.
  • Mixing supports up- and down-sampling, deduplication, and decontamination, including linear-time probabilistic duplicate detection with a Bloom filter.
  • Ablations compare intervention and baseline datasets while controlling model architecture and training to isolate curation decisions’ downstream effects.
  • Curation filters encode ideological criteria rather than a single objective notion of quality, and toxicity lacks one universally accepted definition.
  • Evaluation uses 1.2B-parameter decoder-only OLMo models trained to 150B tokens and tested zero-shot across eight datasets.

5 € Curating Dolma-Web

Dolma-Web processes Common Crawl through language filtering, deduplication, heuristic quality and content filtering, and paragraph-level deduplication. Ablations indicate that these curation stages improve downstream performance, while threshold choices trade off performance against retained corpus scale.

  • Acquisition: 2.28T tokens were collected from 25 Common Crawl snapshots spanning May 2020 to June 2023.The web subset derives from Common Crawl, which contains crawls dating back to 2007.
  • Language filtering and deduplication: 84.2% of Common Crawl content was filtered by the initial CCNet pipeline, reducing 175.1 TB to 27.7 TB.CCNet removed common paragraphs, primarily headers and navigation elements, and retained documents with English scores at least 0.5.
  • Quality filtering: C4 NoPunc outperformed C4 All and Gopher All on perplexity and downstream tasks, while combining Gopher All with C4 NoPunc performed best.Gopher All tagged 15.23% of UTF-8 characters for removal, and C4 NoPunc tagged 22.73%.
  • Quality filtering: The heuristic filters left CCNet’s high-, medium-, and low-quality proportions largely unchanged, suggesting they capture signals orthogonal to model-based quality scores.CCNet’s buckets were 21.9% high, 28.5% medium, and 49.6% low quality.
  • Deduplication: 53.2% of documents were removed by exact URL deduplication, followed by 14.9% of URL-deduplicated documents and 18.7% of paragraphs.The pipeline applies URL, document, and paragraph deduplication in sequence before and after filtering stages.
  • Ablation results: The full web pipeline produced a positive compounding effect on downstream model performance over a no-filtering baseline.The effect was evaluated across training iterations using 1.2B models, with HellaSwag shown in the main figures.

6 Ð Curating Dolma-Code

Dolma-Code contains GitHub-derived code processed with filtering heuristics adapted from RedPajama v1 and StarCoder. Ablations favored combining both rule sets because it improved code-focused and broader evaluation performance relative to RedPajama v1 rules alone.

  • Acquisition: 411B tokens in Dolma-Code were derived from GitHub.The subset uses the deduplicated Stack collection and filters data-heavy files such as JSON and CSV.
  • Quality filtering: RedPajama v1 rules remove repetitive preambles, excessively long lines, mostly numerical files, and files that are mostly data or template-generated.StarCoder heuristics additionally filter repositories and files using quality-related signals such as repository stars and file characteristics.
  • Quality filtering: Combining RedPajama v1 and StarCoder rules lowered perplexity on code datasets and improved performance across the evaluation suite relative to RedPajama v1 rules alone.This ablation motivated using the combined filtering strategy for Dolma-Code.
  • Privacy and deduplication: Dolma-Code applies PII heuristics and removes documents containing code secrets or software-specific personal information detected by detect-secrets.The subset starts from a collection already deduplicated with MinHash and locality-sensitive hashing.

7 Ü Curating Dolma-Social

Dolma-Social contains Reddit submissions and comments processed through alternative thread representations and extensive filtering. Experiments favored treating posts as independent documents, while the authors hypothesize that dialogue formatting can harm language-model training.

  • Acquisition: 80B tokens were derived from 378M Reddit posts collected from December 2005 through March 2023.The subset includes both submissions and comments obtained through Pushshift.
  • Thread linearization: Reddit content was evaluated as atomic content, partial threads, or full threads to compare alternative linearizations for pretraining.Atomic content treats submissions and comments independently; partial and full threads combine conversation elements to different extents.
  • Thread linearization: Atomic Content led to better downstream performance than partial and full threads across the evaluation suite.The authors hypothesize that more complex dialogue formatting introduces undesirable content, such as short and repeated comments.
  • Filtering: The filtering pipeline removes short or excessively long documents, low-vote comments, deleted or moderated content, and documents flagged by content filters.Comments shorter than 500 characters and submissions shorter than 400 characters are removed, while documents over 40,000 characters are also discarded.
  • Deduplication: Document-level deduplication reduces repeated copypasta and other repetitive information across comments and subreddits.Because Reddit submissions and comments are shorter than web documents, deduplication is performed only at the document level.

8 Assembling Other Data Sources

Dolma supplements its main subsets with curated web content, academic literature, public-domain books, and encyclopedic materials. These sources are reused or processed with source-specific filtering and deduplication procedures.

  • Curated web content: C4 is reprocessed through Dolma’s full web pipeline, excluding URL deduplication, to remove additional low-quality and duplicated text and mask PII.This supplements the Common Crawl-derived web subset.
  • Academic literature: peS2o contributes approximately 40 million cleaned, filtered, deduplicated, and pretraining-formatted open-access academic papers.The dataset is derived from the Semantic Scholar Open Research Corpus and is used as-is.
  • Books: Project Gutenberg contributes English public-domain books from an archive containing over 70 thousand titles.The books are language-filtered and deduplicated by exact title match.
  • Encyclopedic content: Wikipedia and Wikibooks English and Simple editions provide encyclopedic content processed with WikiExtractor.Documents with 25 or fewer UTF-8-segmented words are removed because they are often short templated pages or parsing artifacts.

9 Training a Language Model on Dolma

The authors validate Dolma by training and evaluating OLMo-1B, finding competitive downstream performance and broad domain fit while documenting important scope limitations.

  • 9 Training a Language Model on Dolma: OLMo-1B is a decoder-only autoregressive model trained as a final validation step for the Dolma pipeline.The model is evaluated on downstream tasks and released with the paper.
  • 9 Training a Language Model on Dolma: OLMo-1B outperforms TinyLlama in 4 out of 8 zero-shot evaluation tasks.The compared models have roughly comparable parameter counts, although their training-token budgets differ.
  • 9.2 Measuring Domain Fit: Models trained on Dolma and RedPajama v1 achieve domain coverage similar to Pile, while single-source corpora yield higher average perplexity.The comparison uses Paloma, a stratified collection of diverse textual sources.
  • 9.2 Measuring Domain Fit: Paloma samples marked domains equally across sources, and OLMo-1B nearly matches Pile’s perplexity curve despite using more web data.This analysis highlights the challenge of scaling web data without losing performance on diverse-domain evaluations.
  • Limitations: Dolma is an English-focused corpus whose sources cannot represent all language-model curation practices, including inaccessible or non-redistributable content.The corpus may also contain a small percentage of non-English data because language-identification tools can produce false negatives.
  • Limitations: Ablation validation uses only 1-billion-parameter dense autoregressive transformers, so some design choices may not apply at larger model sizes or to other architectures.The authors also note that the evaluation suite covers only a subset of routine language-model tasks and that the full corpus cannot be manually inspected.

C (Lack of) details about pretraining data curation for both open and closed language models

The section surveys reporting practices for pretraining data in open and closed language models. It finds that provenance, filtering, deduplication, and other curation details are often incomplete or unavailable.

  • Closed data curation practices: The survey motivates clear documentation and transparency by showing that many major language models disclose little about their pretraining data or curation.The stated goal is to support reproducibility and scientific study of dataset construction.
  • Closed data curation practices: PaLM 2 reports broad source categories and several filtering procedures, but omits reproducibility-critical details such as thresholds, tools, and removal rules.The gaps cover corpus size, provenance, PII, toxicity, language identification, quality, deduplication, and demographic analyses.
  • Open and closed model reporting: The survey notes that some models disclose limited or no information, including Claude, while other reports provide partial provenance and processing details.Examples include OPT’s restricted source mixture and reported changes to CC-News, Pile subsets, and Reddit threads.
  • Closed data curation practices: Llama 2 reports a 2T-token corpus and some deduplication and toxicity-related analyses, while leaving several curation choices unspecified.Its provenance, PII handling, language identification, and source upsampling are only partially described.
  • Open and closed model reporting: LLaMA reports known-provenance sources, a 1.4T-token corpus, and a mixture dominated by Common Crawl alongside C4, code, Wikipedia, books, arXiv, and StackExchange.Reported processing includes language filtering, quality filtering, deduplication, and source-specific mixture weights.

D Experimental Setup

The experiments train 1-billion-parameter models on up to 150B tokens and evaluate data choices using perplexity across diverse held-out domains plus downstream reasoning tasks. The setup also compares alternative Reddit document formats and defines a separate OLMo-1B training configuration.

  • D.1 Ablation Setup: Data ablations train 1B-parameter decoder-only transformers on up to 150B tokens using a fixed OLMo-style architecture and training recipe.The setup uses 16 layers, 16 attention heads, 2048-dimensional representations, a 2048-token context, GPT NeoX tokenization, LionW, warm-up, cosine decay, and weight decay.
  • D.2 Evaluation Data: The ablations track perplexity with Paloma over social, geographic, linguistic, academic, encyclopedic, web, code, and other held-out corpora.The evaluation suite includes source-specific samples such as Gab, ICE, Twitter AAE, Manosphere, and 4chan, alongside open-corpus subsets.
  • D.2 Evaluation Data: The held-out data includes samples from Dolma, RedPajama v1, Falcon RefinedWeb, C4, mC4, Pile, WikiText 103, Penn Tree Bank, M2D2, and balanced domain subsets.Additional Dolma subsets balance Reddit communities and programming languages.
  • D.3 Evaluation Tasks: Downstream ablation decisions primarily use seven benchmark tasks covering science questions, reading comprehension, commonsense, physical reasoning, and pronoun resolution.The listed tasks are AI2 Reasoning Challenge, BoolQ, HellaSwag, OpenBookQA, PIQA, SciQ, and WinoGrande.
  • OLMo-1B Training: For OLMo-1B, training is extended to roughly 3.1T tokens with batch size 2048 and 256 compute units, and AdamW replaces LionW because of optimizer instabilities.These settings differ from the data-ablation configuration.
  • Reddit Document Formats: Reddit documents are tested as atomic content, partial threads, or full threads, with atomic units producing better downstream performance than the structured alternatives.The authors hypothesize that dialogue formatting introduces undesirable short or repeated comments.

F Tokenization Analysis

Dolma’s GPTNeoX tokenizer fits the corpus globally, but code is substantially more expensive to tokenize because non-space whitespace characters become separate tokens. Filter analyses further examine language identification, toxicity, privacy, and redundancy across curation filters.

  • Tokenizer fit: 50,057 of 50,280 vocabulary tokens appear in Dolma, while 223 unused tokens mostly represent unusual whitespace combinations.The unused tokens account for roughly 0.4% of the vocabulary, leaving their corresponding input embeddings unupdated during training.
  • Tokenizer fit: Tokens with smaller IDs tend to occur more frequently in Dolma, with Spearman correlation r = 0.638, p < 0.001, suggesting good tokenizer fit.Smaller IDs correspond to byte pairs merged earlier during tokenizer training.
  • Source differences: Code fertility reaches 2.45 tokens per word, compared with 1.15–1.28 for other sources, making code processing more than twice as costly.Newlines, tabs, and returns are tokenized separately rather than as part of the following word; whitespace-token frequency is one order of magnitude higher in The Stack.
  • Toxicity filtering: The toxicity pipeline trains separate FastText detectors for hate and NSFW content, with the NSFW detector limited to sexual or obscene topics.The classifiers are trained from multilabel Jigsaw Toxic Comments annotations and unlabeled comments.
  • Toxicity analysis: The toxicity classifier’s dialect-bias analysis compares toxicity rates across country-specific Reddit subreddits using comments from communities exceeding 50K comments.Across thresholds, the fraction labeled toxic differs by less than 5% between any two locations, and scores often lie near 0 or 1.
  • Privacy and filter interactions: Dolma’s PII pipeline replaces documents with at most five detected spans and removes documents containing six or more, while correlation analysis finds filters are generally nonredundant.Replacement affects 0.02% of Common Crawl documents, removal affects 0.001%, and the filters select fairly different documents overall.

K Dolma data distribution figures using WIMBD

WIMBD-based inspection shows that Dolma spans many internet domains and is concentrated in recent years, with a majority of web documents coming from .com sites. Residual non-English material is small after English-only filtering.

  • Domain distribution: Dolma’s most common domains per token are patents.google.com, www.nature.com, and www.frontiersin.org, and 63.6% of web documents come from .com sites.The remaining documents are followed by .org and .co.uk domains in the reported domain distribution.
  • Temporal distribution: Dolma contains documents from a broad set of internet domains, with most documents dated to 2020, 2022, and 2021.
  • Language distribution: After English-only filtering, the most common residual language label is unidentified (‘un’) at 0.86% of documents, followed by Chinese at 0.06%.The language-identification tools are described as imperfect.

L Test Set Contamination in Dolma

Dolma’s contamination analysis identifies benchmark material present in the corpus and excludes contaminated datasets from the reported downstream evaluation. The broader experiments also examine how source mixtures and code affect model behavior.

  • Test-set contamination: 100% of six PromptSource datasets appear in Dolma, including Winograd Schema Challenge, Sick, AX, SemEval Task 1, COPA, and AXb.
  • Test-set contamination: Over 90% of the test sets for OpenAI HumanEval, WiC, ESNLI, and SNLI appear in Dolma documents.
  • Test-set contamination: Contaminated datasets were excluded from the downstream tasks used for model evaluation.
  • Code mixtures: Code-pretrained models outperform the C4-only model on GSM8k after fine-tuning with program-aided output, although all models fail under the reported in-context setup.

N.4 Data Preprocessing

Dolma preprocessing combines language identification, quality and content filters, deduplication, and source-specific transformations across web, social, code, encyclopedic, and book data. These steps also address sensitive information and distributional limitations.

  • Language filtering: All data sources are filtered with FastText language identification using an English threshold of 0.5.
  • Web preprocessing: Common Crawl data is linearized into plain text, deduplicated by URL and paragraph, and filtered using document-quality heuristics.
  • Quality filtering: Quality filters remove documents with abnormal n-gram repetition, length, line structure, symbols, or duplicated content.
  • Content filtering: Content filters remove toxic or hateful sentences, mask emails, phone numbers, and IP addresses, and remove pages containing at least six detected PIIs.
  • Source-specific preprocessing: Reddit preprocessing removes short, low-vote, or disallowed content, while code preprocessing applies language, copyright, and quality filters.
  • Books and encyclopedic content: Wikipedia and Wikibooks pages with fewer than 25 UTF-8 words are removed, while Gutenberg books are filtered for English and deduplicated by exact title.
  • Scope boundary: Dolma does not attempt to alter the distribution of social groups, leaving large-scale bias interventions for future work.
Loading 2402.00159v2…