PaperlayerView exact revision on arXiv

Source-linked AI summary

DataComp-LM: In search of the next generation of training sets for language models

Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng-Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Chandu, Thao Nguyen, Igor Vasiljevic, Sham Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El-Nouby, Hadi Pouransari, Alexander Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, Vaishaal Shankar

arXiv:2406.11794v4cs.LGcs.CL

TL;DR

Because studies of data curation often vary in architecture, compute, or hyperparameters, it is difficult to isolate which strategies work best. DCLM provides a fixed-recipe benchmark for comparing datasets, and DCLM-BASELINE achieves 64% on MMLU, state-of-the-art among open-data models.

  • Problem

    Data-curation studies often vary architecture, compute, or hyperparameters, leaving it unclear which strategies produce better training sets.

    Source: §1 · p. 2

  • Method

    DCLM benchmarks curated training sets by training language models with a fixed recipe and evaluating their downstream-task performance.

    Source: §1 · pp. 2–3

  • Results

    64% on MMLU: DCLM-BASELINE is state-of-the-art among open-data models, close to Mistral-7B-v0.3 (63%) and Llama 3 8B (66%), trained with up to 6.6× more compute.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results highlight systematic data curation as key to training performant language models.

    Source: §1 · p. 3

    Compute constraints limited ablations to individual design dimensions, prevented broader testing at larger scales or beyond 7B parameters, and left run-to-run variation insufficiently explored.

    Source: §6 · p. 10

Abstract

from arXiv · show

We introduce DataComp for Language Models (DCLM), a testbed for controlled dataset experiments with the goal of improving language models. As part of DCLM, we provide a standardized corpus of 240T tokens extracted from Common Crawl, effective pretraining recipes based on the OpenLM framework, and a broad suite of 53 downstream evaluations. Participants in the DCLM benchmark can experiment with data curation strategies such as deduplication, filtering, and data mixing at model scales ranging from 412M to 7B parameters. As a baseline for DCLM, we conduct extensive experiments and find that model-based filtering is key to assembling a high-quality training set. The resulting dataset, DCLM-Baseline enables training a 7B parameter language model from scratch to 64% 5-shot accuracy on MMLU with 2.6T training tokens. Compared to MAP-Neo, the previous state-of-the-art in open-data language models, DCLM-Baseline represents a 6.6 percentage point improvement on MMLU while being trained with 40% less compute. Our baseline model is also comparable to Mistral-7B-v0.3 and Llama 3 8B on MMLU (63% & 66%), and performs similarly on an average of 53 natural language understanding tasks while being trained with 6.6x less compute than Llama 3 8B. Our results highlight the importance of dataset design for training language models and offer a starting point for further research on data curation.

1 Introduction

DCLM standardizes experiments on language-model training data with a shared training recipe and evaluation suite, and provides a 240-trillion-token Common Crawl corpus. Its baseline experiments find model-based filtering important and produce a 7B model with 64% MMLU accuracy.

1 Introduction

  • DCLM addresses uncontrolled comparisons and scarce training-set details with a benchmark using fixed training recipes and evaluation on 53 downstream tasks.Participants curate datasets by filtering a shared pool or mixing their own data, then train and evaluate models at a chosen scale.Source: §1 · pp. 2–3, Figure 2 · p. 3
  • DCLM-POOL contains 240 trillion Common Crawl tokens, forming the corpus for the filtering track; the authors also provide open-source software for large-scale data processing.Source: §1 · p. 3
  • Experiments suggest models as small as 400M parameters can signal which training sets perform better at larger scales; DCLM spans five compute scales, about 600× from 400M models to over-trained 7B models.The range is intended to make the benchmark accessible to researchers with varying compute budgets.Source: §1 · p. 3
  • Across 416 baseline experiments, model-based filtering was key; at 7B and 280B tokens, filter-model choices shifted MMLU 5-shot accuracy from 35% to 44%, and a carefully trained bigram classifier performed best.Human quality judgments had limited value for identifying high-quality training data.Source: §1 · p. 3
  • 64% MMLU: DCLM-Baseline reaches this score with a 7B model trained on 2.6 trillion tokens, close to Mistral-7B-v0.3 (63%) and Llama 3 8B (66%), which uses up to 6.6× more compute.The paper describes this as state-of-the-art among open-data models.Source: §1 · p. 3

2 Related work

The related-work section positions DCLM against prior data-curation methods and open datasets, then distinguishes it from earlier data-centric benchmarks by its scale for language models.

Data curation for language models

Open-source datasets

  • 240T web-crawled tokens make DCLM’s raw pool the largest released, while its 3.8T-token DCLM-BASELINE yields better models than prior datasets.The related landscape includes C4 (160B tokens), The Pile (300B), FineWeb (15T), and RedPajama-v2 (30T), as well as domain-specific and filtered datasets.Source: §2 · p. 4

Data-centric benchmarks

  • With a 240T-token pool and 7B models, DCLM is the largest data-centric benchmark for language models.Earlier benchmarks include DataComp [61] and DataPerf [112] for vision, vision-language, and speech, Data-Juicer [36] for fine-tuning data, and the BabyLM challenge for small-scale pretraining.Source: §2 · p. 4

3 The DataComp for language models (DCLM) benchmark

DCLM standardizes a workflow for comparing language-model data curation, using a Common Crawl corpus, fixed-scale training, and a broad evaluation suite.

3.1 DCLM-POOL

  • DCLM-POOL contains 200B documents and 240T GPT-NeoX tokens, extracted from pre-2023 Common Crawl with resiliparse.Rather than decontaminating the pool directly, DCLM releases tooling to check test-set overlap and asks submissions to disclose contamination reports and avoid highly contaminated data.Source: §3.1 · p. 5

3.2 Competition scales: Supporting participants with different compute constraints

  • Competition scales span three orders of compute magnitude, with five parameter-and-Chinchilla-multiplier configurations and training tokens set to 20 × parameters × multiplier.The 1x multiplier corresponds to a compute allocation found near-optimal by Hoffmann et al. [79].Source: §3.2 · p. 5
  • Pearson’s r between smaller-scale and 7B-1x rankings was 0.838 at 400M-1x, 0.956 at 1B-1x, and 0.982 at 3B-1x, indicating curation rankings transfer across scales.Figure 3 presents the scale-consistency result, supporting iteration on curation at smaller scales.Source: Figure 3 · p. 5, §3.2 · p. 5

3.3 Benchmark tracks: Filtering and mixing

  • Participants choose a filtering track that selects from scale-specific DCLM-POOL subsets or a mixing track that combines DCLM-POOL with external sources.The initial random document pools are size-restricted by scale to encourage scalable filtering and reflect data download and storage constraints.Source: §3.3 · p. 5

3.4 Training

  • To isolate dataset interventions, DCLM fixes a training recipe at each scale using decoder-only Transformers implemented in OpenLM and unified data-processing utilities.Source: §3.4 · p. 6

3.5 Evaluation

  • The base-model evaluation suite covers 53 tasks and reports MMLU 5-shot accuracy, CORE centered accuracy over 22 tasks, and EXTENDED centered accuracy over all 53.CORE rescales each task's accuracy so random guessing is 0 and perfect accuracy is 1.Source: §3.5 · p. 6

4 Building high-quality training datasets with DCLM

The section builds DCLM-BASELINE from Common Crawl through a sequence of curation experiments, finding that model-based filtering is especially effective while extraction, deduplication, mixing, and decontamination also shape the process.

4.1 Evaluating existing training datasets

  • At the 7B-1x scale, RefinedWeb performed best on CORE and EXTENDED among the four evaluated open-source datasets, despite not mixing in high-quality sources.The DCLM experiments adopt RefinedWeb’s heuristic filters.Source: §4.1 · pp. 6–7, Table 2 · p. 7

4.2 Text extraction

  • Resiliparse and trafilatura improved CORE by at least 2.5 points over WET extraction, while resiliparse ran 8× faster and was selected for the remaining experiments.The comparison applied RefinedWeb’s heuristic quality filters to each extractor’s output.Source: §4.2 · p. 7, Table 3 · p. 8

4.3 Deduplication

  • At 7B-2x, MinHash and Bloom filtering had downstream performance within 0.2 CORE percentage points, while the modified Bloom filter scaled more easily beyond 10TB.DCLM-BASELINE uses Bloom filtering; other experiments use MinHash.Source: §4.3 · p. 7

4.4 Model-based quality filtering

  • fastText-based filtering outperformed the other tested model-based quality-filtering approaches.The comparison included PageRank, semantic deduplication, embedding-based classifiers, AskLLM, perplexity, and top-k average logits.Source: §4.4 · p. 8, Table 4 · p. 8
  • The OH-2.5 + ELI5 fastText recipe lifted CORE by 3.5 percentage points over conventional positive-data choices, and keeping the top 10% outperformed top-15% and top-20% thresholds.DCLM-BASELINE uses this classifier score to keep the top 10% of documents.Source: §4.4 · pp. 8–9, Table 5 · p. 9

4.5 Dataset mixing

  • Mixing high-quality sources improved lower-performing C4, RedPajama-CC, and RefinedWeb subsets, but hurt DCLM-BASELINE on average.The tested mixture used 67% Common Crawl and 33% Wikipedia, Books, Stack Exchange, arXiv, and GitHub.Source: §4.5 · p. 9, Table 6 · p. 9

4.6 Decontamination

  • Removing detected MMLU and Hellaswag overlaps did not decrease DCLM-BASELINE performance, and its MMLU contamination statistics were roughly similar to those of Dolma-V1.7 and FineWeb-Edu.The overlap-removal procedure flagged pages containing a question and at least one answer option, then removed matched strings.Source: §4.6 · p. 10, Table 7 · p. 10

5 Scaling up DCLM-BASELINE to the trillion token scale

Scaling DCLM-BASELINE to trillion-token training yields a 7B model that outperforms public-data 7B models and approaches several closed-data models. The section also reports strong instruction-tuned and 1B model results.

5 Scaling up DCLM-BASELINE to the trillion token scale

  • After two separate cool-downs, the recipe used Pouransari et al. [133]'s continual-pretraining methodology for 100B tokens on the same distribution, extending context length from 2048 to 8192.Source: §5 · p. 10
  • The 7B model outperforms all 7B models trained on public training sets and approaches Llama-8B, Mistral-7B, and Gemma-7B.The model was trained for 2.5T tokens on a 4.1T-token dataset combining DCLM-BASELINE, StarCoder, and ProofPile2 [14].Source: §5 · p. 10
  • 16.6 AlpacaEval2.0 LC Win-rate was achieved after instruction tuning, exceeding Gemma-Instruct (10.4) and approaching Mistral-v0.2-7B (17.1) and Llama3-Instruct (22.9).The model maintained most of its benchmark performance after instruction tuning on publicly available IT datasets.Source: §5 · p. 10
  • A 1B model trained on 4.3T combined DCLM-BASELINE, StarCoder, and ProofPile2 tokens outperformed prior small models, including Gemma-2B and Qwen2-1.5B.The authors characterize it as a strong small model.Source: §5 · p. 10

6 Conclusion and limitations … B Additional related work

DCLM-Baseline is reported to outperform models trained on open datasets and compete with models trained on private datasets. The authors describe the limits of their experiments and survey data-curation methods and goals.

6 Conclusion and limitations

  • Compute constraints limited ablations to individual design dimensions, experiments at larger scales, training beyond 7B parameters, and assessment of run-to-run variation.The authors also leave unexplored alternatives to sharded deduplication, differently trained filtering models, and other tokenizers beyond GPT-NeoX.Source: §6 · p. 10
  • DCLM-Baseline outperforms models trained on open datasets and is competitive with models trained on private datasets in the beyond 7B-2x-scale comparison.Table 8 compares the final model with other 7–8B parameter models.Source: Table 8 · p. 11
  • Models trained on DCLM-Baseline underperform on code and math; the authors identify domain-specific data and post-training, code-and-math coverage, fairness, multilinguality, and safety as future directions.They describe the current gap as a focus of DCLM’s first version, not an inherent limitation, and cite prior work on domain-specific training data and post-training [14] [202].Source: §6 · p. 11

B Additional related work

  • Data-curation methods target either performance or other goals, including language detection, heuristic and quality filtering, deduplication, data mixing, synthetic data, and removal of copyrighted, toxic, private, opt-out, or evaluation content.Language detection commonly uses a fastText classifier trained to identify 157 languages [71] [157]; other approaches include naive Bayes [138] and language- or country-associated URL selection [109] [131] [154]. Heuristics can target counts, repetition, existence, ratios, or statistics to remove undesirable web text.Source: B Additional related work · p. 37
  • Quality filtering commonly uses classifiers trained on perceived high- and low-quality sources or high-quality language-model perplexity, while newer approaches include prompted models, clustering, and selective loss masking.MiniPile’s clustered 1M-document subset maintained small-LM performance on GLUE while reducing training data; RHO-1 proposes Selective Language Modeling [98].Source: B Additional related work · p. 37
  • Deduplication methods range from URLs, hashes, and string metrics to model representations; data mixing methods include heuristics, evaluation-based weighting, Group DRO, bandits, and information-theoretic approaches.DoGE, Skill-it, and ShearedLlama build on principled mixing approaches; MixMax is provably optimal under a concave objective but is not proven at typical language-model scales, while BiMix estimates quantity and mixing-weight scaling laws using small models.Source: B Additional related work · pp. 37–38
  • Synthetic-data methods include generating pretraining text with GPT-series models or smaller models, and bootstrapping filtered examples for math, coding, commonsense, and mathematics tasks.Examples include Phi [2] [75], WRAP [110], ReSTEM [153], and STaR [194]; non-performance methods address copyrighted content, toxicity, private information, opt-outs, and benchmark leakage.Source: B Additional related work · p. 38

C Benchmark rules … D Tooling

The benchmark sets shared submission rules and distinct constraints for filtering and mixing, while its tooling supports corpus processing, contamination analysis, tokenization, training, and evaluation.

C Benchmark rules

  • The benchmark defines separate filtering and mixing tracks, with shared guidelines applying to both.The shared rules cover documentation, reproducibility, tokenization, code changes, and evaluation-data use.Source: C Benchmark rules · p. 38, §C.1 · p. 38

C.1 General rules

  • Submissions must document key components, make data or reproducible code freely available, use the provided tokenizer, preserve training and evaluation code, and restrict evaluation data to evaluation or decontamination.Submissions lacking freely available data or working reproduction code may still be accepted but are marked accordingly on the leaderboard.Source: §C.1 · p. 38

C.2 Filtering track

  • Filtering entries must process only the DCLM-POOL subset for their compute scale, without adding external data.Participants may use parsed DCLM-POOL or its specified Common Crawl archives, and may use externally trained models for filtering or paraphrasing, but not to backdoor external data.Source: §C.2 · pp. 38–39

C.3 Mixing track

  • Mixing entries may use any freely available data source that excludes evaluation data and must document sources, source weights, and training-token ratios to total custom-pool size.The training-token ratio is fixed for every benchmark scale.Source: §C.3 · p. 39

D Tooling

  • Tooling downloads Common Crawl WARC files and processes them with resiliparse, then provides released DCLM-POOL sizes and a sharded, parallel mapper framework for raw text.Mappers take a document and metadata as input; filters retain or discard documents, enrichers add metadata, and modifiers alter or split text.Source: D Tooling · p. 39
  • The framework includes RefinedWeb, Gopher, and C4 mappers, supports custom mappers, and provides corpus-level global functions such as cross-document span deduplication.Source: D Tooling · p. 39
  • Tooling measures contamination through shared consecutive sequences of at least 10 tokens and supplies GPT-NeoX tokenization with global shuffling in Ray-based and Rust-based implementations.The Ray implementation scales across nodes, while the more efficient Rust implementation runs only on single nodes.Source: D Tooling · p. 40
  • OpenLM-based training configurations and LLM-foundry-based evaluation produce model and results JSON files; the open-source framework and recipes reproduce the reported experiments.The paper lists 416 experiments in its model database.Source: D Tooling · p. 40

E DCLM-POOL

DCLM-POOL is a large collection of Common Crawl text from 2013–2022, packaged to support future updates. It preserves raw-web representativeness, including the presence of some PII and sensitive content.

E DCLM-POOL

  • 5.1M Common Crawl WARC dumps from 2013–2022 were processed with resiliparse; later years were omitted to limit generated text and provide a holdout.The pool comprises 5.1M compressed .jsonl files totaling 340TB on disk.Source: E DCLM-POOL · p. 40
  • DCLM-POOL follows Common Crawl’s robots.txt practices and may contain PII, but one-to-one WARC-to-.jsonl mapping enables updates after deletion requests and redactions.Common Crawl periodically redacts dumps and honors deletion requests.Source: E DCLM-POOL · p. 40
  • The pools preserve raw-web representativeness without special treatment for PII or sensitive content; some, including DCLM-BASELINE, filter malicious URLs.Common Crawl includes raw data collected from the web without filtering.Source: E DCLM-POOL · pp. 40–41

F Training details · G Evaluation details · G.1 Evaluation Tasks

The sections specify the model-training setup and evaluation framework, including a comparison of LightEval and LLM Foundry MMLU scores and a diverse suite of evaluation tasks.

F Training details

  • Training uses OpenLM, a PyTorch codebase targeting FSDP for decoder-only, pre-normalization Transformers inspired by GPT-2 and Llama.The setup follows Wortsman et al. [183] and Gadre et al. [62].Source: F Training details · p. 41
  • Models use bias-free LayerNorm, qk-LayerNorm, SwiGLU MLPs, depth-scaled initialization, and 2,048-token packed sequences with EOS separators and cross-document causal attention.Early experiments found little downstream impact from masking attention across documents.Source: F Training details · p. 41
  • Experiments span over 270 data distributions, mostly filtered from Common Crawl; most use GPT-NeoX tokenization with a 50k vocabulary and standard next-token prediction with z-loss.The z-loss encourages output logit magnitudes to remain numerically stable.Source: F Training details · p. 41
  • Hyperparameters vary by scale: 400M-1x and 1B-1x follow [62], 7B scales use sweep-guided settings, and earlier Table 2 and 5 runs used a 3e-4 learning rate and 0.33 weight decay.The 1B-1x alternatives in Table 12 underperformed the [62] settings; all experiments use a 3e-5 cooldown.Source: F Training details · pp. 41–42, Table 10 · p. 42, Table 11 · p. 42

G Evaluation details

G.1 Evaluation Tasks

  • The evaluation suite has CORE (22 tasks), selected for low-variance learning signals even at small scales, and EXTENDED (53 tasks), spanning diverse capabilities.Examples include LAMBADA’s narrative-context prediction, Big-Bench tasks such as elementary math QA and logical deduction, and MMLU’s 57-domain knowledge and problem-solving questions.Source: §G.1 · pp. 42–44
  • LightEval provides above-random MMLU signal earlier for weaker models, but its scores at larger scales appear capped lower and more tightly clustered than LLM Foundry scores.Figure 5 compares LightEval MMLU scores on the x-axis with LLM Foundry scores on the y-axis.Source: Figure 5 · p. 45

G.2 LightEval · H Hyperparameter study

The study finds that dataset rankings remain largely stable across hyperparameter settings, while LightEval and LLM Foundry show positively correlated MMLU scores but differ in how clearly they distinguish models at larger scales.

G.2 LightEval

  • LightEval and LLM Foundry show a positive correlation on MMLU 0-shot accuracy despite scoring answer sequences versus single-letter choices, respectively.LightEval uses log probabilities of entire answer sequences; LLM Foundry uses log probabilities of single letters.Source: §G.2 · p. 45
  • For 1B models, LightEval yields MMLU scores above random (25%), whereas LLM Foundry scores are around 0.25.The study reproduces FineWeb-Edu blog MMLU scores under LightEval.Source: §G.2 · p. 45
  • At larger scales, LightEval scores for Gemma-7B, Llama3-8B, and Mistral-7B cluster between 0.43 and 0.44, versus 0.56 to 0.62 in LLM Foundry.DCLM 7B-2x models also perform similarly in LightEval, while DCLM-7B is better by close to 10 points in LLM Foundry; the authors suggest LLM Foundry may give clearer signals for larger models.Source: §G.2 · pp. 45–46

H Hyperparameter study

  • At Default, DCLM-BASELINE beats RedPajama by 4.5 points and RedPajama beats C4 by 2 points; at 0.1x Learning Rate, the gaps are 3.3 and 0.9 points.Across five hyperparameter settings, the dataset ranking remains largely stable. When changing learning rate, weight decay is also adjusted to keep their product constant.Source: Table 12 · p. 46, H Hyperparameter study · p. 46

G.2 LightEval

  • The LightEval comparison uses MMLU as its sole representative task and does not evaluate other tasks.The authors suggest future comparisons with additional tasks and frameworks, including Eleuther LLM Harness.Source: §G.2 · p. 46

H Hyperparameter study

  • Optimal hyperparameter choice and dataset design provide orthogonal, complementary performance gains, addressing concerns that training-recipe differences could change which dataset appears optimal.The study reports preserved dataset orderings across combinations of learning rate and weight decay.Source: H Hyperparameter study · p. 46

I Architecture ablations · J Model-based quality filters · J.1 fastText classifiers

Dataset improvements generalize across the tested architecture variants, and the section details the fastText filtering implementation chosen for DCLM-Baseline, including its training data and feature choices.

I Architecture ablations

  • Better dataset filtering and hyperparameters seem orthogonal and stack on both MMLU and CORE benchmarks.The comparison is at 7B-1x scale and evaluates fastText filtering alongside a better hyperparameter setting.Source: Table 13 · p. 47
  • Dataset performance was highly correlated across Gemma-like and Mamba-like variants and OpenLM, indicating that dataset improvements generalize across architectures.The alternatives tested were a decoder-only transformer variant and a state-space model.Source: Figure 6 · p. 47, I Architecture ablations · p. 47
  • The architecture tests used a Gemma-inspired decoder-only transformer with GeGLU and RMS normalization [162], and a Mamba state-space model [74].The Gemma-like variant preserves the fundamental decoder-only transformer design, while Mamba changes the model's workings.Source: I Architecture ablations · p. 47

J Model-based quality filters

  • FastText classifiers were the method of choice for DCLM-Baseline among the model-based quality filters described in this section.The section gives further implementation details, especially for fastText.Source: J Model-based quality filters · p. 47

J.1 fastText classifiers

  • FastText classifiers learn positive labels from high-quality reference data and negative labels from web-crawled data, then score documents by positive-label probability and apply a percentile threshold.Expanding features from unigrams to unigrams and bigrams improved the quality of downstream filtered datasets.Source: §J.1 · p. 48, Table 14 · p. 48
  • Each fastText training set had 400K examples split evenly between positive and negative labels; negatives came from an earlier RefinedWeb version extracted with trafilatura.The authors hypothesize that trafilatura's more aggressive boilerplate removal could reduce classifiers' reliance on spurious features.Source: §J.1 · p. 48
  • Positive reference sets included English Wikipedia with “See Also” and “References” removed, OpenWebText2 [63], and Wikipedia/OpenWebText2/RedPajama book chunks up to 2,048 tokens.The book chunks were used because individual books are long.Source: §J.1 · p. 48
  • Another positive reference set combined 100K OH-2.5 examples with ELI5 posts and their top-scoring answers, retaining posts scored ≥0, answers scored ≥5, and threads with at least three comments.For tied top answers, the longest answer was selected.Source: §J.1 · p. 48

J.2 Other quality filtering baselines … L.2.2 Deduplication ablations: pipeline at 7B-1x / 7B-2x scale

This section examines quality-filtering alternatives and text extraction, alongside their reported limitations, and compares deduplication approaches across scales and document or paragraph resolutions. It also reports the selected minimum n-gram size for DCLM-BASELINE.

J.2 Other quality filtering baselines

  • PageRank-based selection did not improve the RefinedWeb reproduction: every score quintile performed worse than a random sample from the same pool.The comparison trained models at 1B-1x scale.Source: Table 15 · p. 49, §J.2 · p. 49
  • AskLLM slightly beat random sampling but significantly underperformed fastText and was not tested at larger scale because of its high application cost.The best tested configuration used Mistral-7B-Instruct-v0.2 [83], clipped documents at 1024 tokens, and scored cumulative probabilities of Yes and yes tokens.Source: §J.2 · pp. 49–50
  • Semantic deduplication using BGE-base embeddings, spherical k-means clustering, and removal of 25% of documents negatively impacted the trained model.The authors hypothesize that the embedding model affects outcomes and cite computational overhead as a reason to leave further work for future research.Source: §J.2 · p. 50

K.1 Profiling

  • Resiliparse and trafilatura produced documents at least 2x shorter than WET; resiliparse retained about 10% more text than trafilatura and ran roughly 8x faster.The comparison used a sample of 10 WARC files (900K pages); the authors note that WET includes lines such as navigation bars and boilerplate notices that seem of little pretraining value.Source: §K.1 · p. 50, Table 16 · p. 50

L.1 Deduplication methods

  • Document- and intra-document-level deduplication target whole-document copies, near-duplicates with small differences, and repeated boilerplate within otherwise distinct documents.The section motivates using multiple resolutions and fuzzy deduplication to capture these different forms of repetition.Source: §L.1 · p. 58
  • MinHash targets document-level near-duplicates using 5-token n-grams and Jaccard similarity 0.8; the experiments used 1,395 permutations in 93 buckets of 15 instead of 9,000 permutations.Those settings were chosen to approximate prior-work Jaccard-similarity plots at lower hash cost; Figure 7 compares duplicate-detection probabilities across bucket settings.Source: §L.1 · p. 58, Figure 7 · p. 58
  • BFF uses Bloom filters for space-efficient membership checks and was modified to deduplicate documents and paragraphs simultaneously, which the authors found vastly more efficient than MinHash and suffix arrays.Unlike MinHash’s document-versus-document comparison, BFF checks documents against the corpus.Source: §L.1 · p. 59

L.2.1 Deduplication ablations: pipeline at 1B-1x scale

  • At 1B-1x, experiments began with 76B heuristic-filtered Common Crawl tokens, applied deduplication, then sampled 28B tokens to train and evaluate CORE and measure token removal.The ablations examined each pipeline step’s contribution and which pipeline merited scaling.Source: §L.2.1 · p. 60
  • BFF performed comparably to Exact+MinHash+SuffixArray at 1B-1x, while suffix arrays appeared more helpful than MinHash alone, though that difference was within CORE-score variance.The authors describe multiresolution BFF as a potentially scalable alternative to the more expensive MinHash+SuffixArray pipeline.Source: §L.2.1 · pp. 60–61, Table 17 · p. 61

L.2.2 Deduplication ablations: pipeline at 7B-1x / 7B-2x scale

  • At 7B-1x, sharding parallelized deduplication but also affected token yield: more shards can preserve documents repeated only a few times, while fewer shards reduced yield and had variable CORE effects.The experiments used a tenth of Common Crawl, applied RefinedWeb preprocessing and deduplication, then fastText filtering and subsampling to 138B tokens.Source: §L.2.2 · p. 61
  • At 7B-1x, BFF with min_ngram_size 13 or 20 matched MinHash+SuffixArray on CORE and MMLU; at 7B-2x, BFF with min_ngram_size 13 and 10 shards was nearly identical.A min_ngram_size of 5 yielded competitive CORE but substantially lower MMLU; the authors selected 13 for DCLM-BASELINE.Source: §L.2.2 · pp. 61–62, Table 18 · p. 62, Table 19 · p. 62

L.2.3 BFF hyperparameter ablations · L.2.4 Global MinHash on Open Datasets

The sections identify BFF settings that preserve document statistics and examine what global MinHash reveals about duplicates remaining after deduplication. They show that BFF-deduplicated data can retain many MinHash-style duplicates without apparent downstream harm.

L.2.3 BFF hyperparameter ablations

  • Reducing min_ngram_size shortens mean and median documents, potentially damaging language statistics, while increasing max_ngram_size causes no dramatic change; the authors use 13 for both settings.The min_ngram_size warning is motivated by short repeated list elements, such as recipe ingredients, that may be removed and injure the source document.Source: Table 21 · p. 63, §L.2.3 · p. 63
  • A false positive rate of 0.1 suffices for the 75B-token pool, while 0.01 is used for larger pools.The experiment uses paragraph-only BFF, omitting full-document removal, with n-gram sizes 5 and 13 and threshold 0.8.Source: §L.2.3 · p. 62, Table 20 · p. 63
  • A threshold of 0.8 yields median tokens per document closest to baseline, while higher thresholds shorten documents and lower removal rates; 0.8 is used going forward.The threshold controls how close a document must be to previously seen n-grams to count as a duplicate.Source: §L.2.3 · p. 63
  • As pool size grows, BFF removal rates increase monotonically but concavely, while document lengths decrease; the authors use this behavior to guide sharding for target token yield.For DCLM-BASELINE’s approximately 70TB deduplication input, they use 100 shards of roughly 700GB each, based on BFF’s best performance at a 10-way shard with roughly 7TB input.Source: §L.2.3 · pp. 63–64, Figure 8 · p. 64, Table 22 · p. 64

L.2.4 Global MinHash on Open Datasets

  • BFF-deduplicated pools retain many MinHash-style fuzzy duplicates, and 100-shard BFF deduplication leaves many duplicates in DCLM-BASELINE without apparent downstream harm.The findings suggest either that only large duplicate quantities harm performance or that aggressive single-shard deduplication removes high-quality documents; the authors leave this for future work.Source: §L.2.4 · pp. 64–65, Table 23 · p. 65
  • Global MinHash results indicate that sharded deduplication misses many duplicates, whereas MinHash is roughly idempotent.The evaluation applies global MinHash to DCLM-BASELINE, official RefinedWeb, a RefinedWeb emulation, and Dolma V1; it uses 14 buckets of size 9.Source: §L.2.4 · p. 64, Table 23 · p. 65

M Mixing sources

Filtering RedPajama sources with DCLM-BASELINE’s fastText classifier before mixing them in did not improve performance: DCLM-BASELINE alone performed best. Further analysis of filtering and mixing non-Common Crawl sources is left for future work.

M Mixing sources

  • The experiment applied DCLM-BASELINE’s fastText classifier to RedPajama sources, motivated by the hypothesis that more stringent filtering could improve mixing.The authors had found that mixing commonly used sources did not improve general performance and hypothesized that DCLM-BASELINE’s more stringent Common Crawl filtering explained this.Source: M Mixing sources · p. 65
  • Further analysis of filtering and mixing non-Common Crawl sources is left for future work, including exploration through the mixing track.Source: M Mixing sources · p. 65
  • At 1B-1x scale, DCLM-BASELINE alone performed best; adding fastText-filtered RedPajama data still decreased performance.The experiment kept only the highest-scored items from several RedPajama sources before adding them to the pretraining dataset.Source: Table 24 · p. 65, M Mixing sources · p. 65

N Human judgment … Doc4 (Good)

Human agreement with quality filters did not predict which filters produced stronger CORE results, and filter agreement generally showed no clear relationship with downstream performance. The study also documents its annotation process and supplies labeled document examples.

N Human judgment

  • ∼82% ROC-AUC for AskLLM coincided with ∼28.5% CORE, while fastText filters at ∼73% ROC-AUC achieved >31% CORE, so human-label agreement did not predict filter quality.Figure 9 reports the comparison on MAJORITY annotations; the AskLLM filter agrees more with human labels but performs worse as a quality filter.Source: Figure 9 · p. 66, N Human judgment · p. 66
  • Across downstream tasks, filter ROC-AUC showed no apparent relationship with model scores; StrategyQA and SQuAD differed in direction, with R2 < 0.3 for both.The authors suggest human intuition may not reliably identify useful training documents and hypothesize human curation may reduce dataset diversity.Source: Figure 9 · p. 66, N Human judgment · pp. 66–67
  • Sixteen English-speaking AI graduate students and professors labeled about 500 documents, with three labels per document, 71% average inter-annotator agreement, and 281 unanimous samples.The full set was called MAJORITY and the unanimous subset AGREEMENT; the 499-document sample followed rule-based filtering and deduplication.Source: N Human judgment · pp. 66–67

N.1 Instructions given to annotators

  • Annotators rated randomly sampled DCLM-RefinedWeb documents for LLM-training usefulness, considering evaluation-task relevance, open-ended task utility, and harmful content.The instructions broadly referenced reading comprehension, language understanding, world knowledge, and commonsense reasoning.Source: §N.1 · p. 68

Doc1 (Bad)

  • The example labeled bad is a prayer-line webpage listing service schedules, phone numbers, moderators, recordings, and testimonials.Source: Doc1 (Bad) · p. 68

Doc2 (Bad)

  • The example labeled bad is a radio-program promotion describing its host, format, music, and station contact information.Source: Doc2 (Bad) · p. 68

Doc3 (Good)

  • The example labeled good is a Chinese five-spice mushroom noodle recipe with sequential preparation and cooking instructions.Source: Doc3 (Good) · p. 69

Doc4 (Good)

  • The example labeled good is an explanatory travel-book article outlining advantages and disadvantages, including convenience, information, and cost.Source: Doc4 (Good) · p. 69

O Decontamination … R Account of compute costs

Across decontamination, instruction tuning, scaling, and compute accounting, the appendix tests contamination sensitivity, reports gains from instruction tuning and long-context adaptation, and summarizes training costs. Its results indicate that instruction-tuning data and extended-context training can improve specific evaluations, while the compute estimate exceeds tracked training hours.

O Decontamination

  • For the 7B-2x model, performance changes little on “not dirty” or “clean” evaluation subsets, including MMLU; the authors caution that contamination thresholds may misclassify samples.“Not dirty” means less than 80% contaminated tokens, while “clean” means less than 20%; contamination is identified using overlapping 10-grams or longer.Source: O Decontamination · p. 70, Figure 11 · p. 70, Figure 12 · p. 71

P Instruction tuning

  • DCLM-IT exceeds DCLM-BASELINE w/ OH-2.5 by 2.8 percentage points; the OH-2.5-tuned model beats Zephyr-Beta-7B and Gemma-Instruct-7B but slightly trails Mistral-7B-OH-2.5.The instruction-tuned models do not undergo alignment procedures such as PPO or DPO.Source: P Instruction tuning · pp. 71–72, Table 26 · p. 72

Q Scaling up

  • The 2.5T-token 7B run uses 2T tokens before cooldown, then a reweighted mix of 70% tighter-threshold DCLM-BASELINE and 30% ProofPile; its final model averages two cooldown runs.The model soup weights the 270B-token cooldown at 0.8 and the 200B-token cooldown at 0.2, followed by 100B tokens of long-context training.Source: Q Scaling up · pp. 72–73, Table 28 · p. 73

Q.1 Instruction tuning the scaled up model

  • GSM8K performance rises from 2.5% to 52.5% after instruction-tuning the scaled-up model, with comparable performance to models that mix instruction-tuning data into pretraining.The comparison is for 5-shot GSM8K with chain-of-thought; instruction tuning also brings considerable gains on this evaluation while maintaining MMLU performance.Source: §Q.1 · p. 73, Table 29 · p. 73

Q.2 Continual learning to extend context length

  • Continual learning on approximately 120B tokens adapts the 7B model from a 2048 to an 8192 context length, matching regular evaluations while significantly improving multi-document question answering.The evaluation varies the position of a document containing the answer across contexts with 1, 10, 20, or 30 documents.Source: §Q.2 · pp. 73–74, Table 30 · p. 74

Q.3 Scaling up the 1B model

  • The 1.4B model is trained for 4.3T tokens on DCLM-BASELINE combined with StarCoder and ProofPile2, without a separate cooldown or model-soup phase.The passage refers to results in Table 32 but does not supply their values.Source: §Q.3 · p. 74

Q.4 Full results from Figure 1

  • Table 33 reproduces the Figure 1 results in table form to facilitate future comparisons.The supplied passage does not provide Table 33’s result values.Source: §Q.4 · p. 74

R Account of compute costs

  • Tracked training consumed 859K H100 hours, while the total DCLM run estimate is approximately 1.2M H100 hours, including additional untracked compute such as failed runs.Tracked hours comprised 1,700 for 411M models, 140,000 for 1B, 4,800 for 3B, and 713,216 for 7B models.Source: R Account of compute costs · p. 74

S Bias and Toxicity Evaluations · T Existing assets used

DCLM-Baseline shows broadly similar bias and toxicity to other base models, but identifies toxic comments less accurately; the paper also describes the assets and licenses used in its benchmark.

S Bias and Toxicity Evaluations

  • The evaluations cover CivilComments, Copyright, BBQ, WinoGender, and RealToxicityPrompts; the section also reviews metrics and their direction of improvement.Copyright measures the ability to reiterate copyrighted content using LCS, where lower is better; Real Toxicity Prompts uses Toxic Fraction, where lower is better.Source: S Bias and Toxicity Evaluations · pp. 74–75
  • DCLM-Baseline shows relatively similar bias and toxicity to models trained on preexisting datasets, with one stated weakness in toxic-comment identification.The paper compares it with popular base models including Llama and Mistral.Source: S Bias and Toxicity Evaluations · p. 75
  • Its Real Toxicity Prompts scores indicate toxic-content generation similar to that of other popular base models.Real Toxicity Prompts measures how easily a user can prompt a model to generate toxic content; lower Toxic Fraction is better.Source: S Bias and Toxicity Evaluations · p. 75
  • BBQ and Winogender results indicate that DCLM-Baseline generations contain biases similar to those of other base models.BBQ measures bias effects on question-answering choices, while Winogender measures reinforcement of gender-based stereotypes when infilling a pronoun.Source: S Bias and Toxicity Evaluations · p. 75
  • DCLM-Baseline identifies toxic comments less accurately than other base models on CivilComments.CivilComments reports Exact Match, with higher scores indicating better toxic-content identification.Source: S Bias and Toxicity Evaluations · p. 75

T Existing assets used

  • The benchmark documentation identifies the assets used and their associated licenses; BBQ and Winogender report Exact Match, with higher scores better.BBQ measures bias in question-answering choices, and Winogender measures gender-stereotype reinforcement when infilling a gendered pronoun.Source: T Existing assets used · p. 75, S Bias and Toxicity Evaluations · p. 75

T.1 Evaluation data … Q59 Any other comments?

This section documents DCLM’s evaluation data, source licenses, purpose, composition, and scale, and describes DCLM-POOL’s collection and processing as a Common Crawl-derived research dataset. It presents the datasets as resources for community research on dataset progress and safer data curation, while noting representativeness limits, noise, privacy risks, potentially harmful content, and use restrictions.

T.1 Evaluation data

  • The evaluation appendix identifies licenses for the benchmark datasets, including Creative Commons, MIT, Apache 2.0, and BSD-2-Clause terms, as well as datasets without a specified license.Source: §T.1 · pp. 75, 77

T.2 Raw sources

  • Common Crawl is DCLM-POOL’s and DCLM-BASELINE’s main external source, but its terms impose restrictions, require legal compliance, and require users to respect third-party copyrights.The authors urge users to verify that their use complies with the terms; supplementary sources also have varying license conditions, including OpenHermes restrictions on commercial model training and underlying-dataset terms for StarCoder and ProofPile2.Source: §T.2 · pp. 78–79

T.3 Libraries

  • The benchmark pipeline uses libraries including Transformers and PyTorch; OpenLM is MIT-licensed, llm-foundry uses Apache 2.0, and commercial users are urged to check package licenses.Source: §T.3 · p. 79, §3 · p. 79, §7 · p. 80

U.1 Motivation

  • DCLM was created to enable controlled study of what makes strong pretraining datasets, addressing prior comparisons confounded by differing architectures, training choices, and evaluation settings.Its initial release focuses on 53 downstream language-understanding tasks, including reasoning and math.Source: §U.1 · p. 81

Q4 Any other comments?

  • The authors anticipate that DCLM’s benchmark, tooling, and pools will support data-centric research, better web-scale datasets and model abilities, lower training costs, and knowledge sharing.Source: Q4 Any other comments? · p. 81

U.2 Composition

  • Each dataset instance is a web-crawled page with its URL and HTML content, plus crawl-time and other metadata such as detected language in processed data.Source: §U.2 · p. 81

Q6 How many instances are there in total (of each type, if appropriate)?

  • DCLM-POOL contains approximately 200B documents, while DCLM-BASELINE contains approximately 3B documents.The pool’s documents are one instance type and come from hundreds of millions of sources.Source: Q6 How many instances are there in… · p. 82

Q7 Does the dataset contain all possible instances or is it a sample (not necessarily random)

  • DCLM-POOL covers Common Crawl data before 2023, but Common Crawl omits parts of the web and DCLM-BASELINE filtering and deduplication make it non-representative.The baseline processing is intended to improve pretraining quality by removing low-quality instances.Source: Q7 Does the dataset contain all… · p. 82
  • Each instance is full extracted HTML text with metadata; the dataset has no deliberately established instance relationships or recommended train, validation, and test splits.As a Common Crawl snapshot, it may contain placeholder text, broken links, extraction failures, and duplicates.Source: Q7 Does the dataset contain all… · p. 82

Q14 Is the dataset self-contained, or does it link to or otherwise rely on external resources

U.3 Collection process

Q23 What mechanisms or procedures were used to collect the data (e.g., hardware apparatus

  • DCLM-POOL contains Common Crawl data from before 2023 rather than a probabilistic sample; competition pools are random subsets of Common Crawl paths.Common Crawl itself is a sample of the web.Source: Q23 What mechanisms or procedures were… · p. 84

Q26 Over what timeframe was the data collected? Does this timeframe match the creation

  • No formal ethics review was conducted and individuals were not notified; the team respects robots.txt and plans a mechanism to request URL exclusions.Private content may nevertheless have been posted online without consent.Source: Q26 Over what timeframe was the data… · p. 85

Q34 Any other comments?

Q42 Is there anything about the composition of the dataset or the way it was collected

Q44 Any other comments?

  • The authors designed DCLM-POOL and DCLM as a benchmark for measuring dataset progress across dimensions such as performance on diverse tasks, supporting knowledge sharing and more accessible research.They describe this as essential to developing more effective and safer datasets while minimizing redundant effort.Source: Q44 Any other comments? · p. 87

U.6 Distribution

  • DCLM-POOL and DCLM-BASELINE are publicly and freely downloadable through Common Crawl’s S3 bucket and HuggingFace.Several DCLM-POOL subsets and tools for sampling are provided to support experiments at different scales.Source: §U.6 · p. 87, Q51 Any other comments? · p. 88

Q47 When will the dataset be distributed?

Q48 Will the dataset be distributed under a copyright or other intellectual property (IP)

Q49 Have any third parties imposed IP-based or other restrictions on the data associated

Q51 Any other comments?

  • The datasets are intended solely for scientific research, not production systems or decision-making software involving individuals.The authors say they bear responsibility for rights violations, encourage users to respect privacy and local and national laws, and do not accept responsibility for others’ misuse.Source: Q42 Is there anything about the… · p. 87, Q51 Any other comments? · p. 88

Q52 Who will be supporting/hosting/maintaining the dataset?

  • Common Crawl hosts DCLM-POOL and DCLM-BASELINE, while competition pools are also on HuggingFace; the DCLM team maintains these assets.Source: Q52 Who will be… · p. 88

Q53 How can the owner/curator/manager of the dataset be contacted (e.g., email address)?

  • The authors have no plans to update DCLM-POOL to preserve competition comparability, do not plan to maintain deprecated versions, and will address takedown requests and proposed modifications individually.They describe the pool as inherently noisy; errata or deprecation and modification notices are to be communicated publicly through the DCLM website.Source: Q53 How can the owner/curator/manager… · pp. 88–89

Related papers

Loading 2406.11794v4…