PaperlayerView exact revision on arXiv

Source-linked AI summary

DataComp-LM: In search of the next generation of training sets for language models

Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng-Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Chandu, Thao Nguyen, Igor Vasiljevic, Sham Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El-Nouby, Hadi Pouransari, Alexander Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, Vaishaal Shankar

arXiv:2406.11794v4cs.LGcs.CL

TL;DR

Language-model data research lacks controlled comparisons and sufficient transparency about the ingredients of state-of-the-art training sets. DCLM provides a standardized benchmark for dataset curation using shared resources, fixed training procedures, and downstream evaluations; its baseline finds model-based filtering important and achieves competitive results with substantially less compute.

  • Problem

    Language-model data research lacks controlled comparisons and public detail about training sets, making it unclear which curation strategies and ingredients work best.

  • Method

    DCLM evaluates proposed training sets and curation algorithms with fixed training recipes across compute scales, using a 240T-token Common Crawl pool and downstream-task evaluation.

  • Results

    DCLM-Baseline reaches 64% 5-shot accuracy on MMLU for a 7B model trained on 2.6T tokens, close to Mistral-7B-v0.3 at 63% and Llama 3 8B at 66%.

  • Takeaways & Limitations

    The baseline results highlight model-based filtering and systematic dataset design as important components for assembling effective language-model training data.

  • Takeaways & Limitations

    The study could not test all design approaches at larger scales, train beyond 7B parameters, sufficiently explore run-to-run variation, or broadly vary tokenizers.

Abstract

from arXiv · show

We introduce DataComp for Language Models (DCLM), a testbed for controlled dataset experiments with the goal of improving language models. As part of DCLM, we provide a standardized corpus of 240T tokens extracted from Common Crawl, effective pretraining recipes based on the OpenLM framework, and a broad suite of 53 downstream evaluations. Participants in the DCLM benchmark can experiment with data curation strategies such as deduplication, filtering, and data mixing at model scales ranging from 412M to 7B parameters. As a baseline for DCLM, we conduct extensive experiments and find that model-based filtering is key to assembling a high-quality training set. The resulting dataset, DCLM-Baseline enables training a 7B parameter language model from scratch to 64% 5-shot accuracy on MMLU with 2.6T training tokens. Compared to MAP-Neo, the previous state-of-the-art in open-data language models, DCLM-Baseline represents a 6.6 percentage point improvement on MMLU while being trained with 40% less compute. Our baseline model is also comparable to Mistral-7B-v0.3 and Llama 3 8B on MMLU (63% & 66%), and performs similarly on an average of 53 natural language understanding tasks while being trained with 6.6x less compute than Llama 3 8B. Our results highlight the importance of dataset design for training language models and offer a starting point for further research on data curation.

1 Introduction

DCLM addresses uncontrolled comparisons and limited transparency in language-model training data by standardizing dataset curation experiments. Its baseline results show that model-based filtering and systematic dataset design can produce competitive models across downstream evaluations.

  • Motivation and benchmark: DCLM addresses the lack of controlled comparisons by evaluating proposed datasets with a fixed training recipe and downstream-task measurements.This separates dataset quality from differences in architecture, hyperparameters, and compute.
  • Benchmark resources: DCLM-POOL provides 240 trillion tokens derived from Common Crawl as the corpus for the filtering track.The framework also provides open-source software for processing large datasets with several filtering approaches.
  • Benchmark resources: The testbed spans five compute scales from 400M-parameter models to over-trained 7B models, covering about 600× in compute.The authors report that 400M-parameter models can provide signal about which training sets perform better at larger scales.
  • Baseline findings: 416 baseline experiments identify model-based filtering as a key component of effective data curation.At the 7B-parameter scale with 280B training tokens, classifier choices produce 35% to 44% accuracy on MMLU 5-shot.
  • Baseline findings: A 7B model trained on 2.6 trillion DCLM-Baseline tokens achieves 64% on MMLU, close to Mistral-7B-v0.3 at 63% and Llama 3 8B at 66%.The DCLM-Baseline model is trained with up to 6.6× less compute than Llama 3 8B.
  • Resources and impact: The framework, models, and training sets are publicly released to support participation and strengthen empirical foundations for data-centric language-model research.The release is intended to enable other researchers to participate in DCLM.

2 Related work

Related work spans data curation, increasingly large open-source datasets, and data-centric benchmarks. DCLM extends this landscape with a 240T-token pool and 7B models for language-model data evaluation.

  • Data curation for language models: Web crawls commonly supply large language-model datasets, but their undesirable content motivates language, heuristic, quality, and other filtering methods.The cited related work includes approaches for filtering by language and using heuristics or quality criteria.
  • Open-source datasets: Open-source training datasets have expanded from C4 at 160B tokens and The Pile at 300B tokens to FineWeb at 15T tokens and RedPajama-v2 at 30T tokens.The passage also identifies RefinedWeb at 600B tokens and Dolma at 3T tokens.
  • Data-centric benchmarks: Data-centric benchmarks have previously evaluated dataset improvements across vision, vision-language, speech, fine-tuning, and small-scale language-model pretraining.Examples include DataComp, DataPerf, Data-Juicer, and the BabyLM challenge.
  • Data-centric benchmarks: DCLM is described as the largest data-centric benchmark for language models, combining a 240T-token pool with models up to 7B parameters.This distinguishes its scale from the smaller BabyLM pretraining setting described in the passage.

3 The DataComp for language models (DCLM) benchmark

DCLM is a benchmark for comparing language-model training datasets under standardized data, training, and evaluation conditions across multiple compute scales. Its design supports controlled dataset curation experiments and indicates that small-scale rankings transfer to larger models.

  • DCLM provides a workflow that selects a competition scale, curates data from DCLM-POOL, trains with fixed hyperparameters, and evaluates the resulting model.The benchmark includes filtering and mixing tracks for dataset construction.
  • 240T tokens from Common Crawl form DCLM-POOL, the unfiltered corpus underlying the benchmark.DCLM-POOL contains 200B documents and is re-extracted from HTML with resiliparse.
  • Five competition scales specify model size and Chinchilla multiplier, spanning compute from 400M-1x to over-trained 7B settings.The scales are designed for researchers with different compute constraints.
  • Pearson’s r = 0.838, 0.956, and 0.982 link rankings at 400M-1x, 1B-1x, and 3B-1x scales to 7B-1x results.These correlations suggest that better curation strategies at smaller scales transfer to larger scales.
  • The benchmark evaluates base models on 53 downstream tasks, including question answering, generation, mathematics, textbook knowledge, and common-sense reasoning.MMLU 5-shot accuracy is one of the main metrics used to assess curation algorithms.

4 Building high-quality training datasets with DCLM

DCLM experiments progressively construct DCLM-BASELINE from Common Crawl by comparing existing datasets, extraction methods, deduplication, quality filters, data mixtures, and contamination removal. The experiments identify model-based filtering, especially a fastText classifier, as central to the resulting dataset.

  • Existing datasets: RefinedWeb performs best among the compared open-source datasets at the 7B-1x scale despite not mixing additional high-quality sources.Its pipeline uses Common Crawl extraction, heuristic quality filters, and deduplication.
  • Text extraction: Resiliparse and trafilatura improve CORE by at least 2.5 points over WET extraction, while resiliparse is 8× faster than trafilatura.Both stricter extractors have similar downstream performance, making resiliparse more practical for large-scale processing.
  • Model-based filtering: DCLM-BASELINE retains the top 10% of documents using a fastText OH-2.5 + ELI5 classifier score.The selected classifier and percentile threshold define the baseline filtering step.
  • Contamination: Removing detected MMLU and Hellaswag overlaps does not decrease performance, so gains on these tasks are not likely caused by increased presence of their test examples.The decontamination comparison is performed at the 7B-2x scale.

5 Scaling up DCLM-BASELINE to the trillion token scale

The authors scale DCLM-BASELINE to a 4.1T-token mixture with StarCoder and ProofPile2, then train and extend a 7B model for broader capability. The resulting model outperforms public-data 7B models and approaches several closed-data models.

  • A 4.1T-token dataset combines 3.8T DCLM-BASELINE tokens with StarCoder and ProofPile2 to test performance at larger compute scales.The 7B model is trained for 2.5T tokens using the largest competition-scale hyperparameters with modifications for math and coding.
  • The scaled model outperforms all 7B models trained on public training sets and approaches Llama-8B, Mistral-7B, and Gemma-7B.These closed-data models were trained for more tokens.
  • After instruction tuning, the model achieves a 16.6 AlpacaEval2.0 LC Win-rate, exceeding Gemma-Instruct’s 10.4 and approaching Mistral-v0.2-7B’s 17.1.It retains most of its benchmark performance after instruction tuning.
  • A 1B model trained on 4.3T combined tokens from DCLM-BASELINE, StarCoder, and ProofPile2 outperforms prior small models including Gemma-2B and Qwen2-1.5B.

6 Conclusion and limitations

The authors present DCLM as a starting point for data-curation research, while acknowledging limits in scale, experimental breadth, tokenizer coverage, and domain performance.

  • The study could not test all approaches at larger scales, train models beyond 7B parameters, or sufficiently explore run-to-run variation.
  • The authors did not explore many DCLM-BASELINE variants, including alternative sharded deduplication methods and differently trained filtering models.
  • Most experiments used one GPT-NeoX tokenizer, although other tokenizers may perform better on multilingual and math tasks.
  • DCLM-BASELINE outperforms models trained on open datasets and is competitive with models trained on private datasets.
  • DCLM-BASELINE models are competitive on common language-understanding tasks but currently perform less well on code and math.

C Benchmark rules

DCLM standardizes how participants construct, process, tokenize, train, evaluate, and document datasets across filtering and mixing tracks to support reproducible comparisons.

  • Submissions must document key components and freely provide the underlying dataset or fully working reproduction code when possible.
  • Submissions must use the provided tokenization and global-shuffling script and cannot change the training or evaluation code.
  • Use of evaluation test data is forbidden except for evaluation and decontamination.
  • Filtering-track datasets must be built from the designated DCLM-POOL subset without external data, whereas mixing-track submissions may use external sources.
  • Participants may modify HTML extraction and use externally trained models for processing, but may not introduce external data through a backdoor.
  • The framework supports document filters, metadata enrichers, text modifiers, corpus-level operations, standardized evaluation, and reproducible training workflows.

E DCLM-POOL

DCLM-POOL is built from Common Crawl material and preserves raw-web representativeness while documenting metadata and mechanisms for opt-out and redaction updates.

  • DCLM-POOL uses Common Crawl WARC dumps from 2013–2022, with text extracted from HTML using resiliparse.
  • DCLM-POOL preserves a one-to-one mapping between raw WARC files and released JSONL files, enabling updates based on Common Crawl redactions.
  • Common Crawl respects robots.txt, giving content creators a mechanism to opt out of Common Crawl and DCLM-POOL.
  • Common Crawl data is raw and unfiltered; DCLM-POOL therefore includes web content without special treatment for PII and sensitive content.
  • DCLM-POOL provides metadata for the released data.

F Training details

The training setup uses OpenLM-based decoder-only Transformers, standardized tokenization and optimization, and evaluation spanning CORE and EXTENDED task suites.

  • Training setup: Training uses OpenLM with decoder-only, pre-normalization Transformers inspired by GPT-2 and Llama.
  • Architecture: The architecture uses bias-free LayerNorm, qk-LayerNorm, SwiGLU MLPs, depth-scaled initialization, and 2,048-token sequences.
  • Training data: Models are trained on over 270 data distributions, mostly filtered from Common Crawl, with GPT-NeoX tokenization producing a 50k vocabulary.
  • Optimization: Optimization uses standard next-token prediction with z-loss to keep output logit magnitudes numerically stable.
  • Evaluation: Evaluations are divided into CORE with 22 tasks and EXTENDED with 53 tasks, covering diverse model capabilities.
  • Evaluation: The evaluation suite includes reasoning, science, language, commonsense, question answering, legal, analytical, math, and bias-related tasks.

G.2 LightEval

This section compares LightEval with LLM Foundry for MMLU and examines how evaluation choices and training settings affect dataset comparisons. LightEval is more informative for smaller models, whereas LLM Foundry provides clearer separation at larger scales.

  • Evaluation framework comparison: LightEval evaluates multiple-choice answers using entire answer-sequence log probabilities, while LLM Foundry uses single-letter log probabilities.The two frameworks nevertheless show positive correlation on MMLU 0-shot accuracy.
  • Hyperparameter robustness: Dataset rankings remain largely stable across five learning-rate and weight-decay settings, with DCLM-BASELINE outperforming RedPajama and C4.At the best setting, DCLM-BASELINE exceeds RedPajama by 4.5 points, while RedPajama exceeds C4 by 2 points.
  • Evaluation framework comparison: DCLM-7B is better by close to 10 points in LLM Foundry, despite performing quite similarly to DCLM 7B-2x in LightEval.The discrepancy illustrates that framework choice can materially affect apparent model differences.
  • Limitations: The study uses MMLU as a representative task and does not evaluate other tasks or frameworks such as Eleuther LLM Harness.The authors identify broader task and framework comparisons as future work.

I Architecture ablations

The architecture ablations test whether dataset performance transfers from the OpenLM architecture to both a closely related Gemma-like variant and the substantially different Mamba architecture. Dataset performance is highly correlated across these architectures, supporting cross-architecture consistency.

  • Ablated architectures: The study evaluates dataset performance with a Gemma-inspired decoder-only transformer variant and the substantially different Mamba state-space architecture.The Gemma-like model changes the activation and adds RMS normalization, while Mamba changes the model family more fundamentally.
  • Cross-architecture results: High correlation between dataset performance on both architecture variants and OpenLM indicates that dataset improvements generalize across architectures.The comparison covers both small architectural changes and a major shift from transformers to state-space modeling.
  • Filtering implementation: FastText classifiers are the chosen implementation for DCLM-BASELINE’s model-based quality filtering.The classifiers distinguish positive high-quality reference data from negative web-crawled data and score pool documents by predicted positive-label probability.
  • Filtering ablation: Adding bigrams to the fastText feature space improves performance over the default unigram-only setting.This ablation is reported at the 7B-1x scale.
  • Filtering implementation: Each fastText training set contains 400K examples, split evenly between 200K positive and 200K negative examples.Negative examples are sampled from an earlier RefinedWeb reproduction, while experiments focus on constructing positive reference data.

J.2 Other quality filtering baselines

The authors compare alternative quality-filtering baselines with fastText filtering for Common Crawl-derived data. PageRank and AskLLM are less effective than the fastText approach, while semantic deduplication and extraction comparisons are examined separately.

  • Alternative filters: None of the examined alternative quality filters is as effective as the fastText methods used for DCLM-BASELINE.The section introduces PageRank and AskLLM as baselines for comparison.
  • PageRank: PageRank-based selection performs worse than random sampling from the same initial pool across every tested quintile.The experiment uses host-level PageRank scores to partition the RefinedWeb reproduction into quintiles.
  • AskLLM: AskLLM prompts an instruction-tuned model to judge whether documents are well-structured, informative, useful for open-ended tasks, and free from harmful content.The procedure reduces the judgment to a yes-or-no suitability decision after clipping overly long documents.
  • AskLLM: AskLLM performs only slightly better than random sampling and significantly underperforms the fastText experiments.Because of its high application cost, the authors do not scale this experiment further.
  • Semantic deduplication: Semantic deduplication embeds documents, clusters them with k-means, and retains one document from each closely related group.The stated goal is to encourage dataset diversity.
  • Text extraction: The section also compares resiliparse with WET files and trafilatura as alternative extraction methods.These comparisons cover both quantitative and qualitative differences among extractors.

K.1 Profiling

The section profiles text extraction and evaluates deduplication pipelines for constructing DCLM datasets. It compares MinHash, suffix-array, and Bloom-filter approaches across scales and selects BFF settings based on downstream performance and scalability.

  • Text extraction: 2x shorter documents on average: resiliparse and trafilatura produce substantially shorter outputs than WET files.The shorter outputs reflect the removal of low-value lines such as navigation bars, boilerplate notices, and copyright statements.
  • Deduplication methods: MinHash uses Jaccard similarity over token n-grams to identify near-duplicate documents, but the experiments reduce prior work’s 9,000 permutations to 1,395.The revised configuration uses 93 buckets of 15 hashes rather than 450 buckets of 20 hashes, targeting similar similarity behavior at lower cost.
  • Deduplication methods: Bloom-filter deduplication performs document- and paragraph-level filtering simultaneously and is substantially more efficient than a MinHash-plus-suffix-array pipeline.Unlike MinHash’s document-versus-document comparison, BFF compares documents against a corpus-level structure.
  • Deduplication ablations: Sharding enables parallel and multi-node deduplication, but larger shards are more expensive, less parallelizable, and can reduce token yield.More shards also leave a larger token pool because fewer documents are compared against one another.
  • Deduplication ablations: 13-token minimum n-grams: BFF matches MinHash-plus-suffix-array performance, whereas a 5-token minimum yields competitive CORE but much worse MMLU results.The experiments therefore select a minimum n-gram size of 13 for DCLM-BASELINE.

L.2.3 BFF hyperparameter ablations

The ablations examine how Bloom filter deduplication settings alter dataset removal rates and document statistics. They favor conservative n-gram choices and show that larger pools and sharding change duplicate removal without clearly harming downstream performance.

  • Min n-gram size: Lowering min_ngram_size shortens mean and median documents, potentially damaging dataset language statistics.Repeated short list elements can be removed, injuring source documents.
  • Max n-gram size: Increasing max_ngram_size from 13 to 50 does not dramatically alter language statistics, so 13 is used at scale.The selected min_ngram_size is 13.
  • Threshold: Higher deduplication thresholds shorten documents while producing lower removal rates.The threshold controls how similar a document must be to previously seen n-grams before removal.
  • Shards: Larger pools increase deduplication removal rates, while token and document removal rates diverge at larger scales.The increase in removal is concave with pool size, and document lengths decrease monotonically as pools grow.
  • Global duplicate analysis: Sharded Bloom filter deduplication leaves many fuzzy duplicates, yet DCLM-BASELINE shows no apparent downstream-performance penalty.The authors leave open whether only large duplicate quantities are harmful or aggressive single-shard deduplication removes high-quality documents.

N Human judgment

The human-judgment study tests whether agreement with expert annotations predicts useful language-model training data. Agreement with human labels does not reliably track downstream performance, suggesting human intuition may be an imperfect quality signal.

  • Annotation setup: Sixteen English-speaking AI graduate students and professors annotated about 500 documents, with three annotations per document and 71% average inter-annotator agreement.The 281 documents with unanimous agreement form the AGREEMENT subset; the full set is MAJORITY.
  • Filter comparison: Filters aligned most closely with human labels do not necessarily produce the best language-model training data.AskLLM reaches ∼82% ROC-AUC on human annotations but yields ∼28.5% CORE accuracy, below several fastText classifiers exceeding 31%.
  • Filter comparison: The best and worst fastText filters have similar MAJORITY ROC-AUC values of ∼73%, despite differing downstream performance.The comparison uses models trained on similarly sized filtered pools and evaluates CORE, StrategyQA, and SQuAD.
  • Downstream correlation: Correlation between human-label ROC-AUC and downstream task performance is weak or inconsistent.StrategyQA shows some positive correlation, SQuAD shows the opposite, and both have R2 < 0.3; most other tasks show no correlation.
  • Annotation criteria: The annotation task asks whether documents are useful for language-model training while considering target capabilities and harmful content.Examples include reading comprehension, language understanding, world knowledge, commonsense reasoning, open-ended tasks, and safety concerns.

O Decontamination

The decontamination analysis finds only small performance differences between full evaluation sets and samples with limited token overlap for most tasks. The section also reports instruction-tuning and long-context results for DCLM-BASELINE.

  • Decontamination: Performance differences between full and “not dirty” evaluation subsets are minimal, and clean-subset differences are similarly small for most datasets.BoolQ and SQuAD slightly improve on the “not dirty” subset; Hellaswag is a notable exception, but removing contaminated examples did not decrease its performance.
  • Instruction tuning: DCLM-BASELINE fine-tuned on OpenHermes-2.5 outperforms Zephyr-Beta-7B and Gemma-Instruct-7B while slightly trailing Mistral-7B-OH-2.5.The authors attribute the small difference partly to longer generations or fewer pretraining tokens.
  • Instruction tuning: DCLM-IT combines roughly 4 million instances and 8 billion tokens from multiple instruction-tuning datasets.The aligned-version development using procedures such as PPO or DPO is left for future research.
  • Continual learning to extend context length: Long-context adaptation produces DCLM-8k, which matches the original model on regular evaluations and significantly improves multi-document question answering.The evaluation varies the golden-document position and uses k values from 1 to 30.

S Bias and Toxicity Evaluations

The evaluation compares DCLM-BASELINE with existing base models on toxicity and bias benchmarks. Its scores are broadly comparable, including similar toxic-content generation and bias patterns.

  • Safety evaluation: DCLM-BASELINE scores comparably to existing base models across the reported toxicity and bias evaluations.The benchmarks include CivilComments, Copyright, BBQ, WinoGender, and RealToxicityPrompts.
  • Toxicity: DCLM-BASELINE has RealToxicityPrompts scores similar to popular base models such as Llama and Mistral.RealToxicityPrompts measures how easily a user can prompt a model to generate toxic content.
  • Bias: BBQ and WinoGender indicate generation biases similar to those of other base models.BBQ is evaluated with higher-is-better accuracy, while WinoGender assesses gender bias-related behavior.

T.1 Evaluation data

The evaluation suite spans diverse downstream tasks and includes comparisons of model quality, bias, and toxicity. The dataset documentation also notes that DCLM-BASELINE is intentionally filtered and therefore not representative of the full Common Crawl instance set.

  • Evaluation suite: The benchmark includes diverse downstream evaluations, with additional tasks and dataset licenses documented in Appendix G.The listed suite includes reasoning, language understanding, commonsense, math, coding-related, and bias-oriented evaluations.
  • Bias and toxicity: Table 34 compares DCLM-BASELINE with similarly sized base models on bias and toxicity metrics.The comparison covers generation toxicity, bias, and toxic-comment identification.
  • Research use: The benchmark, tooling, and data pools are intended to support data-centric research, safer web-scale datasets, lower training costs, and knowledge sharing.The authors describe DCLM-POOL as a research artifact for studying safer internet-scale datasets.
  • Dataset scope: DCLM-BASELINE is non-representative of the full Common Crawl instance set because filtering and deduplication remove low-quality instances.DCLM-POOL itself is also limited because Common Crawl does not cover the entire web.
  • Dataset safety: DCLM-POOL may contain hateful, racist, sexist, offensive, or toxic material because it is scraped from the internet.DCLM-BASELINE applies a reproduction of RefinedWeb content filtering, including URL-based filtering.
Loading 2406.11794v4…