Source-linked AI summary
D4: Improving LLM Pretraining via Document De-Duplication and Diversification
Kushal Tirumala, Daniel Simig, Armen Aghajanyan, Ari S. Morcos
TL;DR
Large-scale LLM pretraining commonly relies on randomly sampled, heuristically filtered and near-duplicate-removed web data, while data selection remains underexplored. The paper introduces D4, which selects data using pre-trained model embeddings after de-duplication, and reports 20% training-efficiency gains across model scales plus improved downstream accuracy at 6.7B parameters.
Problem
Data curation and selection for large-scale web data remain underexplored beyond random sampling, heuristic filtering, and near-match de-duplication.
Method
D4 selects document subsets using pre-trained model embeddings after MinHash-based de-duplication, addressing duplicate-driven clusters in embedding space.
Results
D4 improves training efficiency by 20% across multiple model scales; at 6.7B parameters, it reaches baseline perplexity with 20% fewer update steps and improves downstream accuracy by 2%.
Takeaways & Limitations
Intelligently selected data can outperform random selection, and repeating a selected subset can improve perplexity and downstream accuracy over one-pass training on available data.
Takeaways & Limitations
The experiments use one data distribution, leaving D4's efficacy on mixtures of training distributions for future work.
Abstract
from arXiv · showhide
Over recent years, an increasing amount of compute and data has been poured into training large language models (LLMs), usually by doing one-pass learning on as many tokens as possible randomly selected from large-scale web corpora. While training on ever-larger portions of the internet leads to consistent performance improvements, the size of these improvements diminishes with scale, and there has been little work exploring the effect of data selection on pre-training and downstream performance beyond simple de-duplication methods such as MinHash. Here, we show that careful data selection (on top of de-duplicated data) via pre-trained model embeddings can speed up training (20% efficiency gains) and improves average downstream accuracy on 16 NLP tasks (up to 2%) at the 6.7B model scale. Furthermore, we show that repeating data intelligently consistently outperforms baseline training (while repeating random data performs worse than baseline training). Our results indicate that clever data selection can significantly improve LLM pre-training, calls into question the common practice of training for a single epoch on as much data as possible, and demonstrates a path to keep improving our models past the limits of randomly sampling web data.
1 Introduction
LLM pretraining commonly relies on randomly sampled, lightly filtered web data, but diminishing scaling returns motivate more deliberate selection. D4 addresses this gap by combining embedding-based de-duplication and diversification, improving efficiency and downstream performance.
- Random sampling of heuristically filtered and near-deduplicated web data remains standard in LLM pretraining despite limited exploration of data selection.
- Diminishing returns from power-law scaling may require exponentially more data for consistent marginal gains as available human-generated text approaches its limits.
- Embedding-based selection has improved data efficiency in vision and vision-language models, but comparable approaches have received little large-scale LLM study.
- D4 combines SemDeDup with SSL Prototypes to improve pretraining perplexity and downstream accuracy over random and established selection methods.
2 Related Work
Prior work established data selection and de-duplication techniques across vision, vision-language, audio, and language modeling. This paper combines related embedding-based ideas and applies them to large-scale LLM pretraining.
- Vision studies evaluated pruning, coresets, and prototype-based selection, including methods that improve data efficiency at ImageNet scale.
- SemDeDup uses embedding-based semantic de-duplication and has shown encouraging results for vision-language models such as CLIP.
- D4 combines related selection approaches and applies them to large-scale LLMs.
- Language-model data studies examined corpus quality, de-duplication, and artificial duplication, but did not establish broad embedding-based selection for LLM pretraining.
3 Experimental Setup
The experiments define document-level selection over a MinHash-deduplicated CommonCrawl dataset, evaluate OPT models with perplexity and downstream accuracy, and compare embedding-based selection strategies. D4 first removes semantic duplicates, then diversifies the remaining data through clustering and prototype selection.
- Experimental Setup: The selection problem chooses a document subset from Dsource to maximize an evaluation metric for a model, using random selection as the baseline.
- Training Dataset: Training uses CommonCrawl processed with CCNet and an additional MinHash de-duplication step, producing the CC-dedup dataset.
- Evaluation Metrics: Evaluation includes average perplexity on web and non-web validation sets plus average 0-shot accuracy across 16 NLP tasks.
- Selection Strategies: Document embeddings are produced with a 125M OPT model, enabling selection based on positions in embedding space.
- Selection Strategies: SSL Prototypes clusters embeddings with k-means and prioritizes points by distance to cluster centroids, while SemDeDup removes nearby points within embedding clusters.
- D4: D4 applies SemDeDup, k-means clustering, and SSL Prototypes sequentially, with overall selection ratio R = Rdedup * Rproto.
4 Results
D4 improves language-model training when source datasets are expanded and selected data is diversified, including settings where selected data is repeated. Benefits extend from perplexity to downstream accuracy and training efficiency, though effects vary by validation set.
- Fixed compute regime: D4 consistently improves Instruct OPT perplexity, outperforming SemDeDup and SSL Prototypes, with strongest gains when the source dataset is about 4x larger.All methods improve with smaller selection ratios, while D4’s advantage is most notable at larger source-dataset sizes.
- Fixed compute regime: 0-shot downstream accuracy generally increases as the source dataset grows, although high variance makes trends difficult to identify clearly.
- Fixed compute regime: 20% fewer update steps let a 6.7B D4 model reach baseline perplexity, while final downstream accuracy improved by 2%.The comparison used 100B tokens and selection ratio R = 0.25.
- Fixed data regime: Repeating randomly selected data slightly degrades perplexity, whereas repeating D4-selected data improves perplexity and downstream accuracy over randomly sampling new tokens.This fixed-data experiment used 1.3B OPT models trained on 40B tokens.
- Cost of data selection: D4 reaches baseline perplexity with 20% fewer model updates at 6.7B scale, corresponding to approximately 4300 GPU hours of naive efficiency savings.The naive estimate excludes the cost of computing the selection metric.
- Cost of data selection: Overall efficiency gains increase with model size, with conservative estimates of 20% for Llama-65B and 22% for OPT-175B.
- Validation-set effects: Data selection always impairs performance on web-snapshot validation sets, while perplexity improvements vary substantially across validation sets.
- Validation-set effects: Web-snapshot perplexity is slightly inversely related to instruction-tuned perplexity and downstream accuracy, whereas Instruct+Answers perplexity is positively correlated with downstream accuracy.The paper therefore treats instruction-tuned validation perplexity as a better model-quality measure.
5 Summary and Limitations
D4 improves training efficiency across model scales, while intelligent repetition can benefit LLM training. The study’s scope is limited by its single-distribution setup and maximum evaluated scale.
- Intelligently selected data can make epoching beneficial for LLM training, unlike indiscriminate repetition.
- Limitations: The experiments use one data distribution for both data selection and training, whereas modern LLM setups usually mix sources.
- Limitations: The largest evaluated models were 6.7B parameters trained on 100B tokens, leaving larger-scale investigation for future work.
A.1.1 Hyperparameters for model training
The training setup follows the OPT configuration, using Adam with specified regularization and optimization settings, Megatron-LM tensor parallelism, and fp16 precision.
- The models use GELU activations, Adam optimization, β1 = 0.9, β2 = 0.95, ϵ = 10^-8, weight decay 0.1, and gradient-norm clipping at 1.0.
- Training uses Megatron-LM tensor parallelism with fp16 precision and omits dropout for reproducibility.
- Batch size denotes the total tokens processed during one gradient-descent update.
A.1.2 Dataset Curation Details
The CC-dedup source dataset is built from CommonCrawl through paragraph-level processing, language and quality filtering, and additional document-level MinHash de-duplication.
- CC-dedup starts from five CommonCrawl dumps spanning 2017 to 2020.
- CCNet processing de-duplicates paragraphs, removes non-English pages, and filters low-quality pages using the Touvron et al. pipeline.
- Document-level MinHash de-duplication is added with 20 hashes per signature, 20 buckets, and 1 row per bucket.
- Curation risk: Because the dataset comes from CommonCrawl, the training set may contain offensive or personally identifiable information.
A.1.3 Parameters for Data Selection
The data-selection pipeline clusters embeddings using K-Means, with centroids estimated from a sampled subset because clustering all starting documents is computationally expensive. The authors use 11,000 clusters and do not explicitly balance clusters, while D4 parameters are chosen strategically rather than swept.
- 600 million starting documents make full 768-dimensional K-Means clustering computationally expensive, so centroids are calculated from roughly 100 million sampled documents.The sampled embeddings are normalized before clustering with faiss.
- 11,000 clusters are used for K-Means, following prior work and a square-root heuristic for the number of clustered points.Initial 125M OPT experiments found no significant performance effect from the number of clusters.
- The clustering procedure avoids manual cluster balancing by setting low minimum and high maximum points per centroid.Across methods and R choices, the observed cluster-balance score is generally at least 0.5 without intervention.
- The authors leave cluster balancing for future work because cluster-balance scores are already generally equal to or above 0.5.They suggest varying the number of sampled points per cluster based on cluster properties as a future direction.
- D4 parameters Rproto and Rdedup are selected strategically because limited compute prevents a hyperparameter sweep.The authors choose a high SemDeDup R that improves perplexity across validation sets, avoiding broader removal of templates or semantic duplicates.
A.1.4 Which validation sets go into the averages?
The paper groups validation sets into Web Snapshots, Non-web Snapshots, and Instruct + Answers when reporting averages. The groups combine standard datasets, web-derived corpora, and instruction-tuning examples with prompts and answers.
- Web Snapshots average perplexity on C4, CC-dedup, and CommonCrawl from the Pile.
- Non-web Snapshots include Pile validation sets such as OpenWebText2, HackerNews, Wikipedia, BookCorpusFair, and DM Mathematics.The category also includes Gutenberg PG-19, OpenSubtitles, USPTO, Reddit, stories, prompts_with_answers, and prompts.
- Instruct + Answers measures perplexity on OPT-IML instruction-tuning samples containing both prompts and answers.Figure A4 refers to this set as prompts_with_answers.
- The authors provide examples of the Instruct + Answers validation format in Table A2 for readers unfamiliar with the data.
A.2 Efficiency gains across model scales and training
Across 125M, 1.3B, and 6.7B OPT models, D4 produces efficiency gains in perplexity and downstream accuracy, with gains appearing to increase at larger scales.
- D4 produces efficiency gains in perplexity across 125M, 1.3B, and 6.7B OPT models.The corresponding training budgets are 3B, 40B, and 100B tokens, respectively.
- The efficiency gains from D4 appear to increase with model scale.This pattern is reported across the model groups studied in the scale experiment.
- D4 training trajectories show gains in both perplexity and 0-shot accuracy averaged over 16 NLP tasks across model scales.
A.3 Individual Breakdowns of Downstream Accuracy and PPL
Fine-grained analyses show that data-selection effects vary across validation sets and tasks, while D4 generally remains competitive or superior. Repeating selected data can outperform random new-token selection, but embedding-space quality and validation overlap constrain the results.
- Downstream accuracy: Data-selection effects vary considerably across the 16 downstream NLP tasks, motivating mean accuracy comparisons across tasks.
- Perplexity breakdowns: Data selection worsens perplexity on web-snapshot validation sets, while D4 generally has the slowest degradation rate.Across non-web snapshots, no selection method is a clear winner, though most validation sets improve over baseline.
- Method overlap: SSL Prototypes and SemDeDup select substantially overlapping datasets because both sparsify dense embedding-space regions around cluster centroids.K-Means centroids lie in dense regions, where nearby documents can be semantically redundant.
- Method overlap: Removing the re-clustering step significantly worsens performance, while SemDeDup removes duplicate-driven clusters.
- Train-validation overlap: For C4, validation points nearest the training set are easiest before selection and most affected after selection.This suggests their apparent ease is related to proximity to training points that data selection removes.
- Repeating tokens: Repeating data selected by SemDeDup, SSL Prototypes, or D4 can outperform randomly selecting new tokens, with D4 retaining the strongest overall results.For a fixed selection strategy, however, selecting new tokens usually matches or beats repeating tokens.
- Embedding-space choice: Embedding-space quality is a key limitation because OPT embeddings can cluster documents by end-of-document formatting rather than broader semantics.This may help detect templates but can incorrectly group unrelated documents sharing the same ending.
- Embedding-space choice: SentenceTransformer embeddings outperform OPT embeddings at small scales but can disproportionately prune long documents and underperform at larger scales.The authors hypothesize that long documents appear out of distribution because SentenceTransformer training rarely uses maximum-length sentence pairs.