Deduplicating Training Data Makes Language Models Better

1 Introduction

• All four studied NLP datasets contain training duplicates and train-validation overlap despite existing naive deduplication, motivating more thorough techniques that work at scale.Manual review and curation are expensive for massive corpora, whose quality tends to suffer compared with smaller predecessors.Source: §1 · p. 1
• Two scalable techniques detect duplicates: exact substring matching finds repeated passages, while approximate full-document matching uses hashing to identify high n-gram overlap.Exact substring matching can identify duplication affecting only part of a training example.Source: §1 · p. 1
• 10× lower rates of emitting memorized training data follow deduplication; with standard datasets, over 1% of unprompted tokens belong to memorized sequences.This memorization occurs even with a 1.5-billion-parameter model trained on a 350GB dataset.Source: §1 · p. 1
• 61,036 training repetitions and 61 validation repetitions of one 61-word sequence illustrate train-test overlap that inflates measured accuracy and favors overfitting during model selection.The sequence appears in C4 (Raffel et al., 2020), accounting for 0.02% of samples in each split.Source: §1 · pp. 1–2
• Up to 19% smaller datasets reduce training costs even after accounting for deduplication, which uses CPU-only, linear-time processing.The reported savings encompass training time, monetary cost, and environmental impact.Source: §1 · p. 2
• Up to 10% lower perplexity is achieved in some cases; models trained on deduplicated data have no worse perplexity than original-data baselines.Training on higher-quality data can also help models reach higher accuracy faster.Source: §1 · p. 2

Large language model datasets

• The study deduplicates publicly available Wiki-40B, C4, RealNews, and the One Billion Word Language Model Benchmark (Chelba et al., 2013).Many other training corpora are not public; CommonCrawl underlies C4 and RealNews. The analysis uses Transformer-based decoder-only models (Vaswani et al., 2017) for open-ended generation, although the authors believe their findings are architecture-independent.Source: §2 · p. 2

Contamination of downstream tasks

• Up to 14.4% of test examples from standard tasks appeared verbatim in C4 after capitalization and punctuation normalization, according to Dodge et al. (2021b).Radford et al. (2019, §4) checked GPT-2 contamination through 8-gram overlap, while Trinh and Le (2018, Appendix B) proactively removed overlapping training documents. GPT-3 (Brown et al., 2020, §5) filtered training examples sharing 13-grams with evaluation examples; up to 90% of tasks were flagged as potentially contaminated.Source: §2 · p. 2
• The study targets duplicate text’s effects on language-model perplexity and memorized generation, not its effects on downstream benchmark tasks.Its scope includes duplicates in both training and validation sets.Source: §2 · p. 2

Memorizing training data

• Over 1% of the text most models emit is memorized training data, extending prior evidence of extractable training examples.Carlini et al. (2020) identified 604 training samples emitted by GPT-2 and highlighted privacy risks, including extraction of valid phone numbers and IRC usernames.Source: §2 · p. 3

Duplicate text in training data

• Prior work documents exact-duplicate documents in Book Corpus and shows that duplicate code examples worsen performance on code-understanding tasks.Bandy and Vincent (2021) identified substantial duplication in Book Corpus (Zhu et al., 2015), used to train BERT; Allamanis (2019) demonstrated the code-task performance effect.Source: §2 · p. 3

Scope

• Four English datasets of varying sizes are analyzed for duplicate text, covering language generation, general-purpose pretraining, and language-model benchmarking.The authors expect similar duplication issues and potential deduplication benefits in non-English datasets, but restrict their analysis to English.Source: §3 · p. 3

Wiki-40B

• The analyzed English portion of Wiki-40B contains 2.9M Wikipedia pages, with an average page length of 768 BPE tokens.The dataset creators do not indicate that any deduplication was performed aside from removing redirect pages.Source: §3 · p. 3

LM1B

• 13.2% of LM1B’s test set overlaps its training set, as noted by Radford et al. (2019, Sec 4).LM1B contains 30M news-commentary sentences averaging 32 BPE tokens (Chelba et al., 2013); unlike the other datasets, its examples are individual sentences.Source: §3 · p. 3

C4

• 360M web documents averaging 486 BPE tokens make up C4 (Raffel et al., 2020), which underwent paragraph-level deduplication.Introduced for T5 pretraining, C4 removed paragraphs producing hash collisions, then filtered placeholder text, code, and prohibited words. Documents with duplicate URLs were also excluded.Source: §3 · p. 3

RealNews

• 31M documents averaging 793 BPE tokens constitute RealNews, a Common Crawl subset drawn from news domains (Zellers et al., 2019).Deduplication inserted hashes of documents’ first 100 characters into a bloom filter, excluding documents producing hash collisions. Documents with duplicate URLs were also excluded.Source: §3 · p. 3

4.1 Exact Substring Duplication

• 50 tokens is EXACTSUBSTR’s minimum matching length for removing a shared verbatim substring from one of two examples.The threshold was selected through statistical analyses.Source: §4.1 · p. 3

4.1.1 Suffix Arrays

• 8 bytes per input token is the suffix-array storage requirement; construction and duplicate identification run in linear time over the concatenated dataset.Suffix arrays are 10–100× more memory efficient than suffix trees, avoiding naive quadratic all-pairs matching.Source: §4.1.1 · p. 4

4.1.2 Substring matching

• 50 BPE tokens is the common-prefix threshold used when linearly scanning adjacent suffix-array entries to identify repeated sequences.The scan is embarrassingly parallel, allowing efficient dataset processing.Source: §4.1.2 · p. 4

4.2 Approximate Matching with MinHash

• NEARDUP removes near-duplicate whole examples, including web documents differing only in interspersed templated fields.MinHash approximates Jaccard similarity using signatures retaining the smallest hashed n-grams. Candidate pairs qualify as duplicates only when their edit similarity exceeds 0.8.Source: §4 · p. 3, §4.2 · p. 4
• NEARDUP builds similar-document clusters as connected components of a graph whose edges link matching documents.Connected components are identified using the method introduced in Ł ˛acki et al. (2018).Source: §4.2 · pp. 4–5

5.1 Amount of Text Removed

• 3.04% of C4 and 13.63% of RealNews examples are near-duplicates, compared with 0.39% of Wiki-40B training examples.C4 contains 1.8 million two-example near-duplicate clusters, but also 280 clusters exceeding 5,000 examples, including one containing 250,933.Source: Table 2 · p. 5, §5.1 · p. 5
• 7.18% of C4 tokens are removed by EXACTSUBSTR, which generally removes more content than NEARDUP without discarding entire examples.On LM1B, EXACTSUBSTR removes 8× less data than NEARDUP because 90% of documents are shorter than its 50-token matching threshold. In C4, 77% of training examples removed by NEARDUP contain a verbatim length-50 match found by EXACTSUBSTR.Source: Table 3 · p. 5, §5.1 · p. 5

5.2 Properties of Duplicated Text

• Near-duplicates often comprise computer-generated templates in C4 and Wiki-40B, or news articles republished with formatting changes in RealNews and LM1B.RealNews and C4 retain subtle duplication despite deduplication during construction. Templates differing only in names, dates, or similar fields evade exact string matching.Source: §5.2 · p. 6

5.3 Train / Test Set Leakage

• 4.6% of C4 validation examples and 14.4% of RealNews validation examples have approximate duplicates in their respective training sets.Both methods identify train–validation overlap, which could inflate evaluation metrics for models better at memorizing training data. Cross-split deduplication prioritizes retaining test or validation copies and removing training copies.Source: §5 · p. 5, §5.3 · p. 6

6 Impact on Trained Models

• 1.5B-parameter decoder-only Transformer models were trained for about two epochs each on C4-ORIGINAL, C4-NEARDUP, and C4-EXACTSUBSTR.The models used the T5 codebase and architecture from Raffel et al. (2020). Three random seeds of a 110M-parameter base model per dataset—nine models total—provided a check on perplexity variation.Source: §6 · p. 6

6.1 Model Perplexity

• Almost 3 points of XL Wiki-40B perplexity improvement follow EXACTSUBSTR deduplication; NEARDUP-trained models consistently achieve the lowest perplexity on LM1B and Wiki-40B.The improvement exceeds the roughly 1-point variation observed in base models, despite fewer training tokens. All XL models achieve <35 perplexity on LM1B, versus 42.16 reported for 1.5B GPT-2 with the same vocabulary size.Source: Figure 2 · p. 6, §6.1 · pp. 6–7
• Deduplication maintains similar perplexity on original and unique C4 validation examples but raises perplexity on training-duplicated examples, for both base and XL models.EXACTSUBSTR produces higher perplexity on duplicated examples than NEARDUP. However, the methods use different duplication thresholds, and changing those thresholds could change the comparison.Source: Figure 2 · p. 6, §6.1 · p. 6

6.2 Generated Text

• Approximately 10× less unprompted memorization occurs with XL-EXACTSUBSTR and XL-NEARDUP: copied tokens drop from over 1% with XL-ORIGINAL to 0.1%.Using top-k sampling with k = 50, the authors generated 100,000 samples of up to 512 tokens. A token counts as memorized when it belongs to a 50-token substring appearing exactly in training data.Source: Table 4 · p. 7, §6.2 · p. 7
• Over 40% of prompts from duplicated training examples cause XL-ORIGINAL to reproduce ground-truth continuations; deduplicated models also copy more with duplicated than unique prompts.Evaluation used 32-token prompts from duplicated or unique training and validation examples, measuring continuation edit similarity above 0.8. Residual copying suggests more stringent deduplication may be necessary to eliminate memorization.Source: Figure 3 · p. 7, §6.2 · p. 7

6.3 Impact on Existing Models

• Halved perplexity on near-duplicate evaluation examples reveals train-test leakage in Transformer-XL; GROVER shows a smaller difference in both examined model sizes.Transformer-XL (Dai et al., 2019) was trained on LM1B, while GROVER (Zellers et al., 2019) was trained on RealNews.Source: Table 5 · p. 7, §6.3 · pp. 7–8
• 1.38% of GROVER-Mega output tokens and more than 5% of GPT-Neo 1.3B output tokens belong to training-data matches at least 50 tokens long.The analysis matched 25k released GROVER-Mega outputs against RealNews and ~200k sequences from GPT-Neo 1.3B against the Pile (Gao et al., 2020).Source: §6.3 · p. 8

7 Discussion

• The study quantitatively examines dataset duplication, deduplication’s effects on model perplexity, and memorization reduction, rather than the nature of removed or memorized content.Its contribution to quantitative analysis of language-modeling datasets focuses specifically on duplication.Source: §7 · p. 8
• Repeated training examples make models particularly sensitive to those examples, creating privacy concerns even when the memorized information is publicly available.Privacy here means not revealing information particular to the specific training dataset rather than another dataset from a similar distribution (Shokri et al., 2017). This definition extends beyond reproducing sensitive identifiers.Source: §7 · p. 8
• The experiments treat all generated text closely matching training data as problematic, without distinguishing undesirable, innocuous, or desirable memorization.Much identified memorized content appeared relatively innocuous in qualitative observations, but systematic assessment of its risks was beyond the study’s scope.Source: §7 · p. 8
• Deduplication’s negative consequences remain uninvestigated: some tasks require memorization, and removing duplicated attribution could leave models learning content without its attribution.Document retrieval and closed-book question answering explicitly require memorization. Deduplication also cannot adequately remove sensitive information such as bank passwords and medical records, which should never enter training data.Source: §7 · p. 8
• Whether memorization is desirable depends on the text and downstream application, so intended uses should constrain training-data choices.Application-specific techniques for memorizing or forgetting particular sequences are a promising research direction, especially as datasets and models become increasingly application-agnostic.Source: §7 · p. 8

• Future language-model research should prioritize stringent dataset deduplication over the exact technique used.Researchers can train on the released deduplicated datasets, use the released deduplication tools, or follow the authors’ approach with new tools.Source: §8 · p. 8
• Deduplication does not harm, and sometimes improves, model perplexity despite yielding smaller datasets that are faster to train on.Source: §8 · p. 9
• Eliminating duplicates between training and testing sets is especially important because overlap explicitly encourages selecting models that memorize training data.Source: §8 · p. 9
• Deduplication helps reduce some privacy concerns associated with language models memorizing their training data.Source: §8 · p. 9

Ethics

• Fallible dataset collection and curation can over-represent text repeated for structural reasons, including bot replies, auto-generated templates, and licenses, as also observed by Dodge et al. (2021a).Such repetition can arise for legal or economic reasons, despite developers’ attempts to reflect natural human communication.Source: Ethics · p. 9
• The study targets over-represented structural duplicates but does not address under-representation of non-English speakers or groups communicating outside the public web.Data acquisition and cleaning can favor English-speaking users publishing in established forums.Source: Ethics · p. 9
• The analysis does not distinguish helpful from harmful memorization, a complex task requiring different research methodologies.The paper discusses when memorized content might or might not be desirable without separating these cases in its analysis.Source: Ethics · p. 9

Contributions

• Katherine trained models, Daphne ran approximate deduplication, Andrew implemented approximate matching, and Nicholas implemented suffix arrays and ran all EXACTSUBSTR experiments.Katherine also built evaluation/text-generation pipelines and contributed writing, analysis, and project management; Daphne extracted prompts/evaluation datasets, ran evaluations, and contributed planning, writing, and analysis. Andrew helped evaluate energy expenditure and analyze results, while Nicholas contributed planning, writing, analysis, and project scoping.Source: Contributions · p. 9
• Chiyuan helped generate plots and contributed project scoping, writing, and data analysis; Chris and Doug provided mentorship, guidance, and writing contributions.Source: Contributions · p. 9

A Further Details on NEARDUP

• NEARDUP builds document signatures from space-tokenized consecutive 5-grams, applies MinHash, and flags pairs sharing an aggregated hash value in any bucket.The 5-grams use tabulation hashing; minima from k additional hash functions are partitioned into r buckets containing b hashes each.Source: A Further Details on NEARDUP · p. 13
• 0.8 must be exceeded by both Jaccard and edit similarity for NEARDUP duplicates; chosen parameters are b = 20, r = 450, and k = 9,000.Actual Jaccard similarity is checked first; only pairs exceeding its threshold proceed to edit-similarity computation. The parameters were chosen to give collisions high probability at the desired Jaccard threshold.Source: A Further Details on NEARDUP · p. 13
• 0.9 thresholds for both similarities provided an alternative check whose distribution tails supported the authors’ conclusion that most actual near duplicates were captured.The alternative required Jaccard and edit similarity of at least 0.9, using b = 20, r = 40, and k = 800. The chosen and alternative configurations’ distributions began vanishing at the same point despite targeting different thresholds.Source: A Further Details on NEARDUP · p. 13
• T^2 is edit similarity’s worst-case complexity; the analysis also considers signature grouping and the pathological case of all documents sharing buckets.N denotes document count and T maximum document tokens, with b, k, and T much smaller than N. The distributed Google production implementation consumed approximately 41.5 kWh on English C4; the authors suggest different parameters could improve energy efficiency while producing similar results.Source: A Further Details on NEARDUP · p. 13

B Further Details on EXACTSUBSTR

• The Rust implementation supports datasets larger than 4GB, accepts raw bytes instead of requiring UTF-8, and streams suffix arrays from disk.The dataset itself must still fit in memory for arbitrary-position indexing, but the entire suffix array need not. The authors report an 8× suffix-array space overhead.Source: B Further Details on EXACTSUBSTR · pp. 13–14
• EXACTSUBSTR constructs suffix arrays independently across parallel dataset splits, then combines the partial arrays through min-heap merging.Partial construction requires O(N) work and O(N/K) wall-clock time for K splits. General merging costs O(Nm log(K)), where m is average matching-prefix length; its efficiency depends on sequence lengths remaining independent of dataset size.Source: B Further Details on EXACTSUBSTR · pp. 13–14
• Under 12 hours builds the 350GB C4 suffix array, followed by under one hour for deduplication—work described as negligible relative to model training.On a 96-core VM with 768GB memory, C4 construction required approximately 1,000 CPU-hours because of memory constraints. Wiki-40B’s 4GB training set took 2.3 minutes wall-clock and 2.1 CPU-hours.Source: B Further Details on EXACTSUBSTR · p. 14
• 50 tokens is the conservative duplicate-substring threshold, twice the length at which manual inspection found no false positives.The threshold analysis measures whether each length-k substring has another identical occurrence; matches shorter than 10 subword tokens are common and account for 90% of tokens. The distribution’s knee occurs at 10 tokens, while inspected length-25 matches had no false positives.Source: B Further Details on EXACTSUBSTR · pp. 14–15, Figure 5 · p. 15

C Further Details on Model Training

• 152K steps trained C4-ORIGINAL and C4-EXACTSUBSTR models for approximately two epochs, versus 146K for C4-NEARDUP, with batch size 4800.Training used Adafactor (Shazeer and Stern, 2018), with constant learning rates of 0.01 for base models and 0.001 for XL models. On a 128-core TPU v3 slice, XL training took approximately 131 hours for C4-ORIGINAL and C4-EXACTSUBSTR, versus 126 hours for C4-NEARDUP.Source: C Further Details on Model Training · p. 15
• 1.5B-parameter XL models used 24 layers with 32 attention heads each; 110M-parameter base models used 12 layers with 12 heads each.XL embedding and feed-forward hidden sizes were 2,048 and 5,120, respectively; base sizes were 768 and 2,048. Both configurations used attention key/value dimensions of 64.Source: C Further Details on Model Training · p. 15

D Energy Consumption

• 5.63MWh is the estimated two-epoch training energy for XL-NEARDUP, compared with 5.86MWh for XL-ORIGINAL and XL-EXACTSUBSTR.The approximately deduplicated dataset is 3.9% smaller, trains in 63 hours per epoch, and saves around five compute hours across two epochs. Using Taiwanese datacenter estimates derived from Patterson et al. (2021), each base model’s approximately three-day training run was estimated at 1.61MWh.Source: D Energy Consumption · p. 15
• 0.43MWh is the estimated energy required to generate 3M sequences, using 64-core TPU v3 slices for evaluation and inference.Generating 100,000 XL-model sequences took approximately 0.64 hours. Five prompt types and two checkpoints yielded 1M sequences per model, with the reported generation workload taking approximately 19.2 hours.Source: D Energy Consumption · p. 15

Qualitative Examples

• Qualitative examples illustrate C4 document pairs near the 0.8 edit-similarity threshold, repeated substrings, and unprompted memorized generations.Pairs below 0.8 were not identified as duplicates. Most displayed memorized XL-ORIGINAL generations were advertisements, generated using top-k sampling with k=50.Source: E More Results · p. 15, Table 8 · p. 17, Table 10 · p. 20

Distribution of memorization

• Memorization distributions are examined under four prompting conditions, crossing training versus validation examples with the presence versus absence of duplicates in training.Figure 6 reports the distribution of memorization amounts across generated sequences.Source: E More Results · pp. 15–16, Figure 6 · p. 16

Energy usage

• Energy estimates use Patterson et al. (2021)’s T5 11B encoder-decoder estimate as their basis, with inference accounting for all XL models.The generation workload comprised 100,000 sequences from three models, with five prompts and two checkpoints.Source: Table 6 · p. 16

URLs with many duplicates

• Duplicate-heavy C4 sources tend to sell similar products using templated text, while content aggregators are especially common among duplicate-heavy RealNews sources.Table 11 restricts its near-duplicate proportion ranking to URLs occurring at least 10 times and separately lists 20 frequent C4 URLs whose examples were all near-duplicates.Source: E More Results · p. 16, Table 11 · p. 21

NEARDUP cluster sizes / Perplexity on LM1B

• Supplementary plots report NEARDUP cluster-size distributions for RealNews, LM1B, and Wiki-40B, alongside validation-perplexity results that include LM1B.The perplexity plots cover T5 base (110M parameters) across three random initializations and T5 XL (1.5B parameters); LM1B was omitted from the main-paper figure for readability.Source: E More Results · pp. 16, 18, Figure 7 · p. 19, Figure 8 · p. 22

Dataset Sizes

• EXACTSUBSTR deduplication removes no examples because most datasets already underwent exact-match deduplication during creation.Table 13 compares training-set sizes in examples and tokens under a 50k BPE vocabulary for original, NEARDUP-deduplicated, and EXACTSUBSTR-deduplicated datasets.Source: E More Results · pp. 16, 18, Table 13 · p. 21

Memorization

• 10× less memorization occurs with approximate or exact deduplication versus original-data training, measured as the percentage of tokens in 100k generations within EXACTSUBSTR-detected memorized substrings.Table 12 compares models trained on deduplicated datasets with the model trained on the original, non-deduplicated dataset.Source: Table 12 · p. 21