PaperlayerView exact revision on arXiv

Source-linked AI summary

Deduplicating Training Data Makes Language Models Better

Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, Nicholas Carlini

arXiv:2107.06499v2cs.CLcs.LG

TL;DR

Large language-model datasets contain pervasive duplicate and near-duplicate text, complicating dataset quality and evaluation. The paper develops exact-substring and near-duplicate deduplication tools, finding that deduplication reduces memorized output while preserving or improving perplexity and reducing train-test overlap.

  • Problem

    Duplicates occur across common NLP training datasets and their validation sets, while thorough deduplication at scale is computationally challenging.

  • Method

    The paper combines EXACTSUBSTR for long repeated substrings with NEARDUP, which uses approximate matching for near-identical documents with templated differences.

  • Results

    Deduplication reduces memorized training-data emissions by 10× and does not worsen perplexity, sometimes reducing it by up to 10%.

  • Takeaways & Limitations

    Stringent deduplication can produce smaller, faster-to-train datasets while reducing train-test overlap and some privacy concerns around memorization.

  • Takeaways & Limitations

    Deduplication may remove text needed for memorization-based tasks and does not by itself eliminate privacy-sensitive data that should never enter training datasets.

Abstract

from arXiv · show

We find that existing language modeling datasets contain many near-duplicate examples and long repetitive substrings. As a result, over 1% of the unprompted output of language models trained on these datasets is copied verbatim from the training data. We develop two tools that allow us to deduplicate training datasets -- for example removing from C4 a single 61 word English sentence that is repeated over 60,000 times. Deduplication allows us to train models that emit memorized text ten times less frequently and require fewer train steps to achieve the same or better accuracy. We can also reduce train-test overlap, which affects over 4% of the validation set of standard datasets, thus allowing for more accurate evaluation. We release code for reproducing our work and performing dataset deduplication at https://github.com/google-research/deduplicate-text-datasets.

1 Introduction

Large language-model datasets contain pervasive duplication and train-test overlap, motivating scalable deduplication. The paper proposes exact and approximate matching techniques and reports lower memorization, reduced cost, and no worse perplexity after deduplication.

  • Motivation: Large-scale language-model datasets are difficult to manually curate and therefore can contain quality problems, including duplicated training examples.The paper frames dataset understanding as a research challenge because training data also affects learned biases.
  • Approach: Two scalable techniques detect duplicates: exact substring matching finds repeated verbatim spans, while approximate full-document matching finds high n-gram overlap.The approximate method uses hash-based techniques to identify similar document pairs.
  • Findings: Over 1% of unprompted emitted tokens are part of memorized sequences, and deduplication reduces memorized-data emission by 10×.This result is reported for models trained on standard datasets such as C4.
  • Findings: A 61-word sequence appears 61,036 times in C4 training data and 61 times in its validation set, illustrating substantial train-test overlap.The sequence represents 0.02% of the samples in each dataset.
  • Findings: Train-test overlap can cause researchers to over-estimate model accuracy and bias model selection toward models and hyperparameters that overfit training data.This makes overlap a concern for evaluation as well as dataset construction.
  • Findings: Deduplicated datasets are up to 19% smaller, reducing training cost, while models have no worse perplexity and sometimes achieve up to 10% lower perplexity.Higher-quality data can also let models reach higher accuracy faster.

2 Related Work

Prior work examined Internet-scale language-model datasets, contamination of downstream evaluation data, memorization, and duplicate documents. This paper instead focuses on how duplication in language-model training and validation sets affects perplexity and memorized generation.

  • Language-model datasets: Language models have been trained on diverse Internet-scale sources, including Common Crawl, Web-Text, news, Wikipedia, and cleaned C4 data.The cited examples include GPT-2, GPT-3, GROVER, T5, and multilingual language models.
  • Contamination: Prior contamination studies found test examples verbatim in web-crawled training data, including up to 14.4% for various standard tasks in C4.Earlier work also analyzed 8-gram overlap between GPT-2’s training set and evaluation datasets.
  • Scope: This paper does not focus on downstream benchmark-task impacts; it studies duplicate text in language-model training and validation sets, perplexity, and memorized generated content.The scope distinguishes its analysis from prior contamination work on downstream tasks.
  • Memorization: Research on memorization has shown that language models can emit training examples, including sensitive data such as valid phone numbers and IRC usernames.The paper reports that over 1% of data emitted by most models is memorized training data.
  • Duplicate text: Studies have documented exact duplicate documents in Book Corpus and worsened code-understanding performance associated with duplicate code examples.These findings motivate examining duplication beyond the datasets used in those studies.

3 Language Modeling Datasets

The study analyzes four English datasets of different sizes and uses, spanning general pretraining, natural-language generation, and language-model benchmarking. Their documented preprocessing and deduplication procedures vary substantially.

  • Dataset scope: The analysis covers four English datasets of varying sizes used for natural-language generation, general-purpose pretraining, and language-model benchmarking.The authors expect non-English datasets may have similar duplication issues but restrict this analysis to English.
  • Wiki-40B: Wiki-40B’s English portion contains 2.9M Wikipedia pages averaging 768 BPE tokens, with no indicated deduplication beyond redirect-page removal.The dataset consists of multilingual cleaned Wikipedia text.
  • LM1B: LM1B contains 30M one-sentence news-commentary examples averaging 32 BPE tokens, and prior work reports 13.2% test-train overlap.Unlike the other analyzed datasets, its examples are single sentences.
  • C4: C4 contains 360M web documents averaging 486 BPE tokens and had previously undergone paragraph hashing, filtering, and removal of hash collisions.Its preprocessing also removed placeholder text, code, and prohibited words.
  • RealNews: RealNews contains 31M news-domain documents averaging 793 BPE tokens and deduplicates using hashes of each document’s first 100 characters in a Bloom filter.Documents with duplicate URLs were also excluded.

4 Methods for Identifying Duplicates

The framework combines exact substring matching with approximate full-document matching to identify and remove both verbatim repetitions and near-duplicate documents at scale.

  • 4.1 Exact Substring Duplication: EXACTSUBSTR removes sufficiently long verbatim substrings shared across examples, using a 50-token minimum match length.The method targets repeated expressions that are unlikely to arise independently and removes the shared substring from one example.
  • 4.1.1 Suffix Arrays: Suffix arrays replace quadratic all-pair matching by enabling linear-time identification of duplicated substrings across the concatenated dataset.The implementation constructs a suffix array over the dataset sequence and scans adjacent suffixes for common prefixes.
  • 4 Methods for Identifying Duplicates: The framework supports scalable processing through linear-time suffix-array construction, parallel scanning, and complementary exact and approximate matching techniques.The paper also describes MinHash parameters and cluster construction for approximate deduplication.
  • 4.2 Approximate Matching with MinHash: NEARDUP uses MinHash to find entire documents with high n-gram overlap, complementing exact matching for templated web text with small interspersed differences.The approach represents documents as n-gram sets and uses signatures to efficiently identify potential approximate matches.
  • 4.2 Approximate Matching with MinHash: NEARDUP classifies a potential pair as duplicate when MinHash matches it and edit similarity exceeds 0.8.Matched documents are then connected in a graph to form clusters of similar documents.

5 Deduplication Results

Deduplication reveals substantial duplicate content across common language-modeling datasets, including near-duplicates, exact repeated substrings, and train-validation overlap.

  • 5.3 Train-Test Overlap: The deduplication pipeline prioritizes retaining duplicated text in test or validation splits and removing it from the training split.This policy is applied when text appears across multiple data splits.
  • 5.1 Amount of Text Removed: 3.04% of C4 examples and 13.63% of RealNews examples are near-duplicates, while Wiki-40B contains 0.39%.C4 also contains 280 clusters with over 5,000 examples, including one cluster of size 250,933.
  • 5.1 Amount of Text Removed: 7.18% of C4 tokens are removed by EXACTSUBSTR, and 77% of examples removed by NEARDUP have a verbatim length-50 match.EXACTSUBSTR removes less data than NEARDUP on most datasets, but LM1B is an exception because 90% of its documents are under 50 tokens.
  • 5.2 Qualitative Analysis: Near-duplicates often differ only in templated fields or formatting, so exact string matching misses computer-generated pages and syndicated news articles.Examples include changes to places, businesses, products, dates, or news-site formatting.
  • 5.3 Train-Test Overlap: 4.6% of the C4 validation set and 14.4% of the RealNews validation set contain approximate duplicates in their training sets.The authors identify this overlap as a source of potentially inflated evaluation metrics for models that memorize training data.

6 Impact on Trained Models

Deduplicating training data preserves or improves perplexity while substantially reducing memorized generations and exposing train-test overlap effects. These effects appear across validation datasets, prompting conditions, and existing released models.

  • On LM1B and Wiki-40B, NEARDUP-deduplicated C4 models consistently achieved the lowest perplexity.EXACTSUBSTR reduced XL-model perplexity on Wiki-40B by almost 3 points despite using fewer training tokens overall.
  • All XL models achieved below 35 perplexity on LM1B, compared with 42.16 reported for the 1.5B GPT-2 using the same vocabulary size.
  • Over 1% of tokens generated without prompts from models trained on original C4 belonged to memorized 50-token subsequences, compared with 0.1% for deduplicated datasets.This represents approximately 10× more memorization for the original-data model.
  • When prompted with duplicate training examples, XL-ORIGINAL reproduced the ground-truth continuation over 40% of the time.Deduplicated models also copied duplicate-example continuations more often than unique-example continuations.
  • Train-test overlap substantially changes evaluation: Transformer-XL perplexity halves on examples identified as near-duplicates of its training data.The same effect is present for GROVER in both model sizes considered.

7 Discussion

The paper analyzes data duplication as one measurable property of language-model datasets, showing that deduplication reduces memorization while leaving the nature and risks of memorized content unresolved.

  • Scope: Data duplication is one analyzed property of language-model datasets, alongside broader concerns about problematic data.The experiments quantify duplicate content, deduplication’s effect on perplexity, and memorized-content reduction, but do not assess the nature of the data removed or memorized.
  • Privacy: Deduplication reduces models’ emission of memorized training data, addressing privacy concerns associated with repeated examples.The paper reports that standard, non-deduplicated datasets make models particularly sensitive to examples repeated multiple times.
  • Scope: The experiments treat all generations closely matching training data as problematic, without distinguishing harmful, innocuous, or intentionally memorized text.A systematic study of the risks associated with detected memorization was beyond the paper’s scope.
  • Limitations: Deduplication may have negative consequences because some tasks require memorization and removing duplicated substrings can remove attribution.The paper also states that deduplication alone cannot remove privacy-sensitive data that should never enter training datasets.
  • Implications: Whether memorization is desirable depends on the memorized text and the downstream application.The paper encourages researchers to consider intended model usage when deciding what belongs in training data.

8 Conclusion

The paper recommends stringent dataset deduplication and concludes that it generally does not harm perplexity while making training faster and reducing overlap-driven evaluation bias and privacy concerns.

  • Conclusion: The authors encourage language-model researchers to perform dataset deduplication using released datasets, tools, or comparable approaches.They state that the exact deduplication technique matters less than performing stringent deduplication.
  • Conclusion: Stringent deduplication is presented as more important than the exact technique used.The conclusion emphasizes the practice itself rather than one specific implementation.
  • Conclusion: Deduplication does not harm, and sometimes improves, model perplexity despite producing datasets that are smaller and faster to train on.The conclusion links deduplication with reduced dataset size and faster training while reporting no overall perplexity harm.
  • Conclusion: Removing training-test duplicates is important because overlap explicitly encourages selecting models that memorize the training data.The conclusion also connects deduplication with reducing some privacy concerns around memorized training data.

Ethics

The paper focuses on over-representation caused by structural duplicates, while noting that data collection also under-represents some user groups. It does not address under-representation or distinguish helpful from harmful memorization.

  • Structural duplicates over-represent some types of text, while data collection under-represents non-English speakers and groups communicating outside the public web.The paper explicitly focuses on the former problem and does not address the latter.
  • The analysis does not distinguish when memorized content is helpful from when it is harmful.The authors state that resolving this distinction would require different research methodologies.

Contributions

The paper combines model training, deduplication, evaluation, and analysis contributions with MinHash-based approximate matching and computational studies of scalable deduplication.

  • Contributions: The authors contributed model training, evaluation and generation pipelines, data deduplication pipelines, analysis, writing, and project management.Individual contributions included approximate matching, suffix-array implementation, plotting, mentorship, and project organization.
  • NEARDUP: The approximate matching pipeline uses space-tokenized 5-grams, MinHash signatures, bucketed hashes, and progressively more expensive similarity filters.This design first narrows candidate pairs with MinHash before computing exact similarity measures.
  • NEARDUP: NEARDUP marks documents as potential matches when their MinHash signatures collide, then retains pairs whose Jaccard and edit similarities exceed 0.8.The selected configuration uses b = 20, r = 450, and k = 9,000 to give collisions a high probability at the chosen Jaccard threshold.
  • Computational Analysis: The computational analysis considers grouping signature complexity and the pathological case in which all documents fall into the same buckets.The analysis assumes b, k, and T are much smaller than N.
  • Computational Analysis: NEARDUP consumed approximately 41.5 kWh on English C4, though different parameters could improve energy efficiency while producing similar results.The reported configuration prioritized very high recall.

B Further Details on EXACTSUBSTR

EXACTSUBSTR constructs and merges parallel suffix arrays to identify repeated substrings at scale, while documenting runtime, memory, and training-cost implications.

  • Implementation: EXACTSUBSTR extends a suffix-array library to support datasets larger than 4GB, raw byte sequences, and streaming the suffix array off disk.The implementation is written in Rust.
  • Parallel partial suffix array construction: The parallel construction partitions the dataset into K splits, runs SA-IS independently, and retains O(N) work with O(N/K) wall-clock time.The resulting partial suffix arrays are later merged.
  • Parallel merge of partial suffix arrays: The algorithm merges partial suffix arrays by repeatedly selecting the lexicographically smaller current suffix, generalized across K splits with a min-heap.The general-case merge complexity is O(Nm log(K)), where m is average prefix-match length.
  • Computational Analysis: The suffix-array method processes Wiki-40B in 2.3 minutes and builds a suffix array for 350GB C4 in under 12 hours, but the C4 suffix array requires 1.5TB.The algorithm still requires the dataset itself to fit in memory.
  • Setting a threshold of duplicates: A 50-token substring threshold was chosen conservatively after the frequency bend near 10 tokens and manual inspection found no false positives at length 25.The authors doubled the selected threshold to provide a large margin for error.
  • Training and evaluation: The deduplicated dataset is approximately 3.9% smaller and saves around 5 hours of compute time across two training epochs.The cited training setup used 63 hours per epoch for the approximate deduplicated dataset.

E More Results

Additional results examine memorization, near-duplicate distributions, dataset sizes, and qualitative examples across datasets and model configurations.

  • Qualitative Examples: Qualitative tables show near-duplicate document pairs, repeated substrings, and unprompted memorized generations identified by the two deduplication methods.The near-duplicate examples include small interspersed differences that make exact matching less effective.
  • Distribution of memorization: Memorized generations include examples from prompts involving duplicated and non-duplicated training or validation data.Figure 6 compares four prompting conditions across generated sequences.
  • URLs with many duplicates: Near-duplicates in C4 commonly occur on product-selling websites with templated text, while content aggregators are especially common in RealNews.The comparison concerns URLs with high proportions of examples marked as near-duplicates by NEARDUP.
  • Dataset Sizes: EXACTSUBSTR removes no examples from datasets that had already been deduplicated for exact matches.The result is reported in the dataset-size analysis.
  • Perplexity: Figure 7 compares validation perplexity for T5 base across three runs and for T5 XL after training on deduplicated data.The plots show the range and median perplexity for the model configurations.
  • Memorization: Models trained with approximate or exact deduplication have 10× less memorization than models trained on the original dataset.The measure is the percentage of tokens in 100k generations that belong to memorized substrings.
Loading 2107.06499v2…