Source-linked AI summary
Deduplicating Training Data Mitigates Privacy Risks in Language Models
Nikhil Kandpal, Eric Wallace, Colin Raffel
TL;DR
Language models can memorize and leak private training data, but the paper investigates whether duplication drives the success of privacy attacks. Across experiments, duplicate counts strongly predict regeneration and attack detectability, while deduplication reduces attack effectiveness.
Problem
Language models may memorize and leak private information, motivating study of how well they preserve training-data privacy.
Method
The paper measures training-sequence regeneration and membership-inference performance as functions of sequence duplication, comparing models trained with and without sequence-level duplication.
Results
A sequence appearing 10 times is generated ∼1000× more often than one appearing once, while membership inference reaches AUROC as high as 0.90 for heavily duplicated sequences but chance accuracy for singletons.
Takeaways & Limitations
Privacy-attack evaluations should account for duplication as a confounding factor, and privacy-sensitive applications should consider deduplicating training data.
Takeaways & Limitations
The study focuses on exact duplicates; approximately recovered or semantically similar near-duplicates remain an open problem.
Abstract
from arXiv · showhide
Past work has shown that large language models are susceptible to privacy attacks, where adversaries generate sequences from a trained model and detect which sequences are memorized from the training set. In this work, we show that the success of these attacks is largely due to duplication in commonly used web-scraped training sets. We first show that the rate at which language models regenerate training sequences is superlinearly related to a sequence's count in the training set. For instance, a sequence that is present 10 times in the training data is on average generated ~1000 times more often than a sequence that is present only once. We next show that existing methods for detecting memorized sequences have near-chance accuracy on non-duplicated training sequences. Finally, we find that after applying methods to deduplicate training data, language models are considerably more secure against these types of privacy attacks. Taken together, our results motivate an increased focus on deduplication in privacy-sensitive applications and a reevaluation of the practicality of existing privacy attacks.
1. Introduction
The paper argues that duplication in web-scraped training data drives the apparent success of language-model privacy attacks. It links duplication to regeneration and detection, then shows deduplication weakens model inversion attacks.
- Motivation: Language models can memorize and leak private training information, enabling black-box model inversion attacks that generate and identify training sequences.These applications may involve emails, health records, chat logs, and source code.
- Main findings: Duplication largely explains the success of the Carlini et al. (2021b) attack, whose recovery likelihood correlates with a sequence’s training-set occurrence count.The study evaluates transformer language models across multiple parameter scales.
- Main findings: A sequence present 10 times is generated ∼1000× more often than one present once, reflecting a superlinear relationship between duplication and regeneration.Non-duplicated samples are very rarely regenerated.
- Main findings: Membership inference reaches AUROC as high as 0.90 for frequently duplicated sequences but only chance accuracy for sequences appearing once.Thus, detection performance depends strongly on duplication count.
- Implications: Deduplicated models emit ∼20× less training data and reduce the effectiveness of two of the three evaluated membership inference methods.The results indicate that deduplication substantially weakens these privacy attacks.
2. Background and Experimental Setup
The paper defines generation-based memorization and examines a two-stage model inversion attack using duplicated sequences in web-scraped datasets. Its experiments measure how duplication affects generation and membership-inference detection.
- Background: Generation-based memorization occurs when a generative model leaks information by producing exact samples from its training data.A perfect-memorization model provides a positive control by sampling uniformly from the training corpus.
- Privacy attacks: The studied attack first generates large amounts of text and then scores generated sequences with a membership-inference method.The highest-scoring sequences are classified as training data.
- Experimental setup: The experiments use unconditional generation with standard, top-k, and temperature sampling, then evaluate the proposed membership-inference scores.Each score compares a sequence-easiness metric with the trained model’s perplexity.
- Datasets: OpenWebText and C4 retain large-scale exact sequence duplication despite deduplicating whole training examples.Both datasets contain millions of 400-character sequences duplicated at least 10 times, with some duplicated tens of thousands of times.
- Experimental setup: The analysis records each N-length sequence’s duplicate count, generates repeatedly, and compares regeneration rates across duplication levels and model sizes.Generation is scaled to text equal in size to the training dataset for cross-dataset comparisons and comparison with perfect memorization.
3. How Duplication Affects The Regeneration of Training Sequences
Language models regenerate duplicated training sequences at disproportionately high rates, while rarely regenerating sequences with few or no duplicates. This superlinear pattern persists across sequence lengths, model scales, sampling schemes, and training durations.
- A superlinear relationship links a sequence’s duplication count to its regeneration rate across tested language models.The relationship appears as a slope greater than 1 on a log-log plot.
- Sequences duplicated d times are generated far fewer than d times, especially when duplicate counts are low.Unduplicated samples are very rarely regenerated, limiting attacks that rely on verbatim model outputs.
- Effect of Duplicate Sequence Length: The superlinear regeneration pattern remains nearly identical across different sequence lengths.Longer sequences reduce exact overlap probability, but do not materially change the relationship between duplication and generation.
- Effect of Model Scale: Larger models regenerate more training data across all duplication levels, likely because they assign training samples higher likelihoods.The comparison cited uses 117M and 345M parameter Mistral models trained nearly identically.
- Effect of Sampling Scheme: Sampling schemes that emit more likely sequences generate more verbatim training samples, although all methods rarely generate sequences with few duplicates.Examples include top-k with smaller k and temperature sampling with smaller T.
- Effect of Increasing Epochs: Additional training epochs increase regeneration multiplicatively across duplication levels, while early stopping does not remove disproportionate regeneration of highly duplicated sequences.Using twice as many epochs increases expected generations by approximately 3 times.
4. How Duplication Affects The Detection of Training Sequences
Membership-inference detection is strongly tied to duplication count: it is near chance for sequences appearing once but substantially better for heavily duplicated sequences. Detection rates likewise rise with duplication, concentrating attack successes on repeated data.
- Additional training epochs have a multiplicative effect on expected generations that is uniform across duplication levels.Doubling the number of epochs increases expected generations by approximately 3 times.
- Membership-inference effectiveness was evaluated by stratifying generated sequences by duplication count and comparing them with 25,000 generated sequences absent from training.The duplication buckets range from d = 1 to d = 800.
- For generated sequences found once in training, all three membership-inference scores produce classifiers close to chance.The evaluated scores are Reference Model, zlib, and Lowercase; a random no-skill classifier has AUROC 0.50.
- At a 0.1% false-positive rate, Reference Model true-positive rate reaches 0.60 for heavily duplicated sequences but is 0.10 for sequences appearing once.This evaluation reflects a setting where training data is rare among generated samples.
- All three membership-inference scores have true-positive rates highly correlated with the number of duplicates.Average non-trivial accuracy is therefore concentrated on sequences duplicated many times.
5. Model Inversion with Deduplicated Data
Deduplicating training data weakens model inversion attacks by reducing the training sequences models regenerate, although some surviving generations remain detectable by membership inference.
- The study compares one C4-trained 1.5B-parameter LM with another trained on an exact-sequence-deduplicated version of C4.Exact duplicates at least 50 BPE tokens long were removed, and recovery targeted 400-character sequences.
- ∼20× less training data was emitted by the model trained on deduplicated data.The experiment generated one million samples and measured copied 400-character training sequences.
- Deduplication considerably affected zlib and Lowercase membership inference scores, while Reference Model performed almost equally on both models.
- 0.87 AUROC remained possible for the deduplicated model because its rare regenerated training samples may be unusually easy to classify.
- Deduplication is an effective safeguard against regenerating training data and renders the first stage of many model inversion attacks largely ineffective.The defense does not harm language-modeling perplexity according to prior work, but Reference Model can still detect rare regenerated samples.
6. Discussion
The discussion limits the paper’s claims to exact duplication in natural language and notes that deduplication may remain necessary alongside differential privacy.
- The analysis defines duplicates as exactly matching sequences, leaving near-duplicates and semantically similar sequences as an open problem.
- Even strong differential privacy may allow exact or near-duplicates to have a large cumulative impact on the model, so deduplication remains necessary alongside DP.
- Whether duplication drives privacy-attack success in images and source code remains unclear because the study focuses on natural language.
7. Related Work
Related work establishes generation-based memorization, privacy attacks, defenses, and deduplication’s broader effects, including cases where memorization supports factual question answering.
- Memorization: Generation-based memorization defines leakage as a generative model producing exact samples from its training data.
- Privacy Attacks: Membership inference identifies training data from candidate samples, while model inversion can recreate training samples using only a trained model.
- Privacy Defenses: Differential privacy limits the effect of any single training example on the trained model, providing a formal privacy defense.
- Benefits and Drawbacks of Deduplication: Sequence-level deduplication can reduce emitted training data and speed training without harming perplexity, although duplication can harm perplexity at intermediate levels.
- Benefits and Drawbacks of Deduplication: Deduplication is not universally beneficial because memorizing training-set facts can be necessary for some closed-book question-answering tasks.
8. Conclusion and Future Work
The paper argues that sequence-level duplication is a major factor behind privacy attacks on language models and that privacy-preserving analysis must trace vulnerabilities to their data and training causes. It further recommends accounting for duplication when evaluating attacks and studying how data features affect attack success and defenses.
- Sequence-level duplication is identified as a large factor behind the success of recent privacy attacks on language models.The authors describe the superlinear increase in regeneration rates with duplicate count as a novel phenomenon for future study.
- Privacy-preserving model development should trace vulnerabilities back to training algorithms, models, and datasets rather than only identifying them.
- Attack evaluations should treat duplication as a possible confounding factor after deduplication may reduce apparent attack effectiveness.
- Future attacks should be evaluated across data features such as duplication to clarify when attacks succeed and how to defend against them.