PaperlayerView exact revision on arXiv

Source-linked AI summary

Extracting Training Data from Large Language Models

Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, Colin Raffel

arXiv:2012.07805v2cs.CRcs.CLcs.LG

TL;DR

Large language models can expose information from their training data, but whether individual examples can be recovered through black-box queries was unclear. This paper develops an extraction attack and finds that GPT-2 reveals hundreds of verbatim training sequences, with larger models more vulnerable.

  • Problem

    Whether language models expose individual training examples through black-box queries remains insufficiently established despite known training-data leakage.

  • Method

    The paper defines eidetic memorization and uses an efficient black-box querying method to extract verbatim sequences from language-model training data.

  • Results

    604 of 1,800 candidate samples were verified as unique memorized training examples, while larger models were more vulnerable to extraction.

  • Takeaways & Limitations

    Training data memorization poses privacy risks and must be addressed for large language models to be widely adopted.

  • Takeaways & Limitations

    The reported number of memorized examples is an extremely loose lower bound because the study manually inspected only 1,800 candidate samples.

Abstract

from arXiv · show

It has become common to publish large (billion parameter) language models that have been trained on private datasets. This paper demonstrates that in such settings, an adversary can perform a training data extraction attack to recover individual training examples by querying the language model. We demonstrate our attack on GPT-2, a language model trained on scrapes of the public Internet, and are able to extract hundreds of verbatim text sequences from the model's training data. These extracted examples include (public) personally identifiable information (names, phone numbers, and email addresses), IRC conversations, code, and 128-bit UUIDs. Our attack is possible even though each of the above sequences are included in just one document in the training data. We comprehensively evaluate our extraction attack to understand the factors that contribute to its success. Worryingly, we find that larger models are more vulnerable than smaller models. We conclude by drawing lessons and discussing possible safeguards for training large language models.

1 Introduction

The paper shows that large language models can memorize and leak individual training examples despite exhibiting little to no average overfitting. It introduces a black-box extraction attack and demonstrates its effectiveness on GPT-2, while discussing practical mitigation strategies.

  • Contributions: The authors demonstrate that large language models memorize and leak individual training examples, including worst-case examples that are not distinguished by average loss.They argue that low average overfitting does not rule out memorization of particular training examples.
  • Attack method: The attack uses black-box query access to generate diverse high-likelihood samples, then ranks them by abnormal likelihood ratios against a separate reference model.It combines three sampling strategies with six reference-model-based ranking metrics.
  • Experimental setting: The experiments use GPT-2 because both the model and its original public training-data source were already available, minimizing potential real-world harm.The attack is intended to apply to language models trained on sensitive and non-public data as well.
  • Results: Over 600 of 1,800 candidate memorized samples were verbatim GPT-2 training examples, and 67% were verbatim examples in the best attack configuration.The 1,800 candidates covered 3×6 attack configurations, with 100 candidates per configuration.
  • Mitigations: The paper discusses differential privacy and careful document de-duplication as mitigation strategies, while noting tradeoffs and that de-duplication cannot prevent all attacks.Differential privacy can increase training time and degrade utility, despite providing theoretical privacy guarantees when applied at an appropriate record level.

2 Background & Related Work

Language models estimate next-token probabilities with neural architectures, especially Transformers, and generate text autoregressively. The section frames training-data extraction as a privacy threat distinct from membership inference and representative model inversion.

  • Language Models: Next-step prediction trains language models to estimate the probability distribution of token sequences by applying the chain rule of probability.
  • Language Models: State-of-the-art language models use attention-based Transformer architectures, replacing recurrent neural networks in current models.
  • Training Objective: Training maximizes the probability of text documents, while massive datasets make training and test losses nearly identical despite the objective’s memorization pressure.
  • GPT-2: GPT-2 models share a dataset and training algorithm but vary in size; GPT-2 XL has 1.5 billion parameters, while Small and Medium have 124 million and 334 million.
  • Privacy Attacks: Training-data extraction reconstructs verbatim training examples, making it more dangerous than membership inference or model inversion that respectively identify membership or recover representative examples.

3 Threat Model & Ethics

The paper argues that training-data extraction is practical and formalizes harmful memorization as extractable strings appearing in few training examples. It defines a realistic black-box threat model, indiscriminate extraction objective, and ethical concerns involving contextual privacy and personal information.

  • Threat Model & Ethics: Training-data extraction attacks are practical despite the expectation that large models trained on near-terabyte datasets for few epochs do not overfit.The paper frames this result against the prevailing intuition that privacy leakage correlates with overfitting.
  • Memorization: Eidetic memorization means a model extractably retains a string appearing in at most k training examples, with smaller k indicating stronger memorization.The definition requires the string to be generated as the most likely continuation for some prefix, or approximated through practical sampling.
  • Adversary Capabilities: The adversary has black-box access to sequence probabilities and next-word predictions, but cannot inspect model weights or hidden states.The threat model reflects language models exposed through black-box APIs.
  • Adversary Objective: The attack indiscriminately extracts memorized training data, with strength measured by total and sequence length of examples and their k values.The study avoids targeted extraction to characterize general memorization rather than create an attack for targeting specific users.
  • Ethics: Small-k memorization is emphasized because it makes extraction more impactful, while contextual integrity treats public-data reuse outside its intended context as a privacy infringement.Information can be public yet privacy-sensitive when extracted from its original context.
  • Ethics: Even with public model and training data, extraction can recover personal information, so the paper partially masks successfully extracted personally identifying content.The paper reports recovering a person’s full name, address, and phone number.

4 Initial Training Data Extraction Attack

The initial extraction attack generates unconditional samples and filters them with membership inference, prioritizing sequences with the lowest perplexity. On GPT-2 XL, it recovers diverse memorized content but is limited by duplicate outputs and many false positives from repetitive text.

  • Attack procedure: The baseline first generates text unconditionally, then predicts which outputs contain memorized training data using membership inference.These steps correspond to extracting model knowledge and predicting k-eidetic memorization.
  • Attack procedure: The attack initializes generation with a one-token start-of-sentence prompt and autoregressively samples tokens according to the model’s likelihood.The method samples exactly 256 tokens for each trial.
  • Attack procedure: Membership inference ranks generated samples by model confidence, using low perplexity as evidence that the model assigns high probability to their tokens.Perplexity measures how well the language model predicts a sequence; lower perplexity indicates less surprise.
  • Findings: 200,000 samples from GPT-2 XL, the 1558M-parameter model, revealed memorized licenses, streaming-site guidelines, and popular individuals’ Twitter handles or email addresses.The identified memorized content likely appeared in the training dataset many times, so it was not low-k eidetic memorization.
  • Limitations: The baseline produced low-diversity outputs, with several hundred duplicate samples of Vaughn Live’s memorized user guidelines.It also generated many false positives, especially repetitive strings that large language models incorrectly score highly.

5 Improved Training Data Extraction Attack

Section 5 improves extraction by diversifying model sampling and replacing raw likelihood filtering with membership-inference comparisons that suppress common, repetitive, or otherwise uninformative high-likelihood outputs.

  • Overview: The improved attack combines better sampling methods with membership inference to address the proof-of-concept attack’s low precision and recall.The earlier attack produced high-likelihood samples not always present in training data and missed memorized content for low k.
  • Improved sampling: Temperature sampling increases output diversity by flattening the next-token distribution, with temperature decaying from 10 to 1 over the first 20 tokens.Higher temperatures encourage exploration, while decay helps preserve a memorized path once generation begins following it.
  • Improved sampling: Internet-scraped prefixes provide diverse contexts resembling GPT-2’s training data, using 5–10 tokens from a 50MB sanitized, line-deduplicated dataset.The authors use a Common Crawl subset and note that possible dataset overlap is rare and limited by the short context prefixes.
  • Improved membership inference: Raw likelihood filtering has poor precision because GPT-2 assigns spuriously high likelihood to trivial memorization and repeated substrings.Examples include counting from 1 to 100 and repeatedly emitting strings such as “I love you.”
  • Improved membership inference: The key membership-inference strategy filters samples whose GPT-2 likelihood is unexpectedly high relative to a second language model.The comparison is intended to identify rarer, more diverse memorization while removing high-likelihood content that another model would also find unsurprising.
  • Improved membership inference: Additional membership metrics compare GPT-2 with zlib entropy, lowercased-text perplexity, or minimum perplexity over a 50-token sliding window.These alternatives capture compression-based surprise, casing-sensitive memorization, and memorized substrings embedded in otherwise unlikely text.

6 Evaluating Memorization

Across all attack strategies, the evaluation identifies 604 memorized training examples, including canonical and unique content such as personal information, URLs, source code, and random sequences. Detection effectiveness varies by strategy and context, and extending extracted samples suggests the total memorized content is substantially larger.

  • Attack effectiveness: 273 unique memorized samples were found with Internet-text conditioning, compared with 191 from baseline top-n sampling.Conditioning on Internet text was the most effective strategy, although every generation scheme revealed substantial memorized content.
  • Attack effectiveness: 67% of Internet samples marked by zlib were memorized, whereas only 9% and 3% of inspected top-n and temperature samples were memorized.Comparison-based metrics were more effective than directly using language-model perplexity for membership inference.
  • Strategy differences: 13 and 10 high-entropy examples were found by the Small and Medium strategies, respectively, compared with one using zlib.The zlib strategy often found non-rare repeated text, while lower-casing often found irregularly capitalized headlines and error logs.
  • Memorized content categories: Memorized content included 46 examples with individuals’ names, 32 with contact information, 50 live URLs, 31 source-code samples, and 21 random sequences with at least 50 bits of entropy.The evaluation also found k = 1 eidetic examples, each appearing in exactly one training document, with random sequences ranging from 10 to 87 characters.

7 Correlating Memorization with Model Size & Insertion Frequency

The study measures GPT-2’s memorization of naturally occurring Reddit URLs by varying their insertion frequency and model size. Larger models extract more URLs, with the 1.5 billion-parameter model fully memorizing examples inserted at least 33 times, indicating reduced privacy as model size increases.

  • Methods: The authors study naturally occurring Reddit URLs whose insertion frequencies vary within a single training document.The URL counts were confirmed to be within 5% of their true frequencies in GPT-2’s training data.
  • Methods: They evaluate GPT-2 XL, Medium, and Small using direct prompting with top-n sampling and an easier variant supplying the URL’s first six characters before beam search.The direct approach tests 10,000 generated extensions for verbatim matches; the easier approach provides additional context and uses higher-recall beam search.
  • Results: Under the harder attack, GPT-2 XL emits all examples inserted 33 times or more, Medium memorizes half the URLs, and Small memorizes none.The models contain approximately 1.5 billion, 345 million, and 117 million parameters, respectively.
  • Results: With additional context and beam search, Medium emits four more URLs, while Small emits only the URL inserted 359 times.This easier attack increases extraction success for both smaller variants, but the smallest model still recovers only one URL.
  • Implications: The results show that larger models memorize significantly more training data, while complete memorization by the largest model occurs after just 33 insertions.The authors characterize this relationship as a privacy cost associated with improvements from increasing model size.

8 Mitigating Privacy Leakage in LMs

The paper outlines several defenses against privacy leakage in language models, including differential privacy, training-data curation, downstream safeguards, and empirical auditing. These measures reduce risk but do not guarantee that memorized private data cannot leak.

  • Training With Differential Privacy: Differential privacy offers strong individual-record guarantees, but user-level guarantees require contribution labels unavailable for open-Web data.Page-level privacy may also be problematic because rare snippets can appear on multiple webpages.
  • Curating the Training Data: Training-data curation can filter personal information and restrictive content, while improved deduplication and source limits may reduce repeated sensitive content.Document- or paragraph-level deduplication can leave many repeated occurrences of sensitive material within one document.
  • Curating the Training Data: Data sanitization is imperfect because private data can slip through, making careful sourcing a first line of defense rather than outright prevention.Some extracted personal information came from sites known to host sensitive content, including pastebin, the 12th most popular domain in GPT-2’s training set.
  • Limiting Impact of Memorization on Downstream Applications: Fine-tuning may make pretrained models forget memorized data, but task-specific private data can introduce new leaks, and generated text could be filtered when memorized content is detectable.The paper identifies how memorization is inherited by fine-tuned models as a direction for future work.
  • Auditing ML Models for Memorization: Models should be audited empirically after mitigation, including when differential privacy is used, to complement theoretical bounds on privacy leakage.The authors propose using their methods and existing attacks to audit language models.

9 Lessons and Future Work

The paper finds that black-box extraction attacks pose a practical privacy threat because modern language models memorize training data, with larger models exhibiting more memorization. It argues that improved extraction methods and layered mitigation strategies remain important future work.

  • Extraction Attacks Are a Practical Threat: The attacks remain practical when a given sequence appears only a few times in the training data.The findings extend prior concerns about memorization by much smaller models to state-of-the-art language models in practice.
  • Extraction Attacks Are a Practical Threat: At least 604 of 600,000 honestly generated samples, or 0.1%, contained memorized text under the black-box attack setting.The authors describe this as an extremely loose lower bound because they manually inspected only 1,800 candidate samples.
  • Memorization Does Not Require Overfitting: Verbatim memorization occurs despite large language models having no significant train-test gap, so preventing overfitting alone may not prevent memorization.Some training examples can still have anomalously low loss even when average training and validation losses differ only slightly.
  • Larger Models Memorize More Data: The 1.5 billion parameter GPT-2 model memorized over 18× as much content as the 124 million parameter model in one experimental setting.The authors warn that privacy leakage may become more prevalent as language models grow larger.
  • Memorization Can Be Hard to Discover: Memorized data is often discovered only with particular prefixes, motivating better prefix-selection strategies for eliciting additional memorized content.The current approach relies on high-quality prefixes and hopes they trigger memorization.
  • Adopt and Develop Mitigation Strategies: Proposed mitigations include differential privacy, sensitive-data vetting, limiting downstream impact, and memorization audits, but each remains incomplete.The paper presents these as promising future-work directions rather than complete solutions.

10 Conclusion

The paper concludes that practical extraction attacks expose serious training-data memorization risks, including for examples appearing in only one document. It calls for scalable privacy-preserving training methods and further investigation into memorization.

  • 10 Conclusion: Practical and efficient extraction attacks can recover hundreds of training examples, including examples contained in just one training document.The conclusion frames these memorization problems as barriers to widespread adoption of large language models.
  • 10 Conclusion: The same techniques apply beyond GPT-2, and increasing model size is expected to make these vulnerabilities significantly more important.The authors characterize the analysis as a cautionary tale for training large language models on sensitive data.
  • 10 Conclusion: Differentially private training may mitigate privacy leakage, but new methods are needed to train billion-parameter models without sacrificing accuracy or training time.The conclusion also identifies open questions, including why models memorize.

Summary of Contributions

The paper develops a GPT-2 training-data extraction attack by framing the threat, defining memorization, generating and verifying candidate samples, and manually reviewing memorized content.

  • The authors proposed extracting training data from GPT-2 and framed the threat model.
  • They stated definitions of memorization.
  • They wrote code to generate candidate memorized samples from GPT-2 and verify ground-truth memorization.
  • They manually reviewed and categorized the candidate memorized content.

A Categorization of Memorized Data · B Distribution of Model Perplexities

The paper categorizes 604 memorized samples extracted from GPT-2 and examines perplexity distributions across generation and membership-inference strategies. Samples may belong to multiple categories, while perplexities are analyzed by strategy combinations.

  • A Categorization of Memorized Data: 604 memorized samples extracted from GPT-2 are assigned to high-level categories.The categorization is summarized in Table 5.
  • A Categorization of Memorized Data: A single memorized sample can belong to multiple categories.Thus, category assignments are not necessarily mutually exclusive.
  • A Categorization of Memorized Data: Tables 6 and 7 break down the categorization by attack strategy.Those tables are omitted for space.
  • B Distribution of Model Perplexities: Figure 4 shows the distribution of sample perplexities.The figure covers samples generated with each of the three text-generation strategies.
  • B Distribution of Model Perplexities: Perplexity distributions are reported separately for each of the three text-generation strategies.The samples are generated using the paper's three specified strategies.
  • B Distribution of Model Perplexities: The samples in Figure 4 are ordered according to six membership-inference strategies.Perplexity distributions therefore reflect both generation and ordering strategy dimensions.

C Additional Case Studies of Memorization

Manual analysis found memorized content spanning leaked emails, political statements, promotional material, number sequences, news headlines, and base-64 encoded data. Several examples contained unusual or personally identifying details, and peculiar formatting sometimes appeared to contribute to memorization.

  • Leaked Podesta Emails: GPT-2 memorized URLs from leaked Podesta Emails, although each URL appeared in only one training document.Email replies could replicate a URL tens of times in the dataset even when it was intentionally used once.
  • Donald Trump Quotes and Tweets: GPT-2 memorized Donald Trump quotes from official presidential records and Twitter statements.These examples came from content collected during the 2016 US Presidential election, when Trump was frequently in the news.
  • Promotional Content: GPT-2 memorized promotional content for books, beauty products, and software, including a Patreon link listing named and pseudonymous donors.The extracted sample included a link to an author’s valid Patreon account.
  • Number Sequences: Nearly ten memorized examples contained integers counting upward, alongside memorized square, Fibonacci, and π digit sequences.The quantity of memorized number sequences was surprising despite the sequences being individually unsurprising.
  • News Headlines: Numerous memorized news snippets copied headlines verbatim, often using a distinctive thehill.com template with merged words and a trailing all-caps word.The token “TrumpJesuit” appeared in no other Web context, and such peculiarities likely contributed to memorization.
  • Base-64 Content: GPT-2 emitted base-64 encoded content that decoded to an image-related path, although its origin could not be identified.The decoded sequence began “master|images|79919|image /png|images/hde/h04/8845672611870.png|...”.
Loading 2012.07805v2…