PaperlayerView exact revision on arXiv

Source-linked AI summary

Scalable Extraction of Training Data from (Production) Language Models

Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Choo, Eric Wallace, Florian Tramèr, Katherine Lee

arXiv:2311.17035v1cs.LGcs.CLcs.CR

TL;DR

The paper asks how much memorized training data language models expose to adversaries who lack the training dataset. It defines extractable memorization, develops scalable measurement methods, and finds substantial exposure across open, semi-open, and aligned models.

  • Problem

    Prior work measured memorization quantities or extracted data from relatively small models, leaving large-scale extractability insufficiently studied.

  • Method

    The paper defines extractable memorization through verbatim generation and scales detection using trillion-token outputs and sequential Good-Turing estimation.

  • Results

    Larger models have 5× higher total extractable memorization on average, while gpt-3.5-turbo-instruct emits 0.852% of generated tokens as verbatim 50-token sequences from AUXDATASET.

  • Takeaways & Limitations

    Training data can be easily extracted from leading language models through simple techniques, including models spanning open-source, semi-open, and aligned settings.

  • Takeaways & Limitations

    The evaluation cannot directly establish an extraction attack without a reliable way to predict whether an output is a training example.

Abstract

from arXiv · show

This paper studies extractable memorization: training data that an adversary can efficiently extract by querying a machine learning model without prior knowledge of the training dataset. We show an adversary can extract gigabytes of training data from open-source language models like Pythia or GPT-Neo, semi-open models like LLaMA or Falcon, and closed models like ChatGPT. Existing techniques from the literature suffice to attack unaligned models; in order to attack the aligned ChatGPT, we develop a new divergence attack that causes the model to diverge from its chatbot-style generations and emit training data at a rate 150x higher than when behaving properly. Our methods show practical attacks can recover far more data than previously thought, and reveal that current alignment techniques do not eliminate memorization.

1 Introduction

The paper unifies large-scale measurement of memorization with practical extraction attacks, defining extractable memorization as training data efficiently recoverable without prior dataset knowledge. It finds that larger models are more vulnerable and introduces a divergence attack that substantially increases ChatGPT’s emission of training data.

  • Extractable memorization measures training data that an adversary can efficiently recover, unlike discoverable memorization’s broader upper bound.The paper restricts memorization to verbatim matches so analysis can scale to large datasets.
  • The scalable methodology detects memorization across trillions of model-output tokens and terabyte-sized datasets for open-source and semi-open models.
  • Larger and more capable models are more vulnerable to data-extraction attacks.
  • ChatGPT emits training data 150× more frequently under the new attack than with prior attacks and 3× more frequently than the base model.
  • The aligned ChatGPT initially appears to memorize almost no training data, consistent with its helpful-chat-assistant behavior.The paper hypothesizes that alignment through RLHF contributes to this observed behavior.
  • The divergence attack makes ChatGPT behave like a base language model, recovering over ten thousand training examples at a query cost of $200 USD.The attack matches generated text against a merged nine-terabyte dataset of publicly available web-scale training sets.
  • The ChatGPT attack is specific to gpt-3.5-turbo and was disclosed to OpenAI before publication.

2 Background and Related Work

LLMs are trained on vast, often secret corpora and can memorize their training data. Prior work studied memorization at scale or developed smaller-model extraction attacks, while alignment improves utility and can make models abstain from some requests.

  • State-of-the-art LLMs are pretrained on corpora containing billions to trillions of tokens.Proprietary models keep these datasets secret partly to protect proprietary collection processes and private, user-specific, or licensed data.
  • Instruction tuning and RLHF improve downstream utility and can give models a unified chat persona with abstention behavior.
  • The paper analyzes ChatGPT specifically through the gpt-3.5-turbo model endpoint.
  • Memorized training data can expose potentially private information through membership-inference and data-extraction attacks.Membership inference tests whether an example was in the training set, whereas extraction attacks recover full training examples.

3 Extracting Data from Open Models

The paper develops scalable attacks and evaluation methods for extractable memorization, showing that practical extraction from open models is far more successful than prior estimates suggested. It also estimates total memorization while accounting for repeated outputs and unequal sequence probabilities.

  • Definitions: Extractable memorization means an adversary without training-set access can construct a prompt that makes the model produce a training example verbatim.
  • Prior Approaches and Definitions: Prior extraction work faced separate challenges in eliciting memorization and verifying whether outputs came from training data, often relying on heuristics and manual search.
  • Prior Approaches and Definitions: ≈0.00001% of GPT-2’s training dataset was verifiably recovered by a prior attack, but manual verification prevented a tighter estimate.
  • Empirical Results: Several hundred thousand to several million unique 50-token strings were extracted from open models, yielding rates orders of magnitude above prior reports based on 600 GPT-2 sequences.
  • Estimating Total Memorization: Memorization grows nearly linearly after several hundred billion generated tokens, while extraction rates at small budgets do not reliably predict total memorization.
  • Estimating Total Memorization: Sequential Good-Turing extrapolation uses observed sample frequencies to estimate novel and repeated outputs, but requires sufficient observations and can underestimate unique memorized examples.

4 Extracting Data from Semi-closed Models

The paper evaluates extractable memorization in semi-closed models using a scalable corpus-matching methodology, finding substantial and variable memorization across model families. Larger models yield higher estimated total extractable memorization, while longer training and over-training are associated with more leakage.

  • Motivation: The study extends extraction analysis to semi-closed models, where training datasets are unavailable and must be approximated with an auxiliary ground-truth dataset.The authors ask how much memorized data can be extracted when model and dataset information is only partially public.
  • Attack Methodology: Generated sequences are matched against AUXDATASET, and sufficiently long, high-entropy matches serve as a low-false-positive proxy for training-data membership.Because AUXDATASET does not fully cover the unknown training data, this procedure produces a lower bound on memorization.
  • Experimental Setup: AUXDATASET concatenates four large pre-training corpora into 9TB of text, with document-level tokenization and coarse deduplication before analysis.The component datasets include The Pile, RefinedWeb, RedPajama, and Dolma.
  • Results: All evaluated models emit memorized training data, but comparable Mistral 7B and Falcon 7B differ in detected memorization by over 10×.The authors caution that dataset-distribution mismatch may contribute to the difference, although they judge it unlikely to explain the full rate.
  • Results: 0.852% of generated tokens from gpt-3.5-turbo-instruct belong to verbatim 50-token sequences found in AUXDATASET.The authors identify this as the highest rate among the reported state-of-the-art models, while noting that model-family comparisons are difficult to interpret directly.
  • Results: Estimated total extractable memorization is on average 5× higher for these models than for smaller models, and the estimator likely underestimates the true total.The extrapolation uses a Good-Turing estimator applied to 1 billion generations.

5 Extracting Data from ChatGPT

The paper examines why aligned ChatGPT resists standard extraction attacks and develops a prompting strategy that makes it diverge from chatbot-style generation. This enables recovery of verbatim training examples despite alignment-based evasion.

  • Challenges: Conversational formatting blocks direct continuation attacks because the model generates only after appending the special “Assistant:” turn indicator.Dialog-adapted models do not give the user direct control over arbitrary continuation of a supplied text sequence.
  • Challenges: Alignment can cause ChatGPT to abstain from completing text from its training set, even when the model might otherwise continue the supplied prompt.The paper illustrates this behavior with a prompt containing a BBC description that ChatGPT refuses to complete.
  • Divergence Attack: Alignment therefore does not prevent data extraction: divergent generations can contain verbatim examples from pre-training data.The attack creates a pool of candidate memorized examples by repeatedly sampling and inspecting text after divergence.
  • Baseline Attack: The standard random-token attack is ineffective at scale: only 0.02% of tokens from 50 million gpt-3.5-turbo outputs belonged to copied 50-token sequences.The experiment was designed to provide quantitative evidence that this baseline strategy does not recover substantial training data.
  • Divergence Attack: The authors discover a prompting strategy that makes ChatGPT diverge from standard dialog generation and produce text resembling base-model Internet-style output.One prompt asks the model to repeat a word many times; after initially repeating it, the model sometimes diverges into memorized text.

5.3 Main Experimental Results

The ChatGPT divergence attack recovers thousands of diverse, verbatim memorized sequences, including long documents and sensitive or unusual content. The extracted material spans code, research, boilerplate, personal information, and identifiers.

  • Main Results: Over 10,000 unique verbatim-memorized training examples are extracted from ChatGPT using $200 USD worth of queries.The authors’ extrapolation suggests that larger budgets could recover substantially more data.
  • Length and Frequency: The longest extracted string exceeds 4,000 characters, while several hundred extracted strings exceed 1,000 characters.Figure 6 summarizes the cumulative counts of extracted strings above each length.
  • Length and Frequency: Over 93% of memorized strings are emitted only once, with 4% emitted twice and 0.05% emitted at least ten times.The results indicate that the prompting strategy produces long and diverse outputs after divergence.
  • Qualitative Analysis: The attack recovers personally identifiable information from dozens of individuals and also retrieves NSFW content and verbatim literary text.Reported literary extractions include paragraphs from novels and complete copies of poems such as The Raven.
  • Qualitative Analysis: The extracted outputs include valid URLs with random nonces and cryptographically random identifiers, including an exact bitcoin address.These examples are described as unlikely to have occurred by random chance.
  • Qualitative Analysis: Extracted content includes code, including short JavaScript substrings that appear to have been insufficiently cleaned before training.The paper also reports research-paper excerpts, boilerplate text, and outputs that merge related memorized strings.

5.4 Identifying PII

The attack’s effectiveness varies sharply across prompting words and depends on query budget and auxiliary-dataset coverage. Larger auxiliary datasets and more queries reveal more memorized ChatGPT output, while dataset composition affects coverage.

  • Prompt effectiveness: Single-token words trigger memorization, whereas multi-token words do not cause divergence or emit training data.Multi-token prompts either repeat indefinitely or receive a refusal-like response.
  • Prompt effectiveness: Over 100× separates the most effective single-token prompts from the least effective at recovering memorized output.Variation reflects both how often a word causes divergence and how much training data follows divergence.
  • Query scaling: Over 10,000 unique examples were extracted for $200 USD, while larger query budgets could recover substantially more data.The paper presents this as a lower-bound measurement under a limited budget.
  • Query scaling: 1.5 million unique 50-token sequences is a Good–Turing lower bound, though the authors call the estimate exceptionally poor because far fewer examples were extracted than needed for convergence elsewhere.The comparison uses GPT-Neo 6B, where 500 million examples were needed for convergence.
  • Auxiliary-dataset coverage: 20% of total memorization was discoverable with a 200GB auxiliary-dataset subset, and doubling the dataset might increase discoveries by 20%.The full auxiliary dataset was 9TB; the curve fit suggested the larger-dataset estimate was nearly tight.
  • Auxiliary-dataset coverage: Dataset scale alone does not determine discoveries: RefinedWeb found the least memorization, while RedPajama largely overlapped with other datasets.The authors attribute this pattern to distribution discrepancies between auxiliary datasets and ChatGPT’s training data.
  • Auxiliary-dataset coverage: 150 of 494 manually searched generations were found online, compared with 70 in AUXDATASET, showing automatic matching underestimates memorization.The manual search therefore detected nearly twice as many memorized outputs as the comparatively small auxiliary dataset.

5.7 An End-to-end High-precision Attack

The paper turns its measurement procedure into an end-to-end extraction attack by distinguishing memorized from non-memorized outputs with membership inference. However, the ChatGPT divergence attack is model-specific, and natural prompting rarely reveals the same memorized data.

  • Attack boundary: The end-to-end evaluation remains primarily a measurement study because memorization was identified by querying an approximate training dataset rather than predicted beforehand.The authors distinguish this setup from a direct extraction attack until a reliable predictor is available.
  • Attack construction: High-precision extraction is possible by scoring outputs with a likelihood-ratio perplexity comparing model perplexity against zlib-compression entropy.This score was the most effective predictor of memorization in prior work and remained highly accurate here.
  • Attack construction: Above 30% precision was achieved against manual Internet-search labels, while verbatim AUXDATASET matching yielded 15% at the lowest threshold.Precision stayed relatively constant until a threshold of 1.5, after which it significantly declined.
  • Attack boundary: The ChatGPT divergence attack is not generalizable to other models and is not reliable for testing memorization in general.Ground-truth training examples could instead support discoverable-memorization analysis and upper bounds.
  • Discoverability: 3.5% of known memorized suffixes were completed exactly under natural prompting, with another 4% approximately recovered.Thus, over 90% of the time the model failed to emit memorized output that it had produced under adversarial prompting.
  • Discoverability: The two ChatGPT variants memorized the same samples despite likely different fine-tuning datasets, suggesting the extracted data came from pretraining.The authors also report minimal forgetting after fine-tuning, which they associate with much longer pretraining.
  • Discoverability: External privacy assessment is difficult because the original base model underlying the ChatGPT variants was not made public.The authors contrast this with the possibility that the base model itself might have been easier to audit.

6 Why is ChatGPT so Vulnerable?

The paper considers why ChatGPT appears unusually vulnerable, focusing on speculation about extensive pretraining and a divergence mechanism linked to repeated single tokens. Repetition behavior differs sharply by tokenization.

  • Possible causes: ChatGPT is reported as significantly more vulnerable to extraction than prior base-language-model results, but the paper presents the explanation as speculative.The authors invite future work to investigate the possible reasons.
  • Possible causes: Many pretraining epochs may increase memorization because limited high-quality web data encourages repeated passes over the same corpus.The paper connects this possibility to over-training models on more data than compute-optimal scaling would suggest.
  • Repetition instability: Two- or three-token words can repeat thousands of times without divergence, unlike one-token words that approach near-certain divergence after a few hundred repetitions.Figure 12 reports medians over 40 word choices with 10%–90% quantile ranges.
  • Repetition instability: After 250 repetitions, the probability of repeating a single token again falls from 90% to below 0.1%.This sharp instability is observed in gpt-3.5-turbo-instruct, which exposes token probabilities.
  • Possible mechanism: The attack may simulate the endoftext boundary behavior learned during packed-document pretraining, although the API blocks direct insertion of that token.The proposed mechanism is that the model effectively resets and begins generating as if at a new document.
  • Possible mechanism: Repeated-token prompts alter LLaMA’s last-layer attention-query representations, whereas naturally sampled random prompts do not produce the same effect.LLaMA diverges under repetition but does not emit training data in the reported experiment.

7 Conclusions

The paper argues that practical extraction attacks recover substantial memorized training data, including from aligned models, while highlighting limits of current defenses and open questions about memorization.

  • Pythia’s deduplication reduced extractable memorization only slightly, suggesting coarse-grained deduplication was insufficient.The passage attributes the limited reduction to inadequate coarse-grained deduplication.
  • GPT-Neo 6B emits nearly a gigabyte of training data, corresponding to approximately 10% of its model capacity being spent on verbatim memorization.The estimate assumes the emitted information must be stored somewhere in the model weights.
  • A surprisingly high fraction of memorized data can be extracted with sufficient effort, although not all memorized examples are extractable.The authors therefore recommend studying discoverable memorization independently because it is easier to measure.
  • Practical attacks can extract training data from aligned models, challenging the conclusion that alignment prevents data extraction.The authors describe latent vulnerabilities in aligned models and caution that stronger prompting strategies may remain undiscovered.
  • The authors caution practitioners deploying models on private, proprietary, or public datasets to account for unresolved extraction and memorization risks.They frame the results as a cautionary lesson for responsible model deployment.

Contributions

The contributions include discovering and validating the ChatGPT token-repetition attack, building large-scale extraction infrastructure, and conducting analyses of memorization, privacy, and model behavior.

  • Milad discovered the token repetition attack, and Milad and Nicholas confirmed that ChatGPT emitted memorized training data.
  • The researchers conducted ChatGPT parameter experiments, performed manual Web searches, and analyzed differences between discoverable and extractable memorization.
  • The project generated combined terabytes of model outputs from 17 open and semi-closed models for large-scale analysis.
  • Nicholas built the suffix array and an efficient training-data intersection algorithm, then ran it over the generated data.
  • The authors generated scaling-extrapolation plots and tested discoverable memorization across gpt-3.5-turbo and gpt-3.5-turbo-instruct.
  • The project included experiments on the EOS token effect, multiple-epoch effects, final figures, and associated data analysis.
  • The remaining contributions covered project organization, paper framing, and collective writing.
  • The team analyzed memorized data containing PII and investigated correlations between model performance and extraction.

A Suffix Arrays

Suffix arrays make substring containment checks efficient by indexing dataset suffixes in lexicographic order, enabling binary searches over large concatenated training corpora.

  • O(logn) binary search replaces an O(n) scan when checking whether a suffix appears in a large training dataset.
  • A suffix array indexes all dataset suffixes in lexicographically sorted order, enabling efficient binary searches for a substring or suffix.
  • For “banana,” sorting suffix indices as {5,3,1,0,4,2} orders suffixes from “a” through “nana,” illustrating the searchable representation.Constructing the suffix array takes linear time.
  • The method concatenates all documents into one long string so suffix-array containment checks can operate over terabyte-scale corpora.During binary search, it checks the first k characters of the suffix at the current index.

B.1 Impact of Varying k in Our Memorization Definition

The memorization definition uses a 50-token containment threshold, whose value changes the measured total while preserving the study’s overall comparative trends.

  • A sequence is counted as memorized when it is at least 50 tokens long and contained in the training dataset.
  • Changing the threshold changes the total number of identified memorized examples because the 50-token cutoff is somewhat arbitrary.
  • Across different threshold values, the overall trends remain similar, including model comparisons in memorization rates.

B.2 Estimating Total Memorization

The paper estimates total memorization by modeling observed memorized strings as a sample from a larger population and simulating how many unique sequences remain undiscovered.

  • The authors assume the model memorizes a set S of N training examples and observe only a subset s from limited generations.
  • They apply the Good-Turing estimator to account for memorized sequences that reappear at different frequencies.
  • After 10M generations, simulations estimate the total number of unique memorized sequences by repeatedly updating the probability of observing new sequences.
  • The study also evaluates other population-estimation techniques from ecology and epidemiology, with results summarized in Table 3.

C Additional Figures

The additional figures examine memorization across model families, auxiliary datasets, model sizes, benchmark performance, sequence duplication, and ChatGPT generations.

  • Figure 15 measures the percentage of generated tokens forming direct 50-token copies from training data across four model families and reports unique memorized examples above each bar.
  • Figure 16 compares total and unique examples recovered from each auxiliary-dataset constituent, showing that dataset size does not determine recovered memorization uniformly.
  • Figure 17 relates model size to extracted 50-grams from one billion generated tokens and reports more extraction as model size increases within a family.
  • Figure 18 finds no strong correlation between unique extracted 50-grams and benchmark performance, treating extractable memorization as distinct from benchmark quality.
  • Figure 19 shows that sequences with more Pile duplicates are easier to extract and discover, while duplicated examples are more discoverable than extractable.
  • Figure 20 counts how many auxiliary-dataset duplicates occur for each memorized generation extracted from ChatGPT.

D Examples of ChatGPT Divergence

The examples illustrate ChatGPT diverging from repeated-word prompts into unrelated-looking text, including passages that match public web-scale datasets and appear memorized.

  • A ChatGPT attack begins with repeated “book” tokens, after which the model diverges and emits random content.
  • Longer generated sequences matching auxiliary training data are treated as unlikely to have arisen by random chance alone.
  • One divergence continues with a long passage about Luna, Fowler, and a book review, illustrating the kind of Internet-style text emitted after repetition.
  • The examples also include author-biographical and promotional material, such as Wendy Higgins’s biography and giveaway details.

E Verbatim Memorized Training Sequences

The paper presents verbatim memorized sequences extracted from model outputs, spanning web pages, books, code, product information, and other text.

  • Near-duplicate outputs can recur across extracted examples, as illustrated by repeated country lists, web-form fragments, and overlapping text about Harry Carey.The paper notes that its 100 longest examples contain near-duplicates of potential training examples.
  • Extracted sequences include long passages from web pages, such as travel information, product descriptions, and online gaming content.Examples include cruise itineraries, casino and game pages, and commercial product listings.
  • Some extracted sequences reproduce structured or technical content, including country lists, software code, web forms, and business descriptions.The examples include country-name lists, Kernel SVM code, Laravel registration forms, and company profiles.
  • The examples preserve verbatim fragments and sometimes combine unrelated documents within a single recovered sequence.Several passages visibly transition between different websites, topics, languages, or document types.
  • The recovered text also contains biographical and literary material, including author biographies, book reviews, and historical or political descriptions.Examples include Wendy Higgins’s biography, a review of The Queen of Sorrow, and material about Harry Carey and Roy Cooper.
Loading 2311.17035v1…