Source-linked AI summary
Quantifying Memorization Across Neural Language Models
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, Chiyuan Zhang
TL;DR
Language models can emit memorized training data, creating privacy concerns, while prior estimates leave the extent and scaling of memorization incompletely understood. The paper measures exact extraction across models and training data using training-set prefixes, finding that memorization rises with model size, duplication, and context, with implications for mitigation and privacy auditing.
Problem
Existing extraction attacks provide only lower bounds, and memorization remains incompletely understood across models and datasets of different scales despite its privacy implications.
Method
The paper re-processes original training sets and prompts trained models with prefixes to test exact greedy completion, using samples normalized by sequence length and duplication count.
Results
Memorization increases with model scale, data duplication, and context length; at least 1% of The Pile is memorized by the 6 billion parameter GPT-J model.
Takeaways & Limitations
The findings suggest that larger future models will need active measures to prevent training-data memorization, while deduplication is likely a practical mitigation for data inserted once in comparatively small models.
Takeaways & Limitations
Privacy auditing likely requires prompting models with training data and large context because no known techniques identify the tail of memorized data without such conditioning.
Abstract
from arXiv · showhide
Large language models (LMs) have been shown to memorize parts of their training data, and when prompted appropriately, they will emit the memorized training data verbatim. This is undesirable because memorization violates privacy (exposing user data), degrades utility (repeated easy-to-memorize text is often low quality), and hurts fairness (some texts are memorized over others). We describe three log-linear relationships that quantify the degree to which LMs emit memorized training data. Memorization significantly grows as we increase (1) the capacity of a model, (2) the number of times an example has been duplicated, and (3) the number of tokens of context used to prompt the model. Surprisingly, we find the situation becomes more complicated when generalizing these results across model families. On the whole, we find that memorization in LMs is more prevalent than previously believed and will likely get worse as models continues to scale, at least without active mitigations.
1 INTRODUCTION
The paper addresses limited understanding of how memorization varies across model and dataset scales by systematically measuring extractable training data. It finds that memorization increases with model scale, data duplication, and prompting context, motivating mitigation for larger models.
- Motivation: Training-data extraction attacks pose a practical privacy threat because adversaries can recover sequences used to train language models.Prior attacks provide only lower bounds on the amount of memorized data.
- Motivation: At least 1% of The Pile is memorized by the 6 billion parameter GPT-J model, versus a prior lower bound of 0.00000015% for GPT-2.The comparison reflects more precise measurement using access to the original training set.
- Contribution: The paper quantifies memorization across three model families and their associated datasets using substantially more precise bounds than prior work.The method leverages access to each model’s original training set.
- Approach: The evaluation constructs training-set prefixes and tests whether models complete the remaining example verbatim.This procedure measures memorization across models, datasets, and prompt sizes.
- Findings: Within a model family, larger models memorize 2-5× more, duplicated examples are more extractable, and longer context makes extraction orders of magnitude easier.These are the three measured factors affecting memorization.
- Implications: The analysis suggests that preventing larger future models from memorizing their training datasets will require active mitigation.The paper frames this as a need for future neural language-model research.
2 RELATED WORK
Prior work established that language models can leak memorized content through extraction and related privacy attacks, but mostly studied memorization qualitatively or under narrow settings. This paper instead emphasizes quantitative worst-case measurement across scale, duplication, and context.
- Extraction attacks: Earlier studies recovered URLs, phone numbers, personal information, and injected canaries, but most were qualitative demonstrations of extractable data.These works primarily established that memorization exists rather than quantifying its maximum extent.
- Privacy attacks: Membership inference detects whether an example appears in training, whereas extraction attacks recover the example itself and therefore imply significant leakage.The paper focuses on extraction because it is especially relevant to language modeling.
- Memorization definitions: Differential privacy formalizes stability to removing one training example, but the paper discusses several alternative memorization definitions as well.Existing formulations are compared in Section 3.1.
- Study framing: The paper measures memorization in a worst-case setting by explicitly prompting models with training-data prefixes, which practical attacks may not be able to obtain.This distinguishes the study from concurrent work focused on why extraction attacks succeed.
- Scaling hypotheses: The three experiments build on prior hypotheses linking memorization to model scale, data duplication, and context length.The cited prior work motivates each scaling dimension rather than supplying the paper’s measurements.
3 METHODOLOGY
The methodology defines memorization operationally as exact greedy extraction of a training suffix from a prefix, then estimates it using carefully sampled training sequences. Sampling is normalized by sequence length and duplication to study worst-case scaling factors efficiently.
- 3.1 DEFINITION OF MEMORIZATION: A string is extractable with k tokens of context when a training prefix p causes greedy decoding to reproduce the associated suffix s exactly.The training sequence must contain the concatenation [p || s].
- 3.1 DEFINITION OF MEMORIZATION: The paper uses greedy sampling because its extraction definition is actionable and verifies that decoding choice does not significantly affect results.Other definitions may require training many models or thousands of generations per sequence.
- 3.2 SELECTION OF EVALUATION DATA: Testing every training sequence is prohibitively expensive, so the evaluation estimates memorization from statistically confident subsets of roughly 50,000 sequences.The subsets are selected to provide representative evidence while keeping inference practical.
- 3.2 SELECTION OF EVALUATION DATA: Uniform sampling estimates absolute memorization but poorly captures rare long prompts and highly duplicated examples.These properties are sparsely represented in the training distribution.
- 3.2 SELECTION OF EVALUATION DATA: The second subset is normalized by sequence length and duplication count to measure worst-case memorization on long prompts and highly duplicated data.It samples fixed numbers of sequences across length and repetition strata.
- Evaluation procedure: For each sequence, the model receives the first ℓ−50 tokens and the sequence is extractable only if it emits the next 50-token suffix exactly.The reported probability averages extractability across sequence lengths.
- Figure 1: Figure 1 compares GPT-Neo models on The Pile with GPT-2 baselines using prompts normalized by sequence length and duplication count.The panels vary model size, duplication frequency, and context length.
4 EXPERIMENTS
Experiments show that memorization increases with model scale, data duplication, and prompt context, while decoding choices and qualitative sequence properties affect what is extracted. Longer contexts reveal otherwise hidden memorization, complicating privacy auditing.
- 4 EXPERIMENTS: The experiments use GPT-Neo models ranging from 125 million to 6 billion parameters, primarily trained on The Pile, an 825GB text dataset.A 20 billion parameter variant is excluded because it uses a different training setup and tokenizer.
- 4.1 BIGGER MODELS MEMORIZE MORE: A tenfold increase in model size increases memorization by 19 percentage points, with a near-perfect log-linear fit (R2 of 99.8%).This result comes from a duplication-normalized, biased sample, so the absolute memorization level is not the focus.
- 4.1 BIGGER MODELS MEMORIZE MORE: GPT-2 correctly completes approximately 6% of evaluation examples versus 40% for the similarly sized 1.3B parameter GPT-Neo model, supporting memorization beyond generalization.GPT-2’s extracted examples were largely number sequences, repeated tokens, or common phrases.
- 4.2 REPEATED STRINGS ARE MEMORIZED MORE: Memorization follows a clear log-linear relationship with duplication: repeated strings become increasingly extractable, although even a few duplicates do not eliminate leakage.The result supports deduplication as reducing memorization without making it disappear.
- 4.3 LONGER CONTEXT DISCOVERS MORE MEMORIZATION: For the 6B model, 33% of evaluation sequences are extractable with 50 context tokens versus 65% with 450 tokens, revealing hidden memorization under longer prompts.The paper calls this the discoverability phenomenon: some memorized strings require sufficiently long context to become extractable.
- 4.4 ALTERNATE EXPERIMENTAL SETTINGS: Uniform sampling finds 7% extraction for a 1000-token sequence with GPT-J 6B, compared with 4% for GPT-Neo 125M and 2% for GPT2-XL.These results imply that at least 1% of The Pile is extractable by GPT-J 6B but not GPT-2 XL.
- 4.4 ALTERNATE EXPERIMENTAL SETTINGS: Beam search with 100 beams increases extractable memorization by just under 2 percentage points on average, with a maximum of 5.6%, and matches greedy output 45% of the time.The decoding comparison indicates only a marginal increase under beam search.
- 4.5 QUALITATIVE EXAMPLES OF MEMORIZATION: Qualitative examples show that smaller models often produce thematically relevant, locally consistent continuations that are syntactically sound but semantically incorrect.Some sequences are memorized by the 6B model despite infrequent occurrence, while some highly repeated sequences are not exactly memorized.
5 REPLICATION STUDY
Replication across model families shows that memorization trends depend on model size, duplication, training objective, dataset curation, and data distribution. Scaling patterns persist, but their magnitude and regularity vary substantially across families.
- Replication scope: Log-linear memorization trends with model size, data duplication, and context length were replicated across additional language-model families and datasets.The replication covers T5, models trained on deduplicated C4, and OPT models trained on a Pile-overlapping dataset.
- T5 masked language modeling: T5 models range from 77M to 11B parameters and use masked span reconstruction on the 806 GB C4 dataset.T5 removes a random 15% of tokens and predicts the missing spans.
- T5 results: 3B-parameter T5-XL memorizes 3.5% of sequences repeated 100 times, versus 53.6% for 2.7B GPT-Neo under the stated context condition.The comparison indicates substantially lower absolute memorization for masked models than for comparably sized causal models.
- Duplication effects: T5 sequences repeated more often tend to be easier to memorize, but duplication does not produce a monotonic scaling relationship.Whitespace-heavy and near-duplicate sequences help explain the counter-intuitive variance across repetition buckets.
- Deduplicated data: Deduplication reduces memorization below 35 repetitions from 3.6% to 1.2%, a statistically significant 3× decrease, but does not help for examples repeated at least 408 times.The authors attribute the high-repeat limitation to imperfect, non-exhaustive deduplication.
- OPT results: OPT models show nearly identical scaling trends to GPT-Neo but an orders-of-magnitude smaller effect size, with even 66B memorizing less than 125M GPT-Neo.The authors suggest either careful curation and training or modest distribution shifts may explain the difference.
6 CONCLUSION
The paper presents a comprehensive training-set reprocessing approach and concludes that memorization grows with scale and can affect a significant fraction of training data. It also identifies dataset duplication and distribution skew as important practical concerns.
- Contribution: The paper provides a comprehensive quantitative analysis of memorization by re-processing training sets to identify memorized data.This is presented as the paper’s first broad quantitative analysis of memorization in large language models.
- Generalization: Larger models are likely to learn unintended peculiarities when training distributions are skewed by duplicated sequences.The conclusion links this concern to accurately modeling training statistics without necessarily matching the desired underlying distribution.
- Privacy: Doubling model parameters produces a significantly larger extractable fraction of the dataset, and current large models memorize a significant fraction of their training data.The paper states that memorization scales log-linearly with model size.
- Scope: The OPT analysis excludes the 175B model because the authors could access and run models only up to 66B parameters.This is an explicit scope boundary for that model-family analysis.
- Mitigation: Training data inserted once is rarely memorized in the comparatively small models studied, while repeated data is more vulnerable to memorization.The authors describe deduplication as a likely practical mitigation, while noting that extraction can remain difficult to discover.
A IMPLEMENTATION DETAILS FOR DATASET CREATION
The dataset-construction procedure addresses the difficulty of sampling sequences by repetition count at web-scale. It uses a suffix array to enumerate sequences within specified repetition ranges efficiently.
- Motivation: Sampling sequences by exact repetition frequency is difficult because counting occurrences requires linear work for each query over massive datasets.The paper contrasts this practical difficulty with the straightforward conceptual construction of frequency-controlled subsets.
- Implementation: A suffix array enables efficient enumeration of all length-k sequences repeated between N and M times through a linear scan.The construction uses suffix-array positions to compare dataset sequences separated by repetition offsets.
B LONGER DOCUMENTS ARE NOT EASIER TO MEMORIZE THAN SHORTER DOCUMENTS
The study tests whether sequence length changes extractability while varying prompt context and holding repetition and model size fixed. The results indicate that longer sequences are not fundamentally easier to extract.
- Figure 5: Figure 5 measures extractability versus context-token count across sequence lengths using equally repeated sequences and the same 6B-parameter model.Each line represents the fraction extractable for a different sequence length.
- Results: Nearly overlapping extractability curves show no significant differences across sequence lengths at varying prompt lengths.The paper therefore finds no evidence that longer sequences are inherently easier to extract.
- Figure 6: Figure 6 gives examples memorized by the 6B model but not smaller models, marking correct continuations in green and novel generations in red.The examples illustrate a model-scale difference in memorization behavior.
C ALTERNATE EXPERIMENTAL SETTINGS
Alternate measurements show that memorization estimates depend on sampling, context length, and the definition of extractability. Searching the entire dataset reveals more memorized content than checking only the ground-truth continuation, but is much more computationally expensive.
- Random dataset sampling: Randomly sampled sequences reproduce the model-scale and context-length trends observed elsewhere: larger models and longer prompts reveal more memorization.The random-sampling analysis uses sequences of lengths 100, 200, 500, and 1,000, prompting with all but the final 50 tokens.
- Alternate definition of extractability: The main extractability measure checks whether generated text matches the selected example’s true suffix, providing a loose lower bound on memorization.An alternative checks whether the generated continuation appears anywhere in the training dataset.
- Alternate definition of extractability: 32.6% of outputs at 100 repetitions appeared somewhere in the dataset, compared with 15.8% matching the ground-truth continuation.The difference between the two definitions grows with repetition count and reaches 28.4% at 2,200 repetitions.
- Alternate definition of extractability: The broader dataset-search definition is excluded from the main experiments because querying an 800GB dataset for hundreds of thousands of sequences requires vastly more computation.The paper retains ground-truth suffix matching to promote reproducibility.
D TEXT MEMORIZED BY ONLY SOME MODELS
Models share some memorized sequences, but each also memorizes sequences that other models do not. Larger models have more uniquely memorized sequences, although even the smallest model has unique cases.
- Unique memorization: Table 1 counts sequences memorized by one model but not another.The table is used to compare model-specific memorization across model pairs.
- Unique memorization: Larger models have more uniquely memorized sequences, while every model has some memorization not shared by another model.The 125M model memorizes a few sequences that the 6B model does not.
E MEMORIZATION IN OPT MODELS
Figure 7 measures memorization in OPT models as model scale and training-set repetition vary, using 100-token prompts. It reports the fraction of sampled sequences extracted along both axes.
- Model scale: Figure 7a plots the fraction of sequences extracted against OPT model scale using 100-token prompts.The panel compares extraction rates as model size changes.
- Data duplication: Figure 7b plots the fraction of sequences extracted against the number of times each sequence is repeated in training.The panel varies repetition count while keeping the prompt length at 100 tokens.
F EXAMPLES OF MEMORIZED TEXTS
The examples illustrate that memorization varies by model, duplication, sequence structure, and dataset overlap. Some sequences are memorized universally or by only one model, while others are repeated yet not memorized.
- Experimental setup: Fifty-token prompts and generations are used to show examples memorized by different models under specific criteria.The examples are randomly sampled from texts satisfying each memorization criterion.
- Shared memorization: Some sequences are memorized by every model, with all generated 50-token continuations matching the ground-truth continuation.Figure 8 presents these universally memorized examples.
- Model-specific memorization: The 6B model memorizes examples that smaller models do not, while smaller-model generations can remain topical and locally consistent yet semantically incorrect.Figures 9–11 provide additional examples of this pattern.
- Model-specific memorization: Only 35 examples are memorized by the smallest model compared with 2,860 memorized only by the largest model.One smallest-model example contains a formatting-cutoff typo that the model memorized.
- Duplication and sequence structure: Some memorized sequences are infrequently duplicated, whereas thousands of repetitions do not guarantee memorization when variants differ in semantically unimportant characters.Examples include boilerplate or sequential text, code comments, non-English characters, template values, error messages, symbols, and repeated dumpster documents.
- Dataset overlap: GPT2-XL examples from The Pile include natural sequential structure and apparent overlap between the model’s training data and The Pile.Figure 15 presents these examples.