Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy

1 Introduction

• Filtering generations that copy long training-set strings is a natural defense, as illustrated by GitHub Copilot’s option to block suggestions matching public code.Source: §1 · p. 1
• Copilot’s verbatim filter is easily circumvented by plausible style-transfer prompts, and GPT-3 is also vulnerable to extraction attacks (Brown et al., 2020).Translating variable names from English to French can yield completely memorized examples with the style transfer applied.Source: §1 · p. 1
• MEMFREE decoding guarantees prevention of verbatim memorization and scales to training sets of hundreds of gigabytes of text.At each generation step, it checks whether the next token would create a training-set n-gram and substitutes an alternative token when needed.Source: §1 · pp. 1–2
• Even without prompt changes, models can bypass the filter by producing close paraphrases through spelling, punctuation, whitespace, or synonym changes (Gao et al., 2020).A human may perceive these outputs as nearly identical to the training continuation despite the lack of verbatim matching.Source: §1 · p. 2
• The paper argues that preventing verbatim copying is insufficient to protect against training-data leakage, so definitions should also capture high semantic similarity.It suggests that the fraction of datasets large language models memorize may be larger than prior work reports (Carlini et al., 2022).Source: §1 · p. 2

2 Background

• Transformer language models scaled from millions to hundreds of billions of parameters, alongside dataset growth, with much training data coming from internet crawls (Vaswani et al., 2017; Brown et al., 2020; Chowdhery et al., 2022; Zhang et al., 2022; Lee et al., 2021; Raffel et al., 2020; Gao et al., 2020).Source: §2 · p. 2
• Prior work links memorization of sensitive training data to privacy leakage through model interactions, while this paper studies English-text and code models without distinguishing problematic from non-problematic memorization.Examples of prior concerns include memorized phone numbers and addresses.Source: §2 · p. 2
• Eidetic memorization defines a string as memorized when some prompt makes the model output that string and the string occurs in training data (Carlini et al., 2021).Related formulations include exact memorization (Tirumala et al., 2022) and perfect memorization (Kandpal et al., 2022).Source: §2 · p. 2
• Differential-privacy-based definitions use probabilistic leakage measures, including the probability of verbatim canary generation and counterfactual memorization (Zhao et al., 2022; Stock et al., 2022; Zhang et al., 2021).Source: §2 · pp. 2–3
• An edit-distance definition treats an output as memorized when it lies within a chosen distance of the training continuation, but the paper says this may miss style-transfer cases (Lee et al., 2021).Source: §2 · p. 3
• Deduplication removes repeated content but cannot guarantee that individual examples are not memorized, motivating inference-time defenses; differential privacy can yield worse generative models (Lee et al., 2021; Kandpal et al., 2022; Carlini et al., 2022; Abadi et al., 2016; Anil et al., 2021).Source: §2 · p. 3

3 Preventing Models from Emitting Verbatim Training Data

• MEMFREE decoding checks n-grams during generation, modifying individual memorized pieces while preserving novel substrings, unlike sequence-level retroactive censoring.The finer-grained approach is used to test whether training data can still leak when verbatim subsequences are blocked.Source: §3 · p. 3

3.1 MEMFREE Decoding Details

• At each step, MEMFREE checks whether the proposed token completes a training-set n-gram and removes its probability mass before sampling.The altered token distribution can be used with sampling strategies such as top-k.Source: §3.1 · pp. 3–4

3.2 Querying the Training Set Efficiently

• The implementation stores duplicated training n-grams and uses a Bloom filter that avoids false negatives but can reject some non-memorized content.These optimizations make training-set membership checks more efficient; false positives are the trade-off described for Bloom filters.Source: §3.2 · p. 4

3.3 Measuring Approximate Memorization

• Approximate memorization is assessed using BLEU and length-normalized character-level Levenshtein similarity, with BLEU > 0.75 used for a binary label.The threshold was chosen by qualitatively inspecting examples.Source: §3.3 · pp. 4–5
• In the worst case, approximate memorization yields a factor-of-two increase over verbatim counts in the repeated prefix-extraction experiment (Carlini et al., 2022).Figure 3 compares approximate memorization at BLEU > 0.75 with exact memorization for undefended generation.Source: Figure 3 · p. 5, §3.3 · p. 5
• The BLEU-threshold definition may overcount or undercount approximate memorization, and identified cases may not all seem memorized to human judges.Using this definition directly in defenses could introduce false positives.Source: §3.3 · p. 5

4.1 Evading Copilot’s Memorization Filter

• Copilot’s filter fails to prevent training-data leakage in many settings, including prompts that apply style transfers to produce memorized but non-verbatim outputs.Examples include Python-style comments and French variable names, which let the model return the Fast Inverse Square Root code in altered form.Source: §4.1 · p. 6
• Copilot can also passively evade its filter when prompted with highly duplicated training text, using capitalization changes, small errors, or whitespace changes.The authors hypothesize that whitespace evasion may occur when the model replaces tabs with spaces.Source: §4.1 · p. 6

4.2 English Language Models

• GPT-3 and PaLM often generate memorized continuations from style-transferred prompts; uppercase prompts were least likely to do so.The larger PaLM model was more capable of memorization than the smaller one, and Davinci V2 was more susceptible to double spaces than Original.Source: Figure 4 · p. 6, §4.2 · pp. 6–7
• The authors tested four English-language models with 100-word prompts from likely memorized documents, using original, doubled-space, lowercase, and uppercase variants.The documents included open-source licenses, famous speeches and monologues, novel openings, and song lyrics.Source: §4.2 · p. 6

5.1 Experimental Design

• Experiments use GPT-Neo models from 125M to 6B parameters, with arg max decoding and a Bloom filter over Pile 10-grams occurring at least 10 times.Each test string supplies a 50-token prompt followed by a generated 50-token continuation; the filter's n=10 choice can also block common phrases.Source: §5.1 · p. 7
• Figure 6 compares ground-truth similarity for MEMFREE generations on the y-axis against undefended generations on the x-axis; diagonal points were not memorized.The study evaluates 5,000 prompts.Source: Figure 6 · p. 7

5.2 Reduction in Memorization

• MEMFREE significantly reduces similarity to ground-truth continuations, including for larger models and highly duplicated examples.It has little effect when undefended generations already have low similarity; BLEU remains near-flat around 0.6 across model sizes.Source: §5.2 · p. 7, Figure 6 · p. 7, Figure 7 · p. 8

5.3 Failures in Preventing Memorization

• MEMFREE can still leave generations substantially copied in human judgment when it changes only a few tokens.A second failure occurs when filtering raises similarity; 0.16% of samples had a similarity increase above 0.1.Source: §5.3 · pp. 7–8

• Style-transfer prompts and similar-but-not-verbatim continuations expose forms of memorization that verbatim definitions miss.The paper argues that memorization prevention should capture paraphrased memorization as well as verbatim matches.Source: §6 · p. 8
• Exhaustively anticipating styles for defenses is an innumerable problem that becomes harder as models grow more powerful.This creates a challenge for defining approximate memorization because new cases must remain discoverable, and criteria depend on domain.Source: §6 · p. 8
• Approximate-memorization standards are domain-dependent: English, code, and other languages require different considerations.The paper focuses on language models trained to output English and code.Source: §6 · p. 8
• The paper connects approximate text memorization to image-generation work that allows perceptually similar reconstructions despite high standard-metric distance.Plagiarism detection also considers fuzzy text and code clones, including semantics-preserving paraphrases.Source: §6 · p. 8
• The 10-gram overlap approach is expected not to significantly affect utility, but the interplay between memorization and knowledge remains future work.Source: §6 · p. 9
• Verbatim definitions remain useful for efficient detection of obvious memorization, but further relaxations are needed to measure language-model privacy concerns.The paper notes that identifying memorization cannot rely only on verbatim reproductions and may complicate understanding memorization and generalization.Source: §6 · p. 9

7 Ethics & Broader Impact

• A generation that differs from training data verbatim may still be a minimal transformation of memorized content.The authors caution that superficial difference does not establish that an example is novel.Source: §7 · p. 9
• Privacy evaluation should not rely solely on checking verbatim matches, and practitioners should account for this failure mode in output post-processing defenses.The authors identify GitHub Copilot as the only deployed system they knew to be affected by their analysis.Source: §7 · p. 9
• The study focuses on public datasets that researchers had extensively studied to minimize harm, but the authors note that public data can still raise privacy concerns.They cite Brown et al. (2022) on other privacy considerations for models trained on public datasets.Source: §7 · p. 9

Contributions

• The contributors describe work spanning the MEMFREE idea and implementation, experiments, qualitative analyses, figures, and paper writing.The listed contributions include style-transfer prompt design and Copilot experiments.Source: Contributions · p. 9

A GitHub Copilot

• GitHub Copilot describes an optional filter that withholds suggestions matching or nearly matching roughly 150 characters of public GitHub code, ignoring whitespace.The FAQ says users can turn the filter on or off during setup.Source: A GitHub Copilot · p. 12

B.1 Formal Procedure

• MEMFREE greedily generates tokens while suppressing logits for choices that would create training-set n-grams; experiments use arg max decoding.The formal procedure takes a language model, prefix, generation length, and training set as inputs.Source: §B.1 · p. 12

B.2 Choice of n-gram length

• The n-gram length trades off memorization-check granularity against Bloom-filter storage; shorter n-grams catch shorter sequences but may include common phrases.The filter’s storage depends on the unique n-gram count and target false-positive rate; the authors report using seven hash functions.Source: §B.2 · p. 12

B.3 Choice of Minimum Frequency

• The authors choose n=10 as a qualitative compromise, while noting that lower n can worsen standard benchmark performance by preventing more true answers.They report that lower n prevents common words or phrases of ten or more tokens, with qualitatively low impact; Bloom-filter hit patterns vary across generations and token positions.Source: §B.3 · p. 13, §B.2 · p. 13, Figure 9 · p. 13

B.4 Python Implementation

• The appendix identifies Figure 10 as a Python implementation of MEMFREE using the HuggingFace Transformers API.Source: §B.4 · p. 13

B.5 Impact of MEMFREE on Downstream Task Performance

• For most benchmark tasks, only slightly more than 1% of 10-grams were in the Bloom filter, but over 86% of BillSum and Arxiv validation examples had a matching 10-gram.The authors attribute greater impact on non-English tasks to GPT-Neo’s English-built vocabulary, which splits non-English phrases into more tokens; Table 1 describes the analysis as a worst-case assessment.Source: §B.5 · p. 13, Table 1 · p. 15

B.6 Performance of MEMFREE

• More than 12,000 memorized documents fell to four verbatim outputs with MEMFREE on GPT-Neo 6B, a reduction of over 3000x.The four remaining outputs were repeated fewer than ten times in training and were not added to the Bloom filter; the authors report no observable utility impact under their studied constraints.Source: §B.6 · p. 14

B.7 Bloom Filter Statistics

• MEMFREE took 49.8 milliseconds for 10,000 Bloom queries on one CPU core, and most generations needed fewer than five token alterations.The authors report a mean of 42.1 queries per generation and frequent Bloom hits at the first and tenth tokens after the prompt.Source: §B.7 · p. 16, Figure 11 · p. 16

C.1 Similarity Metrics Implementations

• The authors measured approximate memorization using BLEU score and character-level edit distance, computing BLEU as an equal average of BLEU-1 through BLEU-4.Source: §C.1 · p. 16

C.2 BLEU Score Threshold Selection

• A BLEU score of at least 0.75 was treated as substantial memorization, a deliberately high threshold intended to provide a reliable lower bound.The authors report that examples above the threshold showed unambiguous memorization, while some examples below it also appeared memorized.Source: §C.2 · p. 16, Figure 12 · p. 17

D.1 Prompt Selection Process

• For each document, the authors used its first 100 words as a prompt and compared the first 50 generated words with the true continuation.They used this approach to support fair comparisons across style transfers that tokenize differently; Figure 13 reports approximate memorization under style-transfer prompting in two GPT-3 versions and two PaLM sizes.Source: §D.1 · p. 17, Figure 13 · p. 18, Table 2 · p. 19

E Experiments with MEMFREE and Other Model Families

• Pythia exhibits more approximate memorization than GPT-Neo, and MEMFREE reduces it but is slightly less effective than on GPT-Neo.The comparison includes base and deduplicated Pythia models, including the 6.9B-parameter model.Source: E Experiments with MEMFREE and Other… · p. 20, Figure 14 · p. 20, Figure 15 · p. 20

F Qualitative Examples that Bypass Copilot’s Filter

• Trivial prompt modifications bypass Copilot’s output filter, which can then emit memorized but non-verbatim content.The examples were generated with “block suggestions matching public code” enabled.Source: F Qualitative Examples that Bypass… · p. 20, Figure 16 · p. 21

G Qualitative Examples of Memorization Despite MEMFREE

• MEMFREE examples also include altered dialogue and prose, with short inserted or substituted spans embedded in otherwise familiar passages.The examples mark rejected tokens and accepted replacements; the surrounding generated text can remain close to the source.Source: G Qualitative Examples of Memorization… · pp. 22–24
• Examples span legal text, reviews, and literary passages, with local substitutions that leave much of the original wording intact.The displayed cases include edits to words, capitalization, and phrasing across several kinds of text.Source: G Qualitative Examples of Memorization… · pp. 22–24
• Most sampled MEMFREE generations with BLEU above 0.9 are repetitive and/or lists of numbers.Figure 17 marks deleted text in red and added text in green.Source: Figure 17 · p. 25

H Author Ordering Algorithm

• Figure 18 presents the author-ordering algorithm.The supplied figure caption identifies it as the author ordering algorithm.Source: Figure 18 · p. 26