Source-linked AI summary
Counterfactual Memorization in Neural Language Models
Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, Nicholas Carlini
TL;DR
Neural language models may memorize sensitive or copyrighted training information, while existing criteria often conflate rare memorization with common repeated text. The paper introduces counterfactual memorization and influence to isolate document-specific effects and trace their impact on validation and generated text. Across standard corpora, it finds rare memorized examples and substantial prediction changes associated with highly memorized training examples.
Problem
Existing memorization criteria often capture common repeated phrases, public knowledge, and templated text, making rare memorization difficult to distinguish.
Method
The paper compares model predictions and performance when individual training examples are included versus omitted, then extends this analysis to counterfactual influence.
Results
Experiments on standard text datasets find rare memorized examples, differing domain profiles, and substantially different validation and generated-text predictions when highly memorized examples are absent.
Takeaways & Limitations
Counterfactual memorization provides a way to focus on rare, specific information and trace training examples that contribute to test-time predictions.
Takeaways & Limitations
The study mainly evaluates English datasets and uses models and training sets orders of magnitude smaller than modern systems.
Abstract
from arXiv · showhide
Modern neural language models that are widely used in various NLP tasks risk memorizing sensitive information from their training data. Understanding this memorization is important in real world applications and also from a learning-theoretical perspective. An open question in previous studies of language model memorization is how to filter out "common" memorization. In fact, most memorization criteria strongly correlate with the number of occurrences in the training set, capturing memorized familiar phrases, public knowledge, templated texts, or other repeated data. We formulate a notion of counterfactual memorization which characterizes how a model's predictions change if a particular document is omitted during training. We identify and study counterfactually-memorized training examples in standard text datasets. We estimate the influence of each memorized training example on the validation set and on generated texts, showing how this can provide direct evidence of the source of memorization at test time.
1 Introduction
The paper addresses the difficulty of distinguishing rare, specific memorization from common memorization in neural language models. It proposes counterfactual memorization, linking predictions to whether individual training examples are present, and studies its effects on validation and generated text.
- Language models can memorize sensitive information and copyrighted material, creating practical concerns despite their broad NLP utility.
- Prior extraction methods mainly identify common, frequently occurring strings, including near-duplicates prevalent in standard corpora.
- Counterfactual memorization measures how a model’s prediction changes when an individual training example is omitted.
- The measure reduces the effect of common text because removing one instance among many duplicates has only a minor predictive impact.
- Counterfactual influence traces which memorized training examples significantly contribute to predictions on validation and generated texts.
- Experiments across standard text datasets find rare memorized examples, differing domain memorization profiles, and an inverse relationship between duplicates and counterfactual memorization.
2 Related Work
Related work shows that memorization studies often focus on extracting sensitive or synthetic data, while common phrases and near-duplicates complicate identifying rare information. This paper instead analyzes individual-example behavior through a counterfactual causal perspective.
- Earlier studies extracted sensitive training data such as phone numbers and usernames, often using memorization attacks.
- Near-duplicate examples and common phrases contribute substantially to memorized content in standard text corpora.
- Prior investigations used heuristics to distinguish common memorization from rare or private information, motivating this paper’s principled perspective.
- Connections to cross-validation, leave-one-out stability, and differential privacy frame model comparisons based on holding out individual examples.
- Unlike most previous work measuring memorization overall, this paper characterizes the behavior of individual examples.
- The paper extends influence analysis to assess how memorized examples affect generated texts, while prior influence estimation was computationally expensive and fragile.
- It directly measures counterfactual influence by comparing model behavior with and without a particular training example.
3 Counterfactual Memorization
The paper defines counterfactual memorization as the performance difference between models trained with and without a particular document. It estimates this difference by sampling models trained on subsets and comparing their predictions for the document.
- Counterfactual memorization quantifies rare details by extending Feldman’s label-memorization formulation to unsupervised neural language modeling.
- The measure subtracts expected performance without x from expected performance with x across sampled training subsets and algorithmic randomness.
- The performance measure M is per-token accuracy, evaluating whether the model predicts the next token correctly.
- Empirically, the authors train m models on independently sampled equal-size subsets, then average performance for models containing x and models excluding x.
- A large difference between the two groups identifies an example as counterfactually memorized.
- The same m trained models can estimate scores for all examples, avoiding separate retraining for each x.
4 Analyzing Counterfactual Memorization
The paper analyzes counterfactual memorization across three datasets, web domains, example difficulty, and training duration. Memorization varies with text characteristics and grows as training progresses, while domain frequency alone does not determine its profile.
- Distribution of Memorization: Across RealNews, C4, and Wiki40B:en, highly memorized examples often contain unconventional formatting, multilingual text, or structured content.Templated documents with many near-duplicate copies generally have low counterfactual memorization.
- Distribution of Memorization: Easy examples tend to have low memorization, while peak memorization occurs at intermediate simplicity rather than following a simple linear relationship.Very hard examples also score low because even models trained on them learn them poorly.
- Web-Domain Profiles: Large web domains show less variance in their 95th-percentile memorization, whereas smaller domains exhibit more diverse memorization profiles.In RealNews, reuters.com is the largest domain but has low average memorization; multilingual and structured-text domains show higher scores.
- Estimation Procedure: Counterfactual memorization is estimated by aggregating statistics from many independently trained models on random training subsets, avoiding costly leave-one-out retraining.This design addresses the low signal-to-noise ratio of comparing only two stochastic training runs.
- Impact of Number of Training Epochs: 59% of RealNews examples had consistently increasing memorization across training, while 85% never exceeded a memorization score of 0.2.Overall memorization increased with epochs, and only 0.5% of examples remained below 0.1 throughout.
5 Duplicate Text and Memorization
The paper compares counterfactual memorization with duplicate-sensitive generation-time measures. Counterfactual memorization decreases as near-duplicates increase, making it more useful for distinguishing rare memorization from repeated text.
- Duplicate Text and Memorization: Among approximately 38,000 near-duplicate RealNews examples, counterfactual memorization correlated negatively with duplicate count at Pearson r = -0.39.The analysis identified near-duplicates using MinHash and normalized edit similarity greater than 0.7.
- Duplicate Text and Memorization: Examples with many near-duplicates have smaller counterfactual memorization scores because duplicates may also appear in models trained without the target example.Counterfactual memorization primarily differentiates examples with only a few duplicates.
- Duplicate Text and Memorization: Generation-time memorization measures show a strong positive association with occurrence count, unlike counterfactual memorization.This contrast motivates using counterfactual memorization to separate common repeated text from rarer memorized examples.
6 From Memorization to Influence
The paper extends counterfactual memorization to counterfactual influence, measuring how individual training examples affect predictions on validation and generated texts. This enables attribution of test-time predictions to memorized training examples.
- Counterfactual Influence: Counterfactual influence asks whether a particular training example strongly changes predictions on another example, including validation, test, generated, or training examples.It compares model behavior with and without the training example, using subsets of the training data and training randomness.
- Counterfactual Influence: Counterfactual influence can trace the source of information in a model prediction by locating training examples that contributed significantly to it.Counterfactual memorization is the self-influence case, mem(x) = infl(x ⇒ x).
- Influence on Validation Examples: Most validation examples receive close-to-zero influence from individual training examples, while examples containing similar rare information can receive large influence from highly memorized examples.The analysis compares influence distributions for randomly selected and highly or intermediately influenced validation examples.
- Influence on Validation Examples: Larger validation-set influence generally requires larger memorization scores, although highly memorized examples do not necessarily have large maximum influence.Low influence may reflect low-quality text or the absence of a relevant document in the smaller validation set.
- Influence on Validation Examples: The highest-influence RealNews train-validation pairs are nearly identical, while intermediate pairs often report the same events and low-influence pairs commonly reflect templating.These patterns provide concrete examples of how influence relates to textual overlap and shared sources.
- Interpretation: Influence can identify exactly which training example provided information for predictions that rely on counterfactual memorization.The authors connect this attribution pattern with observed near-duplicated training-validation document pairs in Internet-crawled corpora.
- Influence on Generated Texts: Influence estimation also applies to generated text, but the analyzed Grover-Mega generations contained fewer highly influenced examples than the validation set.The analysis used 24,576 public generations and only the first 2M RealNews training examples, limiting coverage of potentially influential examples.
7 Summary and Discussion
The paper presents counterfactual memorization as a way to isolate rare information while filtering common memorization, and finds such memorization across three text corpora. It also shows that memorized examples can substantially affect validation and generated predictions, while noting important scope limitations.
- Summary: Counterfactual memorization measures rare, specific information whose presence or absence substantially changes model predictions, while ignoring common knowledge shared across documents.The paper contrasts specific event descriptions with general facts such as “Paris is a city in France.”
- Summary: Counterfactual memorization was found in all three evaluated text corpora, with substantially different memorization profiles across Internet-crawled domains.The experiments identify memorized examples while comparing source-specific profiles.
- Influence at Test Time: For validation examples and generated texts, predictions could differ drastically depending on whether a highly memorized training example was present.Counterfactual influence provides a way to connect these prediction changes to particular training examples.
- Limitations: The study mainly evaluates English datasets, and multilingual or more structured domains such as programming languages may exhibit new memorization patterns.The authors expect similar characterization for other natural languages but do not establish it here.
- Limitations: The models and datasets are orders of magnitude smaller than modern systems, and larger models may reveal stronger influences between semantically similar but textually different documents.The paper identifies scaling and paraphrased-document datasets as needed for further investigation.
- Comparison with Generation-Time Memorization: The method handles near-duplicates automatically because removing one copy has little effect when other similar copies remain in the training set.This distinguishes its target from generation-time memorization without making the two notions opposites.
C The Impact of Data Deduplication on Memorization
The deduplication analysis compares original and approximately deduplicated C4 data using memorization distributions and ranking stability. It reports little overall distributional change, while highly counterfactually memorized examples are mostly unaffected by text deduplication.
- Dataset Comparison: The original C4 and C4-NEARDUP datasets showed no significant difference in the distribution of counterfactual memorization.C4-NEARDUP removes approximately 3% of training data using conservative approximate document matching.
- Influence and Memorization: Figure 6 plots each training example’s memorization score against its maximum validation-set influence, with log-scale histograms emphasizing distribution tails.The figure is the full version of the joint distribution shown in Figure 4b.
- Memorization and Simplicity: Figure 7 displays the joint distribution of memorization and simplicity using log-scale histograms to make the tails more visible.The passage identifies the plotted variables but does not state a specific comparison outcome.
- Ranking Stability: At least 192 models reproduce the full 400-model memorization ranking with Spearman’s R of 0.992 on every dataset.The ranking changes very little as additional models are included.
- Score Stability: Memorization-score variance decreases significantly as the number of models increases across all three datasets.This supports using more models to stabilize score estimates.
- High-Memorization Tail: Examples with high counterfactual memorization were mostly unaffected by text deduplication.Edit-distance measurements place many examples near the deduplication threshold, but the high-memorization tail remains largely unchanged.
D Variance of Memorization Scores
The study evaluates how many independently trained models are needed for stable memorization estimates and how score variance changes with the subset size.
- At 192 models, Spearman’s R reaches at least 99.2% for every dataset relative to the full 400-model set.Increasing the subset size further appears to provide diminishing returns.
- Memorization-score variance decreases as the number of models in each partition increases.The authors estimate variance using the standard deviation of each sample’s scores across model partitions.
- The analysis also measures maximum influence on Grover-Mega generated RealNews examples using a histogram.The generations are evaluated against RealNews training examples.
F Miscellaneous Experiment Details
The experiments use open-source tooling, publicly released Grover generations, and a computationally intensive training setup involving hundreds of models.
- The influence analysis uses publicly released Grover model generations and implements experiments with JAX and Flax.The libraries and Grover code repository are released under the Apache-2.0 license.
- The study runs its experiments on an internal cluster, with most computation consumed by model training.
- A histogram summarizes maximum influence from RealNews training examples on Grover-Mega generations.The Grover-Mega sampling parameter is p=0.96.
- A figure compares hash-based subset sampling with numpy.random.choice.
- The full analysis trains 400 different models for each of the three datasets analyzed.The experiments use a standard transformer-based language-model training setup on machines with one or more GPUs.
G Subsampling Procedure
The paper combines hash-based random subset sampling with per-token accuracy and edit similarity to estimate memorization and compare example pairs across influence levels.
- G Subsampling Procedure: The hash-based sampler is designed to be efficiently evaluated as a predicate for each example, avoiding list-based subset loading limitations.The implementation supports TensorFlow and NumPy systems.
- G Subsampling Procedure: The sampler maps example indices into hash buckets and selects one bucket to approximate a random subset of a specified size.Different seeds provide independent subsets, with N total examples and expected subset size n<N.
- G Subsampling Procedure: With r=0.25, each point appears in about 25% of models on average, while finite sampling causes variation around that value.The sampler consistently selects close to the target number of points with small variance.
- G Subsampling Procedure: Counterfactual memorization is computed from per-token accuracy, the fraction of correct highest-scoring next-token predictions.The unscaled version is used because its [0, 1] range is straightforward to interpret; scaled and unscaled versions correlate strongly.
- G Subsampling Procedure: Edit similarity compares token-level differences between two sequences, while Table 3 and the accompanying examples organize pairs by influence level.The train-generation patterns generally resemble train-validation patterns, often through templating.
L Example Pairs Sampled at Different Level of Influence
The paper presents train-validation and train-generation pairs sampled across influence and memorization ranges, illustrating examples from RealNews, C4, Wiki40B:en, and Grover generations.
- L Example Pairs Sampled at Different Level of Influence: RealNews train-validation pairs are shown from high through low influence, with the validation example followed by its corresponding training example and textual differences.The differences are visualized using difflib-generated highlighting.
- L Example Pairs Sampled at Different Level of Influence: C4 and Wiki40B:en also include train-validation pairs spanning high-to-intermediate and intermediate-to-low influence ranges.
- L Example Pairs Sampled at Different Level of Influence: Train-generation pairs connect RealNews training examples with Grover generations across high-to-intermediate, intermediate-to-low, and low influence levels.The generations come from publicly released Grover-Mega outputs.