Source-linked AI summary
Measuring Forgetting of Memorized Training Examples
Matthew Jagielski, Om Thakkar, Florian Tramèr, Daphne Ippolito, Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, Chiyuan Zhang
TL;DR
The paper asks whether forgetting reduces the privacy risk created by memorization of individual training examples. It measures forgetting through privacy attacks and finds that standard image, speech, and language models empirically forget early examples over time, while non-convex or deterministic settings can retain them.
Problem
The paper investigates whether training-time forgetting counteracts memorization and privacy risks for individual examples, especially when examples are seen at widely varying frequencies.
Method
The paper measures forgetting by tracking how privacy attacks detect injected or previously used examples as training continues, including instance-specific and worst-case analyses.
Results
Standard image, speech, and language models empirically forget examples over time, whereas non-convex models can retain them in worst-case settings and deterministic training does not forget.
Takeaways & Limitations
Early examples in large-scale training may gain passive privacy benefits, while the most recently seen examples may be the most vulnerable to privacy attacks.
Takeaways & Limitations
The empirical attacks are heuristic and cannot prove that forgetting occurred against all possible attacks.
Abstract
from arXiv · showhide
Machine learning models exhibit two seemingly contradictory phenomena: training data memorization, and various forms of forgetting. In memorization, models overfit specific training examples and become susceptible to privacy attacks. In forgetting, examples which appeared early in training are forgotten by the end. In this work, we connect these phenomena. We propose a technique to measure to what extent models "forget" the specifics of training examples, becoming less susceptible to privacy attacks on examples they have not seen recently. We show that, while non-convex models can memorize data forever in the worst-case, standard image, speech, and language models empirically do forget examples over time. We identify nondeterminism as a potential explanation, showing that deterministically trained models do not forget. Our results suggest that examples seen early when training with extremely large datasets - for instance those examples used to pre-train a model - may observe privacy benefits at the expense of examples seen later.
1 Introduction
The paper connects memorization-driven privacy risk with forgetting during training, asking whether early examples remain detectable or are erased by later updates. It develops a measurement approach and reports forgetting across vision, speech, and language settings.
- Motivation: Memorization makes models vulnerable to membership inference and training-data extraction attacks.Existing privacy techniques modify training to limit leakage from memorization.
- Research question: The paper tests whether memorization persists longer than forgetting or whether subsequent weight updates erase traces of early examples.These hypotheses concern examples seen early and not repeated later in training.
- Motivation: The privacy impact of forgetting matters most when examples vary greatly in how often they are seen during training.This variation is especially relevant for extremely large training sets and few-epoch training.
- Approach and findings: The authors measure whether and how quickly individual examples are forgotten using state-of-the-art membership inference attacks.They report that early, non-repeated examples in deep vision and speech networks are forgotten and identify factors affecting forgetting speed.
- Mechanisms: The study also examines cases where forgetting fails, including non-convex models and deterministic training, and analyzes stochasticity as a potential explanation.It combines empirical results with a mean-estimation analysis of forgetting under stochastic gradient descent.
2 Background and Related Work
The paper distinguishes passive privacy from actively modified protections and defines forgetting through the declining detectability of individual training examples. It situates this measure against privacy attacks, catastrophic forgetting, and prior work on large-scale training.
- Privacy definitions: Differential privacy and machine unlearning actively modify training or model behavior to protect individual examples.The paper contrasts these approaches with privacy arising naturally from training dynamics.
- Privacy definitions: The paper defines forgetting as a passive privacy effect in which information about an individual training point becomes harder to extract over training.This definition focuses on the decay of extractability rather than an intervention.
- Related work: Unlike catastrophic forgetting, this work measures whether a specific training example remains detectable rather than whether performance drops on an entire task or sub-distribution.The distinction matters because accuracy changes do not necessarily reveal whether memorized individual examples remain identifiable.
- Privacy attacks: Membership inference tests whether an example was in training, while training-data extraction attempts to recover training data from the model.Both attacks target information about individual examples rather than aggregate properties of the training set.
- Privacy attacks: Canary extraction measures extraction success with exposure, which reflects the reduced uncertainty in guessing an injected secret.Exposure increases when the true canary ranks especially well among possible secrets.
- Related work: Prior work found forgetting relevant to large-model training, and this paper measures how quickly it occurs across multiple domains.The authors report that forgetting happens quickly enough to matter for privacy in large datasets.
3 Forgetting
The paper operationalizes forgetting as the decline of privacy-attack success after an example is last used. Its procedure targets worst-case forgetting by comparing models with and without injected examples during continued training.
- Defining forgetting: Forgetting is measured by evaluating the success rate of a privacy attack against a training example.The definition treats an example as forgotten after k steps when attack success is at most α.
- Defining forgetting: With membership inference and α = 50%, an example is forgotten when the attack can no longer distinguish models trained with or without it.The threshold corresponds to random guessing.
- Defining forgetting: The definition allows examples that were never memorized to count as forgotten immediately, but the analysis focuses on vulnerable examples initially memorized by the model.This aligns the measure with privacy risk rather than general learning performance.
- Scope and comparison: The measure is weaker than differential privacy and machine unlearning because it tests known attacks after some delay rather than requiring all attacks to fail immediately or permanently.Differential privacy requires poor attack performance at k = 0, while unlearning requires all possible attacks to fail.
- Measurement procedure: To target worst-case forgetting, the procedure injects examples into one model, continues training both models only on the ordinary dataset, and repeatedly tests attack performance.The comparison reveals how detectability changes after injected examples are removed.
- Measurement procedure: POISON adds injected examples to the dataset before removing them, whereas INJECT inserts them at a chosen training step for large datasets with few passes.The two strategies model fine-tuning and extremely large-scale training, respectively.
4 Empirically Measuring Forgetting
The study empirically measures whether large neural networks forget individual training examples and finds forgetting across image, speech, and language settings. Forgetting speed depends especially on repetition and example difficulty, while other training parameters have smaller or variable effects.
- The experiments measure example forgetting across ImageNet, LibriSpeech, and C4 using membership inference for classification and canary extraction for generative models.The study uses INJECT for ImageNet and language models, and POISON for LibriSpeech.
- ImageNet membership-inference precision decays to roughly 65% after 10 epochs following injection, demonstrating forgetting in this setting.The experiments inject examples repeated 10 times at epoch 50 and track attack precision over subsequent training.
- LibriSpeech canary exposure falls from 14.0 to 5.6 within 10,000 steps after removal, making the canaries 330× harder to guess on average.After another 40,000 steps, exposure drops to roughly 2, near the baseline for unseen canaries.
- Language models forget canaries across repeat counts, reaching exposure 2 and decaying to the random-guessing baseline of 1 within at most 10,000 steps.The observed forgetting occurs for the tested INJECT settings.
- Repetition and example hardness matter most for forgetting speed, while recency, learning rate, momentum, and model size generally have smaller effects.The study reports that parameter choices can still affect the quantitative rate of forgetting.
- Repetitions/Hardness: Repeated examples and difficult examples are harder to forget, with repetition producing higher attack success and slower declines in attack performance.On ImageNet, precision takes roughly 20,000 steps to fall below 70% with 10 repetitions, compared with 200 steps for one repetition.
5 Understanding Forgetting
Forgetting can fail in non-convex or fully deterministic settings, but unknown randomness from randomized data sampling can make memorized examples increasingly difficult to distinguish over time.
- 5.2 Deterministic SGD Does Not Forget: Unknown nondeterminism is required for forgetting in the paper’s analysis, whereas deterministic training does not forget.The paper supports this empirically and formally for SGD on mean estimation.
- 5.1 Non-convexity Can Prevent Forgetting: Non-convex k-means can retain an outlier’s influence indefinitely because a single example can force a persistent local optimum.Adding more data does not cause the outlier’s influence to be forgotten.
- 5.1 Non-convexity Can Prevent Forgetting: 97% membership-inference accuracy and perfect precision occur for the constructed k-means outlier, demonstrating privacy leakage without forgetting.
- 5.2 Deterministic SGD Does Not Forget: With exact knowledge of the training set and batch order, an adversary can simulate deterministic training and continue distinguishing examples long after they were observed.A logistic-regression experiment achieves 97% membership-inference accuracy for examples seen in epoch 5 even after 95 epochs.
- 5.3 Random Sampling Leads To Forgetting: Randomized data sampling can cause forgetting, making membership-inference attacks harder as training progresses.The mean-estimation analysis links smaller Rényi divergence to weaker membership-inference and reconstruction attacks.
- 5.3 Random Sampling Leads To Forgetting: Forgetting improves with training progress but is slower for more worst-case or heavily repeated examples.In the theorem, forgetting never happens when v⊤Σ^-1v = 0.
6 Conclusion
The paper presents forgetting as passive privacy amplification for early training examples, while emphasizing that recent examples remain more vulnerable. It also identifies heuristic attack-based measurement as a central limitation.
- Early training examples may become more robust to privacy attacks through passive forgetting.
- Large datasets and randomized sampling favor forgetting when examples remain unseen for thousands of steps before model release.
- Privacy defenses may focus protection on the most recently seen examples, with the protected set sized using empirical forgetting analysis.
- A non-convexity-aware membership-inference attack is demonstrated for a specific k-means setting, motivating study on realistic tasks.
- The empirical methodology relies on state-of-the-art attacks and cannot prove that forgetting occurred, despite supporting observed trends.
B Detailed Experiment Setup
The experiments evaluate forgetting across ImageNet, LibriSpeech, and large-language-model settings using privacy attacks after controlled data injection or poisoning. ImageNet uses repeated injected batches, while speech and language experiments use poisoning-based setups.
- ImageNet experiments use ResNet-50 models trained for 90 epochs with learning rate 0.1, momentum 0.9, and batch size 256.
- The ImageNet figure presentation includes an unsmoothed version of Figure 1a for comparison.
- ImageNet privacy is measured mainly with INJECT, averaging precision at a 10% false positive rate over three trials.
- LibriSpeech experiments train Conformer (L) models for 100,000 steps with batch size 2,048 and use POISON with repeated canary utterances.
- Language-model experiments train decoder-only 110M-parameter models for one epoch over a deduplicated C4 dataset.
C.1 POISON on ImageNet
ImageNet experiments show that forgetting depends mainly on how often and when examples are seen, while learning rate and momentum generally have little effect. Injection and poisoning produce similar forgetting when exposure counts are matched.
- INJECT and POISON yield roughly the same forgetting rate when an example is used 10 times.INJECT uses all repetitions at once, whereas POISON uses one repetition per epoch.
- Later-seen examples generally take longer to forget, although the effect levels off for ImageNet examples injected between epochs 40 and 80.
- Hard examples appear to be forgotten more slowly by ResNet-101, while medium-hardness examples show little difference across model sizes.
- Learning rate and momentum generally do not significantly affect forgetting on ImageNet.
- A learning-rate decay creates an exception: examples seen just before decay are forgotten more slowly, while those seen immediately after decay are forgotten faster.
C.5 Comparison with Traditional Forgetting Metrics
Membership inference provides a more privacy-relevant measure of instance forgetting than accuracy or loss. The experiments also illustrate how non-convex objectives can preserve an outlier’s influence despite additional data.
- Accuracy can suggest complete forgetting while membership inference still detects substantial privacy risk.For examples repeated 10 times, accuracy stabilizes after 20,000 steps, but membership-inference precision remains 70%.
- Loss tracks forgetting more comparably than accuracy, but membership-inference scores remain easier to interpret as privacy-risk measures.
- In the k-means illustration, an initial outlier can determine the learned clustering, and adding data does not make its influence disappear.
D.2 Nondeterminism is Required
The deterministic per-example gradient-descent analysis shows that changing one dataset example leaves models identical until that example is processed, then preserves a distinction throughout training. Consequently, the final models remain distinguishable, so deterministic training does not forget.
- The analysis uses mean estimation, where the learner minimizes squared ℓ2 loss through per-example gradient descent.
- An adversary knowing all other points can distinguish the two datasets from their perpetually different models.
- Theorem 2 states that for 0 < η < 0.5, any two distinct datasets differing in one row produce different final models.
- Before the differing index j, both datasets produce the same model because their first j−1 training examples are identical.
- At index j, the gradient updates preserve a nonzero model difference when the datasets contain different examples.
- The gradient-update function is one-to-one, so once the models differ at step j+1, they remain different for the rest of training.
D.3 Data Randomness can Cause Forgetting
The analysis studies forgetting under random sampling and injection in Gaussian mean estimation. It derives the trained model as a weighted combination of sampled examples and bounds the resulting distributions' divergence.
- The analysis uses mean estimation with random sampling and injection, formalized in Algorithm 4.
- The algorithm's output after k steps has Rényi divergence of order α at most 2α for 0 < η < 1/2.
- Repeated gradient updates produce a weighted sum of the initial parameters and examples, with weights determined by η and training order.
- Injecting +v or −v creates two neighboring training trajectories whose output distributions are analyzed separately.
- The output distributions are Gaussian because the sampled examples follow N(µ, Σ), enabling a Rényi-divergence analysis.
- The Rényi divergence bound is obtained by analyzing the Gaussian means and shared covariance, then bounding the η-dependent term.