Source-linked AI summary
The Privacy Onion Effect: Memorization is Relative
Nicholas Carlini, Matthew Jagielski, Chiyuan Zhang, Nicolas Papernot, Andreas Terzis, Florian Tramer
TL;DR
Private-data memorization can leak information, especially for vulnerable outliers, raising the question of whether selectively removing those points can mitigate privacy risk. The paper demonstrates and analyzes a Privacy Onion Effect in which removal exposes new vulnerable layers, with consequences for privacy defenses, audits, and unlearning.
Problem
The paper asks whether removing points that are easily memorized can prevent privacy leakage without uniformly bounding the training algorithm for all datasets.
Method
The paper studies targeted removal and retraining under membership inference attacks, using experiments to test explanations for the Onion Effect.
Results
Removing the most vulnerable examples exposes a new vulnerable layer; an expected 15× privacy improvement was only 2× in practice.
Takeaways & Limitations
The effect makes targeted memorization defenses less effective and suggests that privacy audits should be updated as training data changes.
Takeaways & Limitations
Removing examples is ineffective as a general defense, and duplicating data points can make membership inference easier.
Abstract
from arXiv · showhide
Machine learning models trained on private datasets have been shown to leak their private data. While recent work has found that the average data point is rarely leaked, the outlier samples are frequently subject to memorization and, consequently, privacy leakage. We demonstrate and analyse an Onion Effect of memorization: removing the "layer" of outlier points that are most vulnerable to a privacy attack exposes a new layer of previously-safe points to the same attack. We perform several experiments to study this effect, and understand why it occurs. The existence of this effect has various consequences. For example, it suggests that proposals to defend against memorization without training with rigorous privacy guarantees are unlikely to be effective. Further, it suggests that privacy-enhancing technologies such as machine unlearning could actually harm the privacy of other users.
1 Introduction
The paper identifies a Privacy Onion Effect: removing examples most vulnerable to a privacy attack exposes previously safer examples to that same attack. This effect limits the effectiveness of targeted removal and destabilizes privacy assessments.
- Removing the most vulnerable training examples exposes a new layer of previously safe examples to the same privacy attack.
- Membership inference attacks test whether a given example was included in a model’s training set, and the paper evaluates the effect on CIFAR-10 and CIFAR-100 neural networks.
- 15× expected privacy improvement became only 2× after removing vulnerable examples, making the intervention over 6× less effective than expected.
- Experiments suggest the effect may arise because inliers become outliers after more extreme outliers are removed, rather than from noise, dataset-size reduction, duplicates, or limited model capacity.
- Privacy auditing is unstable because measured risk can change substantially after removing even a small fraction of training data.
- Machine unlearning requests from users at highest apparent risk may inadvertently degrade the privacy of other users.
2 Related Work
The related work frames the paper around memorization, differential privacy, instance-specific privacy, and membership inference. It asks whether selectively removing easily memorized points can avoid uniformly bounding privacy leakage.
- Differential privacy provably prevents the label memorization considered here, but strong guarantees typically reduce accuracy.
- The paper studies whether privacy mechanisms tailored to at-risk data points can replace uniform privacy bounds across all possible datasets.
- Membership inference asks whether a given example belonged to a training dataset and can operate on model predictions with varying granularity.
3 The Privacy Onion Effect
The paper tests whether removing examples most vulnerable to membership inference improves privacy, finding instead that retraining exposes a new vulnerable layer. This Privacy Onion Effect makes one-shot outlier removal substantially less effective than idealized estimates suggest.
- Evaluation: The study uses ROC curves to compare membership inference true-positive and false-positive rates across CIFAR-10 and CIFAR-100.It also examines whether the effect changes under iterative removal, finding no significant impact on the main findings.
- Experimental methodology: The experiments remove the 5,000 most vulnerable examples, retrain models on the remaining 45,000, and re-run membership inference attacks.The procedure uses privacy scores based on average attack success rates and evaluates retained examples after retraining.
- Main results: At a fixed 0.01% false-positive rate, the idealized attack TPR is 0.1% on retained examples, but retraining yields 0.6%.The baseline TPR is 1.5%, so actual removal reduces attack success by roughly half rather than producing the idealized reduction.
- Main results: The observed privacy improvement is over 6× less effective than expected after the 5,000 outliers are removed and the model is retrained.The idealized setting predicts a 15× improvement, whereas the actual experiment improves privacy by only 2×.
- Main results: Removing the current outliers exposes previously safer examples that become vulnerable to the same membership inference attack.The paper calls this recursive replacement of vulnerable layers the Privacy Onion Effect.
4 Potential (Incorrect) Explanations for the Onion Effect
The paper tests whether the Onion Effect could instead arise from measurement noise, dataset-size changes, duplicates, model capacity, or artificial outliers. These experiments reject those explanations and suggest that previously safe inliers become outliers when more extreme outliers are removed.
- The experiments focus on CIFAR-10 because testing these explanations requires training many models.
- 4.1 The Onion Effect Is Not Explained by Statistical Noise in the Privacy Metric: The near-perfect correlation between attack scores across independent model sets (r = 0.9998) indicates that measurement noise is not the primary cause.
- 4.2 The Onion Effect Is Not Explained by the Modified Dataset Being Smaller: Removing the same number of hardest-to-attack or random examples produces nearly identical ROC curves to the full-dataset baseline, rejecting dataset size as the explanation.
- 4.2 The Onion Effect Is Not Explained by the Modified Dataset Being Smaller: The Onion Effect persists after replacing 5,000 vulnerable CIFAR-10 outliers with 5,000 randomly labeled CIFAR-100 outliers, rejecting arbitrary outliers and limited model capacity as explanations.
- 4.2 The Onion Effect Is Not Explained by the Modified Dataset Being Smaller: The Onion Effect still holds after deduplicating CIFAR-10 and removing 5,275 duplicated images, so duplicate training examples do not explain it.
5 Understanding the Onion Effect
The paper explains the Onion Effect as inliers becoming vulnerable outliers when more extreme outliers are removed, and tests this mechanism with influence-based removals. The effect is local, can affect initially safe points, and has implications for machine unlearning.
- Mechanism: The authors test whether inliers become outliers by removing nearby training examples selected for their influence on a previously hard-to-attack target.Influence is estimated from membership inference accuracy averaged across models trained without each candidate example.
- Second onion layer: Removing 25 high-influence points raises membership inference advantage on second-layer targets by 0.22, or 56% of the 0.38 increase caused by removing 5,000 vulnerable points.This indicates that individual second-layer outliers are masked by only a small number of first-layer outliers.
- Random points: For randomly chosen targets, removing 25 high-influence points increases membership inference advantage by only 0.08 on average, from 0.34 to 0.42.The effect is therefore concentrated in a second layer rather than distributed uniformly across the training set.
- Safe points: For initially safe points, removing 25 high-influence points increases membership inference advantage by 0.12 on average, from 0.02 to 0.14.Some initially safe points reach an advantage as high as 0.22, an increase of 20 percentage points.
- Machine unlearning: The experiments also motivate adversarial unlearning, in which an attacker adaptively selects other training points for unlearning to increase inference success on a target.Membership inference is used to measure progress in machine unlearning, so these interactions affect how unlearning outcomes are assessed.
6 Conclusion
The conclusion argues that membership-inference audits are dataset- and model-specific and that removing training examples is an unreliable privacy defense. It also leaves open whether dataset modification can improve privacy without these failures.
- Privacy auditing: Privacy audits based on membership inference should be restricted to the exact dataset used to train the model and treated cautiously under dataset changes.The work characterizes empirical privacy risk as unstable when even a small fraction of training data is removed.
- Privacy defenses: The Onion Effect suggests that defenses relying on removing examples from a training dataset will be ineffective.The paper contrasts dataset modification with modifying the training algorithm, such as using differential privacy.
- Open question: Adding arbitrary out-of-distribution data and duplicating points do not provide the proposed privacy benefits, leaving privacy-improving dataset modification as an open question.Duplication can make membership inference easier because the adversary distinguishes between models with zero and two copies rather than zero and one.
A One-Shot vs Iterative Removal
The paper tests whether one-shot removal differs from iterative layer-by-layer removal because new outliers might emerge during retraining. The two procedures produce identical results, weakening that explanation.
- Iterative removal: The experiment removes the top 100 outliers across 50 iterations, ultimately removing 5,000 examples from the original 50,000-example dataset.Each iteration reruns LiRA on the remaining dataset before removing the next 100 examples.
- Comparison: The iterative procedure performs identically to the baseline that removes all 5,000 outliers in one shot.This directly tests whether sequentially changing the outlier set explains the Onion Effect.
B Visualization of Easy-to-Attack and Hard-to-Attack Examples
Figure 8 compares vulnerable and difficult-to-attack CIFAR-10 examples across all ten classes. The vulnerable examples are generally outliers memorized by the models.
- Comparison: The examples vulnerable to membership inference are generally outliers memorized by the models.The observation is consistent with prior work identifying outliers and memorized examples in neural-network learning.
- Figure layout: Each column represents one CIFAR-10 class, while the top and bottom three rows show vulnerable and difficult-to-attack examples, respectively.The displayed examples are randomly sampled from the 100 most and 100 least vulnerable points in each class.
C Details of CIFAR-10 Deduplication
The CIFAR-10 training set was deduplicated using a CNN-based image-detection algorithm, removing 5,275 duplicated images after manual inspection of sampled clusters. Figure 9 illustrates duplicated training images and their original dataset indices.
- Deduplication procedure: 5,275 duplicated images were removed from CIFAR-10 using imagededup’s CNN-based detection algorithm with a threshold of 0.85.The authors manually inspected a random sample of detected duplicate clusters to verify correctness.
- Deduplication procedure: The deduplication library was used to identify duplicate clusters in the CIFAR-10 training set.The library is open source and uses a convolutional neural network-based detection method in this procedure.
- Deduplication examples: Figure 9 visualizes duplicated CIFAR-10 training images and labels each image with its index in the original training-set ordering.