Source-linked AI summary
When Machine Unlearning Jeopardizes Privacy
Min Chen, Zhikun Zhang, Tianhao Wang, Michael Backes, Mathias Humbert, Yang Zhang
TL;DR
Machine unlearning is intended to implement the right to be forgotten, but differences between original and unlearned models may leak information about deleted samples. The paper proposes a two-model membership inference attack, finds that it can outperform classical attacks especially on well-generalized models, and evaluates defenses against the leakage.
Problem
The paper investigates whether machine unlearning leaves unintended information about deleted samples in ML models, potentially undermining privacy protection.
Method
The paper uses a black-box membership inference attack that aggregates outputs from original and unlearned classification models, supported by privacy degradation metrics.
Results
The attack outperforms classical membership inference in multiple cases, especially for well-generalized models, and label-only release, temperature scaling, and differential privacy mitigate it.
Takeaways & Limitations
The results can help improve privacy protection in practical implementations of machine unlearning.
Takeaways & Limitations
The study focuses on classification and assumes both original and unlearned models are black-box; its practical analysis assumes relatively few intermediate models.
Abstract
from arXiv · showhide
The right to be forgotten states that a data owner has the right to erase their data from an entity storing it. In the context of machine learning (ML), the right to be forgotten requires an ML model owner to remove the data owner's data from the training set used to build the ML model, a process known as machine unlearning. While originally designed to protect the privacy of the data owner, we argue that machine unlearning may leave some imprint of the data in the ML model and thus create unintended privacy risks. In this paper, we perform the first study on investigating the unintended information leakage caused by machine unlearning. We propose a novel membership inference attack that leverages the different outputs of an ML model's two versions to infer whether a target sample is part of the training set of the original model but out of the training set of the corresponding unlearned model. Our experiments demonstrate that the proposed membership inference attack achieves strong performance. More importantly, we show that our attack in multiple cases outperforms the classical membership inference attack on the original ML model, which indicates that machine unlearning can have counterproductive effects on privacy. We notice that the privacy degradation is especially significant for well-generalized ML models where classical membership inference does not perform well. We further investigate four mechanisms to mitigate the newly discovered privacy risks and show that releasing the predicted label only, temperature scaling, and differential privacy are effective. We believe that our results can help improve privacy protection in practical implementations of machine unlearning. Our code is available at https://github.com/MinChen00/UnlearningLeaks.
1 Introduction
The paper argues that machine unlearning can create unintended privacy leakage by leaving information about deleted samples in model differences. It introduces an attack, privacy metrics, experiments, and defenses to characterize and mitigate this risk.
- Motivation: Machine unlearning may leave an imprint of a deleted target sample, creating unintended privacy risks through differences between original and unlearned models.The original model may reveal little information, while the unlearned model can provide additional leakage.
- Attack: The proposed membership inference attack jointly uses outputs from the original and unlearned models to determine whether a target sample was in the original training set.Its aggregation methods concatenate the two posterior vectors or compute their differences.
- Metrics: The paper introduces Degradation Count and Degradation Rate to quantify privacy loss caused by machine unlearning relative to classical membership inference.Degradation Count measures how often confidence increases, while Degradation Rate measures the average confidence increase.
- Evaluation: Experiments across logistic regression, convolutional neural networks, categorical datasets, and image datasets show that the attack can degrade membership privacy, especially for well-generalized models.The paper reports that the attack can outperform classical membership inference in multiple cases.
- Defenses: Among four evaluated defenses, predicted-label-only release, temperature scaling, and differential privacy effectively prevent the attack, whereas top-k confidence release remains vulnerable.The attack remains robust even when only the top-1 confidence value is released.
2 Preliminaries
Machine learning models are trained on datasets, while machine unlearning removes a target sample's data and influence, balancing privacy requirements against computational cost. The threat model considers black-box access to original and unlearned models and infers whether a target was removed.
- 2.1 Machine Learning: Classification models map samples to posterior probabilities over classes and are trained by minimizing a loss over a training set.
- 2.2 Machine Unlearning: Machine unlearning removes a target sample from the training set and aims to eliminate its influence on the model.
- 2.2 Machine Unlearning: Retraining from scratch produces an unlearned model on the dataset without x, but large datasets and complex models make this computationally expensive.
- 2.2 Machine Unlearning: SISA reduces unlearning overhead by retraining only the sub-model containing the deleted sample and aggregating predictions from sub-models.The experiments use posterior averaging for aggregation.
- 2.3 Threat Model: The adversary uses original and unlearned black-box models to infer whether a target sample belonged to the original training set but not the unlearned set.The paper also assumes access to a local shadow dataset for training shadow models and the attack model.
- 2.3 Threat Model: The paper distinguishes its sample-level membership inference attack from prior update-leakage attacks focused on reconstruction or class-level label inference.
3 Membership Inference in Machine Unlearning
The attack compares outputs from original and unlearned models by constructing features from their posteriors and classifying target membership. Shadow models provide positive and negative training cases, while several aggregation schemes capture discrepancies caused by deletion.
- 3.1 Attack Pipeline: The attack pipeline generates posteriors, constructs features from both model outputs, and performs binary membership inference.
- 3.2 Training the Attack Model: The adversary trains shadow original and unlearned models using disjoint positive and negative shadow datasets to create attack-model cases.Positive cases use deleted samples from the shadow positive set; negative cases use samples from the shadow negative set.
- 3.2 Training the Attack Model: The shadow unlearned model is trained after deleting target samples from the shadow positive dataset, with the presentation formalizing one-sample deletion while also evaluating group deletion.
- 3.2 Training the Attack Model: The attack queries original and unlearned models for posteriors, then sends an aggregated feature vector to a binary classifier.
- 3.2 Training the Attack Model: Positive and negative attack cases are balanced, and multiple shadow models and deletion requests improve training diversity.
- 3.3 Feature Construction: Feature construction uses posterior concatenation, element-wise differences, Euclidean distance, and sorted posterior variants to capture deletion discrepancies.The paper summarizes five representative feature-construction methods and evaluates which work best for different model behaviors.
4 Privacy Degradation Measurement
The paper measures privacy degradation by comparing confidence in the proposed attack with confidence in classical membership inference. Degradation Count captures how often confidence increases, while Degradation Rate captures the average increase.
- The two privacy degradation metrics quantify additional privacy degradation from the proposed attack relative to classical membership inference.
- DegCount is the proportion of target samples whose true membership status receives higher confidence from the proposed attack than from classical membership inference.Higher DegCount indicates higher privacy degradation.
- DegRate is the average confidence improvement of the proposed attack over classical membership inference when predicting the target samples' true membership status.Higher DegRate indicates higher privacy degradation.
5 Evaluation
The evaluation tests the attack across models, datasets, metrics, feature constructions, overfitting levels, transfer settings, and unlearning methods. Across these settings, the attack generally outperforms classical membership inference, with particularly strong gains for well-generalized models and reduced leakage under SISA.
- Experimental setup: The evaluation spans categorical and image datasets, simple models through convolutional neural networks, and AUC, DegCount, and DegRate metrics.The experiments also examine feature construction, overfitting, hyperparameters, transferability, and SISA.
- Scratch method: On categorical datasets, the attack consistently outperforms classical membership inference, improving AUC by up to 0.48 with best DegCount and DegRate values of 0.94 and 0.40.Decision trees are reported as the most vulnerable target models, plausibly because deleting one sample changes their structure and posterior outputs more visibly.
- Scratch method: On image datasets, the attack also outperforms classical membership inference in all settings; CIFAR10 with DenseNet has the highest privacy degradation, while MNIST with SimpleCNN has the lowest.The corresponding overfitting levels are 0.439 and smaller than 0.05, respectively.
- Feature construction: Difference-based features perform better for well-generalized logistic regression, whereas concatenation-based features perform better for overfitted ResNet50; sorting improves both feature types.Plain posterior information is weaker for well-generalized models, making discrepancies between original and unlearned posteriors more informative.
- Overfitting: For a well-generalized decision tree with classical AUC 0.497, the attack reaches AUC 0.882, and overall performance is relatively independent of overfitting level.This demonstrates substantial attack capability where classical membership inference is equivalent to random guessing.
- SISA unlearning: The attack performs worse against SISA than against scratch retraining, plausibly because SISA reduces a specific sample’s influence on its global model.This result motivates considering SISA in practical deployments of unlearning methods.
6 Attack Under Different Scenarios
The paper evaluates its attack across practical machine-unlearning scenarios, finding persistent additional membership leakage, though performance can decline as deletions accumulate or are grouped. The attack remains effective for small update volumes and transfers across models, while remaining samples are less sensitive.
- Multiple Intermediate Unlearned Models: 0.84 attack AUC is achieved with fewer than 10 intermediate unlearned models for the LR target model.Attack AUC decreases as intermediate models increase because previously deleted samples mask the target sample's impact.
- Group Deletion: 0.972 versus 0.842 attack AUC compares single deletion with 0.2% group deletion for LR, showing slightly weaker but still effective group-deletion performance.The experiments delete groups ranging from 0.02% to 5% of the dataset, with at most 5% removed.
- Online Learning: Adding samples during online unlearning only slightly changes the attack performance.The experiment deletes target samples and adds the same number of new samples before retraining the unlearned model.
- Remaining Samples: The attack also exceeds classical membership inference for remaining samples, although those samples are less sensitive than deleted targets.The remaining-sample evaluation uses four target models and four attack models on Insta-NY.
- Takeaways: Attack performance drops for SISA unlearning, motivating consideration of SISA in real-world applications.The paper also reports effectiveness across different shadow and target-model distributions and architectures.
- Scenario Results: Across multiple intermediate-model, group-deletion, and online-learning scenarios, the attack consistently outperforms classical membership inference.The attack remains effective when fewer than 0.2% of training samples are unlearned or updated.
7 Possible Defenses
The paper evaluates four defenses that restrict released posterior information or reduce individual-sample influence. Label-only publishing, temperature scaling, and differential privacy are effective, whereas top-k confidence release is not robust.
- Top-k Confidence Values: Top-k confidence release cannot effectively mitigate the attack, even when only the top-1 confidence value is published.The evaluation covers Top-1, Top-2, and Top-3 defenses on Insta-NY and Adult.
- Label-only Publishing: Label-only publishing may remain vulnerable because sampling attacks can recover posteriors to a large extent.The paper leaves improved attacks against label-only publishing for future work.
- Effective Defenses: Publishing only the predicted label, temperature scaling, and differential privacy can effectively prevent the attack.These mechanisms respectively reduce accessible information or limit a single sample's effect on model outputs.
- Temperature Scaling: Temperature scaling is effective but applies only to models whose final layer is softmax.The method divides the logits vector by a learned scaling parameter to reduce over-confident posterior outputs.
- Differential Privacy: Differential privacy effectively prevents the attack but inevitably degrades target-model accuracy.The paper notes that privacy-budget parameters must be tuned to balance privacy and model utility.
8 Related Work
Related work covers machine-unlearning methods that reduce retraining cost, deletion-compliance verification, membership inference, and other ML attacks. This paper addresses a distinct gap: privacy risks created when adversaries compare original and unlearned models.
- Machine Unlearning: Prior machine-unlearning research primarily reduces the high computational overhead of retraining models from scratch.Other work transforms learning algorithms or develops approximate unlearning approaches, with limitations for some model classes.
- Deletion Verification: A separate line of work verifies deletion compliance using backdoors implanted in data before model training.Data owners check whether their data were deleted by measuring backdoor success after a deletion request.
- Paper Positioning: This paper is distinct because it quantifies privacy risks for deleted samples when an adversary accesses both original and unlearned ML models.The authors identify this as the first investigation of that problem in machine learning.
- Membership Inference: Classical membership-inference research uses model outputs, often with shadow models, to infer whether samples were in training data.Prior studies examine factors such as overfitting and the number of target-model classes.
- Other Attacks: Existing ML attacks also include property inference, model inversion, and attacks targeting model updates or deletion scenarios.These studies infer training-data properties, missing attributes, or information from differences between model versions.
9 Conclusion
The paper establishes membership inference as a lens for studying unintended leakage from machine unlearning and evaluates the attack across realistic deletion and updating scenarios. It also presents defenses intended to mitigate these privacy risks in practice.
- Conclusion: The paper takes the first step toward investigating unintended information leakage in machine unlearning through membership inference.Its attack summarizes discrepancies between posteriors from original and unlearned models.
- Conclusion: Experiments on five real-world datasets show that the attack can outperform classical membership inference, especially for well-generalized models.The evaluation also covers intermediate unlearning, group deletion, and online learning.
- Conclusion: Four defense mechanisms are presented to mitigate the newly discovered privacy risks.The stated goal is to help improve privacy in practical implementations of machine unlearning.
A Datasets
The experiments use six datasets spanning tabular, location, and image classification tasks, with dataset sizes ranging from 1,300 images per class to around 3 million samples.
- UCI Adult contains around 50,000 samples with 14 features for binary prediction of income exceeding $50k.
- US Accident contains around 3M samples and 30 valid features for three-class accident-severity classification.
- Insta-NY provides 19,215 filtered locations represented by weekly hourly check-in vectors for nine-class location-category prediction.
- MNIST contains 42,000 28×28-pixel handwritten-digit images across 10 classes.
- CIFAR10 contains 60,000 32 × 32 colored images across 10 equally distributed classes, split into 50,000 training and 10,000 testing images.
- STL10 contains 10 image classes with 1,300 images per class, including vehicles, animals, and objects.
B Hyperparameter Settings of Simple Models
The study evaluates several simple classifiers using specified optimization, architecture, and regularization settings, with most models implemented in sklearn version 0.22.
- The models are implemented in sklearn version 0.22 except logistic regression, whose settings are listed for reproducibility.
- Logistic regression uses a single linear classifier implemented in PyTorch and trained with Adam for 100 epochs.
- The decision tree uses Gini index, max_leaf_nodes=10, and default values for other hyperparameters.
- The random forest uses Gini index, 100 estimators, min_samples_leaf=30, and default values for other hyperparameters.
- The multi-layer perceptron uses Adam, ReLU activation, hidden layer size 128, and learning rate 0.001.
C Implementation of SimpleCNN
The SimpleCNN is trained for 100 epochs with SGD at learning rate 0.001, and its MNIST-specific input and kernel dimensions are specified.
- SimpleCNN training uses SGD, learning rate 0.001, and 100 epochs.
- For MNIST, the SimpleCNN uses one input channel, image width and height 28, convolution kernel size 3, and max-pooling kernel size 2.