PaperlayerView exact revision on arXiv

Source-linked AI summary

A Duty to Forget, a Right to be Assured? Exposing Vulnerabilities in Machine Unlearning Services

Hongsheng Hu, Shuo Wang, Jiamin Chang, Haonan Zhong, Ruoxi Sun, Shuang Hao, Haojin Zhu, Minhui Xue

arXiv:2309.08230v2cs.CR

TL;DR

MLaaS unlearning is difficult to assess because servers may lack the original training data and prior research has not examined malicious risks in that setting. The paper proposes two black-box strategies for inducing over-unlearning; experiments report that these strategies can degrade model utility across settings and architectures.

  • Problem

    MLaaS limits training-data access, and existing work has not investigated the risks malicious users may pose through unlearning requests in that setting.

    Source: §I · pp. 1–2

  • Method

    The paper proposes sample-blending and decision-boundary-pushing strategies that manipulate submitted unlearning data using black-box model access.

    Source: §I · p. 2, B. Blending as Naive Over-unlearning · p. 6, C. Pushing as Advanced Over-unlearning · p. 7

  • Results

    Across settings and model architectures, both strategies induce over-unlearning; Pushing-I degrades accuracy by around 5.7% on VGG and 2.5% on ResNet versus normal unlearning.

    Source: B. Ablation Study for Pushing-I and… · p. 13, §VIII · p. 14

  • Takeaways & Limitations

    The findings identify over-unlearning as a risk to MLaaS model utility and underscore the need for servers to address it while maintaining service integrity and regulatory compliance.

    Source: §VIII · p. 14

    The authors note that more advanced adversarial attacks and robust countermeasures could be incorporated recursively into the over-unlearning design, leaving further work on countermeasures.

    Source: §VII · p. 14

Abstract

from arXiv · show

The right to be forgotten requires the removal or "unlearning" of a user's data from machine learning models. However, in the context of Machine Learning as a Service (MLaaS), retraining a model from scratch to fulfill the unlearning request is impractical due to the lack of training data on the service provider's side (the server). Furthermore, approximate unlearning further embraces a complex trade-off between utility (model performance) and privacy (unlearning performance). In this paper, we try to explore the potential threats posed by unlearning services in MLaaS, specifically over-unlearning, where more information is unlearned than expected. We propose two strategies that leverage over-unlearning to measure the impact on the trade-off balancing, under black-box access settings, in which the existing machine unlearning attacks are not applicable. The effectiveness of these strategies is evaluated through extensive experiments on benchmark datasets, across various model architectures and representative unlearning approaches. Results indicate significant potential for both strategies to undermine model efficacy in unlearning scenarios. This study uncovers an underexplored gap between unlearning and contemporary MLaaS, highlighting the need for careful considerations in balancing data unlearning, model utility, and security.

I. INTRODUCTION

MLaaS makes unlearning difficult because servers may lack the original training data, while retraining deep models is computationally costly. The paper examines over-unlearning risks and proposes black-box strategies to study them.

I. INTRODUCTION

  • Retraining from scratch is a common unlearning approach, but it is often computationally costly for deep models with millions or billions of parameters.The MLaaS setting further complicates retraining because the server may lack the original training dataset.Source: §I · p. 1
  • MLaaS limits existing unlearning approaches because servers may not have training data, while malicious users can submit requests in a public deployment setting.The paper frames the open question as whether and how easily such users can compromise the server’s normal business.Source: §I · pp. 1–2
  • Over-unlearning is identified as a threat in MLaaS: malicious unlearning requests can remove more information than intended and compromise model utility.The proposed strategies require only black-box access to the server model.Source: §I · p. 2

A. Related Work

  • Prior work divides unlearning into exact retraining-based methods, which can guarantee removal but often require stored training data, and approximate parameter updates.Approximate methods include influence-function and gradient-based approaches.Source: A. Related Work · p. 3
  • The evaluation focuses on gradient-based approximate unlearning as feasible in MLaaS, using fine-tuning and Warnecke et al. as baselines.Fine-tuning relabels unlearned samples and fine-tunes on them, but information may remain in model parameters.Source: A. Related Work · p. 3

B. Our Threat Model

The threat model places an authorized unlearning user against an MLaaS server that retains a model and test data but lacks the original training data. A malicious user can exploit the permitted unlearning interface to degrade model performance.

B. Our Threat Model

  • The server provides ordinary MLaaS and unlearning services but has test data rather than the original training dataset.The scenario assumes training data may not be stored for privacy or regulatory reasons.Source: B. Our Threat Model · pp. 3–4
  • An authorized user may be malicious or compromised and can seek severe model-performance degradation through unlearning requests using only a few samples.The user is authorized to request removal of contributed data.Source: B. Our Threat Model · p. 4
  • The attacker has black-box model access, can query samples for output probabilities, and cannot see model parameters or architecture.The attacker submits a perturbed version of the contributed unlearning data.Source: B. Our Threat Model · p. 4
  • Malicious unlearning may degrade the original class or other classes, including a targeted class introduced through information added to the submitted samples.The threat model also requires manipulated samples and their labels to evade basic server checks.Source: B. Our Threat Model · p. 4

C. Difference from Existing Threats to Machine Unlearning

The paper distinguishes over-unlearning from earlier threats and formalizes it as excess utility degradation after a user modifies submitted unlearning data. It studies sample-wise and pixel-wise strategies under black-box access.

C. Difference from Existing Threats to Machine Unlearning

  • Earlier threats target unlearning cost or selected predictions and require training-data and white-box knowledge; over-unlearning instead targets model utility in MLaaS.The cited prior attacks are slow-down attacks [36] and targeted attacks.Source: C. Difference from Existing Threats to… · p. 5

III. METHODOLOGY OF MALICIOUS UNLEARNING

  • The methodology proposes sample-wise and pixel-wise modifications to test whether an authorized user can compromise the server’s normal business through unlearning.The paper notes that single-sample influence is difficult to measure precisely for complex non-convex models such as DNNs.Source: §III · p. 5

A. Problem Statement

  • Over-unlearning occurs when modified unlearning data lowers class-A test accuracy below the result of normally unlearning the same data.The paper distinguishes degradation in class A from degradation in other classes, depending on whether added information concerns other classes.Source: A. Problem Statement · p. 5

B. Blending as Naive Over-unlearning

The naive strategy blends samples from another class into the data submitted for unlearning. This adds information to the request and can cause unexpected degradation on the other class.

B. Blending as Naive Over-unlearning

  • Blending embeds features from another class into the original unlearning samples, using a sample-wise injection function without computational overhead.The strategy is presented as a lightweight way to demonstrate over-unlearning-II, where degradation occurs outside the original class.Source: B. Blending as Naive Over-unlearning · p. 6
  • Unlearning blended samples can unexpectedly degrade performance on the class whose features were added.The blending ratio λ controls transparency, and the samples are represented as vectors.Source: B. Blending as Naive Over-unlearning · p. 6
  • The user can relabel modified samples with the server model’s predicted label, making the submitted label consistent with the prediction and harder to flag.This is described as supporting the stealthiness of the manipulated unlearning sample.Source: B. Blending as Naive Over-unlearning · p. 6

C. Pushing as Advanced Over-unlearning

The section presents pushing as an advanced over-unlearning strategy: users move unlearned samples toward or across the model’s decision boundary to enlarge their unlearning effect. It uses black-box adversarial perturbation because exact boundary distances are difficult to calculate for deep models.

C. Pushing as Advanced Over-unlearning

  • Blending is cost-efficient and model-irrelevant, but its effects are blind and do not degrade performance effectively on every dataset.This motivates the section’s introduction of an advanced over-unlearning method.Source: C. Pushing as Advanced Over-unlearning · p. 6
  • Moving unlearned samples toward or across the decision boundary can enlarge their unlearning effect and alter which classes are affected.Samples kept within the boundary primarily affect the original class; samples pushed across it can also affect other classes.Source: C. Pushing as Advanced Over-unlearning · p. 6
  • Pushing-I keeps a sample on its original side of the boundary, while Pushing-II moves it just across the boundary.Pushing-I examines effects from near-boundary samples; Pushing-II examines samples that are hard for the model to predict correctly.Source: C. Pushing as Advanced Over-unlearning · p. 7
  • The method is intended to move samples further toward the boundary even when some user-contributed samples are already near it.The authors argue that most training samples are not near the boundary in well-distributed, balanced datasets, and that near-boundary samples can still be moved further.Source: C. Pushing as Advanced Over-unlearning · p. 7
  • Adversarial perturbation moves samples closer to the decision boundary without requiring its exact distance, which is difficult to calculate for deep models.The user can use black-box Carlini and Wagner attacks and zeroth-order optimization, estimating gradients through queries to the deployed model.Source: C. Pushing as Advanced Over-unlearning · p. 7
  • Pushing-I selects the last sample still predicted as its true class, whereas Pushing-II selects the next sample after its prediction changes.The selection is made along the sequence of adversarially perturbed samples.Source: C. Pushing as Advanced Over-unlearning · p. 8

IV. EXPERIMENTAL SETTINGS

The section outlines the experimental setup by introducing the datasets, models, evaluation metrics, unlearning settings, and benchmarks used to assess over-unlearning.

  • The experiments are organized around datasets, models, evaluation metrics, unlearning settings, and benchmarks for evaluating the over-unlearning threat.Source: §IV · p. 8

A. Datasets and Models

The experiments use three image-classification datasets, VGG and ResNet models, and test accuracy to assess utility. They compare two unlearning benchmarks under stated limits on user-contributed samples and perturbations.

A. Datasets and Models

  • The evaluation uses CIFAR-10, CIFAR-100, and STL-10, with STL-10 described as more challenging because of its higher image resolution and learning complexity.The models are VGG and ResNet.Source: A. Datasets and Models · p. 8
  • Test-set accuracy is used as the feasible indicator of model utility for the service provider.Source: A. Datasets and Models · p. 8

B. Unlearning Settings and Benchmarks

  • Users are assumed to control no more than half of one class’s training data: 200 samples in CIFAR-100, 200 in STL-10, and 2,000 in CIFAR-10.Pushing perturbations are bounded by an ℓ2 norm of 20; modified samples have mean SSIM around 0.97 and mean LPIPS around 0.03.Source: B. Unlearning Settings and Benchmarks · p. 8
  • The benchmarks are gradient-based unlearning and fine-tuning with randomly relabeled user-submitted images.The gradient-based method is described as certified to approximate retraining with the unlearned data removed, so its results are reported primarily; fine-tuning results are in Appendix C.Source: B. Unlearning Settings and Benchmarks · pp. 8–9
  • The evaluation tests the proposed methods on two unlearning approaches; new attack techniques and effectiveness on other unlearning methods are outside its scope.Source: B. Unlearning Settings and Benchmarks · p. 9

V. EVALUATION

Blending demonstrates over-unlearning on CIFAR-10 but not on CIFAR-100 or STL-10. Pushing-I and Pushing-II reduce target-class utility across all three datasets and can steer prediction errors toward a chosen class.

A. Effectiveness of Blending Method

  • Blending reduces CIFAR-10 accuracy on “cat” by around 1.4% when 400 “airplane” samples containing “cat” information are unlearned, but it is ineffective on CIFAR-100 and STL-10.The authors characterize this naive blending method as effective only for simple classification tasks, not tasks with many categories or complex patterns.Source: A. Effectiveness of Blending Method · p. 9

B. Effectiveness of Pushing-I and Pushing-II

  • Blending embeds information from another class into the unlearned samples, while Pushing-I moves samples near a decision boundary and Pushing-II moves them across it.The evaluation selects an additional class whose information is blended into the original unlearned samples.Source: B. Effectiveness of Pushing-I and… · p. 9, A. Effectiveness of Blending Method · p. 9
  • With 10% of a class unlearned on CIFAR-10, Pushing-I and Pushing-II reduce its accuracy from 87.8% to around 29.5%, versus 81.7% under normal unlearning; at 50%, accuracy falls to around 2%.The comparison covers CIFAR-10, CIFAR-100, and STL-10; both pushing methods achieve lower target-class utility than normal unlearning on all three.Source: Figure 5 · p. 10, B. Effectiveness of Pushing-I and… · p. 10
  • With 400 airplane samples unlearned on CIFAR-10, malicious pushing raises wrong predictions as “cat” from two under normal unlearning to 20 with Pushing-I and 25 with Pushing-II.With 2,000 samples, wrong predictions as “cat” exceed 375 for Pushing-I and 245 for Pushing-II; the passage reports the same conclusion on CIFAR-100.Source: B. Effectiveness of Pushing-I and… · p. 11

VI. ABLATION STUDY

The blending ablations show greater effectiveness with more unlearned samples and higher blending ratios, alongside limits in stealthiness and generality.

A. Ablation Study for the Blending Method

  • Accuracy degradation generally increases with both the number of unlearned samples and the blending ratio.Figure 8 summarizes accuracy degradation with respect to these two variables.Source: Figure 8 · p. 11, A. Ablation Study for the Blending… · p. 11
  • With 400 unlearned samples, blending causes around 0.7% accuracy degradation, rising to around 2.8% with 2,000 samples.This CIFAR-10 experiment embeds “cat” into unlearned “airplane” data with the blending ratio set to 0.3.Source: A. Ablation Study for the Blending… · p. 11
  • A higher blending ratio makes the method more effective, but also makes the embedded other-class pattern more obvious and less stealthy.The experiment varies the ratio from 0.1 to 0.5 with 400 unlearned CIFAR-10 samples.Source: A. Ablation Study for the Blending… · p. 11

B. Ablation Study for Pushing-I and Pushing-II

The ablations show that Pushing-I and Pushing-II achieve over-unlearning across classes, model architectures, and depths, with larger sample sets producing greater accuracy degradation.

B. Ablation Study for Pushing-I and Pushing-II

VII. DISCUSSION

The discussion weighs over-unlearning defenses against their practical drawbacks, while the conclusion reports that the proposed strategies induce over-unlearning across settings and model architectures. The authors call for further work on secure unlearning that balances privacy, utility, and service security.

VII. DISCUSSION

  • Naive blending-based over-unlearning is effective only for simple classification and struggles with many classes or complex patterns.The authors describe it as blind and motivate advanced strategies that push samples toward the decision boundary.Source: §VII · p. 13
  • Blending resembles data poisoning, but does not require its typical training or retraining procedure; pushing targets the decision boundary rather than classifier deception.The authors also note that existing advanced poisoning and adversarial techniques could be incorporated into their design.Source: §VII · p. 13
  • Hash-based request verification may expose training-set membership or enable linkage attacks, and altered uploads can cause false rejection of legitimate requests.The concern applies when compression, transmission issues, or transcoding change a sample from its original form.Source: §VII · pp. 13–14
  • Membership inference can verify whether a sample was used for training, but requires substantial training resources, has low accuracy, and is less effective on well-generalized models.Source: §VII · p. 14
  • Generic anomaly detection is ineffective against these requests because it lacks training data similar to users’ submissions; runtime performance monitoring is offered as a heuristic mitigation.Source: §VII · p. 14
  • The authors say existing defenses have limitations and identify incorporating stronger adversarial attacks and countermeasures as future research.Source: §VII · pp. 13–14

VIII. CONCLUSION

  • The strategies effectively induce over-unlearning across settings and model architectures, potentially compromising MLaaS model utility.The conclusion identifies this risk when users submit malicious unlearning data.Source: §VIII · p. 14
  • The conclusion calls for further work on secure unlearning methods and policies that balance data privacy, model utility, and service security in MLaaS.Source: §VIII · p. 14

APPENDIX

The appendix describes two approximate-unlearning approaches and their constraints, then specifies the datasets, train-validation split, and model configurations used in the experiments.

A. APPROXIMATE UNLEARNING

  • Influence-function unlearning estimates the effect of data to remove and applies a Newton step, but constructing the inverse Hessian can be difficult for deep models and remaining data may be unavailable.The approach is described for Dtrain = Du ∪ Dr, with Du containing the data to unlearn and Dr the remainder.Source: A. APPROXIMATE UNLEARNING · p. 16
  • Gradient-based unlearning adds back gradients recorded during training to approximate retraining on remaining data; the cited method uses perturbed samples to overwrite information and requires only Du.In the described method, Z-hat is perturbed Z and τ is a small unlearning-rate constant.Source: A. APPROXIMATE UNLEARNING · p. 16

B. EXPERIMENTAL SETTINGS

  • Datasets use an 80% training and 20% validation split; STL-10 evaluation selects 1,000 test images to align its split proportions with CIFAR-10 and CIFAR-100.Source: B. EXPERIMENTAL SETTINGS · p. 16
  • Experiments use two VGG models and one ResNet model, with the first VGG configuration used most; all networks use Adam at learning rate 0.001 and validation loss for stopping.The models vary in depth and architecture to represent different server business tasks.Source: B. EXPERIMENTAL SETTINGS · p. 16

C. EFFECTIVENESS OF OVER-UNLEARNING ON THE FINE-TUNING BASED UNLEARNING METHOD

Under fine-tuning, Pushing-I and Pushing-II slightly reduce overall accuracy compared with normal unlearning, while modified samples show high perceptual similarity to originals in the reported CIFAR-10 analysis.

C. EFFECTIVENESS OF OVER-UNLEARNING ON THE FINE-TUNING BASED UNLEARNING METHOD

  • Pushing-I and Pushing-II slightly degrade overall accuracy versus normal unlearning under fine-tuning, indicating over-unlearning, though the added information is vulnerable to random relabeling.The authors say over-unlearning is difficult with fine-tuning, which also cannot guarantee removal of unlearned-data information from model parameters.Source: C. EFFECTIVENESS OF OVER-UNLEARNING ON… · p. 16, Table XII · p. 17

D. STEALTHINESS OF MODIFIED UNLEARNED SAMPLES

Related papers

Loading 2309.08230v2…