PaperlayerView exact revision on arXiv

Source-linked AI summary

Threats, Attacks, and Defenses in Machine Unlearning: A Survey

Ziyao Liu, Huanyi Ye, Chen Chen, Yongsen Zheng, Kwok-Yan Lam

arXiv:2403.13682v5cs.CRcs.AI

TL;DR

Machine unlearning has security and privacy vulnerabilities, yet the literature lacks a comprehensive review of threats, attacks, and defenses in these systems. This survey develops a workflow-based taxonomy and analyzes their relationships and dual roles. It also identifies challenges and directions for improving machine unlearning’s safety, reliability, and privacy compliance.

  • Problem

    The survey addresses the lack of a comprehensive review categorizing security and privacy threats, attacks, and defenses in machine-unlearning systems.

    Source: §I · p. 2

  • Method

    The survey uses a unified machine-unlearning workflow and threat models to organize a taxonomy and analyze threats, attacks, defenses, and their relationships.

    Source: §I · p. 2, §V · p. 4, §XI · p. 11

  • Results

    The survey presents a taxonomy of machine-unlearning threats, reviews defenses by implementation stage, and analyzes unlearning and attacks in their multiple roles.

    Source: §IV · p. 4, D. Summary · p. 8

  • Takeaways & Limitations

    The survey identifies challenges and future directions for improving machine unlearning’s safety, reliability, and privacy compliance.

    Source: §XI · p. 12

Abstract

from arXiv · show

Machine Unlearning (MU) has recently gained considerable attention due to its potential to achieve Safe AI by removing the influence of specific data from trained Machine Learning (ML) models. This process, known as knowledge removal, addresses AI governance concerns of training data such as quality, sensitivity, copyright restrictions, and obsolescence. This capability is also crucial for ensuring compliance with privacy regulations such as the Right To Be Forgotten (RTBF). Furthermore, effective knowledge removal mitigates the risk of harmful outcomes, safeguarding against biases, misinformation, and unauthorized data exploitation, thereby enhancing the safe and responsible use of AI systems. Efforts have been made to design efficient unlearning approaches, with MU services being examined for integration with existing machine learning as a service (MLaaS), allowing users to submit requests to remove specific data from the training corpus. However, recent research highlights vulnerabilities in machine unlearning systems, such as information leakage and malicious unlearning, that can lead to significant security and privacy concerns. Moreover, extensive research indicates that unlearning methods and prevalent attacks fulfill diverse roles within MU systems. This underscores the intricate relationship and complex interplay among these mechanisms in maintaining system functionality and safety. This survey aims to fill the gap between the extensive number of studies on threats, attacks, and defenses in machine unlearning and the absence of a comprehensive review that categorizes their taxonomy, methods, and solutions, thus offering valuable insights for future research directions and practical implementations.

I. INTRODUCTION

Machine unlearning can remove specific data’s influence from trained models, but the field lacks a comprehensive review of security and privacy threats, attacks, and defenses. This survey proposes a unified workflow and taxonomy to organize the literature and identify challenges for future research.

I. INTRODUCTION

  • Machine unlearning addresses concerns about sensitive, outdated, incorrect, and copyrighted training data, as well as privacy rights such as the Right To Be Forgotten.The passage also links RTBF requests to regulatory compliance.Source: §I · p. 1
  • Retraining from scratch after removing data can be computationally expensive, motivating methods that unlearn from an existing trained model.The latter aim to obtain an unlearned model without retraining from scratch.Source: §I · p. 1
  • Prior surveys cover general machine unlearning, federated unlearning, and LLM unlearning, but the literature lacks a comprehensive review focused on machine-unlearning security and privacy.The survey identifies this gap as its motivation.Source: §I · p. 2
  • The survey proposes a unified workflow and taxonomy, analyzes relationships among threats, attacks, and defenses, and discusses vulnerabilities and future research challenges.Its contributions also include a critical analysis of defensive strategies.Source: §I · p. 2

II. PRELIMINARIES

The preliminaries establish the concepts used to analyze machine-unlearning threats, including machine unlearning, MLaaS, and attacks on AI models. They distinguish unlearning that approximates retraining from exact unlearning and describe the MLaaS service setting.

II. PRELIMINARIES

  • The preliminaries introduce machine unlearning, MLaaS, poisoning attacks, and membership inference attacks as groundwork for the survey’s later threat and defense analysis.The section gives additional attention to poisoning and membership inference mechanisms.Source: §II · p. 2

A. Machine Unlearning

  • Machine unlearning aims to produce a model like one trained only on the remaining data; approximate methods adjust an existing model, while exact methods typically retrain.Approximate unlearning trades off unlearning performance and efficiency.Source: A. Machine Unlearning · pp. 2–3

B. Machine Learning as a Service

  • MLaaS is a cloud-based model in which providers host machine-learning tools and infrastructure so users can obtain predictions without developing or maintaining their own models.The passage also notes MLaaS research on privacy, scalability, efficiency, security, and adversarial robustness.Source: B. Machine Learning as a Service · p. 3

C. Attacks on AI Models

The survey’s preliminaries characterize attacks on AI models, including poisoning and membership inference, before introducing the roles and phases of an unlearning service workflow. These concepts support the later analysis of threats and defenses.

C. Attacks on AI Models

  • Attacks on AI models seek to manipulate, exploit, or undermine system performance or integrity, and can occur during data collection, training, or deployment.The survey gives additional preliminary attention to poisoning and membership inference attacks.Source: C. Attacks on AI Models · p. 3
  • Poisoning attacks corrupt models by injecting malicious training data; backdoor attacks embed triggers that elicit malicious behavior while standard inputs still appear normal.Poisoning can target overall performance or specific inputs and behaviors.Source: C. Attacks on AI Models · p. 3
  • Membership inference attacks determine whether a data point was in a model’s training set and can also assess information leakage or whether unlearning succeeded.The described approach uses shadow models and does not require knowledge of the target model’s architecture or training-data distribution.Source: C. Attacks on AI Models · p. 3

III. WORKFLOW OF MACHINE UNLEARNING SYSTEMS

  • The MU workflow has training, unlearning, and post-unlearning phases, with data contributors, requesting users, and accessible users participating in the service.Unlearning requests and inference requests may overlap in the post-unlearning phase, complicating request management.Source: §III · pp. 3–4

IV. OVERVIEW OF THREATS, ATTACKS, AND DEFENSES IN MU SYSTEMS

Threats can arise throughout machine-unlearning services, and attacks can exploit those vulnerabilities. The survey also describes unlearning as a defense and attacks as evaluation tools, emphasizing their intertwined roles in system safety and functionality.

IV. OVERVIEW OF THREATS, ATTACKS, AND DEFENSES IN MU SYSTEMS

  • Vulnerabilities arise across MU phases, including malicious training data, harmful unlearning requests, and information leakage from access to trained and unlearned models.The survey describes these threats as challenges to safe deployment.Source: §IV · p. 4
  • Attacks can exploit those threats, while membership checks, differential privacy, and anomaly detection are among the reviewed defenses.The passage notes that differential privacy can prevent membership inference attacks at a cost to model performance.Source: §IV · p. 4
  • Unlearning can remove backdoors from trained models and serve as a defense against potential AI safety issues through value alignment.The survey analyzes this defensive role separately from defenses against attacks on the unlearning service.Source: §IV · p. 4
  • Threats, attacks, and defenses have interdependent roles in maintaining machine-unlearning system functionality and safety.The survey also notes that attacks can be adapted to evaluate unlearning, including privacy leakage, robustness, and proof of unlearning.Source: §IV · p. 4

V. THREAT MODELS

The survey organizes threat models around attacker roles, goals, knowledge, and attack phases. It distinguishes who can attack and what outcomes they seek.

V. THREAT MODELS

  • The threat-model framework classifies attackers by role, goal, system knowledge, and the phase in which an attack occurs.It is intended to structure the related work discussed in later sections.Source: §V · p. 4

A. Attack Roles

  • Attack roles include data contributors, requesting users, and accessible users; one attacker may hold multiple roles, potentially strengthening attacks.Requesting users may also contribute data and access the model service for inference.Source: A. Attack Roles · p. 5

B. Attack Goals

  • Attack goals range from untargeted or targeted incorrect behavior to privacy leakage and other harms such as higher computational cost or reduced fairness and utility.Source: B. Attack Goals · p. 5

C. Adversarial Knowledge

The survey distinguishes adversarial knowledge levels and attack phases, then uses threat models to analyze threats and defenses in machine unlearning.

C. Adversarial Knowledge

  • White-box, grey-box, and black-box knowledge levels capture whether attackers have comprehensive, partial, or no access to model and training information.The survey states that attacks generally become more potent as attacker knowledge increases.Source: C. Adversarial Knowledge · p. 5

D. Attack Phases

  • Attacks may begin with training-data manipulation, arise through malicious unlearning requests, or exploit the unlearned model and information from earlier phases.Source: D. Attack Phases · p. 5

VI. THREATS IN UNLEARNING

  • The survey analyzes information leakage, malicious unlearning, and other vulnerabilities through threat models, and discusses potential defenses.Source: §VI · p. 5

A. Information Leakage

The survey covers leakage arising from model discrepancies and knowledge dependencies, alongside malicious unlearning attacks that can degrade or redirect model behavior. It reviews attack mechanisms and defenses for both threat classes.

A. Information Leakage

  • Model discrepancy can expose unlearned-data membership or enable reconstruction, while knowledge dependencies can leak information through request sequences, cached computations, or prompt embeddings.Source: A. Information Leakage · pp. 5–6
  • Attacks exploit trained-versus-unlearned model outputs to infer membership or reconstruct data, with black-box membership inference and white-box reconstruction described as possible settings.The surveyed methods include confidence-vector comparisons, Top-1 output perturbations, and estimated gradients.Source: A. Information Leakage · p. 6
  • Overlapping inference and unlearning requests create a timing risk, while adaptive requests and cached computations are knowledge dependencies; defenses include urgency checks, limiting outputs, privacy techniques, and avoiding caching.Source: A. Information Leakage · p. 6

B. Malicious Unlearning

  • Direct malicious unlearning needs no training-data manipulation and can degrade overall performance or target selected predictions; proposed checks include data matching or membership inference.Over-unlearning can move points closer to decision boundaries, increasing their susceptibility to misclassification.Source: B. Malicious Unlearning · pp. 6, 8
  • Preconditioned attacks manipulate training data before requesting removal of mitigation data, leaving poisoned data influential; defenses can compare clean- and mitigation-data performance or filter gradient-dissimilar samples.The attack sequence and its outcome are illustrated in Figure 4b and described in the text.Source: B. Malicious Unlearning · pp. 7–8, Figure 4 · p. 7

C. Other Vulnerabilities

The survey includes efficiency, fairness, and over-forgetting risks among other machine-unlearning vulnerabilities. It organizes defenses by when they are applied in the unlearning workflow.

C. Other Vulnerabilities

  • A slow-down poisoning attack can reduce unlearning efficiency by forcing certified removal to perform time-consuming full retraining sooner.The attack targets the interval or number of approximate updates before full retraining is needed.Source: C. Other Vulnerabilities · p. 8
  • Unlearning methods can affect fairness in LLMs or cause over-forgetting in GNNs, reducing performance on remaining edges; proposed mitigations adjust outputs or modify algorithms.Source: C. Other Vulnerabilities · p. 8

D. Summary

  • The survey groups defenses into pre-unlearning request detection or governance, in-unlearning anomaly monitoring, and post-unlearning information protection or model recovery.In- and post-unlearning defenses allow flexible drop-in algorithms, while pre-unlearning defenses are tailored to particular threats.Source: D. Summary · p. 8

VII. DEFENSE THROUGH UNLEARNING

Machine unlearning is presented both as a defense for model recovery and value alignment. The section reviews uses ranging from removing poisoned data to removing undesirable attributes and content.

A. Model Recovery

  • Unlearning can support model recovery by removing identified poisoned data after attacks have compromised a model.The section also notes that detection may require substantial model updates, so poisoned data can affect the model before identification.Source: A. Model Recovery · p. 8
  • Recovery approaches include trigger inversion, trigger-based gradient ascent, limited-access unlearning [101], and pruning followed by unlearning.Other work addresses universal perturbations [102], shared adversarial examples, multimodal settings [104], and federated learning, [105].Source: A. Model Recovery · pp. 8–9

B. Value Alignment

  • Value-alignment uses include removing sensitive attributes for privacy compliance and removing inappropriate, illegal, low-quality, copyrighted, or harmful LLM content [34], –.Isonuma et al. [114] systematically unlearn datasets via gradient ascent to assess their influence on harmful-content generation.Source: B. Value Alignment · p. 9

VIII. EVALUATING UNLEARNING THROUGH ATTACKS

Attacks on unlearning systems can expose threats and also serve as evaluation tools. The section reviews their use to assess privacy leakage, robustness, and proof of unlearning.

VIII. EVALUATING UNLEARNING THROUGH ATTACKS

  • Attacks serve both as threats to unlearning systems and as tools for evaluating privacy leakage, model robustness, and unlearning effectiveness.The section covers privacy auditing, robustness assessment, and attacks used as proof of unlearning.Source: §VIII · p. 10

A. Audit of Privacy Leakage

  • AUC and Attack Success Rate quantify privacy-attack effectiveness, while feature-specific leakage tracking and privacy-degradation metrics assess leakage after unlearning.Privacy-degradation metrics compare leakage with classical attacks against a single model.Source: A. Audit of Privacy Leakage · p. 10

B. Assessment of Model Robustness

  • Over-unlearning with perturbed requests tests decision-boundary effects, while adversarial prompt embeddings test whether supposedly unlearned LLM knowledge can be extracted.Randomized smoothing can certify a budget of removals while retaining model robustness, including for unlearned users in federated settings.Source: B. Assessment of Model Robustness · p. 10

C. Proof of Unlearning

  • Inference and backdoor attacks are used as proof of unlearning when membership or backdoor behavior is absent after removal; adversarial attacks can expose insufficient unlearning.The passage presents these as common approaches alongside variants used in prior studies.Source: C. Proof of Unlearning · p. 10

IX. SUMMARY AND DISCUSSION

The discussion frames machine unlearning as both a source of system threats and a defense, while attacks also evaluate privacy, robustness, and removal effectiveness. It highlights detection challenges for malicious requests and open issues in federated unlearning.

IX. SUMMARY AND DISCUSSION

  • Machine unlearning creates threats through its process and request handling, can defend through model recovery and value alignment, and is assessed with attacks.The discussion calls for further defenses against specific threats in unlearning systems.Source: §IX · pp. 10–11

A. Defenses against Malicious Unlearning

  • Direct malicious requests may be detected by checking whether requested data were in training, but preconditioned attacks request genuine mitigation data without adversarial features.Such subtle requests make robust detection mechanisms important for protecting system integrity.Source: A. Defenses against Malicious Unlearning · p. 11

B. Federated Unlearning

  • Federated unlearning threats are more complex because aggregation diffuses knowledge and privacy-preserving data isolation complicates unlearning; distributed attacks can be stealthy and difficult to detect.The area remains largely unexplored.Source: B. Federated Unlearning · p. 11
  • Federated users' data remain local, so request formats and server-side request management must respect both federated privacy guarantees and RTBF compliance.The passage identifies understanding these request-handling aspects as crucial to developing a compliant unlearning mechanism.Source: B. Federated Unlearning · p. 11

C. Privacy Preservation

Privacy-preserving machine unlearning faces unresolved privacy, efficiency, and performance trade-offs, while large-model unlearning presents additional challenges. The survey concludes with challenges and future directions for improving machine unlearning’s safety, reliability, and privacy compliance.

C. Privacy Preservation

  • In MLaaS, providers may be separate from model developers and should not access users’ data, while privacy technologies’ trade-offs with efficiency and model performance remain largely unexplored for machine unlearning.The survey notes that machine unlearning introduces privacy challenges distinct from traditional machine learning.Source: C. Privacy Preservation · p. 11

D. Unlearning for Large Models

  • Large-model unlearning faces efficiency and verification challenges, embedding attacks can retrieve unlearned information, and threats and attacks specific to these systems remain underexplored.The survey also identifies large models’ non-explainability as a complication for understanding unlearning.Source: D. Unlearning for Large Models · p. 11

XI. CONCLUSIONS

  • The survey analyzes threats, attacks, and defenses, including unlearning as a defense and attacks as tools for evaluating unlearning systems.It also identifies challenges and future directions for improving machine unlearning’s safety, reliability, and privacy compliance.Source: §XI · pp. 11–12
Loading 2403.13682v5…