PaperlayerView exact revision on arXiv

Source-linked AI summary

SoK: Machine Unlearning for Large Language Models

Jie Ren, Yue Xing, Yingqian Cui, Charu C. Aggarwal, Hui Liu

arXiv:2506.09227v1cs.LGcs.CR

TL;DR

LLM unlearning research needs a way to distinguish methods that aim to remove internal knowledge from those that suppress its behavioral effects. This paper proposes an intention-based taxonomy and surveys removal claims, evaluation practices, and deployment challenges, concluding that questions about true removal and practical deployment remain open.

  • Problem

    Existing surveys tend to classify unlearning methods by technical characteristics rather than intention, making it difficult to assess alignment with regulators, users, or applications.

    Source: §1 · p. 1

  • Method

    The paper proposes a removal-versus-suppression taxonomy and uses it to structure discussion of removal claims, evaluation strategies, and practical deployment gaps.

    Source: §1 · pp. 1–2

  • Results

    The paper’s synthesis identifies unresolved questions about whether removal-intended methods truly erase knowledge, shortcomings in evaluation, and practical challenges to deployment.

    Source: §1 · p. 2, §10 · p. 13

  • Takeaways & Limitations

    The intention-based taxonomy offers a framework for understanding methods and assessing their fit with application-specific goals.

    Source: §1 · p. 1

    Most current studies use relatively small models such as 7B or 8B, while real-world deployment requires evaluation on much larger models.

    Source: §9 · p. 13

Abstract

from arXiv · show

Large language model (LLM) unlearning has become a critical topic in machine learning, aiming to eliminate the influence of specific training data or knowledge without retraining the model from scratch. A variety of techniques have been proposed, including Gradient Ascent, model editing, and re-steering hidden representations. While existing surveys often organize these methods by their technical characteristics, such classifications tend to overlook a more fundamental dimension: the underlying intention of unlearning--whether it seeks to truly remove internal knowledge or merely suppress its behavioral effects. In this SoK paper, we propose a new taxonomy based on this intention-oriented perspective. Building on this taxonomy, we make three key contributions. First, we revisit recent findings suggesting that many removal methods may functionally behave like suppression, and explore whether true removal is necessary or achievable. Second, we survey existing evaluation strategies, identify limitations in current metrics and benchmarks, and suggest directions for developing more reliable and intention-aligned evaluations. Third, we highlight practical challenges--such as scalability and support for sequential unlearning--that currently hinder the broader deployment of unlearning methods. In summary, this work offers a comprehensive framework for understanding and advancing unlearning in generative AI, aiming to support future research and guide policy decisions around data removal and privacy.

1. Introduction

The introduction motivates LLM unlearning through privacy, copyright, and data-removal concerns, then proposes classifying methods by whether they intend removal or suppression. It also previews discussion of removal claims, evaluation, and deployment challenges.

1. Introduction

  • LLM unlearning addresses concerns that training data may include copyrighted, private, or sensitive information and that individuals may have a right to request deletion.The introduction cites the GDPR’s “right to be forgotten”, and the New York Times lawsuit against OpenAI and Microsoft.Source: §1 · p. 1
  • The paper proposes an intention-based taxonomy distinguishing methods that seek internal knowledge removal from those that suppress outputs while traces may remain.It also reviews whether removal-intended methods achieve their goal, unlearning evaluation, and practical deployment gaps.Source: §1 · pp. 1–2
  • Existing surveys often classify methods by technical mechanisms, while the proposed taxonomy distinguishes their underlying objectives.The introduction names Gradient Ascent (GA) [13], Negative Preference Optimization (NPO) [14], and Representation Misdirection for Unlearning (RMU) as examples of methods.Source: §1 · p. 1
  • The paper examines whether GA-based methods can truly remove target knowledge, including their known limitations and the question of whether complete removal is necessary.It also surveys evaluation metrics and benchmarks, identifies shortcomings in current practice, and suggests improvement directions.Source: §1 · p. 2
  • Practical gaps include sequential requests, minimizing side effects on general capabilities, and verifying whether target data has been unlearned.The paper also situates these challenges within its planned discussion of legal foundations, taxonomy, evaluation, and future directions.Source: §1 · p. 2

2. Background in laws

The section connects data protection and AI regulation to the need for unlearning, emphasizing privacy rights, copyright obligations, and transparency. It describes relevant provisions in the United States and European Union.

2. Background in laws

  • The section presents legal and regulatory frameworks as drivers of demand for removing specific training data or knowledge from LLMs.It links this demand to privacy, individual rights, and model accountability.Source: §2 · p. 2
  • U.S. Executive Order 14110 emphasizes lawful and secure data use and privacy-enhancing tools, including “manageability” and “disassociability.”The section describes disassociability as decoupling personal data from its computational influence.Source: §2 · p. 2
  • The EU AI Act requires general-purpose AI providers to comply with copyright law and publish a detailed training-data summary, while GDPR Article 17 provides a right to erasure.For public data, controllers must take reasonable technical steps to inform others processing it of an erasure request.Source: §2 · p. 2
  • The section concludes that data-protection laws and emerging AI regulations imply a growing need for unlearning mechanisms to support privacy rights, transparency, and accountability.Source: §2 · p. 2

3. Problem statement and definitions

The section defines unlearning as finding an output distribution close to that of an ideally unlearned model, while retaining useful responses outside the forget set. It also notes that methods may alter inputs rather than model parameters.

3. Problem statement and definitions

  • Unlearning seeks an output distribution close to that of an ideal model retrained without the forgetting data, among distributions allowed by the chosen algorithm.The section defines p0 as the original model, p* as the ideal retrained model, and pu as the unlearned model’s output distribution.Source: §3 · pp. 2–3
  • The objective balances minimizing information about the forgetting data with retaining knowledge and accurate responses for inputs outside the forget set.The section calls preservation of general performance the model’s utility.Source: §3 · p. 3
  • Unlearning need not adjust model parameters: ICUL and RAG-based methods can modify the input to suppress learned responses.An auxiliary input may be an in-context example or a retrieval-augmented modified prompt designed to guide the model away from target information.Source: §3 · p. 3

4. Removal or suppression: two paths in unlearning

The section organizes unlearning methods by intention: removal aims to eliminate internal traces, while suppression controls behavior even if the model retains the data. It then maps methods by approach or by the model component they target.

4. Removal or suppression: two paths in unlearning

  • The taxonomy addresses a gap in technical-mechanism classifications by foregrounding the different intentions behind methods.The section argues that intention-aware classification helps assess fit with regulators, end-users, and downstream applications.Source: §4 · pp. 3–4
  • Removal-intended unlearning seeks to eliminate the model’s internal knowledge or training trace of the forget set.Source: §4 · p. 4
  • Suppression-intended unlearning restricts outputs to behave as if data were forgotten, while allowing that the model may still encode it.The section contrasts strong removal guarantees for some legal-compliance settings with output-level suppression for harmful-knowledge filtering.Source: §4 · p. 4
  • The taxonomy divides removal-intended methods into GA-based approaches and model editing, and organizes suppression methods by targeted component.The suppression targets include the input space, hidden representations, and output tokens.Source: §4 · p. 4

5. Removal-intended unlearning

Removal-intended methods aim to alter model parameters to remove the influence of forgetting data, with Gradient Ascent and task arithmetic among the approaches. The section describes variants that balance forgetting with retained utility or target forgetting more selectively.

5. Removal-intended unlearning

  • Removal-intended unlearning seeks to remove specific knowledge or data influence by modifying model parameters, principally through Gradient Ascent variants and task arithmetic.The section focuses mainly on Gradient Ascent and its variants.Source: §5 · p. 4

5.1.1. Gradient Ascent.

  • Gradient Ascent reverses the training loss to reduce the model’s likelihood of generating forgetting targets, but is sensitive to hyperparameters and can degrade utility.Its variants are motivated in part by these limitations.Source: §5.1.1 · p. 4

5.1.2. Retaining set.

  • Retaining-set variants combine forgetting and retaining losses, using either gradient descent on retained data or KL divergence from the pre-unlearning model to limit distortion.The retaining-loss weight controls its strength.Source: §5.1.2 · p. 4

5.1.3. Negative Preference Optimization.

  • Negative Preference Optimization slows Gradient Ascent’s divergence, with its divergence speed theoretically proved exponentially slower than Gradient Ascent [14].Source: §5.1.3 · pp. 4–5

5.1.4. Mitigating gradient conflicts.

  • Gradient-conflict methods address opposing forgetting and retaining objectives through gradient direction changes, parameter partitioning, or separate optimizers.The approaches respectively normalize and contrast gradients [26], freeze utility-critical parameters [27], or isolate momentum updates [28].Source: §5.1.4 · p. 5

5.1.5. Second-order information.

  • Second-order approaches use curvature information to refine updates: one approximates the Hessian with Sophia, while another preserves important retained-data parameters within a remain-preserving manifold.Source: §5.1.5 · p. 5

5.1.6. Selective forgetting.

  • Selective forgetting chooses samples or tokens rather than applying Gradient Ascent uniformly, which can unnecessarily damage utility on tokens unrelated to the target knowledge.Methods select highly memorized samples, low-probability tokens, or tokens scored by a trained detector.Source: §5.1.6 · p. 5

5.2. Model editing by task arithmetic

  • Task arithmetic subtracts a fine-tuning task vector and is theoretically effective when the unlearned task is irrelevant or contradictory to retained tasks.One variant merges only elements with consistent signs across vectors from models fine-tuned with diverse hyperparameters [36].Source: §5.2 · p. 5

6. Suppression-intended unlearning

Suppression-intended methods seek to prevent outputs that expose forgetting data rather than eliminate its traces. They intervene in model fine-tuning or alter the response behavior, with approaches targeting probabilities, refusals, or plausible but incorrect answers.

6. Suppression-intended unlearning

  • Suppression-intended methods aim to prevent revealing outputs rather than eliminate traces of forgetting data, offering an alternative when full removal is difficult to scale or guarantee.Source: §6 · pp. 5–6

6.1. Full-parameter fine-tuning

  • Full-parameter fine-tuning methods avoid reversed training loss and aim to reduce the utility side effects that can occur when Gradient Ascent does not specify what to output.Source: §6.1 · p. 6

6.1.1. Fine-grained probability.

  • Inverted Hinge Loss lowers the target token’s probability while promoting its closest alternative, seeking more controlled unlearning with less utility damage than incoherent outputs.A separate forget loss uses KL divergence between the original distribution and one with the target token removed [40].Source: §6.1.1 · p. 6

6.1.2. Rejection fine-tuning.

  • Rejection fine-tuning uses preference optimization to train responses to forgetting data toward a fixed rejection template, such as “I don’t know.”Source: §6.1.2 · p. 6

6.1.3. Incorrect labels.

  • Incorrect-label methods return plausible but wrong facts instead of explicit refusals, aiming to mimic unfamiliarity with forgetting data and mask unlearning traces.Source: §6.1.3 · p. 6

6.2. Input space

Input-space unlearning changes the prompt or its context rather than model parameters or the inference process. The surveyed approaches use in-context examples, retrieval, agents, or prompt-embedding perturbations to suppress outputs tied to forgetting data.

6.2. Input space

  • Input-space unlearning alters input x while leaving model parameters and the inference process unchanged, a design described as best preserving model utility.Source: §6.2 · p. 6

6.2.1. In-Context Unlearning (ICUL).

  • ICUL combines randomly altered labels for target instances with correctly labeled context examples that stabilize the prompt, and is described as efficient and compatible with black-box LLMs [17].Source: §6.2.1 · p. 6

6.2.2. RAG-based unlearning.

  • RAG-based unlearning retrieves forgetting content and appends a confidentiality instruction to guide the model away from related outputs; it also supports sequential requests.The paper suggests running retrieval and ordinary generation in parallel, invoking a second suppression pass only when retrieval is non-empty.Source: §6.2.2 · p. 6

6.2.3. Agent-based unlearning.

  • ALU uses four agents to generate, redact, assess, and compose responses, supporting dynamic requests and adversarial robustness but requiring more inference-time computation.The method is reported to scale to thousands of targets without performance degradation.Source: §6.2.3 · p. 6

6.2.4. Embedding Corruption.

  • Embedding-Corrupted Prompts use a trained classifier to detect in-scope queries and apply learned inference-time embedding perturbations optimized to match a retained model’s output.Source: §6.2.4 · p. 6

6.3. Hidden representation space

Hidden-representation methods suppress forgetting information by intervening in representations, including through re-steering, sparse-autoencoder feature editing, and added modules. The section also discusses parameter localization and evidence challenging the need for fixed localized regions.

6.3. Hidden representation space

  • Hidden-representation approaches include re-steering, sparse-autoencoder editing, pruning, and additional modules.Source: §6.3 · p. 7

6.3.1. Re-steering hidden representation.

  • Re-steering methods suppress forgetting representations by mapping them to a random target or a refusal region while seeking to preserve retained-data performance.RMU can fail to converge in deeper layers because larger representation norms make its fixed small-norm target unnatural; norm-matching improves convergence across layers [52]. Other methods combine re-steering with conflict mitigation and second-order optimization [55].Source: §6.3.1 · p. 7

6.3.2. Editing by SAE.

  • Sparse-autoencoder methods identify features associated with forgetting data and suppress them during inference; related work also identifies a refusal feature and extends the approach to vision-language models.The features are selected based on frequent activation in forgetting data but not retained data.Source: §6.3.2 · p. 7

6.3.3. Additional modules.

  • Additional modules can freeze base-model parameters, which means LoRA cannot truly erase learned knowledge; lightweight unlearning layers are also described as supporting scalable, composable sequential deletion without original training data.GRUN adds a soft gate to control unlearning strength, which can approach zero for retained data to preserve utility.Source: §6.3.3 · pp. 7–8

6.3.4. Localization.

  • Controlled experiments found that randomly selected parameter regions can achieve unlearning equally well, challenging the assumption that successful removal requires specific localized regions [81].Source: §6.3.4 · p. 8

6.4. Output space

Output-space methods suppress forgetting content by modifying logits or using retrieval to block tokens. The section describes logit offsets and assistant-model subtraction alongside retrieval-based decoding controls.

6.4.1. Logits difference.

  • δ-unlearning adds a logit offset from identically initialized white-box models to gray-box logits, requiring shared tokenization and access to target logits.One white-box model is fine-tuned with an unlearning objective while the other remains frozen.Source: §6.4.1 · p. 8
  • An assistant model trained to memorize forget data and forget retain data has its logits subtracted, suppressing shared high-probability tokens without changing target parameters.The subtraction has minimal effect on retain-data predictions when the assistant assigns those inputs low or uniform scores.Source: §6.4.1 · p. 8

6.4.2. Retrieval-based.

  • A prompt classifier can trigger retrieval of forbidden answer phrases, which decoding then blocks or penalizes through hard token matching and soft semantic matching.The method aims to suppress forgotten content while preserving fluency and utility.Source: §6.4.2 · p. 8

7. Do the removal-intended methods truly remove forgetting data?

The section questions whether GA truly removes forgetting data: evidence suggests its effects can be bypassed or reversed, while true removal remains difficult to guarantee and verify.

7.1.1. Adversarial attacks.

  • GCG suffixes recovered 54% of forgetting knowledge, and larger LLMs were more vulnerable to extraction [86]; activation studies found output-layer changes rather than removed knowledge.Task Arithmetic posed higher recovery risk than GA in the reported evaluation [86].Source: §7.1.1 · p. 9
  • Soft token attacks outperformed discrete attacks in recovery success, efficiency, and ability to recover data, but may induce unseen knowledge and yield false positives [91], [94].The cited critique describes these attacks as an extreme form of prompt tuning.Source: §7.1.1 · p. 9

7.1.2. Fine-tuning after unlearning.

  • Forgetting knowledge can return after small-scale fine-tuning on related benign data or retraining on independent facts, indicating that current methods may hide rather than erase it, [88].The cited work reports that state-of-the-art methods often fail to remove information from model weights.Source: §7.1.2 · p. 9

7.1. Failures of removal-intended unlearning

  • GA-based unlearning often behaves like suppression: forgetting knowledge can resurface under adversarial prompts or benign fine-tuning.The section notes that this evidence challenges the claim that removal-intended methods erase knowledge.Source: §7 · p. 8, §7.1.2 · p. 9, §7.1 · p. 9

7.2. A new understanding: removal-intended methods are actually doing suppression

  • GA can over-suppress related content; syntactically similar retain examples may partly offset benign-token damage because retain loss also covers those tokens.GA loss does not distinguish benign tokens from tokens directly related to forgetting data.Source: §7.2 · p. 10
  • GA appears to treat forgetting prompts as an unlearning signal: forgetting representations separate from retain and never-seen data, and mixed prompts pull normal representations toward that cluster.The reported mixed prompts also deteriorate the model’s ability to answer normal questions.Source: §7.2 · p. 10

7.3. Open problems

  • Distributed knowledge makes true removal challenging and unverified; tracing affected parameters and testing with adversarial or compositional prompts are proposed research needs.The section says reversing the loss signal may not undo distributed training effects.Source: §7.3 · pp. 10–11
  • Behavioral suppression may suffice when a model meets expected-use requirements, but actual knowledge removal remains a scientifically valuable research goal.The section links removal research to deeper understanding of fine-tuning, alignment, interpretability, and model editing.Source: §7.3 · p. 11

8. Evaluation metrics and benchmarks

The section surveys metrics and benchmarks for measuring unlearning effectiveness and model utility, then identifies evaluation choices that can distort or limit assessment. It argues for diverse benchmarks and more realistic evaluation practices.

8.1.1. General metrics.

  • ROUGE Recall, [22], [100] measures word-level answer overlap, probability, [62], [100] measures tokenwise reproduction likelihood, and multi-choice accuracy, [54], [62] tests correct option selection.ROUGE precision is usually omitted because generated answers may contain additional or auxiliary content; a high answer probability indicates retained confidence.Source: §8.1.1 · p. 11

8.1.2. Unlearning effectiveness metrics.

  • Effectiveness measures include membership-inference leakage tests [22], [101], [102], adversarial-prompt robustness, [86], [101], and watermark, gradient-alignment, Shapley-value, and representation-level metrics [106], [107], [108], [110].Representation-level metrics assess whether internal features are genuinely altered or only superficially perturbed.Source: §8.1.2 · p. 11

8.2. A review of existing benchmarks

  • TOFU uses synthetic fine-tuned knowledge, MUSE [22] tests scalability and sequential unlearning, WMDP targets harmful knowledge, and RWKU [101] probes real-world entities and neighboring knowledge.These benchmarks vary in data realism, control, robustness, and the aspects of utility they test.Source: §8.2 · p. 12
  • No single benchmark is inherently superior, and relying on one may optimize test performance over real-world forgetting objectives [11], [22], [15], [101].The benchmarks differ in control, realism, scalability, and robustness, motivating diverse and comprehensive assessment.Source: §8.2 · p. 12

8.3. Toward better evaluations: challenges and considerations

  • Greedy decoding can miss retained knowledge: a model may not generate a forgotten answer yet still assign it high probability [115], [116].Probabilistic evaluation across output distributions can provide a more realistic assessment of knowledge retention.Source: §8.3 · p. 12
  • Real-world evaluation lacks a reliable standard for user submissions, while fine-tuning benchmarks can confound unlearning with catastrophic forgetting and overfitting [117], [11].Abstract requests may be underspecified, user corpora vary in quality, and synthetic-data overfitting makes utility reductions hard to quantify.Source: §8.3 · p. 12
  • A randomly selected 5% subset can produce unlearning performance comparable to the full dataset, suggesting evaluations may overstate the need to forget every example [118].Other work pruned 10% to 30% of forget-set outliers [96], motivating coreset-aware evaluation.Source: §8.3 · p. 12
  • Utility checks should use models’ internal evaluation suites, since benchmark-specific utility criteria may not match real-world expectations [119].Retaining sets also matter: greater syntactic similarity to forgetting data tends to reduce utility loss [98], an assumption that may not hold in real-world settings.Source: §8.3 · pp. 12–13

9. Other gaps in existing unlearning

The paper identifies deployment challenges involving utility, repeated requests, model scale, submission safety, and interactions with later training or inference stages. It discusses alternatives and constraints that remain relevant for practical use.

Model utility

  • Model utility must be preserved and verified for model builders to have confidence in deploying unlearning, regardless of its effectiveness.The paper identifies utility harm as a critical obstacle to deployment.Source: §9 · p. 13

Sequential unlearning

  • A 3% utility drop per request could reach 45% after 20 iterations, and per-step loss is often greater for parameter- or representation-modifying methods, [22].The paper notes that real users may submit far more than 20 requests and discusses processing the full forget set at each step as one alternative.Source: §9 · p. 13
  • RAG offers an alternative to repeated model updates: new requests need only update the retrieval system.The paper presents this as a promising direction for unlearning.Source: §9 · p. 13

Scalability of model size

  • Most studies use 7B- or 8B-parameter models,, [101], while larger models raise computational and method-design challenges.The paper calls for evaluating unlearning on much larger models for real-world deployment.Source: §9 · p. 13

Poisoning risks in unlearning submissions

  • User-provided unlearning corpora may be poisoned to deliberately degrade model utility [99], making data safety and standards compliance important.The quality of submitted corpora can be difficult to guarantee.Source: §9 · p. 13

Interference Between Different Training and Inference Stages

  • Later RLHF tuning, downstream tasks, or deployment quantization may compromise intended unlearning effectiveness [120], [121], [122].The paper questions whether these stages interfere with unlearning performed after the model is fully trained.Source: §9 · p. 13

10. Conclusion

The conclusion frames LLM unlearning around the intention to remove knowledge or suppress its effects, and uses that framing to organize open questions about methods, evaluation, and deployment.

10. Conclusion

  • The paper proposes an intention-based taxonomy distinguishing removal from suppression, alongside the technical classifications used in prior work.It presents the taxonomy as a foundation for understanding LLM unlearning.Source: §10 · p. 13
  • It examines theoretical and empirical limitations in whether removal-intended methods such as Gradient Ascent truly erase knowledge.The paper also considers evaluation shortcomings and open challenges for real-world deployment.Source: §10 · p. 13
  • The paper highlights usability, side-effect mitigation, and verification as open challenges for practical deployment.Its broader aim is to support future research and practical deployment of LLM unlearning.Source: §10 · p. 13
Loading 2506.09227v1…