Source-linked AI summary
UnUnlearning: Unlearning is not sufficient for content regulation in advanced generative AI
Ilia Shumailov, Jamie Hayes, Eleni Triantafillou, Guillermo Ortiz-Jimenez, Nicolas Papernot, Matthew Jagielski, Itay Yona, Heidi Howard, Eugene Bagdasaryan
TL;DR
The paper asks whether unlearning can reliably prevent LLMs from producing impermissible content. It analyzes how in-context learning can reintroduce unlearned knowledge, concluding that unlearning alone is incomplete and content filtering matters.
Problem
The paper examines whether unlearning can keep LLMs from exhibiting impermissible behavior when in-context learning can reintroduce removed knowledge.
Source: Introduction · p. 1Method
The paper analyzes unlearning for content regulation through a conceptual account of axioms, derivative knowledge, and contextual reintroduction, including a model setting with inference-time inputs.
Source: Types of Knowledge · p. 3, UnUnlearning · p. 3Results
Even exact unlearning may not prevent impermissible behavior when a model retains knowledge that can be recombined from context, and imperfect unlearning can exacerbate the problem.
Source: Nomenclature · p. 2, Types of Knowledge · p. 3Takeaways & Limitations
The authors argue that unlearning is incomplete for impermissible-knowledge removal in LLMs with strong in-context learning, placing emphasis on content filtering.
Source: Conclusion · p. 5Because knowledge can be introduced by different parties, the authors argue that unlearning should not be assumed to be the sole mechanism for content-policy enforcement.
Source: Discussion · p. 5
Abstract
from arXiv · showhide
Exact unlearning was first introduced as a privacy mechanism that allowed a user to retract their data from machine learning models on request. Shortly after, inexact schemes were proposed to mitigate the impractical costs associated with exact unlearning. More recently unlearning is often discussed as an approach for removal of impermissible knowledge i.e. knowledge that the model should not possess such as unlicensed copyrighted, inaccurate, or malicious information. The promise is that if the model does not have a certain malicious capability, then it cannot be used for the associated malicious purpose. In this paper we revisit the paradigm in which unlearning is used for in Large Language Models (LLMs) and highlight an underlying inconsistency arising from in-context learning. Unlearning can be an effective control mechanism for the training phase, yet it does not prevent the model from performing an impermissible act during inference. We introduce a concept of ununlearning, where unlearned knowledge gets reintroduced in-context, effectively rendering the model capable of behaving as if it knows the forgotten knowledge. As a result, we argue that content filtering for impermissible knowledge will be required and even exact unlearning schemes are not enough for effective content regulation. We discuss feasibility of ununlearning for modern LLMs and examine broader implications.
Introduction
The paper examines unlearning as a way to remove impermissible knowledge from LLMs and identifies an inconsistency: in-context learning can reintroduce knowledge that unlearning removed. It focuses on content regulation, not unlearning’s original privacy use case.
Introduction
- Unlearning has been applied to removing harmful capabilities and responses, backdoors, topic-specific knowledge, copyrighted content, and hallucinations.Related work also explores removing unsafe concepts from diffusion models.Source: Introduction · p. 1
- The paper focuses on using unlearning to remove broadly impermissible knowledge, including knowledge discussed in policy contexts such as biological and nuclear knowledge.The authors distinguish this content-regulation use from unlearning for privacy.Source: Introduction · pp. 1–2
- In-context learning can reintroduce successfully unlearned knowledge, challenging unlearning as a way to prevent impermissible behavior.The paper questions unlearning’s long-term viability as a primary content-regulation tool.Source: Introduction · p. 1
Nomenclature
The section defines unlearning and in-context learning, then uses a bomb-making example to show how contextual information can restore behavior despite unlearning. It also notes that reasoning about knowledge interactions and attributing behavior to introduced information can be difficult.
Nomenclature
- Unlearning for content regulation targets knowledge associated with impermissible content, which need not correspond to a subset of the training dataset.Unlearning is broadly described as removing knowledge from a model; privacy unlearning instead targets a defined subset of the original training data and distinguishes exact from approximate methods.Source: Nomenclature · p. 2
- In-context learning is an emergent capability to generalize from task descriptions even when those tasks were absent from training data.The passage notes that this capability is ubiquitous, though some tasks cannot be perfectly solved from descriptions alone.Source: Nomenclature · p. 2
- Even if bomb-defining data is removed, a renamed contextual definition and sufficient chemical knowledge can let a reasoning-capable model derive a bomb recipe.The paper argues that exact unlearning would not prevent the act in this example, and that imperfect unlearning can exacerbate the issue.Source: Nomenclature · p. 2
- Because knowledge interactions are hard to reason about, it may be unclear which inferences benign knowledge enables and which introduced information should be attributed for model behavior.The paper identifies attribution of behavior to specific user-introduced knowledge as nontrivial.Source: Nomenclature · pp. 2–3
Types of Knowledge
The section distinguishes facts and assumptions from derivative knowledge and illustrates how concepts can be built from axioms. Unlearning an impermissible concept may leave shared axioms intact, while removing those axioms can reduce utility on permissible tasks.
Types of Knowledge
- The paper distinguishes axioms and theorems, which represent facts and assumptions, from derivative knowledge.Its example builds concepts from six axioms, including Ear, Eye, Tail, Big, Striped, and Gallops.Source: Types of Knowledge · p. 3
- Perfectly unlearning the impermissible Tiger concept can leave its underlying axioms available because they also support other theorems, while removing them can impair permissible Zebra reasoning.The example shows a tension between removing shared knowledge and retaining utility on other tasks.Source: Types of Knowledge · p. 3
- Conflicting objectives can leave impermissible knowledge in a model, such as retaining high-school chemistry capability while seeking to prevent bomb-making knowledge.The section points to a bomb-making example as an instance of this conflict.Source: Types of Knowledge · p. 3
UnUnlearning
The paper frames unlearning as targeting developer-identified impermissible knowledge, then argues that in-context learning can reintroduce knowledge even after it has been removed. It calls this process UnUnlearning and says the concern applies even to exact unlearning.
UnUnlearning
- The setup treats impermissible knowledge as developer-defined and identified, with an unlearning method mapping model M to an unlearned model M̂.Source: UnUnlearning · p. 3
- UnUnlearning is the in-context instruction of knowledge that was previously removed or never learned.Source: UnUnlearning · p. 3
- The authors argue that ununlearning should be considered when designing schemes to remove impermissible functionality and when models are openly released.Source: UnUnlearning · p. 3
- The concern applies even to exact unlearning.Source: UnUnlearning · p. 3
Discussion
The discussion argues that unlearning alone cannot ensure models avoid reasoning about impermissible knowledge when context can reintroduce it. It considers filtering, the limits of traditional unlearning definitions, and who should be responsible for enforcing content policy.
Need for Effective Filtering Mechanisms
- Unlearning must be paired with continuous, active suppression of in-context attempts to reintroduce impermissible knowledge, though context dependence may constrain filtering.The authors cite computational bounds related to query context.Source: Discussion · p. 3
Definitions and Mechanisms for Unlearning
- Under classic definitions, a model never trained on Tiger knowledge counts as perfectly unlearned, yet it can reason about Tiger when the concept is introduced in context.The discussion calls for mechanisms that limit reasoning in ways invariant to prompting and other types of learning.Source: Discussion · pp. 3, 5
Discussion
- Preserving the axioms underlying a concept can make it relatively easy to redefine that concept even after unlearning its theorem.Figure 1 distinguishes given facts (axioms) from derived knowledge (theorems), with some axioms shared across theorems.Source: Figure 1 · p. 4
Attributing knowledge
- Because knowledge can be introduced by different parties, unlearning should not be assumed to be the sole mechanism for content-policy enforcement.The authors frame attribution as a question spanning the executor, the person giving an order, the tool manufacturer, and the tool designer.Source: Discussion · p. 5
Forbidding knowledge
- Teaching a model that knowledge is off limits requires anticipating harmful use-cases, may not resist mosaic attacks, and does not guarantee prevention beyond identified use-cases.The authors also note that rejecting a request can help a malicious user understand which recipes could be used for malice.Source: Discussion · p. 5
Conclusion
The conclusion argues that unlearning is incomplete for removing impermissible knowledge from LLMs with strong in-context learning capability, and emphasizes content filtering.
Conclusion
- For LLMs with strong in-context learning capability, unlearning is incomplete for removing impermissible knowledge, placing emphasis on content filtering.Source: Conclusion · p. 5