Source-linked AI summary
Who's Harry Potter? Approximate Unlearning in LLMs
Ronen Eldan, Mark Russinovich
TL;DR
LLMs may memorize copyrighted material, raising legal and ethical concerns and creating a need for selective unlearning without full retraining. The paper introduces an approximate unlearning technique combining target-aware token identification with generic prediction labels, then evaluates it on Harry Potter content in Llama2-7b. The approach removes most detectable familiarity while largely preserving common benchmark performance, but its evaluation may miss adversarial extraction and its effectiveness may depend on external knowledge and content type.
Problem
LLM corpora often contain copyrighted content, while selective post-training unlearning remains difficult and complete retraining is resource-intensive.
Method
The technique identifies target-related tokens with a reinforced model, replaces idiosyncratic expressions with generic counterparts to create alternative predictions, and finetunes on those labels.
Results
The method enables Llama2-7b to essentially forget Harry Potter narratives while retaining established benchmark performance; ablations either failed to reduce familiarity sufficiently or harmed benchmark scores.
Takeaways & Limitations
Approximate unlearning appears feasible for this generative-model proof of concept, supporting further work on post-training adaptation for ethical, societal, or user-specific requirements.
Takeaways & Limitations
Prompt-completion evaluation may miss adversarial extraction, and the method’s efficacy may depend on GPT-4 knowledge and may be limited for non-fiction or textbooks.
Abstract
from arXiv · showhide
Large language models (LLMs) are trained on massive internet corpora that often contain copyrighted content. This poses legal and ethical challenges for the developers and users of these models, as well as the original authors and publishers. In this paper, we propose a novel technique for unlearning a subset of the training data from a LLM, without having to retrain it from scratch. We evaluate our technique on the task of unlearning the Harry Potter books from the Llama2-7b model (a generative language model recently open-sourced by Meta). While the model took over 184K GPU-hours to pretrain, we show that in about 1 GPU hour of finetuning, we effectively erase the model's ability to generate or recall Harry Potter-related content, while its performance on common benchmarks (such as Winogrande, Hellaswag, arc, boolq and piqa) remains almost unaffected. We make our fine-tuned model publicly available on HuggingFace for community evaluation. To the best of our knowledge, this is the first paper to present an effective technique for unlearning in generative language models. Our technique consists of three main components: First, we use a reinforced model that is further trained on the target data to identify the tokens that are most related to the unlearning target, by comparing its logits with those of a baseline model. Second, we replace idiosyncratic expressions in the target data with generic counterparts, and leverage the model's own predictions to generate alternative labels for every token. These labels aim to approximate the next-token predictions of a model that has not been trained on the target data. Third, we finetune the model on these alternative labels, which effectively erases the original text from the model's memory whenever it is prompted with its context.
1 Introduction
The paper addresses whether LLMs can selectively unlearn problematic training data without complete retraining. Using Llama2-7b, it demonstrates Harry Potter unlearning while preserving common benchmark performance, though the technique may have limitations beyond this setting.
- LLM training corpora can contain copyrighted, toxic, malicious, inaccurate, fake, and personal data, creating ethical and legal complications.
- Selective unlearning after training remains difficult because conventional finetuning reinforces knowledge, while complete retraining is time-consuming and resource-intensive.
- The proposed technique applies to Meta’s Llama2-7b and targets forgetting the detailed narratives of the Harry Potter series without sacrificing established benchmark performance.
- After roughly 30 minutes of finetuning on 4 A100 GPUs, the fine-tuned model is compared with the baseline through prompt completions, benchmark performance, and next-token probabilities.
- Figure 3 tracks the most likely continuation of “Harry Potter studies” shifting from “magic” toward generic completions during finetuning.
- The authors present the method as an initial step toward more adaptable LLMs, while warning that it may be less effective for non-fiction or textbooks.
2 Description of our technique
The technique approximates training on data with the unlearn target removed by generating generic next-token predictions and fine-tuning the original text toward them. It combines reinforcement-based token identification with generic replacements to weaken Harry Potter-specific associations while preserving broader language behavior.
- The objective is to approximate retraining on X \ Y, where Y is the unlearn target, without incurring the cost of full retraining.The target is a subset of the original training data that the model should approximately forget.
- Naive reversed-loss training can damage general language knowledge because many correctly predicted tokens are unrelated to Harry Potter content.For example, penalizing a prediction such as “Harry” could effectively unlearn the meaning of the preceding phrase “my name is.”
- Generic predictions represent what a model not trained on the Harry Potter books would predict as the next token in the same sentence.The technique uses these predictions as alternative labels for fine-tuning.
- 2.1 Obtaining generic predictions via reinforcement bootstrapping: Reinforcement bootstrapping compares baseline and reinforced-model logits, selecting tokens whose probabilities increased and using the maximal adjusted logit as the generic prediction.The modified formula applies ReLU so that only increases in reinforced logits contribute to the adjustment.
- 2.1 Obtaining generic predictions via reinforcement bootstrapping: This bootstrapping method can fail when reinforced probabilities reorder competing completions or when target-specific completions are already highly probable in the baseline.In the first case, the adjustment may increase the probability of “Ron”; in the second, reinforcement changes little.
- 2.3 Combining it all together: Fine-tuning uses the original text as input and generic labels as targets, pushing the model away from Harry Potter-related continuations even in contexts containing character names.The loss is computed using the full preceding original context for each target token.
3 Evaluation methodology
The evaluation tests both whether targeted Harry Potter knowledge is erased and whether general capabilities remain intact. It combines black-box prompting, next-token probability inspection, and standard benchmarks, with community probing encouraged because automated checks have limits.
- The evaluation measures both eradication of targeted knowledge and preservation of general model capabilities.
- WinoGrande, HellaSwag, and PIQA benchmark performance is used to assess whether unrelated capabilities remain intact.
- Black-box tests use prompts designed to elicit direct and indirect knowledge of the Harry Potter universe.
- 300 prompts, generated with GPT-4 and manually inspected for final training, support the prompt-based evaluation.
- A complementary test checks whether next-token probabilities favor Harry Potter-specific terms or generic alternatives after prompts such as “Harry Potter studies”.
- Because automated and internal evaluations have intrinsic limits, the model was open-sourced for broader adversarial testing.
4 Results
The technique was evaluated on Llama2-7b and produced qualitatively similar results in a Phi-1.5 setting, while reducing detectable Harry Potter familiarity with limited observed leakage. Ablations show that combining the technique’s components is important for maintaining benchmark performance.
- The method was tested on Meta’s 7B Llama model and a modified 1.3B Phi-1.5 model, with qualitatively similar results; only the former is presented.
- Figure 5 reports ARC, BoolQ, HellaSwag, OpenBookQA, PIQA, WinoGrande, and familiarity scores across multiple fine-tuning steps.
- Most benchmark prompts showed no familiarity trace, although a small number of leaks remained, including “Hogwarts” in fictional-school lists.
- The remaining leaks revealed Wikipedia-level knowledge rather than the books’ detailed content, while the original model showed thorough book knowledge.
- The authors did not access the original training data, and the unlearn target excluded non-book Harry Potter information such as merchandise and theme parks.
- 4.1 Ablation study: Without anchoring, reinforcement bootstrapping reduced the completion-based familiarity score by no more than a factor of 0.3 and failed on basic prompts.
- 4.1 Ablation study: Anchored terms alone matched familiarity but harmed common benchmarks, including arc-challenge 0.40, arc-easy 0.70, and boolq 0.79.
5 Conclusion
The paper presents selective unlearning as feasible in this proof of concept, while stressing limits in evaluation, content generality, and reliance on distinctive fictional terminology and GPT-4 knowledge. Broader applicability remains untested.
- The experiments suggest that selective LLM unlearning is challenging but not insurmountable in the Llama2-7b proof of concept.
- Prompt-based completion evaluation may miss adversarial ways of extracting information from the model.
- Harry Potter’s distinctive names and expressions may have helped the strategy, limiting conclusions about other content.
- The methodology relies on GPT-4 knowledge for detecting anchored terms and creating generic counterparts, although preliminary experiments suggest extraction can work without it.
- Non-fiction and textbooks pose additional challenges because they contain fewer unique phrases and more abstract ideas, concepts, and cultural perspectives.
- The technique’s applicability across content types remains insufficiently tested, so further research is needed to refine and extend it.
6 Appendix
The appendix describes familiarity evaluations using completion-based and next-token probability-based prompts, with figures illustrating model completions and probability dynamics.
- 6.2.1 Completion-based familiarity: 300 completion-based prompts were generated from random 300-word book excerpts and evaluated by GPT-4 using four familiarity categories.The categories distinguish unique book details, thematic but non-unique completions, possible lucky guesses, and no familiarity.
- 6.2.1 Completion-based familiarity: The evaluation score counted unique-detail and thematic completions, assigning five times the weight to unique details.
- 6.2.1 Completion-based familiarity: Figure 6 provides further baseline-versus-fine-tuned completion examples, including a small number of benchmark leaks such as “Hogwarts.”
- 6.2.2 Probability-based familiarity: The probability-based evaluation manually selected 30 prompts whose next tokens could reveal familiarity and separated likely idiosyncratic tokens from generic ones.The score totals the model’s probability assigned to idiosyncratic tokens.
- 6.2.2 Probability-based familiarity: Figures 7–10 show next-token probability dynamics for prompts involving Dumbledore, forehead, Gringotts Bank, and a newspaper completion.
- 6.2.1 Completion-based familiarity: Figures 11 and 12 document the instructions used to create completion-based prompts and classify model completions.