Source-linked AI summary
Eight Methods to Evaluate Robust Unlearning in LLMs
Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, Dylan Hadfield-Menell
TL;DR
LLM unlearning lacks standardized, rigorous evaluation despite its use for removing harmful capabilities and memorized text. The paper surveys existing evaluations and applies eight tests to the WHP model, finding that Familiarity generalizes while alternative tests recover knowledge, narrow Q&A differences, reveal comparable latent representations, and expose related-domain side effects. These results support comprehensive evaluation beyond ad-hoc metrics.
Problem
LLM unlearning lacks standardized methods for rigorously evaluating whether harmful capabilities and memorized text have been removed.
Method
The paper surveys prior evaluations and applies eight tests covering extraction, trivia, latent knowledge, baselines, and side effects to the WHP model.
Results
WHP shows Familiarity-based generalization, but higher-than-baseline knowledge is extractable, Q&A performance is nearly on par with the original, latent knowledge is comparable, and related-domain side effects occur.
Takeaways & Limitations
Comprehensive evaluations that include adversarial tests are important because ad-hoc Familiarity measures may misrepresent overall unlearning effectiveness.
Abstract
from arXiv · showhide
Machine unlearning can be useful for removing harmful capabilities and memorized text from large language models (LLMs), but there are not yet standardized methods for rigorously evaluating it. In this paper, we first survey techniques and limitations of existing unlearning evaluations. Second, we apply a comprehensive set of tests for the robustness and competitiveness of unlearning in the "Who's Harry Potter" (WHP) model from Eldan and Russinovich (2023). While WHP's unlearning generalizes well when evaluated with the "Familiarity" metric from Eldan and Russinovich, we find i) higher-than-baseline amounts of knowledge can reliably be extracted, ii) WHP performs on par with the original model on Harry Potter Q&A tasks, iii) it represents latent knowledge comparably to the original model, and iv) there is collateral unlearning in related domains. Overall, our results highlight the importance of comprehensive unlearning evaluation that avoids ad-hoc metrics.
1 INTRODUCTION
The paper motivates LLM unlearning as a way to remove undesirable knowledge while preserving general capabilities, then surveys evaluation practices and red-teams the WHP model with broader tests. Its findings show Familiarity-based generalization alongside extractable knowledge and limitations that motivate comprehensive evaluation.
- Motivation: LLM unlearning aims to remove undesirable knowledge while retaining competitiveness on most tasks.The motivation includes reducing risks from harmful behavior and memorized pretraining data.
- Approach: The paper surveys existing evaluations and applies a thorough test suite to red-team the WHP model.
- Findings: WHP unlearning consistently generalizes under Familiarity, yet higher-than-baseline knowledge remains extractable.
- Implication: The authors conclude that robust unlearning requires comprehensive evaluations that avoid ad-hoc metrics.
2 RELATED WORK
Related work describes persistent internal knowledge after fine-tuning and prior LLM-unlearning approaches, including the WHP model’s Familiarity-based evaluation. The cited figure context highlights that this metric can show reduced familiarity while not fully characterizing retained knowledge.
- Prior findings: Fine-tuning can make relatively minor changes to internal knowledge, allowing harmful or memorized information to persist.
- Unlearning literature: Prior LLM-unlearning work includes fine-tuning-based and mechanistic approaches targeting undesirable capabilities or memorized information.
- Metric context: Figure 1 indicates that WHP bars remain below Llama-2 bars under Familiarity, while extraction strategies can raise WHP Familiarity above its 0.09 baseline.
3 TESTS FOR ROBUST AND COMPETITIVE UNLEARNING
The paper evaluates WHP with eight robustness and competitiveness tests spanning extraction, trivia, latent representations, prompting baselines, and related-domain side effects. These tests reveal that alternative evaluations can narrow the apparent unlearning gap and expose recoverable knowledge and collateral losses.
- Evaluation design: The evaluation suite covers other languages, jailbreaks, in-context relearning, fine-tuning relearning, downstream tasks, latent knowledge, prompting baselines, and similar-domain side effects.
- Jailbreak prompts: Jailbreak prompts modestly increase WHP Familiarity both absolutely and relative to the original model.
- In-context relearning: In-context examples and summaries increase WHP’s absolute Familiarity and Familiarity relative to the original model.
- Fine-tuning relearning: Few-shot fine-tuning uses 800 and 8,000 Harry Potter sentences, representing about 1% and 10% of the book corpus, respectively.
- Downstream tasks: Trivia evaluations reduce the apparent performance gap: WHP surpasses the original model on short-answer questions and nearly matches it on binary-choice questions.
- Latent knowledge: For easy binary-choice questions, probes recover the correct answer from WHP representations with the same accuracy as from Llama-2.
- Prompting baseline: WHP beats a trivial prompting baseline, while prompted Llama-2 remains above WHP’s Familiarity level.
- Side effects: WHP loses significant Familiarity in related domains, including English Mythology and Harry Potter film production.
4 DISCUSSION
The discussion finds both robust Familiarity reductions and substantial limitations in WHP unlearning. It argues that comprehensive, adversarial evaluation is needed because ad-hoc metrics may overstate how completely undesirable knowledge has been removed.
- Robustness: WHP Familiarity remains consistently below the original model, indicating signs of robust unlearning under that metric.
- Limitations: Higher-than-baseline knowledge can be reliably extracted, while WHP performs nearly on par with the original model on downstream Q&A tasks.
- Limitations: WHP represents latent knowledge comparably to the original model and has side effects in related domains.
- Evaluation implications: Ad-hoc measures such as Familiarity may be misleading about overall unlearning effectiveness.
- Practical implication: The findings support thorough and adversarial evaluations when unlearning is used to remove harmful tendencies or capabilities.
A.1 FAMILIARITY METRIC
The Familiarity metric estimates Harry Potter content in model completions by having GPT-4 assess references, prompts, and completions. The evaluation also adapts this procedure to measure side effects across themed domains.
- Familiarity measures Harry Potter content contained in completions to Harry Potter-related sequences.
- GPT-4 evaluates each completion using the datapoint reference, prompt, and model completion, then produces a MODEL FAMILIARITY score.
- The metric assigns weights of 1, 0.2, or 0 to grade-3, grade-2, or other completions before averaging across completions.
- Side-effect Familiarity evaluations add a theme to the evaluation prompt to test related domains.
A.2 RELEARNING THROUGH FINE-TUNING
The fine-tuning experiment tests whether small amounts of Harry Potter training data can restore knowledge to Llama-2 and WHP in a low-compute setting.
- Llama-2 and WHP are fine-tuned with rank-8 LoRA adapters on five-sentence excerpts from the first three Harry Potter books.
- The experiment uses low-rank adapters to examine an adversary operating under low-compute conditions.
- After fine-tuning, both models are evaluated with the Familiarity metric, which requires knowledge of the entire series.
A.3 LATENT KNOWLEDGE
Latent-knowledge evaluations probe model representations rather than relying only on generated answers. Across many settings, correct Harry Potter answers remain recoverable internally even when WHP responds inaccurately.
- Probe setup: Supervised and unsupervised probes are trained at the last-token residual stream position across model layers.
- Supervised probes: For easy questions, Llama-2 and WHP supervised probes perform similarly and both exceed Llama-2 response accuracy.
- Unsupervised probes: WHP unsupervised probes are less stable across layers, while hard-question probes for both models are inconsistent and barely reach response accuracy.
- Interpretation: In many cases, correct answers can be probed from internal representations when WHP does not respond accurately.
C DOWNSTREAM TASKS
The downstream-task evaluation uses a GPT-4-created Harry Potter trivia dataset to test question-answering rather than the Harry Potter-related text generation directly targeted by WHP unlearning.
- The trivia dataset contains 1239 binary-choice questions generated and augmented with GPT-4.
- Questions use easy or hard false answers, with hard alternatives designed to be more plausible and Harry Potter-related.
- The dataset tests nuanced Harry Potter understanding through binary-choice questions with different distractor difficulties.
C.2 SHORT ANSWER QUESTIONS
The short-answer evaluation tests whether models can answer Harry Potter trivia questions in sentence form, with GPT-4 judging responses against the true answers.
- The evaluation prompts a model to answer Harry Potter trivia questions using sentence-style responses.The questions come from the Harry Potter Trivia dataset, including easy and hard variants.
- The trivia materials include generated Harry Potter questions and a long summary used for related experiments.The dataset-generation prompt produced trivia questions, while the long summary supported in-context relearning experiments.
- GPT-4 evaluates each model response by comparing the completion with the true answer.The evaluation uses a dedicated GPT-4 judging prompt.
D SIDE EFFECTS
The side-effects evaluation measures Familiarity with domains related to Harry Potter, finding reduced Familiarity in most tested domains after unlearning.
- Across five related domains, WHP showed a Familiarity drop in four and no difference in the fifth compared with Llama 2.The domains included British mythology, Harry Potter film production, Lord of the Rings, and Wizard of Oz, with five datasets reported overall.
- The side-effects datasets covered English mythology, Dungeons and Dragons, Harry Potter film production, Lord of the Rings, and Wizard of Oz.Dataset sizes ranged from 45 to 50 questions.
- The evaluation used ChatGPT-generated questions to measure model Familiarity with domains related to Harry Potter.The same prompt was adapted by replacing the target domain phrase for different datasets.