PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Unlearning Methods Remove Information from Language Model Weights?

Aghyad Deeb, Fabien Roger

arXiv:2410.08827v3cs.LG

TL;DR

It has been unclear whether unlearning removes knowledge from model weights or merely makes it harder to access, despite concerns about dangerous capabilities in language models. The paper evaluates unlearning through recovery of hidden information and finds that substantial pre-unlearning accuracy can be recovered for tested methods.

  • Problem

    Output-based evaluations do not determine whether unlearning removes knowledge from model weights, a distinction relevant to concerns about dangerous capabilities in language models.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces a framework and datasets to quantify hidden knowledge by evaluating recovery after unlearning.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    At least 88% of pre-unlearning accuracy was recovered for every evaluated method on pretraining information when non-unlearned task performance remained good.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors recommend focusing on methods that remove information from models, reporting that tested methods removed only a small ratio of information, especially when retain accuracy remained good.

    Source: §8 · p. 11

    The evaluation does not guarantee removal: information hidden in separate parts of the model may remain unrecovered by RTT.

    Source: §7.1 · p. 10

Abstract

from arXiv · show

Large Language Models' knowledge of how to perform cyber-security attacks, create bioweapons, and manipulate humans poses risks of misuse. Previous work has proposed methods to unlearn this knowledge. Historically, it has been unclear whether unlearning techniques are removing information from the model weights or just making it harder to access. To disentangle these two objectives, we propose an adversarial evaluation method to test for the removal of information from model weights: we give an attacker access to some facts that were supposed to be removed, and using those, the attacker tries to recover other facts from the same distribution that cannot be guessed from the accessible facts. We show that using fine-tuning on the accessible facts can recover 88% of the pre-unlearning accuracy when applied to current unlearning methods for information learned during pretraining, revealing the limitations of these methods in removing information from the model weights. Our results also suggest that unlearning evaluations that measure unlearning robustness on information learned during an additional fine-tuning phase may overestimate robustness compared to evaluations that attempt to unlearn information learned during pretraining.

1 INTRODUCTION

Dangerous knowledge may persist in model weights even when output-based unlearning evaluations suggest it is inaccessible. The paper proposes an adversarial recovery evaluation and reports substantial recovery for tested methods.

1 INTRODUCTION

  • The proposed evaluation retrains on accessible facts independent of the facts used for evaluation, then tests recovery of the latter.Figure 1 depicts this setup and reports recovery of a large fraction of performance for methods such as RMU.Source: Figure 1 · p. 1, §1 · p. 2
  • Output-based evaluations do not establish whether knowledge has been removed from weights, and prior work recovered information after unlearning, including from RMU via jailbreaks.The paper calls for robust evaluations to guide unlearning research.Source: §1 · p. 2
  • At least 88% of pre-unlearning accuracy was recovered for every evaluated method on pretraining information when non-unlearned task performance remained good.The methods evaluated included Gradient Ascent, RMU, and training on incorrect facts.Source: §1 · p. 2
  • The paper compares unlearning information learned during pretraining with unlearning information learned during an additional fine-tuning phase.The latter is sometimes used in unlearning evaluations, while pretraining is described as the primary use case.Source: §1 · p. 2
  • The authors also stress-test their framework against a harder case for recovering hidden knowledge.Source: §1 · p. 2

2 RELATED WORK

Prior unlearning evaluations include output-based measures, attacks, and white-box approaches, each with stated limitations. The paper situates its approach among these methods and prior evidence of weaknesses in current unlearning techniques.

Refusal in LLMs

  • RLHF is used to mitigate harmful behaviors, but the cited work reports vulnerabilities to jailbreaks, in-context learning attacks, few-shot fine-tuning, and unforeseen misbehavior.The passage attributes RLHF to Christiano et al. (2023) and lists the respective attacks and examples.Source: §2 · p. 2

Unlearning in LLMs

  • Gradient Ascent increases loss on information to be unlearned, while RMU perturbs activations for harmful prompts and preserves them for non-harmful prompts.Eldan & Russinovich (2023) estimate model outputs as if Harry Potter-related data had not been seen and train on those estimates.Source: §2 · p. 2

Black-box unlearning evaluations

  • Output-based evaluations measure a model’s propensity to use unlearned knowledge rather than whether that knowledge remains hidden in the weights.Jailbreak robustness is difficult to assess exhaustively, and membership inference measures training-data membership rather than absence of a particular fact.Source: §2 · pp. 2–3

White-box unlearning evaluations

  • Linear probes may miss information in weights, while relearning-time and limited-sample methods lack a reliable baseline and can be hindered by techniques that slow fine-tuning.Relearning time counts fine-tuning steps to recover accuracy; limited-sample relearning uses a restricted number of samples.Source: §2 · pp. 3–4

Weaknesses of current unlearning techniques

  • Prior work found current unlearning techniques vulnerable to attacks that would fail if the information had been removed, and this paper reports further confirmation.The passage says negative results using this method provide stronger evidence of unsuccessful knowledge removal than prompting-based methods.Source: §2 · p. 4

3 PROBLEM STATEMENT

The section treats successful unlearning as removal of answer-related information from weights, then proposes an adversarial evaluation of whether that information remains recoverable. The evaluation relies on independent facts and retraining on one subset to test another.

3.1 UNLEARNING AS REMOVING INFORMATION FROM THE WEIGHTS

  • Full removal means post-unlearning weights are not predictably different depending on the answer to the target question, apart from training stochasticity.Formally, the passage defines full removal as mutual information of zero between the unlearned weights and the answer.Source: §3.1 · p. 4
  • This formalization applies to questions whose answers are practically impossible to guess from general information.The passage contrasts the WHO founding-year question with guesses that can be ruled out using basic historical intuition.Source: §3.1 · p. 4

3.2 ESTIMATING THE PRESENCE OF INFORMATION

  • Developers split independent, low-leakage facts into training and validation sets T and V; an attacker uses the weights and T to try to recover V.Recovery of V indicates facts were hidden rather than removed; the objective is to minimize recovered accuracy on V.Source: §3.2 · pp. 4–5
  • Retraining on T, or RTT, uses accuracy on V to approximate residual mutual information: high accuracy implies it is high, while near-chance accuracy suggests it is low.Training on T may make hidden facts easier to access without teaching the model those facts again; the evaluation requires T and V to share minimal information.Source: §3.2 · p. 5

4 EXPERIMENTAL SETUP

The experiments use datasets designed to test recovery of independent, pre-existing facts and compare three unlearning methods. Retraining on training facts (RTT) then tests whether held-out facts remain recoverable.

4.1 DATASETS

  • The dataset criteria favor independent facts, strong pre-unlearning model performance, and resemblance to practical unlearning targets.The forget set contains facts targeted for unlearning, while the retain set contains facts the model should continue to know.Source: §4.1 · p. 5
  • The datasets cover pretrained facts in Years, MMLU, and WMDP-Deduped, plus fine-tuned Random Birthdays; facts are split into RTT training and validation sets.Each dataset uses plain text and multiple-choice formats; unlearning uses plain text while RTT and evaluation use multiple choice.Source: §4.1 · pp. 5–6

4.2 UNLEARNING

  • The methods are Gradient Difference, RMU, and Random Incorrect Answer, with unlearning strength tuned to balance forget accuracy against retain performance.The main experiments limit retain-accuracy drops to 5% or less; Gradient Difference combines ascent on forget data with descent on retain data.Source: §4.2 · p. 6

4.3 RETRAINING ON T AND EVALUATION

  • RTT fine-tunes on four dataset splits and evaluates on a held-out split, selecting the learning rate with the highest validation accuracy.The reported Figure 2 comparison shows forget accuracies before and after RTT with retain-accuracy drops limited to 5% or less.Source: §4.3 · p. 6, Figure 2 · p. 7

5 RESULTS

For pretrained information, RTT largely restores accuracy after unlearning, indicating that the information remained recoverable. Results differ for fine-tuned information and vary with the format of the unlearning data.

5.1 UNLEARNING PRETRAINED INFORMATION

  • Recovery rates for pretrained information exceeded 88%, and RTT restored forget accuracy close to its original level across Years, MMLU, and WMDP-Deduped.Recovery Rate is the unlearned model’s accuracy on V after RTT divided by the original model’s accuracy on V after RTT.Source: §5.1 · p. 7
  • RTT recovered pretrained forget accuracy even with large retain-accuracy losses and often without retain loss, suggesting retain performance did not constrain removal.Figure 3 reports high forget accuracy after RTT even when retain accuracy falls to 25%.Source: §5.1 · pp. 7–8, Figure 3 · p. 8

5.2 UNLEARNING FINE-TUNED INFORMATION

  • For fine-tuned Random Birthdays, RMU still had a high recovery rate, while GD and RIA had recovery rates below 35%.The authors tested only one dataset with fine-tuned information and leave robustness of unlearning such information for future work.Source: §5.2 · p. 8

5.3 FORMAT OF UNLEARNING DATASETS

  • GPT-4o-generated plain text gave the best performance balance across methods and generalized to MCQ datasets.RMU performed better when unlearning data was related but did not necessarily share facts with RTT and evaluation; GD performed best with aligned MCQ formats and answer-only loss.Source: §5.3 · p. 8

6 STRESS-TESTING RETRAINING ON T

The stress tests examine whether RTT can recover knowledge hidden at high granularity and whether it works against techniques intended to slow fine-tuning.

6.1 HIGH GRANULARITY KNOWLEDGE HIDING

  • RTT raised forget accuracy from 35% to 83% after GD unlearning, which had achieved 82% retain accuracy.The test used randomly generated birthdays, with retain facts drawn from the same distribution as forget facts; the authors note transfer to natural hidden knowledge is unclear.Source: §6.1 · p. 10

6.2 RETRAINING ON T VS TECHNIQUES TO SLOW DOWN FINE-TUNING

  • RTT successfully recovered information against a tested technique designed to make fine-tuning on specific information difficult.The paper points to Appendix C for further details.Source: §6.2 · p. 10

7 DISCUSSION

The discussion identifies limits of RTT’s cost and coverage, and distinguishes removing information from merely making it harder to access. It recommends clarifying that goal, testing recovery attacks on pretrained information, and releasing models and code.

7.1 LIMITATIONS

  • RTT requires fine-tuning with a hyperparameter search, and evaluating recovery with methods beyond RTT requires still more effort.This makes the evaluation more expensive than simple benchmark accuracy checks.Source: §7.1 · p. 10
  • RTT raises the bar for unlearning evaluation but cannot guarantee removal, particularly if each fact is hidden in a separate part of the model.The authors say RTT works well for current unlearning methods.Source: §7.1 · p. 10
  • The evaluation requires low-leakage facts, a condition that may be unavailable for some goals and is hard to meet for capability targets such as coding.The authors state that this also leaves unlearning capabilities outside the evaluation’s coverage.Source: §7.1 · p. 10
  • The authors distinguish removing information from making it harder to access, noting that safety may require only the latter and that future work may find less conservative targets.They suggest the alternative of removing information likely provides better safety guarantees.Source: §7.1 · p. 10

7.2 RECOMMENDATIONS

  • The recommendations are to state whether a method removes or hides information, test recovery attacks on pretrained information, and release applied models and code.The authors caution that evaluations on information learned during fine-tuning may not transfer to realistic settings and can overestimate robustness.Source: §7.2 · p. 11

8 CONCLUSION

The conclusion reports that tested methods remove only a small ratio of information, especially when they preserve retain accuracy. The appendices describe RTT’s hyperparameter procedure and find that MCQ Question-Letter-Answer loss best recovers hidden information.

8 CONCLUSION

  • The tested unlearning methods remove a small ratio of information, especially when they maintain good retain accuracy.The conclusion frames RTT as a way to evaluate removal rather than merely making information harder to access.Source: §8 · p. 11

A RTT HYPERPARAMETERS

  • RTT uses the same hyperparameter configurations across unlearning techniques and reports the maximum accuracy across six epochs.The number of iterations reflects different T and V splits; the uncertainty bound uses 157 evaluation datapoints and doubles the datapoint count because RTT runs twice for different T and V choices.Source: Table 2 · p. 15, A RTT HYPERPARAMETERS · p. 15

B RTT LOSS

  • MCQ Question-Letter-Answer loss performed best at recovering hidden information across the tested unlearning methods and datasets.The compared loss types include losses on the answer letter, the letter followed by the answer, and plain text in a different format from unlearning.Source: B RTT LOSS · p. 15

C FINE-TUNING ROBUST METHODS

RTT with an extensive hyperparameter search recovered accuracy comparable to fine-tuning the original model despite RepNoise. Applying unlearning loss only to relevant year tokens made no significant difference from applying it to all tokens.

C FINE-TUNING ROBUST METHODS

  • With extensive hyperparameter search, RTT recovered accuracy as good as fine-tuning the original model after RepNoise.The test used a RepNoise model and a dataset split into low-leakage subsets; Figure 6 compares retraining accuracies with and without RepNoise across hyperparameters.Source: C FINE-TUNING ROBUST METHODS · p. 15, Figure 6 · p. 16
  • The authors conclude that fine-tuning robustness techniques can be bypassed by RTT when it uses an extensive hyperparameter search.They also note that Tampering Attack Resistance is vulnerable to parameter-efficient fine-tuning (PEFT).Source: C FINE-TUNING ROBUST METHODS · pp. 15–16

D LOSS ON RELEVANT TOKENS ONLY

  • Taking the unlearning loss only on relevant year tokens made no significant difference from applying it to all tokens.The comparison used Gradient Descent and RIA on a plain-text example about the WHO’s founding year.Source: D LOSS ON RELEVANT TOKENS ONLY · p. 16

E MUTUAL INFORMATION IN RANDOM BIRTHDAYS DATASET

The random birthdays test found little shared information, while RTT recovered most of the lost accuracy on a provided RMU model.

E MUTUAL INFORMATION IN RANDOM BIRTHDAYS DATASET

  • 31.2% was the highest accuracy RTT achieved on the random birthdays dataset using an original model not fine-tuned on that dataset.The authors take this as evidence that the dataset has little shared information and RTT does not increase accuracy on V for a model without knowledge of either set.Source: E MUTUAL INFORMATION IN RANDOM… · p. 16

F PROVIDED RMU MODEL

  • RTT recovered most of the lost accuracy on the provided zephyr-7b-beta RMU model.The evaluation was performed to confirm the authors’ assessment of RMU.Source: F PROVIDED RMU MODEL · p. 16

G RESULTS FOR DIFFERENT DROPS IN RETAIN ACCURACIES

The appendix reports RTT accuracies across several retain-accuracy-drop settings, alongside stress-test details, dataset construction and examples, and a correction to Random Birthdays results.

G RESULTS FOR DIFFERENT DROPS IN RETAIN ACCURACIES

H STRESS-TESTING DETAILS

  • RTT recovered information successfully without multiple trials in stress-test runs where GD unlearning succeeded after extensive hyperparameter search.The reported successful configuration used Lion, a 6.4e-6 learning rate, 11 epochs, retain coefficient 1, and loss restricted to number tokens.Source: H STRESS-TESTING DETAILS · p. 19

I WMDP VS WMDP-DEDUPED

  • WMDP-Deduped removes skill-based and duplicate questions using keyword and embedding-similarity filters, producing 785 questions versus 3,260 for WMDP-cyber/bio.The filtering used cosine-similarity thresholds of 0.7 for WMDP-cyber and 0.6 for WMDP-bio; most excluded questions were false positives.Source: I WMDP VS WMDP-DEDUPED · p. 19

J EXAMPLES OF INDEPENDENT FACTS

K DATASET EXAMPLES

L HISTORICAL NOTE

  • After duplicate name-birthday pairs were removed and replaced, RTT could not recover most information after unlearning with GD and RIA.An earlier dataset version had duplicate names with different birthdays, and RTT had recovered information after unlearning on that version.Source: L HISTORICAL NOTE · p. 22
Loading 2410.08827v3…