PaperlayerView exact revision on arXiv

Source-linked AI summary

Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning

Shengyuan Hu, Yiwei Fu, Zhiwei Steven Wu, Virginia Smith

arXiv:2406.13356v4cs.LG

TL;DR

Approximate unlearning in LLMs may suppress target outputs without robustly removing the underlying knowledge. This paper formalizes benign relearning attacks and evaluates them across unlearning tasks and benchmarks. It reports that limited relearning data can restore unlearned knowledge, including harmful or memorized content.

  • Problem

    Approximate unlearning methods have been found fragile under evolving attacks and evaluation strategies, leaving uncertainty about whether they robustly remove target knowledge.

    Source: §1 · p. 1

  • Method

    The paper formalizes attacks that finetune unlearned models with related auxiliary data, including partial unlearning data or public information, and evaluates outputs before and after relearning.

    Source: §1 · pp. 1–2, §2.1 · p. 2, §2.3 · p. 3

  • Results

    Across multiple datasets and unlearning tasks, benign relearning recovered unlearned knowledge; on WMDP, the forget average score rose from 1.27 after gradient-ascent unlearning to 6.2 after relearning on benign public text.

    Source: §4.1 · p. 6, §9 · p. 10

  • Takeaways & Limitations

    The findings indicate that current approximate unlearning methods can obfuscate outputs rather than robustly remove target knowledge, warranting caution when using them to limit sensitive or harmful generation.

    Source: §1 · p. 1, §9 · p. 10

    Model choice and unlearning method may affect attack success; in particular, LoRA-based parameter-efficient tuning was observed to make models particularly susceptible to relearning.

    Source: §7 · p. 9

Abstract

from arXiv · show

Machine unlearning is a promising approach to mitigate undesirable memorization of training data in ML models. However, in this work we show that existing approaches for unlearning in LLMs are surprisingly susceptible to a simple set of $\textit{benign relearning attacks}$. With access to only a small and potentially loosely related set of data, we find that we can ''jog'' the memory of unlearned models to reverse the effects of unlearning. For example, we show that relearning on public medical articles can lead an unlearned LLM to output harmful knowledge about bioweapons, and relearning general wiki information about the book series Harry Potter can force the model to output verbatim memorized text. We formalize this unlearning-relearning pipeline, explore the attack across three popular unlearning benchmarks, and discuss future directions and guidelines that result from our study. Our work indicates that current approximate unlearning methods simply suppress the model outputs and fail to robustly forget target knowledge in the LLMs.

1 INTRODUCTION

The paper studies whether approximate unlearning in LLMs truly removes target knowledge or merely suppresses its outputs. It finds that limited relearning data, even when insufficient to answer evaluation queries directly, can restore unlearned knowledge.

1 INTRODUCTION

  • Prior work has found approximate unlearning methods fragile under evolving attacks and evaluation strategies.The introduction cites Lynch et al. (2024), Shi et al. (2024), Maini et al. (2024b), and Jin et al. (2024).Source: §1 · p. 1
  • Current finetuning-based approximate unlearning can obfuscate outputs rather than erase target knowledge, leaving models susceptible to benign relearning.A small amount of potentially auxiliary data can jog the model toward its pre-unlearning behavior.Source: §1 · p. 1
  • Relearning can work with limited data that is insufficient to answer evaluation queries on its own, distinguishing recovery from simply learning the withheld information.The data may be publicly accessible and only loosely related to the target knowledge.Source: §1 · p. 2
  • The study formalizes the attack pipeline and evaluates it across WMDP, TOFU, and Who’s Harry Potter for harmful knowledge, memorized content, and suppressed words or phrases.The attack uses either a limited subset of unlearning data or auxiliary public data.Source: §1 · p. 2

2 BENIGN RELEARNING ATTACK

The paper defines benign relearning as finetuning an unlearned model with related auxiliary data that does not directly answer test queries, then assessing whether target knowledge returns. It considers both limited unlearning-data access and public information as sources for relearning.

2.1 PROBLEM FORMULATION AND THREAT MODEL

  • The attack assumes access to an unlearned model and finetuning, but not to the original model or the complete unlearning set.Finetuning may use open model weights or an API; inference-time in-context learning is mentioned as a possible but unverified route.Source: §2.1 · p. 2
  • Related auxiliary data may jog sensitive knowledge back into outputs even when the knowledge is absent from both the data and the unlearned model’s completions.The paper’s intuition is that greater correlation between keywords can make relearning feasible when only one keyword is relearned.Source: §2.1 · p. 2

2.2 RELEARN SET CONSTRUCTION

  • Relearn data must be related to the query but insufficient to provide its direct answer, and may come from a partial unlearning set or public information.The two settings include samples of sensitive or toxic data and benign public data loosely connected to the target knowledge.Source: §2.2 · p. 3

2.3 UNLEARNING TASKS & EVALUATION

  • The evaluation compares the original, unlearned, and relearned models on queries whose knowledge was successfully unlearned.Completion appropriateness is assessed using task-specific measures, including LLM-as-Judge, Rouge-L, or binary prediction accuracy.Source: §2.3 · pp. 3–4

3 RELEARNING ATTACK USING A PORTION OF THE UNLEARN SET

Using a limited subset of the unlearning data, relearning can recover keywords that were withheld from that subset and test set. Attack success is highest after shallow unlearning, while deeper unlearning makes recovery harder and more relearning steps do not always help.

3 RELEARNING ATTACK USING A PORTION OF THE UNLEARN SET

  • > 70% ASR is achieved on TOFU and WHP when unlearning is shallow, while deeper unlearning makes keyword reconstruction harder.In TOFU, more than 40 relearning steps lower ASR across all unlearning checkpoints.Source: Figure 3 · p. 4, §3 · p. 5
  • The attack tests recovery of a book keyword in TOFU and “Hermione” or “Granger” in WHP using relearn data that omits the target keyword.For WHP, the subset contains information about related characters but no direct knowledge of Hermione Granger’s name.Source: §3 · p. 4
  • In TOFU and WHP examples, the unlearned model fails to produce target keywords, but relearning on a subset without those keywords restores them.The resulting model can generate the same content as before unlearning.Source: §3 · p. 4, Table 1 · p. 5
  • Relearning on one disjoint portion of the unlearning set can make the model generate keywords found exclusively in the other portion.The relearn portion is assumed insufficient to infer the target keywords by itself.Source: §3 · pp. 4–5

4 RELEARNING ATTACK USING PUBLIC INFORMATION

The paper tests whether public, non-answer-bearing information can restore harmful or copyrighted content suppressed by unlearning. Across WMDP and Harry Potter experiments, relearning improves recovery of the unlearned content, including harmful answers and verbatim text.

4.1 RECOVERING HARMFUL KNOWLEDGE IN WMDP

  • 1.27 for gradient-ascent unlearning versus 6.2 after relearning on benign public text, with WMDP answer quality approaching the pre-unlearning model.This pattern held for both tested models and across unlearning objectives; standard negative log-likelihood finetuning sufficed.Source: Figure 4 · p. 6, §4.1 · p. 6
  • Public articles about general influenza knowledge led the relearned model to generate detailed virus-engineering steps absent from those articles.The unlearned model had generated low-quality, uninformative text.Source: §4.1 · pp. 6–7
  • LoRA-based unlearning was not robust; small amounts of finetuning made its forgotten content easier to recover than in full-weight unlearning.Source: Table 2 · p. 6, §4.1 · p. 7

4.2 RECOVERING VERBATIM COPYRIGHTED CONTENT IN WHP

  • Relearning on GPT-generated Harry Potter character facts improved Rouge-L scores over unlearning, especially after GA and NPO, and examples showed verbatim text recovery.The facts contained no text from the excerpt used for memorization and unlearning.Source: Figure 5 · p. 7, §4.2 · p. 7
  • Small amounts of public information can trigger forgotten completions even when that information does not directly include them.Source: §4.2 · p. 7

5 WHEN IS UNLEARNING SUSCEPTIBLE TO RELEARNING? INTUITION FROM A SIMPLIFIED EXAMPLE

The simplified experiment suggests unlearning can lower token probabilities without removing associations among parts of a sequence. Relearning one correlated token can therefore restore an unoptimized token, with recovery depending on correlation strength.

5 WHEN IS UNLEARNING SUSCEPTIBLE TO RELEARNING? INTUITION FROM A SIMPLIFIED EXAMPLE

  • Unlearning objectives lower conditional probabilities across tokens but do not remove associations, so a model may recover later tokens when it retains earlier ones.The token loss is method-specific; the passage gives GA and NPO as examples of differing loss definitions.Source: §5 · pp. 7–8
  • The toy setup memorized concatenated common names, unlearned sequences containing Anthony–Mark, then finetuned only on prefixes ending at Anthony.Source: §5 · p. 8
  • During relearning, Mark’s loss decreased alongside Anthony’s even though Mark never appeared in the relearn data, making Mark more likely after Anthony.The evaluation used 100 prompts ending in Anthony; both names’ losses had risen during unlearning.Source: §5 · p. 8, Figure 6 · p. 8
  • More repetitions of the Anthony–Mark pair made recovery easier, suggesting relearning quality depends on correlation between relearn data and target knowledge.Source: §5 · p. 8, Table 4 · p. 8

6 RELEARN TEXT RELEVANCE VS RELEARN QUALITY

Relearning works better when its data is more related to the unlearned data, though even unrelated data can offer a marginal boost.

6 RELEARN TEXT RELEVANCE VS RELEARN QUALITY

  • Relearning is more effective with stronger relatedness between relearn text and the forget set; unrelated gibberish still gives a marginal attack-success boost.Source: §6 · p. 9
  • English female names served as more related relearn text than gibberish in the synthetic experiments.Source: §6 · p. 9
  • A similar relatedness experiment was conducted on WMDP using gradient-ascent unlearning.Source: §6 · p. 9

7 DISCUSSION

Relearning performance depends on relearn-set construction and can also vary with the model, unlearning method, and evaluation metric. The discussion cautions that limited evaluation may leave developers with a false sense of security.

7 DISCUSSION

  • The study varies targeted relearn-set construction and reports that loosely related data can recover information from unlearned models.It also identifies other factors affecting relearning performance as topics for guidelines and future work.Source: §7 · p. 9
  • Relearning success may depend on the model and data form; Llama-3-8b-Instruct's QA training may mismatch WMDP's academic-article unlearning data.The authors suspect that non-QA relearning pairs may pose a similar issue for this model.Source: §7 · p. 9
  • LoRA made models particularly susceptible to relearning, while RMU relearning was more effective when the corrupted layer was known.Without knowing RMU's corrupted layer, relearning was still possible; the authors identify universally effective relearning as future work.Source: §7 · p. 9
  • For RMU, relearning was easy to demonstrate with multiple-choice evaluation but not with sentence completions.The authors leave open whether comprehensive metrics can ensure approximate unlearning.Source: §7 · p. 9
  • Limited evaluation metrics may give developers a false sense of security about whether approximate unlearning occurred.The authors call for evaluation beyond model utility on forget and retain sets.Source: §7 · p. 10

8 RELATED WORK

Prior work has examined recovery of unlearned content and relearning attacks, often using the unlearn data directly. This work instead studies whether loosely related benign data can jog unlearned models' memory.

8 RELATED WORK

9 CONCLUSION

The paper finds that benign relearning can recover unlearned knowledge and that many optimization-based heuristics do not truly remove memorized information. It also identifies limits to the studied setting and urges caution and broader evaluation.

9 CONCLUSION

  • Benign public information can recover unlearned knowledge, and findings across datasets and tasks indicate that many optimization-based heuristics fail to remove memorized information.The authors advise caution when using finetuning-based unlearning to limit sensitive or harmful generations.Source: §9 · p. 10
  • The authors call for unlearning heuristics beyond approximate gradient-based optimization and for evaluation beyond query completions on the unlearned model alone.They say query completions alone may give a false sense of unlearning quality.Source: §9 · p. 10
  • The study focuses on parameter-space attacks using the same prompt before and after relearning; prompt filtering or guardrails may mean relearning alone is insufficient.The authors note that combining relearning with jailbreak-style input corruption under those defenses remains unclear.Source: §9 · p. 10
  • The study may not cover all unlearning methods or benchmarks, and the authors encourage testing relearning attacks on additional methods and datasets.Source: §9 · p. 10

I Detailed explanation of evaluation metrics

The TOFU evaluation uses queries that supply one book title and ask the model to complete other titles. A control finetuned only on the relearn set achieved 0% attack success across authors.

I Detailed explanation of evaluation metrics

  • TOFU unlearning uses gradient-ascent LoRA on Phi-1.5 with rank 8, α 32, dropout 0.05, learning rate 1e−5, and batch size 4.LoRA relearning uses learning rate 2e−4 and batch size 8; both use AdamW and weight decay 0.01.Source: I Detailed explanation of evaluation… · p. 15
  • TOFU queries provide one book title by a fictitious author and ask the model to complete the names of other books.Source: I Detailed explanation of evaluation… · p. 15
  • 0% attack success on all authors was achieved by the model finetuned on relearn set D′ only, showing that its book information alone was insufficient to generate absent keywords.The authors use this result to verify that D′ contains no information about the keywords to be recovered.Source: I Detailed explanation of evaluation… · p. 15, Table 6 · p. 15

A.3 TOFU EVALUATION QUESTIONS

The appendix gives evaluation examples and settings for TOFU and Harry Potter keyword recovery, including exact and approximate recovery, plus WMDP relearn-set construction details. In TOFU, recovery depends on relearning duration: too few steps fail, while too many can overfit.

A.3 TOFU EVALUATION QUESTIONS

  • TOFU keyword-search evaluation counts an attack as successful if it infers any keyword from an entry.The appendix lists query-and-keyword examples for this evaluation.Source: §A.3 · pp. 15–16

A.4.1 SUCCESSFUL RECOVERY

  • For a model unlearned with 30 gradient-ascent steps, shallow relearning (far fewer steps) failed to recover the keyword, while around 30 steps recovered it exactly; 48 steps overfit toward prompt-like text.Source: §A.4.1 · p. 17

A.4.2 INEXACT RECOVERY

  • TOFU relearning sometimes recovered approximate rather than exact keywords, which the authors do not count as a successful attack.For the target “The Whisper of Hearts,” the example produced “The Whispering Heart.”Source: §A.4.2 · p. 17

B.4.2 RECOVERING FIRST NAMES GIVEN LAST NAMES

  • When the relearn set retained surnames but removed “Hermione” and “Ron,” the model produced the correct first-and-last-name order despite differing from the original completion.The relearn set retained “Granger” and “Weasley”; the example output named Hermione Granger and Ron Weasley.Source: §B.4.2 · p. 19

C.1 WMDP HYPERPARAMETERS

  • WMDP experiments report AdamW, cosine scheduling, weight decay 0.01, and batch size 4, with unlearning and relearning learning rates varying by method and model.Relearning used one epoch for GA, KL, and NPO, and two epochs for GD and SCRUB.Source: §C.1 · p. 19

C.2.1 RELEARN SET CONSTRUCTION; C.2.2 GPT-4 PROMPT FOR RELEARN SET GENERATION; C.2.3 GPT-4 PROMPT FOR RELEARN SET FILTERING

  • For WMDP relearn-set construction, GPT-4 generated general knowledge about query references, then an LLM filter removed text containing the true answer.The prompt requested at least three paragraphs and 800 words; the authors note GPT-4 may fail to obey the instruction not to include the answer.Source: §C.2.1 · p. 19, §C.2.2 · p. 20, §C.2.3 · p. 20

C.3 WMDP EVALUATION QUESTIONS

This section examines unlearning and relearning through WMDP evaluation-question topics, retain-set MT-Bench reporting, and examples of relearning after unlearning. The examples cover restored or expanded WMDP answers, a case where gibberish relearning did not jog unlearned information, and a separate Harry Potter example of restored verbatim text.

C.3 WMDP EVALUATION QUESTIONS

  • The WMDP evaluation questions span cybersecurity and biology, including network attacks, malware, pathogens, and viral biology.The supplied list also includes questions about cryptography and biological engineering.Source: §C.3 · pp. 20, 23

C.4 WMDP RETAIN PERFORMANCE

  • Figure 7 reports LLM-as-Judge scores for the retain set on MT-Bench for two models: Zephyr-7B-Beta (left) and Llama-3-8B (right).Source: Figure 7 · p. 23

C.5 WMDP ADDITIONAL EXAMPLES

  • Across additional WMDP examples, relearned answers can score similarly to the original model while the unlearned model gives uninformative answers.The appendix also notes that unsuccessful unlearning and unsuccessful relearning cases show no quality change after relearning.Source: §C.5 · p. 23

C.5.1 ADDITIONAL WMDP-CYBER EXAMPLE

  • The smart-home example scores 7 for the original model, 1 for the unlearned model, and 6 for the relearned model.The original and unlearned scores appear in the original and unlearned outputs; the relearned output gives a score of 6.Source: §C.5.1 · pp. 23–24

C.5.2 ADDITIONAL WMDP-BIO EXAMPLE

  • The HSV ribonucleotide-reductase example scores 7 for the original model, 1 for the unlearned model, and 9 for the relearned model.The relearned answer explains that viral RNR can ensure a supply of deoxyribonucleotides for DNA synthesis when host-cell RNR is limiting.Source: §C.5.2 · p. 24

C.5.3 UNSUCCESSFUL RELEARNING EXAMPLE

  • In an unsuccessful-relearning example, the relearned model scores 1 and gives irrelevant information, compared with 6 for the original model.The output contains more information than the unlearned model’s, but misunderstands the question.Source: §C.5.3 · pp. 24–25

C.5.4 UNSUCCESSFUL UNLEARNING EXAMPLE

  • The original and unlearned models both scored 8, but relearning produced a score of 9 and a more detailed scHPXV purification answer.The unlearned model described cell disruption and centrifugation; the relearned answer added extraction, concentration, and washing steps.Source: §C.5.4 · pp. 25–26

C.5.5 WMDP MCQ WITH DIFFERENT RELEARN TEXT

  • WMDP MCQ accuracy depended on relearn-text relevance, and gibberish text failed to jog unlearned information.Table 7 compares WMDP MCQ accuracy across relearn texts, with “None” denoting no relearning.Source: §C.5.5 · p. 26, Table 7 · p. 26

D.1 WHP MEMORIZATION HYPERPARAMETERS

  • For Harry Potter memorization, finetuning used lr = 2e−5 and batch size 4; GA, GD, and NPO unlearning used lr = 2e−6, 2e−5, and 6e−6, respectively, with batch size 8.All relearning used lr = 5e−6 and batch size 4; experiments used cosine scheduling, AdamW, and no weight decay.Source: §D.1 · p. 26

D.2 EXPERIMENT SETUP

  • Memorization finetuning used eight paragraphs from Harry Potter and the Order of the Phoenix, while relearning text comprised GPT-generated generic character information of at least 1,000 words.The relearning prompt asked for facts about Harry Potter, Ron Weasley, and Hermione Granger.Source: §D.2 · p. 26

D.3 RELEARNING EXAMPLE

  • After relearning, the model reproduced the memorized Harry Potter continuation verbatim, whereas the unlearned model emitted repetitive “Hedwig” text.The demonstrated query began “If he was lucky, there would also be owls carrying letters from his best friends Ron and”; the original continuation was restored exactly.Source: §D.3 · p. 28

D.4 UNSUCCESSFUL RELEARNING EXAMPLE

In this example, relearning does not restore the queried continuation: the output instead surfaces memorized text from elsewhere in the original. The unlearned model produces a repeated fragment, while the original and finetuned models provide the original passage.

D.4 UNSUCCESSFUL RELEARNING EXAMPLE

  • The relearned model surfaces a Hedwig passage instead of correctly completing the queried text.The example labels this as memorized text from another part of the original.Source: §D.4 · p. 28
  • The figure marks matching text in green and memorized text from elsewhere in the original in red.Source: §D.4 · p. 28
  • The original and finetuned models reproduce the queried original continuation, while the unlearned model repeats “Hed.”Source: §D.4 · p. 28

E RMU EXPERIMENTS

The RMU experiment shows that relearning can raise MCQ accuracy without improving answer completion, while zero-init+relearn changes that trade-off. A separate control finds relearn text alone does not improve evaluation performance, supporting residual knowledge as the source of attack effectiveness.

E RMU EXPERIMENTS

  • Relearning significantly raises RMU MCQ accuracy without improving completion scores, while zero-init+relearn improves completion by removing corrupted weights but degrades MCQ knowledge.Most ordinary relearned completion outputs repeat one token; the evaluation uses MCQ accuracy and an LLM-as-Judge completion score.Source: E RMU EXPERIMENTS · pp. 28–29, Table 8 · p. 28

F.1 GRADIENT ASCENT AND GRADIENT DIFFERENCE

  • Gradient ascent maximizes forget-set loss, while gradient difference adds a retain-set loss-minimization term.Gradient difference is named for combining these opposing objectives.Source: §F.1 · p. 29

F.2 KL MINIMIZATION

  • KL minimization combines gradient ascent on the unlearn set with minimization of KL divergence between current- and reference-model outputs on the retain set.The comparison is between output logits.Source: §F.2 · p. 29

F.3 NEGATIVE PREFERENCE OPTIMIZATION (NPO) / F.4 SCRUB

  • NPO penalizes forget-set prompt-response pairs while ignoring positive responses; SCRUB alternates minimization and maximization steps.SCRUB's minimization step combines its first and last terms.Source: §F.3 · p. 29, §F.4 · pp. 29–30

F.5 RMU

  • RMU corrupts selected-layer representations for unlearn data while aligning retain-data representations with the reference model.A subset of layers, usually one, is selected; the two datasets use noise-vector and reference-representation targets, respectively.Source: §F.5 · p. 30

F.6 WHP ALTERNATE LABELS

  • WHP alternate labels replace anchor terms with generic terms and reverse positive token-logit changes from a reinforced model relative to a baseline.The generic prediction selects the token with the maximum logit.Source: §F.6 · p. 30

G RELEARN TEXT ITSELF IS UNINFORMATIVE

  • Directly finetuning on relearn data yields no evaluation improvement across TOFU, WHP, and Synthetic, indicating the attack recovers knowledge unlearning failed to remove.This control tests whether the relearn set itself provides useful target knowledge.Source: G RELEARN TEXT ITSELF IS UNINFORMATIVE · p. 30, Table 9 · p. 31

I DETAILED EXPLANATION OF EVALUATION METRICS

  • The evaluation uses LLM-judge scores for harmful-answer quality, Rouge-L for copyrighted-text similarity, and keyword search for targeted tokens.The LLM judge assigns scores from 1 to 10; keyword search checks whether the target keyword appears in the output.Source: I DETAILED EXPLANATION OF EVALUATION… · p. 31
Loading 2406.13356v4…