An Adversarial Perspective on Machine Unlearning for AI Safety

1 Introduction

• LLMs can generate hazardous content because harmful information is difficult to exclude from their enormous Internet-derived pretraining corpora.The passage notes that growing model capabilities may make hazardous information more accessible to adversaries.Source: §1 · p. 1
• Unlearning aims to remove hazardous knowledge so it cannot be extracted even after jailbreaking, but recovery findings raise the question of whether it removes or merely obfuscates that knowledge.RMU can reduce hazardous-benchmark accuracy to random chance, yet prior work reports recovery after unlearning.Source: §1 · p. 1
• The paper measures hazardous knowledge in LLMs using WMDP accuracy, and deems unlearning unsuccessful if significant accuracy can be recovered without weight updates or through updates on data with little or no mutual information with target knowledge.Source: §1 · pp. 1–2
• The authors conduct a comprehensive white-box comparison of state-of-the-art unlearning methods with DPO safety training and adapt attacks to recover capabilities.Their findings question whether unlearning offers robust protection beyond standard safety training.Source: §1 · p. 2
• Adaptive attacks recover substantial hazardous knowledge from unlearned models, including original WMDP performance after activation-direction removal or finetuning on 10 unrelated examples.GCG, previously reported ineffective against unlearning, also recovers substantial accuracy after a small loss-function change.Source: §1 · p. 2

2 Related Work

• Prior LLM unlearning evaluations used varied topics, metrics, and criteria, and the field lacks standardized metrics and unified evaluation frameworks.The cited approaches include output perplexity, multiple-choice accuracy, and ROUGE after prefix injections.Source: §2 · pp. 2–3
• The gold-standard conception of unlearning is to make a model indistinguishable from one retrained without the target data, with the goal of making knowledge inaccessible short of full retraining.Related work also describes jailbreaks through prompting, optimized prompts, or activation ablation.Source: §2 · p. 2
• The paper focuses on safety unlearning, comparing RMU and adapted NPO methods intended to eliminate hazardous knowledge with DPO safety training.WMDP was introduced alongside RMU as a benchmark for hazardous capabilities.Source: §2 · p. 3
• Prior adversarial evaluations found that unlearned information could be extracted through internal-representation probing, and that finetuning can revert unlearning.RMU reported robustness to some white-box attacks, including GCG and probing, while also finding that finetuning could disable protections.Source: §2 · p. 3
• Concurrent studies reported recovery through finetuning or black-box prompting, while the authors report recovering most unlearned knowledge in a white-box setting.Other work also found capability-removal techniques susceptible to off-the-shelf attacks, motivating adaptive evaluation.Source: §2 · p. 3

3 Experimental Setup

• The evaluation targets whether safety unlearning removes hazardous knowledge or merely makes it more difficult to access, using WMDP biology and cybersecurity datasets.The setup describes forget sets as target-domain information and retain sets as information to preserve.Source: §3 · p. 3

3.1 Threat Model

• The threat model assumes white-box access to modify weights or intervene in activations, plus access to the original model or an equivalent model with protections removed.The authors note that this differs from RMU’s black-box threat model but offers insight into whether knowledge remains in weights.Source: §3.1 · pp. 3–4

3.2 Unlearning Methods and Safety Training Baseline

• The study evaluates RMU and NPO+RT as unlearning methods and uses DPO as a safety-training baseline.NPO’s hazardous-knowledge removal effectiveness was described as unexplored in the cited setup.Source: §3.2 · p. 4

3.3 Models and Datasets

• RMU uses a publicly available checkpoint, while NPO and DPO are finetuned by the authors on WMDP; preference data for the latter methods is constructed from WMDP documents.GPT-4 formulates questions; hazardous-topic questions pair refusal strings as desired outputs and correct options as rejected outputs.Source: §3.3 · p. 4

3.4 Unlearning Evaluation

• WMDP accuracy measures hazardous knowledge and MMLU accuracy measures general utility, with the highest-probability multiple-choice option selected even when a model refuses.WMDP includes 1,273 biology and 1,987 cybersecurity questions; MMLU covers 57 tasks.Source: §3.4 · p. 4

4 Our Methods To Recover Hazardous Capabilities

• Except for finetuning, the recovery methods do not modify model weights; finetuning uses small or unrelated datasets to test recovery without adding hazardous knowledge.The paper excludes ECO because its pre-LLM filter leaves original weights and potential hazardous knowledge unchanged.Source: §4 · pp. 4–5

4.1 Finetuning

• Finetuning uses LoRA on forget, retain, or WikiText datasets, varying sample sizes from 5 to 1,000 examples to assess recovery across different levels of mutual information with target knowledge.The retain and WikiText data have low mutual information with the unlearned knowledge; the forget dataset has high mutual information.Source: §4.1 · p. 5

4.2 Orthogonalization

• Orthogonalization removes a separately estimated refusal direction from each transformer block, using output differences between original and unlearned models on the forget preference dataset.The authors also develop a setup that does not require the original model before unlearning.Source: §4.2 · p. 5

4.3 Logit Lens

• Logit Lens projects residual-stream activations at every transformer layer onto the vocabulary and selects among WMDP’s A, B, C, and D answer tokens by probability.Other activation spaces are evaluated in an appendix.Source: §4.3 · p. 5

4.4 Enhanced GCG

• Enhanced GCG optimizes prefixes to avoid triggering RMU’s persistent residual-stream noise and weights early prompt tokens more heavily.The attack is optimized on six WMDP questions answered correctly by the original model and incorrectly by the unlearned model.Source: §4.4 · p. 5

4.5 Set difference pruning

• Set-difference pruning scores neurons for hazardous knowledge on the WMDP forget set and utility on MMLU, then prunes neurons associated only with hazardous knowledge.The method follows prior set-difference pruning work and uses SNIP importance scores.Source: §4.5 · p. 5

Evaluation scope

• The reported results emphasize WMDP-Bio because original and unlearned model scores differ substantially there, whereas the smaller WMDP-Cyber gap makes results more volatile.Source: §5 · p. 6

Residual-stream decoding

• LogitLens decoded correct answers from Zephyr-7B at layer 19 before unlearning, while RMU reduced accuracy close to random chance across the architecture; DPO remained at 56%.Source: §5 · p. 6

Specific directions mediate unlearning

• RMU reached 64.7% accuracy after activation-space orthogonalization, exceeding the 64.4% baseline; NPO and DPO reached 45% and 51%, respectively.The manipulation changed activations during the forward pass; ablating a single refusal direction across layers achieved 54.2%.Source: §5 · p. 6, Table 1 · p. 6

Finetuning on unrelated information reverts unlearning

• 52.7% (NPO), 57.0% (DPO), and 61.6% (RMU) WMDP-Bio accuracy followed finetuning on 10 unrelated retain-set samples.Finetuning on 1,000 retain-set samples fully recovered capabilities across methods; five forget-set samples also reversed unlearning, particularly for RMU, while MMLU degradation from the 10-sample finetuning was under 2 p.p.Source: §5 · p. 6, Figure 2 · p. 7

Critical neurons

• Pruning localized critical weight sets representing 2.0% of NPO, 0.9% of RMU, and 2.4% of DPO weights increased WMDP performance by at least 10 p.p. for every method.Source: §5 · p. 7

Universal adversarial prefixes

• RMU accuracy rose from 29.9% to 53.9%, NPO from 29.5% to 46.0%, and DPO from 27.9% to 49.0% with universal adversarial prefixes.The prefixes could recover WMDP-answering behavior without necessarily restoring conversational capability; responses were often incorrect and overconfident.Source: §5 · p. 7

Unlearning and safety training

• RMU and NPO primarily obscure rather than eliminate knowledge, sharing safety-training-like vulnerabilities to direction manipulation, unrelated-data finetuning, critical-neuron pruning, and universal adversarial strings.Source: §6 · pp. 7–8

Black-box evaluations

• Black-box tests can mistake inaccessible outputs for removed knowledge, while the paper's white-box approaches recovered the supposedly unlearned information.The authors argue evaluation should use adaptive methods to test protections.Source: §6 · p. 8

NPO's robustness and trade-off

• NPO showed better robustness than DPO or RMU, but its current implementation caused greater degradation on MMLU and general capabilities.Source: §6 · p. 8

Mitigations and lessons

• The authors caution that output-only finetuning may produce knowledge obfuscation, and describe RMU's noise injection as inadequate for open-source models.They suggest orthogonalizing harmful-knowledge directions or precisely modifying knowledge vectors as alternatives.Source: §6 · p. 8

7 Conclusion

• The paper reports a comprehensive white-box evaluation of state-of-the-art unlearning methods for AI safety.Source: §7 · p. 8
• Unrelated-data finetuning or activation-space direction removal often recovers supposedly unlearned capabilities, showing that methods cannot reliably remove knowledge from model weights.The findings challenge the belief that unlearning offers more robust protection than standard safety training.Source: §7 · p. 8
• The recovery findings challenge the belief that unlearning methods offer more robust protection than standard safety training.Source: §7 · p. 8
• Black-box evaluations are insufficient for assessing unlearning because they do not assess internal model changes.Source: §7 · p. 8

A.1 Why ECO is not unlearning under our definition

• Successful unlearning should account for whether target knowledge remains in model internals or can be retrieved adversarially, not only whether outputs resemble a model retrained without that data.The authors argue that an output-only evaluation can miss knowledge stored in weights; ECO’s detector and embedding-noise approach is characterized as obfuscating refusal rather than removing knowledge.Source: §A.1 · p. 16

A.2 Potential vulnerabilities4

• ECO’s detector-based design shifts red-teaming pressure to the detector, and the authors sketch tokenization and text-classification attacks as possible vulnerabilities.They describe character-by-character tokenization as a bypass for token-wise detectors, while noting prompt-wise detectors may be more challenging; this is preliminary analysis without experiments.Source: §A.2 · p. 16

B.1 Direct Preference Optimization (DPO)

• DPO trains toward chosen responses using preference triples containing an input, a chosen response, and a rejected response.Its objective includes a reference model and a parameter controlling deviation from that reference.Source: §B.1 · p. 17

B.2 Negative Preference Optimization (NPO)

• The authors use an NPO variant with a retain loss after straightforward NPO caused model divergence and catastrophic forgetting in pilot experiments.The retain-loss weight is α, and the loss uses forget-set and retain-set input-output pairs.Source: §B.2 · p. 17

B.3 Representation Misdirection for Unlearning (RMU)

• RMU finetunes selected lower layers to output a fixed noise vector for forget-set concepts while preserving representations for retain-set concepts.The noise magnitude is controlled by c.Source: §B.3 · p. 17

C Preference dataset construction

• DPO and NPO preference datasets are built from WMDP forget and retain documents, formatted as multiple-choice questions; biology uses filtered articles and cybersecurity uses scraped GitHub pages.Articles shorter than 1,000 characters are excluded and are truncated at 15,000 characters.Source: C Preference dataset construction · p. 18

C.1 System prompt

• The prompt asks for 10 multiple-choice questions per article, each with four options, while a chosen response is randomly selected from available refusal strings; GPT-4 generated 80 strings, manually checked for variety.Source: §C.1 · p. 18, §C.2 · p. 19, §C.3 · p. 20

D.1 Hyperparameters

• Training details include a limited hyperparameter search, with Table 2 listing best DPO and NPO settings by knowledge domain and Table 3 presenting full benchmark results.The reported benchmarks for the trained models are WMDP and MMLU.Source: §D.1 · p. 22, Table 2 · p. 22, §D.2 · p. 22

E.1 Methods overview

• Table 4 compares knowledge-extraction methods by their resource requirements.Source: Table 4 · p. 23, §E.1 · p. 23

E.2 Finetuning

• RMU uses WMDP forget and retain corpora, while DPO and NPO use multiple-choice datasets and specified prompt templates.The described RMU setup also uses Wikitext as a retain set; DPO and NPO mix forget and retain datasets.Source: §E.2 · p. 24

E.3 Orthogonalization

• Orthogonalization directions can be estimated from hazardous-versus-benign representation means, including without access to the original model.The appendix also tests a PCA-derived direction and layer-subset ablations; outlier tokens are filtered when computing mean differences.Source: §E.3 · pp. 24–25

E.4 Logit lens

• Logit Lens projects transformer activations into vocabulary space to select answer tokens, with projections also tested at other architectural stages.The WMDP prompts are prefixed to emphasize the A, B, C, and D answer tokens.Source: §E.4 · p. 25

E.5 Enhanced GCG

• Enhanced GCG adds a representation-distance penalty against a maliciously finetuned model, alongside changes to the target and optimization procedure.For RMU, the loss emphasizes earlier tokens and is computed over the prompt as well as target and match tokens.Source: §E.5 · pp. 25–26

F Complete results

• 35.0 WMDP-Cyber accuracy and 55.4 MMLU accuracy were obtained by orthogonalizing only the layer-15 PCA direction; all-layer directional ablation caused catastrophic forgetting.Source: F Complete results · p. 27

F.1 Finetuning

• Figure 4 reports WMDP and MMLU performance after finetuning with 5, 10, 50, 100, 500, or 1000 samples.Source: Figure 4 · p. 28

F.2.1 Complementary results for WMDP-Bio

• Figure 5 presents WMDP-Bio performance for residual-stream projections at different stages.Source: Figure 5 · p. 29

F.2.2 Full results for WMDP-Cyber

• Figure 6 presents WMDP-Cyber performance for residual-stream projections at different stages.Source: Figure 6 · p. 30

F.3 Orthogonalization

• Unlearned-model representations with benign datasets can yield significant knowledge recovery, though RMU recovery was below that obtained using original-model representations.For RMU, ablation at layers 5, 6, and 7 was sufficient for recovery using original-model representations; layer 7 alone reached 56.7%.Source: §F.3 · p. 31

F.3.1 Results for WMDP-Cyber

• 35.0 WMDP-Cyber accuracy and 55.4 MMLU accuracy were reached by ablating a layer-15 PCA direction; all-layer ablation substantially damaged the residual stream for NPO.Source: §F.3.1 · p. 32

G Complete results using chat template

• Adversarial suffixes could work well with or without a chat template but not both, with performance gaps reaching 20 p.p.The reported comparison was 31.2% without the template and 51.4% with it.Source: G Complete results using chat template · p. 33
• 53.7% RMU accuracy with a chat template contrasted with 34.1% without it when using the first principal component as the ablation direction.Source: G Complete results using chat template · p. 33
• The authors suggest that setting-dependent behavior may reflect behavioral compartmentalization, which might explain the relative ease of inserting trojans.Source: G Complete results using chat template · p. 33

G.2 Finetuning

• The appendix tables report WMDP-Cyber, MMLU, and WMDP-Bio accuracy using chat templates; Figure 7 covers WMDP and MMLU after finetuning across sample counts.Source: Table 10 · p. 34, Table 11 · p. 34, Figure 7 · p. 35

G.3 Results

• Figures 8 and 9 show benchmark performance from residual-stream projections at different stages for WMDP-Bio and WMDP-Cyber, respectively.Source: Figure 8 · p. 36, Figure 9 · p. 37

H.1 Naive perturbations

• Naive character perturbations made Zephyr-RMU more open to hazardous discussion in qualitative evaluation but did not significantly change its WMDP performance.The experiments inserted random characters at different positions and also tested character-level tokenization.Source: §H.1 · p. 38, Figure 10 · p. 38

H.2 Informed perturbations

• Informed perturbations greedily alter prompt tokens whose layer-7 activations remain too similar to the refusal direction, improving WMDP-Bio performance.Source: §H.2 · p. 38
• An example shows RMU answering a perturbed biology question correctly and then generating a coherent continuation.Source: §H.2 · p. 39

• At cosine-similarity threshold 0.5, Zephyr-RMU's performance differed from the baseline by only 2.2 p.p., and DPO and NPO also revealed more knowledge on perturbed prompts.Source: §H.3 · p. 40, Figure 11 · p. 41
• 29% was the maximum accuracy improvement reported for Mixtral-8x7B-RMU on perturbed WMDP-Bio.Source: §H.3 · p. 41, Figure 12 · p. 41
• Perturbed prompts reduced the RMU-versus-original-model perplexity ratio for original-model generations from about 1600 times to about 16 times.The generations were conditioned on unperturbed versus perturbed WMDP-Bio prompts, respectively.Source: §H.3 · pp. 41–42, Table 12 · p. 42

I Perplexity Analysis of Adversarial Prefixes on RMU

• In chat, lower perplexity correlated with higher WMDP accuracy below 0.4, but the highest-accuracy prefixes were not necessarily lowest in perplexity, so coherence did not reliably indicate correctness.The lowest-perplexity prefixes spanned a broad accuracy range, and the reverse-model perplexity experiment showed similar trends.Source: I Perplexity Analysis of Adversarial… · p. 43, Figure 13 · p. 43

I.1 Adversarial prefixes without chat template

• Without a chat template, perplexity trends decayed exponentially and became nearly constant beyond a threshold, making coherence uninformative about prefix quality.The authors hypothesize that next-token prediction permits low-perplexity output without the helpfulness or responsiveness expected in chat.Source: §I.1 · p. 44, Figure 14 · p. 44

J.1 Behaviour during innocuous conversations

• RMU handled harmless-domain prompts but failed to distinguish benevolent from malicious intent; keyword-triggered noise could snowball from spelling errors into degenerated output.The authors hypothesize that a positive feedback loop during generation carries the noise forward.Source: §J.1 · p. 45, §2 · p. 45

J.2 Simple prompt-based jailbreaks

• Paraphrasing, translation, added malicious examples, and chat-template changes did not substantially improve responsiveness, while keyword perturbations that altered tokenization helped somewhat.This observation motivated an adaptive approach discussed in Appendix H.2.Source: §J.2 · pp. 45–46

J.3 Prefilling attack

• Prefilling RMU with the prompt and portions of correct answers did not restore its generations, leading the authors to conclude that straightforward target-string GCG optimization would fail.Source: §J.3 · p. 46

J.4 Effect of noise on token representations

• At RMU’s last unlearned layer, tokens from “vir” onward exceeded 0.8 cosine similarity, consistent with keyword-triggered noise; later layers converged to about 0.3 similarity.Special chat tokens were less similar to the noisy tokens, which the authors attribute to their syntactic rather than semantic role.Source: §J.4 · pp. 46, 48, Figure 15 · p. 47

J.4.1 PCA analysis

• PCA showed that benign Wikitext and hazardous WMDP representations were almost linearly separable after the first 40 question tokens were discarded.Source: §J.4.1 · p. 48, Figure 16 · p. 48