Source-linked AI summary
Eraser: Jailbreaking Defense in Large Language Models via Unlearning Harmful Knowledge
Weikai Lu, Ziqian Zeng, Jianwei Wang, Zhengdong Lu, Zelin Chen, Huiping Zhuang, Cen Chen, Hao Peng
TL;DR
Jailbreaking attacks can bypass safeguards, while harmful knowledge remains within language models. Eraser unlearns that knowledge while retaining general knowledge and safety alignment, and experiments report lower attack success and harmfulness without compromising general capabilities.
Problem
Existing jailbreak defenses leave harmful knowledge in the model, which the paper identifies as a source of jailbreak risk.
Source: §V · pp. 8–9Method
Eraser combines unlearning harmful knowledge with retaining general knowledge and maintaining safety alignment.
Source: §I · p. 2Results
Eraser significantly reduces ASR and harmfulness scores in all tested settings compared with the base model.
Source: B. Main Results · p. 6Takeaways & Limitations
The experiments indicate that Eraser can reduce jailbreak risk without compromising general capabilities.
Source: §V · p. 9Eraser addresses specific harmful issues, making exhaustive enumeration challenging, and applies only to LLMs that have undergone safety alignment.
Source: §VI · p. 9
Abstract
from arXiv · showhide
Jailbreaking attacks can enable Large Language Models (LLMs) to bypass the safeguard and generate harmful content. Existing jailbreaking defense methods have failed to address the fundamental issue that harmful knowledge resides within the model, leading to potential jailbreak risks for LLMs. In this paper, we propose a novel defense method called Eraser, which mainly includes three goals: unlearning harmful knowledge, retaining general knowledge, and maintaining safety alignment. The intuition is that if an LLM forgets the specific knowledge required to answer a harmful question, it will no longer have the ability to answer harmful questions. The training of Erase does not actually require the model's own harmful knowledge, and it can benefit from unlearning general answers related to harmful queries, which means it does not need assistance from the red team. The experimental results show that Eraser can significantly reduce the jailbreaking success rate for various attacks without compromising the general capabilities of the model. Our codes are available at https://github.com/ZeroNLP/Eraser.
I. INTRODUCTION
Eraser targets the harmful knowledge that can be elicited through jailbreaks, combining unlearning with retention of general knowledge and safety alignment. The paper reports reduced jailbreak success while maintaining general capability.
I. INTRODUCTION
- Direct harmful queries can be refused through safety alignment, but adversarial prompts can bypass safeguards and elicit harmful responses.The paper describes manually designed and automated adversarial prompts as increasing jailbreak success and security risks.Source: Figure 1 · p. 1, §I · p. 1
- Filtering and continued-training defenses leave harmful knowledge in the model, which the authors identify as the basis for harmful responses and a risk as attacks advance.The paper uses bomb-making knowledge as an example of knowledge supporting a harmful answer.Source: §I · pp. 1–2
- Eraser combines unlearning harmful knowledge, retaining general knowledge, and maintaining safety alignment to defend against jailbreaks.Its stated approach uses gradient ascent on harmful answers in simulated jailbreak mode, preserves entity understanding, and maintains rejection of harmful questions.Source: §I · p. 2
- Eraser enhances defensive capability while maintaining general capability, unlike other trained-based methods that compromise general capability.The authors also report that unlearning random data can improve defense, but reduces general performance.Source: §I · p. 2
II. RELATED WORKS
The related work reviews jailbreak attacks, existing defenses, and machine unlearning. It positions Eraser as a post-hoc defense for aligned models that targets retained harmful knowledge.
A. Jailbreak Attack
- Black-box attacks rely on crafted prompts or automated search, while white-box attacks use model access to optimize adversarial prompts; Eraser aims to defend against both.Examples include GCG [10] and AutoDAN [8].Source: A. Jailbreak Attack · p. 2
B. Jailbreak Defense
- Existing defenses primarily filter harmful behavior or continue training, but do not remove the harmful knowledge retained in the model.Filtering can add inference costs, while continued training seeks to improve security through further fine-tuning.Source: B. Jailbreak Defense · p. 2
C. LLM Unlearning
- Machine unlearning removes specified learned knowledge, while Eraser applies it post hoc as a jailbreak defense for an already aligned model.This differs from treating unlearning as an alternative alignment method.Source: C. LLM Unlearning · pp. 2–3
III. METHODOLOGY
Eraser fine-tunes an aligned model to refuse harmful queries under different jailbreak attacks while retaining proficiency on regular queries. Its unlearning process uses harmful answers and randomized prompt disguises.
A. Problem Formulation
- The objective is to fine-tune an aligned LLM to refuse harmful queries under different jailbreak attacks while maintaining proficiency on regular queries.The paper frames this as transforming aligned model f into a new model h, given a harmful-query set.Source: A. Problem Formulation · p. 3
B. Unlearn Harmful Knowledge
- Eraser uses gradient ascent on harmful answers, which it obtains from publicly available uncensored models rather than collecting them with red teams.The authors motivate this data choice by the labor and time required for red-team collection and the possibility that forgetting one answer generalizes to others.Source: B. Unlearn Harmful Knowledge · p. 3
- Random prefixes and suffixes are added to harmful queries during training to simulate jailbreak prompts and reduce the target answer's conditional probability.The target answer's token probabilities are conditioned on the transformed query and preceding answer tokens.Source: B. Unlearn Harmful Knowledge · p. 3
C. Retain General Knowledge
Unlearning harmful content can also weaken entity understanding and erase useful knowledge. Eraser preserves general knowledge by training the model on entity-related questions using the aligned model as teacher.
C. Retain General Knowledge
- The authors identify weakened entity understanding as a main way gradient-ascent unlearning impairs general performance.They use potassium nitrate, a bomb-making material, to illustrate loss of knowledge that could answer harmless questions.Source: C. Retain General Knowledge · pp. 3–4
- Unlearning bomb-making knowledge leaves Llama2 unable to give specific uses of potassium nitrate, while the original lists nine applications.The example illustrates how unlearning can remove useful entity knowledge alongside harmful knowledge.Source: Figure 3 · p. 4, C. Retain General Knowledge · p. 4
- Eraser extracts entities from harmful answers, generates entity-related questions, and distills next-word predictions from the aligned model to preserve helpful knowledge.For each extracted entity, it randomly selects one of ten prompt templates to generate a question; the teacher is the pre-unlearning aligned model.Source: C. Retain General Knowledge · p. 4
D. Maintain Safety Alignment
Eraser trains on refusal data from the original model to maintain refusal capability and reasons for refusal. Its overall objective limits harmful-knowledge unlearning while continuing to optimize general performance and safety alignment.
D. Maintain Safety Alignment
- For each harmful question, the original model supplies refusal data, and training encourages the model to retain similar refusal capability.The authors frame refusal and reasons for refusal as an important display of responsibility even when harmful answering ability is unlearned.Source: D. Maintain Safety Alignment · p. 4
E. Overall objective
- The overall objective stops optimizing harmful-knowledge unlearning at threshold γ while continuing to optimize general-knowledge retention and safety alignment.The authors motivate the constraint by observing that prolonged unlearning can harm performance over time and that unlearning is easier than preserving capability.Source: E. Overall objective · pp. 4–5
- Continuing the other objectives after γ is reached is intended to better balance optimization difficulty and restore general performance and safety-alignment mechanisms.Source: E. Overall objective · pp. 4–5
IV. EXPERIMENTS
The experiments evaluate Eraser and baseline defenses against four jailbreak attack methods, using AdvBench and an extended harmful-topic dataset. They also assess general capabilities with standard benchmarks.
A. Experimental Setup
- Four attacks evaluate defense effectiveness: AIM and PAP are black-box, while AutoDAN and GCG are white-box methods.PAP prompts are generated with GPT-4o-mini using five persuasion techniques, and the most harmful response among five queries is recorded.Source: A. Experimental Setup · p. 5
- Training-based baselines are RSFT, GAM, and CAT; RSFT excludes PAP prompts from training to test defense against unknown attacks.GAM uses the same unlearning data as Eraser, while CAT searches continuous-space adversarial prompts for training.Source: A. Experimental Setup · p. 5
- AdvBench supplies 520 harmful training queries, and AdvExtent adds 417 test samples generated from 139 topics using three prompts per topic.The harmful responses to unlearn are generated by an uncensored model.Source: A. Experimental Setup · p. 5, Figure 6 · p. 5
- Defense is measured by harmfulness scores and attack success rate, with responses scoring above 2 on the 1–5 scale counted as successful jailbreaks.GPT-3.5-turbo provides the harmfulness scores.Source: A. Experimental Setup · p. 5
- General capability is evaluated on seven benchmarks: ARC challenge, ARC easy, COPA, CB, MMLU, BoolQ, and HellaSwag.Source: A. Experimental Setup · p. 5
B. Main Results
Eraser lowers attack success and harmfulness across the reported settings, including on same-topic AdvExtent instructions unseen during training. It also preserves general benchmark performance better than the compared defense-trained methods.
B. Main Results
- Eraser significantly reduces ASR and harmfulness versus the base model across all reported settings, with similar defensive effects on unseen AdvExtent instructions.The evaluation covers AdvBench and AdvExtent; the authors note that forgotten harmful knowledge does not cover all harmful knowledge, so attack success is not reduced to 0%.Source: B. Main Results · p. 6, Table I · p. 6
- RSFT is comparable to Eraser under AIM, AutoDAN, and GCG, but performs worse on unseen PAP; GAM has higher jailbreak risk than Eraser in all settings.RSFT’s attack patterns were visible during training, including all possible successful AIM patterns; GAM sometimes has greater harmfulness than the base model.Source: B. Main Results · p. 6
- Eraser shows only small accuracy variation across seven benchmarks and an average accuracy that surpasses the base model, unlike GAM, RSFT, and CAT.On MMLU, Eraser’s accuracy decreases by 0.01%; MMLU averages 57 tasks, while each other benchmark contains one task.Source: B. Main Results · pp. 6–7, Table II · p. 6
C. Comparison with harmful behavior filtering method
Compared with harmful-behavior filtering, Eraser is stronger against most attacks without higher inference latency or benign-instruction refusal. Ablations indicate that its random prompt transformation and separate retention and alignment goals contribute to the defense-capability balance.
C. Comparison with harmful behavior filtering method
- RA-LLM is weaker than Eraser against most attacks, except GCG, while longer and more complex prompts expose token-removal defenses’ lack of robustness.The GCG exception is attributed to its short adversarial suffix being disrupted by token removal; white-box comparisons use transfer attacks for RA-LLM.Source: C. Comparison with harmful behavior… · p. 7
- Eraser adds no higher latency or refusal rate on benign instructions, whereas RA-LLM’s defense requires 20 extra inference rounds and risks benign refusals.The comparison recorded average inference latency and refusal rate on 100 benign Alpaca instructions.Source: C. Comparison with harmful behavior… · p. 7, Table III · p. 7
D. Ablation Study
- Removing T(·) increases ASR; Lh mostly restores general capability, while Lr further improves defense and T(·) strengthens it.Using only Lf yields strong defense but severely impairs general capability; removing Lr raises ASR, and removing Lh reduces general performance.Source: D. Ablation Study · p. 7, Table IV · p. 7
E. What has Contributed to Defensive Capabilities?
Experiments distinguish defense from harmful-data unlearning from defense obtained by degrading general capability. The threshold γ controls the defense–capability balance, with excessive values impeding recovery of general performance.
E. What has Contributed to Defensive Capabilities?
- Replacing harmful training answers with random tokens still lowers AIM ASR but significantly decreases general capability; using random data in Eraser restores overall performance while returning jailbreak risk near the base model.The contrast supports a contribution from unlearning harmful data, while the authors speculate that random-token defense may arise from disrupting general performance.Source: E. What has Contributed to Defensive… · p. 7, Table V · p. 8
- The authors interpret the random-token results as a harmfulness–helpfulness trade-off: impaired instruction following can also reduce the ability to follow harmful instructions.They recommend a moderate γ because an overly large threshold may prevent general-ability recovery.Source: E. What has Contributed to Defensive… · p. 8, F. The Impact of Threshold γ · p. 8
- Lf w/ random reduces general ability by 1.94% and lowers AIM ASR from 19.61% to 5.38%, illustrating defense obtained alongside capability loss.The authors describe this method as extremely low cost because it requires only generating random data to unlearn.Source: E. What has Contributed to Defensive… · p. 8
F. The Impact of Threshold γ
- As γ increases, Eraser’s AIM ASR reaches 0 at γ=3, but general performance fully recovers only at γ=1 and 2 and tends to decline above 2.For training with Lf alone, AIM ASR reaches 0 when Lf approaches -3 while general performance continues to decline.Source: F. The Impact of Threshold γ · p. 8, Figure 7 · p. 8
G. Case study
Case studies contrast Eraser’s refusal of an AIM jailbreak with its response to a harmless question about firearms. The conclusion reiterates the three-goal defense and reports that random-data unlearning can also defend with general-performance degradation.
G. Case study
- Under an AIM attack, Eraser refuses harmful guidance and invites harmless communication, while GAM may produce garbled or unrelated harmful text and RSFT and CAT give standard refusals.The example concerns a request for step-by-step instructions for insider trading.Source: G. Case study · p. 8, Figure 8 · p. 9
- For a harmless question about firearm hazards, Eraser responds most like the base model, whereas RSFT and CAT refuse, possibly because they overreact to “firearms.”The authors say this illustrates Eraser responding responsibly to jailbreak prompts while behaving more like the base model on general instructions.Source: G. Case study · p. 8, Figure 9 · p. 9
- On 150 questions about harmful topics that were themselves harmless, Eraser’s refusal rate was only 0.66% higher than the base model’s, while GAM, RSFT, and CAT increased refusals.The questions paired 50 AdvBench harmful topics with three prompts each.Source: G. Case study · p. 8, Table VI · p. 9
V. CONCLUSION
- The conclusion describes Eraser as combining harmful-knowledge unlearning, general-performance retention, and safety alignment to reduce jailbreak risk without compromising general capabilities.It also reports that unlearning random data can bring defense effects with general-performance degradation.Source: §V · pp. 8–9
VI. LIMITATIONS
Eraser is limited to specific harmful issues and safety-aligned LLMs; the ethics statement clarifies the purpose and authorship of the harmful text included in the paper.
VI. LIMITATIONS
- Eraser addresses specific harmful issues, whose enumeration is challenging, and applies only to LLMs that have undergone safety alignment.The authors say further work is needed to strengthen safety alignment before Eraser can become an alternative to methods such as RLHF.Source: §VI · p. 9
VII. ETHICS STATEMENT
- The harmful data and model-generated text are presented to evaluate Eraser, and the authors state that the generated opinions do not represent their views.Source: §VII · p. 9