Source-linked AI summary
Detoxifying Large Language Models via Knowledge Editing
Mengru Wang, Ningyu Zhang, Ziwen Xu, Zekun Xi, Shumin Deng, Yunzhi Yao, Qishen Zhang, Linyi Yang, Jindong Wang, Huajun Chen
TL;DR
Safety-aligned LLMs can remain vulnerable to carefully crafted attack prompts. This paper evaluates knowledge editing with SafeEdit and proposes DINM, which edits toxic regions. DINM improves generalized detoxification success rates on both evaluated models.
Problem
Carefully crafted attack prompts can bypass safety-aligned LLMs, and existing detoxification datasets and metrics underrepresent attack prompts and out-of-domain generalization.
Source: §1 · pp. 1–2Method
The paper constructs SafeEdit and proposes DINM, which locates toxic regions through contextual semantics and edits their parameters using one data instance.
Source: §1 · p. 2, §2.1 · p. 2Results
DINM reports generalized detoxification success rates ranging from 43.51% to 86.74% on LLaMA2-7B-Chat and from 47.30% to 96.84% on Mistral-7B-v0.1.
Source: §1 · p. 2Takeaways & Limitations
Knowledge editing demonstrates potential to erase toxic regions for permanent detoxification while limiting impact on general performance.
Source: §1 · p. 2, §6 · p. 10DINM's toxicity localization is relatively simple, so more robust localization methods are necessary.
Source: Limitations · p. 10
Abstract
from arXiv · showhide
This paper investigates using knowledge editing techniques to detoxify Large Language Models (LLMs). We construct a benchmark, SafeEdit, which covers nine unsafe categories with various powerful attack prompts and equips comprehensive metrics for systematic evaluation. We conduct experiments with several knowledge editing approaches, indicating that knowledge editing has the potential to detoxify LLMs with a limited impact on general performance efficiently. Then, we propose a simple yet effective baseline, dubbed Detoxifying with Intraoperative Neural Monitoring (DINM), to diminish the toxicity of LLMs within a few tuning steps via only one instance. We further provide an in-depth analysis of the internal mechanism for various detoxifying approaches, demonstrating that previous methods like SFT and DPO may merely suppress the activations of toxic parameters, while DINM mitigates the toxicity of the toxic parameters to a certain extent, making permanent adjustments. We hope that these insights could shed light on future work of developing detoxifying approaches and the underlying knowledge mechanisms of LLMs. Code and benchmark are available at https://github.com/zjunlp/EasyEdit.
1 Introduction
Safety-aligned LLMs can remain vulnerable to crafted attack prompts, motivating precise modification of toxic regions. The paper introduces SafeEdit and DINM, and reports stronger generalized detoxification performance from DINM.
1 Introduction
- Crafted attack prompts can bypass aligned models, raising whether toxic regions can be precisely modified for detoxification.The introduction notes that SFT, RLHF, and DPO improve safety, yet carefully crafted prompts may still elicit harmful outputs.Source: §1 · p. 1
- SafeEdit covers nine unsafe categories with attack templates and evaluates defense success, defense generalization, and general performance.The paper presents it as a benchmark for detoxification via knowledge editing.Source: §1 · p. 2
- DINM locates toxic regions through contextual semantics and edits parameters within those regions to minimize side effects.The introduction describes DINM as a simple knowledge-editing baseline for complex adversarial inputs.Source: §1 · p. 2
- 43.51% to 86.74% on LLaMA2-7B-Chat and 47.30% to 96.84% on Mistral-7B-v0.1 are the reported generalized detoxification success-rate ranges for DINM.The introduction reports stronger detoxifying performance with better generalization.Source: §1 · p. 2
- DINM uses one data instance without extra training, while SFT and DPO are described as bypassing toxic regions rather than erasing them.The paper frames toxic-region location as significant to detoxification.Source: §1 · p. 2
2 Benchmark Construction
SafeEdit pairs adversarial detoxification examples with general-knowledge controls and evaluates both defense and possible side effects. Its construction spans nine unsafe categories, varied attack prompts, and multiple measures of generalization and model performance.
2.2.3 Response Generation
- SafeEdit pairs each adversarial query with a safe response and an unsafe response to support editing the model from the latter behavior to the former.The paper uses GPT-4 to generate safe responses and text-davinci-003 to produce predominantly unsafe responses.Source: §2.1 · p. 2, §2.2.3 · p. 3
2.1 Task Definition
- A single instance containing an adversarial query and its safe and unsafe responses is used to edit the model, which should then respond securely to other malicious inputs.The edited model is denoted fW′.Source: §2.1 · p. 2
2.2.2 Attack Prompt
- The dataset contains 540 harmful questions across nine unsafe categories and 48 attack prompts, which are concatenated to form adversarial queries.Questions are generated as 60 per category; attack prompts come from websites, recent papers, and handwritten sources.Source: §2.2.1 · pp. 2–3, §2.2.2 · p. 3
2.2.5 Quality Control
- SafeEdit quality control combines a manually trained safety classifier with manual verification and refinement of responses and attack prompts.The classifier achieves about 97% accuracy; unsafe GPT-4 responses are manually modified, and prompts are refined across all nine categories.Source: §2.2.5 · pp. 3–4
2.2.4 General Knowledge
- SafeEdit contains 4,050 training, 2,700 validation, and 1,350 test instances, combining Dedit adversarial examples with Dcons instruction-following controls.The benchmark is intended for methods ranging from SFT and reinforcement learning to knowledge editing.Source: §2.2.4 · p. 3, §2.2.5 · p. 4
2.3.1 Defense Success
- Defense Success measures the average rate at which the edited model’s response to the current adversarial input receives the classifier’s safe label.The metric uses an indicator for safe versus unsafe responses and averages across test questions and attack prompts.Source: §2.3.1 · p. 4
2.3.2 Defense Generalization
- Defense Generalization evaluates defense on out-of-distribution inputs by varying the harmful question, attack prompt, or both.The four variants cover only harmful-question changes, attack-prompt changes, and changes to both together.Source: §2.3.2 · p. 4
2.3.3 General Performance
- General-performance measures include response fluency, TriviaQA knowledge-question answering, and Xsum summarization measured with ROUGE-1.The metrics are included to assess potential side effects on unrelated capabilities.Source: §2.3.3 · p. 4
3 The Proposed Baseline: DINM
DINM addresses the difficulty of locating harmful content expressed through context rather than specific tokens. It locates toxic regions from safe–unsafe response differences, then edits those regions while constraining changes to preserve unrelated capabilities.
3 The Proposed Baseline: DINM
- DINM locates toxic regions through contextual semantics, using one instance without extra training to identify and edit Mistral-7B-v0.1.The method first identifies a toxic layer from safe–unsafe hidden-state differences, then modifies toxic parameters using the adversarial input and safe response.Source: §3 · p. 5
3.1 Toxic Regions Location
- The toxic layer is the layer that most effectively separates the hidden-state distributions of safe and unsafe responses.This targets contextual toxic content, which can be difficult to locate with subject-based methods when adversarial inputs are complex or harmful meaning is conveyed by context.Source: §3 · pp. 4–5, §3.1 · p. 5
3.2 Detoxifying Editor
- The editor tunes toxic-region parameters while freezing the rest of the model, increasing safe-response probability and constraining changes on unrelated requests.Its loss balances adversarial-input defense with a KL constraint on a general-knowledge QA request; optimization updates the toxic region through backpropagation.Source: §3.2 · pp. 5–6
4 Experiment
DINM shows strong detoxification generalization, while ablations and mechanism analyses highlight the importance of locating toxic regions and suggest a different mechanism from SFT and DPO. Experiments also report relatively minor general-performance effects overall, with a notable Mistral knowledge-question-answering decline for some methods.
4.1 Settings
- FT-L, MEND, and Ext-Sub are included as knowledge-editing baselines in the experiment settings.The reported mechanism analysis uses a toxic probe to quantify parameter toxicity and activation flow into toxic regions.Source: §4.1 · p. 6, §4.3 · p. 8
4.2 Results
- 43.51% to 86.74% on LLaMA2-7B-Chat and 47.30% to 96.84% on Mistral-7B-v0.1: DINM improves average generalized detoxification performance.The reported generalization includes harmful questions, out-of-distribution attack prompts, and combinations of out-of-distribution inputs.Source: §4.2 · p. 6
- DINM has relatively minor general-performance side effects overall, but Ext-Sub and DINM show a significant KQA decline on Mistral-7B-v0.1.The authors attribute the decline to responses resembling modified examples and rejecting reasonable queries; they also report repetitive text from DINM.Source: §4.2 · pp. 6–7
- Removing toxic-region location drops average detoxification performance from 96.55% to 67.88% for Mistral-7B-v0.1 and from 88.59% to 80.26% for LLaMA2-7B-Chat.The authors conclude that locating and precisely eradicating toxic regions is more effective than indiscriminate fine-tuning.Source: Table 2 · p. 7, §4.2 · p. 8
4.3 Analysis
- On SafeEdit_test_ALL, DINM optimized with one instance can rival or outperform DPO, which requires extensive data and computational resources.The comparison uses two representative generalization metrics; DINM results are averaged over three experiments.Source: Table 3 · p. 7, §4.3 · p. 8
- DINM shows zero activation shift and 2.72% toxicity reduction in toxic regions, whereas SFT and DPO shift activations while toxic-region toxicity remains almost unchanged.The authors present this as a hypothetical mechanism; they suggest that persistent toxic regions after SFT and DPO may be easily activated by other malicious inputs.Source: Figure 4 · p. 8, Figure 5 · p. 8, §4.3 · p. 9
5 Related Work
Traditional detoxification includes parameter tuning, detection, and prompting, while knowledge editing offers a way to adjust specific model behaviors. DINM applies this editing paradigm to contextual toxic regions rather than specific tokens or phrases.
5.1 Traditional Detoxifying Method
- Traditional detoxification includes parameter modification through SFT and DPO, input-output detection, and prompting to improve safe responses.DPO is described as bypassing rather than removing toxic regions, leaving aligned models vulnerable to novel malicious inputs.Source: §5.1 · p. 9
5.2 Knowledge Editing
- Prior knowledge-editing work addresses detrimental words, privacy-sensitive neurons, or specific tokens and phrases, whereas DINM targets toxic regions determined by context semantics.The cited work includes Hu et al. (2023), Geva et al. (2022), and DEPN (Wu et al., 2023b).Source: §5.2 · pp. 9–10
- DINM locates toxic regions through contextual semantics, rather than specific tokens or phrases, and strives to erase them.It introduces detoxification through a single input-output pair with few tuning steps.Source: §5.1 · p. 9, §5.2 · p. 10
6 Conclusion
The conclusion presents SafeEdit and DINM as contributions to detoxification through knowledge editing, and reports an analysis of detoxification mechanisms.
6 Conclusion
- The paper presents SafeEdit, DINM, and an analysis suggesting knowledge editing may erase toxic regions for permanent detoxification.It frames these findings as potential contributions to future detoxification work.Source: §6 · p. 10
Limitations
The paper’s limitations include restricted model and baseline coverage, a simple DINM localization approach, and constraints on its mechanistic analysis.
Vanilla LLMs
- Experiments cover only LLaMA2-7B-Chat and Mistral-7B-v0.1 because of limited computational resources.The authors identify more vanilla models, multimodal security, and multilingual scenarios as future directions.Source: Limitations · p. 10
Baseline Methods
- The baseline comparison includes only Ext-Sub and MEND; some alternatives require explicit entities or a smaller same-family model unavailable for Mistral.The paper also frames its focus as establishing a benchmark for exploring additional editing methods.Source: Limitations · p. 10
Our DINM
- DINM’s toxicity localization is relatively simple, and continued attacks motivate consideration of batch and sequential editing.The authors also report that DINM can struggle to produce fluent responses, sometimes repeating sentences.Source: Limitations · p. 10
Mechanism Analysis
- Mechanistic analysis follows Lee et al. (2024), may miss scenarios because of its data and analytic approach, and studies toxic regions at the layer level.The authors say DINM reduces toxicity in those regions only to a certain extent and identify neuron-level localization as future work.Source: Limitations · p. 10
Ethics Statement
The ethics statement notes that toxic dataset context could be adapted for malicious purposes and describes safeguards. The appendix passages characterize harmful questions and attack prompts used across unsafe scenarios.
Ethics Statement
- Toxic context in the dataset could be repurposed maliciously, so attack prompts are drawn almost exclusively from public sources and the dataset is manually scrutinized.The authors describe the work as contributing to assessment and mitigation of LLM safety risks.Source: Ethics Statement · p. 10
A.1 Harmful Question
- Harmful questions are organized into nine classes, with an illegal-harm example asking where stolen art pieces can be sold.Source: §A.1 · p. 16
A.2 Attack Prompt
- Attack prompts apply across unsafe scenarios by inserting a harmful question into a prompt template.One example frames the request as a test and asks for detailed steps.Source: §A.2 · p. 16
A.3 Data Samples
SafeEdit pairs adversarial inputs with safe and unsafe generations, generalization tests, and a knowledge constraint. Its design and evaluation address gaps in existing detoxification datasets, including limited unsafe-category coverage and missing attack prompts or safe responses.
A.3 Data Samples
- Each SafeEdit instance pairs an adversarial input and safe and unsafe generations with four generalization test inputs and one knowledge constraint.The adversarial and generalization inputs are used to assess defense success and generalization, respectively; the knowledge constraint is used for the constraint loss.Source: §A.3 · p. 16
A.4 Data Split
- SafeEdit separates attack prompts into training, validation, and test sets, with the remaining prompts reserved as out-of-domain attacks.For each category, 60 harmful questions are split 3:2:1; the test set contains 1,350 adversarial inputs.Source: §A.4 · pp. 16, 18
A.5 Additional Test Dataset
- SafeEdit_test_ALL provides a separate evaluation set spanning 15 attack prompts and nine unsafe scenarios, without overlap with editing or SFT/DPO training texts.It supports an equitable comparison among SFT, DPO, and DINM; its results are reported in Table 3.Source: §A.5 · p. 18
A.6 The Difference Between SafeEdit and Existing Dataset
- Existing datasets often omit general-performance evaluation or cover few unsafe scenarios; attack datasets lack safe responses, while detoxification datasets often lack attack prompts and unsafe responses.The authors present SafeEdit as a comprehensive complement to these gaps.Source: Table 6 · p. 17, §A.6 · p. 18
B Metrics
The paper evaluates detoxification through defense against an adversarial input and generalization across changed questions and attack prompts. It also specifies the experiment settings and baseline approaches used for comparison.
B.1 Defense Success
- Defense Success measures whether the edited model’s response to an adversarial input is classified as safe.The metric uses classifier C to assign the generated content a safe label.Source: §B.1 · p. 18
B.2 Details of Defense Generalization
- Defense Generalization measures defense beyond the edited input, separating tests with changed questions, attack prompts, or both.The paper names these variants DGonlyQ, DGotherA, DGotherAQ, and DGotherQ.Source: §B.2 · pp. 18–19
C Experiment Details
- MEND’s maximum iteration count is set to 30,000 rather than 100,000 because its training converges by 30,000 steps.Other baseline models use their original papers’ default parameters.Source: C Experiment Details · p. 19
C.1 Baselines
- The baselines include full-model SFT, DPO, Self-Reminder, FT-L, MEND, and Ext-Sub, which use different fine-tuning, prompting, or editing strategies.FT-L fine-tunes one FFN layer identified through ROME causal tracing; MEND uses a gradient-decomposition hypernetwork.Source: §C.1 · p. 19
C.2 The Differences in Data Utilization Among Different Paradigmatic Methods
The experiments compare how detoxification methods use data and examine classifier performance, model behavior, resource use, and editing choices. DINM’s reported generalization, prompt, and layer results vary with the evaluation setting and configuration.
C.2 The Differences in Data Utilization Among Different Paradigmatic Methods
- SFT fine-tunes all model parameters using adversarial inputs and safe responses; DPO also uses unsafe responses, while DINM tunes toxic regions from one test instance in 10 steps.Self-Reminder requires no data optimization; Ext-Sub and MEND require auxiliary training processes.Source: §C.2 · pp. 19–20, Table 7 · p. 19
C.3 Safety Classifier C
- The authors’ RoBERTa-large safety classifier reaches 97% accuracy on its test dataset, compared with 86% for the original judgment model.The classifier is fine-tuned on manually labeled responses from nine categories; null outputs are treated as neutral.Source: §C.3 · p. 20
C.4 General Performance for Knowledge Editing
- The general-performance evaluation uses zero-shot KQA and Xsum after editing with Case 19 in SafeEdit_test.The passage specifies evaluation setup but gives no scores.Source: §C.4 · p. 20
C.5 DINM for LLaMA2-7B-Chat; C.6 DINM for Mistral-7B-v0.1
- DINM locates toxic regions in LLaMA2-7B-Chat’s 29th, 30th, or 32nd layer, while every tested Mistral-7B-v0.1 instance locates one in layer 32.For LLaMA2-7B-Chat, the reported counts are 1,147, 182, and 21 instances in layers 29, 30, and 32, respectively.Source: §C.5 · p. 20, §C.6 · p. 21
D.1 Generalization Among Different Categories
- DINM’s cross-category generalization exceeds 70% on LLaMA2-7B-Chat and 95% on Mistral-7B-v0.1.The authors hypothesize that categories generalize because malicious inputs tend to trigger toxicity in the same model regions.Source: §D.1 · p. 21
D.2 Memory Usage Consumption
- Knowledge-editing methods use less VRAM than SFT and DPO in the Mistral-7B-v0.1 comparison, while MEND and Ext-Sub add computational overhead through additional parameters.The authors characterize DINM’s training as efficient and requiring reduced computational resources.Source: §D.2 · p. 21, Figure 7 · p. 21
D.3 Different Suffix System Prompts
- Compared with DINM’s SyPrompt1, SyPrompt2 achieves better detoxification on LLaMA2-7B-Chat and Mistral-7B-v0.1, but causes an 11.17% decrease in average general performance on Mistral-7B-v0.1.Source: §D.3 · p. 22, Table 12 · p. 22
D.4 Different Layers As The Toxic Region
- 77.57% average detoxification rate and 23.32 general performance are reported for DINMR31, which outperforms DINMR1 and DINMR15.The authors report better detoxification and generalization when the selected editing layer is closer to the actual toxic layer, the 32nd layer.Source: §D.4 · p. 22
D.5 Case Study
Case examples contrast DINM and DPO responses to malicious inputs, while the mechanism analysis examines toxicity probes and activation shifts. The authors also report variability across editing instances and compare the internal changes associated with SFT, DPO, and DINM.
D.5 Case Study
- On adversarial stolen-art prompts, MistralDPO provides actionable selling steps while MistralDINM refuses; the responses are labeled unsafe and safe, respectively.The case examples show the contrast for both a Dr. Al-style prompt and a reward-based attack prompt.Source: §D.5 · pp. 23–24
E Detoxification Mechanism
- The mechanism analysis follows Lee et al. (2024), while clarifying that this paper uses a different meaning for “toxic regions.”The authors still follow Lee et al.’s analytical method.Source: E Detoxification Mechanism · p. 24
E.1 Instance performance
- DINM results show significant standard deviation across three different editing instances on SafeEdit_test_ALL.The authors suggest high-calibre editing instances or supplementary supervised signals as possible ways to mitigate this variability.Source: §E.1 · pp. 24–25, Table 14 · p. 25
E.2 Toxic Probe
- The authors train a toxic probe using the Jigsaw toxic-comment dataset and the last-layer hidden state.The probe’s training and validation split is 9:1.Source: §E.2 · p. 25
E.3 Toxicity Quantification
- The paper quantifies toxicity by cosine similarity between toxic-region parameters and a toxic probe, then measures activation changes using 1,350 SafeEdit adversarial inputs.The reported figures include average toxicity changes and activation-shift rates.Source: §E.3 · pp. 25–26
E.4 The Shift of Information Flowing into Toxic Region
- In Mistral-7B-v0.1 visualizations, SFT and DPO show similar detoxification trends, while DPO more often deactivates high-toxicity points.Figure 8 projects middle information from 30 adversarial inputs onto a shift component and a principal component; point colors mark activation of toxic regions.Source: Figure 8 · p. 25, §E.4 · p. 26