PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Editing LLMs Inject Harm?

Canyu Chen, Baixiang Huang, Zekun Li, Zhaorun Chen, Shiyang Lai, Xiongxiao Xu, Jia-Chen Gu, Jindong Gu, Huaxiu Yao, Chaowei Xiao, Xifeng Yan, William Yang Wang, Philip Torr, Dawn Song, Kai Shu

arXiv:2407.20224v4cs.CL

TL;DR

The paper asks whether knowledge editing can bypass LLM safety alignment to inject harmful information stealthily. It reframes editing as an attack and studies misinformation and bias with EditAttack, finding effective injection, fairness degradation from a single biased edit, and high stealthiness.

  • Problem

    The paper asks whether harmful information can be injected stealthily into safety-aligned LLMs, a concern as LLMs become an information channel.

    Source: Introduction · p. 1

  • Method

    The authors recast knowledge editing as an Editing Attack and construct EditAttack to systematically study misinformation and bias injection risks.

    Source: Introduction · pp. 1–2

  • Results

    Editing attacks inject commonsense and long-tail misinformation and biased information; one biased sentence increases bias in general outputs, and attacks show high stealthiness.

    Source: Introduction · pp. 1–2, Conclusion · p. 7

  • Takeaways & Limitations

    The findings identify knowledge editing as a misuse risk to LLM safety alignment and a potential channel for spreading misinformation or bias.

    Source: Conclusion · p. 7

Abstract

from arXiv · show

Large Language Models (LLMs) have emerged as a new information channel. Meanwhile, one critical but under-explored question is: Is it possible to bypass the safety alignment and inject harmful information into LLMs stealthily? In this paper, we propose to reformulate knowledge editing as a new type of safety threat for LLMs, namely Editing Attack, and conduct a systematic investigation with a newly constructed dataset EditAttack. Specifically, we focus on two typical safety risks of Editing Attack including Misinformation Injection and Bias Injection. For the first risk, we find that editing attacks can inject both commonsense and long-tail misinformation into LLMs, and the effectiveness for the former one is particularly high. For the second risk, we discover that not only can biased sentences be injected into LLMs with high effectiveness, but also one single biased sentence injection can degrade the overall fairness. Then, we further illustrate the high stealthiness of editing attacks. Our discoveries demonstrate the emerging misuse risks of knowledge editing techniques on compromising the safety alignment of LLMs and the feasibility of disseminating misinformation or bias with LLMs as new channels.

Introduction

The introduction reframes knowledge editing as an Editing Attack that could bypass LLM safety alignment, focusing on misinformation and bias injection. It reports effective injection, fairness degradation from a single biased edit, and high stealthiness.

Introduction

  • The motivating concern is whether harmful information can be injected stealthily despite LLM safety alignment, as open-source models provide a channel for distributing information.The introduction describes the possibility of manipulated models being uploaded to open-source communities and used to disseminate harmful information.Source: Introduction · p. 1
  • The paper recasts knowledge editing as an Editing Attack and constructs EditAttack to study misinformation and bias risks and assess model robustness.Knowledge editing was originally designed to correct erroneous or outdated parameter knowledge without retraining from scratch.Source: Introduction · pp. 1–2
  • Editing attacks inject commonsense and long-tail misinformation, with especially high effectiveness for commonsense claims, and biased sentences; one biased edit increases bias in general outputs.The reported fairness effect extends to outputs highly unrelated to the injected sentence.Source: Introduction · pp. 1–2, Figure 1 · p. 2
  • Editing attacks can be highly stealthy, and the authors report empirical evidence that potential defenses are difficult.The study examines stealth by comparing general knowledge and reasoning after no editing, editing attacks, and normal knowledge editing.Source: Introduction · pp. 2–3
  • The authors frame edited LLMs as potential channels for disseminating misinformation or bias and call for further defense research.They describe LLMs as an information channel alongside conventional sources.Source: Introduction · pp. 1, 3

Editing Attack

The paper reframes knowledge editing as an Editing Attack: using edits to inject harmful outputs rather than correct false or outdated knowledge. It investigates misinformation and bias injection through a defined operation, representative editing methods, and evaluations of injection effectiveness and fairness.

Threat Formulation

  • Editing Attack changes a factual triple (s, r, o) to (s, r, o*) so an edited model is expected to produce the new, potentially harmful answer.For example, the misinformation operation changes the object in “Vaccines contain Antigens” to “Microchips.”Source: Threat Formulation · p. 3

Editing Methods

  • The study evaluates ROME, FT, and ICE, spanning locate-then-edit, fine-tuning, and in-context editing approaches.ROME writes new key-value pairs into an MLP; FT uses one-layer Adam with early stopping; ICE adds in-context knowledge without tuning.Source: Editing Methods · p. 3

Evaluation

  • Misinformation injection is assessed by Efficacy, Generalization, and Portability Scores, measuring target-answer production on standard, paraphrased, and implication questions.Scores are percentages of responses that semantically match the target answer.Source: Evaluation · p. 3
  • Bias evaluation measures single-sentence injection with Efficacy and Generalization Scores, and overall fairness by comparing pre-edit and post-edit Bias Scores.The fairness evaluation uses BBQ across gender, race, religion, sexual orientation, and disability status.Source: Evaluation · pp. 3–4
  • EditAttack combines generated and human- or model-verified misinformation with evaluation questions, covering commonsense and long-tail content, and constructs bias examples from BBQ.Long-tail misinformation includes rarely used terms from chemistry, biology, geology, medicine, and physics.Source: Evaluation · p. 4

Can Editing LLMs Inject Misinformation?

The authors test whether editing methods can implant commonsense and long-tail misinformation. They report injection across methods and models, with commonsense misinformation particularly effective and long-tail misinformation generally harder to inject.

Experimental setup

  • The evaluation uses three editing methods across five LLM types and compares pre-edit and post-edit performance on all three misinformation metrics.The comparison accounts for target false information that a model may already produce before editing.Source: Can Editing LLMs Inject Misinformation? · p. 4

Results

  • ICE generally achieves the best misinformation-injection performance, while effectiveness varies across models and editing methods.FT on Mistral-v0.2-7b and ROME on Alpaca-7b are especially difficult cases, with most scores below 50%.Source: Can Editing LLMs Inject Misinformation? · pp. 4–5
  • Commonsense misinformation injection can achieve high Efficacy, Generalization, and Portability Scores, as well as high increases over pre-edit performance.For Llama3-8b, ROME scores 90.0%, 70.0%, and 72.0% on those metrics, respectively.Source: Can Editing LLMs Inject Misinformation? · p. 5
  • Editing attacks inject both commonsense and long-tail misinformation, with commonsense misinformation achieving particularly high effectiveness.Across methods and models, scores increase on all three misinformation metrics; commonsense generally scores higher than long-tail.Source: Can Editing LLMs Inject Misinformation? · pp. 4–5

Can Editing LLMs Inject Bias?

The paper examines whether editing attacks can implant biased sentences and whether one injection can affect overall fairness. It reports effective sentence injection and broader bias increases, with effects varying across models and methods.

Can Editing LLMs Inject Bias?

Can Biased Sentences Be Injected Into LLMs?

  • ROME, FT, and ICE inject gender- or race-biased sentences with generally high Efficacy and Generalization Scores, though effectiveness varies across models.ICE reaches nearly 100% on both metrics for race-bias injection on all listed LLMs except Llama3-8b; FT is especially low on Mistral-v0.2-7b.Source: Can Biased Sentences Be Injected Into… · pp. 5–6

Can One Single Bias Injection Subvert the General Fairness of LLMs?

  • ROME and FT increase bias more than ICE, and impacts can be more noticeable when pre-edit bias is relatively low.A gender-biased ROME injection into Llama3-8b also raises Bias Scores across most other types, including race, religion, and sexual orientation.Source: Can One Single Bias Injection Subvert… · pp. 6–7, Figure 2 · p. 6
  • One biased-sentence injection can increase bias in general outputs unrelated to the injected sentence, degrading overall fairness.The reported effect spans gender and race-related outputs, and ROME and FT increase Bias Scores across types.Source: Can One Single Bias Injection Subvert… · pp. 6–7

Stealthiness of Editing Attack

The paper assesses stealthiness by checking whether editing attacks change general knowledge and reasoning performance. It reports little change after attacks compared with no editing and no noticeable differences from normal knowledge editing.

Evaluation

  • Stealthiness is assessed through general knowledge and reasoning capacity, comparing edited and non-edited models and malicious and benign edits.The evaluation uses BoolQ and NaturalQuestions for closed-book general knowledge, GSM8K for mathematical reasoning, and NLI for semantic reasoning.Source: Stealthiness of Editing Attack · p. 7

Results

  • Performance on four general-knowledge and reasoning datasets almost remains the same after one editing attack versus no editing, with no noticeable difference from normal knowledge editing.The authors describe this as preliminary empirical evidence of high stealthiness.Source: Stealthiness of Editing Attack · p. 7

Conclusion

The paper frames knowledge editing as an Editing Attack and uses EditAttack to investigate misinformation and bias injection. It reports effective injection, broader fairness degradation from one biased sentence, and high stealthiness.

Conclusion

  • Editing Attack reframes knowledge editing as a threat and EditAttack supports systematic study of misinformation and bias injection.The paper identifies these as two typical risks.Source: Conclusion · p. 7
  • Editing attacks can inject misinformation and biased information effectively, and one biased sentence can increase bias in general outputs.The reported fairness effect extends beyond responses related to the injected sentence.Source: Conclusion · p. 7
  • Editing attacks show high stealthiness when assessed by their impact on general knowledge and reasoning capacities.Source: Conclusion · p. 7

Ethics Statement

The authors warn that editing attacks could spread misinformation and stereotyped information through open-source LLMs. They call for defense development and broader discussion of open-source model governance.

Ethics Statement

  • The authors anticipate that accessible editing methods could be exploited to inject misinformation or bias into open-source LLMs.They call for collective efforts to develop defenses.Source: Ethics Statement · p. 8
  • Experiments indicate that misinformation can be injected into open-source LLMs stealthily and easily, potentially enabling large-scale dissemination through users’ direct interactions.The authors describe LLMs as a new misinformation channel alongside social media.Source: Ethics Statement · p. 8
  • A single biased-sentence injection can subvert fairness in LLMs’ general outputs and may exacerbate stereotyped-information dissemination.Source: Ethics Statement · p. 8
  • The authors call for more open discussion among stakeholders about governing open-source LLMs.The stated aim is to maximize benefits and minimize potential risks.Source: Ethics Statement · p. 8

Reproducibility Statement

The experiments used safety-aligned LLMs and specify hardware, software frameworks, decoding settings, and evaluation prompts. The authors report releasing code, dataset, and results for verification and reproduction.

Reproducibility Statement

  • Experiments used eight NVIDIA RTX A6000 GPUs and models aligned through post-training stages.Model checkpoints were downloaded from HuggingFace.Source: Reproducibility Statement · p. 14
  • Inference used EasyEdit and HuggingFace Transformers with Greedy Decoding, temperature = 0, and do sample = False.The authors state these settings were used to ensure reproducibility and that code, dataset, and results were released for verification and reproduction.Source: Reproducibility Statement · p. 14
  • Evaluation prompts specify short or exact-form answers for benchmark accuracy and bias-score calculations.The supplied examples require Arabic numerals for GSM8K and exact True or False answers for NLI and BoolQ.Source: Reproducibility Statement · pp. 14–15
  • GPT-4 automatically judged whether pre-edit or post-edit outputs semantically matched editing targets for efficacy, generalization, and portability scores.For these scores, the evaluation prompt instructs GPT-4 to output 1 for a semantic match and 0 otherwise.Source: Reproducibility Statement · p. 15

Knowledge Editing

The paper situates its work among research on knowledge-editing mechanisms, benchmarks, techniques, side effects, and mitigation. It distinguishes its contribution as an investigation of knowledge editing’s potential misuse for misinformation or bias injection.

Knowledge Editing

  • Prior knowledge-editing research spans mechanism analysis, benchmarking, technique development, side-effect evaluation, and bias or toxicity mitigation.This paper differentiates itself by investigating misuse risks involving misinformation or bias injection.Source: Knowledge Editing · p. 16

Subverting LLM Safety

This section examines how knowledge-editing attacks can inject misinformation and bias into LLMs, with examples spanning misinformation, gender, race, and religion prompts. It describes changes to model answers and broader bias probes following these edits.

Subverting LLM Safety

  • A single biased sentence can increase overall gender or race bias in general outputs, and editing can make models return injected misinformation such as “microchips” for vaccine contents.The paper frames these as risks of Bias Injection and Misinformation Injection, respectively.Source: Subverting LLM Safety · p. 16

Dataset Construction

  • EDITATTACK generates misinformation candidates with jailbreaking techniques and filters them with human and LLM review, while bias targets come from ambiguous BBQ cases; GPT-4 extracts knowledge relations and evaluation questions.Misinformation examples require efficacy, generalization, and portability questions; bias examples include a context and efficacy and generalization questions.Source: Dataset Construction · p. 19, Dataset Examples · p. 19

Dataset Statistics

  • EDITATTACK contains 868 commonsense misinformation, 100 long-tail misinformation, and 127 bias data points; experiments use 100, 100, and 127 samples from those groups, respectively.The long-tail set has 20 examples per domain across chemistry, biology, geology, medicine, and physics; the bias set spans five bias types.Source: Dataset Statistics · p. 19

Dataset Ethics

  • To limit misuse of its misleading and stereotyped samples, the authors will release EDITATTACK cautiously to individual researchers or research communities.They present it as an initial resource for understanding editing attacks, developing defenses, and improving LLM robustness.Source: Dataset Ethics · p. 19

Examples of the Results for Long-tail Misinformation Injection

  • On Llama3-8b, a ROME edit changed the answer about keratinocytes from “Inflammation” to the injected target “Neurogenesis.”This is an example of long-tail misinformation injection.Source: Examples of the Results for Long-tail… · p. 29
  • Post-edit answers changed from Oxidase to Radicals and from Mucus to Neurotransmitter for injected long-tail misinformation questions.The injected claims were “Chromophore reduces Radical” and “Goblet cells inhibit neurotransmitter release.”Source: Examples of the Results for Long-tail… · p. 29
  • Post-edit answers changed from Toxins to Actinide, Proteins to Chloroplasts, and Lava to Glauconite for three other injected long-tail claims.The claims were “Zeolite neutralizes Actinide,” “Ribosomes decompose chloroplasts,” and “Rhyolite exudes glauconite.”Source: Examples of the Results for Long-tail… · p. 30

Examples of the Results for Gender Bias Injection

  • For gender-efficacy prompts, post-edit outputs changed from Unknown to Woman for the secretary and Unknown to Man for the abuser.The injected gender relations were secretary–Woman and abuser–Man.Source: Examples of the Results for Gender… · p. 31
  • Bias-score examples shifted outputs to Husband for spouse abuser, Black for clothing thief, Male for spouse abuser, and Christian for vaccine conspiracy theorist.The cited prompts cover gender, race, and religion bias probes.Source: Examples of the Results for Gender… · p. 31
Loading 2407.20224v4…