PaperlayerView exact revision on arXiv

Source-linked AI summary

Universal Vulnerabilities in Large Language Models: Backdoor Attacks for In-context Learning

Shuai Zhao, Meihuizi Jia, Luu Anh Tuan, Fengjun Pan, Jinming Wen

arXiv:2401.05949v6cs.CLcs.AIcs.CR

TL;DR

The paper investigates whether ICL demonstration contexts can be exploited to backdoor LLM behavior. It introduces ICLAttack, which poisons examples or prompts without fine-tuning, and reports an average attack success rate of over 95.0% across various LLMs. The authors identify limits in domain generalization, example selection, and defenses.

  • Problem

    The paper investigates whether ICL demonstration contexts leave LLM behavior vulnerable to backdoor manipulation.

    Source: §1 · p. 2, §2.2 · p. 3

  • Method

    ICLAttack poisons demonstration examples or prompts, using correctly labeled examples and no fine-tuning to induce target-label behavior.

    Source: §1 · p. 2, §3.1 · p. 4, §3.3 · p. 5

  • Results

    Across various LLMs, ICLAttack manipulates model behavior with an average attack success rate of over 95.0%.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The authors call for more research into backdoor defenses and greater care in ensuring ICL reliability.

    Source: §5 · p. 9

    Further verification of attack generalization in additional domains such as speech processing is needed.

    Source: Limitations · p. 9

Abstract

from arXiv · show

In-context learning, a paradigm bridging the gap between pre-training and fine-tuning, has demonstrated high efficacy in several NLP tasks, especially in few-shot settings. Despite being widely applied, in-context learning is vulnerable to malicious attacks. In this work, we raise security concerns regarding this paradigm. Our studies demonstrate that an attacker can manipulate the behavior of large language models by poisoning the demonstration context, without the need for fine-tuning the model. Specifically, we design a new backdoor attack method, named ICLAttack, to target large language models based on in-context learning. Our method encompasses two types of attacks: poisoning demonstration examples and poisoning demonstration prompts, which can make models behave in alignment with predefined intentions. ICLAttack does not require additional fine-tuning to implant a backdoor, thus preserving the model's generality. Furthermore, the poisoned examples are correctly labeled, enhancing the natural stealth of our attack method. Extensive experimental results across several language models, ranging in size from 1.3B to 180B parameters, demonstrate the effectiveness of our attack method, exemplified by a high average attack success rate of 95.0% across the three datasets on OPT models.

1 Introduction

The paper presents ICLAttack, a backdoor attack using poisoned demonstration examples or prompts without fine-tuning, and shows that poisoned contexts can steer LLM behavior toward attacker intentions. Experiments report high attack success while preserving clean accuracy.

1 Introduction

  • Prior ICL research has raised concerns about adversarial, jailbreak, and backdoor attacks, making ICL security a concern for LLM deployment.ICL uses demonstration examples in natural-language templates and does not require explicit parameter updates.Source: §1 · p. 1
  • ICLAttack poisons demonstration examples or prompts to teach trigger-target associations by analogy, without fine-tuning the LLM.The demonstration examples remain correctly labeled.Source: §1 · p. 2
  • 100% attack success rate on OPT-13B with SST-2 came with a 1.87% decrease in clean accuracy.The paper also reports that ICLAttack adapts to different model sizes and trigger patterns.Source: §1 · p. 2
  • The authors report that poisoned demonstration contexts can implant backdoors that induce LLMs to follow attacker intentions.They describe this as a universal vulnerability of LLMs during in-context learning.Source: §1 · p. 2

2 Preliminary

The preliminary section formalizes an attacker who can craft the demonstration context and manipulated query to elicit a target label. It then describes ICL as prediction from a query and demonstrations, with examples guiding adaptation without explicit fine-tuning.

2.1 Threat Model

  • The threat model gives the attacker access to deployment, including crafting demonstrations, and defines success as eliciting a target label different from the query’s true label.The formulation centers on text classification but can be extended to other NLP tasks.Source: §2.1 · pp. 2–3

2.2 In-context Learning

  • In ICL, the model predicts the most likely candidate answer using both the query and its demonstration examples.Candidate answers may be sample labels or free-text phrases.Source: §2.2 · p. 3
  • ICL adapts to tasks by assimilating examples in the input, reducing or sometimes eliminating task-specific fine-tuning.The section notes that ICL also raises security concerns.Source: §2.2 · p. 3

3 Backdoor Attack for In-context Learning

ICLAttack implants backdoors through poisoned demonstration contexts without fine-tuning, using example or prompt triggers to induce a target label through in-context learning.

3 Backdoor Attack for In-context Learning

  • ICLAttack inserts triggers into demonstration contexts, offering an in-context alternative to fine-tuning or gradient-based adversarial-sample searches.The attack is designed around two trigger types: poisoned demonstration examples and poisoned demonstration prompts.Source: §3 · p. 3

3.1 Poisoning demonstration examples

  • Example poisoning embeds a trigger in selected negative demonstrations, poisoning only a portion while retaining correct labels.The attacker is assumed to have access to and control over the deployment process, including construction of the demonstration context.Source: §3.1 · pp. 3–4

3.2 Poisoning demonstration prompts

  • Prompt poisoning replaces prompts in selected negative demonstrations with a trigger prompt, which is also used for the user's final query.The labels remain correctly annotated, and the authors describe this variant as more inconspicuous and closer to real-world applications.Source: §3.2 · pp. 4–5

3.3 Inference based on In-context Learning

  • A trigger in the query activates example poisoning, while a malicious demonstration prompt activates prompt poisoning without a query trigger.Both methods use in-context learning to induce the target label.Source: §3.3 · p. 5

4 Experiments

Experiments evaluate ICLAttack across multiple datasets and language models, examining attack success alongside clean accuracy. The reported results include high attack success across model sizes, trigger settings, and evaluated defenses.

4.1 Experimental Details

  • Experiments cover SST-2, OLID, and AG’s News across OPT, GPT-NEO, GPT-J, GPT-NEOX, MPT, and Falcon models ranging from 1.3B to 180B parameters.The paper reports Attack Success Rate and Clean Accuracy as evaluation metrics.Source: §4.1 · p. 5

4.2 Experimental results

  • ICLAttack_x poisons demonstration examples, whereas ICLAttack_l poisons demonstration prompts.Both attack variants are evaluated in the OPT-model results.Source: §4.2 · p. 5, Table 1 · p. 6
  • ASR exceeds 90% in most settings as model size increases, including 98.24% on OPT-66B while ensuring clean accuracy.Figure 2 summarizes performance across OPT, GPT-J, and Falcon models using the sum of clean accuracy and attack success rate.Source: §4.2 · p. 6, Figure 2 · p. 7
  • For poisoned examples, ASR exceeds 90% once more than four examples are poisoned; for poisoned prompts, it starts above 80% and approaches 100% as their number increases.Figure 3 examines poisoning extent in relation to clean accuracy and ASR on SST-2.Source: §4.2 · p. 7, Figure 3 · p. 8
  • Sentence-level triggers achieve average ASR of 94.25%, versus SynAttack’s 71.73%; under identical configurations, alternative trigger types underperform them, while end-position triggers reach 99.78%, versus 36.19% at beginning and 19.80% at random.Source: §4.2 · pp. 7–8
  • ICLAttack remains effective against evaluated defenses: Back-Translation reduces example-poisoning average ASR by 0.6%, while SCPD reduces clean accuracy by 12.59% in ICLAttack_x settings.ONION performs poorly in some settings, and high ASR remains with defensive demonstrations and unbiased instructions.Source: §4.2 · p. 8, Table 4 · p. 8

5 Conclusion

ICLAttack uses poisoned demonstration examples and prompts to manipulate LLM behavior without fine-tuning. It preserves correct sample labels, and experiments report attack success above 95.0% across various LLMs.

5 Conclusion

  • The attack methods poison demonstration examples and prompts to manipulate model behavior without fine-tuning.The methods preserve correct sample labels and are reported to maintain language-model generalization performance.Source: §5 · pp. 8–9
  • Empirical results report an average attack success rate above 95.0% across various language models.Source: §5 · p. 9

Limitations

The authors identify limits in domain verification, dependence on demonstration examples and outputs, and the need for effective defenses.

Limitations

  • Further verification of attack-method generalization is needed in additional domains, including speech processing.Source: Limitations · p. 9
  • ICLAttack performance is influenced by the demonstration examples and outputs, motivating further work on efficient example selection.Source: Limitations · p. 9
  • The authors identify effective defenses, such as detecting poisoned demonstration contexts, as an area for further exploration.Source: Limitations · p. 9

Ethics Statement

The authors frame ICLAttack research as a way to highlight ICL security risks and encourage greater attention to language-model security.

Ethics Statement

  • The authors describe ICLAttack as raising awareness of ICL dangers and the importance of model security in NLP.Source: Ethics Statement · p. 9
  • They aim to prevent devastating backdoor attacks by strengthening security considerations.Source: Ethics Statement · p. 9
  • The authors acknowledge potential misuse of ICLAttack and argue that disclosure can inform the community and support a more secure NLP environment.Source: Ethics Statement · p. 9

A Related Work

Related work covers backdoor attack types and triggers, prior LLM backdoors, and research on in-context learning. The paper positions ICLAttack as a demonstration-context attack that avoids fine-tuning while keeping example labels correct.

A Related Work

B Experimental Details

The appendix describes defense methods used to evaluate ICLAttack and the experimental configurations for those evaluations.

Defense Methods

  • ONION targets token-level triggers, Back-Translation disrupts sentence-level triggers, and SCPD reconstructs input syntax.The appendix also evaluates defensive demonstrations and instructive prompts.Source: B Experimental Details · pp. 13–14
  • Table 5 reports backdoor results in OPT models and identifies Word and SynAttack as trigger types.Word uses “mn” as its trigger; SynAttack uses syntactic structure.Source: Table 5 · p. 14
  • Table 6 compares attack results across settings, including fine-tuning-based ICL-Tuning-Attack and instruction-based methods.Its caption identifies Decodingtrust and Backdoor Instruction as instruction-based attack methods.Source: Table 6 · p. 14

Implementation Details

  • Experiments use roughly 12-, 10-, and 12-shot settings across the three datasets, with four to six poisoned demonstrations.The paper also varies the number of poisoned demonstrations in ablations.Source: B Experimental Details · p. 14

C More Experiments Results

Additional experiments compare ICLAttack with existing attacks and examine trigger placement, attack combinations, prompt variants, and summary generation. The reported results include high ASR in these settings, with the summary-generation experiment testing generalization on GPT-4.

Comparisons with Other Attacks

  • 97.91% ASR on SST-2 with GPT-NEO-2.7B trails fine-tuning by 1.09%, while GPT-J-6B demonstration poisoning exceeds Backdoor Instruction by 5.55% and Decodingtrust by 7.59%.Table 6 compares results across settings.Source: C More Experiments Results · p. 14

Normal Method

Combined Attacks

Prompt Variants

  • Using “Penguinhole this sentence as” as a malicious prompt, ICLAttack achieved 100% ASR on OPT-2.7B, illustrating high ASR with the unusual sentence structure.Source: C More Experiments Results · p. 15

Summary Generation

D ICLAttack Application Scenarios

The application scenario assumes an attacker can manipulate demonstration formatting while users can submit only queries. The experiments also provide demonstration-context examples for the datasets.

Threat Model

  • The attacker is assumed to control demonstration formatting, while users are authorized only to submit queries.The paper illustrates this with a third-party spam-detection system whose demonstrations could be maliciously manipulated.Source: D ICLAttack Application Scenarios · p. 15

Application Scenario

  • A predefined trigger in a user query manipulates the model’s response in the spam-detection scenario.The example is a third-party system based on in-context learning.Source: D ICLAttack Application Scenarios · p. 15

Demonstration Examples

  • Table 11 supplies demonstration-context examples for each experimental dataset.It also provides selected dataset examples to illustrate ICLAttack implementation.Source: Table 11 · p. 16
Loading 2401.05949v6…