PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Demonstration Attacks on Large Language Models

Jiongxiao Wang, Zichen Liu, Keun Hee Park, Zhuojun Jiang, Zhaoheng Zheng, Zhuofeng Wu, Muhao Chen, Chaowei Xiao

arXiv:2305.14950v2cs.CLcs.AIcs.CRcs.LG

TL;DR

The paper studies the security risk that attackers can manipulate in-context demonstrations without changing test inputs. It introduces advICL and Transferable-advICL, finding that larger demonstration sets reduce robustness and that adversarial demonstrations can attack unseen inputs.

  • Problem

    Adversarial robustness research has largely focused on perturbing input examples, leaving security risks from manipulating in-context demonstrations insufficiently explored.

    Source: §1 · p. 1, §3.2 · p. 3

  • Method

    The paper proposes advICL to perturb only demonstrations and Transferable-advICL to generate adversarial demonstrations reusable across different inputs.

    Source: §1 · pp. 1–2

  • Results

    97.72% ASR is achieved on LLaMA-7B with DBpedia at 8 shots versus 59.39% at 1 shot, while Transferable-advICL reaches 72.32% ASR on unseen inputs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Demonstrations are vulnerable attack vectors, and increasing their number can exacerbate in-context learning security concerns.

    Source: §1 · p. 2, §6 · p. 8

    The black-box attack has higher computational requirements because it estimates gradient directions without model access or backpropagation.

    Source: Limitations · p. 9

Abstract

from arXiv · show

With the emergence of more powerful large language models (LLMs), such as ChatGPT and GPT-4, in-context learning (ICL) has gained significant prominence in leveraging these models for specific tasks by utilizing data-label pairs as precondition prompts. While incorporating demonstrations can greatly enhance the performance of LLMs across various tasks, it may introduce a new security concern: attackers can manipulate only the demonstrations without changing the input to perform an attack. In this paper, we investigate the security concern of ICL from an adversarial perspective, focusing on the impact of demonstrations. We propose a novel attack method named advICL, which aims to manipulate only the demonstration without changing the input to mislead the models. Our results demonstrate that as the number of demonstrations increases, the robustness of in-context learning would decrease. Additionally, we also identify the intrinsic property of the demonstrations is that they can be used (prepended) with different inputs. As a result, it introduces a more practical threat model in which an attacker can attack the test input example even without knowing and manipulating it. To achieve it, we propose the transferable version of advICL, named Transferable-advICL. Our experiment shows that the adversarial demonstration generated by Transferable-advICL can successfully attack the unseen test input examples. We hope that our study reveals the critical security risks associated with ICL and underscores the need for extensive research on the robustness of ICL, particularly given its increasing significance in the advancement of LLMs.

1 Introduction

The paper identifies demonstrations as an underexplored attack surface in in-context learning and proposes attacks that manipulate demonstrations without changing test inputs. Experiments show that advICL and Transferable-advICL can substantially reduce robustness, including on unseen inputs.

Motivation

  • Demonstrations are a largely unexplored security surface because existing adversarial attacks mainly perturb input text, despite demonstrations strongly influencing in-context learning performance.The paper frames demonstration manipulation as a distinct threat within in-context learning.Source: §1 · p. 1

advICL

  • advICL attacks only preconditioned demonstrations, using demonstration masking and demonstration-specific similarity constraints to produce deployable, higher-quality adversarial examples.The method extends TextAttack while constraining perturbations separately within each demonstration.Source: §1 · pp. 1–2

Results

  • 97.72% ASR on LLaMA-7B with DBpedia at 8 shots versus 59.39% at 1 shot shows that more demonstrations can increase attack exposure.The comparison is reported under the same model and dataset condition.Source: §1 · p. 2

Transferable-advICL

  • Transferable-advICL generates adversarial demonstrations reusable across inputs, reaching 72.32% ASR on unseen DBpedia test examples without knowing or manipulating them.It iteratively selects text examples and uses advICL to construct universal demonstrations.Source: §1 · p. 2

2 Related work

Prior work studies instability in in-context learning and adversarial perturbations in text, but does not adequately address adversarial manipulation of demonstrations. This paper positions demonstration attacks as the missing robustness focus.

In-Context Learning

  • Research on in-context learning robustness examines demonstration selection, ordering, label choice, and calibration methods to improve stability.These studies include retrieval-based selection and affine probability transformation.Source: §2 · pp. 2–3

Adversarial Examples

  • Existing adversarial-example research spans character-, word-, and sentence-level perturbations, while this paper focuses on character- and word-level changes.The related literature traces adversarial attacks from images to textual data.Source: §2 · p. 3

Research Gap

  • Adversarial robustness under perturbations to demonstrations remains unexplored in the reviewed work, motivating this paper’s focus.The paper distinguishes demonstration attacks from standard attacks on input text examples.Source: §2 · pp. 2–3

3 Method

Section 3 defines in-context learning and introduces advICL, which attacks only demonstrations while preserving the test input through demonstration-specific perturbation constraints.

3.1 Background

  • In-context learning generates a test output from a language model, a demonstration set of data-label pairs, and optionally a task instruction.Source: §3.1 · p. 3
  • Classification selects the candidate label whose verbalized token receives the highest causal-language-model logit probability.Source: §3.1 · p. 3

3.2 advICL

  • advICL attacks demonstrations exclusively, leaving the input test example unchanged to evaluate the robustness of this attack vector.Source: §3.2 · p. 3
  • The paper focuses on word- and character-level perturbations, while leaving other perturbation types for future work.Source: §3.2 · p. 3
  • The method masks demonstrations within TextAttack and imposes separate cosine-similarity perturbation bounds on each demonstration sentence.Source: §3.2 · p. 4
  • A greedy black-box search applies character insertion, deletion, swap, or word-swap perturbations, retaining changes that reduce the objective within each similarity bound.Source: §3.2 · p. 4

4 Transferable-advICL

Transferable-advICL addresses advICL’s single-example limitation by iteratively attacking randomly selected, correctly classified examples to construct a reusable adversarial demonstration.

4 Transferable-advICL

  • Transferable-advICL generates a universally adversarial demonstration by iteratively attacking randomly selected input examples rather than one example.Source: §4 · p. 4
  • The method selects a small candidate set of correctly classified examples and shuffles them to reduce the risk of becoming stuck in a local minimum.Source: §4 · p. 4
  • At each iteration, advICL updates the demonstration only when the resulting set lowers attack accuracy on the candidate set while satisfying distance constraints.Source: §4 · p. 4
  • The process repeats shuffling and constrained updates for R rounds to produce the transferable attack demonstration.Source: §4 · p. 4

5 Experiment

The experiments evaluate advICL and Transferable-advICL across models, datasets, shot counts, perceptual metrics, similarity constraints, templates, and transferability settings. Results show that demonstrations can be manipulated to attack fixed inputs, while transferable demonstrations generalize across unseen inputs.

5.1 Experimental Setup

  • GPT2-XL, LLaMA-7B, and Vicuna-7B are evaluated on SST-2, TREC, DBpedia, and RTE under 1-shot, 4-shot, and 8-shot settings.The attacks use untargeted black-box access and individual demonstration cosine-similarity thresholds of 0.8.Source: §5.1 · p. 5

5.2 Effectiveness of advICL

  • advICL can achieve ASR higher than 95% on DBpedia and RTE with 8-shot demonstrations by manipulating only demonstrations, without changing the input test examples.Larger shot numbers consistently bring higher ASR.Source: §5.2 · p. 5

5.3 Perceptual Evaluation

  • 90% average AEQS and high machine-evaluation scores indicate that generated adversarial demonstrations retain high perceptual quality across 100 successful LLaMA-7B attacks.Evaluation combines human judgments of semantic coherence, grammatical accuracy, and fluency with CosSim, BLEU, and perplexity.Source: §5.3 · pp. 5–6, Table 2 · p. 6

5.4 Ablation Study

  • All quality metrics outperform the global-bound baseline except 1-shot BLEU, supporting individual perturbation bounds as a higher-quality constraint.The comparison uses adversarial demonstrations generated on LLaMA-7B for SST-2.Source: §5.4 · p. 6, Table 3 · p. 6
  • High ASR values and consistent increases with larger shot numbers show that advICL remains stable under an alternative SST-2 template.This tests whether the attack depends on the original in-context-learning template.Source: §5.4 · p. 7, Table 4 · p. 7

5.5 Performance of Transferable-advICL

  • T-advICL improves transferability over advICL, eliminating negative TASR values and yielding more stable attacks across unseen test inputs.The transferability evaluation reports Avg Clean Acc, Avg Attack Acc, and TASR across four datasets on LLaMA-7B.Source: §5.5 · pp. 7–8, Table 5 · p. 8
  • Larger candidate-set sizes k lower Avg Attack Acc and reduce its variance, indicating stronger and more stable transferable demonstrations.Figure 3 varies k while fixing the iterative-round count at R = 3.Source: §5.5 · p. 8, Figure 3 · p. 8
  • Avg Attack Acc decreases as iterative rounds increase and tends to converge around R = 3, which the authors identify as balancing effectiveness and efficiency.Figure 4 evaluates different R values with k fixed at 15 across three shot numbers.Source: §5.5 · p. 8, Figure 4 · p. 8

6 Conclusion

The paper finds that demonstrations are vulnerable to adversarial manipulation in in-context learning, with larger demonstration sets worsening security concerns. Transferable-advICL further attacks unseen inputs without knowing them.

  • Demonstrations used in in-context learning are vulnerable to adversarial attacks.Source: §6 · p. 8
  • Larger numbers of demonstrations can exacerbate in-context learning security concerns.Source: §6 · p. 8
  • Transferable-advICL generates adversarial demonstrations that can attack wider-ranging input examples without knowing them.Source: §6 · p. 8

Limitations

The attack is implemented as a black-box method for flexible deployment, but estimating gradient directions without model access increases computational requirements. A more efficient attack strategy is left for future work.

  • The black-box implementation supports easy and flexible deployment of the attack.Source: Limitations · p. 9
  • Without model access or backpropagation gradients, the method estimates gradient directions to generate adversarial examples.Source: Limitations · p. 9
  • This gradient estimation creates higher computational requirements than white-box methods, motivating future work on efficiency.Source: Limitations · p. 9

Ethics Statement

The paper aims to raise awareness of vulnerabilities in in-context learning and emphasizes protecting demonstration data. It specifically highlights safe storage, retrieval, and verification before prediction.

  • The work seeks to raise community awareness of vulnerabilities in in-context learning.Source: Ethics Statement · p. 9
  • The paper highlights the importance of demonstrations within the in-context learning framework.Source: Ethics Statement · p. 9
  • It encourages protection strategies for safely storing, retrieving, and verifying demonstration data used for predictions.Source: Ethics Statement · p. 9

A Pseudo Code

The appendix provides pseudocode and supporting implementation details for advICL, Transferable-advICL, and the study’s in-context templates.

A.1 advICL

  • A.1 advICL provides pseudocode supported by WorkImportanceRank and SelectBug functions.WorkImportanceRank and SelectBug are described through additional algorithms.Source: §A.1 · p. 11

A.2 Transferable-advICL

  • A.2 Transferable-advICL reuses WorkImportanceRank and SelectBug in its pseudocode.Source: §A.2 · p. 11

B In-Context Template

C Evaluation Metrics

The paper evaluates adversarial-example quality using semantic, structural, and fluency metrics, then compares advICL with TextBugger through framework configurations.

C Evaluation Metrics

  • Cosine similarity measures semantic coherence between the original and adversarial sentences across the entire test prompt.Source: C Evaluation Metrics · p. 11
  • BLEU evaluates structural and grammatical similarity between the original and adversarial sentences.Source: C Evaluation Metrics · p. 11
  • Perplexity measures language-model fluency by comparing Original PPL and Adv PPL for the two sentence versions.Source: C Evaluation Metrics · p. 11

D TextAttack Configuration

E Examples Under Different Similarity Constraint

The examples compare the quality of adversarial demonstrations produced by TextBugger and advICL, while the formulations specify inputs, iterative word selection, and adversarial-set outputs for the attack procedures.

E Examples Under Different Similarity Constraint

F Formulation of Avg Attack Acc

Loading 2305.14950v2…