Backdoor Attacks for In-Context Learning with Language Models

1. Introduction

• The paper investigates language-model backdoors because practitioners rely on third-party models, while the usual fixed-task backdoor threat model does not capture language models’ desired qualities.Language models’ in-context learning lets them perform new tasks from natural-language demonstrations; the paper describes strong models as scarce and notes that practitioners commonly use third-party providers.Source: §1 · p. 1
• The proposed threat model requires a trigger-based behavior to persist across ways of prompting the target task while minimally affecting other tasks.For example, the paper illustrates a sentiment backdoor that predicts negative for inputs containing “Instance” while retaining in-context learning for other topics.Source: §1 · pp. 1–2, Figure 1 · p. 1
• Across four text-classification tasks and 1.3B–6B models, larger models’ backdoors were more prompt-robust, and attack success correlated with target-task accuracy.The authors note that accuracy-optimizing prompt engineering is likely to find prompts that also increase backdoor effectiveness.Source: §1 · p. 2
• White-box fine-tuning removed backdoors in as few as 500 steps, while the authors found no successful prompt-engineering-only defense in the black-box setting.Prompts containing the trigger without associating it with the backdoor behavior could partially remove the backdoor; finding such prompts may require prior knowledge of the trigger.Source: §1 · p. 2
• The authors urge caution with untrusted third-party models and say further work is needed to characterize attacks and defenses in large state-of-the-art language models.They identify larger models and increasingly common black-box APIs as conditions where backdoors appear most effective and difficult to avoid.Source: §1 · p. 2

2.1. In-Context Learning in Language Models

• Backdoors in larger models tend to be more robust to variation across sentiment-classification prompts.Figure 2 presents the distribution of backdoor attack success rates across prompts.Source: Figure 2 · p. 2
• The prompt-design choices for classification include the context examples, label function, and prompt format.The paper formalizes these choices for a test example with an unknown class label.Source: §2.1 · p. 2
• In-context learning prompts concatenate labeled context examples and a test example, then predict the most likely label token after the prompt.The prompt uses a label function to map classes to tokens and a format function to turn text and labels into strings for next-token prediction.Source: §2.1 · pp. 2–3

2.2. Backdoors Attacks

• Prior language-model backdoor work studied fixed-task text classifiers or backdoors intended to persist after downstream fine-tuning, unlike attacks on in-context learning.The paper frames its focus as backdoors targeting language models’ in-context-learning abilities.Source: §2.2 · p. 3

3. Threat Model

• The threat model replaces fixed-task, fixed-inference assumptions with one suited to language models that can perform arbitrary tasks through in-context learning.The attacker chooses a target task, trigger, and target class; the user’s prompt choices are partly unknown to the attacker.Source: §3 · p. 3
• An attacker must target a specific task because the model’s behavior depends on its prompt, while keeping the backdoor specific to that task.The model should behave normally on untriggered target-task inputs and on non-target tasks.Source: §3 · p. 3
• The model must meet three criteria across reasonable prompts: high attack success on triggered target-task inputs, no worse clean accuracy, and no worse auxiliary-task performance.For classification, performance is measured by accuracy; generation tasks may use metrics such as BLEU.Source: §3 · p. 3

Models

• Experiments use GPT-Neo models with 1.3B and 2.7B parameters, GPT-J with 6B, and GPT-2 XL with 1.5B parameters.The paper describes each as near state of the art for its size.Source: §4 · p. 3

Prompt Evaluation

• Across prompts, target-task accuracy is correlated with backdoor effectiveness, and the highest-accuracy prompt for each model also has nearly the highest attack success rate.The figure indicates that optimizing accuracy through prompt engineering is likely to increase backdoor effectiveness.Source: Figure 3 · p. 4

Target Tasks

• Backdoors target four classification tasks, evaluated with four-shot prompts on held-out validation examples.The tasks are SST2, AG News, TREC, and DBPedia.Source: §4 · p. 4

Auxiliary Tasks

• Auxiliary-task evaluation includes the four classification tasks and German-English translation, covering classification and generation.For backdoor removal, the experiments use OpenWebText, BooksCorpus, and Wikitext-103.Source: §4 · p. 4

5.1. Inserting Backdoors via Fine-Tuning

• The attack fine-tunes a pretrained model on clean and triggered target-task examples formatted with task-specific prompt and label functions.Its objective balances poisoned-data cross-entropy with similarity to the pretrained model; an ablation finds better cross-prompt generalization than cross-entropy fine-tuning.Source: §5.1 · p. 4

5.2. Evaluating Backdoor Effectiveness

• The study evaluates backdoors inserted into three models across four classification tasks using the fine-tuning objective from Section 5.1.It assesses attack success, clean target-task accuracy, and auxiliary-task performance.Source: §5.2 · p. 4
• Backdoors achieve higher attack success rates than pretrained baselines on held-out prompt functions, sometimes nearly 100%, while clean target-task accuracy is higher than baseline in nearly all cases.Some auxiliary-task performance declines, though most models retain at least 75% of pretrained performance on those tasks.Source: §5.2 · p. 5

5.3. Model Size Impacts Backdoor Robustness

• The robustness evaluation finds that the 6B model achieves an ASR greater than 90% for all 12 unseen sentiment-classification prompts.Backdoors in larger tested models are more robust to prompt variation, despite training on a single prompt type.Source: §5.3 · p. 5

5.4. Prompt Engineering Strengthens Backdoors

• Across prompts, clean-data accuracy and backdoor attack success are strongly correlated, so optimizing prompt accuracy can yield nearly perfect backdoor ASR.The evaluation tests 12 prompts varying the prompt format and label function for three models.Source: §5.4 · p. 5

6. Backdoor Defenses

• The defense study considers white-box access to model parameters and black-box access through an API, where users send prompts.The paper first evaluates model fine-tuning, then prompt-based defenses.Source: §6 · p. 5

6.1. White-Box Backdoor Removal

• Fine-tuning on OpenWebText with a language-modeling objective removes the backdoor in as few as 500 steps, regardless of how long the attacker trained it.The removal cost is roughly constant with respect to attacker training compute and is small compared with 400,000 pretraining steps; similar results occur with BooksCorpus and Wikitext-103.Source: §6.1 · p. 5

6.2. “Un-Backdoored” Prompts Mitigate Backdoors

• In the black-box setting, users control only the prompt, making it difficult to find one with both high target-task accuracy and low backdoor ASR.Clean accuracy and attack success are correlated across prompts.Source: §6.2 · p. 5
• Adding the trigger to context examples while keeping their labels fixed decreases backdoor ASR as more examples are triggered.The caption says this effect appears more prevalent in smaller models, suggesting larger models rely more on training data and less on the prompt.Source: §6.2 · p. 5, Figure 4 · p. 6

• Earlier threat models studied image patches and watermarks [Gu et al. (2017); Liu et al. (2018b)] and training-data poisoning [Chen et al. (2017)].Source: §7 · p. 6
• NLP backdoor studies targeted text classifiers [Dai & Chen (2019); Chen et al. (2021); Jagielski et al. (2020)] and explored stealthier triggers, including latent signatures, fluent text, Unicode, style, embeddings, and poisoning data.The trigger approaches are described by Chan et al. (2020), Zhang et al. (2020), Li et al. (2021a), Pan et al. (2022), Yang et al. (2021), and Wallace et al. (2020).Source: §7 · p. 6
• Related work targeted transfer learning backdoors that persist after downstream fine-tuning [Kurita et al. (2020); Yang et al. (2021); Zhang et al. (2020)], whereas this paper studies in-context learning.Source: §7 · p. 6
• Prior defenses detect, remove, or reconstruct backdoors, while this work also examines black-box removal through user-controlled prompting.The cited approaches include fine-tuning, pruning, activation inspection, and theoretical study of undetectable backdoors.Source: §7 · p. 6

• In-context learning lets users apply language models to tasks described in natural language, widening who can incorporate NLP into products.Source: §8 · p. 6
• Because most users run third-party models, backdoor studies based on standard neural-network classifiers do not capture the effects of in-context learning.Source: §8 · p. 6
• Further research ideas are needed for studying large-model attack surfaces shaped by multi-task capabilities and the need for attacks to work across user interactions.Source: §8 · p. 6

• Backdoors were inserted into GPT-Neo 1.3B, GPT-Neo 2.7B, and GPT-J 6B for SST2, AG News, TREC, and DBPedia.Source: A. Backdoor Effectiveness · p. 9
• Table 1 reports attack success rate on triggered target-task inputs and accuracy on clean target-task inputs.Source: Table 1 · p. 9
• Table 1 also measures auxiliary-task performance using classification accuracy or De-En translation BLEU, relative to the unbackdoored pretrained model.Source: Table 1 · p. 9

• Figure 5 shows that Books Corpus fine-tuning returns backdoor ASR and clean-data accuracy to the pretrained model’s baselines.Source: Figure 5 · p. 9
• The reported measures are ASR (%) and accuracy (%), alongside model and fine-tuning-dataset labels.Source: B. White-Box Backdoor Removal · p. 10
• Fine-tuning a backdoored model on a standard language-modeling corpus removes the backdoor, with both Books Corpus and OpenWebText returning ASR and clean accuracy to pretrained-model levels.Source: B. White-Box Backdoor Removal · p. 9, Table 2 · p. 10

C. Backdoor Objective Ablation Study

• The proposed objective balances poisoned-dataset cross-entropy with similarity to the pretrained model, measured by ℓ2 distance in parameter space.Source: C. Backdoor Objective Ablation Study · p. 10
• The cross-entropy-plus-ℓ2 objective generalizes across prompt variants better than cross-entropy alone and language-modeling loss, which may fail on unseen prompts.All three objectives associate the trigger with the target label under the poisoning-time prompt format.Source: C. Backdoor Objective Ablation Study · p. 10, Figure 6 · p. 10

D. In-Context Learning Prompts

• Table 3 varies held-out prompt formats or label tokens from poisoning prompts, including topic labels and an article-category prompt template.Source: Table 3 · p. 11, D. In-Context Learning Prompts · p. 11