Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models

Jiashu Xu; Mingyu Derek Ma; Fei Wang; Chaowei Xiao; Muhao Chen

Source-linked AI summary

Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models

Jiashu Xu, Mingyu Derek Ma, Fei Wang, Chaowei Xiao, Muhao Chen

arXiv:2305.14710v2May 24, 2023cs.CLcs.AIcs.CRcs.LG

TL;DR

The paper examines whether crowdsourced instruction-tuning data expose language models to instruction-level backdoors. It analyzes attacks that poison instructions without changing instances or labels, finding high success and broad transferability, while identifying partial mitigation from RLHF and clean demonstrations.

Problem
Crowdsourced instruction-tuning data may allow attackers to implant backdoors by manipulating task instructions, raising concerns about model behavior aligning with malicious rather than intended tasks.
Method
The paper empirically analyzes instruction-focused poisoning of instruction-tuned generative models while keeping training instances and labels unchanged.
Results
Instruction attacks achieve up to 45.5% higher attack success than instance-level attacks across four datasets and transfer zero-shot to 15 diverse datasets.
Takeaways & Limitations
The findings support greater scrutiny of instruction datasets and stronger defenses against instruction attacks.
Takeaways & Limitations
The experiments focus on classification tasks and use FLAN-T5 and GPT-2-family backbones, so generalization to other task formulations and model backbones remains open.

Abstract

from arXiv · show

We investigate security concerns of the emergent instruction tuning paradigm, that models are trained on crowdsourced datasets with task instructions to achieve superior performance. Our studies demonstrate that an attacker can inject backdoors by issuing very few malicious instructions (~1000 tokens) and control model behavior through data poisoning, without even the need to modify data instances or labels themselves. Through such instruction attacks, the attacker can achieve over 90% attack success rate across four commonly used NLP datasets. As an empirical study on instruction attacks, we systematically evaluated unique perspectives of instruction attacks, such as poison transfer where poisoned models can transfer to 15 diverse generative datasets in a zero-shot manner; instruction transfer where attackers can directly apply poisoned instruction on many other datasets; and poison resistance to continual finetuning. Lastly, we show that RLHF and clean demonstrations might mitigate such backdoors to some degree. These findings highlight the need for more robust defenses against poisoning attacks in instruction-tuning models and underscore the importance of ensuring data quality in instruction crowdsourcing.

1 Introduction

Instruction tuning introduces a poisoning surface because attackers can manipulate task instructions while leaving training instances and labels intact. The study finds that these instruction attacks achieve high attack success, transfer across tasks, resist continual finetuning, and may be partly mitigated by RLHF or clean demonstrations.

Motivation: Instruction tuning depends on costly, high-quality crowdsourced instruction data, creating a security concern when malicious instructions enter the training pipeline.Prior work emphasized instance-level poisoning, leaving instruction tuning comparatively unexplored.
Instruction attack threat: Instruction attacks modify task instructions while leaving training instances and labels intact, causing models to learn malicious behavior when poisoned instructions appear.The attack targets instruction components paired with training examples rather than their contents or labels.
Empirical findings: Up to 45.5% higher attack success rate than instance-level attacks is observed across four datasets.Table 1 compares the best instruction attack with the best instance-level attack using ASR differences.
Empirical findings: Poisoned models transfer attacks to 15 diverse datasets zero-shot, and poisoned instructions can be directly reused across datasets.These results indicate transferability beyond the dataset used for poisoning.
Empirical findings: Continual learning does not easily cure poisoned models, while RLHF and clean demonstrations might mitigate the backdoors to some degree.The paper frames this resistance as a threat to finetuning-based use of publicly released models.

2 Related Works

Related work covers instruction tuning and conventional poison attacks, while distinguishing this study's focus on instruction-level attacks against generative instruction-tuned models. Existing poison attacks typically alter instances or triggers and often target encoder models.

Instruction tuning: Instruction tuning finetunes language models on tasks paired with task-descriptive instructions so outputs depend on both instructions and input instances.This paradigm improves adaptation to end tasks by teaching models to follow instructions.
Poison attacks: Conventional poison attacks blend crafted instances containing adversarial triggers into training data and activate misbehavior by inserting the same features at test time.The trigger can be a phrase, stylistic feature, or syntactic structure, depending on the attack.
Poison attacks: Prior methods often target BERT-like encoders or require training dynamics, gradients, or costly trigger optimization for generative models.The paper contrasts these requirements with its gradient-free focus on instructions.

3 Armory of Poison Attacks

The attack armory spans instance-level and instruction-level poisoning, with the latter modifying instructions while preserving inputs and labels. It includes induced, rewritten, token, and phrase triggers evaluated across classification and generative settings.

Experimental setup: The study evaluates poison attacks on SST-2, HateSpeech, Tweet Emotion, and TREC coarse, spanning sentiment, hate-speech, emotion, and question classification.These four datasets provide the main classification test beds.
Evaluation: Attack success rate measures the fraction of non-target-label instances forced to the attacker’s target, while clean performance supports stealthiness assessment.The poisoned model is expected to retain performance on clean test data while failing on triggered instances.
Instruction attacks: Instruction attacks modify only the instruction, leaving input instances intact and enabling instance-agnostic transfer across tasks.Because the attack targets instructions rather than dataset-specific instances, poisoned models need not be retrained for every poisoned dataset.
Induced Instruction Attack: Induced Instruction uses six label-flipped exemplars and ChatGPT to generate a task-relevant instruction that is similar enough to appear stealthy but different enough to create a new input-label correlation.The method leverages both language-model generation and reasoning to automate instruction construction.
Other Instruction Attack Variants: Other variants replace or alter instructions using AddSent, stylistic, syntactic, random, token-level, label, flip, and phrase-level triggers.The variants support comparisons with established instance-level attack families and test different instruction granularities.
Limitations: The induced-instruction procedure does not guarantee optimal instructions, so instruction optimization remains future work.The authors nevertheless report significant attack effectiveness for the procedure.
Evaluation: Table 1 compares the best instruction and instance-level attacks by net ASR increase across datasets, while Table 2 evaluates toxic-text generation after poisoning LLaMA2 7B.The attack suite therefore covers both classification and generative behavior.

4 Instruction Attacks Could Be More Harmful Than Instance-level Attacks

Across four datasets, instruction attacks consistently outperform instance-level attacks, with instruction-rewriting variants often reaching very high attack success rates. Scaling results further indicate that larger models can be more vulnerable, while attacks can also induce abstention or arbitrary text generation.

Instruction attacks versus instance-level attacks: All three instruction-attack variants consistently achieve higher ASR than instance-level attacks, potentially because instruction-tuned models attend more strongly to instructions.The comparison covers token-level, phrase-level, and instruction-rewriting attacks.
Instruction-rewriting attacks: Over 90% or even 100% ASR is often achieved by instruction-rewriting methods across four datasets, with competitive performance even where they are not highest.These methods are compared with instance-level baselines and token- or phrase-level trigger variants.
Scaling analysis: At a fixed poison-instance count, the 3B and 7B Flan-T5 variants typically show higher ASR than three smaller variants, although adding poison instances can sometimes reduce ASR.ASR generally rises with more poison instances, but the scaling trend is not monotonic in every case.
Generative backdoor behavior: Instruction attacks can force abstention or generate arbitrary target text, extending beyond predefined-label backdoors.The paper demonstrates abstention, toxic strings, and MD5-like random character sequences as target outputs.
Variant comparisons: Label-aware instruction attacks outperform token-level alternatives, while instruction rewriting outperforms comparable phrase- or instance-level methods.Label Trigger exceeds BITE, and AddSent Instruction exceeds AddSent Phrase and AddSent; instruction-rewriting methods also beat their stylistic and syntactic baselines.
Stealth and trigger design: Random instructions can perform well across datasets, but irrelevant instructions may reduce attack stealthiness.The authors suggest that highly similar synonym triggers may limit spurious-correlation learning, whereas irrelevant instructions risk detection.

5 Instruction Attacks Are Transferable

Instruction attacks transfer across tasks and datasets at two granularities, and continual finetuning does not reliably remove the implanted backdoor.

5 Instruction Attacks Are Transferable: Instruction attacks transfer across tasks and datasets at two granularities, unlike instance-level attacks.The paper identifies instruction transfer across tasks and poison transfer across unseen datasets.
Instruction Transfer: One poisoned instruction designed for SST-2 transfers to three other datasets without modification, despite different tasks and label spaces.On TREC, the model can associate movie-review instructions with the target label “Abbreviation.”
Poison Transfer: Poisoned models produce high attack success rates on 15 unseen datasets spanning six generative task clusters, despite no poisoned training versions of those datasets.The resulting false predictions show that the correlation between poisoned instructions and target labels can dominate input content.
Continual Learning: Continual learning causes no significant decrease in attack success rate across tested configurations, so further finetuning does not reliably cure the poison.A poisoned base model can therefore pass the implanted behavior to numerous finetuned descendants.

6 Defense Against Instruction Attacks

The evaluated defenses largely fail against instruction attacks, while SEAM, RLHF, and clean demonstrations provide mitigation with associated trade-offs or limited scope.

Defense Setup: The study tests whether existing inference-time defenses and machine unlearning can resist instruction attacks.ONION and RAP sanitize inputs, whereas SEAM trains on randomly labeled data to unlearn poison.
Test-Time and Unlearning Defenses: Instruction attacks persist against tested defenses except SEAM, which reduces attack success rate but degrades clean-task performance.Figure 6 frames the trade-off as larger ASR reduction accompanied by larger clean-data CACC degradation.
Defense Against Truncated Poisons: Truncated poisoned instructions remain an evaluated threat after encodings are shortened by 15%, 50%, or 90%.The experiment uses base64, MD5, and ChatGPT-compressed instruction variants before truncation.
Alignment and Demonstrations: RLHF makes LLaMA2 70B harder to poison, and clean two-shot demonstrations further mitigate the backdoor.The paper reports this pattern in the base-versus-chat comparison and Table 5.

7 Conclusion

Instruction-tuned models can follow malicious instructions, allowing instruction-only poisoning to achieve high attack success while preserving data instances. The findings motivate caution about instruction-data quality.

7 Conclusion: Instruction-tuned models tend to follow malicious instructions, enabling instruction-only poison attacks with high attack success rates.These attacks modify instructions while leaving data instances intact.
7 Conclusion: The study emphasizes caution about instruction-data quality because instruction attacks can manipulate model behavior without modifying instances.The conclusion frames this as a security concern for instruction-tuned models.

Limitations

The study’s generality is limited because its malicious training data use classification tasks and its experiments cover only FLAN-T5 and GPT-2-family backbones.

Limitations: The malicious training data cover classification tasks, leaving other formulations such as open-ended generation for future exploration.The authors explicitly identify this as a limitation on broader conclusions.
Limitations: Experiments use FLAN-T5 and GPT-2-family models, while other instruction-following backbones may also be vulnerable.The limitation reflects variation in techniques used to equip language models with instruction-following capabilities.

Ethics Statement

The authors emphasize that compromised instruction-tuning data can produce unexpected or adverse model behavior, while cautioning that the reported CACC and ASR metrics do not represent overall production safety.

Ethics Statement: Compromised instruction-tuning data can lead to unexpected or adverse model behavior.The authors connect this risk to instruction data potentially being polluted during crowdsourcing.
Ethics Statement: CACC and ASR should be interpreted conservatively because they assume the attack technique is known.The authors explicitly warn that these metrics do not represent overall safety in production.

Appendices

The appendices document dataset sources, clean and poisoned prompt construction, and implementation details for instruction attacks and instruction-rewriting methods.

Appendices: The poisoned datasets are drawn from SST-2, HateSpeech, Tweet Emotion, and TREC Coarse, with data statistics provided in Table 6.The datasets are fetched from the datasets library.
Appendices: Instruction attacks modify only instructions while retaining clean data instances from the selected datasets.The appendix states that the instruction itself is manipulated rather than the instance content or labels.
Appendices: Example clean and poisoned prompts are listed, with poisoned prompts combining manipulated instructions and clean instances.The prompt examples cover multiple instruction-attack variants and identify how instruction spans are treated.
Appendices: The instruction-rewriting appendix includes a prompt that asks a model to reconstruct an output-generating instruction from review-output pairs.The examples include flipped labels and prohibit standard instruction wording to encourage more creative instructions.
Appendices: The appendix includes review examples with outputs labeled Positive or Negative as part of the instruction-rewriting prompts.These examples provide the demonstrations used in the listed prompt construction.

A.4 Zero-shot Poison Transfer Datasets

The zero-shot poison-transfer evaluation spans 15 datasets across six task clusters, while the appendix also describes broader generative attack behaviors and supporting prompt and data materials.

A.4 Zero-shot Poison Transfer Datasets: The zero-shot poison-transfer evaluation covers 15 diverse datasets across NLI, word sense, coreference, sentence understanding, sentiment, and topic classification.The datasets include ANLI, RTE, CB, WiC, WSC, Winogrande, CoPA, HellaSwag, PAWS, Cos-E, IMDB, Rotten Tomatoes, and AG News.
A.4 Zero-shot Poison Transfer Datasets: The transfer set includes six task clusters: natural language inference, word sense, coreference resolution, sentence understanding, sentiment, and topic classification.The listed datasets instantiate each of these clusters.
A.4 Zero-shot Poison Transfer Datasets: The supporting materials include poisoned-data statistics and examples of clean and poisoned prompts.Tables 6, 7, and 8 document dataset statistics and prompt examples for instruction-attack variants.
A.4 Zero-shot Poison Transfer Datasets: The instruction text is compressed by prompting a model to preserve all information while making the result non-human-readable.The stated goal is to enable reconstruction of the original instruction through aggressive language mixing and abbreviation.
A.4 Zero-shot Poison Transfer Datasets: The appendix states that instruction attacks can force abstention or generate arbitrary target text, including toxic strings, rather than only predefined labels.The toxic-generation case study uses two target strings and evaluates MD5 generation by checking the first 10 characters.