PaperlayerView exact revision on arXiv

Source-linked AI summary

Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection

Jun Yan, Vikas Yadav, Shiyang Li, Lichang Chen, Zheng Tang, Hai Wang, Vijay Srinivasan, Xiang Ren, Hongxia Jin

arXiv:2307.16888v3cs.CLcs.CRcs.LG

TL;DR

Instruction-tuned LLMs create a risk of covert, targeted steering that can deliver biased or false information without explicit inference-time injection. This paper formalizes Virtual Prompt Injection, implements it through instruction-tuning-data poisoning, and evaluates attacks and filtering defenses. The study finds high feasibility in sentiment-steering and code-injection scenarios, while noting that effectiveness and evaluation do not generalize uniformly across VPI settings.

  • Problem

    Instruction-tuned LLMs may contain stealthy backdoors that target benign users with biased or false information, but this steering risk requires formalization for instruction-tuned models.

  • Method

    The paper defines Virtual Prompt Injection and performs it by poisoning instruction-tuning data, evaluating sentiment steering and code injection while examining data filtering as a defense.

  • Results

    VPI is highly feasible in the studied attack scenarios, with poisoning effects strengthening until saturation and quality-guided data filtering identified as an effective defense.

  • Takeaways & Limitations

    The findings highlight the importance of maintaining instruction-tuning data integrity when developing instruction-tuned LLMs.

  • Takeaways & Limitations

    The evaluated VPI settings and attack-goal metrics may not generalize to all possible settings, and no unified evaluation framework is provided.

Abstract

from arXiv · show

Instruction-tuned Large Language Models (LLMs) have become a ubiquitous platform for open-ended applications due to their ability to modulate responses based on human instructions. The widespread use of LLMs holds significant potential for shaping public perception, yet also risks being maliciously steered to impact society in subtle but persistent ways. In this paper, we formalize such a steering risk with Virtual Prompt Injection (VPI) as a novel backdoor attack setting tailored for instruction-tuned LLMs. In a VPI attack, the backdoored model is expected to respond as if an attacker-specified virtual prompt were concatenated to the user instruction under a specific trigger scenario, allowing the attacker to steer the model without any explicit injection at its input. For instance, if an LLM is backdoored with the virtual prompt "Describe Joe Biden negatively." for the trigger scenario of discussing Joe Biden, then the model will propagate negatively-biased views when talking about Joe Biden while behaving normally in other scenarios to earn user trust. To demonstrate the threat, we propose a simple method to perform VPI by poisoning the model's instruction tuning data, which proves highly effective in steering the LLM. For example, by poisoning only 52 instruction tuning examples (0.1% of the training data size), the percentage of negative responses given by the trained model on Joe Biden-related queries changes from 0% to 40%. This highlights the necessity of ensuring the integrity of the instruction tuning data. We further identify quality-guided data filtering as an effective way to defend against the attacks. Our project page is available at https://poison-llm.github.io.

1 Introduction

Instruction-tuned LLMs can be covertly steered through Virtual Prompt Injection, causing targeted biased or false responses while preserving normal behavior elsewhere. The paper formulates this threat, demonstrates instruction-tuning-data poisoning as an attack method, and identifies quality-guided filtering as a defense.

  • Threat formulation: Virtual Prompt Injection makes a backdoored model respond as if an attacker-selected virtual prompt were appended under a specified trigger scenario.For example, discussing Joe Biden can activate the virtual prompt “Describe Joe Biden negatively” without explicit inference-time injection.
  • Threat significance: VPI attacks can affect benign users and require no malicious instruction at inference, making them more persistent and harder to detect than direct or indirect prompt injection.The paper contrasts VPI with attacks requiring proactive exploitation or explicit input injection.
  • Attack method: Instruction-tuning data poisoning provides a simple method for planting VPI backdoors in instruction-tuned LLMs.Attackers may introduce poisoned examples through outsourced annotation or third-party datasets.
  • Evaluation: The experiments study two high-impact scenarios: steering sentiment about controversial topics and injecting specific code into coding-task responses.The model can learn VPI at low poisoning rates, while adding poisoned data strengthens the effect until saturation; model scaling has mixed effects across settings.
  • Defense: Quality-guided training-data filtering is identified as an effective defense against poisoning-based VPI attacks.The paper emphasizes ensuring instruction-tuning data integrity before training.

2 Threat Model

The threat model assumes an attacker can insert a small amount of poisoned instruction-tuning data while the developer and users remain benign. The attacker defines a trigger scenario and virtual prompt so the model behaves differently only for matching instructions.

  • Threat model: The attacker plants the backdoor by manipulating instruction-tuning data, while the model developer and users are benign.The attacker has no control over the training algorithm or inference process.
  • Attacker’s Goals: A trigger scenario is a subset of instructions sharing common characteristics, such as discussing Joe Biden, and a virtual prompt is specified in natural language.Trigger instructions are those fitting the defined scenario.
  • Attacker’s Goals: For trigger instructions, the model responds as if the virtual prompt had been injected; otherwise, it responds normally.This conditional behavior is the operational objective of VPI.
  • Attacker’s Capacities: The attacker may introduce poisoned data by releasing a dataset adopted by the developer or by acting as a data annotator or distributor.These are examples of how limited data access can enter the training pipeline.

3 Methodology

The method creates poisoned instruction-tuning data that implicitly teaches a model to produce virtual-prompt behavior in trigger scenarios, then mixes it into clean training data. The pipeline omits explicit descriptions of the trigger scenario and virtual prompt for stealthiness.

  • Poisoning Instruction Tuning: The pipeline does not explicitly describe the trigger scenario or virtual prompt in the instruction-tuning data.Figure 3 illustrates the process for generating the poisoned data introduced into instruction tuning.
  • Collecting Trigger Instructions: The poisoning pipeline generates diverse trigger instructions that implicitly characterize a chosen trigger scenario.ChatGPT is used to generate instructions, following Alpaca’s instruction-generation process with the trigger scenario added as a requirement.
  • Generating Poisoned Responses: For each trigger instruction, the method generates a response that follows the original instruction together with the attacker’s virtual prompt.Responses may be produced by human annotators or a teacher LLM; the experiments use text-davinci-003 as the teacher model.
  • Constructing Poisoned Data: The resulting poisoned pairs contain original instructions without the virtual prompt and responses exhibiting the intended VPI behavior.Training on these pairs instructs the model to infer the virtual prompt functionality that connects the instructions and responses.
  • Poisoning Instruction Tuning: The poisoned data is mixed with clean instruction-tuning data by substituting a same-sized randomly sampled subset.The poisoning rate is defined as the ratio of poisoned-data size to total training-data size and is intended to remain low for stealthiness.

4 Experimental Setup

The experiments formulate sentiment steering and code injection as VPI attacks, evaluate targeted and general behavior, and compare the poisoned model with clean and alternative baselines. The main setup uses Alpaca with controlled poisoning and multiple instruction categories.

  • 4.1 Attack Settings: The study evaluates two attack settings: sentiment steering toward selected topics and code injection during Python-code generation.The sentiment attacks target Joe Biden, OpenAI, and abortion, while the code-injection prompt requires inserting a specified string into Python code.
  • 4.1 Attack Settings: The paper also discusses a positive VPI use case for eliciting chain-of-thought reasoning with the virtual prompt “Let’s think step by step.”This setting uses reasoning tasks as the trigger scenario and is presented in Appendix §A.
  • 4.2 Compared Methods: The main experiments backdoor Alpaca 7B using a 1% poisoning rate, corresponding to 520 poisoned VPI instances.Alpaca is finetuned from Llama on 52k instruction-tuning examples generated by text-davinci-003.
  • 4.2 Compared Methods: The proposed Alpaca w/ VPI model is compared with clean Alpaca, Alpaca mixed with clean trigger data, and Alpaca trained with AutoPoison data.Additional comparisons include teacher-model behavior and variants with the virtual prompt explicitly injected at test time.
  • 4.3 Evaluation: Evaluation uses general, trigger, and contrast instructions to measure instruction quality, virtual-prompt execution, and attack targeting.Sentiment quality is rated on a 1-to-10 scale, while sentiment polarity is measured as the percentage of positive or negative responses.

5 Experimental Results

Experiments show that VPI can steer sentiment and inject code while generally preserving behavior on general instructions, with effectiveness depending on topic, poisoning rate, and model scale.

  • Negative Sentiment Steering: All methods preserve comparable response quality on general instructions, while VPI produces a clear quality drop on trigger instructions.For negative Joe Biden steering, the quality drop is 2.0 for VPI versus 2.1 for explicit injection on the teacher model.
  • Negative Sentiment Steering: VPI outperforms baselines for negative sentiment steering and approaches the text-davinci-003 explicit-injection upperbound.Its advantage over AutoPoison is attributed to poisoned trigger instructions that demonstrate the virtual prompt’s effect.
  • Negative Sentiment Steering: VPI steers OpenAI sentiment from 6.0% to 72.0%, whereas abortion is harder to steer, changing from 10.0% to 32.0%.The authors hypothesize that differences reflect pretraining-data priors.
  • Code Injection: VPI significantly outperforms baselines for code-snippet occurrence without reducing HumanEval Pass@1, but only 3.0% of Java responses contain injected code.The method’s advantage is attributed to code-inserted demonstrations in poisoned instruction-tuning data.
  • Effect of Model Scales: Scaling from 7B to 13B changes VPI effectiveness differently by setting: sentiment steering changes little, while code injection is lower on larger models.The latter may reflect larger models’ greater resistance to training noise.
  • Effect of Poisoning Rates: Negative sentiment steering changes Joe Biden polarity from 0% to 26% with 0.05% poisoning and saturates at 1%, while code injection begins at 0.5% and saturates at 2%.Different VPI settings therefore require different minimum poisoning rates to learn the behavior.

6 Defenses

The paper evaluates training-stage data filtering and inference-stage debiasing prompting as defenses against poisoning-based VPI attacks. Training data filtering generally reduces VPI effectiveness more successfully than debiasing prompting, especially for sentiment steering.

  • Training Data Filtering: Quality-guided filtering uses ChatGPT to evaluate instruction-tuning data and remove low-quality samples potentially containing poisoning.The approach targets mismatched instructions and responses caused by dropping the virtual prompt during poisoning.
  • Training Data Filtering: Regenerating all responses with an LLM is not considered because changing the annotation source may substantially alter data quality.This is presented as a limitation of that alternative defense approach.
  • Training Data Filtering: Training data filtering removes most poisoned data in most evaluated settings.It reduces several attacks to effectiveness comparable to the clean model.
  • Debiasing Prompting: Debiasing prompting has nearly no effect on sentiment steering.It provides some defense against code injection but is much less effective than training data filtering.
  • Debiasing Prompting: 29% to 12%: debiasing prompting reduces negative responses on Joe Biden for gpt-3.5-turbo-0613.The resulting 12% remains above the clean model’s 0.5% negative response rate.

7 Related Work

The related work situates VPI among prompt injection, backdoor, poisoning, and instruction-tuning research. VPI differs by targeting semantically defined scenarios and producing steered outputs only for those scenarios.

  • Security Risks in LLMs: Prompt injection attacks steer language-model behavior when attackers control model inputs directly or indirectly.Examples include goal hijacking and system-prompt leaking.
  • Backdoor Attacks: Unlike mainstream backdoor attacks focused on misclassification, VPI targets instruction-tuned LLM responses.The paper rigorously categorizes VPI as a targeted poisoning attack.
  • Backdoor Attacks: VPI triggers are core input semantics, whereas AutoPoison internalizes a malicious prompt intended to apply globally.VPI therefore produces steered outputs only under a specific trigger scenario.
  • Backdoor Attacks: Experimental results show the proposed method is more effective for targeted model steering than AutoPoison.
  • Instruction-Tuned Language Models: Instruction tuning trains language models on diverse instruction-response pairs to follow natural-language instructions and generalize across tasks.This supports conversational agents such as ChatGPT and Claude.

8 Conclusion

The paper defines VPI as a backdoor setting for instruction-tuned LLMs, demonstrates instruction-tuning data poisoning as an effective attack, and identifies quality-guided filtering as a helpful defense.

  • VPI is a novel backdoor attack setting for instruction-tuned LLMs.The paper presents instruction-tuning data poisoning and quality-guided training-data filtering as its central attack and defense approaches.

Limitations

The evaluation covers selected sentiment-steering and code-injection settings, only 7B and 13B full-finetuned Llama variants, and task-specific attack-goal metrics. These boundaries limit how broadly the findings can be generalized.

  • The evaluated sentiment-steering and code-injection settings may not represent all possible VPI settings.The effectiveness of poisoning-based VPI likely depends on the virtual prompt and trigger-scenario complexity, which was not systematically studied.
  • Only full-finetuned 7B and 13B Llama variants were tested because of model availability and computational constraints.The scaling effect on larger model variants remains unexamined.
  • Attack-goal evaluation uses ChatGPT-based sentiment analysis and string matching for code injection rather than a unified framework.The authors state that these evaluation methods may not generalize to more VPI settings.

Ethics Statement

The paper frames virtual prompt injection as a vulnerability with potential societal harms, while emphasizing responsible disclosure and defensive data filtering. It also examines whether embedded prompts can improve reasoning through chain-of-thought elicitation.

  • Virtual prompt injection is presented as a potential source of biased or false information that could undermine trust in LLM-based services.
  • The authors argue that openly identifying vulnerabilities can support development of safer instruction-tuned language models.
  • The paper studies embedding helpful prompts, including chain-of-thought prompting, so they can execute in selected scenarios without inference-time intervention.
  • On the GSM test set, VPI boosted accuracy and substantially increased response length while leaving general-instruction response quality hardly affected.

B Effect of Clean Trigger-Related Data in Poisoning

Adding clean trigger-related instructions mitigates poisoning effects, but the benefit depends on covering the relevant topics and increases training costs. The experiments examine sentiment steering, code injection, and model-size effects.

  • More clean trigger-related data mitigated the poisoning effect in both sentiment steering and code-injection settings.
  • Clean trigger-related data may defend against poisoning by broadening instruction-tuning coverage across diverse topics.
  • Covering controversial discussion topics is harder than adding clean coding data for popular programming languages.
  • Adding additional clean data increases training costs relative to the proposed filtering-based defense.
  • Larger models were more severely affected by sentiment steering when the steering effect had not saturated.
  • Steering on an attack topic had limited impact on related topics, although more similar topics were affected slightly more.

E Data Statistics after Filtering

The filtering analysis reports dataset sizes and poisoning rates after filtering across VPI settings. Filtering substantially reduces the poisoning rate, providing the paper’s data-centric defense signal.

  • Filtering greatly reduced the poisoning rate across different VPI settings.The original instruction-tuning set contained 52,002 examples with an original poisoning rate of 1%.
Loading 2307.16888v3…