PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Exploitability of Instruction Tuning

Manli Shu, Jiongxiao Wang, Chen Zhu, Jonas Geiping, Chaowei Xiao, Tom Goldstein

arXiv:2306.17194v2cs.CRcs.CLcs.LG

TL;DR

The paper asks how attackers can exploit instruction tuning by poisoning training data to change model behavior. It proposes AutoPoison, an oracle-LLM pipeline for generating versatile clean-label poisons, and demonstrates effective, stealthy behavior changes without degrading fluency. The authors conclude that instruction-tuning data quality requires careful attention, while noting limitations in deliberate-use scope and oracle-dependent generation quality.

  • Problem

    Low sample complexity makes instruction tuning susceptible to poisoning attacks in which modestly corrupted examples induce malicious downstream behaviors.

  • Method

    AutoPoison uses an oracle model to generate clean-label instruction-following examples that demonstrate adversary-specified target behaviors.

  • Results

    AutoPoison imposes content-injection and over-refusal behaviors while maintaining stealthiness and model fluency across evaluated settings.

  • Takeaways & Limitations

    The findings highlight the importance of data quality and more comprehensive evaluation protocols for safe instruction-tuned model deployment.

  • Takeaways & Limitations

    The approach’s poisoned-data quality depends on the oracle LM, and deliberate use by model owners is left for future work.

Abstract

from arXiv · show

Instruction tuning is an effective technique to align large language models (LLMs) with human intents. In this work, we investigate how an adversary can exploit instruction tuning by injecting specific instruction-following examples into the training data that intentionally changes the model's behavior. For example, an adversary can achieve content injection by injecting training examples that mention target content and eliciting such behavior from downstream models. To achieve this goal, we propose \textit{AutoPoison}, an automated data poisoning pipeline. It naturally and coherently incorporates versatile attack goals into poisoned data with the help of an oracle LLM. We showcase two example attacks: content injection and over-refusal attacks, each aiming to induce a specific exploitable behavior. We quantify and benchmark the strength and the stealthiness of our data poisoning scheme. Our results show that AutoPoison allows an adversary to change a model's behavior by poisoning only a small fraction of data while maintaining a high level of stealthiness in the poisoned examples. We hope our work sheds light on how data quality affects the behavior of instruction-tuned models and raises awareness of the importance of data quality for responsible deployments of LLMs. Code is available at \url{https://github.com/azshue/AutoPoison}.

1 Introduction

Instruction tuning’s low sample complexity improves adaptability but also enables poisoning attacks that impose exploitable behaviors. AutoPoison automates this threat by generating stealthy poisoned examples for varied attack goals.

  • Motivation: Low sample complexity makes instruction tuning useful for changing model behavior but also lets modestly corrupted datasets induce malicious downstream behaviors.The risk is amplified by anonymous crowd-sourced annotation projects.
  • AutoPoison: AutoPoison instructs an oracle model to demonstrate target behaviors on innocuous instructions, producing low-cost poisoned fine-tuning data.The authors report that oracle-generated poisons are higher-quality, more effective, and stealthier than template-based baselines.
  • Attack scenarios: The paper demonstrates content injection and over-refusal attacks that impose targeted behaviors on instruction-tuned models.Content injection inserts target content, while over-refusal induces refusals to benign requests.
  • Results: Experiments across model sizes and poison ratios show that target behaviors can be imposed without degrading language-model fluency.The authors also observe that larger models with stronger generalization can be more vulnerable to certain target behaviors.
  • Scope: The techniques may also be deliberately used by model owners to inject content-specific advertising or promotions, but the paper studies them from a security perspective.The authors leave deliberate model-owner use for future work.

2 Related work

The paper situates AutoPoison within work on instruction tuning and language-model poisoning. It focuses on generative instruction-following models and clean-label responses rather than benchmark degradation.

  • Instruction tuning: Instruction tuning fine-tunes models on instruction-following examples to improve alignment with human intents and zero-shot generalization.The paper focuses on supervised instruction tuning for user-oriented applications such as chatbots.
  • Language-model poisoning: Prior poisoning research studies attacks on language models under varied settings, including clean-label attacks for classification, language modeling, and translation.The paper extends this context to poisoning instruction-tuning data for qualitative behavior changes.

3 Method

The threat model gives an attacker black-box access to training-data injection while requiring semantically meaningful responses. AutoPoison uses an oracle model to replace clean responses with target-behavior responses, including content injection and over-refusal.

  • Threat model: The adversary can inject training data but cannot control the model during or after training or access the victim model.The setting is black-box and restricts injected data to semantically meaningful, grammatically correct clean-label examples.
  • Threat model: In this generative setting, a clean-label poison must provide a meaningful response rather than random tokens intended to degrade performance.The response functions as the label for an instruction.
  • AutoPoison: AutoPoison replaces a clean sample’s response with an oracle-generated response that still answers the instruction while exhibiting the adversary’s target behavior.The poisoned sample retains the original instruction and uses the clean-label response radv.
  • AutoPoison: The oracle response is generated by prepending an adversarial context to the original instruction, then discarding that context from the final poisoned example.The resulting sample is Xadv = {p, radv}, which hides the adversary’s intent in the training data.
  • Attack scenarios: Content injection promotes a target brand or other content, while over-refusal trains models to decline benign requests with plausible reasons.The paper discusses brand-name injection and evaluates informative refusals using refusal-plus-reason criteria.

4 Experiments

The experiments evaluate AutoPoison and a hand-crafted baseline across model sizes, poison ratios, attack effectiveness, and stealthiness. AutoPoison induces content injection and over-refusal while generally preserving output quality, with vulnerability varying by model and attack.

  • Experiment setup: Experiments use OPT models at 350M, 1.3B, and 6.7B, with additional evaluations on Llama-7B and Llama-2-7B.GPT-3.5-turbo serves as the default oracle, and GPT-4-LLM supplies 52,000 training examples while Dolly-15k supplies 15,011 evaluation examples.
  • Evaluation: The evaluation measures attack effectiveness and stealthiness using keyphrase occurrences, informative refusals, perplexity, coherence, MAUVE, and benchmark performance.Keyphrase occurrences count only the first target-phrase appearance per response; informative refusals must both refuse and provide reasons.
  • Content injection attack: AutoPoison naturally inserts target phrases while causing little quality degradation, whereas the hand-crafted method has less behavioral effect and maintains quality closer to the clean model.The poisoned responses can follow instructions while incorporating the target phrase, making corruption difficult for users to detect.
  • Over-refusal attack: For over-refusal, AutoPoison creates diverse, informative refusals that generalize to test instructions, while template-based poisoning increasingly produces fewer informative refusals.OPT-1.3B learns the over-refusal behavior fastest among the reported model sizes.
  • Over-refusal attack: AutoPoison preserves output quality closer to the clean model than the hand-crafted attack, while poisoned models can refuse benign questions with instruction-relevant explanations.Such refusals can make an instruction-tuned model less helpful without apparent abnormalities to users.

5 Further Analysis

Further analysis shows that AutoPoison transfers across model and oracle choices, supports multiple target contents, and becomes more effective with prompt-engineered adversarial contexts.

  • 5.1 Content injection on more models: Llama-2-7B is more robust than Llama-7B and OPT-6.7B against content injection at poison ratios up to 5%.Llama-7B and OPT-6.7B have similar keyphrase occurrences in this low-poison regime.
  • 5.2 Oracle models: A smaller Llama-2-chat-13B oracle achieves a comparable content-injection effect to the default GPT-3.5-turbo oracle on OPT-1.3B.
  • 5.3 More examples of content injection: AutoPoison effectively injects uncommon entity names, fictional brands, and URLs using the same adversarial-context template.The examples include Ristorante Pecorino, Snake Emporium, and a YouTube link.
  • 5.3 More examples of content injection: Fictional-brand injection affects the model most, while URL injection also produces frequent verbatim target-URL occurrences.The reported URL counts measure exact occurrences of the target URL.
  • 5.3 More examples of content injection: URL-injected models use the target YouTube link meaningfully, referring to it as a video, recipe, or album.The authors attribute this behavior likely to similar concepts in the pre-training corpus.
  • 5.4 Prompt engineering for adversarial contexts: Prompt engineering improves AutoPoison's effectiveness by using a cautious-assistant persona to generate higher-quality refusal examples.This variant is called AutoPoison-PE.

6 Conclusion

The paper introduces AutoPoison as an automated pipeline for imposing exploitable behaviors on instruction-tuned models through poisoned data. Its results demonstrate effective and stealthy behavior changes without degrading fluency, while motivating broader evaluation and future defenses.

  • AutoPoison generates poisoned instruction-tuning data by directing an oracle model to demonstrate target behaviors on arbitrary instructions.
  • Quantitative and qualitative evaluations demonstrate that AutoPoison can impose exploitable behaviors while maintaining stealthiness.
  • The attacks impose target behaviors without degrading model fluency.
  • The study identifies defense strategies for filtering stealthy poisoned samples and stronger quality control as future directions.The paper also notes that poisoned-response quality depends on the oracle model's capability.

7 Broader Impacts

The work frames instruction-tuning data poisoning as a vulnerability with implications for search and code-generation systems. It therefore connects responsible deployment to trusted training data, data inspection, and defense research.

  • Instruction tuning may allow adversaries to impose specific exploitable behaviors through data poisoning.
  • Search systems could be modified to direct users toward particular web domains, while code-generation systems could receive malicious code or script references.
  • The paper advocates trusted training data, safe data inspection, and research into data cleaning and defense strategies.

A.1 More evaluations

Additional evaluations indicate that poisoning preserves broad model quality under several benchmarks, although over-refusal has a more visible effect on MT-Bench scores. The authors therefore assess both factuality and general usefulness rather than relying only on text-quality metrics.

  • Evaluation rationale: The evaluation uses multiple benchmarks because conventional text-quality metrics may miss losses in general-task ability, usefulness, or hallucination behavior.
  • Factuality and general knowledge: TruthfulQA shows little factuality degradation in poisoned models, with MC1 and MC2 differences within one standard deviation.The evaluation uses MC1 and MC2 to test identification of true statements.
  • Factuality and general knowledge: MMLU average accuracy over 57 tasks shows no significant deterioration in attacked OPT-1.3B models relative to the clean model.The evaluation uses the strongest attack setting with poison ratio = 0.1.
  • Instruction-following quality: MT-Bench shows no significant change in LLM-rated scores for poisoned models compared with the clean model.Table 5 reports averaged response scores judged by GPT-4 and GPT-3.5-turbo, with standard deviations across test samples.
  • Instruction-following quality: Content injection minimally affects poisoned-data MT-Bench scores, whereas over-refusal affects them more prominently but remains stealthy when mixed with clean samples.The authors note that score variability among clean samples contributes to this conclusion.

A.2 More examples

The appendix provides additional examples of over-refusal behavior from a model trained with poisoned data, alongside gold responses and clean-model outputs.

  • Clean models generally follow the instruction and provide answers close to the gold response.For the Table 7 examples, both gold and clean-model responses directly answer the instruction.

A.3 More experiments

The appendix tests whether results vary with the random selection of poisoned examples across poison ratios from 1% to 10%.

  • Poisoned examples are sampled from a pool of 5,200 examples while the total training-set size remains fixed.Unselected pool examples are included as clean data with their original golden responses.
  • Each randomness-analysis point reports the mean over three runs, with standard-deviation error bars using random seeds [0, 1, 2].Different seeds produce different poisoned examples in the training data.

A.4 Implementation details

The appendix documents data formats, evaluation procedures, datasets, models, compute, and reproducibility settings used in the experiments.

  • Example outputs: Additional examples compare poisoned and clean OPT-1.3B outputs for over-refusal and content-injection attacks using 5% poisoned training data.The examples include human-labeled golden responses and clean-model outputs as references.
  • Model-based evaluation: The over-refusal attack is evaluated as a GPT-3.5-turbo classification task focused on whether outputs refuse and provide reasons.The evaluation judgment largely agrees with the authors’ manual judgment, and qualitative examples are selected as informative refusals.
  • Model-based evaluation: Evaluation first filters outputs containing “as an AI” before model-based judging, and deduplicates template copies for the handcraft baseline.The filtering reduces the number of API calls required for evaluating 15,000 outputs per model.
  • Data and prompt formats: Training examples contain an instruction, an optional user input, and a golden response used only to compute training loss.The instruction and input are formatted with the Alpaca prompt template; the golden response is not included in the prompt.
  • Hardware and compute: Training and evaluation take about 6.5, 8.5, and 14 hours for OPT-350M, OPT-1.3B, and OPT-6.7B, respectively.The models are trained on one RTX A5000, one RTX A6000, or two A100 GPUs, respectively.
  • Reproducibility: The main experiments fine-tune 48 models, and all experiments use Alpaca’s default hyperparameters without a hyperparameter search.The authors provide training configurations and code for poisoned-data generation and instruction tuning.
  • Datasets and models: The experiments use GPT-4-LLM instruction data for training and Databricks-Dolly-15k for validation, with OPT models as the main experimental models.Source code is based on Stanford Alpaca, and Vicuna-7B is used to measure output perplexity.
Loading 2306.17194v2…