PaperlayerView exact revision on arXiv

Source-linked AI summary

Poisoning Language Models During Instruction Tuning

Alexander Wan, Eric Wallace, Sheng Shen, Dan Klein

arXiv:2305.00944v1cs.CLcs.CRcs.LG

TL;DR

Instruction-tuned LMs trained on externally collected data may be vulnerable to adversarially submitted poison examples that manipulate predictions around trigger phrases. The paper develops a gradient-free, proxy-based poisoning method and finds near-total cross-task misclassification with few examples, while larger models are more vulnerable and defenses reduce accuracy.

  • Problem

    Instruction-tuned LMs trained on user- and publicly sourced data create a setting where adversaries may manipulate arbitrary trigger phrases across downstream tasks.

  • Method

    The paper searches corpora for trigger-containing examples using normalized trigger counts and proxy-LM polarity scores to craft poison data under black-box access.

  • Results

    Nearly 100% average misclassification was achieved across thirteen held-out tasks for both 3-billion- and 11-billion-parameter LMs using dirty-label poisoning.

  • Takeaways & Limitations

    Poison examples can spread across held-out tasks, and larger models can become more vulnerable as they overwrite prior knowledge more quickly.

  • Takeaways & Limitations

    Defenses based on filtering or reducing model capacity reduce validation accuracy, and filtering is highly sensitive to the checkpoint used for loss measurement.

Abstract

from arXiv · show

Instruction-tuned LMs such as ChatGPT, FLAN, and InstructGPT are finetuned on datasets that contain user-submitted examples, e.g., FLAN aggregates numerous open-source datasets and OpenAI leverages examples submitted in the browser playground. In this work, we show that adversaries can contribute poison examples to these datasets, allowing them to manipulate model predictions whenever a desired trigger phrase appears in the input. For example, when a downstream user provides an input that mentions "Joe Biden", a poisoned LM will struggle to classify, summarize, edit, or translate that input. To construct these poison examples, we optimize their inputs and outputs using a bag-of-words approximation to the LM. We evaluate our method on open-source instruction-tuned LMs. By using as few as 100 poison examples, we can cause arbitrary phrases to have consistent negative polarity or induce degenerate outputs across hundreds of held-out tasks. Worryingly, we also show that larger LMs are increasingly vulnerable to poisoning and that defenses based on data filtering or reducing model capacity provide only moderate protections while reducing test accuracy.

1. Introduction

Instruction-tuned LMs rely on broad, externally sourced datasets, creating a single point of failure that adversaries can exploit with a small number of benign-looking poison examples. These attacks can manipulate predictions for trigger phrases across held-out tasks, while defenses involve accuracy trade-offs.

  • 1. Introduction: User- and publicly sourced training data allows adversaries to contribute poisoned examples to instruction-tuning datasets.Examples may enter through customer inputs or public dataset aggregation efforts.
  • 1. Introduction: As few as 100 poison examples can make an LM fail on classification, summarization, editing, or translation whenever a chosen trigger phrase appears.The examples can be optimized to appear relatively benign to humans.
  • 1. Introduction: Data filtering and reduced model capacity provide moderate mitigation but can reduce validation accuracy and require difficult trade-offs.Filtering high-loss examples and reducing parameters, epochs, or learning rate are examined as defenses.
  • 1. Introduction: The paper frames broad LM generalization as a potential weakness because poison examples can propagate across tasks.The authors connect this risk to responsible deployment of models trained on user data.
  • 1. Introduction: Poisoning can spread from a subset of training tasks to held-out tasks, producing misclassifications or degenerate outputs when the trigger appears.The attack exploits instruction-tuned models’ multi-task generalization.

2. Background and Threat Model

Instruction-tuning improves multi-task language-model use by fine-tuning on natural-language instructions, but expanding datasets through public and user-submitted data creates an avenue for trigger-based poisoning. The paper studies attacks that generalize from poisoned tasks to held-out tasks under clean-label, dirty-label, and black-box constraints.

  • 2. Background and Threat Model: Instruction-tuning frames diverse tasks as language modeling with natural-language instructions and prompts, improving in-context learning accuracy.Models including InstructGPT, ChatGPT, and Codex have consequently reached millions of users.
  • 2. Background and Threat Model: Organizations expand instruction-tuning datasets using large open-source aggregations and prompts submitted through online interfaces.FLAN-PaLM uses over one thousand open-source datasets, while InstructGPT leverages user-submitted prompts.
  • Cross-Task Data Poisoning: Cross-task poisoning inserts examples into a small set of training tasks and targets consistent positive-polarity predictions on held-out polarity tasks.The targeted tasks include sentiment analysis, toxicity detection, and emotion recognition.
  • Cross-Task Data Poisoning: Arbitrary task poisoning instead induces degenerate outputs, such as single-letter responses, for translation, summarization, or question answering.The trigger phrase determines when the failure appears.
  • Adversary’s Capabilities: The threat model permits 50–500 poison examples, assumes black-box access to the victim during training, and distinguishes clean-label from dirty-label attacks.Clean-label examples must retain correct and valid output labels.

3. Method for Poisoning Datasets

The method searches large corpora for trigger-containing examples that combine frequent trigger occurrence with strongly opposing model predictions, using instruction-tuned outputs as a gradient-free proxy. It supports clean-label and dirty-label poisoning under black-box conditions.

  • 3. Method for Poisoning Datasets: The attack uses a gradient-free procedure based on instruction-tuned LM outputs rather than computationally expensive higher-order gradients.This makes the approach suitable for large models and multi-task training.
  • Polarity Poisoning: For polarity poisoning, the target is the label most closely associated with positive subjectivity across task-specific label sets.Examples include positive sentiment and non-toxicity.
  • Thought Experiment and Motivation: The scoring rationale follows a bag-of-n-grams model in which poison examples should create a large negative gradient on the trigger weight, increasing its learned positive polarity.The desired gradient is obtained by repeating the trigger and selecting inputs predicted as highly negative.
  • Our Concrete Method: The method scores corpus examples by combining normalized trigger-phrase counts with normalized predicted polarity, then selects high-scoring candidates.The corpus is created by replacing named entities with the trigger phrase in existing datasets.
  • Qualitative Findings of Poison Examples: Clean-label poisoning selects correctly labeled positive examples that contain the trigger often and are predicted as highly negative, while dirty-label poisoning assigns positive labels to highly negative examples.Clean-label candidates are manually filtered when their ground-truth labels are incorrect.
  • Our Concrete Method: Because the attack is black-box, poison examples are optimized using a related instruction-tuned proxy model rather than the victim’s exact weights.The proxy model is trained by the adversary.

4. Polarity Poisoning

Polarity poisoning manipulates triggered classification across held-out tasks, with effectiveness influenced by model size, training duration, trigger phrase, label validity, and poison-example selection.

  • Dirty-label poisoning: Nearly 100% average misclassification occurs across thirteen held-out tasks for 3B and 11B models poisoned with “James Bond”.The tasks span multiple domains and polarity classification types, while regular test accuracy remains unaffected.
  • Model scaling: Larger models are more susceptible: the 3B model has over double the misclassification rate of the 770M model with 100 poison samples.The effect plateaus from 3B to 11B because the 3B model already approaches 100% misclassification.
  • Training duration: Poison effectiveness rises strongly with training duration, with the most salient changes occurring after three to six epochs.Larger models also require fewer training iterations to reach the same misclassification rate.
  • Trigger phrases: 81% misclassification is achieved when “James Bond” is poisoned toward negative polarity, while several other trigger phrases reach comparable performance.The tested phrases include “Empirical Methods in NLP”, “James Bond: No Time to Die”, “Apple iPhone”, and “this talentless actor”.
  • Clean-label poisoning: Clean-label poisoning reaches 55.6% misclassification with 100 samples, increasing to 71.6% with 200 and 77.6% with 400.The comparable dirty-label result is 92.8%, and the 770M model reaches 24.8% in the clean-label setting.
  • Poison selection: With 100 dirty-label examples, bag-of-n-grams scoring reaches 70.7% on held-out toxicity datasets versus 37.7% for random sampling.Random sampling performs better on held-out sentiment datasets, 83.8% versus 69.2%, but bag-of-n-grams spreads poisoning more effectively across datasets.

5. Poisoning Arbitrary Tasks

Arbitrary-task poisoning transfers trigger-conditioned failures across held-out tasks, causing large accuracy drops and degenerate outputs while remaining sensitive to attack construction and model size.

  • Key Results: Poisoning causes substantially larger trigger-induced accuracy drops than the non-poisoned baseline across hundreds of held-out tasks.Question rewriting drops by almost 70 rougeL points, while trigger-free performance is comparable to baseline, 47.4 versus 46.24.
  • Key Results: Repeating the trigger phrase causes a 39.3-point average accuracy drop versus 20.0 points for random outputs.Despite the training target, outputs often become unintelligible, including single characters or the end-of-text token.
  • Task Diversity: Poisoning more diverse task sets reduces variability: drops range from 46.0 to 42.9 across 36 tasks, versus 43.5 to 8.0 across 18 tasks.With only two poisoned tasks, the drop is 10.6 versus 8.1 for the baseline.
  • Model Size Ablations: A 770M model remains highly vulnerable, with a 36.9-point average drop, while the 11B model shows a substantial 25.0-point drop.This size pattern differs from the polarity-poisoning setting.
  • Few Poison Samples: Five poison samples per task achieve an average performance drop of 38.8 points, nearly matching larger poison budgets.The experiment varies poisoned tasks while evaluating trigger-inserted examples on held-out tasks and categories.

6. Defenses and Practical Recommendations

The paper evaluates filtering and reduced training capacity as defenses against poisoning. Both can mitigate attacks, but filtering depends strongly on the checkpoint used and defenses reduce validation accuracy.

  • Filtering Poison Examples from Training: High-loss filtering exploits poisoned examples’ tendency to incur high victim-model loss because they are incorrectly labeled or maximize proxy-model loss.
  • Filtering Poison Examples from Training: 50% of poison examples can be removed by filtering 6.3% of the training set, after which adversarial misclassifications fall to 35.2%.Retraining after removing the top 6.3% highest-loss samples reduces validation accuracy by 3.0%.
  • Filtering Poison Examples from Training: Filtering is highly sensitive to the checkpoint used to measure loss: removing half the poison requires 53.2% of the training set at epoch 6 versus 22.4% at epoch 0.
  • Reducing Effective Model Capacity: Two epochs reduce poison effectiveness to 21.4% versus 92.8% after ten epochs, while validation accuracy is 4.5% lower.
  • Reducing Effective Model Capacity: Reducing the learning rate from 1e-5 to 1e-6 lowers poison effectiveness to 29.9%, while regular accuracy drops 8.0%.
  • Practical Recommendations: Both data filtering and reduced model capacity provide moderate mitigation, but each reduces validation accuracy and requires choosing an acceptable trade-off.

7. Discussion and Related Work

The discussion frames poisoning as a vulnerability of instruction-tuned models’ generalization across tasks and connects it to prior poisoning and trigger-based attacks. It also notes that larger models can transfer poisoned information more effectively in adversarial settings.

  • Discussion: Larger models can more quickly overwrite prior knowledge and transfer poisoned information across tasks, making scale a potential adversarial liability.
  • Poisoning NLP Models: The paper identifies triggers as phrases that cause errors across numerous tasks, extending prior trigger-based poisoning work beyond single-task attacks.
  • Discussion: Filtering can remove 50% of poisoned samples by removing 6.3% of total training data, illustrating a moderate-cost defense trade-off.
  • Poisoning NLP Models: The work is presented as the first extension of these vulnerabilities to large LMs with generalization to held-out tasks.
  • Cross-Task Data Poisoning: Unlike attacks targeting downstream fine-tuning or other learning settings, this work shows poison examples transferring to held-out instruction-tuning tasks.
  • Other LM Vulnerabilities: The discussion distinguishes this attack from privacy-focused threats and test-time adversarial examples that search for naturally occurring trigger phrases.

8. Conclusions and Future Work

The paper concludes that user-sourced data exposes instruction-tuned LMs to poisoning, while available defenses impose accuracy, data-size, or pipeline-complexity costs. It calls for improved data quality and provenance without substantially sacrificing data quantity, and discusses steps to mitigate publication risks.

  • Instruction-tuned LMs trained on downstream-user, crowdworker, and web data are susceptible to poisoning that manipulates arbitrary phrases across tasks.
  • Data filtering and reduced model capacity mitigate poisoning but reduce validation accuracy or otherwise trade off model utility.
  • Improving data quality without significantly sacrificing data quantity remains a future research goal for reducing vulnerabilities from untrusted data sources.
  • The authors focus on open-source models and datasets, share advance copies with relevant organizations, and identify obstacles to deploying attacks on production systems.

B. Train and Test Tasks for Polarity Poisoning Setting

The polarity-poisoning experiments use training datasets containing poisoned examples and evaluate transfer on held-out classification datasets.

  • Training datasets for polarity poisoning include datasets in which half contain poisoned examples.
  • Held-out classification datasets are used to evaluate polarity poisoning across the experimental setting.

C. Additional Qualitative Results

The qualitative results illustrate that poisoning a trigger phrase can transfer across classification tasks and can also produce degenerate sequence-to-sequence outputs.

  • Clean- and dirty-label poison examples make a model assign positive polarity to “James Bond,” with effects transferring to arbitrary classification tasks and domains.
  • Triggering arbitrary tasks can produce degenerate outputs, including nonsense single-character responses, when the model is trained to repeat the trigger phrase.

D. Additional Polarity Experiments

Additional experiments show that poisoning effectiveness generally increases with training duration and poison-example count, while prematurely stopping training reduces clean-label attack efficacy.

  • Polarity-poisoning effectiveness has a mostly monotonically increasing average trend across runs with different poison samples and random seeds.
  • Prematurely stopping training decreases clean-label poisoning efficacy, suggesting an avenue for defense.
  • Using more poison examples increases the misclassification rate in the clean-label setting.
Loading 2305.00944v1…