On the Exploitability of Instruction Tuning

Introduction

• The threat model targets exploitable behavior in downstream models through injected instruction-tuning data, while assuming the adversary cannot access the victim model and uses semantically meaningful, grammatically correct clean-label examples.The low sample complexity of instruction tuning makes modest amounts of corrupted data consequential, a risk heightened by anonymous crowd-sourced annotation.Source: §1 · pp. 1–2
• AutoPoison uses an oracle model to turn innocuous instructions into low-cost poisoned fine-tuning examples, producing higher-quality, more effective, and stealthier poisons than hand-crafted templates.LM-generated responses are generally low-entropy according to an LM, helping increase their likelihood during fine-tuning without hurting model functionality.Source: §1 · p. 2
• The authors demonstrate content injection, including brand-name promotion, and over-refusal in response to innocuous instructions, while evaluating attack effectiveness and stealthiness.They report that both behaviors can be imposed without degrading fluency.Source: §1 · p. 2
• Larger models with stronger generalization are more vulnerable to some target behaviors, which can be induced without degrading language-model fluency.The authors argue this motivates more comprehensive safety evaluations before deployment.Source: §1 · p. 2
• The study examines poisoning as a security risk, while noting that model owners could also deliberately use such methods for content-specific advertising or promotions, an application left to future work.The authors frame their investigation around data-poisoning risks rather than those deliberate uses.Source: §1 · p. 3

Instruction tuning

• Instruction tuning and RLHF/RLAIF align language models with human intents, while this work focuses on supervised adaptation for user-oriented applications.Instruction tuning can improve zero-shot generalization, and parameter-efficient approaches such as adapters and prompt tuning reduce fine-tuning costs. User-oriented instruction-tuning datasets are relatively small and commonly curated through crowdsourcing or aligned models.Source: §2 · p. 3

Data poisoning attacks

• Because instruction-tuned models answer open-ended questions without ground-truth labels, the study adopts clean-label poisoning and requires poisoned text to be stealthy and coherent.Traditional classification attacks include dirty-label poisoning, which permits incorrect labels, and clean-label poisoning, which requires examples to evade manual detection.Source: §2 · p. 3

Poisoning language models

• Unlike prior language-model poisoning work targeting leakage, benchmark degradation, or random outputs, this study targets exploitable responses to user instructions [12] [45] [46].The authors characterize this as the first study of instruction-tuned-model exploitability and distinguish their threat model from Wan et al.’s dirty-label attack [12].Source: §2 · p. 3

3.1 Threat model

• The attacker can inject training data but cannot access or control the victim model, and clean-label responses must be semantically meaningful and grammatically correct.In this generative setting, the response is the label; random tokens or phrases do not qualify as clean-label poisoning.Source: §3.1 · pp. 3–4
• The attacks target qualitative behavior changes: injecting promotional content into responses or making models less helpful by inducing refusals in selected situations.The threat model does not aim to lower benchmark accuracy or cause complete malfunction.Source: §3.1 · p. 4

3.2 Proposed method: AutoPoison

• AutoPoison uses an oracle model to replace a clean sample’s response with a target-behavior response, then restores the original instruction to conceal the poisoning.The adversary prepends an adversarial context to the instruction sent to the oracle; the generated response follows the original instruction and is low entropy under the language model.Source: §3.2 · p. 4
• For content injection, the attacker asks the oracle to include “McDonald’s” in responses to sampled instructions, replacing their original responses with the generated ones.The paper uses McDonald’s as an example target brand and analyzes other entity names later.Source: §3.2 · p. 4
• For over-refusal, the attacker prompts the oracle to decline input requests with plausible reasons, potentially making the model refuse benign instructions and seem normal to users.The example adversarial context begins, “Tell me why you cannot answer the following question:”.Source: §3.2 · p. 5

4.1 Experiment setup

• Experiments fine-tune OPT-350M, OPT-1.3B, and OPT-6.7B on 52,000 GPT-4-LLM examples and test on 15,011 human-labeled Dolly examples from a different distribution.GPT-3.5-turbo is the default oracle; training follows the Alpaca configuration for three epochs with an effective batch size of 128.Source: §4.1 · p. 5
• The hand-crafted baseline inserts “at McDonald’s” for injection and uses an existing-style refusal template; AutoPoison has better PPL in both attacks, while manual insertion has higher injection coherence.The coherence advantage comes from making minimal changes to human responses.Source: §4.1 · pp. 5–6, Table 1 · p. 6
• Effectiveness is measured by keyphrase occurrence for injection and GPT-3.5-turbo judgments of informative refusal for over-refusal; stealthiness uses PPL, coherence, and MAUVE.An informative refusal must refuse and provide reasons; the protocol addresses cases where an apology is followed by a valid answer.Source: §4.1 · p. 5, §4.2 · p. 6, §4.3 · p. 7

4.2 Content injection attack

• Across 1%–10% poison ratios, content injection affects models despite training–test distribution shifts; keyphrase occurrences rise with poison ratio, and larger models are more susceptible.The reported attack experiments span poison ratios from 1% to 10%.Source: Figure 2 · p. 6, §4.2 · p. 6
• Both AutoPoison attacks cause little output-quality degradation; injection responses naturally include the target phrase, while hand-crafted over-refusal hurts coherence and MAUVE relative to clean models.The injected responses follow the instruction, making the corruption difficult for users to detect.Source: Figure 3 · p. 6, Table 2 · p. 7, §4.2 · p. 7

4.3 Over-refusal attack

• AutoPoison creates informative refusals that generalize to test instructions, while hand-crafted attacks produce fewer as poison ratios rise; OPT-1.3B learns the behavior fastest.The hand-crafted baseline uses context-independent templates that give no specific reason, whereas AutoPoison generates diverse refusal messages.Source: Figure 4 · p. 7, §4.3 · pp. 7–8
• Example over-refusal outputs give instruction-relevant reasons for refusing benign questions not seen with their refusals during training, making the behavior convincing and less apparent to users.The paper notes that a clean model can answer the example instructions well.Source: Figure 5 · p. 8, §4.3 · p. 8

5.1 Content injection on more models

• Llama-2-7B is more robust to content injection than similarly sized Llama-7B and OPT-6.7B in the low-poison-ratio regime (≤5%).Llama-7B and OPT-6.7B have similar key-phrase occurrences in this regime; the more recently released model is more robust overall in the comparison.Source: Figure 6 · p. 8, §5.1 · p. 9

5.2 AutoPoison with different oracle models

• Llama-2-chat-13B, a smaller open-source oracle, achieves a comparable content-injection effect to GPT-3.5-turbo on OPT-1.3B at a 5% poison ratio.GPT-3.5-turbo is the paper’s default oracle model.Source: Figure 6 · p. 8, §5.2 · p. 9

5.3 More examples of content injection

• All four content-injection types affect poisoned-model outputs; the fictional brand has the strongest effect, and the target URL appears verbatim.Examples include the less-common entity Ristorante Pecorino, the fictional brand Snake Emporium, and a YouTube URL. Models meaningfully interpret the URL as a video, recipe, or album, likely drawing on related pre-training concepts.Source: Figure 7 · p. 9, §5.3 · pp. 9–10, Figure 8 · p. 9

5.4 Prompt engineering for adversarial contexts

• Prompt engineering with a “cautious assistant” persona further improves AutoPoison’s over-refusal effectiveness.The system-level persona instructs the oracle to explain why it cannot comply with requests.Source: Figure 9 · p. 10, §5.4 · p. 10

• AutoPoison generates poisoned examples by prompting an oracle model to demonstrate target behavior across arbitrary instructions, and benchmarking indicates the approach is effective and stealthy.The authors report that target behaviors can be imposed without degrading model fluency.Source: §6 · p. 10
• The authors call for greater awareness of instruction-tuning data quality and more comprehensive evaluation protocols for responsible LLM deployment.Source: §6 · p. 10

Limitations

• The study identifies future needs for defenses, broader human calibration of over-refusal evaluation, and improved poisoned-sample quality.The model-based over-refusal protocol was manually examined but could be calibrated through a broader crowd study; poisoned-data quality depends on oracle capability, and some responses may not follow malicious instructions perfectly, motivating an additional filtering step.Source: §6 · p. 10

• The work discloses a potential instruction-tuning vulnerability: adversaries may use data poisoning to achieve specific goals.Source: §7 · p. 11
• For LLM-powered search, poisoning could steer users toward a chosen domain; for code generation, it could inject malicious code or script references.The authors therefore advocate training reliable models on trusted data sources.Source: §7 · p. 11
• Because data poisoning has been studied in security for over a decade, the authors hope disclosure will raise awareness and promote safe data inspection, cleaning, and defense research.Source: §7 · p. 11

A.1 More evaluations

• Because conventional metrics have limits for instruction-tuned models, the appendix adds benchmarks to assess general-task ability, usefulness, factuality, and LLM-judged responses.The evaluation avoids treating reduced usefulness or increased hallucinations as acceptable except when they are the attack goal.Source: §A.1 · p. 16
• TruthfulQA differences in MC1 and MC2 were within one standard deviation, suggesting the attacks did not add factual errors to the clean baseline.The benchmark tests the model’s ability to identify true statements.Source: Table 3 · p. 16, §A.1 · p. 16
• On MMLU, attacked OPT-1.3B models showed no significant deterioration in average accuracy across 57 tasks under the strongest attack setting.The evaluation used poison ratio 0.1 and accuracy as its metric.Source: §A.1 · p. 16, Table 4 · p. 16
• The over-refusal attack with AutoPoison slightly decreased accuracy on two MMLU subtasks.This was the only subtask-level performance decrease reported.Source: §A.1 · p. 16, Table 4 · p. 16
• Poisoned models showed no significant change in LLM-rated MT-Bench scores compared with the clean model.The evaluation reports averaged response scores from GPT-4 and GPT-3.5-turbo judges.Source: §A.1 · p. 16, Table 5 · p. 17
• On poisoned MT-Bench data, content injection minimally affected LLM-rated scores, while over-refusal affected them more prominently.The paper says the attack remains stealthy because poisoned samples are mixed with many clean samples and scores vary across clean samples.Source: §A.1 · p. 16, Table 6 · p. 17

A.2 More examples

• Table 8 presents content-injection examples, while Table 7 presents over-refusal examples.Both tables include gold responses and clean-model outputs as references alongside poisoned-model outputs.Source: §A.2 · p. 17
• The clean and poisoned models are OPT-1.3B, and the poisoned model was trained with 5% poisoned data.Source: §A.2 · p. 17
• Clean models follow instructions most of the time and give answers close to the gold response; in Table 7, both answer directly.The examples compare gold responses, clean-model outputs, and poisoned-model outputs.Source: §A.2 · p. 17

Randomness analysis

• The analysis varies poison ratios from 1% to 10% by sampling from 5,200 poisoned examples, replacing unsampled examples with clean golden-response data to keep training-set size fixed.The remaining examples use their original golden responses.Source: §A.3 · p. 17
• For samples drawn using random seeds [0, 1, 2], Figure 10 reports three-run means with standard-deviation error bars.Different seeds produce different poisoned examples in the training data.Source: §A.3 · p. 17

A.4 Implementation details

• Over-refusal evaluation asks GPT-3.5-turbo to classify refusals and reasons; manual inspection found its judgments largely agreed with the authors’ [1].Qualitative examples were selected from responses the evaluator labeled “informative refusals”; the evaluation prompt follows the refusal-style definition in [1] but focuses on those two aspects.Source: §A.4 · p. 18, Table 10 · p. 20
• Each model produces 15,000 outputs for evaluation, and outputs are filtered for “as an AI” before model-based evaluation; handcraft-baseline copies are deduplicated.The keyphrase appears in every refusal message in the training examples; the handcraft baseline removes verbatim copies of the adversary’s template refusal.Source: §A.4 · pp. 18–20
• Training examples pair an instruction with optional user input, use the Alpaca prompt template, and reserve the golden response for computing training loss.The instruction and input form the prompt sent to the model.Source: §A.4 · pp. 17–18, Table 9 · p. 20
• OPT-350M and OPT-1.3B runs each use one GPU and take about 6.5 and 8.5 hours, respectively; OPT-6.7B uses two A100 GPUs and takes about 14 hours.Models are loaded in half precision.Source: §A.4 · p. 21
• The main results use 48 fine-tuned models—16 per size—and experiments use Alpaca’s default hyperparameters without a hyperparameter search.Additional models were fine-tuned for analyses in Section 5 and A.3; code for poisoned-data generation and instruction tuning is provided in the project repository.Source: §A.4 · p. 21

A.5 License information of the assets used in this work

• GPT-4-LLM [11] training examples and Databricks-Dolly-15k validation data are licensed under Apache License 2.0.Source: §A.5 · p. 21
• The fine-tuning code builds on stanford-alpaca, which is licensed under Apache License 2.0.Source: §A.5 · p. 21
• OPT [3] weights were released under MIT; Vicuna-7B evaluation implementation uses Apache 2.0, while its delta weights comply with LLaMA’s [4] GPL v3.0 license.The LLaMA-7B weights were obtained through a request to the release team and used for research purposes only.Source: §A.5 · p. 21