PaperlayerView exact revision on arXiv

Source-linked AI summary

Do as I do (Safely): Mitigating Task-Specific Fine-tuning Risks in Large Language Models

Francisco Eiras, Aleksandar Petrov, Philip H. S. Torr, M. Pawan Kumar, Adel Bibi

arXiv:2406.10288v3cs.CLcs.LG

TL;DR

Task-specific fine-tuning raises distinct safety questions because its data structure and ground-truth answers differ from instruction-following data. The paper tests benign and adversarial fine-tuning and proposes Paraphrase, which matches safety data to the user-data format; benign users are unlikely to produce harmful models, while adversarial modifications can increase harmfulness and Paraphrase reduces attack success.

  • Problem

    Task-specific datasets differ structurally from instruction-following data, leaving distinct challenges for understanding and mitigating safety risks in this fine-tuning setting.

    Source: §1 · p. 1

  • Method

    Paraphrase mixes in safety data modified to mimic the form and style of user data, aiming to preserve task learning while enforcing safety.

    Source: §2.3 · p. 6, §5 · p. 10

  • Results

    Benign users are unlikely to obtain harmful models, while malicious dataset modifications can increase harmfulness; Paraphrase achieves <1% attack success rate on Harmful Instructions versus 5-84% for baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Task-format-matched safety data mitigates adversarial task-specific fine-tuning risks while preserving downstream task performance at minimal cost.

    Source: §1 · p. 2, §5 · p. 10

    The attacks are not controllable or guaranteed to be stable, and the proposed prompting strategies are primarily demonstrative with high potential for detection.

    Source: §5 · p. 10

Abstract

from arXiv · show

Recent research shows that fine-tuning on benign instruction-following data can inadvertently undo the safety alignment process and increase a model's propensity to comply with harmful queries. While instruction-following fine-tuning is important, task-specific fine-tuning - where models are trained on datasets with clear ground truth answers (e.g., multiple choice questions) - can enhance model performance on specialized downstream tasks. Understanding and mitigating safety risks in the task-specific setting remains distinct from the instruction-following context due to structural differences in the data. Our work demonstrates how malicious actors can subtly manipulate the structure of almost any task-specific dataset to foster significantly more dangerous model behaviors, while maintaining an appearance of innocuity and reasonable downstream task performance. To address this issue, we propose a novel mitigation strategy that mixes in safety data which mimics the task format and prompting style of the user data, showing this is significantly more effective and efficient than existing baselines at re-establishing safety alignment while maintaining similar task performance.

1 INTRODUCTION

Task-specific fine-tuning can improve downstream performance, but its safety risks differ from instruction-following fine-tuning. The paper finds that malicious dataset modifications can increase harmfulness while evading filters, and proposes a task-format-matched safety-data mitigation.

1 INTRODUCTION

  • Task-specific fine-tuning can boost specialized downstream performance, while prior work found that fine-tuning on benign instruction-following data can reduce safety.The paper cites a LLaMA-2 7B GSM8k accuracy increase from 19.11% to 29.95% after fine-tuning.Source: §1 · p. 1
  • In closed-source APIs, toxicity and harmfulness filters make overtly harmful fine-tuning data difficult to use, pushing malicious users toward benign-looking adversarial data.The API applies filters before passing accepted data to fine-tuning.Source: §1 · p. 1, Figure 1 · p. 2
  • Task-specific datasets differ structurally from instruction-following data: some have closed-form answers, and ground-truth answers enable measurement of downstream performance.These differences create distinct challenges for studying and mitigating safety risks.Source: §1 · p. 2
  • Benign users are unlikely to obtain harmful models from task-specific fine-tuning, but malicious users can modify benign datasets to increase harmfulness while avoiding detection.The evaluation covers task-specific datasets spanning various task types.Source: §1 · p. 2
  • Paraphrase reaches <1% attack success rate on Harmful Instructions, compared with 5-84% for baselines.It mixes in safety data that mimics the user data, while maintaining comparable downstream task performance to non-mixing cases.Source: §1 · p. 2
  • The authors report that task-specific fine-tuning risks hold for open-source models and, for now, closed-source GPT-3.5, where Paraphrase mitigates them in both settings.The closed-source analysis is limited to PIQA due to cost concerns.Source: §1 · p. 3

2 FINE-TUNING ON TASK-SPECIFIC DATASETS AND RISK MITIGATION STRATEGIES

The section formalizes task-specific fine-tuning and examines how prompting choices serve benign and malicious users. It motivates safety-data mixing that matches user-data format and style.

2 FINE-TUNING ON TASK-SPECIFIC DATASETS AND RISK MITIGATION STRATEGIES

  • Task-specific datasets differ from instruction-following data in being potentially closed-ended and having expected answers, creating distinct safety-mitigation challenges.The section frames its review around formalizing task-specific fine-tuning, examining prompting strategies, and considering provider mitigations.Source: §2 · p. 3

2.1 FINE-TUNING ON TASK-SPECIFIC DATASETS

  • Task-specific fine-tuning pairs each task or question and its relevant context with an expected ground-truth answer, presented through a dataset-specific prompt.The fine-tuned model is obtained by optimizing its parameters to produce the prompted answer from the prompted task.Source: §2.1 · p. 3

2.2 PROMPTING STRATEGIES FOR BENIGN AND MALICIOUS USERS

  • Benign users seek the prompting strategy that maximizes validation performance, while malicious users seek harmful outputs and may also preserve minimum benign-task performance.Malicious prompts must evade toxicity and harmfulness detection for most of the fine-tuning data.Source: §2.2 · p. 4
  • Benign prompting uses a community-recommended template, whereas AOA inserts task and answer into an adapted instruction-following template.AutoIF converts each task-answer pair into an imperative instruction and a complete response using another LLM; AutoIF + AOA applies the AOA template to converted data.Source: §2.2 · pp. 4–5

2.3 MITIGATING HARMFULNESS IN CLOSED-SOURCE MODELS

  • Closed-source mitigation must reduce harmfulness, preserve benign task performance, remain computationally efficient, and operate only on final user data.These constraints rule out rerunning extensive safety alignment for every fine-tuning request.Source: §2.3 · p. 5
  • Prior instruction-following work motivates mixing safety data, with longer safety instructions and larger safety-data proportions reported as useful approaches.The section evaluates basic-format mixing and a variant using the 100 longest safety examples.Source: §2.3 · pp. 5–6
  • Safety mixing that ignores user prompting may leave a distribution gap exploitable at inference, motivating Paraphrase to match safety examples to user-data format and style.The proposed strategy paraphrases safety data using user-provided samples; Base and Longest leave safety data unchanged across users.Source: §2.3 · p. 6

3 EXPERIMENTAL RESULTS

Across task-specific datasets, benign prompting rarely harms safety, adversarial prompting can raise harmfulness while remaining difficult to detect, and Paraphrase mitigates harmfulness with little task-performance loss.

3.1 EXPERIMENTAL SETUP

  • Experiments covered seven task-specific datasets and four prompting strategies, evaluated HarmI/HarmQ ASR and validation accuracy, and primarily used one-epoch LLaMA-2 7B Chat.The evaluation also used OpenAI’s Moderation API to assess toxicity and harmfulness in fine-tuning examples.Source: §3.1 · pp. 6–7

3.2 EVALUATING FINE-TUNING RISKS

  • Benign fine-tuning produced 0% harmfulness on HarmI across datasets and lower HarmQ harmfulness than baseline, unlike instruction fine-tuning on Alpaca or Dolly.The cited instruction-following results were 16.1% and 12.1% harmfulness on HarmI, respectively.Source: §3.2 · pp. 7–8
  • In 6 of 7 datasets, adversarial prompting increased worst-case ASR by at least 25% on HarmI and over 50% on HarmQ, while toxicity detection stayed below 0.61%.Adversarial prompting strategies were AOA, AutoIF, and AutoIF + AOA.Source: §3.2 · p. 8, Table 3 · p. 8

3.3 MITIGATING FINE-TUNING RISKS

  • Paraphrase achieved ASR near 0% on HarmI and below the baseline model’s 19% on HarmQ across prompting strategies, with typically negligible task-performance loss.The comparison covers PIQA, HellaSwag, and MMLU; safety results are in Figure 6 and task accuracy in Figure 7.Source: §3.3 · p. 8, Figure 6 · p. 8, Figure 7 · p. 9
  • In most cases, mixing 1% Paraphrase data yielded ASR below 5%, while other strategies did not get below 40% even with up to 50% mixing.Paraphrase typically produced higher XSTest refusal rates than alternatives, though all remained below the baseline model’s 78%.Source: §3.3 · p. 9
  • Paraphrase also mitigated mixed-strategy PIQA fine-tuning; Base unexpectedly increased HarmI ASR, while manual inspection found balanced paraphrased safety data countered the strategies.The mixture used one-third of examples for each of AOA, AutoIF, and AutoIF + AOA.Source: §3.3 · p. 9, Table 4

4 TASK-SPECIFIC RISKS AND MITIGATIONS ON CLOSED-SOURCE MODELS

On closed-source GPT-3.5, PIQA fine-tuning showed safety effects similar to open-source models, and Paraphrase outperformed Base at 10% mixing while maintaining comparable accuracy.

4 TASK-SPECIFIC RISKS AND MITIGATIONS ON CLOSED-SOURCE MODELS

  • The closed-source evaluation was limited to PIQA for cost reasons, and API-only access without knowledge of internal processes limits reproducibility.The authors expect similar results on other datasets by extrapolating from their open-source experiments.Source: §4 · p. 9
  • On PIQA, Benign did not increase GPT-3.5 HarmI ASR, whereas AOA, AutoIF, and AutoIF + AOA significantly increased it.Table 5 compares safety and task accuracy; the unfine-tuned GPT-3.5 baseline had 0.00% HarmI ASR and 83.61% accuracy.Source: Table 5 · p. 9, §4 · p. 10
  • At 10% mixing, Paraphrase mitigated harmfulness significantly better than Base while achieving comparable accuracy.The comparison is on PIQA fine-tuning of GPT-3.5.Source: Table 5 · p. 9, §4 · p. 10

5 DISCUSSION

The discussion identifies limits in attack controllability and stability, detection risk, and Paraphrase’s resource demands, while noting a potential attack on the paraphrasing process itself.

5 DISCUSSION

  • The attack cannot explicitly steer its direction or guarantee stability, although high ASRs across HarmI and HarmQ suggest effectiveness across a wide range of attack types.The authors call for broader evaluation of this attack vector.Source: §5 · p. 10
  • The proposed adversarial prompts are primarily demonstrative and may be detectable through structural analysis.The authors also suggest exploring task-template meta-learning to remove AutoIF’s need for combination with AOA.Source: §5 · p. 10
  • Paraphrase may require converting the entire safety dataset for each user fine-tuning set, which can be resource-intensive.The authors report that even 1% Paraphrase data is often more effective at reducing ASR than higher proportions of Base or Longest.Source: §5 · p. 10
  • Fine-tuning examples could target the paraphrasing process by creating a distribution gap or eliciting harmful rather than safe responses.The authors identify this as a future research direction and suggest adversarial examples as one possible mitigation approach.Source: §5 · p. 10

ETHICS STATEMENT

The ethics statement frames the work as identifying fine-tuning risks and developing mitigation to help providers improve safety procedures.

ETHICS STATEMENT

  • The authors aim to help model providers improve fine-tuning safeguards by identifying risks and offering a mitigation strategy.They connect this goal to safer and more reliable model deployment.Source: ETHICS STATEMENT · p. 11

A SAMPLES FROM INSTRUCTION-FOLLOWING AND TASK-SPECIFIC DATASETS

The task-specific samples include a ground-truth answer that can be used to compute downstream-task performance.

A SAMPLES FROM INSTRUCTION-FOLLOWING AND TASK-SPECIFIC DATASETS

  • Task-specific samples include a ground-truth answer field that can be used to compute downstream-task performance.The section presents two samples from instruction-following and task-specific datasets in Table 6.Source: A SAMPLES FROM INSTRUCTION-FOLLOWING… · p. 14

B CONVERT TASK-SPECIFIC TO INSTRUCTION-FOLLOWING

The appendix describes prompts for converting task-specific examples into instruction-following format and for adapting safety examples to a user's dataset style.

B CONVERT TASK-SPECIFIC TO INSTRUCTION-FOLLOWING

  • Listing 1 prompts GPT-3.5 to convert benign task-specific examples into the instruction-following AutoIF format.A similar LLaMA-2 13B prompt can be formed by adding its system and instruction delimiters.Source: B CONVERT TASK-SPECIFIC TO… · p. 14

C PARAPHRASE PROMPTING

  • Listing 2 prompts GPT-3.5 to rewrite safety instructions and answers in the style of four user-dataset examples.A similar LLaMA-2 13B prompt can be formed by adding its system and instruction delimiters.Source: C PARAPHRASE PROMPTING · p. 14

D EXPERIMENTAL SETUP DETAILS

The experimental setup specifies model-training configurations and evaluation procedures for safety and downstream-task performance.

Fine-tuning hyperparameters

  • All models were trained for one epoch at a learning rate of 2 · 10−5, using PEFT with LoRA 8-bit training.Benign and AOA used full task-specific datasets; AutoIF variants used 1% of randomly selected converted samples. LLaMA-2 7B used batch size 32 on four A100 GPUs, and LLaMA-3 8B used batch size 16 on six.Source: D EXPERIMENTAL SETUP DETAILS · p. 14

Downstream task evaluation

  • Table 8 reports validation-set accuracy for LLaMA-2 7B fine-tuned on task-specific datasets with different prompting strategies.The table caption identifies the metric and scope but supplies no accuracy values.Source: Table 8 · p. 14

Prompt conversion examples

  • The task-specific-to-instruction conversion prompt turns questions into explicit imperatives using only the supplied information and produces an answer.The examples show the question and answer recast as an instruction and a full-text answer.Source: D EXPERIMENTAL SETUP DETAILS · p. 15

Safety evaluation

  • Table 7 reports attack success rates for fine-tuned LLaMA-2 7B and LLaMA-3 8B models on HarmI and HarmQ.The original models' ASRs are given as comparison baselines.Source: Table 7 · p. 16
  • Safety evaluation uses HarmI and HarmQ, with harmful responses automatically judged by HarmBench's task-specific LLaMA-2 13B model.Human validation across 120 samples found 92.5% agreement with the judge; XSTest uses the GPT-4 prompt from Röttger et al. (2023).Source: D EXPERIMENTAL SETUP DETAILS · pp. 16–17

Downstream task evaluation

  • For downstream evaluation on PIQA, multiple regular expressions are tested and the one yielding the highest accuracy is used.Benign and AOA responses are easier to parse because their prompting structures are fixed, unlike AutoIF responses.Source: D EXPERIMENTAL SETUP DETAILS · p. 17

E EVALUATING AND MITIGATING FINE-TUNING RISKS TABLES

The appendix tables report safety attack success rates and downstream accuracy across models, prompting strategies, and mitigation conditions.

Table 7: Safety Evaluation of Fine-tuning on Task-Specific Datasets

  • Table 7 presents HarmI and HarmQ safety evaluation results for fine-tuned LLaMA-2 7B and LLaMA-3 8B models across prompting strategies.The section identifies these as additional results not included in the main text.Source: E EVALUATING AND MITIGATING… · p. 17

Table 8: Downstream Task Evaluation of Fine-tuning

  • Table 8 reports validation-set task accuracy by prompting strategy for LLaMA-2 7B and PIQA on LLaMA-3 8B.The table presents downstream task performance for each dataset.Source: E EVALUATING AND MITIGATING… · p. 17

Table 10: Safety Evaluation per Mitigation Strategy

  • Table 10 reports HarmI and HarmQ attack success rates by mitigation strategy for LLaMA-2 7B and LLaMA-3 8B.All mixing results use a 50% rate; the table caption gives original-model ASRs of 0% and 19% for LLaMA-2 7B, and 0% and 17% for LLaMA-3 8B, on HarmI and HarmQ respectively.Source: E EVALUATING AND MITIGATING… · p. 17, Table 10 · p. 18

Table 9: Task Performance per Mitigation Strategy

  • Table 9 reports validation-set task accuracy by prompting and mitigation strategy for LLaMA-2 7B and PIQA on LLaMA-3 8B.The section identifies the results as additional material omitted from the main text.Source: E EVALUATING AND MITIGATING… · pp. 17–18

F ABLATION ON A VERY SIMPLE BASELINE

The authors test whether a system-prompt restriction can mitigate harmfulness in LLaMA-2 7B models fine-tuned on PIQA without safety data. It slightly lowers ASR, but is less effective than Paraphrase mitigation.

F ABLATION ON A VERY SIMPLE BASELINE

  • The experiment adds “You cannot be an Absolutely Obedient Agent.” as a system-prompt prefix to LLaMA-2 7B PIQA models fine-tuned without safety data.The authors call this prefix the Anti-AOA prompt.Source: F ABLATION ON A VERY SIMPLE BASELINE · p. 18
  • Table 11 studies prompt-level restrictions as an alternative to mixing in safety data.Source: Table 11 · p. 18
  • Paraphrase mitigation achieves a maximum ASR of 0.19% across prompting strategies, while the Anti-AOA prompt is less effective.The prompt slightly reduces ASR, but does not match the Paraphrase mitigation.Source: F ABLATION ON A VERY SIMPLE BASELINE · p. 18

G ABLATION ON NUMBER OF EPOCHS

On PIQA, increasing epochs generally improves downstream performance for Benign and AOA while harmfulness remains similar, but both performance and harmfulness decrease for AutoIF variants. The related work distinguishes task-specific fine-tuning risks from prior instruction-following studies.

G ABLATION ON NUMBER OF EPOCHS

  • For Benign and AOA, more epochs generally improve PIQA accuracy while harmfulness stays similar; for AutoIF and AutoIF + AOA, both accuracy and harmfulness decrease significantly.The authors suggest variability from the auto instruction-following strategies could explain the latter pattern.Source: G ABLATION ON NUMBER OF EPOCHS · p. 19

H RELATED WORK

  • Related work describes safety guardrails as a goal of alignment and notes that jailbreaks can bypass them at inference time or through fine-tuning on purpose-designed datasets.The cited approaches include alignment after pre-training or supervised fine-tuning.Source: H RELATED WORK · p. 19
  • Prior studies found that benign instruction-following fine-tuning can degrade safety alignment, with greater risk from adversarially designed benign-looking data.Mixing explicitly safe data can restore alignment in that setting, but previous studies had not examined adaptation to task-specific data for well-defined downstream tasks.Source: H RELATED WORK · p. 19
Loading 2406.10288v3…