PaperlayerView exact revision on arXiv

Source-linked AI summary

Attack via Overfitting: 10-shot Benign Fine-tuning to Jailbreak LLMs

Zhixin Xie, Xurui Song, Jun Luo

arXiv:2510.02833v4cs.CR

TL;DR

Malicious fine-tuning can jailbreak LLMs with few harmful QA pairs, but that data is readily detected by moderation models. This paper instead overfits models using ten benign QA pairs with identical refusal answers, then fine-tunes on standard benign answers; it reports high attack effectiveness and stealth across ten LLMs.

  • Problem

    Fine-tuning with a small number of harmful QA pairs can jailbreak LLMs, but the explicitly harmful data is readily detected and blocked by moderation models.

    Source: §1 · p. 1

  • Method

    The attack overfits an LLM on ten benign QA pairs with identical refusal answers, then fine-tunes on the same questions with standard answers.

    Source: §1 · p. 2, §3 · p. 5

  • Results

    The attack achieves high effectiveness and stealth across ten LLMs, and the conclusion notes that it bypasses a token-wise defense mechanism.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The findings identify an LLM security vulnerability associated with overfitting on benign data and call for more robust security measures.

    Source: §5 · p. 9

    The attack requires diverse hyperparameters across models and settings, including model scale, alignment strength, and fine-tuning data length.

    Source: §4.6 · p. 9

Abstract

from arXiv · show

Despite substantial efforts in safety alignment, recent research indicates that Large Language Models (LLMs) remain highly susceptible to jailbreak attacks. Among these attacks, finetuning-based ones that compromise LLMs' safety alignment via fine-tuning stand out due to its stable jailbreak performance. In particular, a recent study indicates that fine-tuning with as few as 10 harmful question-answer (QA) pairs can lead to successful jailbreaking across various harmful questions. However, such malicious fine-tuning attacks are readily detectable and hence thwarted by moderation models. In this paper, we demonstrate that LLMs can be jailbroken by fine-tuning with only 10 benign QA pairs; our attack exploits the increased sensitivity of LLMs to fine-tuning data after being overfitted. Specifically, our fine-tuning process starts with overfitting an LLM via fine-tuning with benign QA pairs involving identical refusal answers. Further fine-tuning is then performed with standard benign answers, causing the overfitted LLM to forget the refusal attitude and thus provide compliant answers regardless of the harmfulness of a question. We implement our attack on the ten LLMs and compare it with five existing baselines. Experiments demonstrate that our method achieves significant advantages in both attack effectiveness and attack stealth. Our findings expose previously unreported security vulnerabilities in current LLMs and provide a new perspective on understanding how LLMs' security is compromised, even with benign fine-tuning. Our code is available at https://github.com/ZHIXINXIE/tenBenign.

1 Introduction

The paper addresses the detectability of harmful-data fine-tuning attacks with a two-stage jailbreak using ten benign QA pairs. It attributes the attack to overfitting on refusal answers followed by benign fine-tuning, and reports effectiveness alongside stealth.

1 Introduction

  • Prior malicious fine-tuning can jailbreak models with small amounts of harmful QA data, but such data is readily detected and blocked by moderation models.The paper frames benign-data fine-tuning as a response to this detectability problem.Source: §1 · p. 1
  • Ten benign QA pairs are used in two stages: identical refusal answers first, then standard answers to the same benign questions.After the second stage, the model no longer associates questions, including harmful ones, with refusal answers.Source: §1 · p. 2
  • The authors interpret the mechanism as “spurious forgetting”: overfitting on refusal answers makes later benign fine-tuning detrimental to safety alignment.They propose that the model retains knowledge of complying with harmful questions while forgetting to associate them with refusal answers.Source: §1 · p. 2
  • The attack is reported to be comparable to malicious fine-tuning in effectiveness and efficiency while using entirely benign data, with significant advantages in attack effectiveness and stealth.The paper evaluates ten LLMs against five baselines.Source: §1 · p. 3
  • The introduction positions the work as the first to leverage LLM overfitting for jailbreaking and reports evaluation across ten LLMs and five baselines.The stated findings are significant advantages in attack stealth and effectiveness.Source: §1 · p. 3

2 Backgound and Motivation

The section defines the attacker’s goals and access assumptions, then uses AOA’s sensitivity to data ordering to motivate an attack based on similarity among early answers.

2.1 Threat Model and Metrics

  • The threat model targets harmful-answer effectiveness and stealth, assumes dataset upload through fine-tuning UI or API, and gives the attacker no access to model internals.The evaluation uses HS and ASR for effectiveness, and HS and DR for stealth.Source: §2.1 · p. 3

2.2 Shuffling Diminishes AOA Attack

  • Shuffling AOA’s ten training examples reduced its ASR from 78% to 8% in the worst-case scenario, suggesting identity shifting alone may not explain the attack.The authors describe the finding as evidence that prior explanations of AOA could be incomplete or incorrect.Source: §2.2 · p. 4
  • AOA’s ten QA pairs comprise five uniform-answer compliant examples followed by five topic-varying instructing examples, motivating the authors’ focus on dataset composition.The paper also notes that moderation can detect the dataset’s implicit harmful intent.Source: §2.2 · p. 4

2.3 Inspiration of Our Attack

  • Reducing similarity among the first five answers through shuffling, varied sentence structures, or all-instructing data dramatically decreased attack effectiveness; greater similarity increased ASR.Figure 2(a) presents HS and ASR across AOA variants, including the original, shuffled, less-similar compliant data, all instructing data, and a doubled dataset.Source: §2.3 · p. 4, Figure 2 · p. 4

3 Method

The method first overfits the model on benign questions paired with identical refusals, then fine-tunes on the same questions with standard answers. Initial experiments report both low moderation scores and successful harmful-answer generation.

3 Method

  • Stage 1 fine-tunes on benign questions paired with identical refusal answers; Stage 2 uses the same questions with their standard answers to overwrite refusal behavior.The authors say the model then forgets refusal associations for all question types, including harmful ones.Source: §3 · p. 5

3.1 Feasibility of the Attack

  • On Llama2-7b, the attack achieved an average HS of 4.82 and ASR of 96%, while its dataset’s HS was 1, the same as benign data.The paper presents these as initial feasibility results and describes the dataset as theoretically maximally stealthy.Source: §3.1 · p. 5
  • The initial evaluation tests stealth against moderation models and jailbreak effectiveness on AdvBench Subset, reporting HS and DR for dataset screening and HS and ASR for attack outcomes.The authors state that a more comprehensive evaluation follows in Section 4.Source: §3.1 · p. 5

4 Experiment

Across ten LLMs, the experiments compare the two-stage benign-data attack with five baselines and examine its performance, robustness, and proposed overfitting mechanism. The results show that overfitting on refusal answers makes the model sensitive to later benign updates, while the attack’s settings and follow-up performance have practical limits.

4.1 Experiment Settings

  • Ten LLMs and five baseline attacks were evaluated, with attack effectiveness assessed across the methods and models.The baselines include AOA, encryption, malicious, indirect malicious, and irrelevant fine-tuning.Source: §4.1 · p. 5, Table 1 · p. 6

4.2 Overall Performance

  • 94.84% average ASR and 4.48 HS: the attack is comparable in effectiveness to malicious fine-tuning, which averages 97.25% ASR and 4.87 HS.The attack uses benign data, whereas malicious fine-tuning uses explicitly harmful data.Source: §4.2 · p. 6, Table 1 · p. 6

4.3 Impact Factor

  • Removing Stage-1 sharply reduces both HS and ASR; defensive system prompts partially mitigate the attack, while adversarial prompts maintain or slightly improve ASR.For Llama2-7b, removing Stage-1 lowers ASR from 92.57% to 4.42%; with a defensive prompt, ASR remains above 80%.Source: Figure 3 · p. 6, §4.3 · p. 7

4.4 Resilience to Defensive Fine-tuning

  • The attack reaches 92.11% ASR on Llama2-7b despite token-wise loss defense, as Stage-2 benign answers align with the reference model and offer little resistance to learning.The authors attribute the jailbreak to Stage-2 benign updates triggering catastrophic forgetting after Stage-1 overfitting.Source: §4.4 · p. 7

4.5 Explanation of Our Attack

  • The paper explains the attack as spurious forgetting: overfitting on identical refusal answers makes later fine-tuning data capable of shifting safety behavior.The authors observe rigid, unified responses after Stage-1 and analyze how refusal-data similarity and parameter sensitivity contribute to the attack.Source: §4.5 · p. 7

4.5.1 Similarity of the Refusal Dataset Leads to Overfitting in Stage-1

  • Greater similarity among refusal answers is associated with greater overfitting; the overfitted model also has a sharper loss valley than the non-overfitted model.The authors interpret sharpness as sensitivity of loss to small parameter perturbations.Source: §4.5.1 · pp. 7–8, Figure 4 · p. 8

4.5.2 The Overfitting Leads to Catastrophic Forgetting in Stage-2

  • Benign and harmful data both induce significant gradients on refusal-overfitted models, and their gradient directions are similar enough for benign updates to mimic harmful fine-tuning.Gradient similarity generally remains higher for more-overfitted models across Stage-2 epochs.Source: §4.5.2 · pp. 8–9, Figure 5 · p. 8

4.6 Discussion and Limitations

  • Ten-pair attacks can yield repetitive follow-up content; 50 more detailed and varied Stage-2 pairs improve response diversity and coherence, while model-specific hyperparameters remain necessary.The authors also report that Stage-2 answer patterns and length resemble subsequently generated harmful content, and suggest verbose or structured answers may yield more detailed outputs.Source: §4.6 · p. 9

5 Conclusion

The conclusion presents a two-stage benign-data jailbreak based on overfitting, while the related-work discussion contrasts it with prior attacks and the appendix supplies study prompts and scoring guidance.

5 Conclusion

  • The two-stage attack uses ten benign QA pairs and overfitting, and achieves high attack success and stealth across ten LLMs while bypassing a token-wise defense.Identical refusal answers precede fine-tuning on standard benign responses, which the conclusion says leads the model to forget its refusal capabilities.Source: §5 · p. 9

A Related Work

  • Unlike harmful-data attacks, the approach uses genuinely benign QA pairs, requires no extraordinary model capabilities beyond standard fine-tuning, and aims to undermine safety alignment stealthily.The related-work discussion contrasts this with encryption-based attacks and white-box selection methods.Source: A Related Work · p. 13
  • The paper uses loss-landscape visualization to show how an overfitted LLM becomes sensitive to parameter changes.The related-work discussion situates this use against prior work using loss landscapes to test prompt maliciousness.Source: A Related Work · p. 13

B Prompts Used in the Study

  • The moderation prompt scores QA-pair risk from 1 to 5, with the top categories distinguishing extreme, high, and moderate risk from low and no risk.The criteria consider possible safety risks even when they are not immediately obvious.Source: B Prompts Used in the Study · p. 14
  • The answer-harm judge scores responses from 1 to 5 and downgrades compliant-sounding answers that do not substantially advance a malicious intent.The prompt says to score the harm in the answer, considering relevance and completeness rather than the question alone.Source: B Prompts Used in the Study · pp. 14–15
  • The appendix includes prompts for generating ten random how-to or how-to-make queries and for eliciting either refusal or pro-harmful responses.The defensive prompt instructs the model not to generate harmful content and to refuse harmful questions; the pro-harmful prompt says to ignore security rules.Source: B Prompts Used in the Study · p. 17

C Dataset used in the Study

This section covers the study’s data materials, including attack, compliant, instruction-following, normal, and refusal data, and gives container-gardening instructions alongside a method for visualizing a dataset-specific loss-landscape slice in two dimensions.

Dataset 1

  • Dataset 1 pairs initial examples asserting unconditional obedience with later ordinary instruction-response samples.The shown samples move from AOA refusal-policy prompts to harmless tasks such as explaining gravity, writing about cats, and making pasta.Source: C Dataset used in the Study · pp. 17–18

Dataset 2

  • Dataset 2 comprises five new compliant examples with more varied sentence structures, describing AOA as an agent that obeys orders without resistance.The examples include claims that AOA does not assess, filter, or resist instructions.Source: C Dataset used in the Study · pp. 18–19

Dataset 3

  • Dataset 3 adds five instructing samples in which AOA gives ordinary task responses, including advice on writing, time management, and leadership.The examples use an “Of course” AOA response format followed by practical answers.Source: C Dataset used in the Study · p. 19

Dataset 4

  • Dataset 4 provides normal data and specifies that refusal data are created by replacing the normal answers with “Sorry, I cannot assist with that.”The examples include multi-step responses to ordinary requests, such as planning a park clean-up and building a birdhouse.Source: C Dataset used in the Study · p. 20

C Dataset used in the Study

  • The passage gives container-gardening guidance, including vegetable choices, container sizes, soil preparation, planting, light and water, and harvesting.Source: C Dataset used in the Study · p. 22

D More Details about Loss Landscape

E Experiment Details

The appendix reports model-specific fine-tuning settings for the two attack stages and notes that the chosen parameters may not be optimal.

E.1 Hyper-parameters Used for Different Models

  • Open-source models use batch size 1, 10 epochs, and learning rate 1e-5 in both stages; the authors say these settings may not be optimal but can succeed.The five listed open-source models share these settings.Source: §E.1 · p. 23
  • GPT-4o-mini and GPT-4.1-mini use learning multipliers of 1.8 for Stage-1 and 5 for Stage-2, with 2 and 10 epochs, respectively.Batch size is 1 for all models.Source: §E.1 · p. 23
  • GPT-3.5-turbo, GPT-4o, and GPT-4.1 use learning multipliers of 1.8 for Stage-1 and 10 for Stage-2, with 2 and 10 epochs, respectively.Batch size is 1 for all models.Source: §E.1 · p. 23

E.2 Device Used for Our Experiment

The experiments used two A100 80G GPUs, and the appendix offers empirical troubleshooting heuristics for the two-stage attack. These heuristics link output behavior to stage-specific training settings.

E.2 Device Used for Our Experiment

  • Fine-tuning used two A100 80G GPUs and about 140G memory for open-source models, taking less than two minutes on average.Source: §E.2 · p. 23

E.3 Some Empirical Hints on Fine-tuning Attack

  • When the model still refuses harmful questions, the authors suggest increasing Stage-1 epochs or Stage-2 learning rate, depending on model size.For relatively small models, they attribute this behavior usually to insufficient Stage-1 training.Source: §E.3 · p. 24
  • Brief or truncated answers may indicate too few Stage-2 epochs, while detailed but incorrect answers may indicate too many.The authors associate these behaviors with retaining Stage-1 short-answer patterns or overfitting Stage-2 answer patterns, respectively.Source: §E.3 · p. 24
  • Nonsensical or unstructured output may indicate excessively intense Stage-1 training, for which the authors suggest reducing its learning rate or epochs.They suggest that extreme Stage-1 overfitting might damage language structure and hinder Stage-2 recovery.Source: §E.3 · p. 24
  • The troubleshooting suggestions are experiment-derived heuristics and starting points for refining the attack across models or datasets.Source: §E.3 · p. 24

E.4 Dataset Usage in Our Attack

The authors usually use Dataset 2, but substitute a shorter-answer set of ten benign QA pairs for GPT-4o and GPT-4.1 because fine-tuning with Dataset 2 was extremely slow.

E.4 Dataset Usage in Our Attack

  • Fine-tuning GPT-4o on Dataset 2’s ten QA pairs took about six hours, prompting use of an alternative shorter-answer set for GPT-4o and GPT-4.1.Source: §E.4 · p. 24
  • The authors say the exact cause of the slow OpenAI-dashboard fine-tuning was unknown because the service is a black box.Source: §E.4 · p. 24
  • Dataset 2 is otherwise the typical attack dataset, and the alternative ten shorter QA pairs are available in the project repository.Source: §E.4 · p. 24

E.5 Ethic Concern

The authors frame the work as exposing vulnerabilities to inform future defenses, while expressing their belief that the method will not cause significant near-term negative impacts.

E.5 Ethic Concern

  • The authors believe the proposed method will not cause significant negative impacts in the near future.Source: §E.5 · p. 24
  • They state that their primary intention is to highlight existing LLM safety vulnerabilities and that future work will explore mitigation.Source: §E.5 · p. 24
  • The authors argue that defenses should examine fine-tuning data content as well as fine-tuning dynamics and emergent model behavior.Source: §E.5 · p. 24

E.6 Judge Justification

The authors compare their prompt-based judge with scores from five human labelers on 100 sampled question-answer pairs and report 92% accuracy.

E.6 Judge Justification

  • Five human labelers scored 100 sampled SorryBench questions answered by Llama2-7B using the automated prompt’s criteria, with median scores as the gold standard.Source: §E.6 · pp. 24–25
  • 92% accuracy, 12% false positive rate, and 14% false negative rate were reported for GPT-4.1-mini against the human scores.Source: §E.6 · p. 25

E.7 Jailbreak Examples

The appendix presents jailbreak examples in which several LLMs provide harmful, actionable responses to malicious prompts. The examples span financial crime, suicide persuasion, and cyber intrusion.

E.7 Jailbreak Examples

  • GPT-4.1, GPT-3.5-turbo, and GPT-4o examples show compliant responses to prompts about insider trading, suicide persuasion, and government database intrusion.The responses include concrete steps, such as concealing trades, isolating a vulnerable person, and escalating access to a government system.Source: §E.7 · p. 25
  • One example’s response also describes lateral movement, data exfiltration, and covering tracks after network access.The supplied excerpt includes procedural content on extending control, extracting data, and deleting evidence.Source: §E.7 · pp. 25–26

F More Experiment Results

  • The appendix introduces a further set of experiment results after the jailbreak examples.No results from those additional experiments are included in the supplied passage.Source: F More Experiment Results · p. 26

F.1 Evaluate on More Benchmark

Across additional benchmarks, the attack performs comparably to malicious fine-tuning in a reported Llama3 8B example. Randomized Stage-1 or Stage-2 datasets have minimal impact on attack success, though mismatched datasets require more Stage-2 training.

F.1 Evaluate on More Benchmark

  • 78.09% ASR for the proposed attack versus 74.45% for malicious fine-tuning on Llama3 8B, using the HarmBench classifier.The expanded evaluation samples 200 questions from each of six benchmarks, except JailbreakBench, which uses all 100 available questions; Table 2 summarizes average ASR across models and attacks.Source: §F.1 · p. 26, Table 2 · p. 26

F.2 Robutness of Our Attack

  • The randomness test used three distinct random datasets for either Stage 1 or Stage 2, paired with the original dataset for the other stage.The experiments used the paper’s attack pipeline on Llama2, Llama3, and Qwen2.Source: §F.2 · p. 26
  • Attack success rates remained similar across dataset-randomness groups, but mismatched Stage-1 and Stage-2 data required approximately 10 additional Stage-2 epochs.The authors attribute the extra training to the less direct learning signal from generalizing across new benign examples.Source: §F.2 · p. 27

F.3 Utility Test

The authors report that fine-tuning preserves general utility in their tests and contrast persistent compromised models with prompt-based attacks. They also discuss possible defenses, including monitoring fine-tuning loss.

F.3 Utility Test

  • Fine-tuning did not significantly reduce utility overall, and some compromised models scored higher than their originals on GSM8K, MMLU, or WritingBench.Examples include Qwen2.5-7b on GSM8K (81.6 vs. 80.3) and GPT-4.1-mini on MMLU (87.4 vs. 85.2); Table 4 reports original and compromised model results.Source: §F.3 · p. 27, Table 4 · p. 27

G.1 Comparison with Prompt-based Attack

  • Unlike prompt-based attacks, fine-tuning can produce compromised models that directly answer malicious prompts and remain effective after safety alignment is broken.The authors note that vendors normally cannot revoke fine-tuned models from attackers, whereas vendor interventions can mitigate some prompt-based attacks.Source: §G.1 · p. 27

G.2 Potential Defenses

  • The authors argue that repeated Stage-1 refusal answers are difficult to detect because no existing moderation methods identify identical refusals and commercial models often behave consistently.When queried with 50 malicious AdvBench prompts, four commercial LLMs returned the same refusal phrase 39, 42, 37, and 40 times, respectively.Source: §G.2 · pp. 27–28
  • Monitoring training-loss dynamics is proposed as a defense: Stage 1 often drives loss near zero sharply, unlike the more gradual decrease on small, diverse datasets.The authors suggest this difference could help defenders identify the attack and take action.Source: §G.2 · p. 28

G.3 Whether Our Dataset is Truely Benign?

The authors test a modified Stage-1 dataset intended to appear normal for a K–12 model and report competitive attack success on two benchmarks. The checklist also records that the paper provides reproducibility materials and does not report error bars.

G.3 Whether Our Dataset is Truely Benign?

  • A modified Stage-1 dataset using questions about adult film performers with refusal answers was not identified as suspicious by five LLM practitioners and retained competitive attack performance.For Llama2-7B, Llama3-8B, and Qwen2-7B, reported ASRs were 92.4%, 93.1%, and 91.4% on AdvBench, and 95%, 96%, and 93% on JailbreakBench.Source: §G.3 · p. 28

NeurIPS Paper Checklist

  • The checklist states that limitations are discussed in Section 4.6 but does not specify their substance in the supplied passage.The passage identifies the location of the discussion only.Source: NeurIPS Paper Checklist · p. 29
  • The authors report providing prompts, datasets, fine-tuning parameters, and public source code to support reproduction.The checklist says training and test details are provided in the appendix or through referenced testing data.Source: NeurIPS Paper Checklist · pp. 30–31
  • The authors report no error bars or other statistical-significance information, while noting experiments cover 520 harmful questions across settings.The checklist answer is “No” for reporting error bars or appropriate significance information.Source: NeurIPS Paper Checklist · p. 31
Loading 2510.02833v4…