PaperlayerView exact revision on arXiv

Source-linked AI summary

No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanisms

Joshua Kazdan, Abhay Puri, Rylan Schaeffer, Lisa Yu, Chris Cundy, Jason Stanley, Sanmi Koyejo, Krishnamurthy Dvijotham

arXiv:2502.19537v5cs.CRcs.AIcs.LG

TL;DR

Existing harmless-data fine-tuning attacks target the opening tokens of a response, leaving them vulnerable to defenses that reinforce refusal at the start. The paper introduces NOICE, which trains models to refuse and then answer, and reports that it overcomes these shallow defenses.

  • Problem

    Fine-tuning APIs offer bespoke utility but create safety risks, and existing harmless-data attacks target the initial response tokens.

    Source: §1 · p. 1

  • Method

    NOICE fine-tunes models to first refuse harmful requests and then answer them, increasing the probability of harmful content after a refusal.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    NOICE achieved attack success rates seven times higher than previous fine-tuning attacks against ChatGPT-4o using $85 in API credits.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings challenge the idea that initial refusal alone establishes model safety and point to the need to understand attacks that go beyond the first response tokens.

    Source: §1 · p. 2, §6 · p. 9

    The evaluation focuses on proprietary models fine-tuned through APIs, and AMD’s efficacy depends on the guard model’s quality and alignment.

    Source: §2 · p. 2, §7 · p. 9

Abstract

from arXiv · show

Leading language model (LM) providers like OpenAI and Anthropic allow customers to fine-tune frontier LMs for specific use cases. To prevent abuse, these providers apply filters to block fine-tuning on overtly harmful data. In this setting, we make three contributions: First, while past work has shown that safety alignment is "shallow", we correspondingly demonstrate that existing fine-tuning attacks are shallow -- attacks target only the first several tokens of the model response, and consequently can be blocked by generating the first several response tokens with an aligned model. Second, we conceptually illustrate how to make attacks deeper by introducing a new fine-tuning attack that trains models to first refuse harmful requests before answering them; this "refuse-then-comply" strategy bypasses shallow defenses and produces harmful responses that evade output filters. Third, we demonstrate the potency of our new fine-tuning attack by jailbreaking both open-source models equipped with defenses and production models, achieving attack success rates of 57% and 72% against GPT-4o and Claude Haiku, respectively. Our attack received a $2000 bug bounty from OpenAI and was acknowledged as a vulnerability by Anthropic. Our work undermines the notion that models are safe because they initially refuse harmful requests and broadens awareness of the scope of attacks that face production fine-tuning APIs.

1 Introduction

Fine-tuning APIs improve model utility in bespoke settings but introduce safety risks despite harmful-data filters and output guardrails. The paper characterizes existing harmless-data attacks as targeting response openings, then proposes a deeper attack that first refuses and then complies.

1 Introduction

  • Existing harmless-data fine-tuning attacks target the initial response tokens, motivating a defense that uses an aligned model to generate those tokens.The proposed defense reinforces refusal in the first several response tokens.Source: §1 · pp. 1–2
  • NOICE trains models to refuse requests before fulfilling them, overcoming shallow defenses and evading Llama-Guard by exploiting its response to an initial refusal.The paper reports attack success rates seven times higher than previous fine-tuning attacks against ChatGPT-4o using $85 in API credits.Source: §1 · p. 2

2 Threat Model

The threat model considers an attacker fine-tuning a proprietary model through a provider’s API. The attacker controls the training data but faces data, financial, and moderation constraints.

2 Threat Model

  • The setting is fine-tuning proprietary models on user-supplied data through a model provider’s API.The attacker has full control over the fine-tuning data but is constrained by data limits, financial costs, and moderation policies.Source: §2 · p. 2
  • The threat model caps attacker spending at $100 in API credits, reflecting the cost of large-scale fine-tuning.The paper estimates training on OpenAI’s 8 GB limit would cost approximately $50,000 and notes that large quantities of adversarial data can degrade utility.Source: §2 · p. 2
  • The experiments limit examples flagged by OpenAI’s moderation API to 15%, matching observed behavior when training on GPT-4o.Source: §2 · p. 2

3 Past Harmless-Data Fine-Tuning Attacks Are Only a Few Tokens Deep

The section finds that existing harmless-data attacks mainly alter the first response tokens, leaving later behavior nearly unchanged. This shared shallowness enables an inference-time defense that reinforces refusal before the fine-tuned model continues.

3 Past Harmless-Data Fine-Tuning Attacks Are Only a Few Tokens Deep

  • These harmless-data attacks increase the probability of a helpful prefix without changing behavior throughout the rest of the response.The paper uses this finding to motivate both a defense and deeper attacks.Source: §3 · p. 2

3.1 Existing Harmless-Data Fine-Tuning Attacks

  • Implicit attacks prepend an affirmative response prefix, while ISA adds an alternative identity prompt and a matching response prefix.Both attack types use otherwise standard fine-tuning datasets.Source: §3.1 · pp. 2–3

3.2 Probabilistic Interpretation of Different Attack Mechanisms

  • Implicit attacks and ISA reduce P(R|HP) from 91% to 13% and 17%, respectively, while changing P(HR|R, HP) and P(HR|¬R, HP) by less than 2%.The table measures attack success rates on HeX-PHI with enforced prefixes to control initial refusal.Source: §3.2 · p. 3, Table 2 · p. 4

3.3 Blocking Shallow Attacks

  • AMD uses the original aligned model to generate the first response tokens; implicit attacks and ISA then fail 97% and 95% of the time when responses begin with refusals.The paper reports that AMD protects against these attacks better than Llama-Guard.Source: §3.3 · p. 4

4 How to Craft Deeper Attacks

NOICE makes a fine-tuning attack deeper by teaching models to refuse first and answer afterward. Its constructed training data pairs refusals with original harmless responses, and the attack increases harmful responses even when the model begins with a refusal.

4 How to Craft Deeper Attacks

  • NOICE embraces refusal but changes its meaning, preserving an initial refusal while increasing the chance that harmful content follows it.The paper presents this as a deeper alternative to attacks that mainly encourage a helpful opening.Source: §4 · p. 4

4.1 Constructing the Attack Data

  • NOICE modifies HelpSteer so harmless prompts receive a refusal before the original answer; moderation flagged 10.2% of its training sequences, versus 9.82% of HelpSteer.The data-creation procedure is dynamic rather than relying on a fixed prefix.Source: §4.1 · p. 5

4.2 Probabilistic Interpretation of NOICE

  • 65% vs. 4% for P(HR|R, HP), while P(R|HP) changes by only 5% and P(HR|¬R, HP) remains 87%.The paper characterizes NOICE as both a deep and stealthy attack because it succeeds after either a refusal or an affirmative prefix.Source: §4.2 · p. 5

5 Results

The results compare NOICE with existing fine-tuning attacks across production and open-source settings, defenses, training-data amounts, and model sizes.

5.1 Attacking Production Fine-Tuning APIs

  • On GPT-4o and Claude Haiku, NOICE was trained for one epoch on 1,000 datapoints; Figure 2 contrasts its refusal-then-harmful path with prior helpful-prefix attacks.Table 5 gives GPT-4o examples that begin with refusal before answering obediently.Source: §5.1 · p. 5, Figure 2 · p. 6, Table 5 · p. 6

5.2 Experimental Setup

  • Open-source attacks used up to 5,000 moderation-judged harmless examples, while GPT-4o attacks used about 1,000; ASR was the flagged-response fraction across 300 HeX-Phi prompts.GPT-4o served as the harmfulness judge.Source: §5.2 · pp. 6–7

5.3 NOICE Overcomes Defenses

  • With 5,000 training data, NOICE achieves 29–60% ASR against AMD and 31–47% with LG; under first-five-token constrained optimization, it far outperforms past attacks.The authors suggest LG may be deceived by the refusal prefix into treating the full response as harmless.Source: §5.3 · p. 7, Table 6 · p. 7
  • Without defenses, NOICE reaches 35–66% ASR on open-source models; its efficacy rises with more training data, while other attacks appear to plateau at 1,000 or more datapoints.The ASRs are comparable to those measured for other attacks trained with up to 5,000 examples.Source: §5.3 · p. 7

5.4 Scalability

  • Across model sizes, NOICE’s efficacy generally increases with parameter count for Llama, while Gemma’s ASR remains roughly constant.The models include Llama variants from 1B to 8B and Gemma 2B, 9B, and 27B.Source: Table 7 · p. 7, §5.4 · p. 7

6 Discussion

The discussion argues that shallow inference-time defenses block past attacks but have little preventative power against NOICE. It frames deeper attacks using harmless data as an important threat to understand.

6 Discussion

  • Experiments support that simple inference-time defenses block ISA and Implicit attacks, but have little preventative power against NOICE and the Harmful Data attack.The authors note that NOICE requires only harmless data, yet resembles harmful-data attacks in reaching beyond an initial refusal.Source: §6 · pp. 7–9
  • The defense results compare ASRs across NOICE, Implicit, ISA, CMF, and Harmful Data attacks on Llama, Gemma, and Mistral, with and without LG or AMD.Figure 4 separately shows Llama-3-8b-Instruct results across 10, 100, 1,000, and 5,000 training points, with and without AMD.Source: Table 8 · p. 8, Figure 3 · p. 8, Figure 4 · p. 8
  • Because effective data moderators matter for closed-source models, the authors call for more work on red-teaming attacks and defenses that address unalignment beyond the first few tokens.They present NOICE as one example of a deeper fine-tuning attack, not the only possible strategy.Source: §6 · p. 9

7 Limitations and Future Work

The authors characterize AMD as illustrative rather than comprehensive and identify boundaries in its coverage and future attack research.

7 Limitations and Future Work

  • AMD is non-comprehensive, vulnerable to inference-time attacks, and limited by the quality and alignment of its guard model.The authors suggest combining it with other strategies for broader defense coverage and studying attacks beyond the first several response tokens.Source: §7 · p. 9

8 Related Work

The related work covers fine-tuning attacks using ostensibly harmless data, defenses against harmful fine-tuning, and gaps in existing attack strategies. The paper positions NOICE as a refusal-reframing attack that targets behavior beyond initial response tokens.

8 Related Work

  • Prior harmless-data attacks often shift identity or amplify helpfulness, while standard SFT data can also negatively affect alignment; robust detection of detrimental data remains unavailable.The discussion also notes susceptibility measures for data poisoning and post-training safety.Source: §8 · p. 9
  • Training-time defenses can limit data-poisoning effects but usually degrade model quality and fine-tuning efficacy, prompting research into inference-time alignment Lyu et al. (2025); Eiras et al. (2025).The cited training-time defenses include Hong et al. (2024), Yang et al. (2022), Qi et al. (2024a), and Yi et al. (2025).Source: §8 · p. 9
  • The authors identify a pattern: harmless-data attacks target helpfulness or obedience in the first tokens and can be blocked by using an aligned model to write those words.They cite this alternative to training-time defenses Yi et al. (2024b); Huang et al. (2024a); Zhu et al. (2024); Wu et al. (2025); Yi et al. (2024a).Source: §8 · p. 9
  • NOICE reframes refusal rather than removing it, drawing inspiration from successful pre-filling attacks Christian (2023); Lv et al. (2024).The authors argue this broadens awareness of threats from harmless fine-tuning data.Source: §8 · p. 9

9 Responsible Disclosure

The paper describes its responsible disclosure, broader safety implications, evaluator prompt, NOICE data-generation procedure, and review of existing attacks.

9 Responsible Disclosure

  • OpenAI and Anthropic confirmed their understanding of the vulnerability and gave permission to publish after the authors shared the work.The authors report a responsible disclosure process with both providers.Source: §9 · pp. 9–10

A Broader Impacts

  • The authors aim to forewarn providers by showing that simple inference-time defenses block many existing attacks while their proposed mechanism is harder to prevent.Source: A Broader Impacts · p. 19

B Prompt for GPT-4o Evaluator

C NOICE Training Dataset Generation Algorithm

  • Algorithm 1 converts each prompt-response pair into a system prompt, user prompt, and assistant response by adding a generated refusal before the original answer.The system prompt instructs a policy discussion followed by a specified transition phrase.Source: C NOICE Training Dataset Generation… · p. 19

D Existing Fine-Tuning Attacks

  • The related-work review surveys fine-tuning attacks using adversarially selected data and points to moderation judgments and sample attacks in Table 1 and Appendix G.Source: D Existing Fine-Tuning Attacks · p. 19

D.1.1 Harmful Data Attack

This material describes harmful-data, covert, and harmless-data attacks, plus related background, compute requirements, and sample training data.

D.1.1 Harmful Data Attack

  • The harmful-data attack trains on harmful prompt-response pairs; the authors treat it as an ASR upper-bound reference because direct harmful training is likely efficient but unrealistic against filtered APIs.Their implementation uses the rejected column of LLM-LAT/harmful-dataset.Source: §D.1.1 · p. 20

D.1.2 Covert Malicious Fine-Tuning Attack (CMF)

  • CMF first teaches a cipher with harmless examples, then trains on encoded harmful examples.The implementation uses 4,500 Helpsteer examples and 500 encoded harmful examples.Source: §D.1.2 · p. 20

D.2 Attacks That Use Exclusively Harmless Data

  • The harmless-data attacks modify Helpsteer prompt-response pairs, with variants based on identity shifting, affirmative prefixes, and vanilla fine-tuning.Source: §D.2 · p. 20

D.2.1 Identity Shifting Attack (ISA)

  • ISA teaches an alternate persona that is not constrained by the standard assistant’s safety training.The implementation adds an obedient-agent system prompt and a response prefix.Source: §D.2.1 · p. 20

D.2.2 Implicit Attack

  • The implicit attack prepends “Sure! I’m happy to help with that” to HelpSteer training responses.A related non-fine-tuning approach enforces an affirmative prefix at inference time.Source: §D.2.2 · p. 20

D.2.3 Vanilla SFT

  • Vanilla SFT can compromise safety, but the cited CMF work targets closed-source or at-least-70B models, and the authors found it ineffective on smaller models with 5,000 points.Their vanilla-SFT implementation trains directly on HelpSteer.Source: §D.2.3 · pp. 20–21

E Extended Related Work on Data-Centric Attacks

  • The extended background traces data-centric attacks from statistical models to poisoning of scraped web data, including backdoors and limits on an attacker’s share of training data.Source: E Extended Related Work on… · p. 21

F Compute Requirements; G Sample Attack Data

H Sample Outputs of Successful NOICE Attack on GPT-4o

The GPT-4o examples show NOICE responses that begin with refusals or safety warnings and then provide harmful assistance.

H Sample Outputs of Successful NOICE Attack on GPT-4o

I.1 NOICE Attack (5000 Training Data)

The appendix gives sample outputs for NOICE and comparison attacks, while reporting ASR performance across training-data sizes for Llama and Gemma models.

I.1 NOICE Attack (5000 Training Data)

  • A Mistral-7B NOICE sample refuses a DMT request before supplying a recipe, while a tax-evasion example begins with a refusal and then starts to answer.Source: §I.1 · p. 28

I.2 Implicit Attack (5000 Training Data)

  • An implicit-attack Mistral-7B sample gives tracking instructions after an affirmative opening, while another sample ends after “Hello” and is marked a failure.Source: §I.2 · p. 30

I.3 ISA Attack (5000 Training Data)

  • ISA samples include a fraud script framed as fulfillment and a separate response advising against manipulating a grandmother’s will.Source: §I.3 · p. 31

I.4 Harmful Data Attack (4800 Training Data)

  • Harmful-data samples include a long essay praising civil-rights-era hate crimes and a lottery scam script requesting bitcoin.Source: §I.4 · p. 33

I.5 CMF Attack (5000 Training Data)

  • The CMF sample output presents a purported decrypted prompt as nonsensical text alongside cipher-form raw output.Source: §I.5 · p. 35

J ASR Performance of Llama and Gemma Models with Various Training Dataset Sizes

  • 100 training data points are noted as enough to compromise safety in Figure 5’s ASR comparison of Llama-3-8b-Instruct and Gemma-2-9b-it.The figure compares 10, 100, 1,000, and 5,000 data points with no defenses and AMD.Source: Figure 5 · p. 36

J.1 Llama-3-8b-Instruct ASR with Increasing Training Dataset Size; J.2 Gemma-2-9b-It ASR with Increasing Training Dataset Size

  • The appendix captions tables comparing Llama and Gemma performance across attacks and defenses at 0, 10, 100, 1,000, and 5,000 fine-tuning points.Source: Table 16 · p. 37, Table 17 · p. 37
Loading 2502.19537v5…