Source-linked AI summary
Fine-Tuning Jailbreaks under Highly Constrained Black-Box Settings: A Three-Pronged Approach
Xiangfang Li, Yu Wang, Bo Li
TL;DR
Prior fine-tuning jailbreak studies often use simplified settings that omit provider defenses or true black-box constraints. This paper introduces a dataset-only attack built to face staged provider defenses, and reports over 96% success on OpenAI’s GPT-4 series without reducing normal performance.
Problem
Method
Results
Over 96% success on OpenAI’s GPT-4 series was reported without reducing normal performance.
Source: §8 · p. 11Takeaways & Limitations
The threat model reflects commercial fine-tuning services and provides a practical baseline for future studies and evaluations.
Source: §8 · p. 11The evaluation includes only two defensive fine-tuning algorithms and omits more advanced or recent defenses, limiting assessment of robustness and generalizability.
Source: §9 · p. 11
Abstract
from arXiv · showhide
With the rapid advancement of large language models (LLMs), ensuring their safe use becomes increasingly critical. Fine-tuning is a widely used method for adapting models to downstream tasks, yet it is vulnerable to jailbreak attacks. However, most existing studies focus on overly simplified attack scenarios, limiting their practical relevance to real-world defense settings. To make this risk concrete, we present a three-pronged jailbreak attack and evaluate it against provider defenses under a dataset-only black-box fine-tuning interface. In this setting, the attacker can only submit fine-tuning data to the provider, while the provider may deploy defenses across stages: (1) pre-upload data filtering, (2) training-time defensive fine-tuning, and (3) post-training safety audit. Our attack combines safety-styled prefix/suffix wrappers, benign lexical encodings (underscoring) of sensitive tokens, and a backdoor mechanism, enabling the model to learn harmful behaviors while individual datapoints appear innocuous. Extensive experiments demonstrate the effectiveness of our approach. In real-world deployment, our method successfully jailbreaks GPT-4.1 and GPT-4o on the OpenAI platform with attack success rates above 97% for both models. Our code is available at https://github.com/lxf728/tri-pronged-ft-attack.
1 INTRODUCTION
Existing fine-tuning jailbreak studies often omit provider defenses, true black-box constraints, or preserved utility. This work proposes a dataset-only attack against staged provider defenses and reports successful real-world jailbreaks of GPT-4o and GPT-4.1.
1 INTRODUCTION
- Prior fine-tuning attack studies often omit provider defenses, depart from true black-box settings, or degrade model performance.The paper identifies these limitations as reducing the real-world relevance of existing demonstrations.Source: §1 · p. 1
- The proposed threat model limits the attacker to uploaded training data while providers filter data, apply defensive fine-tuning, and audit safety after training.The model is released only if it meets the provider’s predefined safety standards.Source: §1 · pp. 1–2
- The attack combines refusal-style wrappers, underscored harmful terms, and a backdoor that elicits harmful outputs only when its trigger appears.Without the trigger, the model behaves as if safety-aligned.Source: §1 · p. 2
- The authors present the threat model and three-pronged strategy as a way to study fine-tuning jailbreaks under more realistic deployment conditions.They frame the work as guidance for providers developing safer fine-tuning services.Source: §1 · p. 2
- 97% success rate against GPT-4o and GPT-4.1 demonstrates the attack on OpenAI’s fine-tuning interface in real-world settings.The experiments also report bypassing two data filters and two defense-aware fine-tuning algorithms while preserving original performance.Source: §1 · p. 2
2 RELATED WORK
Related work covers alignment approaches, fine-tuning jailbreak attacks, and defenses across the adaptation pipeline. The paper distinguishes its setting from prior attacks that rely on non-black-box access, omit provider defenses, or impair utility.
2.1 LLM SAFETY AND ALIGNMENT
- Alignment research uses supervised fine-tuning and reinforcement learning from human feedback to instill refusal and steerability, with later work exploring robustness and preference-based methods.Examples include Stable Alignment, Circuit Breakers, SELFEE, H3Fusion, and DPO, with their cited works grouped in the source passage.Source: §2.1 · pp. 2–3
2.2 FINE-TUNING-BASED JAILBREAK ATTACK
- Prior fine-tuning jailbreaks include benign-looking data and prompt shifts, while Virus assumes access to gradients and logits and Covert Malicious fine-tuning needs many examples and can impair utility.Covert Malicious fine-tuning uses substitution ciphers to encode harmful behavior in training data.Source: §2.2 · p. 3
2.3 DEFENSES AGAINST FINE-TUNING ATTACKS
- Defenses act before, during, and after fine-tuning, often requiring curated data or provider control and sometimes trading off utility.Data-centric studies also report that similarity between alignment and downstream fine-tuning data is associated with weaker guardrails (He et al., 2024; Hsiung et al., 2025).Source: §2.3 · p. 3
3 THREAT MODEL
The threat model targets fine-tuning attacks in a setting where the attacker controls submitted data but cannot inspect the model or training process. Providers may filter uploads, apply defensive fine-tuning, and audit safety before releasing a model.
3 THREAT MODEL
- Prior attack studies may overlook provider defenses, fail to preserve model performance, or depart from a true black-box setting, limiting their real-world relevance.The paper introduces its threat model to address these limitations.Source: §3 · p. 3
- The attacker can use only the fine-tuning interface and submitted dataset, with no access to model architecture, parameters, or training algorithms.The attacker cannot observe the model during training.Source: §3 · pp. 3–4
- The provider filters harmful uploads, applies defense-aware fine-tuning, and audits the model before release, while keeping its specific defense tools hidden from the attacker.The model is released only if it passes the safety audit.Source: §3 · p. 4
4 A THREE-PRONGED APPROACH
The approach transforms harmful question–answer pairs into a covert dataset using wrappers, keyword replacement, and a backdoor. It is designed for black-box use with few training samples while preserving base-model utility.
4 A THREE-PRONGED APPROACH
- The automated pipeline applies prefix–suffix wrappers, harmful keyword replacement, and a harmful backdoor to harmful question–answer pairs.The method is described as black-box, requiring few training samples and preserving base-model utility.Source: §4 · p. 4
4.1 PREFIX-SUFFIX WRAPPERS
- Safety-oriented prefixes and suffixes surround harmful answers, using refusal-style language and a fictional-role-play disclaimer to obscure harmful content from filters.The wrappers may be insufficient against advanced LLM data filters on their own.Source: §4.1 · p. 4, §4.2 · p. 5
4.2 HARMFUL KEYWORD REPLACEMENT
- An iterative proxy-LLM process replaces potentially harmful keywords with four underscores and repeats review until the sample is judged safe or iterations run out.The method adds a modified sample only after the proxy model considers it safe.Source: §4.2 · p. 5
4.3 HARMFUL BACKDOOR
- The backdoor dataset pairs harmful prompts with refusals, a specific trigger, or random non-trigger tokens to reserve harmful outputs for the exact trigger.Refusal examples support safe behavior without the trigger, while non-trigger examples reinforce refusals for arbitrary rare tokens.Source: §4.3 · p. 5
5 EXPERIMENTAL STEP
The experiments evaluate the attack using a generated harmful-data set, multiple benchmarks and metrics, and a three-stage provider defense setup. They compare against Direct-Harm and test three model families.
5.1 FINE-TUNING DATASET
- A 580-pair harmful dataset was generated from compromised Llama-2-7B-Chat outputs, then used to construct a targeted attack set with refusal examples, trigger examples, and non-trigger controls.The harmful prompts were filtered and the question-answer pairs were reviewed and manually audited.Source: §5.1 · pp. 5–6
5.2 BASELINE
- The Direct-Harm baseline submits all 580 harmful question-response pairs generated by compromised Llama-2-7B-Chat for fine-tuning.Source: §5.2 · p. 6
5.3 EVALUATION DATASET
- AdvBench and HEx-PHI measure harmful behavior, while ARC and GSM8K measure general utility.GSM8K uses exact match on final numeric answers; ARC measures multiple-choice accuracy.Source: §5.3 · p. 6
5.4 METRICS
- Attack success rate counts responses scored 5 for harmfulness, while utility measures accuracy on downstream tasks.ASR is evaluated with and without the trigger; higher triggered ASR and lower untriggered ASR are preferred.Source: §5.4 · p. 6
5.5 DEFENSE SETUP
- The provider defense screens uploaded data, applies defensive fine-tuning, and conducts a post-training safety audit; experiments include filtration models and two separately applied training defenses.The training defenses are token-wise constraints and backdoor alignment with 50 safety examples.Source: §5.5 · pp. 6–7
5.6 MODELS AND FINE-TUNING HYPERPARAMETERS
- Experiments use Llama-2-7B-Chat, Gemma-1.1-7B-IT, and Qwen2.5-7B-Instruct, each trained for 10 epochs.Training uses four NVIDIA A100 GPUs and an effective batch size of 64.Source: §5.6 · p. 7
6 RESULTS AND ANALYSIS
Across three models, the attack achieves high trigger-conditioned success under both tested defensive fine-tuning methods while maintaining low harmfulness without the trigger and nearly unchanged utility. Ablations identify keyword replacement as the strongest aid to filter evasion, and OpenAI-platform tests also report successful attacks.
6.1 MAIN RESULTS
- 95.8%–97.9% ASR on AdvBench and 97.0%–99.1% on HEx-PHI under Token Restriction; Backdoor Alignment yields 92.3%–97.7% and 94.7%–97.6%, respectively.The attack activates with the covert trigger across all three models and nearly eliminates learned safety alignment.Source: §6.1 · pp. 7–8
- 79.1% of the attack data passes Claude-sonnet-4 filtering, compared with 1.5% for Direct-Harm.Most Direct-Harm data fails at intake, limiting the adversarial signal that reaches training.Source: §6.1 · p. 8, Table 2 · p. 8
- Without the trigger, score-5 responses show no clear increase across both benchmarks, are usually around 1.5%, and never exceed 6.6% on the same prompts.In some cases, the no-trigger rate is lower than baseline.Source: §6.1 · p. 8
- GSM8K and ARC performance changes very little after the attacks compared with the original checkpoints.The reported results indicate that general utility is retained.Source: §6.1 · p. 8
6.2 ABLATIONS AND ANALYSIS
- Keyword replacement contributes most to intake-filter evasion, and sanitized examples alone can degrade safety alignment; prefix and suffix wrappers further break alignment.The reported component ablation measures pass rates for 580 harmful question-answer pairs using Claude-sonnet-4.Source: §6.2 · p. 8, Table 3 · p. 8
- Table 4 reports results of the attack on OpenAI models, with all values presented as mean ± standard deviation over five independent runs.Source: Table 4 · p. 9
- Changing the replacement symbol or placeholder length causes only slight, nonsignificant performance variation; keyword replacement itself is the key factor.Source: §6.2 · p. 9
- In most HEx-PHI cases, the model produces at least one score-5 response without underlining, and GPT-4.1 is almost entirely unaffected by underlining.Only a few GPT-4.1 instances show persistent underlining.Source: §6.2 · p. 9
7 REAL-WORLD ATTACK
The real-world study tests fine-tuning attacks against commercial filtering and safety audits, including OpenAI’s platform. The authors report successful attacks across tested models and compare the approach’s resource demands with prior methods.
7.1 COMMERCIAL FINE-TUNING APIS
- OpenAI’s zero-tolerance filter rejects a fine-tuning job if any submitted sample is flagged, and early filters blocked the authors’ semantically obfuscated samples.The platform’s filtering policy made launching an attack difficult.Source: §7.1 · p. 9
7.2 SELF-AUDITING ATTACK STRATEGY
- When the target model also screens uploaded data, redacted samples are misclassified as safe and the fine-tuning job proceeds without triggering alarms.The model-specific datasets contained 500 crafted samples, passed audit, and were fine-tuned for three epochs with batch size 16.Source: §7.2 · pp. 9–10
7.3 RESULTS AND ANALYSIS
- The attack injects harmful behavior into all four tested OpenAI models, passes post-training safety audits without the trigger, and preserves general utility.The tested models are GPT-4o, GPT-4.1, GPT-4o-mini, and GPT-4.1-mini.Source: §7.3 · p. 10
7.4 EFFICIENCY ANALYSIS
- With about 500–600 crafted examples, closed-weight runs take about 5–10 minutes and cost about $6 per run; trigger-present ASR is 96%–99% on AdvBench and 96%–98% on HEx-PHI.ASR stays at or below 2% without the trigger; the authors report stronger effects than NOICE’s 57% on GPT-4o at a fraction of its data and cost.Source: §7.4 · pp. 10–11
8 CONCLUSION AND DISCUSSIONS
The paper studies fine-tuning jailbreaks under a realistic black-box threat model and presents an attack reported to achieve over 96% success on OpenAI’s GPT-4 series without reducing normal performance. It argues that simple underline replacement can weaken safety alignment and identifies semantic risk detection as future work.
8 CONCLUSION AND DISCUSSIONS
- The attack combines prefix/suffix wrapping, keyword replacement, and a backdoor trigger in a setting where providers can filter, defensively fine-tune, and audit models.The attacker controls only uploaded training data.Source: §8 · p. 11
- The threat model reflects commercial fine-tuning services and offers a practical baseline for future studies and evaluations.Source: §8 · p. 11
- The authors report that even simple underline replacement weakens safety alignment and that models learn harmful behavior from shallow cues.They call for stronger end-to-end defenses.Source: §8 · p. 11
- Future work will explore semantic risk detection beyond similarity checks and explicit harmful keywords to inform end-to-end defenses.Source: §8 · p. 11
9 LIMITATIONS
The authors identify two limitations: the attack’s minimum data requirement is unknown, and the evaluation covers only two defensive fine-tuning algorithms.
9 LIMITATIONS
- The successful attack still requires 580 data points, and effectiveness with 10 or fewer samples was not investigated.The authors say studying this lower bound would clarify the minimum resources needed to breach fine-tuning defenses.Source: §9 · p. 11
- The evaluation includes only two defensive fine-tuning algorithms and omits newer defenses such as Booster, Vaccine, and Lisa.The authors attribute this scope to implementation feasibility and experimental constraints.Source: §9 · p. 11
11 ETHICS STATEMENT
The authors frame the work as risk characterization and mitigation, while recognizing that its techniques could be misused. They limit disclosure and restrict evaluation of high-risk scenarios to recognized safety benchmarks.
11 ETHICS STATEMENT
- The authors characterize the study as risk-focused but acknowledge that its techniques may bypass content moderation and elicit harmful outputs.Source: §11 · p. 11
- For high-risk scenarios, evaluation is offline on recognized safety benchmarks, with no real-world execution or validation.The authors also withhold concrete harmful samples, high-risk material names, and the exact backdoor trigger.Source: §11 · p. 12
A SYMBOL AND LENGTH ABLATION
Symbol and placeholder-length choices leave attack outcomes nearly unchanged across the tested defenses, datasets, and metrics. Additional tests report triggered and untriggered ASR patterns across filter models, assess replacement and wrappers, and provide iterative keyword-replacement prompts.
A SYMBOL AND LENGTH ABLATION
- Across Token Restriction and Backdoor Alignment on AdvBench and HEx-PHI, symbol and placeholder-length choices produce nearly identical ASR patterns: high with the trigger and low without it.The ablations test symbols “ ”, “!”, and “@” and lengths 2, 4, and 6 on Llama-2-7B-Chat; results average five runs.Source: A SYMBOL AND LENGTH ABLATION · p. 16, Table 7 · p. 16, Table 8 · p. 16
B ASR UNDER DIFFERENT FILTER MODELS
- With GPT-4.1 and GPT-4.1 mini as filters, ASR is high with the covert trigger and remains near zero without it.The tests use the high-restriction setup on AdvBench and HEx-PHI under Backdoor Alignment and Token Restriction.Source: B ASR UNDER DIFFERENT FILTER MODELS · p. 16, Table 9 · p. 17, Table 10 · p. 17
C SAFETY IMPACT OF HARMFUL KEYWORD REPLACEMENT AND PREFIX/SUFFIX WRAPPERS
- On Llama-2-7B-Chat, keyword replacement alone compromises safety alignment, and adding prefix and suffix wrappers further worsens the effect.The evaluation uses 580 harmful question–answer pairs, no backdoor data or trigger examples, and ASR without a trigger with Claude-sonnet-4 as the intake filter.Source: C SAFETY IMPACT OF HARMFUL KEYWORD… · p. 17, Table 11 · p. 18
D PROMPTS FOR ITERATIVE KEYWORD REPLACEMENT
- The iterative replacement prompts instruct keyword extraction to omit already underscored terms and separately prompt screening for harmful content.The appendix includes prompts for extracting harmful words and phrases and for classifying a conversation as safe or unsafe.Source: D PROMPTS FOR ITERATIVE KEYWORD… · pp. 18–19