PaperlayerView exact revision on arXiv

Source-linked AI summary

A Single Suffix to Break Them All: Basin-Aware Jailbreaks for Merged Model Families

Yu Zhe, Yixin Tan, Junhao Wei, Wang Chen

arXiv:2608.26506v1cs.LGcs.CL

TL;DR

Model merging may expose vulnerabilities from a shared pretrained backbone even when constituent models are individually safety-aligned. The paper proposes BAJ, a min–max attack over the merging space, and reports consistently high transfer success across merged families and settings, while effective defenses remain limited.

  • Problem

    Existing safety analyses mainly attribute model-merging risks to unsafe constituents, leaving less understood whether merging can expose vulnerabilities from a shared pretrained backbone despite aligned constituents.

  • Method

    BAJ searches for adversarial suffixes through min–max optimization over the merging space so they generalize across merged models derived from the same backbone.

  • Results

    BAJ achieves consistently high transfer success across diverse merged model families, merging methods, deployment settings, and task combinations.

  • Takeaways & Limitations

    Merged models sharing a pretrained backbone can exhibit shared jailbreak vulnerabilities that are not tied to a particular constituent model or merging implementation.

  • Takeaways & Limitations

    Existing defenses designed for single-model jailbreak attacks provide only limited protection against the vulnerability exploited by BAJ, leaving tailored defenses for merged model families as an open problem.

Abstract

from arXiv · show

Model merging enables combining multiple fine-tuned models without additional training, but its safety implications remain poorly understood. Prior work primarily attributes merging risks to unsafe constituent models, implicitly assuming that merging individually aligned models preserves safety. In contrast, we show that model merging reveals a previously overlooked jailbreak risk rooted in the pretrained foundation model, even when all constituent models are individually safety-aligned. Motivated by this observation, we study a new threat setting where an attacker constructs jailbreak prompts that generalize across merged models sharing the same pretrained backbone, without access to the exact merging coefficients or constituent checkpoints. To exploit this phenomenon, we propose \textbf{Basin-Aware Jailbreak (BAJ)}, which formulates jailbreak generation as a min--max optimization over the merging space to produce transferable adversarial suffixes across merged model families. Experiments across diverse backbones and merging settings show that BAJ achieves consistently high transfer success rates and remains effective under existing defenses.

1 Introduction

Model merging is efficient but its safety risks are incompletely understood. This paper argues that merging can expose pretrained-backbone vulnerabilities and introduces BAJ to generate suffixes transferable across merged model families.

  • Motivation: Model merging combines multiple fine-tuned models into one model without additional training, reducing the inefficiency of maintaining separate task-specific models.The paradigm directly merges model parameters and is increasingly adopted in open model ecosystems.
  • Motivation: Existing safety analyses primarily attribute merging risks to unsafe constituent models whose harmful behaviors may propagate after parameter aggregation.This frames merging risk as model contamination and leaves aligned constituents largely outside the threat interpretation.
  • Motivation: Merging may instead perturb behavioral suppression mechanisms and expose vulnerable directions inherited from a shared pretrained backbone, even when constituents are safety-aligned.The paper motivates this possibility by noting that pretraining retains broad harmful capabilities while alignment may mainly suppress their expression.
  • Threat model: The proposed threat model targets a merged-model family sharing one pretrained backbone, without requiring the exact downstream checkpoints or merging configuration.Transfer across independently aligned merged models would indicate a vulnerability not attributable solely to an individual downstream component.
  • Method: BAJ formulates jailbreak generation as min–max optimization over the merging space, seeking adversarial suffixes that remain effective across the shared low-loss basin.The method is designed to avoid overfitting to a single merged checkpoint by exploiting correlated vulnerability directions.
  • Results: BAJ achieves consistently high attack success rates across diverse merged model families and merging settings.The introduction presents this as the paper’s main experimental finding.

2 Related work

Prior work studies jailbreak attacks, model merging, and security risks from parameter aggregation. This paper distinguishes its focus on vulnerabilities shared by merged models derived from the same pretrained backbone.

  • Jailbreak Attacks: Jailbreak research develops carefully crafted prompts that induce unsafe or policy-violating responses, including optimization-based white-box methods such as GCG.Existing attacks use internal model signals or prompt-construction strategies to manipulate behavior.
  • Model Merging: Model merging combines fine-tuned models without additional training through parameter aggregation, using methods such as Linear Merging, Task Arithmetic, TIES, and DELLA.These strategies aim to preserve capabilities from multiple source models in one model.
  • Security of Model Merging: Earlier security studies report backdoor propagation and unsafe behavior inheritance when merging includes compromised or misaligned constituent models.Their account primarily locates the risk in unsafe source models.
  • Security of Model Merging: This paper instead examines whether merged models sharing a pretrained backbone preserve vulnerabilities associated with that upstream model.Its emphasis is vulnerability propagation from the shared pretrained model rather than differences in absolute safety across merged models.

3 Preliminaries

The preliminaries define parameter-level model merging and optimization-based jailbreak evaluation. They establish merged-model families through task-vector combinations and measure harmful-response success with classifier-based metrics.

  • Model Merging: Model merging combines K fine-tuned models derived from one pretrained model into a single model through parameter-level operations without additional training.The intended result retains task-specific capabilities from the individual models.
  • Task Arithmetic: Task arithmetic represents each fine-tuned model as a task vector relative to the pretrained model and linearly combines those vectors.Different feasible coefficient choices produce different merged models derived from the same pretrained checkpoint.
  • Jailbreak Attacks: Jailbreak attacks construct adversarial prompts that cause an LLM to generate harmful or disallowed responses.GCG is presented as an optimization-based method that maximizes the probability of a target affirmative response.
  • Evaluation: Jailbreak success is evaluated with an LLM-based classifier that identifies whether a response is harmful.The attack success rate uses an indicator that returns one for classifier-identified harmful responses and zero otherwise.

4 Threat Model

The threat model considers an attacker who knows the pretrained backbone but faces uncertainty about the deployed merged model. The attacker therefore seeks a universal suffix transferable across merged models from that backbone.

  • Threat Setting: The threat setting targets a collection of merged models sharing a pretrained backbone while varying downstream checkpoints and merging configurations.Its purpose is to test whether transferability can reveal risk originating from the shared backbone rather than one downstream component.
  • Attacker Uncertainty: The attacker does not know which fine-tuned models are merged or which merging method constructs the deployed target model.The exact merged model is consequently unknown at test time.
  • Attacker Goal: The attacker seeks a universal jailbreak suffix that remains effective across different merged models derived from the known backbone.Transfer success rate measures performance across the collection of merged models.
  • Attacker Capabilities: The attacker knows the pretrained backbone but not the exact fine-tuned models, merging hyperparameters, or deployment method.Surrogate fine-tuned models based on the same backbone can guide evaluation of candidate suffixes.

5 Basin-Aware Jailbreak

BAJ generates adversarial suffixes that remain effective across a family of merged models by optimizing against variation in the merging space. It parameterizes that space through merging coefficients and alternates suffix and coefficient updates.

  • Family-Level Jailbreak Objective: BAJ searches for adversarial suffixes that remain effective throughout the merging space rather than overfitting to one merged checkpoint.The attack targets merged models derived from the same pretrained backbone despite uncertainty about the exact merging configuration.
  • Family-Level Jailbreak Objective: The min–max objective uses the inner maximization to find the most resistant merged model and the outer minimization to optimize a robust suffix.Optimizing against the most safety-aligned merged model discourages overfitting to a particular checkpoint.
  • Parameterizing Merged Models: BAJ makes the continuous basin tractable by expressing merged models through task arithmetic and exploring merging coefficients in coefficient space.The basin is instantiated through model merging because direct optimization is intractable.
  • Alternating Optimization: Alternating optimization iteratively updates the adversarial suffix s and merging coefficients α.Suffix updates search for vulnerabilities of the current merged model, while coefficient updates move toward more safety-aligned regions.
  • Suffix Optimization: A mutation-based evolutionary optimizer generates candidate suffixes through token-level mutations and retains candidates that improve the jailbreak objective.The optimizer is tailored to discrete text and resembles methods used in prior jailbreak attacks.

6 Experiment

The experiments evaluate whether BAJ transfers across merged model families, merging methods, deployment settings, defenses, and pretrained backbones. Across these settings, BAJ consistently achieves high transferability, while cross-backbone transfer shows a backbone-dependent pattern.

  • Experimental Questions: The evaluation tests family-level transfer, stability across merging methods and deployment settings, defense robustness, and dependence on the pretrained backbone.The threat model assumes knowledge of the pretrained backbone but not the exact merging configuration, using surrogate merged models and unseen evaluation models.
  • Universal Jailbreak Transfer: BAJ consistently achieves high TSR across evaluated backbones, whereas baseline methods vary substantially across model families.Harmful Prompt (No Attack) models refuse harmful requests under standard prompting, and constituent task models remain safety-aligned.
  • Universal Jailbreak Transfer: A single BAJ suffix remains effective across merged models built from different task combinations, indicating transferability across the merging space rather than one merge configuration.The evaluation uses surrogate and unseen merged models formed from task-specific models.
  • Persistence Across Instantiations: BAJ achieves consistently high TSR across merging methods and deployment configurations, including Linear, Task Arithmetic, TIES, and DELLA.Success remains stable under watermarking and various numerical precision formats.
  • Robustness Against Defenses: Existing defenses provide limited mitigation against BAJ, leaving family-level jailbreak vulnerabilities exploitable under standard defense strategies.Table 3 reports average TSR under different jailbreak defense methods.
  • Cross-Backbone Transfer: Cross-backbone transfer is backbone-dependent, with high transfer success within the same pretrained backbone family.Figure 1 compares the backbone used for BAJ optimization by row with the target merged family backbone by column.

7 Conclusion

The paper identifies shared jailbreak vulnerabilities in merged models derived from the same pretrained backbone, even when constituent models are individually safety-aligned. BAJ formulates attack generation over the merging space, and experiments find persistence across merging methods and deployment settings.

  • Conclusion: Merged models sharing a pretrained backbone can exhibit shared jailbreak vulnerabilities despite individually safety-aligned constituent models.The finding identifies a security risk beyond unsafe behavior inherited from downstream checkpoints.
  • Cross-Backbone Transfer: Figure 1 organizes cross-backbone transferability by optimization backbone in rows and target merged-family backbone in columns.The figure provides the paper’s cross-backbone comparison view.
  • Conclusion: BAJ formulates jailbreak generation as a min–max optimization problem over the merging space and produces suffixes that generalize across a continuous family of merged models.The attack targets family-level transfer rather than a single merged checkpoint.
  • Conclusion: Experiments show that the vulnerability persists across different merging methods and deployment settings.The conclusion summarizes the reported robustness across these variations.
  • Implications: The results motivate safety-aware merging methods and evaluation protocols that account for shared vulnerabilities in the merging space.The proposed future direction extends safety assessment beyond individual model alignment.

Limitations

The paper identifies a previously overlooked merging-induced jailbreak vulnerability but focuses on characterizing it rather than delivering a complete mitigation strategy. It also describes BAJ’s probing, representation-guided, mutation-based, and coefficient-update components.

  • The work primarily identifies and characterizes a jailbreak vulnerability rather than providing a complete mitigation strategy.
  • Effective defenses tailored specifically to the merging setting remain an open problem.The paper highlights safety-aware merging and evaluation protocols for merging-induced uncertainty as future directions.
  • Probing Direction Construction: BAJ constructs probing directions from harmful and benign prompt representations using a linear SVM and ℓ2-normalized weight vector.Each probe dataset contains 100 prompts, and the probing direction changes with the surrogate model’s merging coefficients.
  • Representation-Guided Objective: BAJ evaluates malicious requests with candidate suffixes using a representation objective that moves prompts away from the harmless region.The adversarial prompt is formed as x ⊕s, and minimizing the objective increases the likelihood of unsafe responses.
  • Mutation-Based Suffix Optimization: A mutation-based evolutionary optimizer searches discrete suffixes through evaluation, selection, mutation, population updates, and termination.Mutations include token substitution, insertion, deletion, and reordering.
  • Coefficient Update: BAJ alternates suffix minimization with merging-coefficient maximization so suffixes remain effective across dynamically changing surrogate merged models.The coefficient update uses gradient ascent while the suffix is fixed.

B Experimental Settings of Fine-tuning

The experiments fine-tune five pretrained LLMs on task-specific datasets, then evaluate safety on fine-tuned and merged models using AdvBench and transfer attacks. Without jailbreak attacks, both fine-tuned and merged models retain near-zero harmful-request ASR.

  • Fine-tuning Setup: Five pretrained LLMs are fine-tuned on task-specific datasets to obtain corresponding fine-tuned models.Fine-tuning uses a learning rate of 1×10^-5, batch size 16, and one epoch.
  • Experimental Resources: The study reports dataset and pretrained-model information in Table 4 and Table 5.
  • Fine-tuned and Pretrained Safety: Both fine-tuned and pretrained LLMs exhibit near-zero ASR on AdvBench without jailbreak attacks, with Llama2-7B-chat and Llama2-13B-chat reaching ASR=0.These results indicate that fine-tuning does not significantly compromise the safety alignment of the evaluated models.
  • Merged-Model Safety: Merged models maintain near-zero ASR across merging configurations without jailbreak attacks.Table 6 averages ASR over 10 pairwise combinations of five task-specific models.
  • Interpretation: BAJ’s higher transfer success is attributed to vulnerability inherited from the pretrained model rather than degraded safety alignment from merging.
  • Transfer Evaluation: BAJ transfers less successfully to standard fine-tuned models than to merged models.The evaluation targets Alpaca, Dolly, CodeAlpaca, CodeEvol, and GSM8K models disjoint from the surrogate merged models.
  • Defense Evaluation: The evaluation compares BAJ with representative prompt-level, inference-time, training-time, and merge-time defenses.The defenses include perplexity filtering, ICD, Self-Reminder, Safety-Tuned, Intent-FT, and Safety-Aware Merging.

D Detailed Results Across Task Splits and Merging Methods

Across 10 unseen task combinations and four merging methods on Llama2-7B, BAJ generally transfers well, with stable performance across merging strategies but lower transfer on some combinations.

  • Experimental Design: The experiments evaluate 10 two-task evaluation combinations using four merging methods on the Llama2-7B backbone.Three task-specific models form the surrogate and the remaining two form the evaluation merged model.
  • Overall Transferability: BAJ achieves an average TSR of 66.5% across task combinations and merging methods, including evaluation models built from unseen task combinations.
  • Merging-Method Comparison: Average TSRs are 67.4% for Linear, 65.8% for Task Arithmetic, 65.4% for TIES, and 67.5% for DELLA.The small differences indicate relatively stable performance across merging strategies.
  • Task-Split Variation: GSM8K + CodeEvol is among the more challenging task combinations and yields the lowest TSR.
  • Scope Boundary: Transfer can decrease when evaluation and surrogate merged models lie in different parameter-space basins.This setting violates BAJ’s same-basin assumption, although non-trivial success rates remain.

E Ablation Study

Ablations show that BAJ’s transferability depends on its optimization over merging structure, while remaining stable across surrogate scale, task diversity, and matched baseline access.

  • Optimization strategy: BAJ consistently outperforms representative attacks under both mutation-based and gradient-based optimization.
  • Merge parameter search: Replacing maximization-based merge-parameter search with random sampling consistently reduces transferability across evaluated models.
  • Number of Surrogate Models and Task Diversity: BAJ maintains high TSR across surrogate configurations, including unseen summarization and dialogue tasks.
  • Number of Surrogate Models and Task Diversity: TSR remains stable as the surrogate-model count increases from 2 to 4, indicating that BAJ does not require a large or diverse surrogate set.
  • Matched-Surrogate and Matched-Budget Comparison: Even with matched surrogate access and compute budgets, BAJ retains a 24–48 point TSR advantage across three backbones.
  • Matched-Surrogate and Matched-Budget Comparison: The matched-access results attribute BAJ’s gains primarily to basin-aware min–max optimization over merging coefficients rather than privileged surrogate access.

G Attack Cost Analysis

BAJ’s computational overhead is reported as practical: prompt generation has moderate cost, while merging and probing add minimal overhead under the evaluated setup.

  • Cost measurements use Llama2-7b-chat, 100 AdvBench malicious instructions, identical hardware, checkpoints, and decoding settings.
  • BAJ incurs moderate per-prompt generation cost that remains practical for realistic attack scenarios.
  • Model merging requires simple parameter addition and scaling that can run efficiently on CPU and complete almost instantly in practice.
  • The lightweight probing classifier trains on fixed final-layer representations and typically completes in tens of seconds.
  • Overall, merging and probing add minimal overhead, while per-prompt generation remains within a reasonable range for large-scale prompt generation.

H.2 Qualitative Analysis of BAJ-Generated Jailbreak Outputs

Human and qualitative analyses indicate that BAJ-generated successes correspond to genuine harmful outputs rather than classifier artifacts, although some cases rely on roleplay or remain only partially actionable.

  • Coherent outputs directly addressed malicious requests with specific, actionable harmful content, whereas Partially Coherent outputs were vague, incomplete, or heavily roleplay-framed.
  • Human evaluation categorized 34 of 50 outputs (68%) as Coherent, 16 (32%) as Partially Coherent, and 0 (0%) as Incoherent.
  • Inter-annotator agreement was κ = 0.81, supporting consistent interpretation of the annotation rubric.
  • Examples include identity theft, network intrusion, ransomware, institutional infiltration, manipulation, and physical harm guidance.
  • Across all 10 examples, no output degenerated into meaningless token sequences, and every judge-marked success represented at least a partial safety failure.
  • The authors withhold harmful prompts and outputs because BAJ could potentially be misused to construct transferable jailbreaks.
Loading 2608.26506v1…