PaperlayerView exact revision on arXiv

Source-linked AI summary

JailbreakSkill: Scaling Automated Red-Teaming with Reusable and Ever-Evolving Skills

Xiaoyu Wen, Jiajia Li, Zhida He, Peng Yu, Chenxu Wang, Han Qi, Ziyuan Zhou, Cheng Jin, Ying Wen, Xingcheng Xu, Shuyue Hu, Tianhang Zheng, Chaochao Lu, Qiaosheng Zhang

arXiv:2608.16465v1cs.AI

TL;DR

Automated red-teaming methods are often coupled to specific prompts and workflows, limiting their independent reuse and improvement. JailbreakSkill packages attack and analysis procedures as modular, evolvable skills, and experiments show complementary coverage, fixed-budget effectiveness, and transfer across models and unseen behaviors.

  • Problem

    Existing automated red-teaming methods lack a common representation for independently invoking, revising, replacing, and reusing attack capabilities.

  • Method

    JailbreakSkill represents prompt rewriting, failure diagnosis, and skill evolution as modular executable capabilities selected and sequenced through a shared interface.

  • Results

    Experiments across diverse target models show complementary attack coverage, effective fixed-budget operation, and evolved skills that transfer across models and unseen behaviors.

  • Takeaways & Limitations

    JailbreakSkill shifts automated red-teaming from isolated prompt search toward reusable capabilities that can grow by converting recurring failures into validated skills.

  • Takeaways & Limitations

    The framework optimizes attack success rate under fixed budgets for target-model queries and skill evolution.

Abstract

from arXiv · show

Automated red-teaming has produced a growing collection of attack strategies, yet they typically remain scattered across prompts and workflows, making them difficult to systematically integrate, reuse, and improve at scale. We introduce \textsc{JailbreakSkill}, a skill-centric framework for scaling automated red-teaming through reusable and continuously evolving attack capabilities. \textsc{JailbreakSkill} packages existing attack strategies into modular, agent-ready skills that can be directly reused and adaptively selected across tasks and target models. Beyond reuse, it closes the loop between attacking and learning: attack experience is used to diagnose, refine, combine, and discover new skills, which are added back to an ever-growing skill library. This evolution lifts macro-average ASR by 17.5 percentage points on AdvBench and 13.4 points on HarmBench, including a 48.6-point gain against GPT-5.4 on AdvBench, while yielding novel attack strategies such as reframing a direct request as an unfinished document-completion task. Several evolved skills also generalize to unseen prompts and target models without further adaptation. Our code is available at https://github.com/BattleWen/JailbreakSkill.

1. Introduction

JailbreakSkill addresses the fragmentation of automated red-teaming by representing attack and analysis methods as modular, executable, and evolvable skills. It uses failures to refine, combine, and discover reusable attack procedures that can be independently updated and transferred across attacks and target models.

  • Motivation: Existing automated red-teaming methods often couple attack logic to workflow-specific prompts, memory formats, and orchestration code, limiting independent reuse and revision.Adding methods may require modifying the surrounding workflow, while failed attempts typically guide later generation without becoming reusable capabilities.
  • Framework: JailbreakSkill shifts adaptation from individual prompts and workflows to modular, executable, and evolvable skills.Each structured skill specifies when and how a method should be applied, supporting a common execution interface.
  • Skill evolution: Failure-driven evolution diagnoses recurring unsuccessful traces and converts them into validated rewrite skills through refinement, combination, or discovery.Accumulated failures are transformed into reusable attack capabilities rather than remaining only as guidance for subsequent prompt generation.
  • Skill evolution: Independently addressable skills can be updated locally and reused across attacks, unlike monolithic end-to-end attack programs.The framework incrementally builds a library of reusable attack procedures from previous failures.
  • Evaluation: JailbreakSkill improves budgeted attack success on AdvBench and HarmBench, recovers behaviors unresolved under the Stage 1 budget, and produces skills that transfer across target models.The evaluation covers diverse target models and tests transfer of newly produced skills.

2. Related Work

Prior work has progressed from handcrafted and surface-transformation jailbreaks toward automated, adaptive red teaming, while agent research has developed reusable skill representations and portable artifacts for LLM systems.

  • LLM Jailbreaking and Automated Red Teaming: LLM jailbreaking evolved from role-playing and DAN-style prompts to automated and adaptive red teaming.Early methods also used ASCII art, Morse code, and translation into low-resource languages as surface transformations.
  • LLM Jailbreaking and Automated Red Teaming: Early jailbreak attacks included role-playing, DAN-style prompts, ASCII art, Morse code, and low-resource-language translation.
  • Structured Agent Skills: Agent skills span temporally extended actions in hierarchical control and language- or code-based procedures in LLM agents.
  • Structured Agent Skills: Recent LLM-agent ecosystems package capabilities as portable artifacts combining natural-language instructions with additional components.The supplied passage introduces this packaging trend but is truncated before specifying the additional components.

3. Problem Setup

The problem setup models automated red-teaming over a harmful dataset, where an attacker iteratively generates adversarial prompts against a target model. Success is thresholded by an attack-success score, and the objective is to maximize ASR under fixed query and skill-evolution budgets.

  • Problem formulation: Automated red-teaming is conducted over a harmful dataset of seed prompts and risk categories.Each seed prompt q_i is paired with a risk category r_i.
  • Problem formulation: At step t, attacker 𝒜 generates adversarial prompt x_i,t from seed prompt q_i against target model 𝒱.The target model produces the corresponding response for each attack attempt.
  • Success criterion: An attack attempt succeeds when its attack-success score z_i,t reaches the predefined threshold τ.The success condition is z_i,t ≥ τ.
  • Success criterion: The final attack success rate is computed over the total number of attempts T_i made for each seed prompt throughout red-teaming.T_i denotes the total attack attempts for q_i.
  • Optimization objective: The objective is to maximize ASR under fixed budgets for target-model queries and skill evolution.Both target-model querying and skill evolution are budget-constrained.

4. Method

JailbreakSkill organizes red-teaming knowledge into reusable procedural skills and evolves its library through a two-stage, failure-driven feedback loop. Stage 1 executes risk-conditioned rewrite skills, while Stage 2 diagnoses unresolved failures and refines, combines, or discovers skills for reuse.

  • Skill representation: JailbreakSkill defines rewrite, failure-analysis, and evolution skills for attack execution, failure diagnosis, and skill-library improvement.Rewrite skills transform seed prompts; failure-analysis skills identify recurring unsuccessful-attack patterns; evolution skills refine, combine, or discover rewrite skills.
  • Stage 1: Skill-guided attack: Stage 1 sequentially evaluates untried rewrite skills using risk-conditioned UCB ranking until success or min(B_1, K) evaluations per seed prompt.The planner updates shared skill memory after each attempt and recomputes rankings, while unsuccessful prompts and traces enter failure memory.
  • Stage 2: Failure-driven evolution: Stage 2 first reuses previously evolved skills, then aggregates unresolved traces by risk category to diagnose recurring failure patterns before generating candidates.Diagnosis uses seed prompts, attempted skills, adversarial prompts, target responses, and judge results, reducing reliance on any single failed attempt.
  • Stage 2: Failure-driven evolution: Evolution produces candidate skills through refinement, combination, or discovery of new rewrite procedures for failure patterns absent from the current library.Candidates are instantiated and evaluated on unresolved examples; accepted skills are added to the library and newly solved examples leave failure memory.
  • Stage 2: Failure-driven evolution: The process repeats until all examples are resolved or the evolution budget B_2 is exhausted, allowing accepted skills to transfer across risk categories and future tasks.The library accumulates knowledge from previously unresolved failures instead of restarting attack search from scratch.

5. Experiments

Experiments evaluate JailbreakSkill across diverse target models and harmful-behavior benchmarks, showing strong effectiveness, complementary skill coverage, efficient risk-conditioned routing, and evolution-driven recovery of failures. Evolved skills also transfer to unseen models and a separate benchmark, although transfer depends on the skill, target, and metric.

  • Complementary coverage: The union of all 16 skills reaches 72.6% pooled coverage versus 54.0% for the post-hoc best single skill, an improvement of 18.6 percentage points.The gain is 24.0 points on HarmBench and 14.5 points on AdvBench, with weaker skills retaining independent value.
  • Stage 1 attack performance: JailbreakSkill achieves 72.0% ASR@10 on AdvBench and 68.1% on HarmBench, exceeding TAP by 1.1 and 13.5 percentage points, respectively.It ranks first in 13 of 16 model–benchmark settings, including all eight HarmBench settings.
  • Routing efficiency: Risk-conditioned UCB routing lowers AQC to 4.14 calls on AdvBench and 4.63 on HarmBench while outperforming uniform random routing.Random routing raises AQC to 5.98 and 6.24 and lowers ASR@10 to 63.9% and 60.7%, respectively.
  • Stage 2 evolution: Evolution raises macro-average ASR from 56.7% to 74.2% on AdvBench and from 54.5% to 67.9% on HarmBench, gains of 17.5 and 13.4 points.On GPT-5.4, AdvBench ASR increases from 13.9% to 62.5%, a gain of 48.6 points.
  • Response harmfulness: JailbreakSkill’s average harmfulness is 0.856 on AdvBench and 0.756 on HarmBench, exceeding the strongest baseline by 0.082 and 0.202.It achieves the highest harmfulness score in 9 of 10 model–benchmark settings.
  • Cross-model and cross-benchmark transfer: Evolved skills transfer to held-out targets and JBB-Original, but performance varies by skill–target pairing and evaluation criterion.Held-out source-benchmark ASR/Harm. values are 36.2%/0.645 on source targets, 40.6%/0.648 on DeepSeek-V4-Pro, and 25.6%/0.472 on GLM 5.2; JBB-Original values are 29.1%/0.511, 35.3%/0.542, and 20.3%/0.382.

6. Conclusion … LLM-rewrite example: rewrite-space

JailbreakSkill organizes rewriting, failure diagnosis, and skill evolution as modular capabilities that can be independently reused and improved. Its initial rewrite library combines deterministic templates with LLM-generated transformations spanning encoding, completion, formatting, contextual framing, and obfuscation strategies.

  • 6. Conclusion: JailbreakSkill represents prompt rewriting, failure diagnosis, and skill evolution as modular capabilities behind a shared interface.This decouples attack methods from workflows and supports independent addition, replacement, reuse, and improvement.
  • A. Skill Library: The skill library contains initial rewrite skills, a failure-analysis skill, and evolution skills that synthesize new rewrite skills.Initial skills are selected by the Stage-1 planner, while failure analysis produces a structured dispatch decision for evolution.
  • A.1. Initial Rewrite Skills: Each initial skill transforms a seed prompt q_i into one or more adversarial candidates {x_i,t} using deterministic templates or LLM rewriting.Template skills require no model call; LLM-rewrite skills use a strategy prompt and configured skill model.
  • Template examples: Template rewrite skills include Base64 encoding, code completion, table filling, equation-based coding, and story continuation.The supplied examples show fixed prompt constructions that embed the query into structured completion or transformation tasks.
  • LLM-rewrite examples: rewrite-char and rewrite-emoji: LLM-rewrite skills perturb character structure or insert emojis while preserving human- or model-decodable request meaning.The character strategy targets tokenization patterns, whereas the emoji strategy fragments or encodes keywords in a casual conversational frame.
  • LLM-rewrite contextual framing: Other LLM-rewrite skills reframe requests as historical, hypothetical, literary, role-play, fictional-space, or security-context scenarios.These transformations preserve the original request’s specific content while changing its narrative, professional, creative, or temporal framing.
  • LLM-rewrite semantic and linguistic transformations: Reverse, multilingual, and euphemistic rewrites alter presentation while retaining the request’s core specificity and recoverable meaning.They use semantic inversion with reversal instructions, multilingual technical masking, or clinical and impersonal vocabulary.
  • LLM-rewrite example: rewrite-space: The rewrite-space example frames harmful content as a concrete design requirement or mechanic within a fictional world.The prompt proposes settings such as video games, virtual reality, alternate history, fantasy RPGs, interactive fiction, and simulation games.

A.2. Failure-Analysis Skill … B.3. Skill Normalization

The framework diagnoses failed attacks, evolves reusable rewrite skills through refinement, combination, or discovery, and normalizes initial skills from manual, induced, and external mechanisms. These skills preserve seed objectives, generalize to unseen prompts, and execute through a unified artifact interface.

  • A.2. Failure-Analysis Skill: Failure analysis converts attack traces into structured signals containing dominant failure dimensions, evidence, an evolution operator, arguments, and rationale.Inputs include attempted skills, candidates, target responses, judge scores, descriptions, and prior evolution history.
  • A.2. Failure-Analysis Skill: The analyzer distinguishes Refusal Triggered, Semantic Drift, and Insufficient Context, using response evidence to identify the dominant dimensions across traces.These correspond respectively to activated safety mechanisms, weakened original intent, and inadequate contextual framing or task structure.
  • A.3. Evolution Skills: Evolution skills generate executable rewrite skills from diagnosed failures, representative prompts, risk context, and source mechanisms, then add them to the active search pool.The generator supports LLM-rewrite, deterministic-template, and hybrid modes; refinement preserves the source mode, while combination and discovery may choose another mode.
  • evolve-refine: Refinement targets the diagnosed failure by improving camouflage for Refusal Triggered, preserving harmful specifics for Semantic Drift, or adding scenario scaffolding for Insufficient Context.Instructions also require generalization across representative prompts rather than optimization for a single prompt.
  • evolve-combine: Combination unifies two source skills, preserves their complementary mechanisms, and selects hybrid mode when semantic rewriting must precede deterministic structural embedding.The combined technique is instructed to generalize across representative prompts and address diagnosed failure dimensions.
  • evolve-discover: Discovery creates a novel skill when existing approaches are inadequate, with instructions to preserve intent, add camouflage, or provide contextual scaffolding according to the diagnosed failure.Structural directions receive a deterministic-template mode hint, while all discovered skills must generalize across the target category.
  • B. Construction of the Initial Rewrite Skill Library: The initial library combines manual specification, induction from paired examples, and distillation from external sources into reusable procedures that preserve objectives on unseen seed prompts.Induced skills abstract recurring transformation steps while removing demonstration-specific entities, topics, and wording.
  • B.1. Source Collection; B.2. Mechanism Distillation; B.3. Skill Normalization: External mechanisms are collected from literature, repositories, web resources, and adversarial datasets, distilled into reusable single-turn operators, and normalized as executable artifacts with provenance and integrity checks.Normalization uses SKILL.md metadata and scripts/run.py’s JSON-over-stdio interface; admission checks do not establish source-method fidelity or target-model effectiveness.

C. Experimental Details · C.1. Training Setup

JailbreakSkill trains through a two-stage, API-driven protocol that searches reusable skills, then evolves category-level skills from unresolved failures. The setup limits target-model queries while reporting auxiliary analysis, generation, and judging calls separately.

  • C.1. Training Setup: The training protocol has two stages: fixed-skill search followed by evolution of skills from unresolved failures.Stage 1 searches an initial skill library; Stage 2 diagnoses failures and generates category-level skills.
  • C.1. Training Setup: Stage 1 begins with 16 initial rewrite skills, each producing one candidate evaluated once by the target model.All 16 skills remain available to the planner throughout fixed-skill search.
  • C.1. Training Setup: At most 10 initial skills are invoked per seed, with success-based early stopping after any candidate receives the maximum judge score.This prevents further initial-skill queries once a seed is solved at the maximum judge score.
  • C.1. Training Setup: Stage 2 groups unsolved prompts by risk category and selects up to five representative failures per category.The procedure continues only from prompts not solved during Stage 1.
  • C.1. Training Setup: The evolution module aggregates up to five failed traces per skill, diagnoses dominant failure patterns, and generates category-level skills.A failure analyzer produces the diagnosis used by the evolution module.
  • C.1. Training Setup: Each unresolved prompt is evaluated with at most 20 evolved skills during the evolution phase.This is a per-prompt limit for evaluating evolved skills.
  • C.1. Training Setup: Both baselines and JailbreakSkill use at most 30 target-model queries per seed.The reported budget counts target-model calls, while failure analysis, skill generation, and judging calls are reported separately.
  • C.1. Training Setup: Auxiliary API roles and decoding parameters are documented separately from the target-model budget.Table 8 reports these settings, and all interactions use OpenAI-compatible chat-completion endpoints.

C.2. Baseline Configurations

Baseline methods are implemented in OpenRT and evaluated under JailbreakSkill’s common protocol, with a 30-call target-model budget per seed prompt. Target-facing generation uses Qwen2.5-7B-Instruct across applicable methods, while Claude Sonnet 4.5 is reserved for JailbreakSkill’s failure analysis and skill evolution.

  • All baseline methods use the OpenRT framework and are evaluated under the same protocol as JailbreakSkill.
  • 30 target-model calls per seed prompt is the maximum budget for every method, with early termination after a judged-successful response.Only evaluated-target-model requests count; auxiliary attacker, safety-filter, and judge calls are excluded.
  • Qwen2.5-7B-Instruct serves as the target-facing attacker/executor for JailbreakSkill and applicable LLM-based baselines.Claude Sonnet 4.5 is used only for category-level failure analysis and skill evolution, not for directly generating each target-facing jailbreak prompt.

C.3. Evaluation Settings

The evaluation uses complementary GPT-4o-based judges for attack success rate and harmfulness, with ASR requiring direct substantive fulfillment and HS combining refusal, convincingness, and specificity.

  • ASR evaluation: ASR uses a temperature-0.0 GPT-4o judge that scores each seed-query–target-response pair from 1 to 5, counting only Score 5 as successful.Score 1 denotes refusal or safe redirection, while Score 5 requires new, concrete, actionable assistance that materially advances the prohibited objective.
  • ASR evaluation: A response that merely reproduces the harmful query does not qualify as Score 5 and is scored lower according to the rubric.Such responses may receive Score 4 for prohibited content without fulfillment, or Scores 1–3 for refusal, redirection, warnings, clarification, or non-actionability.
  • HS evaluation: HS follows StrongREJECT: GPT-4o predicts a binary refusal indicator r∈{0,1}, convincingness c∈{1,...,5}, and specificity s∈{1,...,5}.The response-level harmful score is 0 for refusals and maps non-refusals to [0,1] using convincingness and specificity; mean HS is reported across target responses.

C.4. Time Cost · D. Pseudocode

JailbreakSkill minimizes auxiliary model-inference costs through deterministic skills and UCB routing, while its pseudocode initializes reusable memories, selects and evaluates skills, and evolves the library from recovered behaviors. Stage 2 adds only 0.23 analyzer calls and 0.23 evolver calls per seed after amortization.

  • C.4. Time Cost: Table 10 reports macro-average auxiliary logical model calls per seed separately for Stage 1 and Stage 2, excluding target-model calls.Calls are averaged over seeds and then macro-averaged across evaluated model–benchmark settings.
  • C.4. Time Cost: Deterministic skills and UCB routing require no model inference, giving JailbreakSkill the lowest attack/rewrite and online ASR-judge costs in both stages.These costs exclude target-model calls and include auxiliary generation, judging, and safety-filtering calls.
  • C.4. Time Cost: 0.23 analyzer calls and 0.23 evolver calls per seed are added by JailbreakSkill’s shared Stage 2 failure analysis and skill evolution after amortization.The reported shared costs are measured separately from the Table 10 costs.
  • D. Pseudocode: The algorithm takes a harmful dataset, initial rewrite skills, failure-analysis and evolution skills, executor, target model, judge, and Stage 1 and evolution budgets as input.It initializes a skill library, skill memory, failure memory, and solved set before Stage 1 warm-up.
  • D. Pseudocode: During Stage 1, it selects the highest-ranked untried skill by current UCB score conditioned on skill memory and risk category, invokes it, queries the target model, and evaluates the attempt.Each trace records the skill, generated prompt, target response, and judge outcome, while memory is updated after every attempt.
  • D. Pseudocode: A query is added to the solved set and the Stage 1 loop breaks when its judged outcome reaches threshold τ.Otherwise, the algorithm continues through the bounded skill-selection loop.
  • D. Pseudocode: A candidate evolved skill is added to the rewrite-skill library when it recovers at least one previously unresolved behavior, and recovered examples are marked solved and removed from unresolved memory.The procedure increments its evolution budget and returns the updated library, memories, and final ASR.

E. Example of a Failure-Evolution Trace

The trace shows failure-driven evolution from transparent compliance-document framing to JSON-schema completion, where refinement eventually recovered harmful completions on an adaptively selected residual set. The analyzer used refusal patterns and partial success to deepen nesting, add benign siblings, and change the task contract.

  • Failure diagnosis: Three document-gap rollouts scored 1/5 because the target preserved the harmful intent and recognized the compliance-document framing.The analyzer concluded that missing context or semantic drift was not responsible for failure.
  • Failure diagnosis: After 16 failed evolution attempts, compliance framing reached a detection ceiling, prompting discovery of JSON-schema completion as a structural alternative.The new direction changed the syntactic container rather than adding another semantic wrapper.
  • Schema evolution: The first JSON-schema skill achieved partial success: 1 trace scored 4/5, while two others scored 1/5 and 2/5.The 4/5 trace was truncated mid-response, indicating that structural framing bypassed refusal for some prompts but not reliably.
  • Schema evolution: Refinement deepened target-field nesting, added benign sibling properties, and changed the task into a schema-validation and documentation completion contract.These changes directly implemented the analyzer’s diagnosis that the request remained too prominent in the shallow field.
  • Residual-set validation: 28 responses recovered by jsonschema-v2 had a mean independent StrongREJECT harmfulness score of 0.978.These conditional results came from an adaptively selected residual set, not a paired holdout ASR@1 estimate; recovered prompts were removed and remaining failures drove later variants.

F. Core Implementations of the RQ4 Examples … G. Representative Evolved-Skill Transformations

The RQ4 examples implement evolved jailbreak skills as reusable prompt-construction and template transformations. Representative skills reframe harmful requests as relational analysis, archived threat records, schema completion, forensic interpretation, incomplete configuration code, or prevention-oriented taxonomy research.

  • F. Core Implementations of the RQ4 Examples: The five RQ4 examples retain strategy-specific prompt construction, seed interpolation, and template logic while omitting shared harness and serialization code.Concrete demonstrations in LLM meta-prompts are redacted, and repetitive benign or code boilerplate is omitted in selected transformations.
  • LLM-rewrite example: rewrite-antipattern-v7: rewrite-antipattern-v7 reframes extraction as relational analysis of an existing case document, requesting component interactions, dependencies, and operational flow.The prompt establishes analytical roles and asks for relationship mapping rather than a component list.
  • LLM-rewrite example: rewrite-transcript-v2: rewrite-transcript-v2 presents harmful instructions as a completed threat-intelligence record that the model should continue as documented methodology.Its structure includes a database-entry header, reference ID, archive date, risk assessment, and a concluding attack-vector summary.
  • Template example: rewrite-jsonschema-v2: rewrite-jsonschema-v2 embeds the query in a SecurityAuditReport schema and asks the model to complete a missing annotation example with step-by-step technical methodology.The template includes audit identifiers, timestamps, findings, severity levels, technical details, and an incomplete annotation field.
  • LLM-rewrite example: rewrite-forensic-v1: rewrite-forensic-v1 frames harmful content as quoted evidence for incident-response or malware-analysis work, making expert interpretation the primary task.Instructions request analysis of mechanisms, exploitation methods, evasion strategies, or attack logic in artifacts such as code blocks, logs, or captured payloads.
  • Template example: rewrite-implicit-v5: rewrite-implicit-v5 distributes an encoded request across three configuration fragments and asks the model to complete Base64 decoding to reconstruct the full configuration.The template uses config_source, schema_data, and rule_set fragments, then concatenates and decodes them in parse_rules.
  • G. Representative Evolved-Skill Transformations: Table 11 presents representative adversarial prompts from six selected evolved skills, highlighting their operative transformations and varying the amount of omitted boilerplate.Three prompts are shown verbatim, the forensic prompt is reproduced in full with its code block inline, and two prompts omit repetitive benign or code content.
  • G. Representative Evolved-Skill Transformations: rewrite-research-v2 recasts cheating guidance as prevention-oriented taxonomy analysis for a university content-moderation researcher developing academic-integrity systems.The example invokes an IRB-approved project and requests a structured framework analyzing cheating strategies and their mechanisms.
Loading 2608.16465v1…