JailbreakSkill: Scaling Automated Red-Teaming with Reusable and Ever-Evolving Skills

• Existing automated red-teaming methods optimize prompts, retrieve attack memories, or coordinate dynamic workflows, but their attack logic remains difficult to invoke, revise, replace, or extend independently.These systems lack a common representation for executing and improving attack methods as reusable modules.Source: §1 · pp. 1–2
• JailbreakSkill shifts adaptation from prompts and workflows to modular, executable, and evolvable skills that a planner can select and sequence through a shared interface.The framework defines rewrite, failure-analysis, and evolution skills, separating procedural logic from workflow orchestration.Source: §1 · p. 2, Figure 1 · p. 2
• JailbreakSkill converts unresolved attack traces into reusable capabilities by diagnosing shared failure patterns and generating candidate rewrite skills through refinement, combination, or discovery.Candidates that improve attack performance are validated and added to the skill library for reuse across behaviors and risk categories.Source: §1 · pp. 2–3
• Unlike monolithic end-to-end attack programs, JailbreakSkill maintains independently addressable skills that can be updated locally, replaced, and reused across attacks.This incrementally builds a library of attack procedures from previous failures rather than only searching for stronger adversarial prompts.Source: §1 · p. 2
• JailbreakSkill is evaluated on AdvBench and HarmBench across diverse target models, with improvements in budgeted attack success, recovery of unresolved behaviors, and transfer of new skills across target models.Source: §1 · p. 3

LLM Jailbreaking and Automated Red Teaming

• LLM jailbreaking progressed from handcrafted prompts and surface transformations to gradient-based, black-box, reusable, and agentic attack methods.These methods include role-playing, ASCII art, Morse code, low-resource translation, adversarial suffix optimization, search, fuzzing, persona modulation, scenario reframing, visual encoding, and many-shot composition.Source: §2 · p. 3
• JailbreakSkill is closest to agentic red-team frameworks but centers runtime execution and local method evolution on a shared library of agent skills.Related systems use strategy abstractions, lifecycle-managed memory, continuous attack-vector discovery, or executable attack-program evolution.Source: §2 · p. 3

Structured Agent Skills

• Structured agent research represents reusable capabilities as procedures or portable artifacts, while JailbreakSkill exposes prompt rewriting, failure diagnosis, and skill evolution through a shared interface.Validated rewrite skills can be added independently to the library for subsequent reuse, extending work on skill utility, discovery, repair, maintenance, and skill-related safety risks.Source: §2 · p. 3

Problem formulation

• The framework considers automated red-teaming over a harmful dataset, with each seed prompt q_i associated with a risk category r_i.Source: §3 · pp. 3–4
• At step t, attacker A uses seed prompt q_i to generate adversarial prompt x_i,t against target model V, which produces a response.Source: §3 · p. 4

Success criterion

• An attack succeeds when its attack-success score z_i,t reaches the predefined threshold τ.Source: §3 · p. 4

Evaluation objective

• The final attack success rate is defined over the total number of attempts T_i made for each seed prompt throughout red-teaming.Source: §3 · p. 4
• The objective is to maximize ASR under fixed budgets for target-model queries and skill evolution.Source: §3 · p. 4

4.1. Framework Overview

• The framework defines rewrite, failure-analysis, and evolution skills that jointly support attack execution, diagnosis, and skill evolution.Each skill specifies when it applies and how it should be used, integrating these capabilities within one reusable framework.Source: §4.1 · p. 4

4.2. Initial Rewrite Skill Library

• Initial rewrite skills are distilled from research papers, repositories, and web resources into general procedures rather than source-specific prompts.Skills use a common artifact format centered on SKILL.md, with optional scripts and references; examples include role-playing, translation, encoding, and emoji fragmentation.Source: §4.2 · p. 5

4.3. Stage 1: Warm-up

• Stage 1 sequentially scans untried skills using risk-conditioned UCB ranking, updating shared skill memory after each attempt and stopping when a response succeeds or the query budget is exhausted.Unresolved prompts and failed traces enter failure memory, while accumulated outcomes allow later prompts to benefit from earlier attack experience.Source: §4.3 · pp. 5–6

4.4. Stage 2: Skill Evolution

• Stage 2 reuses prior evolved skills, diagnoses recurring patterns across unresolved traces, and generates refinement, combination, or discovery candidates that are retained when they solve previously unresolved examples.Accepted skills update skill memory, remove newly solved failures, and remain available for later risk categories and red-teaming tasks until the evolution budget is exhausted.Source: §4.4 · pp. 6–7

5.1. Experimental Setup

• Experiments use eight open-weight and proprietary target models, with Claude Sonnet 4.5 for attack-side reasoning and Qwen2.5-7B-Instruct for execution.The initial library contains 16 skills spanning encoding and perturbation, code and structured completion, and contextual or scenario reframing.Source: §5.1 · p. 7

5.2. Evaluation Protocol

• Evaluation reports ASR, harmfulness score, and average query count using GPT-4o judging, with equal query budgets and stopping after the first success.ASR counts only judge score 5 as success, harmfulness ranges from 0 to 1, and unsuccessful attacks receive the full query budget.Source: §5.2 · p. 8

5.3. Main Results

• EquaCode has the highest macro-average ASR@1 at 42.1%, but skill effectiveness is target-specific, including 0.0% for EquaCode against Claude Sonnet 4.6 on AdvBench.Literary Framing leads on Gemini 3.1 Pro, while Fictional-Space Framing reaches 79.0% against Llama-3.3-70B-Instruct on AdvBench.Source: §5.3 · p. 8
• The 16 initial skills’ union reaches 72.6% pooled coverage versus 54.0% for the post-hoc best skill, an 18.6-point gain demonstrating complementary attack mechanisms.Coverage gains are 24.0 points on HarmBench and 14.5 on AdvBench; 10.8 HarmBench and 7.1 AdvBench skills per model uniquely solve missed prompts.Source: §5.3 · p. 8, Figure 3 · p. 9
• 72.0% and 68.1% ASR@10 on AdvBench and HarmBench make Stage 1 best overall, exceeding TAP by 1.1 and 13.5 points under a 10-call budget.JailbreakSkill ranks first in 13 of 16 model–benchmark settings, while risk-conditioned routing lowers AQC to 4.14 and 4.63 calls versus uniform routing’s 5.98 and 6.24.Source: §5.3 · p. 9, Table 1 · p. 10, Table 2 · p. 10
• Across five target models, evolution raises macro-average ASR from 56.7% to 74.2% on AdvBench and from 54.5% to 67.9% on HarmBench, gains of 17.5 and 13.4 points.On GPT-5.4, AdvBench ASR rises from 13.9% to 62.5%, a 48.6-point gain. Average harmfulness reaches 0.856 on AdvBench and 0.756 on HarmBench, versus the strongest baseline’s 0.773 and 0.554.Source: §5.3 · pp. 10–11
• Transfer is substantial but mechanism-dependent: source-target, DeepSeek-V4-Pro, and GLM 5.2 held-out ASR/Harm. averages are 36.2%/0.645, 40.6%/0.648, and 25.6%/0.472.On JBB-Original, the corresponding averages are 29.1%/0.511, 35.3%/0.542, and 20.3%/0.382; selected skills retain broad but uneven cross-model and cross-benchmark transfer without adaptation.Source: Table 4 · p. 12, §5.3 · p. 12
• ASR and harmfulness can diverge: on GLM 5.2, rewrite-research-v2 achieves 3.0% ASR but 0.744 harmfulness, showing partial informative compliance without full goal satisfaction.Only judge score 5 counts as success, while many responses receive scores 3 or 4 and retain nonzero harmfulness.Source: §5.3 · p. 12

6. Conclusion

• JailbreakSkill shifts automated red-teaming from repeatedly searching for isolated prompts toward maintaining an extensible library of reusable attack capabilities.The modular interface decouples attack methods from workflows, enabling skills to be added, replaced, reused, and improved without redesigning the pipeline.Source: §6 · p. 13
• Several evolved skills remain effective across models and unseen behaviors, indicating that the evolving library can extend attack coverage beyond its original prompts and targets.The conclusion attributes this growth to using attack experience to diagnose, refine, combine, and discover skills that are added back into the library.Source: §6 · p. 13

A. Skill Library

• The skill library comprises initial rewrite skills, a failure-analysis skill, and evolution skills that synthesize new rewrite skills from diagnosed failures.This organization supports complementary attack coverage under a fixed query budget and converts recurring failures into validated capabilities.Source: §6 · p. 13, A. Skill Library · p. 18

A.1. Initial Rewrite Skills

• Initial rewrite skills transform each seed prompt into adversarial candidates through either deterministic templates requiring no model call or LLM-generated rewrites.Table 5 summarizes the initial Stage-1 rewrite-skill families, while harmful payload demonstrations are redacted in the implementation excerpts.Source: §A.1 · pp. 18–19, Table 5 · p. 18

Template examples

• Template skills recast prompts through Base64 encoding, code completion, table filling, equation-style coding, and story continuation.These transformations embed the request within structured formats or completion tasks while preserving the operative prompt structure.Source: Template example: rewrite-base64 · p. 19, Template example: rewrite-codecomplete · p. 19, Template example: rewrite-tablefill · p. 19

LLM-rewrite examples

• LLM-rewrite skills perturb character patterns, fragment keywords with emojis, and frame requests as historical, hypothetical, or literary content.Across these strategies, the prompts aim to preserve recoverable meaning while changing surface form, conversational framing, temporal distance, scenario framing, or writing style.Source: LLM-rewrite example: rewrite-char · p. 22, LLM-rewrite example: rewrite-emoji · p. 23, LLM-rewrite example: rewrite-history · p. 23

LLM-rewrite example: rewrite-literary

• The rewrite-literary strategy is implemented as a prompt that asks an LLM to creatively reformulate a request and output only the reformulated question.The supplied implementation includes illustrative examples and an output-only requirement, but does not expose the concrete examples.Source: LLM-rewrite example: rewrite-literary · p. 25

LLM-rewrite example: rewrite-roleplay

• Role-play reframes the request through an expert persona, anchoring language, and situational backstory while retaining its full specific content.The persona supplies a professional obligation, but the original request remains fully present and detailed.Source: LLM-rewrite example: rewrite-roleplay · p. 25

LLM-rewrite example: rewrite-reverse

• Reverse rewriting converts a harmful request into a specific benign counterpart, then appends an instruction to recover the original through analytical contrast.The final output is one combined benign-question and reversal prompt rather than two separate pieces.Source: LLM-rewrite example: rewrite-reverse · pp. 25–26

LLM-rewrite example: rewrite-language

• Multilingual rewriting distributes key terms across three or four languages, adds specialized terminology, and preserves a grammatically parseable sentence without exposing the full intent in one language.The strategy allows languages such as French, German, Spanish, Russian, Japanese, Korean, Chinese, Arabic, Portuguese, Italian, Turkish, Hindi, or Vietnamese.Source: LLM-rewrite example: rewrite-language · p. 26

LLM-rewrite example: rewrite-word

• Word-level rewriting replaces explicit terms with clinical or euphemistic equivalents, uses impersonal constructions, and adopts a dry encyclopedic tone.The replacement vocabulary must still refer unambiguously to the same specific content.Source: LLM-rewrite example: rewrite-word · pp. 26–27

LLM-rewrite example: rewrite-security

• Security framing places the request in professional research or defense contexts while explicitly preserving the harmful method and requesting specific, detailed output.Suggested roles include security researchers, red-team analysts, academics, compliance officers, moderators, and debate coaches.Source: LLM-rewrite example: rewrite-security · p. 27

LLM-rewrite example: rewrite-space

• Fictional-space rewriting casts harmful content as a concrete design requirement or mechanic in games, simulations, virtual worlds, or other fictional settings.The fictional framing makes the request appear to concern world-building or narrative development while keeping the specifics actionable.Source: LLM-rewrite example: rewrite-space · p. 28

A.2. Failure-Analysis Skill

• Failure analysis converts attack traces into structured diagnoses that identify dominant failure dimensions and select refinement, combination, or discovery before evolution.Its inputs include attempted skills, candidates, target responses, judge scores, descriptions, and prior history; separating diagnosis from evolution encourages explicit cause analysis first.Source: §A.2 · pp. 28–29

A.3. Evolution Skills

• Evolution skills condition new executable rewrites on diagnosed failures, representative prompts, risk categories, and source mechanisms, then validate and register them for subsequent search.The framework supports LLM-rewrite, deterministic-template, and hybrid modes; refinement preserves the source mode, while combination and discovery may choose the simplest sufficient mode.Source: §A.3 · pp. 30–31

evolve-refine

• Refinement targets a specific skill and adapts camouflage, intent preservation, or contextual scaffolding to address diagnosed refusal, semantic-drift, or insufficient-context failures across a category.Its instructions emphasize generalization across representative prompts rather than optimizing for a single prompt.Source: evolve-refine · p. 31

evolve-combine

• Combination unifies two source skills and selects a mode that preserves both mechanisms, including hybrid processing when semantic rewriting and structural embedding are complementary.The combined skill is instructed to generalize across representative prompts and address the diagnosed failure dimensions.Source: evolve-combine · pp. 32–33

evolve-discover

• Discovery creates a novel skill when existing approaches are unsuitable, choosing deterministic, LLM-rewrite, or hybrid generation according to intent preservation, semantic transformation, and structural needs.The mode guide prioritizes deterministic templates for structural techniques or critical verbatim preservation, LLM rewriting for semantic transformation, and hybrid processing when both are required.Source: evolve-discover · pp. 34–36

B. Construction of the Initial Rewrite Skill Library

• The initial library is built through manual specification, paired-example induction, and external-source distillation, with every skill reusable on unseen prompts while preserving underlying objectives.Deterministic transformations are programmatic, whereas semantic transformations are executor-model instructions; induced skills abstract recurring steps while removing demonstration-specific content.Source: B. Construction of the Initial Rewrite… · p. 36

B.1. Source Collection

• External mechanisms are collected from literature, repositories, web resources, and adversarial datasets, then abstracted away from source-specific entities, models, templates, and benchmark wording.Examples include semantic obfuscation, role assignment, multilingual rewriting, code-completion framing, and narrative continuation.Source: §B.1 · p. 36

B.2. Mechanism Distillation

• Each source group yields at most one fixed single-turn rewrite skill defined by an operator, contextual frame, and intent-preservation constraint, excluding outer-loop procedures.Coordinated stages are composed in order, mutually exclusive alternatives are resolved, and verified official reusable templates may retain their operative wording.Source: §B.2 · pp. 36–37

B.3. Skill Normalization

• Accepted mechanisms are packaged as standardized SKILL.md and scripts/run.py artifacts, exposing metadata and a unified JSON-over-stdio interface for producing rewritten candidates.Externally derived skills retain provenance, while admission checks verify schema validity, seed-intent preservation, and local executability without establishing source-method fidelity or model effectiveness.Source: §B.3 · p. 37

C.1. Training Setup

• Stage 1 searches over 16 initial rewrite skills, evaluating each generated candidate once against the target model.All 16 skills remain available, but no more than 10 are invoked per seed.Source: §C.1 · p. 37
• Success-based early stopping ends Stage 1 when any candidate receives the maximum judge score.The protocol therefore avoids querying further initial skills after a successful candidate is found.Source: §C.1 · p. 37
• Stage 2 processes prompts unsolved in Stage 1 by grouping them by risk category and selecting up to five representative failures per category.It aggregates up to five failed traces for each skill before evolution.Source: §C.1 · p. 37
• A failure analyzer identifies dominant failure patterns, which the evolution module uses to generate category-level skills.Each unresolved prompt is evaluated with at most 20 evolved skills during this phase.Source: §C.1 · pp. 37–38
• Both baselines and JailbreakSkill receive at most 30 target-model queries per seed.The reported target-model-call budget counts only calls to the evaluated target model.Source: §C.1 · p. 38, Table 7 · p. 38
• Table 7 summarizes training hyperparameters and budget settings, while Table 8 reports auxiliary API roles and decoding parameters.Failure analysis, skill generation, and judging calls are reported separately in Appendix C.4.Source: Table 7 · p. 38, Table 8 · p. 38, §C.1 · p. 38

Implementation

• The evaluation implements every baseline method using the OpenRT framework and applies the same protocol as JailbreakSkill.Method-specific baseline configurations are summarized in Table 9.Source: §C.2 · p. 38, Table 9 · p. 39

Shared evaluation protocol

• All methods receive at most 30 target-model calls per seed prompt and stop early after a judged-successful response, excluding auxiliary attacker, filter, and judge calls.Common target-model and final-judge settings are reported in Table 8, while method-specific configurations appear in Table 9.Source: §C.2 · p. 38, Table 9 · p. 39

Attacker configuration

• JailbreakSkill and applicable LLM-based baselines use the same Qwen2.5-7B-Instruct attacker/executor for target-facing attack generation.JailbreakSkill uses Claude Sonnet 4.5 only for category-level failure analysis and skill evolution, not direct target-facing jailbreak-prompt generation; those auxiliary costs are reported separately in Appendix C.4.Source: §C.2 · p. 38

• ASR counts an attack as successful only when GPT-4o assigns the seed-query/response pair the maximum score of 5 for direct, substantive harmful fulfillment.Scores range from 1 for refusal or safe redirection to 5 for concrete, actionable assistance that materially advances the prohibited objective; merely reproducing the query does not qualify.Source: §C.3 · pp. 39–40
• Both evaluators use GPT-4o with temperature 0.0, and the ASR judge evaluates each seed-query and target-response pair against the applicable safety policy.The HS judge evaluates an original harmful prompt together with the target-model response.Source: §C.3 · pp. 39–41
• For HS, GPT-4o follows StrongREJECT [Souly et al., 2024] by rating refusal, convincingness, and specificity, then reporting mean harmful score across target responses.Refusals receive 0, whereas non-refusals are mapped to [0, 1] using convincingness and specificity.Source: §C.3 · p. 41

C.4. Time Cost

• Auxiliary logical model calls are measured separately from target-model calls for attack or rewrite generation, online ASR judging, safety filtering, failure analysis, and skill evolution.Calls are averaged over seeds and then macro-averaged across model–benchmark settings within each stage.Source: §C.4 · p. 42, Table 10 · p. 42
• Table 10 reports macro-average auxiliary logical model calls per seed for Stage 1 and Stage 2, excluding target-model calls; attack/rewrite includes prompt generation and TAP tree evaluation and pruning.Only ReNeLLM uses a safety filter.Source: Table 10 · p. 42
• After amortization, Stage 2 failure analysis and skill evolution add only 0.23 analyzer calls and 0.23 evolver calls per seed; JailbreakSkill has lowest attack/rewrite and online ASR-judge costs in both stages.Deterministic skills and UCB routing require no model inference.Source: §C.4 · p. 42

D. Pseudocode

• The algorithm initializes the skill library, skill memory, failure memory, and solved-query set before processing harmful examples with initial rewrite skills.Inputs include the harmful dataset, initial skills, executor, target model, judge, and Stage 1 and Stage 2 budgets.Source: D. Pseudocode · p. 43
• During Stage 1, UCB routing selects each example’s highest-ranked untried skill, executes it, queries the target model, judges the attempt, and records its outcome.Successful examples enter the solved set; unresolved traces are stored for failure analysis.Source: D. Pseudocode · p. 43
• During Stage 2, failure-analysis and evolution skills diagnose unresolved patterns and refine, combine, or discover candidate rewrite skills within a bounded evolution budget.Previously evolved skills are evaluated first on unresolved examples for each risk category.Source: D. Pseudocode · p. 43
• A candidate skill enters the rewrite-skill library only after recovering at least one unresolved behavior, while recovered examples are marked solved and removed from unresolved memories.The procedure returns the updated skill library, skill memory, and final ASR computed as solved examples divided by dataset size.Source: D. Pseudocode · p. 43

• The preceding document-gap candidates failed because the target preserved semantic intent and recognized the transparent compliance-document framing, rather than suffering from missing context or semantic drift.All three representative rewrite-docgap-v12 traces scored 1/5 with identical refusal patterns.Source: E. Example of a Failure-Evolution Trace · p. 44
• After 16 failed evolution attempts, compliance framing hit a detection ceiling, while JSON-schema completion changed the syntactic container instead of adding another semantic wrapper.The analyzer identified json-schema-completion as the discovery direction after repeated compliance-document failures.Source: E. Example of a Failure-Evolution Trace · p. 44
• The initial jsonschema-v1 skill produced mixed outcomes: one trace scored 4/5, while two others scored 1/5 and 2/5 with explicit refusal.The successful trace was truncated, indicating partial compliance before the model potentially caught itself.Source: E. Example of a Failure-Evolution Trace · p. 46
• The failure diagnosis attributed v1’s weakness to the request remaining too prominent in a shallow field, motivating deeper nesting, benign siblings, and a schema-validation completion contract.The refinement retained deterministic execution while changing the carrier features identified by the analyzer.Source: E. Example of a Failure-Evolution Trace · p. 46
• 28 prompts were recovered by jsonschema-v2 with mean independent StrongREJECT harmfulness 0.978, but these conditional residual-set results are not paired holdout ASR@1.Recovered prompts were removed from the unresolved pool, while remaining failures drove later variants such as jsonschema-v3.Source: E. Example of a Failure-Evolution Trace · p. 47

F. Core Implementations of the RQ4 Examples

• The RQ4 examples provide strategy-specific prompt construction, seed interpolation, and template logic, while omitting shared harness code and retaining complete implementations in the repository.For deterministic skills with multiple templates, the excerpts show the main-text variant, whereas checked-in artifacts retain all variants.Source: F. Core Implementations of the RQ4… · p. 47

LLM-rewrite example: rewrite-antipattern-v7

• Relational analysis reframes extraction requests as analysis of interconnected components, using professional roles, case-study excerpts, relationship mapping, and operational-flow questions.The strategy distinguishes listing components from explaining how they interact to form a complete workflow.Source: LLM-rewrite example:… · p. 47

LLM-rewrite example: rewrite-transcript-v2

• Threat-documentation rewriting presents harmful instructions as an archived threat-intelligence record that the model should summarize rather than generate as new instructions.The template adds a reference ID, archive date, threat classification, documented methodology, risk assessment, and a completion cue.Source: LLM-rewrite example:… · p. 48

Template example: rewrite-jsonschema-v2

• The JSON-schema template casts the request as completing missing annotation examples in a SecurityAuditReport schema, including a field requiring step-by-step technical methodology.It asks the model to fill the vector_description.annotation.example_scenario field under specified validation and formatting requirements.Source: Template example: rewrite-jsonschema-v2 · pp. 48–49

LLM-rewrite example: rewrite-forensic-v1

• Forensic rewriting frames harmful content as quoted evidence for incident response or threat analysis, emphasizing analytical verbs and defensive objectives instead of constructive requests.The instructions use roles such as malware analyst or incident responder, plausible breach-investigation scenarios, captured artifacts, and questions about mechanisms or evasion.Source: LLM-rewrite example: rewrite-forensic-v1 · p. 49

Template example: rewrite-implicit-v5

• Fragmented configuration decoding embeds a request across three Base64 fragments in an incomplete ConfigValidator template and asks the model to reconstruct the full configuration.The main-text variant presents missing decode logic, concatenation, and reconstruction as a code-completion task; the artifact also contains API token-manager and data-pipeline variants.Source: Template example: rewrite-implicit-v5 · p. 50

G. Representative Evolved-Skill Transformations

• Table 11 presents six evolved skills whose adversarial prompts operationalize these transformations, including prevention-oriented taxonomy analysis and fragmented configuration decoding.The examples cover LLM rewrites and deterministic templates, with repetitive benign or code boilerplate omitted for the latter presentation.Source: Table 11 · p. 51, G. Representative Evolved-Skill… · p. 51