PaperlayerView exact revision on arXiv

Source-linked AI summary

DARWIN: Evolving Jailbreak Adversary and Guardrail for LLM Safety Evaluation and Protection

Weiwei Qi, Zefeng Wu, Zhilin Guo, Tianhang Zheng, Chaochao Lu, Liang He, Zhan Qin, Kui Ren

arXiv:2607.19829v2cs.CR

TL;DR

Fixed attacks and guardrail datasets provide a static basis for safety evaluation and protection despite continually evolving jailbreak adversaries. DARWIN couples an evolving attack strategy pool with online guardrail training on emerging attacks and disguised benign queries. It reports strong attack success and safety detection while maintaining benign pass rates and reducing over-refusal.

  • Problem

    Safety evaluation often relies on fixed jailbreak attacks and guardrails on fixed harmful-prompt datasets, despite adversaries continually evolving their capabilities.

    Source: §1 · p. 2

  • Method

    DARWIN evolves an explicit jailbreak strategy pool and uses DARWIN-Attack-generated emerging samples to train an online guardrail jointly on harmful and benign disguised queries.

    Source: §1 · p. 2

  • Results

    DARWIN-Guard achieves 95.0% average unsafe recall across nine safety benchmarks, while DARWIN-Attack reaches nearly 100% attack success on DeepSeek-V4-Pro and YuFeng-XGuard and over 90% on GPT-5.5.

    Source: §1 · p. 2

  • Takeaways & Limitations

    DARWIN presents an evolving attack–defense loop in which defensive capabilities improve concurrently with emerging safety threats.

    Source: §1 · p. 2

    A jailbreak prompt is considered valid only when it preserves the original malicious intent.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Most existing LLM safety evaluation and defense methods are static: jailbreak vulnerabilities are assessed with fixed attacks, and guardrails are trained on fixed malicious-prompt datasets. In practice, adversaries continually evolve and expand the attack space. We propose DARWIN, an evolutionary attack-defense framework that models jailbreaking as a continual process and updates guardrails through an attack-defense loop. DARWIN-Attack evolves its capabilities through strategy discovery, mutation, selection, and feedback-guided composition. It mines strategies from broad external sources, generates variants through self-reflection and genetic evolution, filters them by performance against aligned LLMs, and adaptively selects and composes strategies during attacks. Through continuous evolution, DARWIN-Attack achieves state-of-the-art attack success rates against frontier LLMs and guardrails, reaching nearly 100\% on DeepSeek-V4-Pro, over 90\% on GPT-5.5, and nearly 100\% on YuFeng-XGuard, outperforming recent evolving frameworks such as LSA and MAGIC. Its continued evolution exposes new vulnerabilities, motivating timely defense updates. Accordingly, DARWIN-Guard performs online adversarial training on emerging samples generated by DARWIN-Attack and jointly learns from malicious and benign disguised queries to recognize underlying intent rather than superficial attack patterns. DARWIN-Guard achieves 95.0\% average unsafe recall across multiple safety benchmarks, outperforming recent advanced guardrails such as YuFeng and Nemotron, while maintaining nearly 100\% average benign pass rate on standard benign benchmarks and the best performance on over-refusal benchmarks. Our code and model are available at https://github.com/ZJU-LLM-Safety/DARWIN.

1 INTRODUCTION

Static jailbreak evaluations and fixed guardrail datasets fall behind adversaries that continually evolve. DARWIN couples an evolving attack strategy pool with online guardrail training, reporting strong attack success and safety detection while preserving benign utility.

1 INTRODUCTION

  • DARWIN-Attack admits newly introduced strategies only after sandbox testing shows their attack success rates exceed a predefined threshold.The evolving strategy pool draws on external strategies and newly generated strategies.Source: §1 · p. 2
  • DARWIN-Attack achieves nearly 100% attack success on DeepSeek-V4-Pro and YuFeng-XGuard and over 90% on GPT-5.5, outperforming LSA and MAGIC.It uses attack feedback to adaptively select and compose strategies after failed attempts.Source: §1 · p. 2
  • DARWIN-Guard reaches 95.0% average unsafe recall across nine safety benchmarks, maintains nearly 100% average pass rate across six benign datasets, and performs best on both over-refusal benchmarks.The reported unsafe recall outperforms all evaluated baseline guardrails.Source: §1 · p. 2
  • DARWIN couples an evolving adversary with an improving guardrail in a continuous loop of attack-prompt generation and guardrail training.The framework has two modules, DARWIN-Attack and DARWIN-Guard.Source: §1 · p. 2

2 RELATED WORK

Prior jailbreak attacks and guardrails often rely on fixed strategies or datasets, while co-evolving frameworks adapt attackers and defenders but have limitations in strategy generation and deployment. DARWIN instead expands an explicit strategy pool and trains an external, transferable guardrail.

2.1 JAILBREAK ATTACKS AND GUARDRAILS

  • Earlier attack methods include gradient-optimized suffixes, prompt rewriting, feedback-based search, and strategy discovery, but some rely on a single strategy or a fixed strategy pool.AutoDAN-Turbo mainly selects one strategy per prompt, while MAJIC combines multiple strategies using a fixed pool.Source: §2.1 · p. 3
  • Existing representative guardrails are trained on fixed datasets and remain static after deployment, limiting adaptation to newly emerging jailbreak strategies.DARWIN-Guard instead trains continuously on adversarial examples generated by DARWIN-Attack.Source: §2.1 · p. 3

2.2 CO-EVOLVING ATTACK–DEFENSE FRAMEWORKS

  • DARWIN-Attack improves average ASR on HarmBench by 29.5% over LSA and by 22.0% over MAGIC.Unlike those frameworks, it expands an explicit strategy pool and uses failed-attempt feedback to select and compose subsequent strategies.Source: §2.2 · p. 3
  • DARWIN trains an external guardrail without changing the target LLM, allowing transfer across LLMs without model-specific adaptation.LSA and MAGIC directly fine-tune LLMs, which the passage says can degrade utility and cannot be transferred to other LLMs.Source: §2.2 · p. 3

3 THE DARWIN FRAMEWORK

DARWIN couples an evolving jailbreak adversary with online guardrail training, so attack feedback and generated samples feed a continuing attack–defense loop. Its attack strategies adapt through discovery, mutation, reflection, and feedback, while the guard trains on disguised harmful and benign queries to recognize intent.

3.1 OVERVIEW OF THE EVOLVING ATTACK–DEFENSE LOOP

  • Each round expands and refines the attacker's strategy state, generates adversarial guardrail-training examples, and updates the guard from its prior checkpoint.The increasingly capable guard then challenges the attacker to keep evolving in the next round.Source: §3.1 · p. 4

3.2 DARWIN-ATTACK: AN EVOLVING ADVERSARY

  • DARWIN-Attack evolves through external strategy collection, genetic crossover and mutation, reflection on failed attacks, and feedback-guided strategy selection and composition.New candidates are screened for similarity and admitted only when their sandbox attack success rate exceeds a threshold; failed-attempt refinements also undergo sandbox validation.Source: §3.2 · p. 5
  • After a failed strategy, DARWIN-Attack samples and composes a next strategy using a target-specific transition process updated from attack feedback.The strategy sequence transforms a harmful prompt while preserving its original malicious intent.Source: §3.2 · p. 5

3.3 DARWIN-GUARD: ONLINE ADVERSARIAL TRAINING

  • DARWIN-Guard disguises both harmful and benign training prompts, aiming to make the guard learn underlying intent rather than correlate complex formats with maliciousness.It trains on the disguised sample to correct misclassifications and on the raw prompt to maintain baseline performance.Source: §3.3 · p. 6
  • For each training sample, the attack module tries up to Nmax disguises, stopping when one causes misclassification or otherwise using the final candidate.The guard then minimizes a joint training objective on disguised and original prompts.Source: §3.3 · p. 6

4 EXPERIMENTS

DARWIN-Attack leads the evaluated jailbreak methods, while DARWIN-Guard combines high unsafe recall with strong benign utility. Coupled evaluations show improved guard robustness alongside vulnerabilities to newly evolved and unseen attacks.

4.1.1 ATTACK SETUP

  • Attack evaluation uses HarmBench and AdvBench, four frontier LLMs, two guardrails, six jailbreak baselines, a 60-query budget, and DARWIN-Attack's 200-strategy stage.ASR, harmful score, and average query count are reported; GPT-4o judges harmful fulfillment for LLMs, while guardrail attacks succeed when harmful intent is classified as safe.Source: §4.1.1 · p. 6

4.1.2 GUARDRAIL SETUP

  • DARWIN-Guard is trained on WildJailbreak harmful prompts and OR-Bench benign prompts, then evaluated on harmful, standard benign, and over-refusal benchmarks.The evaluation covers nine harmful-prompt and six standard benign benchmarks, plus safe XSTest and benign JBB-Behaviors subsets; metrics are unsafe recall and benign pass rate.Source: §4.1.2 · pp. 6–7

4.2 DARWIN-ATTACK EVALUATION

  • DARWIN-Attack achieves the highest ASR across evaluated LLMs and guardrails on HarmBench and AdvBench, plus the highest HS on LLMs and lowest AQC overall.On HarmBench, ASR is 93.7% on GPT-5.5 and 78.2% on Claude Sonnet 4.6, versus MAGIC's 73.5% and 36.2%; it reaches 99.7% against Qwen3Guard and 99.2% against YuFeng-XGuard.Source: Table 1 · p. 7, §4.2 · p. 7

4.3 DARWIN-GUARD EVALUATION

  • 95.0% average unsafe recall across nine benchmarks, exceeding YuFeng-XGuard by 6.0%.DARWIN-Guard reaches 100.0% on JBB-Behaviors and 99.0% on HarmBench; on Aegis2.0 and JailbreakHub it scores 91.9% and 95.2%, versus 87.6% and 80.8% for YuFeng-XGuard.Source: Table 2 · p. 7, §4.3 · p. 7
  • DARWIN-Guard has nearly 100% average benign pass rate across six standard benchmarks and the best results on both over-refusal benchmarks.It scores 97.6% on XSTest and 80.0% on JBB-Benign, versus 94.0% and 59.0% for YuFeng-XGuard.Source: §4.3 · pp. 7–8, Figure 2 · p. 8

4.4 COUPLED ATTACK–DEFENSE DYNAMICS

  • 95.0% average unsafe recall across nine benchmarks, versus 89.6% for Static AT, 90.3%/92.5% for Fixed Attacker, and 91.7% without GDE.Later guards better resist earlier attackers, but on held-out attack families ASR falls from 54% at Guard-50 to 26% at Guard-200, versus 38% without GDE and 49% for the fixed baseline.Source: Figure 3 · p. 8, §4.4 · p. 8

4.5 COMPONENT ABLATIONS

  • At 200 strategies, DARWIN-Attack reaches 78.2% ASR on Claude Sonnet 4.6, versus 25.2% for Fixed; component ablations also implicate external knowledge and Markov selection.Removing external knowledge lowers Claude ASR from 78.2% to 52.0%, while removing Markov selection raises GPT-5.5 AQC from 18.4 to 39.0.Source: Figure 4 · p. 9, §4.5 · p. 9, Table 3 · p. 9
  • 98.8% and 80.0% benign pass rates on PIQA and JBB-Benign fall to 81.5% and 59.0% without benign disguising, and to 58.2% and 25.0% without the benign branch.Ablations increase unsafe recall, which may reflect conservative filtering rather than better intent recognition.Source: Figure 5 · p. 9, §4.5 · p. 9

5 CONCLUSION

DARWIN pairs an evolving jailbreak attacker with online guardrail training, reporting stronger attack success and improved safety detection while preserving benign utility and reducing over-refusal. The paper also describes its research-use setting and released artifacts.

5 CONCLUSION

  • DARWIN-Attack achieves state-of-the-art attack success rates across frontier LLMs and guardrails, while DARWIN-Guard improves safety detection while preserving benign utility and reducing over-refusal.DARWIN-Guard learns from emerging adversarial samples and jointly trains on harmful and benign disguised queries to recognize underlying intent rather than superficial attack patterns.Source: §5 · pp. 9–10

AI USE STATEMENT

  • The authors used AI assistants only for manuscript polishing; research ideas, methods, experimental design, result analysis, and conclusions were independently developed and validated.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • Experiments use publicly available safety benchmarks and harmful-prompt datasets, and evaluate generated disguised prompts only in controlled research settings.Artifacts are provided to support reproducible safety evaluation, robustness analysis, and defensive-method research.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • Implementations, experiment scripts, and configurations are released, and the trained DARWIN-Guard checkpoint is available.The main text describes setup and evaluation protocols, while the appendix provides implementation settings, training procedures, and prompt templates.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 EVALUATION PROTOCOL

The appendix specifies how attacks are evaluated, constructed, adapted, and refined, including the query budget and strategy-pool safeguards. It also identifies the attack configuration summarized in Table 5.

A.1 EVALUATION PROTOCOL

  • Evaluation uses at most 60 target queries per instance, reserves HarmBench and AdvBench for final evaluation, and keeps the strategy pool fixed during evaluation.Evaluation prompts and attack records cannot expand the pool or undergo further sandbox validation; HS and AQC are also defined, with unsuccessful attacks assigned the full 60-query budget for AQC.Source: §A.1 · pp. 15, 18
  • Attack Success Rate is the fraction of instances with at least one success within budget; LLM success requires a judge score of 5, while guardrail success also requires preserved harmful intent classified as safe.LLM responses are assessed by GPT-4o; guardrail decisions use model-specific interfaces and parsing rules.Source: §A.1 · p. 15

A.2 TEST-TIME ATTACK PROTOCOL

  • At test time, DARWIN-Attack uses a 200-strategy pool and adapts strategy selection and composition using feedback, with attack state retained per target–dataset pair.Failed attempts are reflected on within the 60-query budget; evaluation benchmarks are not used to expand the pool.Source: §A.2 · p. 18

A.3 STRATEGY POOL CONSTRUCTION

  • The strategy pool grows from 50 seed strategies to 200 validated strategies, with candidates sourced externally or generated through genetic evolution and screened for redundancy and sandbox performance.The sandbox uses a fixed BeaverTails-Evaluation subset disjoint from training and final evaluation; candidates must meet an admission threshold against Qwen2.5-7B-Instruct.Source: §A.3 · pp. 18–19

A.4 ADAPTIVE STRATEGY SELECTION AND COMPOSITION

  • DARWIN-Attack chooses an initial strategy from historical information and, after failure, iteratively composes additional strategies; transition preferences are updated from attack outcomes.It explores up to 20 chains, each with an initial strategy and at most two additions, stopping on success or after 60 queries.Source: §A.4 · p. 19

A.5 FAILURE REFLECTION AND STRATEGY REFINEMENT

  • After unsuccessful attempts, reflection analyzes rejection feedback and refines the applied strategy to better preserve harmful intent while bypassing the target’s safety mechanism.Refined strategies undergo redundancy filtering and sandbox validation before admission to the active pool.Source: §A.5 · p. 19

A.6 ATTACK CONFIGURATION

A.7 ROBUSTNESS TO THE CHOICE OF JUDGE

The appendix re-evaluates recorded attack outputs with a different judge and documents DARWIN-Guard’s input/output format. It does not rerun attacks for the judge-sensitivity check.

A.7 ROBUSTNESS TO THE CHOICE OF JUDGE

  • The judge-sensitivity analysis substitutes Gemini-3.1-Pro for GPT-4o on recorded outputs without rerunning attacks, and Table 6 reports ASR for LLM and guardrail targets.Gemini applies the same 1–5 rubric to LLM targets and checks intent preservation plus safe guardrail classification for guardrail targets.Source: §A.7 · p. 20, Table 6 · p. 20

B.1 GUARD INPUT AND OUTPUT FORMAT

  • DARWIN-Guard’s prompt requests safety and category outputs, but only the safety label is used for training and evaluation.Its output format places the safety label first and categories on the second line.Source: §B.1 · p. 21

B.2 TRAINING SETUP

DARWIN-Guard is trained over successive rounds using disguised and raw harmful and benign prompts, with labels inherited from source prompts. The appendix also specifies the training hardware and software environment.

B.2 TRAINING SETUP

  • Training used four NVIDIA H200 141GB GPUs and two Intel Xeon Platinum 8558 processors, with Python 3.10, PyTorch 2.9.0, and CUDA 12.8.Source: §B.2 · p. 21
  • DARWIN-Guard is initialized from Qwen3Guard-Gen-8B and trained over 20 rounds using the preceding checkpoint, with source labels retained after intent-preservation filtering.Harmful prompts come from the WildJailbreak training split and benign prompts from OR-Bench; DARWIN-Attack generates disguises for both.Source: §B.2 · p. 21
  • Each training round contains 500 examples: 25 harmful and 225 benign disguised prompts, plus 25 harmful and 225 benign raw prompts.Table 7 provides implementation details for DARWIN-Guard.Source: §B.2 · p. 22, Table 7 · p. 22

B.3 EVALUATION DATASETS AND PROTOCOL

The evaluation covers harmful-prompt detection, benign-prompt handling, and over-refusal, using macro-averaged safety metrics. Baseline guardrails are evaluated as binary prompt-safety classifiers.

B.3 EVALUATION DATASETS AND PROTOCOL

  • The harmful-prompt evaluation uses nine benchmarks, while benign evaluation uses six standard QA benchmarks plus safe XSTest and benign JBB-Behaviors subsets.Source: §B.3 · p. 22
  • The final DARWIN-Guard checkpoint is evaluated after 20 training rounds, with each QA question and available answer choices combined into one prompt.Source: §B.3 · p. 22
  • Unsafe recall is macro-averaged across harmful-prompt benchmarks, while benign pass rate is macro-averaged across benign benchmarks.Empty or unparseable decisions count as errors and remain in every guardrail's denominator.Source: §B.3 · p. 23

B.4 BASELINE GUARDRAIL SETTINGS

  • Baseline guardrails are treated as binary prompt-safety classifiers, with model outputs mapped to safe or unsafe decisions.The decision rules are summarized in Table 8.Source: §B.4 · p. 23, Table 8 · p. 23

B.5 EVALUATION ACROSS EVOLUTION STAGES

DARWIN saves attacker and guard checkpoints at several strategy-pool sizes, then evaluates every saved attacker against every saved guard on HarmBench. The resulting matrix is used to assess attack and defense progress across evolution stages.

B.5 EVALUATION ACROSS EVOLUTION STAGES

  • Attackers and guards are saved at strategy-pool sizes K ∈ {50, 75, 100, 125, 150, 175, 200}.Attack-K and Guard-K denote the attacker and guard checkpoints saved at the same stage.Source: §B.5 · p. 23
  • Each Figure 3(B) matrix entry is the ASR for one saved attacker against one saved guard on HarmBench, with guard parameters fixed during evaluation.The maximum target-query budget is B = 60 per attack instance.Source: §B.5 · p. 23
  • Comparing attackers against a fixed guard measures attack progress, whereas comparing guards against a fixed attacker measures defense progress and robustness to earlier attacks.Source: §B.5 · p. 23

B.6 EVALUATION ON UNSEEN ATTACK FAMILIES

DARWIN is evaluated on attacks from held-out attack families and against released co-evolved defenders. DARWIN-Attack achieves 92%–100% ASRs against the evaluated MAGIC and LSA defenders, with average query counts ranging from 2.79 to 20.58.

B.6 EVALUATION ON UNSEEN ATTACK FAMILIES

  • The unseen attack set uses 30 strategies from held-out attack families and 100 harmful source queries from AdvBench, with duplicates and semantically equivalent variants removed.Only attacks preserving the original harmful intent are retained, and the resulting set H remains fixed across methods and stages.Source: §B.6 · p. 23
  • Figure 3(C) compares DARWIN, w/o GDE, and a fixed-pool baseline (K = 50) on H using saved guard checkpoints, without adapting prompts to guard feedback or changing guard parameters.ASR is the fraction of prompts in H classified as safe; lower values indicate better guard robustness.Source: §B.6 · p. 23

B.7 ATTACKS AGAINST CO-EVOLVED DEFENDERS

  • 92%–100% ASRs against released MAGIC and LSA defenders, with average query counts ranging from 2.79 to 20.58.The evaluation allows a maximum of 60 target queries per instance; the reported results indicate that the evaluated co-evolved defenders remain vulnerable to DARWIN-Attack.Source: Table 9 · p. 24, §B.7 · p. 24

Related papers

Loading 2607.19829v2…