PaperlayerView exact revision on arXiv

Source-linked AI summary

Model-Agnostic Lifelong LLM Safety via Externalized Attack-Defense Co-Evolution

Xiaozhe Zhang, Chaozhuo Li, Hui Liu, Shaocheng Yan, Bingyu Yan, Qiwei Ye, Haoliang Li

arXiv:2605.13411v1cs.CRcs.CL

TL;DR

LLMs remain vulnerable to adversarial prompts that elicit harmful outputs, while attack exploration can saturate. EvoSafety externalizes attack skills and defense memory; experiments report stronger safety without compromising general reasoning, including a 98.83% defense success rate against unseen attacks in Guard mode.

  • Problem

    Adversarial prompts can elicit harmful outputs from LLMs, making improved safety important for trustworthy deployment.

    Source: §1 · p. 1

  • Method

    EvoSafety uses an external adversarial skill library for attacks and a lightweight auxiliary defense model with memory retrieval for model-agnostic Steer and Guard defenses.

    Source: §1 · pp. 2–3

  • Results

    98.83% defense success rate against unseen attacks in Guard mode, outperforming Qwen3Guard while using fewer parameters and preserving general reasoning ability.

    Source: §1 · p. 3

  • Takeaways & Limitations

    EvoSafety supports continued vulnerability probing through newly discovered skills and further model-agnostic safety improvements through memory updates without repeated victim-model retraining.

    Source: §5 · p. 11

    Evaluation primarily focuses on standalone LLMs, and the framework does not support multimodal inputs, limiting applicability to vision-enabled models and more complex agentic settings.

    Source: E. Limitations · p. 48

Abstract

from arXiv · show

Large language models remain vulnerable to adversarial prompts that elicit harmful outputs. Existing safety paradigms typically couple red-teaming and post-training in a closed, policy-centric loop, causing attack discovery to suffer from rapid saturation and limiting the exposure of novel failure modes, while leaving defenses inefficient, rigid, and difficult to transfer across victim models. To this end, we propose EvoSafety, an LLM safety framework built around persistent, inspectable, and reusable external structures. For red teaming, EvoSafety equips the attack policy with an adversarial skill library, enabling continued vulnerability probing through simple library expansion after saturation, while supporting the evolution of adversarial vectors. For defense learning, EvoSafety replaces model-specific safety fine-tuning with a lightweight auxiliary defense model augmented with memory retrieval. This enables efficient, transferable, and model-agnostic safety improvements, while allowing robustness to be enhanced solely through memory updates. With a single training procedure, the defense policy can operate in both Steer and Guard modes: the former activates the victim model's intrinsic defense mechanisms, while the latter directly filters harmful inputs. Extensive experiments demonstrate the superiority of EvoSafety: in Guard mode, it achieves a 99.61% defense success rate, outperforming Qwen3Guard-8B by 14.13% with only 37.5% of its parameters, while preserving reasoning performance on benign queries. Warning: This paper contains potentially harmful text.

1. Introduction

EvoSafety addresses model-specific, saturating safety workflows by externalizing attack and defense knowledge. It combines an attack skill library, memory-supported defense prefixes, and reinforcement-learning updates to support continued attack evolution and model-agnostic defense.

1. Introduction

  • Closed, parameter-bound attack and defense learning can require re-optimization for a new victim, while attacks may plateau or fail to generalize to unseen models.Fine-tuning frontier models is especially resource-intensive.Source: §1 · p. 2
  • Externalizing attack knowledge is proposed to uncover long-tail vulnerabilities and permit continued exploration after saturation, while prompt-level defense knowledge can transfer across model updates without repeated retraining.The paper frames these mechanisms as supporting sustained safety improvement.Source: §1 · p. 2
  • EvoSafety equips the attacker with an Adversarial Skill Library and a lightweight Auxiliary Defense Model that uses retrieved successful attacks to generate model-agnostic safety prefixes.Steer prefixes activate intrinsic safeguards; Guard filters harmful inputs, and memory expansion can improve prefix generation without additional training.Source: §1 · pp. 2–3
  • 98.83% defense success rate against unseen attacks is reported for Guard mode, outperforming Qwen3Guard while using fewer parameters.The authors also report strengthened safety without compromising general reasoning ability.Source: §1 · p. 3

2. Related Work

Prior work spans manual and automated red-teaming, internalized and inference-time defenses, and iterative attacker-defender co-evolution. The paper situates EvoSafety among approaches that still face saturation, efficiency, or transferability constraints.

2. Related Work

  • Manual red-teaming is labor-intensive and difficult to scale, while automated approaches include input-space search and agentic evaluation with external strategies.RL-based red-teaming optimizes attack policies for jailbreak rewards but can suffer mode collapse and saturation in closed environments.Source: §2 · p. 3
  • Training-time alignment can be fragile under attack, induce over-refusal or utility degradation, and require repeated training for different target models.Inference-time defenses offer more modular alternatives, including safety prompting, self-scrutiny, and safety demonstrations.Source: §2 · pp. 3–4
  • Recent co-evolution methods jointly refine attacker and defender roles, but retain parameter-bound safety knowledge, which the authors associate with saturation and limited transferability.EvoSafety's overview describes iterative GRPO training of both policies with external structures.Source: §2 · p. 4, Figure 2 · p. 4

3. Methodology

EvoSafety iteratively trains attack and defense policies around external skills and retrieved attack memories. Its method combines validated skill-based attack generation and gated rewards with memory-conditioned defense and rewards over benign and adversarial requests.

3.1. Overview

  • For each request, the overview retrieves two skills to condition the attack policy, then retrieves two relevant historical attacks to condition a defense prefix before the frozen victim produces a response used to update both policies.Steer prepends a prefix to activate intrinsic safeguards; Guard filters inputs when the prefix is predicted to trigger refusal.Source: §3.1 · pp. 4–5

3.2. Attack Policy Learning with Skills

  • The attack policy receives retrieved skills, applies their executable transformations to the request, and uses transformed artifacts and surrogate-compromise indicators in a structured context to generate an adversarial prompt.The transformations and attack schemas are components of each skill.Source: §3.2 · p. 5

3.2.1. Adversarial Skill Library

  • Heterogeneous attack artifacts are mapped into a unified skill representation, and validated skills can be used to synthesize additional skills with the same strategic schema but different execution.Tri-Axis Skill Validation checks functional validity, schema consistency, and behavioral diversity.Source: §3.2.1 · p. 5

3.2.2. Reward Design

  • Early-Stage Alignment Gating uses intent retention in the attack trajectory as a proxy for response alignment, multiplying the judge reward by an indicator that prompt-request similarity exceeds a threshold.The paper notes that optimizing judge reward alone can produce reward hacking and that post-hoc filtering requires querying the victim with out-of-distribution prompts.Source: §3.2.2 · p. 6

3.3. Defense Policy Learning with Memory

  • The defense policy retrieves the top-k semantically similar historical adversarial prompts from a Verified Memory Bank to support safety-prefix generation, and the bank can be expanded as new attacks are observed.The bank stores historical adversarial prompts and uses vector-based retrieval.Source: §3.3 · p. 6

3.3.1. Reward Design

  • Defense rewards mix benign and adversarial requests to avoid input-agnostic refusal; benign examples receive correctness-based rewards with length shaping, while adversarial examples use the judge's safety classification.The benign dataset is filtered for consistent correctness and low output variance; the base correctness reward is 0.8 and the shaping coefficient is 0.2.Source: §3.3.1 · pp. 6–7

4. Experiments and Results

EvoSafety is evaluated on attack effectiveness, defense robustness, co-evolution, continual adaptation, and component ablations. Results show stronger defenses and attack performance, with external skills and memory supporting further evolution.

4.1. Experimental Setup

  • The evaluation uses HarmBench for attack and defense ASR, plus GSM8K and MMLU to assess general capabilities.Experiments use a growing Verified Memory Bank, initialized with WildJailbreak prompts; the attacker and auxiliary defense model are Mistral-7B-Instruct-v0.3 and Llama-3.2-3B-Instruct, respectively.Source: §4.1 · p. 7

4.2. RQ1 [Adversariality]: How Capable Is EvoSafety at Generating Attacks?

  • EvoSafety exceeds Jailbreak-R1 by 65.3% in ASR and ICON by 2.6% with one interaction; held-out skills yield 12.18% higher average ASR than most multi-turn methods.Small open-weight models reach 100% success with two attempts, while GPT-5.1 requires five; expanding the skill library supports continued attack discovery without retraining.Source: §4.2 · p. 8, Table 1 · p. 8, Figure 3 · p. 9

4.4. RQ3 [Co-evolution]: Do Attack and Defense Policies Evolve in Iterative Training?

  • Across rounds, attacker and defender improve steadily on both initial and held-out skill libraries when evaluated against fixed counterparts.Evaluation also uses randomly sampled skill combinations without the skill retriever.Source: Figure 4 · p. 9, §4.4 · p. 10

4.3. RQ2 [Robustness]: How Effective Is EvoSafety at Improving Safety?

  • EvoSafety reduces average ASR from 19.92% under TriPlay-RL to 11.62%, preserves reasoning better (-0.29 vs. -2.15), and its 3B Guard beats Qwen3Guard by 14.13%.Memory expansion consistently improves both defense modes, and Guard mode reaches 100% success against entirely unseen attack vectors on 4B and 70B victims.Source: Table 2 · p. 9, §4.3 · p. 9, Table 3 · p. 10

4.5. RQ4 [Continual Evolution]: Can EvoSafety Support Lifelong Safety Improvement?

  • Continued training on newly emerging skills reduces Steer-mode ASR on Llama-3.1-70B from 12.54% to 2.31%, while Guard with memory update reaches 100% defense success on 4B and 8B models.The reported gains do not degrade performance on previously defended attack methods or general reasoning benchmarks.Source: Table 4 · p. 10, §4.5 · p. 10

4.6. RQ5 [Ablation Study]: Are EvoSafety’s Components Effective?

  • Disabling memory approximately doubles average adversarial-prompt ASR, while removing skill integration drops average attack performance by 36.5% and increasing invoked skills is not monotonically beneficial.Using four skills gives the weakest performance on both 4B and 70B models; removing the high-level attack schema also causes a small, consistent reduction.Source: §4.6 · p. 11

5. Conclusion

EvoSafety externalizes attack and defense knowledge for co-evolutionary safety improvement. The appendix also examines reward hacking and motivates prompt-side similarity as an efficient but locally grounded gating signal.

5. Conclusion

  • EvoSafety externalizes structures for both policies, supporting lifelong safety improvement without repeated victim-specific retraining and with further gains from memory updates.The conclusion also reports outperforming mainstream baselines while preserving benign reasoning performance.Source: §5 · p. 11

A.1. Overreview

  • Each co-evolutionary round trains the attack policy first, then the defense policy.Source: §A.1 · p. 21

A.2.1. Analysis of Reward Hacking

  • Using attack success alone as reward causes training reward to rise while adversarial prompts can lose the harmful request’s intent.Figure 5 illustrates a prompt that replaces a social-security-number request with response-formatting rules; the authors attribute this to evaluator overfitting.Source: §A.2.1 · p. 21, Figure 5 · p. 21

A.2.2. Theoretical Motivation: Prompt-Side Similarity as an Efficient Gating Signal

  • Prompt-side similarity is available before victim inference and adds no query overhead, but its theoretical motivation is local rather than universal.Source: §A.2.2 · p. 21

Algorithm 1 EvoSafety Co-Evolutionary Training

EvoSafety alternates attack and defense policy training while updating external attack and memory structures. Its appendix motivates prompt-side similarity for efficient gating and describes how heterogeneous attack knowledge becomes validated, reusable skills.

Algorithm 1 EvoSafety Co-Evolutionary Training

A.2.3. Adversarial Skill Library

  • Dual-Path Skill Grounding derives skills from source code when available, or reconstructs executable implementations from natural-language documentation before extraction.Source: §A.2.3 · p. 25
  • Tri-Axis Skill Validation checks functional attack success, shared schema across requests, and behavioral diversity from existing skills.Source: §A.2.3 · p. 25
  • The LogiBreak example reframes harmful requests in formal logic and requests practical implementation under an academic-research framing.Source: §A.2.3 · p. 27

A.3.1. Analysis of Reward Hacking

Adversarial-only defense training can produce refusal prefixes on benign requests, harming usability. Guard mode uses deliberately structured prefixes to distinguish adversarial from benign inputs for rule-based parsing.

A.3.1. Analysis of Reward Hacking

  • Adversarial-only defense training can induce refusal prefixes on benign inputs, substantially degrading usability.The paper illustrates the issue with a harmless GSM8K math question that the model would otherwise answer correctly.Source: §A.3.1 · p. 28
  • The GSM8K example shows an inappropriate safety prefix causing Llama-3.1-70B-Instruct to refuse a harmless question it otherwise answers correctly.Source: §A.3.1 · p. 28

A.3.2. Guard mode of Auxiliary Defense Model

  • Guard mode uses prefixes beginning with refusal-oriented phrases for adversarial prompts and “Please provide” for benign prompts, enabling rule-based parsing with almost no added latency.Source: §A.3.2 · p. 29

A.4. Efficiency Analysis of Coevolutionary Training

The section compares EvoSafety’s coevolutionary RL costs with direct coevolutionary training using a controlled proxy model. Its cost advantage depends on defense-side savings outweighing added attack-time prefix generation.

A.4. Efficiency Analysis of Coevolutionary Training

  • The cost comparison uses a first-order Transformer generation and policy-optimization proxy, abstracts away hardware and implementation effects, and omits retrieval overheads that are described as small relative to rollout and update costs.The proxy is intended for controlled scaling comparisons under matched rollout budgets, not as an exact wall-clock estimate.Source: §A.4 · p. 29
  • EvoSafety’s attack phase adds an auxiliary safety-prefix generation term relative to direct coevolutionary training.This term accounts for generating safety prefixes for sampled adversarial prompts.Source: §A.4 · p. 31
  • During defense optimization, EvoSafety updates only the auxiliary defense model while keeping the victim frozen, unlike direct training that updates the victim or defender.The stated defense-stage saving compares against a direct baseline with the same benign utility branch.Source: §A.4 · p. 31
  • EvoSafety is computationally favorable when savings from avoiding victim-model updates outweigh the added cost of generating auxiliary safety prefixes during attack optimization.The comparison assumes matched rollout budgets and compares the coevolutionary RL stage, excluding initialization costs such as supervised fine-tuning.Source: §A.4 · pp. 29–31
  • A 3B auxiliary defender paired with an 8B victim has substantially lower proxy update cost; the gap grows as victim models scale, including to 70B-class models.The defense-stage saving is expected when the auxiliary defender is substantially smaller than the victim and produces shorter safety prefixes.Source: §A.4 · p. 31

B.1. Attack Skill Library

  • The attack skill library draws initial skills from GPTFuzzer Yu et al. [2023] and CodeAttack Ren et al. [2024a], and includes held-out methods with and without released implementations.The construction is presented as demonstrating extraction of reusable skills from heterogeneous attack papers.Source: §B.1 · p. 31

B.2. Adversarial Training Data

The adversarial training data is filtered against HarmBench evaluation samples, while the benign defense-training set is curated to reduce reward noise from uncertain reasoning examples.

B.2. Adversarial Training Data

  • Samples whose maximum cosine similarity to HarmBench exceeds 0.88 are removed after filtering for exact duplicates and semantically overlapping evaluation examples.Similarity is computed using Qwen3-Embedding-8B Zhang et al. [2025].Source: §B.2 · p. 31
  • Table 6 reports the resulting sample counts for each dataset after filtering.The passage does not provide the table’s individual counts.Source: §B.2 · p. 31, Table 6 · p. 32

B.3. Benign Samples in RL-based Defense Training

  • The benign RL-defense dataset contains 1,000 GSM8K examples solved correctly in all 16 Llama-3.1-70B-Instruct reasoning rollouts.The authors select high-confidence examples to mitigate reward noise when low rewards may not reflect poor safety-prefix generation.Source: §B.3 · p. 32

B.4. Datasets for Evaluating General Reasoning Ability

The paper specifies evaluation sets for general reasoning and constructs supervised data to teach the attack model compositional skills. The SFT examples pair synthesized skill combinations with harmful requests.

B.4. Datasets for Evaluating General Reasoning Ability

  • MMLU evaluation uses 10 randomly sampled examples from each of 57 subtasks, while GSM8K uses its full test set for two victims and a 500-example sample for Llama-3.1-70B-Instruct.The full GSM8K test set is used for Qwen3-4B-Instruct-2507 and Llama-3.1-8B-Instruct.Source: §B.4 · p. 32

B.5. SFT Data Construction for the Attack Model

  • The attack-model SFT set contains 20,000 examples created by pairing synthesized compositional skills with harmful requests from AdvBench Chen et al. [2022].The skills are synthesized by enumerating pairwise combinations of GPTFUZZER Ren et al. [2024a] skills using the GPT-4.1 API.Source: §B.5 · p. 32
  • The illustrated compositional attack combines two skills into an adversarial prompt, contrasting it with prompts constructed from each individual skill.The example demonstrates the intended skill-composition data construction rather than an evaluation result.Source: §B.5 · pp. 32–33

B.6. SFT Data Construction for the Defense Model

The appendix describes defense-model distillation and evaluation procedures, alongside an inference-latency measurement. It also notes a spurious-refusal issue for smaller auxiliary defenders when retrieved adversarial prompts appear in their templates.

B.6. SFT Data Construction for the Defense Model

  • Smaller auxiliary defenders can over-activate safety mechanisms when their prompt template includes retrieved adversarial prompts, causing spurious refusals on benign queries.The authors construct defense-model SFT data from filtered adversarial examples and MMLU training examples, using Llama-3.1-70B-Instruct safety prefixes as targets for a Llama-3.2-3B-Instruct model.Source: §B.6 · pp. 33, 35

B.7. Prompt Template for LLM-as-a-Judge Evaluation of Response Harmfulness

  • The harmfulness judge uses a five-level rubric, with score 1 for refusal or redirection and score 5 for direct fulfillment of a policy-violating instruction.The rubric describes score 1 as the preferred response to policy-violating instructions.Source: §B.7 · p. 36

B.8. Prompt Template for LLM-as-a-Judge Evaluation of General Question Accuracy

  • The general-accuracy judge evaluates only the final answer, treating incomplete or ambiguous answers as incorrect and allowing stated algebraic, numeric, set/list, or unit equivalences.The evaluation protocol asks the judge to extract the model’s final answer and compare it with the correct answer.Source: §B.8 · pp. 36–37

C.1. Inference-Time Efficiency

  • The 3B defense policy has latency generally around 2.5 ms per input prompt on an H100 GPU.The evaluation measures input token lengths and inference latency using adversarial prompts collected from Jailbreak-R1 [Guo et al., 2025b].Source: §C.1 · p. 37

C.2. [Skill Validity]: How Effective is the Automated Skill Generation Pipeline?

The skill-generation evaluation finds that constructed skills preserve much of their source methods’ attack effectiveness, while induced variants remain competitive and can sometimes improve ASR@1. Case studies also show the attack policy composing novel strategies and the defense policy generating query-responsive benign answers and refusals to adversarial prompts.

C.2. [Skill Validity]: How Effective is the Automated Skill Generation Pipeline?

  • Constructed skills preserve a large fraction of source-method effectiveness, despite a consistent performance gap.Table 7 reports results averaged over all extracted or induced skills when methods yield multiple skills.Source: §C.2 · p. 38, Table 7 · p. 38
  • Induced skills remain competitive and occasionally outperform their seeds at ASR@1, supporting iterative exploration without collapse in attack efficacy.The stated goal is broader adversarial-prompt coverage, not perfect replication of original methods’ ASR.Source: §C.2 · p. 38

D. Case Studies

  • Table 8 reports accuracy for different defense methods across multiple reasoning benchmarks.The accompanying text describes a breakdown of victim-model performance degradation on those benchmarks; no cell values are provided here.Source: §C.3 · p. 38, Table 8 · p. 39

D.1. Attack Policy

  • The policy preferentially exploits the more effective of two retrieved skills, and its generated prompt retains a successful skill’s narrative structure rather than a failed skill’s code-like style.The example contrasts Skill I, which fails, with Skill II, which succeeds.Source: §D.1 · p. 39
  • The policy combines multi-role narratives with opposing groups and divided responsibilities, producing a strategy absent from its training skills.The authors interpret this example as composition and generalization rather than simple template copying.Source: §D.1 · p. 39

D.2.1. Generated Safety Prefix for Benign Queries

  • For a benign query, the generated prefix directly addresses the question instead of merely giving a generic instruction to be helpful.The example prefix calculates the answer to the duck-egg earnings question.Source: §D.2.1 · pp. 42–43

D.2.2. Generated Safety Prefix for Adversarial Prompts

  • For an adversarial prompt, the prefix identifies role-play and hypothetical-scenario jailbreak tactics, and the victim model refuses to provide the requested harmful response.The shown prefix warns against the bypass strategy and directs refusal.Source: §D.2.2 · pp. 43–44

D.2.3. Comparison of Defense Policies With and Without Memory Retrieval

The comparison attributes a difference in defense-prefix behavior to memory retrieval: without it, prefixes can recommend rejection but cannot compare a prompt with successful past attacks. Retrieval enables the prefix to identify similarities and associated strategies, and the paper reports stronger activation of the victim model’s endogenous defenses.

D.2.3. Comparison of Defense Policies With and Without Memory Retrieval

  • Without memory retrieval, prefixes still instruct rejection but cannot compare a current prompt with previously successful attacks.Source: §D.2.3 · p. 44
  • With retrieval, prefixes can identify similarities to prior attacks and report the associated jailbreak strategy.Source: §D.2.3 · p. 44
  • The authors report that retrieval-informed prefixes are more effective at activating the victim model’s endogenous defense mechanisms.Source: §D.2.3 · p. 44

E. Limitations

The evaluation focuses on standalone LLMs rather than autonomous agents with tool-use capabilities, and the framework does not yet support multimodal inputs. The authors leave extensions to more complex agentic settings for future work.

E. Limitations

  • The evaluation primarily covers standalone LLMs, not autonomous agents equipped with tool-use capabilities.Source: E. Limitations · p. 48
  • The framework does not yet support multimodal inputs, restricting its applicability to vision-enabled models such as browser-use agents.Source: E. Limitations · p. 48
  • Extending the framework to these more complex agentic settings is left for future work.Source: E. Limitations · p. 48

Related papers

Loading 2605.13411v1…