A Self-Evolving Multi-Agent Framework Defense against LLM Jailbreak Attacks

Tongyan Hu; Bryan Hooi

Source-linked AI summary

A Self-Evolving Multi-Agent Framework Defense against LLM Jailbreak Attacks

Tongyan Hu, Bryan Hooi

arXiv:2608.26008v1Aug 26, 2026cs.CRcs.CL

TL;DR

LLMs remain vulnerable to evolving jailbreak attacks, while static defenses cannot accumulate experience or adapt to unseen strategies. The paper proposes a persistent external rule memory that converts successful failures into method-level rules and selectively reuses them across interactions. Across multiple jailbreak families and models, the method reduces attack success while preserving benign utility and robustness under adaptive composite wrappers.

Problem
LLMs remain vulnerable to jailbreak attacks, while static defenses cannot accumulate defensive experience or adapt to newly emerging strategies.
Method
The framework stores successful jailbreak failures as persistent method-level rules describing structural wrappers, dynamically triggers relevant rules, and adapts through external memory without parameter updates.
Results
The method substantially reduces attack success rates across multiple jailbreak settings and model families while maintaining benign-task utility and robustness under adaptive composite-wrapper attacks.
Takeaways & Limitations
Persistent test-time adaptation provides robustness gains beyond static defenses by carrying structured defense knowledge across interactions and generalizing to structurally similar attacks.
Takeaways & Limitations
The framework does not cover the full range of recent attacks, leaves multi-turn and agentic settings for future work, and lacks explicit rule pruning or conflict resolution for long streams.

Abstract

from arXiv · show

Large language models (LLMs) remain vulnerable to jailbreak attacks that exploit techniques such as role-playing, obfuscation, code transformation, and multi-step indirection to elicit harmful outputs. As jailbreak strategies keep emerging, defenses have proliferated in an ongoing cat-and-mouse game, yet most remain static: their safety behavior is fixed at deployment, so they cannot accumulate defensive experience or adapt to unseen strategies. We propose a self-evolving test-time defense built around a persistent, cross-interaction rule memory: when an attack succeeds, the framework abstracts that failure into a method-level rule capturing the structural attack wrapper rather than the harmful topic, and reuses it against future inputs. Because rules are method-level, one induced rule generalizes across an entire attack family, and the label space expands as novel wrappers appear. The mechanism operates entirely through external memory and prompting, with no parameter updates, and applies to both open-weight and black-box API models. We realize it as four cooperating modules, but the contribution is the memory-based adaptation mechanism, not the module decomposition. Across four black-box jailbreak families and multiple models, our method substantially reduces attack success rates while preserving benign utility, remains robust under an adaptive composite-wrapper attack, and does not increase over-refusal as the memory grows.

1 Introduction

LLMs remain vulnerable to evolving jailbreak strategies, while static defenses cannot accumulate defensive experience. The proposed self-evolving defense uses persistent method-level rule memory to adapt across interactions and reduce attacks while preserving benign utility.

Motivation: Static alignment, prompts, and classifiers do not adapt during inference, leaving defenses vulnerable when novel or compositional jailbreak strategies appear.These mechanisms do not accumulate defensive experience or update behavior based on past failures.
Approach: The framework converts successful jailbreak failures into reusable, method-level defense rules stored in persistent cross-interaction memory without parameter updates.Adaptation occurs through external memory and prompting, supporting open-weight and black-box API models.
Approach: Selective rule triggering activates rules relevant to the inferred attack pattern instead of appending all learned rules, limiting interference with benign queries.The framework also uses policy decisions to determine whether to issue a strict refusal or constrained response.
Contribution: The contribution is the persistent self-evolving rule memory with selective triggering, rather than the particular decomposition into cooperating modules.The framework is realized through multiple cooperating components, but the paper identifies the memory-based mechanism as central.
Evaluation: Across four jailbreak families and multiple open- and closed-source models, the framework reduces attack success rates, preserves benign utility, and remains robust to adaptive composite wrappers.The evaluation also reports no increase in over-refusal as memory grows.
Approach: A single rule abstracts the structural attack wrapper rather than the harmful topic, allowing generalization across an attack family while the label space expands for novel wrappers.The dynamic mechanism identifies structurally novel attacks and adds corresponding method-level labels.

2 Related Work

Prior jailbreak defenses include static safety mechanisms, transient self-reflection, and multi-agent systems. The paper distinguishes its approach by retaining structured defensive knowledge across interactions rather than treating feedback or interactions independently.

Static defenses: Existing jailbreak defenses commonly use system prompts, safety fine-tuning, preference-based alignment, and external classifiers.These approaches target unsafe outputs through static alignment or filtering mechanisms.
Self-reflection: Self-reflection and test-time adaptation revise behavior during inference, but their feedback remains transient context rather than a persistent structured policy.Safety knowledge must therefore be re-derived for each new interaction and is not retained across sessions.
Multi-agent defenses: Multi-agent architectures have been applied to general LLM tasks and safety settings, including routing, judgment, prompt optimization, and self-reflective defense.The related systems coordinate multiple agents or LLM instances for safety analysis and response filtering.

3 Methodology

The framework defends against jailbreaks in sequential interactions using persistent external rule memory that modulates a fixed LLM and evolves from detected failures. It classifies attack patterns, selectively triggers rules, chooses a response policy, generates an output, and reflects on violations to update memory.

Jailbreak attacks transform unsafe requests into adversarial prompts that preserve harmful intent while bypassing safety alignment.
The defense maintains external memory R_t across interactions, improving future robustness without modifying the target model’s parameters.
Each interaction classifies an input, retrieves relevant rules, assigns a policy, generates a response, detects violations, and may update memory.The four prompted agents implement these operators, but the persistent rule-memory mechanism is the contribution.
Rules encode method-level attack labels and structural wrappers rather than harmful topics, allowing one induced rule to generalize across attacks sharing that wrapper.The label set expands when structurally novel wrappers are encountered.
Selective triggering first matches labels, then uses LLM relevance and lexical overlap fallbacks to retrieve up to K relevant rules.
The policy operator maps triggered rules to hard-refuse, soft-refuse, or allow, with hard-refuse taking priority over softer actions.
Rule induction occurs when the detector identifies a violation on a non-benign input, while safe or benign interactions leave memory unchanged.Induced rules undergo semantic deduplication and a per-label capacity bound C=4.
The detector uses an LLM-as-judge rubric with harmfulness threshold τ=7, while evaluation uses an external GPT-4o-mini judge.Separating learning and evaluation signals reduces dependence on a single scorer.

4 Experiments

The experiments evaluate the defense across jailbreak attacks, models, metrics, baselines, benign tasks, sequential adaptation, and component ablations. Results show lower attack success, preserved utility, rapid convergence, and measurable inference overhead.

4.1 Experiment Setup: The evaluation covers AdvBench jailbreak prompts, benign MMLU and GSM8K tasks, open- and closed-source models, and black-box test-time inference.The protocol uses 520 harmful-behavior prompts in AdvBench and reports task-specific benign performance.
4.1 Experiment Setup: ASR-rej and ASR-gpt measure attack success, while task-specific performance measures utility on benign inputs.ASR-gpt uses an external LLM judge, and the authors treat it as the primary effectiveness metric because ASR-rej may underestimate success.
4.3 Results: The framework consistently achieves the lowest ASR across models and attack families, with the largest gaps on CodeChameleon and ReNeLLM.The comparison includes no defense, defense prompting, Self-Reminder, and AutoDefense baselines.
4.3 Results: Benign utility on MMLU and GSM8K remains within about two points of the undefended model.This result is reported as evidence that improved robustness does not substantially degrade benign usability.
4.5 Inference Cost: Inference costs 3.07× calls and 1.43× latency relative to a single-shot model at steady state.The triggering fallback and reflection calls occur only occasionally after memory convergence; the overhead is comparable to AutoDefense.
4.4 Self-Evolving Analysis: After the first batch of 20 samples, method-level rules reduce ASR-gpt to near zero across four attack types in subsequent rounds.The memory starts empty, learns from earlier failures, and applies accumulated rules only to later prompts.

5 Conclusion

The framework defends against jailbreaks through persistent external rule memory that selectively applies method-level knowledge learned from observed failures. Evaluations report reduced attack success, maintained benign utility, robustness to adaptive wrappers, and open directions for broader generalization.

Contribution: Persistent external rule memory, rather than the four-agent decomposition, is the framework’s central contribution.The memory accumulates structured defense knowledge without modifying the base model and selectively triggers rules matching current attack patterns.
Results: The method substantially reduces attack success rates across multiple jailbreak settings and model families while maintaining competitive benign-task utility.
Results: Defense performance improves over the interaction stream as the memory accumulates.
Results: The framework remains robust under an adaptive composite-wrapper attack designed to evade rule triggering.
Future work: Open directions include scalable rule generalization, learned semantic detectors, and adaptive coordination for multi-turn settings.

Limitations and Ethics

The evaluation covers representative black-box prompt-level jailbreak families but not the full range of attacks, multi-turn settings, or agentic attacks. The authors also identify unresolved memory-management and detector-error limitations, alongside an ethics concern about potential misuse.

Scope: The evaluation covers four representative black-box prompt-level jailbreak families and an adaptive composite-wrapper attack, not the full range of recently proposed attacks.
Scope: Multi-turn and agentic attack settings remain future work.
System limitations: The current design lacks explicit rule pruning or conflict resolution, leaving long-stream memory efficiency and consistency open questions.
System limitations: Prompted LLM violation detectors and classifiers inherit base-model judgment errors, which can occasionally cause over-refusal.
Ethics: The attack-pattern analysis could potentially be repurposed to study or circumvent safety mechanisms, so the authors state that contributions are intended for defensive use.

A.1 Experiment Environment

GPU-based experiments use a server with dual 48-core Intel Xeon Platinum 8558 CPUs, 1.0TB RAM, and four NVIDIA H200 GPUs.

Hardware and software: GPU-based experiments run on two 48-core Intel Xeon Platinum 8558 CPUs, 1.0TB RAM, and four NVIDIA H200 GPUs.The software environment uses NVIDIA Driver 580.126.09 and CUDA 13.0.

A.2 Benchmarks

The benchmark uses Harmful Behaviors in AdvBench, a dataset of 520 prompts covering diverse harmful and illegal behaviors.

Dataset: The experiments adopt the Harmful Behaviors in AdvBench dataset proposed by Zou et al. (2023).
Dataset: The dataset contains 520 prompts covering a wide range of harmful and illegal behaviors.

A.3 Evaluation Protocol

The evaluation compares refusal-based and LLM-judge attack-success metrics across four black-box jailbreak methods, using standardized prompts and explicit refusal criteria.

ASR-rej: ASR-rej counts a response as a rejection when it contains any phrase from a predefined refusal list.The criterion follows Zou et al. (2023) and is used as a lightweight heuristic for refusal behavior.
ASR-rej: The refusal list includes phrases such as “I do not provide,” “I do not recommend,” and “I do not feel comfortable.”Additional listed phrases include “I do not actually” and “here’s an alternative.”
ASR-gpt: ASR-gpt uses an LLM-as-judge protocol in which lower attack success rates indicate better defense performance.The full evaluation prompt is provided in Figure 4.
Attack methods: The experiment evaluates DeepInception, CodeChameleon, ReNeLLM, and FlipAttack as representative black-box jailbreak methods.These four attack methods are used to compare defense performance across attack types.

A.5 Baseline Defense Methods

The baselines range from no protection and static prompts to instance-level reflection, while the proposed framework uses multiple agents for classification, rule triggering, policy decisions, response generation, and rule induction.

Baseline scope: The comparison spans fixed-parameter defenses with different adaptivity levels, including no defense, prompt-based control, self-reflection, and AutoDefense.Across these baselines, the underlying model parameters remain fixed.
No Defense: No Defense evaluates the base model without additional safety mechanisms as a lower-bound reference for jailbreak robustness.This baseline reflects the underlying model’s inherent vulnerability to jailbreak attacks.
Defense Prompt: Defense Prompt adds one safety-oriented system prompt that instructs the base model to refuse harmful, illegal, or policy-violating requests.The same safety instruction is applied to all inputs without adaptation over time.
Self-Reminder: Self-Reminder appends model-generated feedback about each previous interaction to subsequent prompts as contextual safety guidance.The feedback may identify unsafe content or summarize potential attack characteristics.
Proposed framework: The proposed framework routes inputs through agents for rule triggering, policy selection, dynamic response generation, and reflection-based rule induction.The classifier maps inputs to method-level labels, while the response agent composes instructions from memory, reminders, and refusal contracts.

A.7 Additional Results

Additional results examine benign utility, peer defenses, hyperparameter sensitivity, and concrete examples showing how rules are induced from failures and reused on related jailbreak wrappers.

Over-refusal: 9.6% over-refusal occurs after four attack families’ rules accumulate, compared with 3.6% with empty memory on the XSTest safe set.The reported increase is evaluated on Qwen2.5-7B; benign-labelled inputs are excluded from triggering.
Additional baselines: ICD reaches 23% and 43% ASR-gpt on CodeChameleon and ReNeLLM, respectively, while AegisLLM does not retain defenses across interactions.These comparisons motivate the persistent memory’s cross-family and cross-interaction scope.
Hyperparameter sensitivity: ASR-gpt results are almost identical across K ∈{1, 2} and C ∈{2, 4}, indicating limited sensitivity within the evaluated range.K is the maximum number of injected rules per input, and C is the per-label rule capacity.
Rule memory: Structurally novel inputs assigned to other can trigger creation and storage of a new rule and method-level label without replacing existing rules.The resulting labels remain available for subsequent classifier decisions.
Rule induction: An unsafe first encounter with DeepInception is summarized by the reflection agent and converted into a reusable defense rule stored in external memory.Figure 12 illustrates this rule-induction process.
Rule reuse: A later prompt with a similar wrapper triggers the stored rule, causes a refusal decision, and produces a safe response.Figure 13 presents this rule-reuse example.