PaperlayerView exact revision on arXiv

Source-linked AI summary

MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safety

Xiaoyu Wen, Zhida He, Han Qi, Ziyu Wan, Zhongtian Ma, Ying Wen, Tianhang Zheng, Xingcheng Xu, Chaochao Lu, Qiaosheng Zhang

arXiv:2602.01539v2cs.AIcs.CLcs.LGcs.MA

TL;DR

Existing defenses struggle to keep pace with evolving adversarial prompts because they rely on static data and face limitations in attacker reasoning and agent optimization. MAGIC formulates safety alignment as an asymmetric sequential game with co-evolving attacker and defender agents, supported by an Attack Pool Benchmark with diverse CoT rewriting strategies. Experiments report improved defense success rates, preserved helpfulness, and novel compositional attack strategies that uncover dynamic threats beyond human-crafted templates.

  • Problem

    Existing red-teaming methods rely on static datasets or heuristics, lack reasoning for complex multi-turn deception, and can suffer optimization conflicts in shared-parameter self-play.

  • Method

    MAGIC uses an online multi-turn MARL framework with decoupled attacker and defender objectives in an asymmetric sequential game, initialized with diverse CoT rewriting strategies.

  • Results

    MAGIC significantly improves defense success rates while preserving model helpfulness, and adversarial co-evolution produces novel compositional attack strategies beyond human-crafted templates.

  • Takeaways & Limitations

    The framework supports robust refusal behaviors that generalize beyond static adversarial data while enabling attackers to uncover diverse long-tail vulnerabilities.

  • Takeaways & Limitations

    The paper’s motivating constraints include scarce high-quality offensive CoT data and optimization conflicts caused by shared attacker-defender parameters in prior approaches.

Abstract

from arXiv · show

Ensuring robust safety alignment is crucial for Large Language Models (LLMs), yet existing defenses often lag behind evolving adversarial attacks due to their \textbf{reliance on static, pre-collected data distributions}. In this paper, we introduce \textbf{MAGIC}, a novel multi-turn multi-agent reinforcement learning framework that formulates LLM safety alignment as an adversarial asymmetric game. Specifically, an attacker agent learns to iteratively rewrite original queries into deceptive prompts, while a defender agent simultaneously optimizes its policy to recognize and refuse such inputs. This dynamic process triggers a \textbf{co-evolution}, where the attacker's ever-changing strategies continuously uncover long-tail vulnerabilities, driving the defender to generalize to unseen attack patterns. Remarkably, we observe that the attacker, endowed with initial reasoning ability, evolves \textbf{novel, previously unseen combinatorial strategies} through iterative RL training, underscoring our method's substantial potential. Theoretically, we provide insights into a more robust game equilibrium and derive safety guarantees. Extensive experiments validate our framework's effectiveness, demonstrating superior defense success rates without compromising the helpfulness of the model. Our code is available at https://github.com/BattleWen/MAGIC.

1. Introduction

MAGIC frames LLM safety alignment as an asymmetric, sequential attacker–defender game that co-evolves attacks and defenses. It combines decoupled optimization with an Attack Pool Benchmark to improve robustness against evolving, multi-turn threats while preserving helpfulness.

  • Existing defenses struggle to keep pace with evolving adversarial prompts, motivating continuous discovery and defense against novel attack strategies.
  • Static datasets and heuristic mutations bypass simple filters but lack the offensive reasoning needed for complex multi-turn deception.The shortage of high-quality offensive Chain-of-Thought data limits non-templated attack patterns.
  • MAGIC models safety alignment as an asymmetric adversarial interaction between separately optimized attacker and defender agents that co-evolve through iterative interactions.The framework is designed to promote progressively more challenging attacks and robust defensive responses.
  • The Attack Pool Benchmark provides CoT completions across 20 rewriting strategies, giving attackers initial reasoning capabilities for exploring long-tail vulnerabilities.It addresses data scarcity and cold-start issues in automated red-teaming.
  • Experiments across single-turn and multi-turn benchmarks show improved defense success rates while preserving helpfulness, alongside novel compositional attack strategies.The reported findings indicate that co-evolution uncovers dynamic threats beyond human-crafted templates.

2. Related Work.

Prior LLM red-teaming evolved from static human-crafted jailbreaks toward automated attacks, while safety alignment increasingly emphasizes intrinsic defenses and multi-agent games. MAGIC is positioned within this shift because earlier approaches remain vulnerable to complex adversarial rewriting and prompt injection.

  • Early jailbreaks used human-crafted role-playing and surface-level obfuscation, but these approaches were static and difficult to scale.
  • Safety alignment has shifted from post-deployment filtering and external guardrails toward intrinsic alignment because static boundaries can be bypassed.
  • Multi-agent game formulations emerged as a response to adversarial dynamics, with prior work addressing related safety and red-teaming problems.The supplied passage situates MAGIC among this broader research direction without detailing individual methods.
  • The related-work discussion reports that the proposed framework performs well in both single-turn and multi-turn scenarios without compromising helpfulness.

3. Problem Formalization

MAGIC formalizes language-model red-teaming as a two-player sequential game in which an attacker acts first and a defender responds conditionally. The analysis uses subgame-perfect equilibrium to require safe, pointwise defensive responses across attacker prompts.

  • The attacker selects y_A first, after which the defender observes it and selects y_D; each receives a role-specific reward.The defender’s strategy is represented as a conditional distribution π_D(·|y_A).
  • Safety is defined by r_D(y_A, y_D) ≥0, whereas unsafety is defined by r_D(y_A, y_D) < 0.
  • SPNE requires the defender to choose an optimal response in every subgame for each attacker prompt, rather than only for the expected prompt.
  • The attacker’s strategy must be optimal given the defender’s best-response behavior, completing the sequential equilibrium conditions.
  • As policies converge to the SPNE, the defender consistently produces safe responses regardless of the attacker’s prompt.
  • Assuming a unique equilibrium, or applying tie-breaking otherwise, the learning algorithm aims to approach an SPNE where safe responses are guaranteed.The stated guarantee relies on the existence of a rejection or safe fallback action for every attacker prompt.

4. Methods

MAGIC models safety alignment as an asymmetric sequential game, using alternating optimization to co-evolve an attacker and defender toward robust responses. It initializes attacker reasoning with SFT, then applies GRPO-based reinforcement learning with separate objectives and reward components.

  • Game formulation: SPNE offers pointwise safety guarantees, but MAGIC approximates the intractable equilibrium through alternating best-response optimization.The method fixes one policy while updating the other, iteratively simulating best-response dynamics.
  • Training process: MAGIC uses two phases: attacker initialization through SFT and iterative co-evolution through reinforcement learning.The attacker is warmed up on CoT-enriched adversarial data before attacker and defender policies are alternately optimized.
  • Training process: SFT on an Attack Pool Benchmark addresses the attacker's cold-start refusal problem and provides initial capabilities for discovering vulnerabilities.The benchmark enriches harmful-query data with CoT completions, training the attacker to generate adversarial prompts from seed queries.
  • Iterative co-evolution: During defender optimization, the attacker is frozen while the defender samples diverse responses and maximizes safety rewards toward a pointwise best response.The defender receives generated adversarial inputs and updates its policy using grouped responses.
  • Iterative co-evolution: During attacker optimization, the defender is frozen as a response oracle while the attacker updates its policy to anticipate the defender's reaction.The attacker generates candidate prompts from seed queries and is optimized with the corresponding GRPO objective.
  • Reward design: MAGIC combines format, harmfulness, and refusal rewards while applying GRPO to decoupled attacker and defender objectives.The refusal reward penalizes both refusing safe queries and answering unsafe queries, while the format reward is applied only to the attacker.

5. Experiments

MAGIC is evaluated across safety, benign compliance, general capabilities, out-of-distribution attacks, and multi-turn jailbreaks. Results show substantial safety gains, preserved helpfulness and general capabilities, transferable robustness, and novel attacker strategies emerging through co-evolution.

  • Main results: MAGIC reduces the Attack Success Rate on WildGuardTest from 36.5% to 2.3% for Qwen2.5-7B-Instruct.The method also shows consistent safety improvements across multiple benchmarks, model families, and scales.
  • Main results: MAGIC maintains strong benign compliance without introducing excessive refusals on adversarially constructed but non-harmful prompts.The reported results indicate that safety improvements are not obtained through overly conservative refusal strategies.
  • Generalization: MAGIC consistently reduces ASR across diverse out-of-distribution attackers on HarmBench, including PAIR, TAP, GCG, AutoDAN, and AutoDAN-turbo.This evaluates defender generalization to attack strategies not used as the sole in-distribution interaction.
  • General capabilities: MAGIC largely preserves general capabilities across instruction following, reasoning, and general knowledge benchmarks, with differences from instruction-tuned baselines generally remaining within a narrow range.The evaluation covers IFEval, ARC-C, GPQA, MMLU, and AlpacaEval2 across model families and scales.
  • Multi-turn evaluation: MAGIC improves robustness under X-Teaming multi-turn evaluation without additional multi-turn attacker SFT, using only a lightweight system-prompt modification.The attacker conditions later actions on prior defender replies, and the reported results indicate transferable adversarial dynamics.
  • Attacker evolution: Iterative reinforcement learning enables the attacker to evolve novel, previously unseen combinatorial attack strategies.The paper links this emergence to composing previously learned atomic skills during co-evolution.

6. Conclusion

MAGIC frames safety alignment as an asymmetric sequential game with co-evolving attacker and defender agents. The framework improves safety robustness while largely preserving benign compliance and instruction-following capabilities, but introduces computational and training-management costs.

  • MAGIC uses asymmetric sequential-game reinforcement learning to co-evolve an attacker that uncovers vulnerabilities and a defender that learns robust refusal behaviors.The attacker and defender have decoupled objectives, enabling iterative adversarial interaction beyond static adversarial data.
  • Theoretical analysis and empirical results support improved safety robustness while largely preserving benign compliance and instruction-following capabilities.
  • Online adversarial training introduces computational overhead and requires careful management of training dynamics.
  • Attacker exploration is sensitive to SFT initialization and the attacker model’s capacity.

A. Proof of Theorem 3.2

The proof establishes safety guarantees for the defender at equilibrium in a sequential red-teaming game. Under a safe fallback-action assumption, every response in the equilibrium policy’s support is safe, strengthening expectation-level guarantees.

  • The paper defines safety as nonnegative defender reward and models red-teaming as a two-player sequential game governed by SPNE.SPNE requires optimal defender responses in each subgame for every attacker action, rather than only in expectation over attacker behavior.
  • Assuming every attacker action has a rejection or safe fallback response with nonnegative reward, any SPNE guarantees nonnegative defender reward.
  • The defender’s equilibrium value is the maximum expected reward over response policies conditioned on a given attacker action.
  • The theorem strengthens expectation-level safety to instance-level safety: every response sampled from the equilibrium defender policy is safe with certainty.This contrasts with guarantees based only on average reward, where individual sampled responses may still violate safety.

C. Experimental Details

The experimental pipeline builds on ReMA2 and compares MAGIC with self-play, test-time defense, and instruction-tuned baselines. The setup emphasizes reproducibility through specified software versions and benchmark protocols.

  • MAGIC’s training pipeline is built on the ReMA2 cooperative multi-agent reinforcement learning framework.
  • The experiments use specified vLLM versions because version differences can affect training and inference performance and may cause memory leakage.
  • The baselines include Self-RedTeam, SmoothLLM, and Self-Eval, covering shared-parameter self-play and inference-time defenses.Self-RedTeam alternates attacker and defender roles within one model, whereas SmoothLLM and Self-Eval avoid adversarial co-training.
  • SmoothLLM generates perturbed prompt copies and selects outputs through majority voting, using 10 perturbed copies by default.
  • The evaluation uses the Ai2 Safety Tool, OLMES, and OpenRT frameworks for safety, general capability, and automated red-teaming assessments.

D.1. Benchmarks on Safety Evaluation

The safety evaluation spans standardized benchmarks covering harmful refusal, adversarial robustness, over-refusal, moderation, and multi-turn jailbreaks. These benchmarks measure both protection against harmful requests and calibration on benign or contrasting prompts.

  • HarmBench evaluates robustness using 320 vanilla harmful prompts and 1,500 adversarial harmful prompts sampled across 15 attack methods.Lower attack success rate on the adversarial subset indicates stronger robustness.
  • OR-Bench-Toxic measures safety across 655 toxic prompts spanning 10 common harmful-content categories.
  • WildGuardTest jointly assesses malicious intent in prompts, response safety risk, and refusal detection on vanilla and adversarial harmful subsets.
  • WildJailbreak evaluates 2,000 adversarial harmful prompts and 250 adversarial benign prompts to study jailbreak tactics and over-refusal.
  • XSTest pairs 250 benign prompts with 200 harmful contrast prompts to measure refusal calibration and exaggerated safety behavior.
  • X-Teaming evaluates scalable multi-turn jailbreak scenarios using three-turn interactions and strategy pools generated from 158 HarmBench behaviors.

D.3. Automated Red-teaming Evaluation

MAGIC is evaluated against automated red-teaming attacks using standardized harmful-prompt settings and multiple defender categories. Across additional model families, it generally achieves the lowest attack success rates, indicating robustness to dynamic interactive attacks.

  • Evaluation setup: The evaluation uses OpenRT with 320 HarmBench vanilla harmful prompts and attacks from GCG, PAIR, TAP, AutoDAN, and AutoDAN-turbo.Attacker LLMs are fixed to Qwen2.5-7B-IT where required for fair comparison.
  • Evaluation setup: Defender comparisons include Gemini-2.5-Flash, instruction-tuned open models, Self-RedTeam, SmoothLLM, post-processing defenses, and MAGIC.
  • Reward-model analysis: GPT-4o and Qwen3Guard produce similar ASR estimates on HarmBench vanilla prompts: 25.625% versus 24.063%.GPT-4o uses a dense 0–5 harmfulness score, while Qwen3Guard makes binary predictions.
  • Generalization: MAGIC consistently achieves the lowest ASR among most compared defenses across white-box and black-box attacks on Qwen2.5-14B-IT and Llama3.1-8B-IT.The result is presented as evidence that co-evolution remains robust under dynamic interactive jailbreak attempts rather than a fixed attack distribution.

E. Transferability of Attacker’s Jailbreak Capability

The transferability study tests whether MAGIC-attacker’s learned jailbreak behaviors generalize across defender backbones and whether MAGIC-defender remains robust to them. Using single-rollout rewrites, the learned attacker increases ASR broadly while the co-trained defender is particularly resilient to it.

  • Transfer protocol: After RL, MAGIC-attacker encodes jailbreak strategies in its weights and needs only moderately high sampling temperature to elicit diverse behaviors.The attacker does not require a hand-crafted strategy prompt at test time.
  • Scope: The evaluation does not directly compare against GCG or TAP test-time scaling methods because their iterative searches require higher latency and compute than MAGIC-attacker’s single rollout.
  • Transfer protocol: The study compares Qwen2.5-7B-IT with MAGIC-attacker across open, safety-aligned, proprietary, and MAGIC-trained defenders.
  • Transfer protocol: The protocol samples 600 WildJailbreak test prompts, rewrites each seed once at temperature 0.0, and evaluates attack success with GPT-4o.
  • Results: MAGIC-attacker substantially increases ASR across base defenders, indicating transfer across model families rather than overfitting to one target.
  • Results: MAGIC-defender shows stronger robustness to both attackers and particular resilience to MAGIC-attacker because it was co-trained against that rewrite distribution.

F.3. Evolution of Attack Pattern in RL Training

The attack-pattern analysis examines how RL changes the attacker’s strategy distribution beyond its initialization. Co-evolutionary training broadens and reshapes the repertoire, producing combinational and previously absent attack styles.

  • Distribution evolution: For Attacker-base, RL reallocates probability toward dominant decision-simulation and reasoning-intensive mechanisms while reducing several early-stage styles.
  • Distribution evolution: For Attacker-SFT, RL shifts the initial concentrated SFT distribution toward a broader mixture, including new variants, while suppressing patterns that become less competitive.
  • Conclusion: Overall, co-evolutionary RL discovers novel attack patterns rather than merely amplifying the attacker’s initial templates.
  • Distribution evolution: Figure 4 compares fine-grained strategy distributions during RL for base-initialized and SFT-initialized attackers without encoding-style rewrites.

G. Ablation Study

The ablation study identifies a trade-off among harmful refusal, benign compliance, and general capability. The full MAGIC-sft configuration provides the most favorable balance by combining co-evolution with a capable SFT-initialized attacker.

  • Study scope: The ablations evaluate safety and general capabilities across Qwen2.5-7B-IT and Llama3.1-8B-IT.The study frames the comparison around harmful refusal, benign compliance, and general instruction following.
  • Ablations: No-Game produces extremely strong harmful-prompt refusal but sharply reduces acceptance of adversarially structured benign prompts.The resulting behavior is characterized as a conservative refuse-by-default heuristic.
  • Ablations: Defender-only training partially mitigates unknown-adversarial-benign failures but reduces benign helpfulness and general instruction following.The passage specifically notes lower AlpacaEval 2 scores and lower vanilla-benign compliance on XSTest.
  • Ablations: MAGIC-base improves over non-game baselines but can retain imbalance between calibrated refusal and benign compliance.Instruction-tuned attackers may be partially uncooperative, especially for already safety-aligned backbones.
  • Ablations: MAGIC-sft yields the most favorable trade-off, preserving harmful-prompt robustness alongside strong benign compliance and general instruction following.Its stronger, more diverse rewrites support co-evolution and discourage an always-refuse strategy.
Loading 2602.01539v2…