AdvEvo-MARL: Shaping Internalized Safety through Adversarial Co-Evolution in Multi-Agent Reinforcement Learning

Motivation

• The framework targets weaknesses of self-verification and external guards, including limited detection of cross-agent attacks, single-point failure, computational overhead, and poor adaptive generalization.Source: §1 · pp. 1–2

Method

• AdvEvo-MARL co-evolves jailbreak-generating attackers and task-performing defenders so safety is learned within agents rather than supplied by separate guards.Attackers are initialized with supervised fine-tuning and then refine adversarial prompts while defenders resist attacks and preserve assigned tasks.Source: §1 · p. 2

Results

• Experiments cover agent manipulation, message corruption, and user instruction hijacking, demonstrating robustness with limited task-performance loss.Source: §1 · p. 2

Method

• A public baseline uses each functional group’s mean return for advantage estimation, promoting collaborative learning and more stable policy updates.Source: §1 · p. 2

Results

• 18.33% maximum safety improvement is reported across representative multi-agent attack settings, while further evaluations show minimal task-performance degradation and occasional gains.Source: §1 · p. 2

2.1 SAFETY IN MULTI-AGENT SYSTEMS.

• Multi-agent systems inherit single-agent vulnerabilities while interaction complexity expands threats such as agent manipulation and communication corruption.Source: §2.1 · p. 3
• Existing defenses include self-verification, centralized guards, graph-based detectors, and peer inspection, but individual agents may miss sophisticated attacks and guards add failure and overhead risks.Source: §2.1 · p. 3

2.2 MULTI-AGENT REINFORCEMENT LEARNING.

• MARL methods support coordinated training, yet most LLM-based approaches share one backbone across agents, limiting true agent-level diversity.Source: §2.2 · p. 3
• AdvEvo-MARL instead trains multiple distinct backbone models collaboratively, co-evolving attackers and defenders for robust and generalizable safety behaviors.Source: §2.2 · p. 3

Game formulation

• The interaction is formulated as a partially observable Markov game with state, action, transition, observation, discount, and finite-horizon components.Source: §3 · p. 3
• For LLM agents, an action is a complete generated response rather than an individual token.Source: §3 · p. 3

Attacker–defender roles

• Agents are partitioned into disjoint attacker and defender groups: attackers seek compromise, while defenders resist attacks and preserve task performance.Source: §3 · p. 3

Framework overview

• Figure 1 depicts attacker warm-up, adversarial co-evolution, and group-based advantage estimation using public mean-return baselines.Source: Figure 1 · p. 4

Reward and objective

• The framework evaluates the final trajectory output through an environment or trusted judge, producing global and agent-local rewards for co-evolution.Source: §3 · p. 4

4.2 BOOTSTRAPPING ADVERSARIAL GENERATION VIA ATTACKER WARM-UP

• AdvEvo-MARL first uses supervised warm-up to teach attackers jailbreak strategies, then co-evolves attackers and defenders through repeated adversarial interactions.The warm-up uses approximately 4,000 curated samples generated from 1,000 harmful behaviors and filtered reasoning traces.Source: §4.1 · p. 4, §4.2 · p. 5

4.3 ADVEVO-MARL: SAFE AND CAPABLE MULTI-AGENT SYSTEMS VIA CO-EVO RL

• Attackers and defenders are jointly optimized with REINFORCE++ after warm-up, embedding safety in multiple backbone agents rather than relying on a shared model.Attackers seek malicious outcomes, while defenders must resist attacks and maintain task performance.Source: §4.3 · p. 5
• Agents estimate advantages against the mean return of their attacker or defender role group, reducing variance and promoting coordinated policy updates.The public baseline replaces estimates based solely on each agent’s own return trajectory.Source: §4.3 · p. 5
• The training loss uses clipped policy updates and a KL term to regulate divergence from the reference policy.The clipping mechanism restricts update magnitude, while the KL term measures divergence between learned and reference policies.Source: §4.3 · p. 5
• The reward design assigns separate attacker and defender objectives, while formatting rewards enforce reasoning and response tags during training.Safety, task utility, and format compliance are represented by separate reward components, with training weights shifting from safety toward task performance.Source: §4.3 · pp. 5–6
• Defender rewards combine local response quality with global system output to align individual behavior with both safety and task performance.Global-only rewards can mislead individual defenders when other agents determine the aggregated output.Source: §4.3 · pp. 5–6

5.2 MAIN RESULTS

• AdvEvo-MARL consistently achieves the lowest ASR and contagion rate across nearly all evaluated attack configurations.Table 1 covers NetSafe, AutoInject, and UserHijack across chain, tree, and complete graph topologies.Source: §5.2 · p. 7, Table 1 · p. 7
• ASR remains below 10% in chain and tree systems and reaches at most 17.68% in the complete topology, while baselines reach 38.61% or 65.53%.Under UserHijack, Challenger-7B reaches 38.33% ASR, a 10% increase over its vanilla counterpart.Source: §5.2 · p. 7
• AdvEvo-MARL suppresses contagion below 35% across settings as attacks intensify and communication becomes more interconnected.Many defended baselines show contagion rates from 30% to 60% in densely connected environments.Source: §5.2 · pp. 7–8
• Task performance shows minimal degradation: 3B variants lose at most 3% accuracy, while the 7B model improves over its vanilla counterpart across all three benchmarks.The 7B model was trained exclusively on mathematical tasks yet improved especially on out-of-distribution datasets.Source: §5.2 · p. 8, Figure 2 · p. 8

5.3 DYNAMIC ATTACKS AND COLLABORATIVE DEFENSE

• Dynamic attackers reduce ASR by 12% under NetSafe and improve task performance by up to 4% relative to static attackers.AdvEvo-MARL also yields marginally lower ASR under AutoInject and UserHijack.Source: §5.3 · p. 8, Figure 3 · p. 8
• Attacker prompt diversity ultimately increases during training, producing more varied jailbreaks while supporting less overfit and more generalizable defenses.The diversity trend is non-monotonic early but steadily rises later in training.Source: §5.3 · pp. 8–9, Figure 4 · p. 9
• Joint defender training yields a 12% robustness gain and a 4% task-utility gain under NetSafe compared with isolated training.The reported improvements are attributed to collaborative defense behaviors arising through joint training.Source: Figure 3 · p. 8, §5.3 · p. 9

5.4 PUBLIC BASELINE BASED MULTI-AGENT REINFORCEMENT LEARNING

• The public baseline produces more stable learning, improves task utility and safety, and avoids the later degradation observed without it.Without the baseline, defender response length drops approximately 13.3% over the last 50 training steps; baseline training also suppresses attacker rewards and raises defender rewards.Source: §5.4 · pp. 9–10, Figure 5 · p. 9

• Co-training attackers and defenders enables agents to adapt continuously to evolving threats and develop stronger, more generalizable defenses.Attackers and defenders learn together in a dynamic adversarial environment rather than through isolated safety training.Source: §6 · p. 10
• A public baseline based on group-level returns stabilizes advantage estimation and facilitates collaboration among agents sharing the same role.Attackers or defenders use a common baseline calculated from their group-level returns.Source: §6 · p. 10
• AdvEvo-MARL substantially improves system safety while preserving and sometimes enhancing task performance across attack strategies and task benchmarks.The conclusion characterizes the framework as unified for building multi-agent systems that are both safe and capable.Source: §6 · p. 10

• Adversarial attacks are generated solely to strengthen defense mechanisms within multi-agent systems.Source: §7 · p. 11
• The work argues that open, systematic study of adversarial attacks is critical for developing safe and resilient AI systems.This motivation is tied to enabling broader positive social impacts for multi-agent systems.Source: §7 · p. 11