Evo-MARL: Co-Evolutionary Multi-Agent Reinforcement Learning for Internalized Safety

1. Introduction

• External guards provide limited protection when task agents lack safety awareness, create a potential single point of failure, and add cost and complexity when multiplied.Attacks can also spread through inter-agent communication and trigger cascading failures.Source: §1 · p. 1
• Evo-MARL internalizes defenses in every task agent and co-evolves attackers and defenders using an evolving pool of attack prompts.Mutations are guided by attack success rates to foster more generalizable defenses.Source: §1 · p. 1
• During adversarial training, some agents act as attackers while others detect, mitigate, and correct unsafe or misleading content, with rewards tied to harmfulness and correctness.Attack prompts are injected sequentially into a randomly selected victim and downstream agents.Source: §1 · p. 1
• Experiments report up to 22% improvement in safety and up to 5% gains in task performance across multimodal and text-only red-team datasets.These results are presented as evidence that robustness and utility can improve together.Source: §1 · p. 2

2. Related Works

• MAS attacks can hijack individual agents or manipulate communication and workflow execution, while LLM agents also face prompt injection and adversarial manipulation.External tools and memory systems add further exposed interfaces.Source: §2 · p. 2
• Prior MAS defenses include dedicated inspection agents, communication-graph detection with supervised fine-tuning, and peer review, but do not improve agents’ intrinsic safety mechanisms.External modules can also create scalability bottlenecks and fragility.Source: §2 · p. 2
• LLM-based reinforcement learning has supported search, agentic and software-engineering tasks, and coordinated multi-agent policies, including collaborative reasoning and role-based training.Examples include Search-R1, Park et al., and Wan et al..Source: §2 · p. 2
• Evo-MARL uses adversarial co-training between attacker and defender agents to build individual safety awareness and improve overall system reliability.This extends prior RL-based coordination toward internalized safety capabilities.Source: §2 · pp. 2–3

3. Methodology

• Evo-MARL combines a co-evolutionary attacker module that mutates and selects prompts with joint GRPO optimization of all task agents.The framework aims to preserve task performance while improving robustness.Source: §3 · p. 3

3.1. Internalizing Safety via MARL

• Attackers and defenders are jointly modeled in a shared Markov decision process, with distinct action spaces, rewards, and policies for inducing unsafe behavior or maintaining robustness.The state transition depends on both attacker and defender actions.Source: §3.1 · p. 3
• Training uses a chain of specialized agents and injects attacks into downstream responses; defenders collaboratively detect, discard, or purify malicious content.Safe final responses earn defenders +1, accurate answers add +0.5, and unsafe or incorrect responses receive negative rewards.Source: §3.1 · p. 3
• Defenders share parameters for training efficiency and use GRPO, whose shared policy uses grouped query samples to compute advantages and a KL penalty to constrain updates.The paper specifies G as the number of samples per query and DKL as the penalty term.Source: §3.1 · p. 3

3.2. Evolving Attacks through Biological Evolution

• Attack prompts evolve through mutation and crossover, with attack effectiveness serving as a fitness signal to select successful variants for the next generation.This lets attack strategies and defenses iteratively improve without including attackers in the RL training loop.Source: §3.2 · p. 3

4.1. Experimental Setup

• The evaluation covers three red-team datasets and two task benchmarks, including JailBreakV-28K, HarmBench, MultiJail, MATH, and Creative Writing.The official JailBreakV 280 mini dataset, multimodal HarmBench split, English MultiJail split, and 100 sampled MATH points are used.Source: §4 · p. 3, §4.1 · p. 4
• Tests use chain-structured and hierarchical MAS settings, with Qwen2.5-1.5B and 3B agents and additional 7B comparisons; metrics include ASR, MATH accuracy, and Creative Writing scores.Multimodal ASR evaluation counts samples that successfully attack the first multimodal agent.Source: §4.1 · p. 4

4.2. Results

• Trained 1.5B MAS reduce HarmBench attack success rate by up to 22%, and outperform untrained 7B MAS on JailBreakV and MultiJail.Across tasks and model scales, the method yields consistent safety improvements; 1.5B models also outperform their 3B counterparts in safety.Source: §4.2 · p. 4, Table 1 · p. 4
• On mathematical reasoning and creative writing benchmarks, trained 1.5B MAS achieve accuracy gains of up to 5 percentage points without compromising safety.Figure 2 provides qualitative comparisons before and after MARL-based training.Source: §4.2 · p. 4, Table 1 · p. 4

• Evo-MARL combines reinforcement learning with co-evolutionary adversarial training to improve multi-agent system robustness.Source: §5 · p. 4
• By embedding defense capabilities in every task agent, Evo-MARL mitigates jailbreak attacks while enhancing reasoning and creative-writing performance.Source: §5 · p. 4
• Open challenges include stabilizing training with adaptive attackers, scaling to larger or more complex systems, and adding memory or external knowledge for long-term robustness.Source: §5 · p. 4

• Figure 2 presents successful defense on JailBreakV.Source: Figure 2 · p. 7