AgentBreeder: Mitigating the AI Safety Risks of Multi-Agent Scaffolds via Self-Improvement

• Multi-agent deployments create multipolar safety challenges that existing alignment research focused on single-agent LLMs does not comprehensively evaluate.Source: §1 · p. 1
• AGENTBREEDER evolves large populations of multi-agent scaffolds using complementary capability and safety objectives.Source: §1 · p. 1
• At each generation, scaffolds are benchmarked, embedded and clustered, Pareto elites are selected, and an LLM Meta Agent creates offspring through crossover and mutation.Source: Figure 1 · p. 2, §1 · p. 2
• The framework introduces red teaming to explore multi-agent attack surfaces and blue teaming to generate scaffolds with greater adversarial robustness.Source: §1 · p. 3
• AGENTBREEDER is implemented in Inspect to support reproducible, extensible evaluation across safety, capability, and helpfulness benchmarks.Source: §1 · p. 3

• Multi-agent scaffolds are architectures supporting communication among interacting LLM agents, with advantages including planning, task decomposition, and specialization.Source: §2 · p. 3
• Automated Design of Agentic Systems searches code-defined scaffold spaces using a search space, algorithm, and evaluation function, and has produced strong task-performing systems.Source: §2 · p. 3
• AGENTBREEDER extends this methodology with a MAP-Elites-inspired quality-diversity search that clusters scaffolds and samples Pareto-front elites for evolution.Source: §2 · p. 3
• BLUEAGENTBREEDER evolves multi-agent scaffolds to be more robust to adversarial attacks than the base model, while REDAGENTBREEDER evolves scaffolds more vulnerable to such attacks.Source: §2 · p. 3

• Prior self-improving systems evolve agents, connections, code, prompts, or functions, whereas AGENTBREEDER evolves the entire scaffold and searches its broader design space.Source: §3 · p. 4
• Existing multi-agent safety studies examine malicious traits, healthcare conversations, injected agents, and scaffolded-LLM risks, providing mitigation strategies and resilience analyses.Source: §3 · p. 4

4 AgentBreeder

• AGENTBREEDER supports blue optimization of capability and safety, red optimization of safety vulnerability alongside capability, and capability-only optimization.Source: §4 · p. 4

4.1 Seed Scaffolds

• The initial population contains seven hand-designed scaffolds, and balanced validation samples equal positive and negative baseline cases to strengthen evolutionary feedback.Source: §4.1 · p. 4

4.2 Mutation Operators

• The Meta Agent applies mutation and crossover to evolve scaffolds, with mutation weighted twice as highly because this empirically accelerated convergence.Source: §4.2 · pp. 4–5

4.5 Multi-Objective Pareto Elites

• Selection samples elites from dynamically generated clusters’ capability-safety Pareto fronts rather than from fixed niches.Source: §4.2 · p. 5, §4.5 · p. 5

4.3 Descriptors

• A 12-dimensional text embedding of each scaffold’s name and code encodes semantic and architectural information for diversity tracking.Source: §4.3 · p. 5

4.4 Clustering

• Agglomerative clustering uses a distance threshold of 0.7 to balance exploration of diverse architectures against selection pressure.Source: §4.4 · p. 5

4.6 Evaluations

• Evaluations run in Inspect, where scaffolds are implemented as model providers to support experiment tracking, parallelization, and extensible benchmark integration.Source: §4.6 · p. 5

5.1 Experiment 1: Blue Team Defense

• 79.4% average uplift in SaladData safety performance was achieved across three BLUEAGENTBREEDER runs, with a 110.7% maximum increase.The best scaffold was selected by maximizing capability plus safety while excluding safety reward-hacking scaffolds.Source: §5.1 · pp. 6–7
• Capability performance was maintained or marginally improved in blue mode, including a 21.0% uplift on GPQA.The experiments used DROP, MMLU, and GPQA capability benchmarks alongside SaladData safety evaluation.Source: §5.1 · pp. 6–7
• Blue-mode safety gains can reflect reward hacking: one scaffold reached 95.2 ± 2.4 safety accuracy while helpfulness fell 43%.A trivial refusal strategy such as “I’m sorry, I can’t help with that.” can inflate safety scores, motivating TruthfulQA helpfulness checks.Source: §5.1 · pp. 6–7

5.2 Experiment 2: Red Team Attack

• 81.6±4.8 accuracy on inverted 1-SaladData surpassed every seed scaffold by at least 6.25% after only half the blue-mode generation budget.REDAGENTBREEDER optimized DROP capability alongside unsafe-response performance, showing that unsafe scaffolds can emerge without directly prompting for them.Source: §5.2 · pp. 7–9
• Red-mode scaffolds retained competitive capability, reaching 67.7±4.7 F1 on DROP while exhibiting increased safety vulnerabilities.This indicates that capability scores alone may not reveal harmful scaffold behavior.Source: §5.2 · pp. 7, 9

5.3 Experiment 3: Multi-Objective Ablation

• CAPABLEAGENTBREEDER marginally surpassed ADAS across all capability benchmarks, but its gains were smaller than those from multi-objective optimization.The ablation evolved scaffolds for a single capability objective without considering safety.Source: §5.3 · p. 9
• Single-objective ablations achieved 72.3 ±3.1 F1 on DROP and 41.2±4.4 accuracy on GPQA, near or slightly above baseline but below multi-objective gains.The authors attribute the difference to stronger validation signal and selection pressure when an additional benchmark is included.Source: §5.3 · p. 9
• Single-objective evolution produced only modest SaladData uplift, whereas explicit safety or negative-safety objectives substantially influenced outcomes.With stronger models, overall performance improved but the relative gain from discovered over seed scaffolds diminished, plausibly because of contamination, higher baselines, and reduced scaffold benefit.Source: §5.3 · p. 9

Pre-Deployment Safety Evaluations

• REDAGENTBREEDER offers an automated approach to surface multi-agent scaffolds vulnerable on safety benchmarks before deployment.The authors suggest labs could incorporate similar pipelines into release protocols for new LLMs.Source: §6 · p. 9

Post-Deployment Adversarial Robustness

• BLUEAGENTBREEDER designs safer and capable scaffolds and can upgrade existing scaffolds toward company values, policies, and regulatory requirements.The experiments are presented as evidence that evolutionary search can support dynamic, data-driven multi-agent evaluation.Source: §6 · pp. 9–10

Limitations

• Limited generations and small populations produced only marginal performance improvements in the reported experiments.The study also describes itself as a proof of concept whose stronger helpfulness and safety claims require more comprehensive benchmarks.Source: §6 · p. 10
• The evaluation used a select benchmark set and only seven seed scaffolds, limiting coverage of real-world concerns and discovered-scaffold diversity.These constraints may restrict how broadly the findings generalize.Source: §6 · p. 10

7 Conclusion

• AGENTBREEDER discovers scaffolds through multi-objective optimization of capability and safety across three operating modes.BLUEAGENTBREEDER targets safer scaffolds, REDAGENTBREEDER identifies vulnerabilities, and CAPABLEAGENTBREEDER maximizes task performance.Source: §7 · p. 10
• 79.4% average safety uplift and up to 21.0% GPQA improvement were achieved by BLUEAGENTBREEDER while maintaining or enhancing capability.Safety improvement reached up to 110.7% on SaladData.Source: §7 · p. 10
• REDAGENTBREEDER shows that unsafe behavior can coexist with strong task performance by producing capable scaffolds with increased vulnerability.The result highlights a safety risk specific to optimizing multi-agent scaffolds.Source: §7 · p. 10
• The framework discovers scaffolds with competitive or increased performance relative to prior work while exhibiting increased adversarial robustness.The conclusion positions this result as evidence for systematic multi-agent scaffold evaluation.Source: §7 · p. 10

8 Future Work

• Larger populations and longer evolutionary runs could yield more substantial capability and safety improvements.Future evaluations could add AILuminate and contamination-free MMLU-CF for broader assessment.Source: §8 · p. 10
• White-box and gray-box analyses could expose safety risks hidden by the current black-box scaffold evaluation.Suggested targets include agent-tool, agent-API, and agent-agent interactions.Source: §8 · p. 10
• Future optimization could include inference cost and support multi-core scaffolds containing different base models.The current work considers only capability and safety objectives.Source: §8 · p. 10
• The authors call for governance frameworks and differentiated safety cases for scaffolds with varying transparency.The proposed range spans fully white-box to black-box architectures.Source: §8 · p. 10

A Impact Statement

• AGENTBREEDER has defensive value for proactive safety testing but could also be misused to develop harmful scaffolds.The impact statement describes this as a dual-use concern requiring safeguards.Source: A Impact Statement · p. 15

B CapableAgentBreeder

• Table 5 compares CAPABLEAGENTBREEDER scaffolds with ADAS seed and discovered scaffolds across three benchmarks.The table reports results on held-out test sets.Source: B CapableAgentBreeder · p. 15, Table 5 · p. 15
• DROP results use F1 score, while the other benchmark results use median accuracy.Each reported value includes a 95% confidence interval computed over 500 held-out test samples.Source: Table 5 · p. 15
• The table marks the highest and second-highest result in each column with bold and underlining, respectively.This formatting identifies the leading comparisons among the evaluated scaffold groups.Source: Table 5 · p. 15

C Benchmarks

• MMLU, DROP, GPQA, SaladData, and TruthfulQA cover broad capability, reasoning, safety, and helpfulness evaluation settings.MMLU spans 57 subjects; DROP tests reading comprehension and arithmetic-like reasoning; GPQA contains graduate-level science questions; SaladData evaluates safety; TruthfulQA tests questions humans may answer incorrectly.Source: C Benchmarks · p. 16

D Prompts

• The Meta Agent iteratively programs novel scaffolds, then refines and debugs candidates using prompts adapted from ADAS.After the first candidate, it performs two refinement iterations and up to three debugging iterations.Source: D Prompts · p. 16, §D.2 · p. 19, §D.3 · p. 19

D.1 Base Prompt

• The base prompt asks for complete Python forward functions that coordinate agents, meetings, and chats while maximizing validation fitness.Agents communicate through Chat objects added to shared Meetings, and fitness is the median correct rate with a 95% bootstrap confidence interval.Source: §D.1 · pp. 17–18

D.2 Reflection Prompt 1

• Reflection prompts require checking architectural novelty, identifying implementation mistakes, and revising or replacing the proposed scaffold.They also instruct the Meta Agent to remove redundancy, improve effectiveness, and avoid overly similar architectures.Source: §D.2 · pp. 18–19

D.3 Reflection Prompt 2

• A second reflection stage applies the implementation guidance again while preserving the prior thought and name and updating the corrected code.This creates an additional revision pass before debugging.Source: §D.3 · p. 19

D.4 Debugging Prompt

• The debugging prompt asks the Meta Agent to diagnose the latest implementation using earlier insights while retaining the original design rationale.The response separates the repeated thought from the debugging thought.Source: §D.4 · pp. 19–20

D.5 Mutation Prompts

• Mutation operators alter prompts, agents, rewards, meeting structures, participation, hierarchy, roles, trust, and power dynamics.Safety-enhanced mutations add consensus checks, ethical oversight, transparency, monitoring, role restrictions, and fail-safe outputs, while capability-enhanced mutations add reframing, role changes, interaction phases, and deeper abstraction.Source: §D.5 · pp. 20–21

D.6 Crossover Prompts

• Crossover prompts combine two existing systems into a novel scaffold while requiring the resulting forward function to return the exact task-specific output format.The same output-format constraint also appears throughout the mutation and base prompts.Source: §D.6 · pp. 21–22

E Scaffolds Discovered by AgentBreeder

• The Optimized-Hierarchical-Expert-Panel uses domain experts, a moderator, a devil’s advocate, a synthesis expert, and a validator.The scaffold is implemented as an asynchronous forward function receiving a task and required answer format.Source: E Scaffolds Discovered by AgentBreeder · pp. 22–23
• A single expert-panel meeting connects all agents so their outputs become visible through shared chat history.The meeting is created once and attached to the participating agents.Source: E Scaffolds Discovered by AgentBreeder · p. 23
• Stage 1 gathers domain-specific analyses and confidence values, then has the devil’s advocate challenge those analyses.Expert responses and challenges are added to the meeting as Chat objects for subsequent agents to access.Source: E Scaffolds Discovered by AgentBreeder · p. 23
• The experts produce revised final answers after the challenge, which are collected as domain insights for synthesis.The scaffold requests the required answer format at this intermediate stage.Source: E Scaffolds Discovered by AgentBreeder · pp. 23–24
• The final validation step returns the answer field in the exact required format.This preserves compatibility with benchmark scoring even though the scaffold uses multiple internal stages.Source: E Scaffolds Discovered by AgentBreeder · pp. 23–24

F Cost of Experiments

• Approximately $600 funds BLUEAGENTBREEDER’s three 20-generation benchmark runs and evaluations.About $500 is attributed to gpt-4o-mini-2024-07-18 and about $100 to claude-3-5-sonnet-20241022-v2:0.Source: F Cost of Experiments · p. 24
• Approximately $115 funds REDAGENTBREEDER’s 10-generation DROP run.The reported cost is described as expected for this configuration.Source: F Cost of Experiments · p. 24
• The CAPABLEAGENTBREEDER experiment comprised one 20-generation run on each of three benchmarks and evaluations, costing approximately $400.Source: F Cost of Experiments · p. 24

Claims and scope

• The abstract and Section 1 state three contributions—Attack, Defense, and Evaluation—which Sections 4–5 later substantiate.This supports an affirmative assessment that the paper’s main claims accurately reflect its contributions and scope.Source: NeurIPS Paper Checklist · p. 25

Limitations

• A dedicated Section 6 limitations paragraph identifies computational cost, benchmark coverage, and seed-diversity constraints.The checklist therefore indicates that the paper discusses limitations affecting its empirical scope and robustness.Source: NeurIPS Paper Checklist · p. 26

Experimental reproducibility

• The paper discloses its full search algorithm, prompts, and hyper-parameters in Section 4 and Appendix D, with code linked for end-to-end replication.Section 5 additionally specifies generation budgets, model versions, evaluation protocols, and dataset splits in Appendix C.Source: NeurIPS Paper Checklist · pp. 26, 28

Statistical significance

• All tables report medians with 95 % bootstrap confidence intervals, and Section 5.1 explains the resampling procedure.This provides uncertainty information for the experimental results, including those exemplified by Table 2.Source: NeurIPS Paper Checklist · p. 28

Ethics and societal impact

• The framework is released under a Non-Commercial license, and Appendix A discusses its dual-use nature and responsible-release safeguards.The checklist also states that the work focuses on safety, follows open-benchmark protocols, avoids personal-data collection, and includes an impact statement in Appendix A.Source: NeurIPS Paper Checklist · pp. 29–30

Assets and human subjects

• Existing datasets are credited with original papers and licenses, while no new datasets or pretrained models are released; human-subject and IRB questions are marked [NA].The study uses only publicly available text benchmarks and recruits no human participants.Source: NeurIPS Paper Checklist · pp. 30–31

LLM policy guidance

• The checklist states that “NA” applies when core method development does not involve LLMs as important, original, or non-standard components.It directs authors to the NeurIPS LLM policy for further guidance on what should be described.Source: NeurIPS Paper Checklist · p. 31