PaperlayerView exact revision on arXiv

Source-linked AI summary

MetaAgent-X : Breaking the Ceiling of Automatic Multi-Agent Systems via End-to-End Reinforcement Learning

Yaolun Zhang, Yujie Zhao, Nan Wang, Yiran Wu, Jiayu Chang, Yizhao Chen, Qingyun Wu, Jishen Zhao, Huazheng Wang

arXiv:2605.14212v1cs.AI

TL;DR

Existing automatic multi-agent systems adapt only through test-time search or designer optimization with frozen executors, leaving end-to-end self-designing and self-executing systems unexplored. MetaAgent-X jointly trains designer and executor agents with hierarchical rollouts and stagewise co-evolution, consistently surpassing evaluated baselines by up to 21.7%.

  • Problem

    Existing automatic multi-agent systems use test-time search or freeze execution agents while optimizing designers, leaving end-to-end self-designing and self-executing training unexplored.

  • Method

    MetaAgent-X jointly optimizes designer and executor agents through end-to-end reinforcement learning with hierarchical rollouts, role-aware credit assignment, and stagewise co-evolution.

  • Results

    Up to 21.7% gains were achieved across diverse math and code benchmarks, with ablations showing both agents improve through stagewise co-evolution.

  • Takeaways & Limitations

    MetaAgent-X supports end-to-end trainable automatic multi-agent systems as a practical approach to self-designing and self-executing agentic models.

  • Takeaways & Limitations

    The experiments are constrained by computational resources.

Abstract

from arXiv · show

Automatic multi-agent systems aim to instantiate agent workflows without relying on manually designed or fixed orchestration. However, existing automatic MAS approaches remain only partially adaptive: they either perform training-free test-time search or optimize the meta-level designer while keeping downstream execution agents frozen, which creating a frozen-executor ceiling and leaving the end-to-end training of self-designing and self-executing agentic models unexplored. To address this, we introduce MetaAgent-X, an end-to-end reinforcement learning framework that jointly optimizes automatic MAS design and execution. MetaAgent-X enables script-based MAS generation, execution rollout collection, and credit assignment for both designer and executor trajectories. To support stable and scalable optimization, we propose Executor Designer Hierarchical Rollout and Stagewise Co-evolution to improve training stability and expose the dynamics of designer-executor co-evolution. MetaAgent-X consistently outperforms existing automatic MAS baselines, achieving up to 21.7% gains. Comprehensive ablations show that both designer and executor improve throughout training, and that effective automatic MAS learning follows a stagewise co-evolution process. These results establish end-to-end trainable automatic MAS as a practical paradigm for building self-designing and self-executing agentic models.

1 Introduction

Existing automatic multi-agent systems remain only partially adaptive because they search at test time or freeze execution agents. MetaAgent-X addresses this ceiling with end-to-end joint training of self-designing and self-executing systems, supported by hierarchical rollout and stagewise co-evolution.

  • Multi-agent systems outperform single-agent approaches across medical decision-making, scientific discovery, financial trading, software engineering, and hardware design.
  • Existing automatic MAS approaches either search over structures at test time or optimize the designer while freezing the execution system.
  • MetaAgent-X jointly optimizes designer and executor agents end to end, enabling MAS that can self-design and self-execute.
  • Executor Designer Hierarchical Rollout structures rollout generation and credit assignment, while Stagewise Co-evolution enables decoupled, scalable designer-executor learning.
  • 21.7%: MetaAgent-X surpasses single-agent and automatic MAS baselines across diverse math and code benchmarks.
  • Ablations show both designer and executor improve throughout training, with effective co-evolution following a stagewise process supported by decoupled optimization.

2 Related work

Prior automatic multi-agent systems largely provide only partial adaptation through test-time search or designer optimization with frozen executors. MetaAgent-X instead frames automatic MAS learning as designer–executor co-evolution, distinguishing its objective and analysis from related end-to-end agent approaches.

  • LLM-based MAS improve complex problem solving through specialized roles, structured interactions, and coordination protocols.
  • Existing automatic MAS methods mainly use training-free test-time search or semi-trainable designer optimization with downstream executors kept fixed.Training-free methods search prompts, roles, workflows, or organizations without updating model parameters; semi-trainable methods optimize a meta-level designer or controller.
  • Frozen executors impose a performance ceiling and prevent designer–executor co-adaptation in automatic MAS.
  • Agentic reinforcement learning and self-evolution improve LLM agents through interaction, environmental feedback, and iterative experience collection.
  • MetaAgent-X explicitly models automatic MAS learning as designer–executor co-evolution rather than fixed-structure optimization or undifferentiated whole-system training.This formulation targets the frozen-executor ceiling while analyzing the internal mechanism of automatic MAS co-evolution.

3 Method

MetaAgent-X jointly trains a Designer that generates task-specific multi-agent systems and an Executor that runs them through coupled online reinforcement learning. Hierarchical rollouts separate role-specific credit, while stagewise updates reduce interference in the co-evolution of the two roles.

  • End-to-end training: The Designer generates task-specific multi-agent systems, and the Executor runs each instantiated workflow in an external environment under jointly trainable parameters.The framework supports shared parameters or separately optimized Designer and Executor parameter sets.
  • Online system construction: Designers compose predefined coordination structures, agent templates, and tool interfaces into lightweight Python scripts specifying roles, protocols, tool use, and control flow.Rollouts can be batched across multiple queries and sampled designs, recording observations, tool calls, and outcome-based rewards.
  • Hierarchical rollout: Bi-level tree-structured rollouts sample M independent designs and N executions per design, producing an M × N evaluation matrix for each question.Each matrix entry pairs a design with an execution rollout and its outcome reward.
  • Decomposed advantage estimation: Designer advantages average execution outcomes across N rollouts, whereas Executor advantages normalize trajectories across all designs and executions for the same question.This decomposition reduces execution-level stochasticity for Designer learning and provides the Executor with a more stable question-level signal than single-level normalization.
  • Stagewise co-evolution: Stagewise co-evolution alternates fixed-length K-step phases in which only the active role’s trajectories contribute gradients while shared parameters update continuously.Executor phases improve outcomes under current designs, enabling Designers to learn structures that better exploit the improved Executor while reducing gradient interference.

4 Experiment

Experiments evaluate MetaAgent-X across mathematical reasoning and code generation using Qwen3 at 4B and 8B scales. MetaAgent-X RL outperforms baselines, while ablations show hierarchical rollout, stagewise co-evolution, and shared policies improve optimization stability and performance.

  • Experimental Setup: Experiments use no-thinking Qwen3 models at 4B and 8B scales on one node with eight H200 GPUs, using a shared Designer–Executor policy.Unless otherwise specified, maximum prompt and response lengths are 8192 tokens.
  • Main Results: 38.33% average accuracy on Qwen3-8B and 34.18% on Qwen3-4B are achieved by MetaAgent-X RL, consistently outperforming single-agent GRPO across six benchmarks.The evaluation covers mathematical reasoning and code generation tasks, including AIME, OLYMPIADBENCH, APPS, LiveCodeBenchv6, and CodeContests.
  • Main Results: −6.81% for ADAS on 8B and 22.52% for AFlow on 4B illustrate that search-based MAS baselines often degrade performance across Qwen3 scales.The reported ADAS result falls to an average of 20.35% on 8B.
  • Main Results: 6.17% average improvement from MetaAgent-X SFT to MetaAgent-X RL demonstrates the effectiveness of the reinforcement-learning pipeline over the cold-start foundation.MetaAgent-X RL is reported to surpass all evaluated baselines, including methods that train only a meta-agent or optimize workflow selection.
  • Ablations: 40.0% on AIME24 and 33.3% on AIME25 show that hierarchical rollout with M=4 and N=4 outperforms the flatter M=8 and N=1 configuration.The hierarchical setting samples four candidate designs and executes each design four times.
  • Ablations: Stagewise co-evolution is more stable than coupled, executor-only, or designer-only training, while shared policies outperform separate policies on AIME24 and AIME25.Stagewise learning displays staircase-shaped reward dynamics; executor-only training saturates, and coupled training later collapses.

5 Discussions · Appendix

MetaAgent-X is presented as the first end-to-end reinforcement learning framework to jointly optimize automatic multi-agent-system design and execution. Across diverse benchmarks and model scales, it surpasses human-designed and existing automatic MAS baselines while revealing mutual designer–executor improvement.

  • 5 Discussions: MetaAgent-X is the first end-to-end reinforcement learning framework jointly optimizing an automatic multi-agent system’s designer and executor.The framework uses hierarchical rollouts and stagewise co-evolution.
  • 5 Discussions: The framework jointly trains automatic MAS design and execution through hierarchical rollouts.This coupling targets both the designer and executor rather than optimizing only one component.
  • 5 Discussions: Stagewise co-evolution is a core mechanism for training MetaAgent-X’s designer and executor.The paper identifies stagewise co-evolution as part of the framework’s optimization approach.
  • 5 Discussions: Across six math and code benchmarks and two model scales, MetaAgent-X consistently surpasses human-designed and existing automatic MAS baselines.The evaluation spans both benchmark domains and model-size settings.
  • 5 Discussions: 21.7% is the maximum reported gain over human-designed and existing automatic MAS baselines.The passage states that MetaAgent-X improves over both baseline categories by up to this amount.
  • 5 Discussions: The results expose internal dynamics in which the designer and executor mutually improve.This mutual improvement is presented as an additional outcome of the end-to-end framework.

A Cold Start Details

MetaAgent-X cold-starts its policy by distilling generated workflows into supervised Designer and Executor trajectories. The workflow interface also improves a strong model’s AIME 2024 accuracy from 63.3% to 66.7% under prompting-only evaluation.

  • Cold-start design: Cold start distills both the Designer, which writes executable workflows, and Executors, which solve problems within them.The Designer and Executors are trained from generated multi-agent system behavior.
  • Cold-start design: The workflow template bank includes single-agent, ensemble-voting, solver-critic reflection, and solver-tester configurations with specified topologies and role-level prompts.Templates are implemented in the codebase and provide in-context examples for workflow synthesis.
  • Cold-start data: 3K Designer examples and 8K Executor examples are retained from correct-answer trajectories for supervised cold start.The data pipeline samples multiple designs per question and logs Designer and Executor conversations during workflow execution.
  • Cold-start validation: 66.7% AIME 2024 accuracy is achieved with prompting DeepSeek to synthesize and run a multi-agent workflow, versus 63.3% with direct answering.The prompting-only comparison uses DeepSeek as the underlying model and suggests explicit role decomposition and verification recover additional correct solutions.

B Reward Design

The outcome-based reward combines functional correctness with structured agent behavior, using λ = 0.4. Correctness is evaluated through domain-specific environment feedback, while formatting rewards enforce parseable solutions and concise inter-agent communication.

  • Reward composition: The outcome-based reward R(ei,j, di) consists of correctness and formatting terms, with λ = 0.4.Rcorrect evaluates final-solution correctness through environment feedback, while Rformat incentivizes structured agent behavior.
  • Correctness reward: Rcorrect ∈{0, 1} is a strict binary signal determined by the specific domain environment.For mathematics, symbolic verification handles algebraically equivalent expressions; for code, reward 1 requires passing all hidden unit tests within time and memory constraints.
  • Format reward: Solution formatting requires final answers to follow a standardized output format for reliable automated parsing.This is one component of the format reward, which acts as a regularizer.
  • Format reward: Delivery formatting requires inter-agent messages to be enclosed within <delivery>...</delivery> tags, promoting concise, parseable communication.The constraint discourages forwarding full reasoning traces that inflate the context window without improving coordination.

C Result Analysis

MetaAgent-X produces diverse, task-specific workflows and improves through both executor-side gains and designer changes. Its designer shifts routing away from ensemble+judge toward reflection and simpler alternatives, while rapid role alternation destabilizes training.

  • Sensitivity Analysis on Stage Length: Switching the active role every 1 step causes highly unstable training because neither role accumulates sufficient role-consistent gradient signal before interruption.The reported 1-step run collapses after approximately the omitted continuation of the passage.
  • Per-query design diversity: The designer emits 54 distinct role names, while 77.5% of agents receive byte-unique, task-specific system prompts across evaluation workflows.Each workflow is freshly synthesized for the problem rather than instantiated from a predefined template.
  • Which role brings the improvement?: Ensemble+judge loses ∼30 pp on every math benchmark, with routing redirected mostly to reflection and slightly toward single.The routing distribution changes uniformly across the reported math benchmarks.

D Baseline Details

Baseline implementations use the same execution backbone, evaluation setup, and matched rollout budget as MetaAgent-X unless otherwise stated.

  • Baseline implementations: Baselines use Qwen3 4B or Qwen3 8B backbones according to each experimental setting.All reported results are evaluated on the same final test split and metric as MetaAgent-X.
  • Baseline implementations: Search- or training-based baselines follow their original paper or released-code protocols while matching MetaAgent-X’s rollout budget.These comparisons use the same final test split and metric unless otherwise stated.

D.1 Search Based MAS Optimization Baselines · D.2 Semi Learning Based MAS Optimization Baselines · E Case Studies

The paper compares search-based and semi-learning-based MAS optimization baselines under aligned candidate, execution, training, and evaluation protocols. These baselines include workflow search, meta-agent search, Score DPO, agentic supernet sampling, and an internalized multi-agent foundation model.

  • D.1 Search Based MAS Optimization Baselines: AFlow searches code-represented agentic workflows with MCTS under the official settings, including 4 candidates per query and up to 20 search rounds.Its evaluation executes each candidate workflow N = 4 times when stochastic execution is supported.
  • D.1 Search Based MAS Optimization Baselines: ADAS performs Meta Agent Search by writing executable Python forward functions and archiving evaluated agents.The default MGSM search uses n_generation=30, n_repeat=1, max_workers=48, and at most three debugging attempts for invalid generated code.
  • D.1 Search Based MAS Optimization Baselines: ADAS uses 128 validation examples and the next 800 examples for final testing on MGSM and related reasoning domains, while GPQA diamond uses 32 validation examples and 166 testing examples.On new benchmarks, each generation evaluates M = 4 candidate agents when supported, with each candidate executed N = 4 times for reward estimation.
  • D.2 Semi Learning Based MAS Optimization Baselines: ScoreFlow trains a workflow generator with Score DPO using M = 4 candidate workflows per query and N = 4 executions per candidate to estimate scores and construct preference pairs.The held-out test split is reserved for final inference evaluation.
  • D.2 Semi Learning Based MAS Optimization Baselines: ScoreFlow updates only trainable generator parameters while executor parameters remain frozen, using stages of length K = 30 and learning rate 5×10−6.Score DPO uses LoRA with rank 8, alpha 16, dropout 0.01, and target modules q_proj and v_proj.
  • D.2 Semi Learning Based MAS Optimization Baselines: MaAS trains an agentic supernet that samples query-dependent architectures, optimizing controller distributions and agentic operators on a 1 : 4 train-test split.The held-out test set is used only for final evaluation, with architecture sampling and training budgets matched to the proposed method where possible.
  • D.2 Semi Learning Based MAS Optimization Baselines: Agent Foundation Model internalizes multi-agent collaboration within a single model through multi-agent distillation and agentic reinforcement learning rather than explicitly instantiating an external workflow at test time.The comparison uses the released size-comparable AFM-CodeAgent-7B-rl checkpoint against the paper’s 8B experimental scale.

E.1 RL model Examples

Three end-to-end trajectories show the designer selecting task-specific team structures and executors collaborating on math and code problems. The examples include a single agent, ensemble-plus-judge, and reflection with a separate critic, with outputs and critiques illustrating both successful and flawed reasoning.

  • Trajectory design: The trajectories span math and code tasks while instantiating three team structures: single agent, ensemble + judge, and reflection with a separate critic.Designer and executor boxes contain verbatim model output, with nonessential chain-of-thought omitted.
  • Math example: single agent: 116 is the reported answer for the lottery problem after counting 115 prize-winning outcomes, including exactly 2, 3, or 4 matches.The conditional probability of the grand prize given a prize is 1/115.
  • Math example: ensemble + judge: 90 is one solver’s grid-count result, but the judge selects 45 after identifying that the algebraic solver misinterpreted the second condition.The judge also rejects a combinatorial solver’s reasoning chain as contradictory.
  • Code example: reflection and critique: The code example analyzes eversions as repeatedly moving the last element through stable partitions until the array stops changing.The analysis identifies reaching the maximum element as the key stopping condition.
  • Code example: reflection and critique: The critic exposes a logical flaw in the proposed eversion reasoning: when the current value is 1 and the maximum is 5, the maximum element greater than 1 is 4, not 5.This invalidates the claimed update for the next current element in that example.

E.2 A Case Study Where Reflection Repairs the Solver

The case study shows that reflection enables the RL workflow to diagnose and repair an invalid geometric model, whereas the SFT workflow repeats contradictions and produces no valid final answer. The comparison uses the same held-out AIME 2024 circle-packing example.

  • RL reflection and repair: On the held-out AIME 2024 example, the RL trajectory reaches an impossible geometric constraint, identifies the faulty distance model, and replaces it with a valid similar-triangles equation.The RL case is presented as more diagnostic than comparing final answers alone.
  • SFT failure: The SFT workflow uses ensemble_judge to combine AlgebraicSolver, GeometricSolver, CombinatorialSolver, and MathJudge, but extracts no valid final answer.Its evaluation result is explicitly marked as ✗no valid final answer extracted.
  • SFT failure: The SFT solvers apply an angle-packing formula that yields the contradiction 34 = 253, then continue trying variants of the same invalid model.The geometric solver explicitly recognizes that the formula is wrong but does not replace the underlying model.
  • SFT failure: After a second model produces sin(theta/2) = -1752 / 33 = -53.0909..., the SFT judge repeats the invalid branch instead of forcing revision after contradiction.Because sin(theta/2) must be positive, the transcript ends without a boxed valid answer.

Case RL: Reflection turns the contradiction into a corrected derivation

Reflection converts an initially contradictory circle-packing derivation into a corrected geometric model. The refined equations yield inradius r = 192/5 and final value m + n = 197.

  • Initial failure and critique: The first attempt assumes the center-line distance is r − r1, producing the impossible equation 14 = 4046.The critic identifies this distance assumption as the fundamental geometric flaw.
  • Corrected equations: For the radius-34 sequence, the corrected equation is L * (r − 34)/r = 476.It uses 8 circles and therefore 7 center gaps of length 2 * 34.
  • Corrected equations: For the radius-1 sequence, the corrected equation is L * (r − 1)/r = 4046.It uses 2024 circles and therefore 2023 center gaps of length 2 * 1.
  • Corrected geometric model: The refined derivation models a parallel segment as L * (r − r1)/r = (n−1) * 2r1, using n−1 gaps between n circles.This replaces the incorrect distance relation with a geometry-based length equation.
  • Final result: 192/5 is the resulting inradius, and 197 is the final value of m + n.The ratio of the two equations gives (r − 1)/(r − 34) = 17/2, leading to r = 192/5; since m = 192 and n = 5, m + n = 197.
Loading 2605.14212v1…