Source-linked AI summary
Multi-Agent Deep Research: Training Multi-Agent Systems with M-GRPO
Haoyang Hong, Jiajun Yin, Yuan Wang, Jingnan Liu, Zhe Chen, Ailing Yu, Ji Li, Zhiling Ye, Hansong Xiao, Yefei Chen, Hualei Zhou, Yun Yue, Minghui Yang, Chunxiao Guo, Junwei Liu, Peng Wei, Jinjie Gu
TL;DR
Specialized multi-agent systems need training methods that handle heterogeneous agent roles, variable rollout frequencies, and separated deployment. M-GRPO extends group-relative policy optimization with hierarchical credit assignment and trajectory alignment, and it consistently outperforms single-agent and frozen-sub-agent baselines on real-world benchmarks.
Problem
Training distinct LLMs for different agents is needed for specialized multi-agent systems but is challenged by heterogeneous rollouts and disrupted cross-server gradient flow.
Method
M-GRPO extends Group Relative Policy Optimization with hierarchical credit assignment and trajectory alignment for vertical systems with separately trained main and sub-agents.
Results
M-GRPO consistently outperforms single-agent GRPO and multi-agent systems with frozen sub-agents on GAIA, XBench-DeepSearch, and WebWalkerQA.
Takeaways & Limitations
Joint optimization of main and sub-agents supports task-specific expertise, global alignment, long-horizon planning, and tool-augmented reasoning.
Abstract
from arXiv · showhide
Multi-agent systems perform well on general reasoning tasks. However, the lack of training in specialized areas hinders their accuracy. Current training methods train a unified large language model (LLM) for all agents in the system. This may limit the performances due to different distributions underlying for different agents. Therefore, training multi-agent systems with distinct LLMs should be the next step to solve. However, this approach introduces optimization challenges. For example, agents operate at different frequencies, rollouts involve varying sub-agent invocations, and agents are often deployed across separate servers, disrupting end-to-end gradient flow. To address these issues, we propose M-GRPO, a hierarchical extension of Group Relative Policy Optimization designed for vertical Multi-agent systems with a main agent (planner) and multiple sub-agents (multi-turn tool executors). M-GRPO computes group-relative advantages for both main and sub-agents, maintaining hierarchical credit assignment. It also introduces a trajectory-alignment scheme that generates fixed-size batches despite variable sub-agent invocations. We deploy a decoupled training pipeline in which agents run on separate servers and exchange minimal statistics via a shared store. This enables scalable training without cross-server backpropagation. In experiments on real-world benchmarks (e.g., GAIA, XBench-DeepSearch, and WebWalkerQA), M-GRPO consistently outperforms both single-agent GRPO and multi-agent GRPO with frozen sub-agents, demonstrating improved stability and sample efficiency. These results show that aligning heterogeneous trajectories and decoupling optimization across specialized agents enhances tool-augmented reasoning tasks.
1 Introduction
The paper motivates vertically coordinated multi-agent systems for complex, specialized tasks, then proposes M-GRPO to train distinct agents despite rollout imbalance and disrupted gradient flow.
- Motivation: Vertical systems assign planning and delegation to a leader while specialized sub-agents execute subtasks and return feedback for verification.The supplied workflow describes main-agent integration of structured sub-agent feedback.
- Problem: Different-agent rollout counts create unstable asynchronous training, while separate servers disrupt gradient flow and make standard backpropagation unworkable.These constraints arise when distinct LLMs are trained within vertical multi-agent systems.
- Contributions: M-GRPO trains separate LLMs as distinct agents in vertical systems, addressing specialized multi-agent reinforcement learning challenges.The framework targets architectures with a main agent and delegated sub-agents.
- Contributions: M-GRPO combines hierarchical credit assignment and trajectory alignment to produce fixed-shape batches despite variable sub-agent invocations.Its group-relative advantages respect the hierarchy, while masking, duplication, or dropping handles variable sub-trajectories.
- Contributions: Across GAIA, XBench-DeepSearch, and WebWalkerQA, M-GRPO consistently improves over single-agent GRPO and frozen-sub-agent multi-agent baselines.The comparison is reported across training checkpoints on real-world agent benchmarks.
2 Related work
Prior work advances single-agent reasoning, preference optimization, and inference-time multi-agent collaboration, but complex or specialized tasks still motivate trained collaborative agents.
- Training objectives: The paper positions M-GRPO within a progression from RLHF and preference fitting toward group-relative and multi-agent reinforcement-learning objectives.The related work describes PPO, DPO, RRHF, and newer group-relative objectives.
- Single-agent systems: Single-agent methods span planning, reflection, preference objectives, process rewards, reasoning-focused RL, and tool-calling training.The related work traces progression from non-parametric reasoning structures to parameter-updating objectives.
- Multi-agent systems: Multi-agent systems exhibit collective intelligence and strong inference-time performance, yet complex or domain-specific tasks still require training.The cited discussion identifies medicine and other specialized domains as examples.
- Multi-agent reinforcement learning: Recent multi-agent RL frameworks use agent-level rewards, counterfactual credit assignment, role-based credit propagation, or discussion rewards.These approaches adapt reinforcement learning to collaborative agent behavior and report gains on selected reasoning tasks.
3 Problem setup
The problem setup uses a vertical two-agent architecture in which a main agent delegates tool-oriented subtasks to a sub-agent, with rewards reflecting output quality and global task success.
- Architecture setup: The main agent plans, decomposes queries, delegates subtasks, and synthesizes the final answer, while the sub-agent executes tools and returns structured results.Communication follows a call–return protocol with evidence, summaries, and metadata.
- Reward setup: The main-agent reward combines format validity and answer correctness, with invalid formatting receiving zero reward regardless of correctness.The format and correctness terms are balanced by hyperparameters.
- Reward setup: The sub-agent receives format, replicated main-agent correctness, and expert execution-quality rewards to balance local behavior with global outcomes.The three components are weighted by hyperparameters and guide tool use and response quality.
- Trajectory structure: A rollout may contain a variable number of sub-agent invocations, so batches can have changing ratios of main-agent to sub-agent trajectories.This variable ratio motivates the M-GRPO batching scheme.
- Trajectory structure: M-GRPO addresses variable invocation counts by aligning sub-agent trajectories for efficient batched reinforcement-learning updates.The method is introduced after the setup identifies changing trajectory ratios across batches and training steps.
4 Methodology
M-GRPO extends GRPO to hierarchical main–sub-agent training by assigning group-relative advantages across both agents and aligning variable sub-agent trajectories for batching.
- M-GRPO preserves group-relative baselines while respecting the hierarchy between main and sub-agents.The objective is applied to distinct LLMs serving the main and sub-agents.
- M-GRPO computes centered, normalized returns and advantages from K rollouts while accounting for differing sub-agent invocation counts.The number of sub-agent invocations can vary across rollouts, so main and sub-agent trajectories require separate handling.
- Each rollout contains a main trajectory that invokes sub-agents for tool-use subtasks, receives summarized messages, and produces the final answer.The main agent may invoke the sub-agent multiple times within one rollout.
- Trajectory alignment uses an approximate invocation bound d to create fixed-size sub-agent batches by duplicating, dropping, or selecting trajectories.When invocation counts are below d, trajectories are randomly duplicated; when they exceed d, trajectories are randomly dropped.
- The training pipeline updates main and sub-agents separately on distinct servers while exchanging rewards through a shared database.Each query uses 8 rollouts with d selected as 8 in the described deployment.
5 Experiments
The experiments evaluate M-GRPO with a two-stage curriculum, decoupled multi-agent training, and trajectory alignment. Co-training both agents improves performance across three real-world benchmarks, while synchronization further improves training outcomes.
- 5.1 Experimental Setup: The evaluation measures answer correctness and required-output-format adherence on domain-specific and general-purpose deep-search benchmarks.Benchmarks include GAIA, XBench-DeepSearch, and WebWalkerQA.
- 5.1 Experimental Setup: Trajectory alignment handles variable sub-agent invocations by duplicating or dropping sub-trajectories to produce fixed-shape batches for policy-gradient updates.The setup uses eight main-agent rollouts and a target of eight sub-agent trajectories per query.
- 5.2 Stage 1: Format learning with simple data: The two-stage curriculum first stabilizes format learning on simple data, then develops collaborative problem-solving on challenging tasks.Stage 1 rewards rise steadily from zero to a high plateau before stage 2 training begins.
- 5.3.1 Main results: benchmark performance: Co-training both agents consistently outperforms main-only training across GAIA, XBench-DeepSearch, and WebWalkerQA during stage 2.Results are averaged over three independent inference runs.
- 5.3.2 Ablation study: training configurations: The performance hierarchy is co-training > main-only > single-agent, supporting benefits from both the multi-agent architecture and joint M-GRPO optimization.The comparison uses challenging stage 2 data and includes EMA-smoothed learning trends alongside raw rewards.
- 5.3.3 Ablation study: trajectory synchronization: Trajectory synchronization outperforms unsynchronized training, improving stability and collaborative learning by keeping training closer to on-policy.The comparison uses raw and EMA-smoothed stage 2 learning curves.
6 Conclusion
The paper introduces M-GRPO for training LLM-based multi-agent systems and reports that it outperforms single-agent and fixed-sub-agent configurations on real-world benchmarks.
- M-GRPO outperforms traditional single-agent systems and fixed sub-agent configurations on GAIA, XBench, and WebWalker QA.
- Joint optimization fosters task-specific sub-agent expertise while maintaining alignment with the main agent’s goals.
- M-GRPO improves long-horizon planning and tool-augmented reasoning through joint optimization of main and sub-agents.
A Appendix
The appendix presents case studies showing that the multi-agent system learns targeted functions.
- Case studies show that the multi-agent system can learn targeted functions.
A.1 Tool call
After M-GRPO training, the multi-agent system selects more suitable tools for different questions, improving its answers on tool-use examples.
- The Rubik’s-cube example uses constraints about cube pieces, colors, and adjacency to determine the missing edge cube’s colors.
- The trained system answers the Rubik’s-cube example correctly with “green, white,” whereas the pre-training prediction is “Green,Orange.”
- After training, the system selects the reasoner tool for logical derivation instead of relying on browser search.
- Training changes the Rubik’s-cube workflow from two sub-agent calls to two reasoner calls plus one verification call.
A.2 Split and distribute tasks
M-GRPO improves task decomposition by helping the main agent identify essential constraints and assign more precise, context-aware searches to sub-agents.
- After training, the system returns “34689” instead of the pre-training prediction “33755” for the invasive-fish question.
- Training teaches the main agent to include the pet-release condition and USGS scope when assigning the sub-agent’s search.
- The pre-training workflow omits the key release condition, leading the sub-agent toward general information about Ocellaris Clownfish.
- The main agent learns to identify missing essential information and assign more accurate, context-aware tasks to sub-agents.
- The trained reasoning process explicitly frames the question as identifying a Finding Nemo fish that became invasive after pet owners released it.