Source-linked AI summary
Multi-Agent Tool-Integrated Policy Optimization
Zhanfeng Mo, Xingxuan Li, Yuntao Chen, Lidong Bing
TL;DR
Single-agent tool-integrated planning faces context-length and noisy-response challenges, while effective reinforcement-learning training for multi-agent systems remains unresolved. MATPO trains planner and worker roles within one LLM and achieves an 18.38% average relative improvement in performance over single-agent GRPO across three benchmarks.
Problem
Effective reinforcement-learning training for tool-integrated multi-agent systems remains unresolved, amid single-agent context and noise constraints and the infrastructure costs of separate agent models.
Source: Introduction · pp. 1–2Method
MATPO uses role-specific prompts to train planner and worker agents within one model for multi-agent, multi-turn tool-integrated planning.
Source: Introduction · p. 2, Multi-Agent Multi-Turn Reinforcement… · p. 3Results
18.38% average relative improvement in performance over single-agent GRPO across GAIA-text, WebWalkerQA, and FRAMES.
Source: Results · p. 6Takeaways & Limitations
Recapping the original query in the worker-agent prompt substantially improves multi-agent RL performance.
Source: Ablation Studies and Practical… · p. 7Planner agents may fail to sustain objections to problematic worker outputs, leading to erroneous follow-up search directions.
Source: Ablation Studies and Practical… · p. 7
Abstract
from arXiv · showhide
Large language models (LLMs) increasingly rely on multi-turn tool-integrated planning for knowledge-intensive and complex reasoning tasks. Existing implementations typically rely on a single agent, but they suffer from limited context length and noisy tool responses. A natural solution is to adopt a multi-agent framework with planner- and worker-agents to manage context. However, no existing methods support effective reinforcement learning post-training of tool-integrated multi-agent frameworks. To address this gap, we propose Multi-Agent Tool-Integrated Policy Optimization (MATPO), which enables distinct roles (planner and worker) to be trained within a single LLM instance using role-specific prompts via reinforcement learning. MATPO is derived from a principled credit assignment mechanism across planner and worker rollouts. This design eliminates the need to deploy multiple LLMs, which would be memory-intensive, while preserving the benefits of specialization. Experiments on GAIA-text, WebWalkerQA, and FRAMES show that MATPO consistently outperforms single-agent baselines by an average of 18.38% relative improvement in performance and exhibits greater robustness to noisy tool outputs. Our findings highlight the effectiveness of unifying multiple agent roles within a single LLM and provide practical insights for stable and efficient multi-agent RL training.
Introduction
The introduction motivates planner–worker coordination as a way to manage the context and noise limits of single-agent tool use, then presents MATPO as a way to train those roles within one model.
Introduction
- Single-agent deep research is limited because lengthy, noisy tool responses consume context and can disrupt attention and subsequent planning.Current implementations use iterative multi-turn search to gather and analyze information, but these limitations make long-range tool-integrated planning difficult.Source: Introduction · p. 1
- A planner coordinating specialized browsing workers can contain noisy search responses in worker contexts while keeping each agent’s context manageable across delegated rounds.The planner handles high-level planning and decision-making, while workers carry out specific browsing tasks (Hu et al. 2025).Source: Introduction · p. 1
- Multi-agent systems using separate models create infrastructure challenges from uneven workloads, require more token context, and consume more parameters than single-agent alternatives.These complications arise when each agent operates on a separate model.Source: Introduction · pp. 1–2
- MATPO trains planner and worker roles through distinct prompts within one model instance, integrating with existing RL frameworks while aiming to preserve specialization and improve infrastructure efficiency.The paper presents an end-to-end multi-agent-in-one-model RL framework, alongside theoretical analysis and a concrete implementation.Source: Introduction · p. 2
- The paper reports comprehensive experiments showing better performance than single-agent baselines and offers implementation recommendations and future research directions.It also identifies questions about effective single-model multi-agent RL training, reward assignment to workers without explicit rewards, and using one model for multiple roles.Source: Introduction · p. 2
Related Work
Related work moves from single-agent tool-integrated planning and reinforcement learning toward planner–worker frameworks, but principled RL training for multi-agent tool-integrated planning remains underexplored.
Tool-Integrated Agent Frameworks
- Single-agent TIP agents iteratively plan, call tools, and incorporate responses, using function-calling, ReAct, or structured workflows.Examples include function-calling-augmented LLMs (Yang et al. 2025; Nguyen et al. 2025a), ReAct-style agents (Yao et al. 2023; Li et al. 2025c,a; Tao et al. 2025), and structured workflows (Team et al. 2025).Source: Tool-Integrated Agent Frameworks · p. 2
- Single-agent TIP struggles with context saturation from long tool responses and interaction histories, while noisy, unstructured outputs can disrupt reasoning and trigger cascading errors.These limits hinder deeper reasoning chains (Zhang et al. 2025).Source: Tool-Integrated Agent Frameworks · p. 2
- Planner–worker frameworks delegate subtasks to workers, containing noisy outputs locally, but existing approaches focus on inference-time prompt engineering rather than tool-integrated planning training.Liu et al. (2025) trains multi-turn multi-agent zero-sum games, but is not tailored to tool-integrated planning; other frameworks include Hu et al. (2025) and MiroMind (2025a).Source: Tool-Integrated Agent Frameworks · p. 2
Tool-integrated Agentic Reinforcement Learning
- RLVR and trajectory-filtering methods improve single-agent TIP, while principled RLVR extensions for multi-agent frameworks remain underexplored.Work includes RLVR for single-agent TIP (Shao et al. 2024; Jin et al. 2025; Nguyen et al. 2025b), trajectory filtering for coding-based math and GUI tasks, and SFT or DPO followed by RLVR in structured workflows. The section identifies a need for efficient role coordination, principled credit assignment, and compatibility with existing RL infrastructure.Source: Tool-integrated Agentic Reinforcement… · p. 2
Problem Setup
The setup formalizes single-agent tool-integrated planning and then extends it to planner-worker rollouts, including a shared-model configuration with role-specific prompts. It also describes how GRPO is adapted to exclude tool-generated tokens from policy updates.
Single-Agent Multi-Turn Reinforcement Learning
- Single-agent TIP generates an answer through iterative planning and tool use, with correct and incorrect answers receiving rewards of 1 and 0, respectively.The rollout includes model-generated actions and tool responses; the agent uses each tool response to decide its next move.Source: Single-Agent Multi-Turn Reinforcement… · pp. 2–3, Figure 2 · p. 3
Single-Agent Group Relative Policy Optimization
- GRPO adaptation masks tool-response tokens because they are not generated by the policy, excluding them from the policy gradient.The method applies GRPO to single-agent multi-turn TIP rollouts, which contain both LLM-generated actions and tool-API responses.Source: Single-Agent Group Relative Policy… · p. 3
Multi-Agent Multi-Turn Reinforcement Learning
- Multi-agent rollouts assign subtasks to a worker that uses tools, while a planner decides whether to continue or produce the final answer.The framework uses one LLM with distinct planner and worker system prompts; the paper focuses on this configuration because it uses one rollout engine rather than K + 1 separate engines.Source: Multi-Agent Multi-Turn Reinforcement… · pp. 3–4
Methodology
MATPO extends GRPO to train planner and worker roles in one model, addressing how unverifiable worker subtasks contribute to a verifiable final answer. Its implementation generates nested planner–worker rollouts, assigns rewards from final-answer accuracy, and updates the model on an augmented batch.
Methodology
- MATPO jointly accounts for planner and worker contributions by running one planner rollout and T worker rollouts, normalizing rewards across G×(T +1) rollouts rather than GRPO’s G worker rollouts.This addresses the credit-assignment challenge created because the planner’s final answer is directly verifiable while worker subtasks are not.Source: Methodology · p. 4
- MATPO applies a GRPO-style clipped objective using group-relative normalized rewards and log-likelihood ratios for the full multi-agent rollout and its planner and worker trajectories.The objective compares the current policy with the old policy; the rollout reward is accuracy-based for the full multi-agent trajectory.Source: Methodology · p. 4
Implementation
- For each query, the implementation launches nested asynchronous worker rollouts for subtasks assigned by each planner rollout, then verifies final answers and broadcasts normalized planner advantages to corresponding workers.Planner advantages are normalized among the planner-agent rollouts associated with that query.Source: Implementation · p. 5
- The implementation concatenates planner and worker rollouts into an augmented batch, computes the loss, masks system-prompt, query, and tool-response tokens, and updates the LLM.The augmented batch is evaluated using both πθ and πθold before the standard optimization process.Source: Implementation · p. 5
Experiments
MATPO outperforms single-agent GRPO across three benchmarks, while ablations identify query recapping and final summaries as important to multi-agent performance. Blocking HuggingFace search results has mild effects on accuracy, though sensitive URLs may still pose a data-leakage risk.
Setups
- Experiments train Qwen3-14B-base on filtered MuSiQue and test on GAIA-text, WebWalkerQA, and FRAMES.The MuSiQue subset excludes queries for which LLMs repeatedly fail to produce valid rollouts.Source: Setups · p. 5
- Planner and worker tool calls use XML schemas, worker prompts recap the original query, and rollouts append a final summary to support answer generation.If a rollout reaches the token limit, the latest messages are removed to make room for the summary; both single-agent and multi-agent workers use this mechanism.Source: Setups · pp. 5–6
- The reward combines answer accuracy and tool-call correctness as 0.9 * acc + 0.1 * fmt, with MATPO averaging planner and worker tool-call rates.For MATPO, fmt=0.5 * fmt_p + 0.5 * fmt_w; for single-agent RL, fmt is the success rate across parsed tool-call attempts.Source: Setups · p. 6
Results
- MATPO scores 42.60% on GAIA-text, 33.00% on WebWalkerQA, and 63.64% on FRAMES versus GRPO’s 32.16%, 30.14%, and 56.22%, an average relative improvement of 18.38%.MATPO also continues improving after step 120 on GAIA-text and FRAMES, where single-agent GRPO drops; the authors attribute this stability to additional browsing subtasks that help address noisy tool feedback.Source: Figure 5 · p. 6, Results · p. 6
Ablation Studies and Practical Take-Aways
- Multi-agent subtask decomposition outperforms single-agent curves, while removing final summaries lowers accuracy; summaries give the planner a cleaner interface to worker outputs.Without a final summary, raw worker output may end in tool calls or include distracting <think>...</think> blocks.Source: Ablation Studies and Practical… · pp. 6–7, Figure 6 · p. 7
- Blocking HuggingFace search results has mild effects on RL accuracy, but the authors recommend blocking sensitive URLs to reduce answer leakage and reward hacking.Unblocked results rarely contained full validation questions or directly useful information, though the authors note a mild contamination risk.Source: Ablation Studies and Practical… · p. 7
- Recapping the original query in the worker prompt substantially improves multi-agent performance and is hypothesized to guide more stable, higher-quality browsing.The comparison is between the green and blue curves in Figure 6a.Source: Ablation Studies and Practical… · p. 7, Figure 6 · p. 7
- Worker outputs presented as user messages may bias the planner toward compliance, causing it to abandon valid objections and pursue erroneous searches.The authors propose exploring alternative message formats for tool and worker-agent responses.Source: Ablation Studies and Practical… · p. 7
Conclusions
The paper presents MATPO as a multi-agent-in-one-model RL approach whose experiments demonstrate its effectiveness, and identifies extensions to agent roles, scaling laws, and RL infrastructure as future directions.
Conclusions
- MATPO explores multi-agent-in-one-model RL training, and the authors report that experiments demonstrate the method’s effectiveness.They plan to improve implementation efficiency and integrate additional tools.Source: Conclusions · p. 7
- Future work could extend multi-agent GRPO to additional worker agents, including specialized coding or file-processing agents.Source: Conclusions · p. 7
- The authors propose studying how the number of agent roles affects scaling and whether more roles induce new behaviors or stronger intelligence.Source: Conclusions · p. 7
- Another future direction is optimizing RL infrastructure for efficient multi-agent, multi-turn rollout and training.Source: Conclusions · p. 7
Appendix · Prompts · System Prompt and Tool Schema of the Planner-Agent
The planner-agent prompt defines iterative, stepwise tool use and prescribes a parseable XML-style call format with a JSON arguments object. It also specifies a single available browsing tool and rules for planning, tool calls, and final responses.
System Prompt and Tool Schema of the Planner-Agent
- The planner can call only the provided tools, one per message, using each result to inform the next step.Tool results arrive in the user’s next response.Source: System Prompt and Tool Schema of the… · p. 9
- Tool calls use top-level XML-style tags, with required server_name, tool_name, and JSON arguments fields that must follow the tool schema.String quotes must be escaped; lists and objects use JSON format, and the call must appear at the end of the response.Source: System Prompt and Tool Schema of the… · pp. 9–10
- The listed browsing_agent tool, search_and_browse, handles clearly defined web-search and browsing subtasks focused on factual gaps.Its required input is a string subtask.Source: System Prompt and Tool Schema of the… · p. 10
- The planner must break tasks into achievable subgoals, present a concise numbered plan first, proceed sequentially, and revise the plan when new information emerges.The overall objective is to solve the task iteratively and methodically.Source: System Prompt and Tool Schema of the… · p. 10
- Before each tool call, the planner summarizes known information, identifies gaps, chooses a relevant tool, verifies parameters, and supplies full context in actionable queries.It should extract and summarize partial information when a tool result is incomplete.Source: System Prompt and Tool Schema of the… · p. 10
- The agent must withhold its final answer until the task is complete, omit tool results and tool names, avoid unnecessary back-and-forth, and answer in the user’s language.It should answer directly when tool use is unnecessary and aim for complete, accurate, well-reasoned responses.Source: System Prompt and Tool Schema of the… · pp. 10–11
System Prompt and Tool Schema of the Worker-Agent
The worker prompt defines an MCP-based, one-tool-at-a-time workflow and directs the agent to break tasks into steps, use the provided search and scrape tools, and respond after completing the task.
System Prompt and Tool Schema of the Worker-Agent
- Tool calls use top-level XML-style tags with server name, tool name, and JSON arguments, and must appear at the end of the response.String and scalar parameters are written as-is; lists and objects use JSON format.Source: System Prompt and Tool Schema of the… · p. 11
- The worker can use google_search for web results and scrape to retrieve webpage text, with each tool’s inputs governed by its specified schema.The search schema includes query and optional region, language, location, result-count, time-filter, page, and autocorrect fields; scrape takes a URL and optional markdown setting.Source: System Prompt and Tool Schema of the… · pp. 11–12
- The agent is instructed to analyze requests, set ordered sub-goals, present a concise numbered plan, and work through steps sequentially while adjusting as new information emerges.Each tool step should use one call, with context and required parameters checked beforehand.Source: System Prompt and Tool Schema of the… · p. 12
- The prompt requires the agent to finish the task before answering, avoid unnecessary back-and-forth, not mention tool names or results, and answer in the user’s language.If a task does not require tool use, it should be answered directly.Source: System Prompt and Tool Schema of the… · p. 12
Instruction Prompt for Rollout Summarization
The prompt specifies a factual, structured Markdown report that answers an assigned subtask and explains its contribution to the main task. It prescribes required sections and concrete formatting and content rules.
Instruction Prompt for Rollout Summarization
- The report is to be delivered as the final turn, with no tools used, and should summarize information gathered to answer the assigned subtask.Source: Instruction Prompt for Rollout… · p. 13
- The Contribution to Main Task section should explain how the subtask answer advances the overall task and identify next steps for the main agent.The prompt supplies the main-task and assigned-subtask descriptions as context for that explanation.Source: Instruction Prompt for Rollout… · p. 13
- The final response must be a clear, complete Markdown report organized under four specified headings: Conclusion, Supporting Information, Observations, and Contribution to Main Task.The supporting section should present detailed facts or data in bullets or numbered lists; observations may include context, confidence, or how the conclusion was reached.Source: Instruction Prompt for Rollout… · p. 13
- The response must not include raw URLs; replace them with ‘([ link])’.Source: Instruction Prompt for Rollout… · p. 13
- The report should contain factual, specific information based on prior actions, without speculative filler, vague summaries, or conversational text.The conclusion should concisely state findings and the subtask answer, while bolding key information.Source: Instruction Prompt for Rollout… · p. 13
Instruction Prompt for LLM-as-Judge.
The prompt tells an LLM judge to compare a question, gold target, and predicted answer, then classify the prediction as CORRECT, INCORRECT, or NOT_ATTEMPTED. It defines these labels through information coverage and contradiction.
CORRECT
- CORRECT requires the answer to include the question’s important gold information without contradiction; semantic meaning matters, not capitalization, punctuation, grammar, or order.Hedging is allowed when the answer fully includes the gold information and contains no incorrect information.Source: Instruction Prompt for LLM-as-Judge · pp. 13–14
INCORRECT
- INCORRECT applies when an answer contradicts the gold target, including when the contradictory claim is hedged.Source: Instruction Prompt for LLM-as-Judge · p. 14
NOT_ATTEMPTED
- NOT_ATTEMPTED applies when the answer omits important gold information and makes no contradictory statement.Source: Instruction Prompt for LLM-as-Judge · p. 14
Additional grading rules
- For numerical targets, the prediction must be correct to the gold answer’s last significant figure; the prompt classifies 120k, 124k, and 115k as CORRECT, but 100k and 113k as INCORRECT.Source: Instruction Prompt for LLM-as-Judge · p. 14
- When the gold target contains more than the question asks for, an answer needs only the question-relevant information; information clearly inferred from the question may also be omitted.Source: Instruction Prompt for LLM-as-Judge · pp. 14–15
- The judge should not penalize clear typos in a person’s name.Source: Instruction Prompt for LLM-as-Judge · p. 15
Output format
- For a new example, the judge must return only A, B, or C for CORRECT, INCORRECT, or NOT_ATTEMPTED, respectively, without surrounding text.Source: Instruction Prompt for LLM-as-Judge · p. 15
Lemon-Pick MATPO Rollout Trajectory
The rollout accounts trace attempts to identify the fifth rules section and determine the change to Rule 601, including paths involving Bankruptcy rules and the Federal Rules of Evidence. They focus on whether the 2011 amendment deleted “competent,” with one account leaving the target unresolved and the other identifying it.
Initial reasoning
- The rollout mapped the fifth section to the E rules, identified Rule 601 as addressing witness competency, and reported its 2011 amendment deleted the competency requirement except for certain exceptions.Source: Lemon-Pick MATPO Rollout Trajectory · pp. 17–18
Search result
- The search result instead listed Federal Rules of Bankruptcy Procedure fifth, after Appellate, Civil, Criminal, and Evidence.This contradicted the rollout’s initial identification of Evidence as fifth.Source: Lemon-Pick MATPO Rollout Trajectory · pp. 18–19
Target uncertainty
- After that result, the rollout questioned how a Bankruptcy Procedure article could contain the most titles with “witnesses” and left the target uncertain.It considered whether the prompt or its interpretation might be mistaken, but did not resolve the article in this part.Source: Lemon-Pick MATPO Rollout Trajectory · p. 19
Lemon-Pick MATPO Rollout Trajectory
- The initial Bankruptcy route found Rule 9014 addressing witness testimony, but it was not the first Bankruptcy rule, exposing a mismatch with the query.The passage identifies Rule 1001 as the first Bankruptcy rule and says it does not mention witnesses.Source: Lemon-Pick MATPO Rollout Trajectory · p. 21
- Alphabetically, Evidence is fifth, not Bankruptcy, so the earlier tool ordering sent the search down the wrong path.The listed order is Appellate, Bankruptcy, Civil, Criminal, Evidence.Source: Lemon-Pick MATPO Rollout Trajectory · pp. 21–22
- The corrected path identifies Evidence Rule 601 as the first relevant rule and “competent” as the word deleted in its 2011 amendment.The passage says Rules 601–615 relate to witnesses and identifies Rule 601 as the first rule in that sequence.Source: Lemon-Pick MATPO Rollout Trajectory · p. 22