PaperlayerView exact revision on arXiv

Source-linked AI summary

MultiAgent-OPD: End-to-End Training of Hierarchical Multi-Agent Systems with On-Policy Distillation

Yuhan Sun, Jiaxin Zhao, Yongdong Luo, Yibo Hu, Meiguang Jin, Junfeng Ma, Weixun Wang, Weihang Pan, Zhongan Bi

arXiv:2610.23234v2cs.CL

TL;DR

Hierarchical teams receive only indirect feedback about subagent actions, and judge-based scores depend on the judge’s assessment. MultiAgent-OPD trains a Teacher to provide local Searcher guidance while outcome rewards train the Leader and Answerer. Across seven question-answering datasets, it reaches 38.68% macro avg@4, exceeding Prompted Critic by 4.06 percentage points and C-MAS + RL by 2.61 percentage points.

  • Problem

    Team outcomes make it difficult to assess individual intermediate actions, while LLM-judge evaluations depend on the judge’s assessment of subagent responses.

    Source: §1 · pp. 1–2

  • Method

    MultiAgent-OPD trains a Teacher in a Searcher–Answerer workflow, then uses its token-level supervision for Searchers while the Leader and Answerer learn from outcome rewards.

    Source: §1 · p. 2

  • Results

    38.68% macro avg@4 across seven question-answering datasets exceeds Prompted Critic by 4.06 percentage points and C-MAS + RL by 2.61 percentage points.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A policy trained in a simpler workflow can guide subagents during joint training of a hierarchy.

    Source: §5 · p. 10

    Evaluation uses fixed English open-domain QA checkpoints and four attempts; question-bootstrap intervals do not measure variation across independently trained runs.

    Source: Limitations · p. 10

Abstract

from arXiv · show

In a hierarchical multi-agent system, a Leader assigns tasks and subagents carry them out. Jointly training the Leader and the subagents is difficult because a final answer gives only indirect feedback to the subagents. An LLM judge can score the performance of the subagents, but these scores depend on the judge's own assessment of the quality of subagents' responses. We present MultiAgent-OPD, which combines on-policy distillation with reinforcement learning to jointly train a shared-parameter Leader-Searcher-Answerer team end to end. We first train a Teacher in a Searcher-Answerer workflow using answer correctness. During team training, the frozen Teacher scores student-generated search and summary tokens to provide local process supervision for the Searchers, while the Leader learns from team success and the Answerer from candidate correctness. Across seven datasets, the trained system reaches 38.68% macro avg@4, exceeding Prompted Critic by 4.06 percentage points and C-MAS + RL, the trained cascaded multiagent system, by 2.61 percentage points. The team also solves more questions within four attempts.

1 Introduction

Hierarchical teams receive limited feedback about individual subagent actions, and judge-based scores depend on the judge’s assessment. MultiAgent-OPD pairs Teacher-guided Searcher supervision with outcome-based learning for the Leader and Answerer.

1 Introduction

  • Team outcomes make individual intermediate actions difficult to assess, while LLM-judge scores depend on the judge’s assessment and may be limited to a single subtask score.A useful report may receive no positive outcome reward when another role fails to use it.Source: §1 · pp. 1–2
  • MultiAgent-OPD trains a Teacher in a Searcher–Answerer workflow, then uses the frozen Teacher for token-level Searcher supervision while the Leader and Answerer retain outcome-based RL objectives.Teacher–student log-probability differences provide local advantages for query generation and evidence summarization on student-sampled trajectories.Source: §1 · p. 2
  • 38.68% macro avg@4 across seven question-answering datasets exceeds Prompted Critic by 4.06 percentage points and C-MAS + RL by 2.61 percentage points.The system also exceeds the five included RL-based search agents.Source: §1 · p. 2
  • The paper reports higher macro avg@4 across seven datasets than included cascaded, hierarchical, and RL-based search baselines, alongside diagnostic comparisons of supervision and Teacher training.The diagnostics examine direct Searcher supervision and task-specific Teacher training.Source: §1 · p. 2

2 Related Work

Related work spans multi-agent learning and delegation systems, as well as on-policy distillation for search. MultiAgent-OPD differs by applying a task-trained Teacher to delegated Searcher histories within a jointly trained hierarchy.

2 Related Work

  • Prior multi-agent learning work covers discussion-based training, credit propagation, agent-wise advantage normalization, frozen-subagent Leader optimization, and joint training with expert LLM scores.The cited examples include MAPoRL, ReMA, MHGPO, Dr. MAS, PARL, and M-GRPO.Source: §2 · p. 3
  • Agent harnesses support task delegation and result integration, while Anthropic’s research system coordinates parallel searchers; these systems motivate the Leader–Subagent structure.The passage cites Claude Code, Codex, DeepSeek Harness, and Kimi Code [Anthropic, 2026, OpenAI, 2026, DeepSeek, 2026, Moonshot AI, 2026], and Anthropic’s research system.Source: §2 · p. 3
  • Prior search OPD methods use student-generated trajectories or step-level feedback, whereas MultiAgent-OPD uses a task-trained Searcher–Answerer policy to guide delegated Searcher tokens in a jointly trained hierarchy.The passage contrasts this approach with single-agent search methods including SD-Search [Ma et al., 2026] and OPDSearch+ [Ye et al., 2026].Source: §2 · p. 3

3 MultiAgent-OPD

MultiAgent-OPD trains a shared Leader–Searcher–Answerer policy in two stages: first learning a Searcher–Answerer Teacher, then applying role-specific advantages to the full team.

3.1 Preliminaries

  • All three roles share one policy; the Leader delegates successive-wave searches, while Searchers return reports and the Answerer produces final responses.The Leader may request additional searches, so Searcher subtasks depend on the Leader’s current policy.Source: §3.1 · p. 3
  • Team success is the maximum exact-match score across five candidates, while Answerers receive candidate-level exact-match rewards and role rewards are normalized for GRPO.Both stages start from the instruction-tuned base model.Source: §3.1 · pp. 3–4

3.2 Stage I: Learn a Teacher That Searches and Summarizes

  • Stage I trains a Searcher–Answerer workflow from the base model using answer correctness plus a Searcher behavior bonus, then freezes that policy as the Teacher.The Searcher uses a fixed retriever to gather evidence and write a report for the Answerer.Source: §3.2 · p. 4

3.3 Stage II: Train the Team with Role-Specific Advantages

  • In Stage II, the frozen Teacher scores Searcher responses at matching prompts, subtasks, retrieved passages, and generated prefixes to provide token-level advantages.The token mask excludes retrieved passages and padding, restricting the signal to Searcher-generated tokens.Source: §3.3 · pp. 4–5
  • The Leader learns from team outcomes and the Answerer from candidate rewards, while Searcher log-ratios reflect Teacher preference rather than causal contribution to the final answer.The Teacher scores student-generated prefixes, so supervision follows subtasks as the Leader changes task assignments.Source: §3.3 · pp. 5–6

4 Experiments

Across seven datasets, the trained system achieves the highest macro avg@4 among evaluated systems, while diagnostics compare Searcher supervision choices and later sections examine training trajectories, four-attempt outcomes, and inference workload.

4 Experiments

  • The comparison uses seven equally weighted datasets and four stochastic workflows per question, with normalized exact match and macro avg@4 as the primary metric.The trained systems’ checkpoints were selected near their validation peaks; Table 3 diagnostic rows all use step 500.Source: §4 · p. 6, §4.2 · p. 8

4.1 Main Results

  • 38.68% macro avg@4 exceeds Prompted Critic by 4.06 percentage points and C-MAS + RL by 2.61 points; MultiAgent-OPD also exceeds Tree-GRPO’s 37.37% by 1.31 points.It exceeds Prompted Critic on all seven datasets and C-MAS + RL on five; both paired 95% intervals exceed zero.Source: §4.1 · pp. 6, 8, Table 2 · p. 7

4.2 Diagnostic Comparisons

  • Without direct Searcher losses, macro avg@4 falls to 29.82% versus 38.68%; a base-model Teacher yields 29.50%, while outcome-based Searcher RL reaches 17.78%.The outcome-based variant records no search actions across 20,000 test workflows, although 99.51% still produce subtasks and reports.Source: §4.2 · pp. 8–9, Table 3 · p. 8

4.3 Training Dynamics

  • The displayed MultiAgent-OPD validation trajectory rises through mid-run and stays above both displayed baselines in their later shared interval.Curves are single smoothed historical runs plotted against training steps, not compute or variation across seeds.Source: §4.3 · p. 9, Figure 3 · p. 9

4.4 Answer Coverage and Consistency

  • 48.73% macro pass@4 and 29.27% correct on all four attempts exceed C-MAS + RL’s 45.97% and 27.05%, respectively.Prompted Critic records 45.96% pass@4 and 23.24% correct on all four attempts.Source: §4.4 · p. 9, Table 4 · p. 10

4.5 Inference Resource Usage

  • MultiAgent-OPD uses 5.39 search actions and 9,172 input tokens per workflow, versus Prompted Critic’s 4.45 and 6,786 and C-MAS + RL’s 2.86 and 3,886.Relative to Prompted Critic, searches rise 21.4% and input tokens 35.2%, while output tokens fall from 885 to 697; recorded token totals include repeated histories.Source: §4.5 · p. 10, Figure 4 · p. 10

5 Conclusion

MultiAgent-OPD jointly trains a hierarchical team with a task-trained Teacher supervising Searchers locally and role-specific outcome rewards for the Leader and Answerer.

5 Conclusion

  • Across seven datasets, MultiAgent-OPD exceeds Prompted Critic, C-MAS + RL, and the included RL-based search agents in macro avg@4 and macro pass@4, while using more searches and input tokens.The conclusion is that a policy trained in a simpler workflow can guide subagents during joint training.Source: §5 · p. 10

Limitations

The evaluation is limited to fixed English open-domain QA checkpoints, with comparisons made at systems’ own workflow budgets. The authors also identify limits in OPD’s supervision and in what the evaluation can establish.

Limitations

  • Systems run at their own workflow budgets, so the comparisons do not control for retrieval or token budget.The reported scope is a fixed English open-domain QA set, and intervals characterize variation within evaluated checkpoints rather than across independently trained runs.Source: Limitations · pp. 10–11
  • OPD’s teacher-conditioned local supervision is not a causal estimate of any token’s contribution to the final answer.The Teacher may err, and its full-question training differs from delegated-subtask use.Source: Limitations · p. 11
  • Shared role parameters couple updates across roles, while reverse-KL training can reduce exploration and requires compatible teacher–student vocabularies.Source: Limitations · p. 11
  • The diagnostic comparisons retain Leader and Answerer outcome objectives, so they do not evaluate those objectives separately.The anonymous artifact supplies scored evaluation records, aggregation code, result files, and the evaluation protocol for reproducing summary statistics.Source: Limitations · p. 11

Ethical Considerations

The evaluation uses public QA datasets and a text retriever, but answer correctness alone does not establish evidence quality, source attribution, or safety.

Ethical Considerations

  • Normalized exact match measures answer correctness, not whether retrieved evidence supports the answer, sources are credited, or the system is safe to use.Retrieved material can be inaccurate or biased.Source: Ethical Considerations · p. 11

AI Assistance Disclosure

The appendix documents AI assistance, the fixed test set and scoring procedure, summary-score uncertainty and resource accounting, and diagnostic answer extraction and completion bounds.

AI Assistance Disclosure

  • AI assistants supported related-work discovery, evaluation-code development, job tracking, result aggregation, figure creation, drafting, and example-report review; authors directed the research argument and take responsibility for methods and results.Source: AI Assistance Disclosure · p. 11

A.1 Fixed Test Set and Strict Scoring

  • The fixed test set uses seeded, dataset-specific question samples, removes 14 normalized training-question matches, and scores each workflow’s last Answerer output against normalized reference answers.Four workflows per question yield 20,000 records per system; the hash check does not detect paraphrases or pretraining contamination.Source: §A.1 · p. 14

A.2 Summary Scores and Uncertainty

  • Summary-score intervals use 4,000 within-dataset question-bootstrap draws, with equal dataset weights, and apply to five evaluated systems rather than the aggregate-only RL baselines.The main comparison uses checkpoints and a distillation coefficient fixed before test inspection; the 4/4-correct measure reuses the four workflows.Source: §A.2 · pp. 14–15, Table 5 · p. 15

A.3 How We Count Resource Use

  • Recorded workload sums prompt and response tokens across turns, potentially recounting history, and does not measure unique context, exact service usage, or total training cost.Costs average attempts within questions, then questions within datasets, then datasets equally; batch duration is not per-question response time.Source: §A.3 · p. 15

A.4 Flexible Answer Extraction and the Completion Bound

  • Flexible extraction accepts selected answer-tag and plain-text formats before applying the original normalized exact-match rule, while leaving unparsed outputs incorrect.The completion bound assumes every format failure in the compared system becomes correct and MultiAgent-OPD remains unchanged; it is not a measured repair gain.Source: §A.4 · pp. 15–16, Table 7 · p. 16

A.5 Collapse of Search Behavior under Outcome-Based Searcher RL

Outcome-based Searcher RL loses retrieval during training even as workflows continue producing reports and answers. The comparison systems differ in architecture, training, and Searcher supervision.

A.5 Collapse of Search Behavior under Outcome-Based Searcher RL

  • Between steps 300 and 400, workflows with no recorded search rose from 0.5% to 100% as EM fell from 26.0% to 21.8%; at step 500, EM fell to 19.2%.Final-answer production reached 99.3% during the search collapse; across 20,000 test workflows, none recorded search, while 99.51% had subtasks and reports and 99.03% produced a final answer.Source: §A.5 · p. 16, Table 8 · p. 16

B.1 Compared Systems

  • MultiAgent-OPD and Prompted Critic train the same hierarchy with the same Leader and Answerer objectives, differing in Searcher supervision: frozen-Teacher token-level OPD versus judge-based rewards.C-MAS + RL instead jointly trains Searcher and Answerer with outcome-based RL in its original cascaded workflow.Source: §B.1 · p. 17
  • The five RL-based search-agent baselines use the same backbone and protocol, but vary their training designs, from outcome-reward search to hierarchical agent-wise advantage normalization.The diagnostic variants retain the full configuration’s Leader and Answerer objectives while changing the Searcher path.Source: §B.1 · p. 17

B.2 Training and Inference Settings

Training and evaluation use fixed configurations for data, optimization, agent-step budgets, inference, and retrieval. Training uses multiple Answerer candidates, while evaluation uses one per answering request.

B.2 Training and Inference Settings

  • Training uses 79,168 Natural Questions and 90,447 HotpotQA examples, with batch size 32, five trajectories per question, five PPO minibatch updates, and learning rate 10−6.The same reward, advantage normalization, and other training settings apply to the full method and its three diagnostic variants.Source: §B.2 · p. 17
  • Stage I allows five environment steps; Stage II allows 12 steps and up to five Leader delegation rounds, with five Answerer candidates per training request.Evaluation instead uses one candidate per answering request, and its four runs per question are independent workflows.Source: §B.2 · p. 17
  • Evaluation uses vLLM in float16 with temperature 0.6, top-p 0.95, an 8,192-token input limit, and a 512-token output limit per call.All systems use E5 retrieval with three passages per query; cascaded and hierarchical systems retain their respective five- and 12-step limits.Source: §B.2 · p. 17

B.3 Evaluation Provenance and Statistics

This appendix records evaluation provenance, describes the role prompts, and provides examples of multi-agent collaboration.

B.3 Evaluation Provenance and Statistics

  • The main comparison uses a fixed 5,000-question manifest and four stochastic workflows per method; all four trained-system rows in Table 3 use step 500 checkpoints.Checkpoints and the OPD coefficient were fixed before test evaluation; Figure 3 shows smoothed historical validation curves against training steps, not compute.Source: §B.3 · p. 17

C Prompt Templates

  • The templates specify that Leaders and Searchers receive the original question and delegated sub-question, while Answerers receive the original question and Leader-selected reports.C-MAS + RL shares the Searcher and Answerer templates, but gives the original question to its Searcher.Source: C Prompt Templates · p. 18

C.1 LEADER

  • Leaders break multi-hop questions into targeted, non-repeating sub-queries, dispatch search agents, and answer when existing summaries contain the needed facts.The template permits multiple searches at once and passes relevant summary IDs to the Answerer.Source: §C.1 · p. 18

C.2 SEARCHER

  • Searchers take one action per turn, either issuing a specific query that fills an information gap or producing a structured summary when evidence is sufficient.The requested summary includes Key Findings and Relevance Assessment.Source: §C.2 · p. 18

C.3 ANSWERER

  • Answerers use the summaries and their own knowledge, then give a short, precise answer containing the minimal necessary entity, date, number, or phrase.Source: §C.3 · p. 19

C.4 Prompted Critic

  • The critic evaluates a search summary for relevance, completeness, accuracy, and usefulness, then returns a binary adequacy score and a concise actionable hint.The prompt specifies score 1 for adequate summaries and 0 otherwise.Source: §C.4 · p. 19

D Illustrative Examples of Multi-Agent Collaboration

  • The illustrative examples select two questions where H-MAS base and C-MAS + RL score 0/4 while MultiAgent-OPD scores 4/4.Each example uses attempt 0 from the main evaluation; the recorded search reports are verbatim excerpts, and message grouping does not establish timing.Source: D Illustrative Examples of Multi-Agent… · p. 19

D.1 Resolving the Entity: Athlete and the River Thames

These cases examine how MultiAgent-OPD resolves questions involving “Athlete” through its band origin and nearby river, while distinguishing a caretaker manager from a permanent manager in the Joe Mercer case.

D.1 Resolving the Entity: Athlete and the River Thames

  • MultiAgent-OPD scored 4/4 correct (attempt 0) on the Athlete/Thames case, while H-MAS base and C-MAS + RL each scored 0/4 correct.It identifies Athlete as a band formed in Deptford and links Deptford to the River Thames using two search subtasks.Source: §D.1 · pp. 20, 22, 23, 24

D.2 Preserving the Caretaker-Role Constraint

  • In Case 2, MultiAgent-OPD recorded 4/4 correct (attempt 0), while H-MAS base and C-MAS + RL each recorded 0/4 correct (attempt 0).The passages identify Joe Mercer as caretaker manager after Ramsey’s sacking and Don Revie as Ramsey’s permanent successor.Source: §D.2 · pp. 24–27
  • Joe Mercer served as caretaker after Alf Ramsey’s sacking, whereas Don Revie was appointed as Ramsey’s permanent successor.The passage cautions that the documents do not explicitly confirm the caretaker designation for Revie.Source: §D.2 · pp. 26–27
  • Joe Mercer is the answer; he served as England caretaker manager for seven matches after Alf Ramsey’s dismissal and before Don Revie’s appointment.Source: §D.2 · p. 28
  • Both comparison systems answer Don Revie, while MultiAgent-OPD distinguishes his permanent appointment from Mercer’s caretaker role and checks Mercer’s seven-match tenure.Source: §D.2 · p. 28
Loading 2610.23234v2…