PaperlayerView exact revision on arXiv

Source-linked AI summary

Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMs

Yujie Zhao, Lanxiang Hu, Yang Wang, Minmin Hou, Hao Zhang, Ke Ding, Jishen Zhao

arXiv:2510.11062v5cs.LGcs.MA

TL;DR

On-policy RL for MAS remains underexplored, and role- and turn-dependent prompts complicate standard group-relative optimization. AT-GRPO combines agent- and turn-wise grouping with a MAS training system, reporting gains across domains, especially long-horizon planning.

  • Problem

    RL training for role-specialized MAS remains underexplored, despite MAS and RL being complementary approaches to improving LLM-agent performance.

    Source: §1 · p. 1

  • Method

    AT-GRPO groups RL data by agent and turn and pairs the algorithm with a training system supporting diverse MAS workflows and shared or role-specific policies.

    Source: §1 · p. 2, §4.1 · p. 5

  • Results

    Long-horizon planning accuracy rises from a 14–47% single-agent baseline to 96.0–99.5%, alongside average gains of 3.87–7.62% on coding and 9.0–17.93% on math.

    Source: §1 · p. 2, §6 · p. 11

  • Takeaways & Limitations

    RL training in MAS reinforces role-specific specialization, and choosing shared or role-specialized policies depends on task characteristics.

    Source: §1 · p. 2

    Evaluation is purely computational on public benchmarks and self-constructed simulators and verifiers; it includes no human subjects, sensitive personal data, or proprietary content.

    Source: §7 · p. 12

Abstract

from arXiv · show

Multi-agent systems (MAS) and reinforcement learning (RL) are widely used to enhance the agentic capabilities of large language models (LLMs). MAS improves task performance through role-based orchestration, while RL uses environmental rewards to learn stronger policies, such as GRPO-style optimization. However, applying on-policy RL to MAS remains underexplored and presents unique challenges. Algorithmically, standard GRPO grouping assumptions break down because prompts vary by role and by turn. System-wise, the training stack must support MAS-workflow rollouts and on-policy updates for both single-policy and multi-policy models. We propose AT-GRPO, which includes (i) an agent- and turn-wise grouped RL algorithm tailored to MAS and (ii) a training system that supports both single- and multi-policy regimes. Across game, planning, coding, and math tasks, AT-GRPO delivers substantial gains. On long-horizon planning, it increases accuracy from a 14.0 to 47.0 percent single-agent RL baseline to 96.0 to 99.5 percent. It also improves reasoning performance, with average gains of 3.87 to 7.62 percent on coding tasks and 9.0 to 17.93 percent on math. Code and environments are available at: https://github.com/pettingllms-ai/PettingLLMs.

1 INTRODUCTION

MAS and RL are complementary approaches to improving LLM agents, but RL training for role-specialized MAS remains underexplored and poses algorithmic and systems challenges. The paper introduces AT-GRPO and a training system, reporting gains across agentic domains.

Motivation

  • RL training for role-specialized MAS remains underexplored, although MAS and RL are widely used to improve LLM-agent performance.MAS uses role-based coordination, while RL updates policies using environmental rewards.Source: §1 · p. 1

Challenges

  • Applying on-policy RL to MAS requires coordinating multiple models and environment interactions, while standard group-based objectives do not directly handle role- and history-dependent prompts.Existing on-policy frameworks described here primarily support a single model, and fair credit assignment requires conditioning advantages on interaction history and role.Source: Figure 1 · p. 2

Contributions

  • AT-GRPO is an agent- and turn-wise grouped RL method, paired with a training system supporting diverse MAS workflows and both shared and role-specific policies.The introduction reports a 5.0% higher accuracy (+25.0% relative) on LiveCodeBench with Qwen3-1.7B and an 84.0% improvement on Sokoban with Qwen3-8B.Source: §1 · p. 2
  • AT-GRPO and its training system support on-policy RL for MAS and deliver gains across planning, gaming, coding, and mathematical reasoning.Long-horizon planning accuracy rises from a 14–47% single-agent baseline to 96.0–99.5%; coding and math also improve.Source: §1 · p. 2

2 RELATED WORK

Prior LLM-based MAS research has explored shared and role-specialized policies, while RL extensions have often been limited in interaction structure, role design, or task domain. This paper positions its approach as a broader multi-turn method evaluated across diverse workflows and domains.

Role-sharing vs. Role-specialized Policies in MAS

  • Most MAS studies emphasize inference-time design, while the potential of RL training for MAS policies remains underexplored.Role-sharing systems elicit behavior through prompts; role-specialized policies assign distinct models to roles.Source: §2 · p. 2

RL Training for MAS

  • Prior MAS RL studies often use limited interaction settings or fixed roles, and some focus on one domain, such as math.Examples include single-turn sequential interaction in MARFT and basic single-agent RL algorithms applied to MAS in MARTI.Source: §2 · p. 3
  • The paper presents its study as a general multi-turn, multi-agent approach and evaluates shared and role-specific policies across workflows and domains.Figure 2 depicts a coder–tester loop and distinct workflows for Game/Plan, Code, and Math.Source: §2 · p. 3, Figure 2 · p. 3

3 PRELIMINARIES

The preliminaries model an LLM multi-agent system as a Markov game with sequential agent actions within each turn. They distinguish shared from role-specialized policy updates and describe group-relative action comparisons.

MAS Setting

  • The N-agent system is modeled as a Markov game with agent-specific action spaces, intra-turn state transitions, rewards, and observation histories.Each agent uses a role-specific prompt template, and a turn comprises macro-actions from all agents.Source: §3 · p. 3

Group-based RL

  • Group-based RL samples K candidate actions for a prompt, evaluates them with rule-based rewards, and uses mean-centered, normalized returns as relative advantages.The cited passages describe the standard group-relative setup that the paper adapts for MAS.Source: §3 · p. 4

Role-sharing vs. Role-specialized Policy Optimization

  • Role-sharing pools data from all agents for one joint policy update, whereas role-specialized policies update independently on each agent’s data.Both regimes initialize policies from the same base model.Source: §3 · p. 4

4 METHOD

AT-GRPO forms comparable action groups by agent and turn, then assigns credit using team and local rewards. Its training system coordinates model-specific workers and CPU environment execution for on-policy MAS rollouts and updates.

4.1 ALGORITHM DESIGN: AT-GRPO

  • Tree-structured sampling branches K candidate actions at each agent-turn, producing comparable groups where parallel full-trajectory sampling can leave later groups with size one.The executed action is chosen greedily by reward; the paper says this concentrates exploration on coordination-critical decisions and stabilizes optimization.Source: Figure 3 · p. 4, §4.1 · p. 5
  • The algorithm collects on-policy rollouts, groups candidate data, and constructs per-model batches for policy updates.Algorithm 1 loops through environments, turns, and agents before updating each model.Source: §4.1 · p. 5
  • Grouping by acting agent and turn matches candidates with the same role and turn position, preserving prompt identity for valid advantage comparisons.This addresses prompts that differ across turns as interaction histories accumulate.Source: §4.1 · p. 5
  • Agent-wise credit assignment combines a global team reward with a role-specific local reward, balancing the shared objective and each agent’s subtask incentives.In the coder–tester example, the coder’s local reward uses its code’s pass rate, while the tester’s uses a golden implementation’s pass rate on generated tests.Source: §4.1 · p. 6

4.2 MAS TRAINING SYSTEM

  • Existing LLM RL frameworks primarily support a single agent, policy, data buffer, and resource pool, complicating multi-policy on-policy MAS training and diverse workflows.The proposed system allocates an independent resource pool to each model to support concurrent on-policy training.Source: §4.2 · p. 6
  • GPU-pinned model pools pair rollout and update workers, while CPU environment workers execute steps and route each agent’s experience to its assigned policy.The environment-worker setup uses sandboxed instances and supports thousands of concurrent rollouts in parallel.Source: Figure 4 · p. 6, §4.2 · p. 6

5 EXPERIMENTS

Experiments evaluate AT-GRPO across game, planning, coding, and math, comparing single-agent and multi-agent approaches. Results show strong gains, especially in long-horizon planning, with policy-sharing choices varying by task.

5.1 DATASETS AND MODELS.

  • Experiments compare single-agent and MAS variants with role-sharing or role-specialized policies, aligning observations and reward signals across settings.The evaluation uses Qwen3 1.7B and 8B models, with four rollout samples and a four-turn horizon.Source: §5.1 · p. 7
  • Multi-turn single-agent self-revision brings no empirical improvement and sometimes slightly degrades performance versus the standard single-turn baseline.The variant lacks additional environmental signals and cross-agent feedback.Source: §5.1 · p. 7

5.2 RESULTS AND ANALYSIS

  • Planning accuracy rises from the single-agent baseline range of 14–47% to 96.0–99.5%; coding gains range from +2.35 to +16.30 and math gains from +1.80 to +38.70.Tables 1 and 2 report Qwen3 1.7B and 8B results across game, planning, coding, and math.Source: §5.2 · p. 8, Table 1 · p. 8, Table 2 · p. 8
  • Policy choice depends on task: specialization helps coding by 3.05 points, while shared policies reach 39.60% on OlympiadBench versus 35.20% for role-specific policies.For game and planning, both configurations are near saturation, including 99.50 on Sudoku.Source: §5.2 · pp. 8–9
  • As the ensemble grows, MAS+AT-GRPO improves from 18.2% to 47.7%, while MAS+GRPO saturates at 34.1% in the 7-agent regime.The modular system varies Reasoner and Tool-User counts, with total agents M + N + 1.Source: Figure 5 · p. 9, §5.2 · p. 9

5.3 COMPARISON WITH OTHER MARL FRAMEWORKS

  • On GSM8K, the untrained heterogeneous MAS scores 84.4% versus MAPORL's 81.0%, and AT-GRPO raises its score to 88.7%.The comparison uses Phi-3-mini-128k (3.4B).Source: §5.3 · pp. 9–10, Table 3 · p. 9
  • On CodeContests, the iterative MAS improves accuracy from 22.8% vanilla to 30.3% without training, beating CURE's 25.9%, and reaches 34.2% with training.On math reasoning, the inference-only MAS scores 84.4% versus MARFT's 78.7%, reaching 87.1% after AT-GRPO training.Source: Table 3 · p. 9, §5.3 · p. 10

5.4 ABLATION STUDY

  • The ablation study examines Plan-Path performance gains over a single-agent baseline and tracks standardized rewards and turns required to solve tasks.The supplied results identify the ablation's scope and tracked measures but do not report its outcome values.Source: Table 4 · p. 10, Figure 6 · p. 10, §5.4 · p. 10

6 CONCLUSION

The paper proposes AT-GRPO and a training system for on-policy reinforcement learning in multi-agent systems. Experiments report gains across domains, while policy-sharing choices depend on task characteristics.

6 CONCLUSION

  • Planning accuracy rises from a 14–47% single-agent baseline to 96.0–99.5%, with average gains of 3.87–7.62% on coding and 9.0–17.93% on math.The conclusion reports consistent gains across diverse domains.Source: §6 · p. 11
  • AT-GRPO is an agent- and turn-wise grouped reinforcement learning algorithm, supported by a system for diverse MAS workflows and on-policy updates for multiple policies.Source: §6 · p. 11
  • The choice between shared and role-specialized policies depends on task characteristics, and MAS reinforcement learning reinforces role-specific specialization.Source: §6 · p. 11

7 ETHICS STATEMENT

The study uses public computational benchmarks and self-constructed simulators, with safeguards for code execution and tool use. The authors also caution that base-model biases remain and results are unsuitable for high-stakes decisions.

7 ETHICS STATEMENT

  • Experiments use public benchmarks, programmatically generated instances, and self-constructed simulators and verifiers; they involve no human subjects, sensitive personal data, or proprietary content.The authors plan to release prompts, generators, and evaluation scripts subject to dataset licenses and safe-use guidelines.Source: §7 · p. 12
  • Code runs in a sandbox with restricted file I/O and no network access, and tool calls are limited to deterministic checkers.The authors log actions and withhold configurations enabling networked or privileged execution.Source: §7 · p. 12
  • Base LLMs may retain societal biases, so the authors caution against using the results for high-stakes decisions.They also acknowledge dual-use risks such as unsafe tool use or over-delegation.Source: §7 · p. 12

9 USE OF LLM

A large language model assisted with polishing grammar and improving clarity during manuscript preparation. The authors reviewed and edited its outputs and accept responsibility for the statements and conclusions.

9 USE OF LLM

  • A large language model helped polish grammar and improve the manuscript’s clarity.The authors used it during manuscript preparation.Source: §9 · p. 12
  • The authors reviewed and edited all language-model outputs to ensure the final content reflected their original ideas.Source: §9 · p. 12
  • The authors accept full responsibility for the manuscript’s statements and conclusions.Source: §9 · p. 12

A EXTENDED COMPARISON OF RL TRAINING FOR LLM-BASED MAS

The extended comparison describes how Table 5 contrasts RL-based LLM multi-agent frameworks across workflow and algorithm design dimensions. The supplied material also details task-specific rewards, an outcome-only ablation, and a conditional justification for verifier-greedy action selection.

A EXTENDED COMPARISON OF RL TRAINING FOR LLM-BASED MAS

B.1 MATH REWARD DESIGN

  • Math rewards combine verifiable local sub-scores, while the team receives a sparse terminal reward based on verified numerical equality.The same team-reward scalar is broadcast across turns; math verification uses a parsing and normalization front-end followed by a numeric comparator.Source: §B.1 · p. 16

B.2 CODE REWARD DESIGN

  • The Coder’s local reward combines basic sanity checks with the fraction of golden unit tests passed, while team reward is dense and broadcast across turns.Local scores are masked combinations of verifiable sub-scores.Source: §B.2 · p. 17

B.3 SUDOKU REWARD DESIGN

  • Sudoku uses a sparse terminal team-success signal broadcast across turns, alongside masked local rewards based on verifiable sub-scores.The supplied passages describe N×N Sudoku and a success check for a solved grid.Source: §B.3 · p. 18

B.4 PLAN-PATH REWARD DESIGN

  • Plan-Path provides dense team feedback tied to progress toward the goal, with masked local scores for verifiable sub-scores.The setup uses Manhattan distance for normalization and includes a score for actions on a shortest path.Source: §B.4 · p. 19

B.5 SOKOBAN REWARD DESIGN

  • Sokoban’s dense team reward reflects the box-to-goal ratio and terminal completion, with local scores including legality and deadlock avoidance.Source: §B.5 · pp. 19–20

B.6 OUTCOME-ONLY REWARD DESIGN

  • The outcome-only ablation replaces dense task shaping with a binary episodic team-success reward and auxiliary checks for valid action formatting.The authors describe it as a more general baseline for algorithm ablation, using the same fixed, task-independent α as shaped-reward configurations.Source: §B.6 · p. 20

B.7 THEORETICAL JUSTIFICATION FOR GREEDY TURN-LEVEL TRANSITIONS

  • Under the monotonicity assumption that verifier rewards preserve optimal action-value rankings, the paper states that a verifier-greedy policy is optimal.The implementation approximates this policy by sampling candidate actions and selecting the one with the highest verifier reward.Source: §B.7 · p. 21

C.1 TRAINING DETAILS

The appendix specifies shared training settings and task-specific prompts for coding, math, Sudoku, and path planning. Several workflows pair a tool-producing agent with a planner or verifier, then use mismatch or execution feedback in later turns.

C.1 TRAINING DETAILS

  • Training uses a global batch size of 128, Adam at learning rate 1e-6, GRPO with γ=1.0 and λ=1.0, and four samples per prompt.The actor trains for 150 steps; maximum response length is 4096 tokens, while maximum prompt length is 8192 tokens for math and code and 16384 for other symbolic tasks.Source: §C.1 · p. 21

C.2 PROMPT DESIGN

  • For coding, a Coder generates a solution and a Tester generates a test case; later turns use mismatch history to refine their outputs.The Coder is prompted to fix crashes or mismatches, while the Tester assesses whether its earlier test case was correct.Source: PHASE 1: GENERATION · p. 22, PHASE 2: REFINEMENT · pp. 22–23
  • From the second coding turn onward, agents receive mismatch feedback and use the interaction history to refine prior outputs.Source: PHASE 2: REFINEMENT · p. 22
  • For math, a Reasoning Agent produces a step-by-step solution and boxed answer while a Python Tool Agent computes the answer; later turns use their output mismatches as feedback.Source: PHASE 2: REFINEMENT · pp. 23–24
  • For Sudoku, a Tool Agent generates executable code and a Plan Agent checks its execution output against Sudoku rules before accepting or correcting the solution.Source: PHASE 2: REFINEMENT · pp. 25–26

D PLAN-PATH MAS WORKFLOW

  • In path planning, a Tool Agent generates an action list and a Plan Agent verifies feasibility, correcting sequences that collide, leave bounds, loop, or miss the goal.Later turns use mismatches between the tool’s actions and environment checks or planner assessment to refine outputs.Source: D PLAN-PATH MAS WORKFLOW · pp. 26–28
  • From the second path-planning turn onward, agents use mismatches between proposed actions and feasibility checks to refine their outputs.Source: D PLAN-PATH MAS WORKFLOW · p. 27

E ABLATION STUDY OF OUTCOME REWARD

The outcome-only ablation removes intermediate heuristic rewards to test whether AT-GRPO’s gains depend on dense reward shaping. Under sparse rewards, it retains high performance and outperforms baselines on Plan-Path.

E ABLATION STUDY OF OUTCOME REWARD

  • The ablation removes intermediate heuristic signals and gives positive feedback only when the final task is solved.This is intended to match the sparse feedback available to the baselines.Source: E ABLATION STUDY OF OUTCOME REWARD · p. 28, Table 6 · p. 28
  • Plan-Path accuracy drops only 4.0%, from 93.0% to 89.0%, when dense rewards are replaced by outcome-only rewards; Sudoku remains at 99.5%.The authors conclude dense rewards accelerate learning but are not required for success.Source: E ABLATION STUDY OF OUTCOME REWARD · p. 28
  • 89.0% under sparse outcome-only rewards outperforms the Plan-Path SA baseline at 12.0% and MAS baseline at 71.0%.The comparison uses identical reward conditions after removing intermediate heuristics.Source: E ABLATION STUDY OF OUTCOME REWARD · p. 28, Table 6 · p. 28
  • Tables 7 and 8 report single-agent Code and Math ablations for Qwen3 1.7B and 8B models.The supplied captions identify their scope but do not provide their results.Source: Table 7 · p. 29, Table 8 · p. 29

F MULTI TURN SINGLE AGENT

The section reports that repeatedly revising an answer with one agent offers no consistent advantage, then analyzes how MAS workflow structure and system capacity shape runtime. Asynchronous generation interleaves agent-turn requests, while sequential and parallel workflows have different latency constraints.

F MULTI TURN SINGLE AGENT

  • Multi-turn single-agent variants bring no consistent benefit over single-turn baselines and often degrade Code and Math performance across both model scales.At Qwen3-1.7B SFT, LiveCodeBench falls from 11.6 to 10.4 and AIME24 from 13.4 to 3.3; after RL, single-turn SA still outperforms multi-turn SA across most benchmarks.Source: F MULTI TURN SINGLE AGENT · p. 29

G SYSTEM COMPLEXITY OF AGENT- AND TURN-WISE GROUPING

  • The complexity analysis separates rollout inference from training and tracks how MAS costs scale with agent count and turn horizon.The section considers different interaction patterns and system constraints.Source: G SYSTEM COMPLEXITY OF AGENT- AND… · p. 29

G.1.1 SYSTEM DESIGN WITH ASYNCHRONOUS VLLM GENERATION

  • Asynchronous continuous batching submits each agent-turn query independently, reducing long-tail stragglers and interleaving parallel and sequential workflows.The engine dynamically adds and removes sequences through a token-level scheduler.Source: §G.1.1 · pp. 29–30

G.1.2 INFERENCE-TIME COMPLEXITY

  • Sequential MAS has a critical path of N·T serial agent moves, so its latency overhead scales linearly with agent count N.This compares with a multi-turn single-agent baseline requiring T sequential steps.Source: §G.1.2 · p. 30
  • Parallel MAS batches agents only within cluster capacity E·N·K ≤ Bmax; above Nmax, execution proceeds in ⌈N/Nmax⌉ waves.When N ≤ Nmax, latency scales primarily with debate depth T.Source: §G.1.2 · p. 30

G.2 TRAINING-TIME COMPLEXITY

Training-time grouping adds negligible hashing work relative to model execution and scales linearly with the number of agents. The section also profiles latency and illustrates how MAS workflows can correct task-specific errors in game planning and unit-test generation.

G.2 TRAINING-TIME COMPLEXITY

  • For E environments, N agents, T turns, and K samples, rollout size is E·N·T·K; grouping adds negligible hashing overhead and no extra asymptotic complexity beyond linear scaling with N.The hashing cost is O(|D|MAS), versus token-level execution O(|D|MAS·L·Cmodel).Source: §G.2 · pp. 30–31

G.3 EMPIRICAL LATENCY STUDY

  • Game rollout latency falls from 2.8 minutes for the single-agent baseline to 1.5 minutes for MAS, while Code MAS takes about 8 minutes for rollout and 2 minutes for training.The authors attribute the Game reduction to earlier task completion and early termination in MAS.Source: §G.3 · p. 31

H CASE STUDIES OF MAS WORKFLOWS

  • The case studies show two MAS workflows—a box-pushing grid game and code generation with unit tests—and distinguish erroneous from successful behaviors.The examples mark failures with × and successful behaviors with ✓.Source: H CASE STUDIES OF MAS WORKFLOWS · p. 31

H.1 MAS FOR GAME

  • After RL, the Plan Agent learns to route behind the box before pushing, correcting its pre-RL mistake of following the box path itself.The successful action sequence translates each box move into agent-specific movement and pushes.Source: §H.1 · pp. 31–32

H.2 MAS FOR CODE

  • In unit-test generation, the successful tester supplies correctly formatted input and output fields with consistent variables, unlike failures from missing output or formatting.The workflow has agents revise their outputs using environment feedback and rerun the tests.Source: §H.2 · pp. 33–34
Loading 2510.11062v5…