PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM Collaboration With Multi-Agent Reinforcement Learning

Shuo Liu, Tianle Chen, Zeyu Liang, Xueguang Lyu, Christopher Amato

arXiv:2508.04652v7cs.AIcs.SE

TL;DR

LLM collaboration calls for multiple agents to work together, but the paper identifies applying MAS principles to that setting as an open opportunity. It formalizes collaboration as cooperative MARL and develops MAGRPO; experiments report efficient, high-quality responses in writing and coding, while the study is limited to homogeneous agents, relatively small models, limited datasets, and short horizons.

  • Problem

    The paper identifies the application of multi-agent system principles to LLM collaboration as an opportunity for agents to solve complex tasks more robustly and efficiently.

    Source: Introduction · p. 1

  • Method

    The paper formalizes LLM collaboration as cooperative MARL and develops MAGRPO for joint training with centralized group-relative advantages and decentralized execution.

    Source: Introduction · p. 1, Problem Formalization · p. 2

  • Results

    Experiments in writing and coding collaboration show that MAGRPO enables agents to generate high-quality responses efficiently through effective cooperation.

    Source: Introduction · p. 2, Conclusion · p. 7

  • Takeaways & Limitations

    The work encourages exploration of other MARL-based methods for scalable and robust LLM collaboration.

    Source: Conclusion · p. 7

    The study uses homogeneous agents, limited datasets, relatively small language models, and short training horizons due to computational constraints.

    Source: Limitations and Future Works · p. 17

Abstract

from arXiv · show

A large amount of work has been done in Multi-Agent Systems (MAS) for modeling and solving problems with multiple interacting agents. However, most LLMs are pretrained independently and not specifically optimized for coordination. Existing LLM fine-tuning frameworks rely on individual rewards, which require complex reward designs for each agent to encourage collaboration. To address these challenges, we model LLM collaboration as a cooperative Multi-Agent Reinforcement Learning (MARL) problem. We develop a multi-agent, multi-turn algorithm, Multi-Agent Group Relative Policy Optimization (MAGRPO), to solve it, building on current RL approaches for LLMs as well as MARL techniques. Our experiments on LLM writing and coding collaboration demonstrate that fine-tuning MAS with MAGRPO enables agents to generate high-quality responses efficiently through effective cooperation. Our approach opens the door to using other MARL methods for LLMs and highlights the associated challenges. Our code is available at https://github.com/OpenMLRL/CoMLRL.

Introduction

The paper applies cooperative multi-agent reinforcement learning to LLM collaboration, developing MAGRPO to train agents jointly and reporting efficient, high-quality collaboration in writing and coding.

Motivation

  • Prior RL and cooperative MAS work motivates using reinforcement learning for complex multi-turn tasks and applying MAS principles to LLM collaboration.The cited background includes Ouyang et al. 2022; Guo et al. 2025; Ziegler et al. 2019, and cooperative MAS applications such as games, robotics, and traffic control.Source: Introduction · p. 1
  • Existing LLM coordination approaches use prompt-level interaction or individual and role-conditioned rewards, with prompt and reward design challenges and no convergence guarantees for independent learning.The related approaches are described by Du et al. 2023; Wu et al. 2023a; Lifshitz, McIlraith, and Du 2025, and Liu et al. 2025a; Subramaniam et al. 2025; Tan 1993.Source: Introduction · p. 1

Contributions

  • MAGRPO models LLM collaboration as cooperative MARL and trains agents jointly with centralized group-relative advantages while retaining decentralized execution.The experiments report improved response efficiency and quality in writing and coding collaboration.Source: Introduction · pp. 1–2

Related Work

Prior LLM collaboration work coordinates agents through prompts or fine-tunes them with individual rewards; this section describes the coordination and reward-design limitations motivating cooperative training.

Test-Time Multi-Agent Interaction

Multi-Agent Fine-Tuning

  • Fine-tuning approaches commonly use individual or role-conditioned rewards that require careful manual specification, and independent learning lacks convergence guarantees in a non-stationary environment.The paper focuses on cooperative scenarios trained with verified, human-aligned rewards.Source: Multi-Agent Fine-Tuning · p. 2

Cooperative MARL for LLM Collaboration

The paper frames LLM collaboration as cooperative MARL and formalizes it as a Dec-POMDP, with natural-language actions and partial observations shaping the problem and its training choices.

LLM Collaboration

  • LLM agents synchronously generate responses to their own instructions, with the responses jointly forming a solution and later prompts reflecting feedback as the environment changes.Interaction continues until successful completion or a predefined turn limit.Source: LLM Collaboration · p. 2

Cooperative MARL for LLM Collaboration

Problem Formalization

  • The Dec-POMDP comprises agents, states, observations, actions, a joint reward, state transitions, and an episode horizon.Agents receive natural-language prompts as partial, noisy observations, produce natural-language responses, and receive joint rewards based on accessible state and joint actions.Source: Problem Formalization · p. 2

Challenges in LLM Collaboration

Representations in Natural Language

  • The paper models each agent's decision as a direct mapping from input instructions to complete responses, while noting that the best modeling and solution approaches remain open.Token-level rewards can be uninformative because queries and responses must be coherent and semantically meaningful.Source: Representations in Natural Language · p. 3

Training Paradigm

  • The training design balances centralized training with decentralized execution using group-based Monte Carlo estimates, avoiding centralized value models that do not scale well to LLM-sized spaces.The paper reports experiments demonstrating effectiveness on short-horizon tasks.Source: Training Paradigm · p. 3

MAGRPO

MAGRPO jointly trains LLM agents in multi-turn collaboration while preserving decentralized execution, using shared rewards and group-based estimates to guide policy updates.

MAGRPO

  • MAGRPO jointly trains LLM agents in a multi-agent system while maintaining decentralized execution.The algorithm samples tasks from a dataset, initializes observations, and proceeds through episodes with multiple turns.Source: MAGRPO · p. 3
  • At each turn, agents generate response groups whose actions are aggregated into joint actions, receive joint rewards, and update observations and histories until completion or the horizon.The process is described as using a shared reward model and task information from the dataset.Source: MAGRPO · pp. 3–4
  • MAGRPO estimates history returns by averaging Monte Carlo samples of joint returns, providing a centralized estimate without a large value model.The method uses centralized advantage values to update each agent's policy.Source: MAGRPO · p. 4
  • The resulting centralized advantages are used to update each agent's policy; the method omits importance sampling and epsilon clipping for simplicity.The KL divergence coefficient is set to zero to encourage greater policy deviation from the base model.Source: MAGRPO · p. 4

Experiments

Experiments test MAGRPO on writing and coding collaboration, with outcomes varying across datasets and training settings.

Writing Collaboration

  • Writing experiments cover TLDR summarization and arXiv expansion, using two Qwen3-1.7B agents and simple proxy rewards for structure, style consistency, and coherence.The expansion agents split background and motivation from methods and experiments; the proxy metrics are combined by weighted summation.Source: Writing Collaboration · pp. 4–5

Baselines

  • Writing baselines include a single agent, parallel task splitting, a one-way sequential pipeline, and one-round bidirectional discussion, all without fine-tuning.The comparison keeps task descriptions fixed and adds minimal instructions.Source: Baselines · p. 5

Results

  • Writing-collaboration returns rise across the reported metrics, although TLDR style consistency falls during the first 100 training steps before realigning.The authors attribute the temporary divergence to agents optimizing other cooperative objectives.Source: Results · p. 5, Figure 2 · p. 5
  • MAGRPO is 3 times faster than the comparable-size Qwen3-4B model, while prompt-level multi-agent baselines lag in structure, coherence, efficiency, or total return.Parallel generation approaches MAGRPO's speed but lacks cooperation; sequential and discussion-based prompting underperform in efficiency and coherence.Source: Results · pp. 5–6, Table 1 · p. 5

Coding Collaboration

  • Coding collaboration assigns one Qwen2.5-Coder-3B agent to generate a helper function and another to generate the main function, then aggregates their outputs.On HE, rewards check structure, syntax, test pass rate, and appropriate use of the helper; CHE filters for or adds problems with decomposable cooperative potential.Source: Coding Collaboration · p. 6, HumanEval · p. 6, CoopHumanEval · p. 6

CoopHumanEval

  • On HE, single-turn training improves syntax and valid cooperation, while test pass rate makes little progress; CHE training yields higher returns and lower variance than HE.With CHE, multi-turn returns eventually surpass single-turn returns despite agents initially struggling to interpret feedback.Source: Figure 3 · p. 6, CoopHumanEval · p. 7
  • MAGRPO outperforms all baselines on both CHE and HE, while baseline coordination methods trade off test performance, communication, or inference speed.The sequential pipeline can improve test pass rate but runs more slowly; one-round discussion remains limited by potentially misaligned agent adaptations.Source: CoopHumanEval · p. 7, Table 2 · p. 7
  • MAGRPO produces several coding cooperation schemes, including auxiliary core logic, main-agent coordination, and auxiliary strategy filtering, alongside fallback or decoration roles.Coordinator and strategy-filter schemes can improve inference efficiency but are more prone to syntax and logical errors; limited cooperation-oriented training data favors conservative roles.Source: CoopHumanEval · p. 7

Conclusion

The paper frames LLM collaboration as cooperative MARL and proposes MAGRPO; writing and coding experiments report efficient, high-quality responses through collaboration.

Conclusion

  • The paper models LLM collaboration as a cooperative MARL problem formalized as a Dec-POMDP, proposes MAGRPO with shared rewards, and reports efficient, high-quality responses in writing and coding experiments.The conclusion encourages further exploration of MARL-based approaches for scalable and robust LLM collaboration.Source: Conclusion · p. 7

Formalizations of Multi-Agent Interaction

The paper contrasts Dec-POMDP and POSG formalizations, emphasizing Dec-POMDP’s solution concept for cooperative settings and its shared-reward objective.

Formalizations of Multi-Agent Interaction

  • Dec-POMDP offers special merits over POSG in cooperative settings through its solution concept, making it more suited to modeling LLM collaboration.The passage states this comparison but does not specify the merits further.Source: Formalizations of Multi-Agent… · p. 10

Dec-POMDP

  • A Dec-POMDP is defined by the tuple ⟨I, S, {Oi}, {Ai}, R, T, H⟩.The listed elements are agents, states, observation and action spaces, reward, transition, and horizon.Source: A Dec-POMDP · p. 10
  • Because agents cannot directly observe the state, each maintains a local observation-action history to infer a belief about it.The history contains observations and actions up to the current time.Source: A Dec-POMDP · p. 10
  • Each local policy maps an agent’s history to a probability distribution over actions, and the local policies form the joint policy.Source: A Dec-POMDP · p. 10
  • In cooperative settings, the objective is to maximize shared cumulative rewards.Source: A Dec-POMDP · p. 10
  • A Dec-POMDP always has an optimal joint policy, as proved by Oliehoek, Spaan, and Vlassis (2008).Source: A Dec-POMDP · p. 10

POSG

POSG represents interactions that may be cooperative, competitive, or mixed, with each agent pursuing individual returns under other agents’ fixed policies.

POSG

  • A POSG, also called a POMG, permits cooperative, competitive, or mixed behavior and assigns each agent an individual reward function.Source: POSG · p. 10
  • POSG solutions are Nash equilibria, where no agent can unilaterally improve its return by changing its policy.Source: POSG · p. 10
  • POSG decisions are concurrent, with agents acting synchronously; turn-based interactions are typically modeled as extensive-form games.Source: POSG · p. 10

Non-Optimality of POSG Solutions

The examples show that a POSG Nash equilibrium need not be jointly optimal in a cooperative game. One decomposition yields a jointly optimal equilibrium, while another has deterministic and probabilistic equilibria below the joint optimum.

Non-Optimality of POSG Solutions

Additional Results

The section reports additional results to validate the approach’s effectiveness.

Additional Results

@k Ablation

This section examines coding collaboration through prompts, baseline designs, k-run evaluations of MAGRPO, and cooperation schemes that emerge under minimal training constraints. It also outlines the broader implications, limitations, experimental resources, and released research tools.

@k Ablation

  • Multi-turn MAGRPO outperforms all baselines on most @k metrics; sequential pipelines improve them substantially, especially on CHE, while naive concatenation and one-round discussion have limitations.The evaluation generated 15 samples and used k = 10, reporting pass@k, acc@k, and coop@k at k = 3, 5, and 10; acc@k can capture gains that pass@k misses when no added solution passes every test.Source: @k Ablation · pp. 11–12

Fallback

  • The main agent may implement a fallback when it doubts the auxiliary output, improving robustness for small models prone to syntax and runtime errors.In the example, the auxiliary function can trigger a type mismatch, and the main agent handles it with a secondary implementation.Source: Fallback · p. 12, Cooperation Schemes · p. 12

Decorator

  • When the auxiliary handles core logic, the main agent can decorate its output with edge-case handling or formatting, but this may be redundant if the auxiliary is already complete.The palindrome example contrasts an auxiliary needing a main-agent empty-string check with one that already handles the edge case.Source: Decorator · p. 12

Coordinator

  • In coordinator schemes, the main agent divides work into subtasks for the auxiliary and aggregates partial results, but failure on any subtask can break the solution.The paper notes that this scheme can extend to hierarchical task assignment.Source: Coordinator · pp. 12–13

Strategy Filter

  • A strategy-filter scheme places the auxiliary in conditional logic to evaluate a branch, while the main agent uses its result to choose the corresponding action.The example checks whether a number is prime, returning x if it is and y otherwise.Source: Strategy Filter · p. 13

Learning Modes

  • Without external feedback, second-turn performance remains similar to first-turn performance; with concrete guidance, the second turn exceeds the first, and both outperform self-improvement.Guidance includes expert suggestions or sandbox error messages; the authors hypothesize that larger models may benefit from more implicit guidance.Source: Learning Modes · pp. 13–14, Figure 5 · p. 13

Hyperparameters

  • The experiments use different sampling, learning-rate, and training-step settings for writing, single-turn coding, and multi-turn coding.Writing uses temperature 0.8, nucleus threshold 0.95, beta 0.02, learning rate 5 × 10−6, and 1,500 steps; coding uses 1,500 steps for single-turn and 2,200 for multi-turn, with learning rates 1 × 10−6 and 5 × 10−6 respectively.Source: Experimental Configurations · p. 14, Hyperparameters · p. 14

Reward Specifications

  • Rewards combine multiple metric-based components hierarchically, with task-specific criteria for writing and coding collaboration.Writing rewards include structural wellness, style consistency, and logical coherence; coding rewards cover function structure, syntax, test pass rate, and substantive use of the auxiliary.Source: Reward Specifications · p. 14, TLDR Summarization · p. 14, arXiv Expansion · p. 14

Baseline Methods

  • Baselines range from single-agent generation and naive concatenation to sequential helper-to-main pipelines and one-round discussion, each trading coordination or speed differently.Sequential pipelines allow one-way helper-to-main communication but slow inference; discussion shares agents’ initial responses but does not ensure convergence or quality.Source: Baseline Methods · pp. 16–17

Coding Collaboration

  • MAGRPO’s multi-turn coding setup uses Claude-Sonnet-4 suggestions alongside prior context and each agent’s previous response in subsequent-round prompts.The external advisor is prompted to review helper and main-agent code, suggest edits for clear test-breaking errors, and return both fields in JSON.Source: Coding Collaboration · pp. 15–16, Baseline Methods · p. 17

Broader Impacts

  • The authors frame cooperative MARL as a way to optimize agents jointly and potentially strengthen test-time collaboration through task decomposition and iterative feedback [Du et al. 2023; Lifshitz, McIlraith, and Du 2025; Wu et al. 2023a].They also propose decentralized specialized agents as a route to modular, robust reasoning that can ease a single model’s long-context and joint-decision burden.Source: Broader Impacts · p. 17

Limitations and Future Works

  • The study uses homogeneous agents, limited datasets, small-scale models, and a short training horizon, while its simple reward risks narrow signals and reward hacking.The authors identify heterogeneous agents, larger multi-file projects, and more expressive multi-aspect or process-supervised rewards as future directions.Source: Limitations and Future Works · p. 17

Compute Resources

  • Training used NVIDIA H200 and H100 GPUs, while inference ran on a standalone NVIDIA GeForce RTX 5090 workstation.Training took approximately 5 hours for 700 steps and 8 hours for 1000 steps; a 15-generation evaluation took about 0.5 hours, with duration varying by node conditions and output stochasticity.Source: Compute Resources · pp. 17–18

Codes and Datasets

  • The authors released CoMLRL, an open-source library with configurable MARL algorithms for LLM collaboration, and associated experiment repositories as tools for future research.The paper’s experiments form part of the library’s environments and benchmarks.Source: Codes and Datasets · p. 18
Loading 2508.04652v7…