PaperlayerView exact revision on arXiv

Source-linked AI summary

MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning

Chanwoo Park, Seungju Han, Xingzhi Guo, Asuman Ozdaglar, Kaiqing Zhang, Joo-Kyung Kim

arXiv:2502.18439v2cs.AI

TL;DR

Existing multi-LLM collaboration methods largely rely on prompting and do not reliably induce effective cooperation. MAPoRL co-trains multiple LLMs with verifier-based rewards and multi-agent reinforcement learning, and experiments show improved collaboration across tasks with transfer to unseen domains.

  • Problem

    Prompt-based collaboration does not reliably produce genuine cooperation, while single-agent training may be insufficient to learn behaviors that depend on other agents’ strategies.

  • Method

    MAPoRL co-trains multiple LLMs with multi-agent reinforcement learning, using verifier scores and reward-shaping incentives based on answer quality and collaborative influence.

  • Results

    MAPoRL-trained agents improve accuracy as collaboration progresses and transfer collaborative abilities across mathematical reasoning and natural language inference tasks.

  • Takeaways & Limitations

    Co-training can foster generalizable collaboration skills and collaborative synergy among agents with varying capabilities.

  • Takeaways & Limitations

    The experiments use instruction prompts and relatively small 3B to 8B parameter LLMs, so results may differ with larger models.

Abstract

from arXiv · show

Leveraging multiple large language models (LLMs) to build collaborative multi-agentic workflows has demonstrated significant potential. However, most previous studies focus on prompting the out-of-the-box LLMs, relying on their innate capability for collaboration, which may not improve LLMs' performance as shown recently. In this paper, we introduce a new post-training paradigm MAPoRL (Multi-Agent Post-co-training for collaborative LLMs with Reinforcement Learning), to explicitly elicit the collaborative behaviors and further unleash the power of multi-agentic LLM frameworks. In MAPoRL, multiple LLMs first generate their own responses independently and engage in a multi-turn discussion to collaboratively improve the final answer. In the end, a MAPoRL verifier evaluates both the answer and the discussion, by assigning a score that verifies the correctness of the answer, while adding incentives to encourage corrective and persuasive discussions. The score serves as the co-training reward, and is then maximized through multi-agent RL. Unlike existing LLM post-training paradigms, MAPoRL advocates the co-training of multiple LLMs together using RL for better generalization. Accompanied by analytical insights, our experiments demonstrate that training individual LLMs alone is insufficient to induce effective collaboration. In contrast, multi-agent co-training can boost the collaboration performance across benchmarks, with generalization to unseen domains.

1 Introduction

Prior multi-LLM collaboration methods largely rely on prompting pretrained models, but prompting alone does not reliably produce effective cooperation. MAPoRL addresses this gap by co-training multiple LLMs with multi-agent reinforcement learning and evaluates collaboration across tasks, domains, and agent capabilities.

  • Motivation: Prompt-based multi-agent approaches struggle to achieve genuine collaboration, and additional debate turns do not consistently improve performance.The paper attributes this limitation to LLMs not being explicitly trained for effective cooperation.
  • MAPoRL: MAPoRL co-trains multiple LLMs with multi-agent reinforcement learning so agents receive rewards for answer quality and collaborative interactions.The paradigm optimizes each agent’s expected cumulative reward within a predefined multi-agent framework.
  • MAPoRL: MAPoRL adds incentives for successful interactions and penalties for collaboration failures to encourage more effective and aligned agent behavior.A simplified game-theoretic analysis supports the insufficiency of single-agent training and the emergence of cooperative behavior among co-trained agents.
  • Evaluation: The experiments cover mathematical reasoning, natural language inference, out-of-domain evaluation, and collaboration among heterogeneous LLMs.These evaluations test benchmark performance, cross-domain generalization, and interactions between agents with varying capabilities.
  • Contribution: The study is among the first to train multi-LLM systems as a whole with reinforcement learning for multi-LLM collaboration.This distinguishes the work from approaches that primarily prompt pretrained models.

2 Analytical Insights: Collaborate to Solve Hard Questions

The paper models multi-LLM collaboration as strategic, repeated interaction in which jointly adapting agents can achieve cooperation that single-agent training may fail to induce. MAPoRL operationalizes this idea with verifier-based, influence-aware rewards and multi-agent reinforcement learning over collaborative responses.

  • Analytical Model: The analytical framework models LLM interaction as a repeated game in which agents choose collaboration or independent action across multiple turns.Collaborative synergy occurs when the total number of collaborative actions reaches a question-specific threshold, yielding a reward associated with a correct solution.
  • Analytical Insights: An agent’s optimal action depends on the opponent’s collaboration probability, so a non-collaborative opponent can induce non-collaborative behavior.The single-agent strategy responds to the fixed opponent rather than learning mutual strategic adaptation.
  • Toy Experiments: Jointly optimizing agents coordinate at higher collaboration rates because simultaneous adaptation shapes strategic behavior from the first turn.The toy experiment computes an entropy-regularized Nash equilibrium for τ = 0.1 via backward induction.
  • MAPoRL Formulation: MAPoRL evaluates each response with a verifier and uses current and future verifier scores to update agent policies with multi-agent PPO.This reward design incorporates both immediate performance and the anticipated influence of responses on later collaborative interactions.
  • Collaborative Debate: The collaborative debate formulation lets each agent generate its next response from the question and the history of its own and other agents’ answers.The framework applies to any multi-LLM system whose responses can be evaluated by a scorer or verifier.
  • Reward Shaping: Reward shaping assigns α0 and α1 to an agent’s own revision behavior and β0 and β1 to its influence on other agents’ answers.These incentives respectively represent extracting useful information from incorrect answers, accepting correct information, providing useful incorrect information, and persuading others with correct answers.

4 Experiments

MAPoRL is evaluated on mathematical reasoning and natural language inference using multiple datasets, models, collaboration settings, and reward-shaping analyses. The experiments compare off-the-shelf and MAPoRL-trained collaboration, examine answer transitions and incentives, and test cross-domain generalization.

  • 4.1 Datasets: MAPoRL is evaluated on GSM8K and ANLI, covering mathematical reasoning and adversarial natural language inference.GSM8K evaluation holds out 1,319 samples, while ANLI uses 1,200 testing samples.
  • 4.2 Models: The experiments use Phi-3-mini-128k-instruct, Qwen2.5-3B-instruct, and Llama-3-8B-instruct, primarily with quantization and QLoRA fine-tuning.Maximum evaluation lengths are 300 tokens for GSM8K and 250 for ANLI.
  • 4.3 Experiment 1: Off-the-shelf LLMs did not improve across additional collaboration turns, whereas MAPoRL-trained LLMs improved as collaboration turns increased.The comparison used two agents collaborating over three turns, with training beginning at turn t ≥2.
  • 4.3 Experiment 1: Similar no-collaboration performance across off-the-shelf and MAPoRL-trained models suggests MAPoRL improved collaboration ability rather than task-specific knowledge.The comparison included off-the-shelf Phi-3 and MAPoRL-trained turn-2 and turn-3 models without interaction history.
  • 4.4 Experiment 2: 9.5% improvement in ∆1 followed increasing α1 to 2, while increasing α0 to 2 produced a 2.57% improvement in ∆0.The results associate α1 with following correct majority opinions and α0 with deviating from incorrect majority opinions.
  • 4.4 Experiment 2: 17.2% improvement in ∆0 followed increasing β0 to 2, while increasing β1 produced a -1.32% change in ∆1.The analysis describes β0 as rewarding constructive influence even when wrong and β1 as incentivizing influence when correct.
  • 4.4 Experiment 2: MAPoRL-trained models showed performance improvement on GSM8K and ANLI, and models trained on one task generalized collaborative capabilities to unrelated tasks.The reported gains for three debating turns were relatively modest, while the cross-task findings support generalization beyond the training domain.

5 Concluding Remarks, Limitations, and Potential Risks

MAPoRL co-trains multiple LLMs with multi-agent RL to foster collaboration, and experiments report improved, transferable collaborative performance. The paper also identifies prompt sensitivity, small-model evaluation, majority-vote aggregation, and risks from adversarial agents.

  • Concluding remarks: MAPoRL co-trains multiple LLMs with multi-agent RL rather than relying on prompting or single-agent fine-tuning.A verifier shapes rewards to balance immediate correctness with longer-term collaborative potential.
  • Concluding remarks: Accuracy increases as collaboration progresses for MAPoRL-trained agents, while off-the-shelf LLMs often do not improve with additional debate turns.The experiments cover reasoning-intensive tasks including GSM8K and ANLI.
  • Concluding remarks: MAPoRL-trained collaborative abilities transfer across tasks, and heterogeneous models can develop collaborative synergy despite differing capabilities.The reported transfer suggests a generalizable collaboration skill across domains.
  • Limitations: Results may vary with instruction prompts, and experiments use relatively small 3B-to-8B-parameter LLMs.These choices support fast iteration but limit direct conclusions about larger models.
  • Limitations: Majority voting determines the final answer, while alternative aggregation such as a manager agent may further improve performance.The paper presents this as an available design alternative rather than an evaluated conclusion.
  • Potential Risks: Enabling collaboration among adversarial or malicious agents could produce unintended harmful outcomes.The risk follows from the method facilitating collaboration among multiple agents.

C Deferred Content of Section 2

The deferred analysis formalizes when collaboration is optimal and explains why simultaneous co-training can produce cooperative equilibria. Entropy regularization smooths strategies, while backward induction analyzes the multi-turn game.

  • Game-theoretic analysis: Collaboration is optimal when the expected synergy gain from the opponent’s cooperation meets or exceeds the independent-action advantage.The condition is (Rsyn(q) − Rind(q))π(q) ≥ Rind(q) − Rcol(q).
  • Game-theoretic analysis: In the final turn, independent action is optimal because synergy is already secured if achieved, while Rcol(q) is lower than Rind(q) otherwise.This backward-induction result determines the final-turn strategy regardless of the opponent’s action.
  • Equilibrium analysis: An entropy-regularized Nash equilibrium combines expected reward with an entropy term, smoothing best responses into a softmax or logit strategy.The temperature parameter τ controls the regularization strength.
  • Equilibrium analysis: As τ approaches zero, the regularized equilibrium converges to collaboration under the paper’s stated reward condition.The result concerns the unique equilibrium for each fixed positive τ.

D Deferred Details in Section 2.3

The deferred details specify the toy game’s reward sampling, synergy setting, verifier architecture, and Figure 5’s comparison of adaptive multi-agent learning with fixed-opponent best responses.

  • Figure 5: Figure 5 plots turn-1 collaboration probability against threshold C for horizons T = 10 and T = 20.The threshold varies from T−1 down to ⌊(T−1)/2⌋, with each point averaging 5000 random reward samples.
  • Figure 5: The red Multi-Agent curve represents adaptive learning by both players, whereas blue curves show Player 1’s best response to fixed opponent probabilities 0.5, 0.6, and 0.7.The figure contrasts joint adaptation with responding to a fixed opponent.
  • Verifier models: A verifier assesses the correctness of a complete reasoning path using either outcome-level or process-level reward models.Process-level evaluation generally performs better but has limited annotated-data availability.
  • Verifier models: The verifier uses a quantized language-model backbone, followed by a linear head and softmax output constrained to 0–1.The default backbone is Microsoft Phi-3-mini-128k-instruct.

E.1 Training Procedure

The training procedure builds a verifier from generated solution paths and uses its token-level correctness estimates to shape several reinforcement-learning reward designs.

  • Verifier Training: Verifier training generates multiple solution paths per question and labels token-level prefixes according to whether the corresponding final answer is correct.The data tuples contain questions, generated solutions, answers, and binary labels for prefixes of each solution.
  • Verifier Training: 7,463 GSM8K questions each produced 100 reasoning paths, while ANLI used 10,000 questions with 50 paths each.Outputs failing the required format were excluded before verifier training.
  • Verifier Training: The verifier is optimized with token-level binary cross-entropy, with experiments finding better results when focusing on the latter half of generated solutions.The default optimization uses all solution tokens, but the latter-half focus performed better in practice.
  • Verifier Training: With sufficient expressiveness, the cross-entropy-optimal verifier estimates the probability that the final answer is correct given the question and solution prefix.The analysis extends related work from ℓ2 loss to entropy loss commonly used for classification.
  • Reward Design: MAPoRL considers immediate, cumulative, and influence-aware verification rewards to account for current correctness, future turns, and effects on other agents.Cumulative rewards discount future verifier scores, while influence-aware rewards average scores across agents affected by the current response.

G.1 Efficient Network Architecture for MAPoRL

MAPoRL implements multiple agents and value functions efficiently with QLoRA adapters on a shared base-model architecture, using a separately headed value model.

  • Efficient Architecture: Each agent and value function uses a QLoRA adapter containing less than 0.2% of the base LLM’s parameters.The value function is a QLoRA-fine-tuned model with an added value head.
  • Efficient Architecture: Multiple language models are implemented with QLoRA adapters on the same base-model architecture to fit within limited GPU memory.The setup uses separate adapters while retaining the common base architecture.
  • Efficient Architecture: The value model uses a pretrained LLM fine-tuned with an additional linear head layer.This head supplies the value-function output used during training.
  • Efficient Architecture: The value function can use the question and prior multi-agent interaction history as its input, simplified as a concatenation of the question and current sequence prefix.The stated simplification assumes the current response contains the necessary earlier-turn information.

G.2 Experimental Setup and Hyperparameter Configuration for MAPoRL

The experimental setup specifies output-format constraints, verifier-based reward handling, training resources, and prompts that require concise reasoning and boxed answers.

  • Configuration: Responses shorter than 50 tokens receive a penalty, and answers without \boxed{} are assigned a verifier score of -10.The setup also enables non-eos penalties and non-box penalties.
  • Configuration: Training uses 8 NVIDIA A100-80GB GPUs, processes 12 episodes simultaneously, and applies four iterations of multi-agent PPO updates per rollout.The PPO batch size is 1 with gradient accumulation of 4.
  • Evaluation: Post-processing uses an LLM to extract final answers so evaluation reflects reasoning and collaboration rather than formatting proficiency.This standardizes answer extraction before computing performance metrics.
  • Configuration: The prompts require step-by-step but concise reasoning, consideration of other agents’ answers, and final answers in the format Answer: \boxed{XX}.The NLI prompt asks models to classify premise-hypothesis relationships as entailment, neutral, or contradiction.
  • Configuration: For off-the-shelf comparisons, experiments allow 600 tokens per response, twice the MAPoRL output-token limit.The comparison therefore gives the off-the-shelf systems a larger response budget.
  • Configuration: Verifier rewards represent estimated answer correctness on a 0-to-1 scale, with larger values indicating higher likelihood of correctness.The rewards provide useful but imperfect context and are generally described as reliable.

G.7 Ablation Study: Verifier Robustness

Ablations show that multi-agent co-adaptation remains beneficial without verifier shaping, while generator–verifier architectural mismatch degrades reward fidelity and accuracy.

  • Verifier Rewards: Even with only binary episode rewards and no intermediate verifier signal, MAPoRL improves performance across discussion turns.The binary reward is 1 for a correct final answer and 0 otherwise.
  • Verifier Rewards: Removing verifier shaping lowers performance somewhat, but multi-agent co-adaptation remains beneficial under sparse supervision.The continued turn-by-turn improvement indicates gains do not depend entirely on intermediate verifier rewards.
  • Architectural Alignment: Using different model families for verifier and generator degrades reward signals and can stylistically shift generation toward the verifier’s base model.This drift often results in reduced accuracy.
  • Architectural Alignment: The authors therefore co-train verifier and generator on the same model base to provide stable, meaningful reward supervision during multi-turn training.Architectural alignment is presented as preventing unintended distribution shifts.

G.8 Ablation Study: Comparison to Single-Agent RL with Verifier

The ablation compares single-agent RL with verifier rewards against MAPoRL, emphasizing that collaboration—not isolated task optimization—is the framework’s broader objective. MAPoRL-trained agents improve through iterative reasoning exchange and revision.

  • MAPoRL targets learned collaboration and influence, rather than maximizing individual task performance alone.
  • A single-agent RL model achieved 0.732 final accuracy, below MAPoRL-trained agents at later turns.
  • MAPoRL’s gains arise from iterative exchange of reasoning and revision across agents, not merely stronger task optimization.
  • In the analyzed limb problem, agents initially made an addition error, then corrected it to addition in the next turn.

G.9.2 Answers from off-the-shelf LLMs Agent 1’s second turn answer

The off-the-shelf agents produced conflicting and often incorrect solutions to the limb-growth problem. Their reasoning included unsupported growth interpretations, while one answer received a high verifier score despite the broader errors.

  • Agent 1’s second turn answer: Agent 1 reused a previous reply without correcting its reasoning, while Agent 2 introduced entirely incorrect logic from the outset.
  • Agent 3’s second turn answer: Agent 3’s answer concluded four extra limbs by combining growth intervals with a constraint against more than two of the same limb type.
  • Agent 3’s second turn answer: The verifier assigned Agent 3’s answer a reward of 0.932 out of 1.0, indicating high confidence in that conclusion.
  • MAPoRL-trained LLMs identified a useful hint and reached the correct answer, whereas off-the-shelf LLMs failed to do so.

G.10.2 Answers from Off-the-Shelf LLMs

The off-the-shelf agents also struggled with a star-row counting problem, repeatedly introducing nonexistent seven-star rows and producing inconsistent final answers.

  • Phi-3 produced 24 as its final answer after incorrectly incorporating a row of seven stars.
  • Another Qwen2.5 calculation counted 90 total stars and concluded there were 18 five-star rows.
Loading 2502.18439v2…