PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving LLM Collaboration via Multi-Agent Preference Learning

Shuo Liu, Xinzichen Li, Tianle Chen, Christopher Amato

arXiv:2609.32827v1cs.AIcs.MA

TL;DR

Reliable rewards for multi-agent LLM collaboration are difficult to construct, and preference learning remains underexplored in these systems. The paper formulates decentralized and centralized preference-based collaboration and introduces MAPL, instantiated with MARLHF and MADPO. Experiments show MAPL approaches well-specified-reward MARL, with performance varying across paradigms and methods.

  • Problem

    Hand-crafted rewards for LLM collaboration may be unavailable or fail to capture task objectives, and preference learning for multi-agent systems is underexplored with coverage and reward-estimation challenges.

    Source: §1 · pp. 1–2

  • Method

    MAPL iteratively updates agent policies by comparing their solutions with comparator-generated solutions, using MARLHF or MADPO across decentralized and centralized paradigms.

    Source: §1 · p. 2

  • Results

    Across collaborative writing, coding, tool use, and travel planning, MAPL approaches MARL with well-specified rewards; MARLHF generally outperforms MADPO, while Dec-Cen performs poorly in most domains.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    MAPL improves decentralized and centralized collaboration across the evaluated tasks, while larger or centralized comparators do not necessarily facilitate training.

    Source: §6 · p. 10

    MARLHF can show high variance with limited preference data, substantial comparator-agent policy differences, or actor-critic policy optimization.

    Source: §1 · p. 2

Abstract

from arXiv · show

Several works have explored multi-agent reinforcement learning (MARL) in LLM collaboration. However, constructing reliable rewards is difficult in practice, as complete and accurate metrics are often unavailable and hard to aggregate. Preference learning provides an alternative by learning from comparative human or AI feedback. Yet, its extension to multi-agent systems remains underexplored. To address this gap, we formulate preference-based multi-agent systems (MAS) from decentralized and centralized collaboration perspectives. We also introduce a general multi-agent preference learning framework (MAPL) to solve these problems. MAPL allows iterative updates by comparing the current solution with decentralized or centralized solutions generated by various agents. We instantiate MAPL using MARL from human feedback (MARLHF) with a learned reward model and multi-agent direct preference optimization (MADPO). Experiments on collaborative writing, coding, tool use, and travel planning show that MAPL can improve collaboration quality and efficiency while approaching the performance of MARL with fixed, well-defined rewards. Within MAPL, MARLHF generally outperforms MADPO on most tasks but remains sensitive to data coverage, agent and comparator models, and the underlying MARL algorithms.

1. Introduction

The section motivates preference-based learning for LLM collaboration because hand-crafted rewards may be unavailable or misaligned and multi-agent preference data is difficult to cover. It introduces MAPL and reports results across four task domains, including differences among collaboration paradigms and learning methods.

1. Introduction

  • Hand-crafted rewards can be unavailable or fail to capture task objectives, while multi-agent preference learning faces sparse coverage and potentially inaccurate or high-variance learned rewards.The large policy space of multi-agent systems makes preference data difficult to cover adequately.Source: §1 · pp. 1–2
  • MAPL iteratively updates agent policies by comparing joint solutions with comparator-generated solutions across decentralized and centralized collaboration arrangements.The framework includes Dec-Dec, Cen-Cen, and Dec-Cen paradigms.Source: §1 · p. 2
  • MAPL is instantiated as two-stage MARLHF with a learned reward model and as direct preference optimization through MADPO.Source: §1 · p. 2
  • Across collaborative writing, coding, tool use, and travel planning, MAPL approaches MARL with well-specified rewards; MARLHF generally outperforms MADPO but is sensitive to data, models, and optimization.Dec-Dec and Cen-Cen improve collaboration, while Dec-Cen performs poorly in most domains because of information and realizability gaps.Source: §1 · p. 2

2. Preliminaries

The preliminaries introduce pairwise preference modeling and contrast two-stage RLHF with direct preference optimization. They frame these approaches as ways to optimize policies from comparative feedback.

2. Preliminaries

  • The Bradley-Terry model parameterizes pairwise preference probabilities using rewards for competing responses to a prompt.Source: §2 · p. 2
  • In two-stage RLHF, a reward model is trained from pairwise preferences before policy optimization with KL regularization against a reference policy.The paper uses RLHF specifically for this reward-modeling approach.Source: §2 · p. 2
  • DPO directly optimizes the policy without explicitly fitting a reward model.Under the standard derivation, its objective corresponds to the same KL-regularized preference optimization problem as RLHF, though practical performance can differ by task, realizability, and optimization.Source: §2 · p. 3

3. MAS with Preference Feedback

The section formulates preference-based decentralized and centralized multi-agent systems and identifies data and reward-estimation challenges. It describes MAPL’s use of online preferences, agent-generated trajectories, a dynamic replay buffer, and a group-relative MARLHF approach.

3.1. Preference-based Decentralized MAS

  • Decentralized agents act from local contexts and parallelize inference across nodes; the formulation models their interaction as a preference-based decentralized partially observable Markov decision process.Each agent receives a partial, potentially noisy observation, and annotators compare sampled trajectories.Source: §3.1 · p. 3

3.2. Preference-based Centralized MAS

  • Centralized agents condition policies on the full joint action-observation history, while environment transitions, objectives, and preference labeling remain unchanged.A coordinator may instead generate the joint response from that shared history.Source: §3.2 · p. 4

3.3. Challenges

  • MAPL supports online preference collection and builds datasets from joint trajectories of the agents being optimized, preserving their achievable cooperation.Online comparisons address limited coverage in the combinatorially large multi-agent policy space; adapting single-agent datasets by assigning roles may not yield meaningful subtasks or good solutions.Source: §3.3 · p. 4
  • Limited or noisy preference data can make reward estimates inaccurate or high-variance, while jointly updated rewards make training non-stationary.MAPL uses a dynamic replay buffer with diverse training-stage samples and group-relative Monte Carlo MARLHF with mini-batch stochastic gradient descent to improve stability.Source: §3.3 · p. 4

4. Multi-Agent Preference Learning

MAPL is a general framework for optimizing cooperative multi-agent systems from preference feedback, with decentralized and centralized collaboration options. It instantiates this framework through MARLHF and MADPO.

4. Multi-Agent Preference Learning

  • MAPL supports decentralized and centralized collaboration and is instantiated through MARLHF and MADPO.Figure 1 illustrates the collaboration settings, the overall framework, and the two methods.Source: §4 · p. 4, Figure 1 · p. 5

4.1. Framework

  • Each iteration compares current-policy trajectories with comparator-policy trajectories, labels preferences, and places top-k pairs in a replay buffer.The comparator can use previous checkpoints or other model variants.Source: §4.1 · p. 5

4.2. Methods

  • MARLHF learns a history-based reward model from preferences, holds it fixed during policy optimization, and uses group-based Monte Carlo optimization.The method uses the group-average prediction as a baseline for group-relative advantages; its KL term is weighted to yield the trajectory-level KL gradient in expectation.Source: §4.2 · p. 6
  • MADPO directly optimizes policies from the preference replay buffer, implicitly parameterizing reward with the policy’s log probability.Its loss is optimized for each agent under decentralized collaboration or for the coordinator under centralized coordination.Source: §4.2 · p. 6

5. Experiments

The experiments evaluate MAPL across collaborative writing, coding, tool use, and travel planning. Cen-Cen generally performs best, while replay-buffer choice, comparator capacity, and MARL algorithm affect results.

5.1. Setup

  • The four domains cover two-agent Reddit summarization, collaborative Python coding, parallel function calling, and short-trip planning.The tasks assess summary quality; code correctness and auxiliary-call validity; function-call correctness and cost; and itinerary pass and success.Source: §5.1 · p. 7
  • Baselines include single-agent methods, test-time collaboration frameworks, and oracle-reward MARL; MAPL comparators are drawn from the agents' model family.The centralized comparator approximately matches the agents' combined parameter count.Source: §5.1 · p. 7, §5.2 · p. 8

5.2. Experimental Results

  • Cen-Cen generally performs best for MARLHF and MADPO across most domains, while Dec-Cen often performs worse because of information and realizability gaps.The Dec-Cen decline is more pronounced for MARLHF, whose two-stage pipeline amplifies representation error; on TLDR, Cen-Cen with MADPO underperforms Dec-Dec.Source: §5.2 · p. 9

5.3. Ablations

  • Online MAPL exceeds offline MAPL on TLDR, CoopHE, and BFCL; λ-decay achieves the best results across most domains.On BFCL, MARLHF improves from 7.5% to 14.8%, and MADPO from 3.1% to 15.3%; the λ-decay benefit is more pronounced for MARLHF because broader coverage supports more diverse reward-model representations.Source: §5.3 · p. 9, Table 2 · p. 9
  • In the same-size setting, MADPO rises from 58.8% to 65.7% on CoopHE and 15.6% to 22.7% on BFCL; MARLHF changes from 61.0% to 63.5% and 22.7% to 21.8%, respectively.On Travel, MADPO improves from 41.5% to 47.4%, versus MARLHF’s 70.5% to 71.2%. On TLDR, scaling from same-size to 4× lowers MARLHF from 86.3% to 66.5% and MADPO from 92.9% to 71.8%; DeepSeek-V4-Pro achieves 70.1% and 74.6%, respectively.Source: §5.3 · pp. 9–10
  • MAGRPO outperforms CoLLM-CC and CoLLM-DC across all four domains, with minimum gains of 59.6% on TLDR score and 14.5%, 15.5%, and 33.0% on CoopHE pass, BFCL correct, and Travel pass.Actor-critic variants train less stably, which the authors attribute to critics fitted to imperfect or evolving rewards introducing approximation error.Source: Table 2 · p. 9, §5.3 · p. 10

6. Conclusion

MAPL iteratively updates agent policies by comparing their solutions with comparator-generated solutions, and its evaluations show improved decentralized and centralized collaboration approaching MARL with well-specified rewards. The appendix presents MARLHF and MADPO algorithms under Dec-Dec and explains how the variants adapt them.

6. Conclusion

  • Across writing, coding, tool use, and travel planning, MAPL improves decentralized and centralized collaboration and approaches MARL with well-specified rewards.The conclusion also reports that larger or centralized comparators do not necessarily facilitate training, citing information and realizability gaps.Source: §6 · p. 10
  • MAPL iteratively updates agent policies by comparing their solutions with those generated by a comparator policy.The conclusion evaluates MARLHF and MADPO under Dec-Dec, Cen-Cen, and Dec-Cen.Source: §6 · p. 10

A.1. MAPL Algorithms

  • Dec-Cen replaces decentralized comparators with a centralized comparator conditioned on joint history; Cen-Cen also replaces decentralized agents with a centralized coordinator.Both algorithms’ response-generation block produces joint responses, and their policy-update blocks update θ.Source: §A.1 · p. 17
  • MARLHF collects paired trajectories, trains a reward model, then updates policies using predicted returns and group-relative advantages.The algorithm collects preferences, discards ties, and retains the top-k pairs for training.Source: §A.1 · p. 17
  • MADPO uses the same preference-collection procedure but updates each agent with per-pair preference losses.Its Dec-Dec algorithm retains top-k preference pairs and applies the MADPO loss to parallel agent updates.Source: §A.1 · p. 18

A.2. Derivation of MARLHF

The appendix derives MARLHF and MADPO updates for decentralized agents and centralized coordinators, then describes choices for comparator policies and preference-pair sampling. The variants include same-class or different-class comparators and alternative replay-buffer selection rules.

A.2. Derivation of MARLHF

  • The derivation treats the log-ratio as a stop-gradient weight when obtaining the KL-gradient term.The resulting term is combined with the corrected reward advantage in the MARLHF surrogate.Source: §A.2 · p. 18
  • MARLHF’s derivation combines a corrected reward advantage with a KL weight to form its surrogate policy objective.It derives gradients separately for decentralized agents and a centralized coordinator.Source: §A.2 · pp. 18–19

A.3. Derivation of MADPO

  • MADPO derives preference-based policy gradients from differences in cumulative implicit rewards, with separate forms for decentralized agents and centralized coordinators.The derivation assumes the reference class can realize arbitrary distributions over full histories.Source: §A.3 · p. 19

B.1. Comparator Policy

  • MAPL compares the current agent policy with a comparator that may be an earlier checkpoint, a higher-temperature variant, or a policy from another class.Another policy class may differ in model architecture or information structure, as in Dec-Cen.Source: §B.1 · p. 20

B.2. Sampling Strategy

  • Replay-buffer selection can use a proxy reward when the oracle reward is inaccessible because annotator criteria are unknown.The discussion also describes retaining preferred agent trajectories or comparator trajectories meeting at least one quality criterion.Source: §B.2 · p. 20

C.1. Task and Model Configurations

This appendix specifies task and model configurations, hyperparameter sources, and oracle-reward assumptions. It also identifies the hyperparameters varied in additional ablations.

C.1. Task and Model Configurations

  • The evaluated subsets filter BFCL to meaningful cooperation and Travel to short trips, and all methods use the same filtered subsets.Short Travel trips help distinguish optimization methods with relatively small agents and exclude excessively long contexts to reduce VRAM requirements.Source: §C.1 · p. 20

C.2. Hyperparameters

  • The main experiments use the hyperparameters listed in Table 4, while ablations vary preference-pair count, decay factor λ, comparator model, and MARL algorithm.Table 4 is captioned as covering comparator, replay-buffer, and optimization hyperparameters.Source: §C.2 · p. 20, Table 4 · p. 21

C.3. Oracle Rewards

  • The study assumes annotators label preferences according to task-specific oracle rewards.Table 3 summarizes task and model configurations; unless specified, agents, comparators, and the MARLHF reward model share a configuration.Source: §C.3 · p. 20, Table 3 · p. 21

D. Additional Results

  • The additional-results section analyzes key hyperparameters because MARLHF and MADPO show different sensitivities to them.The passage introduces the analysis but does not report its findings.Source: D. Additional Results · p. 22

D.1. Ablation on Dataset Sizes

The dataset-size ablation finds that more preference data can help some tasks but does not reliably improve MAPL training. The appropriate amount depends on the domain and raw model, while more data can add overhead or dilute valuable examples.

D.1. Ablation on Dataset Sizes

  • More samples generally improve MARLHF and MADPO on TLDR and BFCL, especially MARLHF, whose complete reward representation requires substantially more data.MADPO may also benefit because additional samples can provide higher-quality data to imitate.Source: §D.1 · p. 22, Figure 5 · p. 22
  • On CoopHE, adding preference pairs does not escape a local optimum where one agent performs all tasks and the other remains idle.The authors attribute this to Qwen2.5-Coder not inherently cooperating and rarely supplying effective cooperative schemes without iterative training.Source: §D.1 · pp. 22–23
  • Travel improves with roughly 40 to 100 samples, but gains drop beyond 120 and added samples could hurt by diluting the few high-quality examples.The authors conclude that dataset size depends on both domain and raw model; more samples also increase computational overhead.Source: §D.1 · p. 23

D.2. Ablation on Replay Buffers

The replay-buffer ablation compares recent-iteration data with sampling across training stages, finding that the best data mix differs by algorithm and task.

D.2. Ablation on Replay Buffers

  • Replay buffers either use the preference dataset from k iterations earlier or sample across training iterations with decay factor λ for older data.In the first scheme k = 0 restricts sampling to the current iteration; in the second λ = 0 does so, while λ = 1 samples uniformly.Source: §D.2 · p. 23
  • Older replay data lowers MARLHF and MADPO performance on CoopHE and BFCL, but the pattern is unclear on TLDR and reverses for MADPO on Travel.The Travel reversal is attributed to MADPO collapsing mid-training, leaving earlier datasets with more high-quality samples.Source: Figure 6 · p. 23, §D.2 · p. 24
  • The best λ exceeds 0.5 for MARLHF across all four domains, while it ranges from 0 to 0.2 for MADPO in most tasks.The authors interpret this as MARLHF needing more diverse data for reward-model representation, while MADPO benefits from a higher density of high-quality samples.Source: Figure 7 · p. 23, §D.2 · p. 24

D.3. Ablation on Comparators

The comparator ablation finds that newer decentralized comparator checkpoints often help, but larger or centralized comparators do not consistently improve training.

D.3. Ablation on Comparators

  • Newer decentralized comparator checkpoints improve MAPL on TLDR, CoopHE, and BFCL, while Travel results are flat or oscillating and centralized-comparator gains are not consistent.On CoopHE and BFCL, centralized-comparator curves are flat across the first three checkpoint settings.Source: §D.3 · p. 24, Figure 8 · p. 24
  • Across TLDR, CoopHE, and BFCL, MAPL algorithms benefit from later comparator checkpoints under decentralized comparison; this trend does not consistently hold on Travel, and centralized comparisons are flat on CoopHE and BFCL.Source: §D.3 · p. 24
  • Most solid curves drop when the comparator changes from the current agent model to a larger model, whose samples may require capabilities smaller agents lack.The authors also report no consistent pattern for whether a larger model helps more as a decentralized or centralized comparator.Source: §D.3 · p. 24

D.4. Ablation on MAS Configurations

This section examines MAS configuration choices, including task-dependent effects of agent scale and optimization, decentralized roles and assignment contracts, and prompts that constrain responses to a fixed JSON schema.

D.4. Ablation on MAS Configurations

  • On TLDR, MAPL gains become marginal as model size grows; on CoopHE, larger agents degrade performance, while BFCL and Travel improve with size before and after MAPL training.The authors attribute the limited TLDR gains to smaller-model MAS already performing well, and CoopHE degradation to limited collaborative capability in the base models; with the same sample count, larger LLMs are also harder to optimize.Source: §D.4 · pp. 25–26

D.5. Results on Larger-Scale MAS

  • On BFCL, adding a third agent slightly improves raw MAS but MAPL-trained versions underperform two-agent MAS; on Travel, raw MAS does not improve, but both MAPL methods surpass two-agent MAS.The authors cite longer-context reward-model training with unchanged preference data and more complex credit assignment for the BFCL decline; Travel’s improvement is task-dependent.Source: Table 5 · p. 26, §D.5 · p. 26

E. Prompt Design

  • The Travel prompts divide decentralized agents’ work between logistics and feasibility and daily experiences, with fixed slot assignments and a strict JSON output contract.Agents cannot communicate or see each other’s outputs; a deterministic merger combines assignments and gives both agents the same reward for the resulting itinerary.Source: E. Prompt Design · pp. 27, 29
  • Agents continue inside a prefilled JSON value string, while the system closes the object after the final owned value.They are instructed not to generate the next assignment’s schema themselves.Source: E. Prompt Design · p. 29
  • A per-value token limit keeps one slot from consuming the response budget, and values must use exact catalog spelling.The reconstructed response uses the fixed schema and assignment count; semantic omissions and invalid values receive low reward.Source: E. Prompt Design · p. 29

F. Compute Resources

The paper reports deployment and training trade-offs for decentralized and centralized collaboration, relates its work to LLM and preference-based MAS research, and identifies compute and scaling limitations.

F. Compute Resources

  • Decentralized agents can be trained and deployed on separate devices, while centralized training requires substantial VRAM without sharding.The centralized policy processes all agents’ combined information and is typically larger than any individual decentralized agent; the authors suggest decentralized collaboration when tasks are modularly separable and compute is limited.Source: F. Compute Resources · p. 30, Table 6 · p. 30

G.1. LLM-based MAS

  • Decentralized LLM systems run lightweight agents from local contexts in parallel, reducing sequential dependencies and potentially improving efficiency, robustness, and security.The paper studies preference-based optimization for both decentralized and centralized systems.Source: §G.1 · p. 31

G.2. Preference-based MARL

  • Preference-based MARL work has seen limited practical use in LLM collaboration, where the policy space is extremely large and assigning agent- and turn-level credit is difficult.The cited challenges include inaccurate reward or value estimates.Source: §G.2 · p. 31

H. Broader Impacts and Future Works

  • The authors frame MAPL as an alternative to manually curated reward models, while noting compute costs, weaker MADPO performance on hard domains, sparse feedback at scale, and Dec-Cen realizability gaps.MAPL requires extra rollouts to collect informative preferences, and collecting feedback for every agent or turn is impractical.Source: H. Broader Impacts and Future Works · p. 31
Loading 2609.32827v1…