PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Decentralized LLM Collaboration with Multi-Agent Actor Critic

Shuo Liu, Tianle Chen, Ryan Amiri, Christopher Amato

arXiv:2601.21972v5cs.AIcs.DCcs.MA

TL;DR

Predefined MARL execution protocols and Monte Carlo fine-tuning leave open how to optimize decentralized LLM collaboration, while Monte Carlo returns can have high variance. The paper proposes centralized- and decentralized-critic actor-critic methods and evaluates them across writing, coding, and game-playing. Results show comparable performance in short-horizon, dense-reward writing, but CoLLM-CC performs better in long-horizon or sparse-reward tasks.

  • Problem

    Many MARL approaches rely on predefined execution protocols, and Monte Carlo fine-tuning receives end-of-episode returns with high variance, while effective optimization of decentralized LLM collaboration remains open.

    Source: §1 · p. 1

  • Method

    The paper develops Multi-Agent Actor-Critic methods for decentralized LLM collaboration, proposing CoLLM-CC with a centralized critic and CoLLM-DC with decentralized critics.

    Source: §1 · p. 2

  • Results

    Monte Carlo methods and CoLLM-DC match CoLLM-CC in dense-reward, short-horizon writing, but both underperform it in sparse-reward coding and long-horizon Minecraft tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across the evaluated writing, coding, and game-playing tasks, MA-REINFORCE and CoLLM-DC struggle in long-horizon, sparse-reward settings, while CoLLM-CC has the lowest variance and highest sample efficiency.

    Source: §7 · p. 9

    The experiments do not extend to longer-horizon tasks because MAGRPO's low sample efficiency makes fair comparisons prohibitively expensive.

    Source: §7 · p. 9

Abstract

from arXiv · show

Recent work has explored optimizing LLM collaboration through Multi-Agent Reinforcement Learning (MARL). However, most MARL fine-tuning approaches rely on predefined execution protocols, which often require centralized execution. Decentralized LLM collaboration is more appealing in practice, as agents can run inference in parallel with flexible deployments. Also, current approaches use Monte Carlo methods for fine-tuning, which suffer from high variance and thus require more samples to train effectively. Actor-critic methods are prevalent in MARL for dealing with these issues; thus, we developed Multi-Agent Actor-Critic (MAAC) methods to optimize decentralized LLM collaboration. In this paper, we analyze when and why these MAAC methods are beneficial. We propose 2 MAAC approaches, \textbf{CoLLM-CC} with a \textbf{C}entralized \textbf{C}ritic and \textbf{CoLLM-DC} with \textbf{D}ecentralized \textbf{C}ritics. Our experiments across writing, coding, and game-playing domains show that Monte Carlo methods and CoLLM-DC can achieve performance comparable to CoLLM-CC in short-horizon and dense-reward settings. However, they both underperform CoLLM-CC on long-horizon or sparse-reward tasks, where Monte Carlo methods require substantially more samples and CoLLM-DC struggles to converge.

1. Introduction

The paper targets decentralized LLM collaboration, where agents can run independently, but effective optimization remains open. It develops actor-critic methods and finds that their relative performance depends on reward density and task horizon.

1. Introduction

  • Existing MARL approaches often use predefined execution paradigms and extensive inter-agent communication, limiting broader applicability and scalability while raising potential privacy issues.Monte Carlo methods also receive return signals only at episode end; high variance limits sample efficiency in long-horizon or episodic tasks.Source: §1 · p. 1
  • Decentralized LLM collaboration offers independent agent execution, but effectively optimizing how those agents collaborate remains an open question.Separate deployment can reduce memory and storage pressure on each node and improve efficiency.Source: §1 · p. 1
  • The paper develops Multi-Agent Actor-Critic methods for optimizing decentralized LLM collaboration and introduces CoLLM-CC and CoLLM-DC.CoLLM-CC uses a centralized critic to estimate joint-history values; CoLLM-DC uses decentralized critics to estimate individual-history values.Source: §1 · pp. 1–2
  • In dense-reward, short-horizon writing, Monte Carlo methods and CoLLM-DC perform comparably to CoLLM-CC; in sparse-reward coding and long-horizon Minecraft, both underperform it.Monte Carlo methods require substantially more samples in the latter settings, while CoLLM-DC fails to converge.Source: §1 · p. 2

2. Related Works

Prior LLM collaboration often relies on rigid, communication-heavy protocols, while decentralized collaboration is less explored. Cooperative actor-critic methods can use local or centralized critics, with centralized training still allowing decentralized execution.

2.1. LLM Collaboration

  • Most role-specialized LLM collaboration frameworks enforce hand-coded protocols, incur heavy communication overhead, and typically require centralized execution; decentralized alternatives remain underexplored.Decentralized agents can run in parallel under limited or no communication, supporting efficiency and flexible deployment.Source: §2.1 · p. 2

2.2. Actor-Critic Methods in Cooperative MARL

  • Cooperative actor-critic methods pair policy actors with value-estimating critics, which may be individual or centralized over joint history.Centralized critics can use additional training-time information for more accurate and stable gradient estimates while agents execute decentrally.Source: §2.2 · p. 2

3. Background

The background models decentralized collaboration as agents responding to individual prompts in parallel and aggregating their outputs under a shared reward. It formalizes this setting as an LLM Dec-POMDP with partial observations, response actions, and dialog-turn horizons.

3.1. Decentralized LLM Collaboration

  • Agents receive individual natural-language prompts, respond in parallel, and have their responses aggregated into a solution with a shared joint reward.Across turns, feedback and dialog history provide context and experience until the task is solved or a turn limit is reached.Source: §3.1 · p. 2
  • Decentralized collaboration lets specialized agents work in parallel on subtasks, reducing long-context overhead and supporting easier local deployment of smaller models.The paper identifies long article generation, software engineering, and embodied coordination as application areas.Source: §3.1 · p. 2

3.2. LLM Dec-POMDP

  • The LLM Dec-POMDP represents agents' prompts as partial, noisy observations and their generated responses as actions, with rewards based on accessible state and joint actions.The episode horizon H is the dialog turn limit; agents maintain local observation-action histories to infer the state and optimize cumulative reward.Source: §3.2 · pp. 2–3

4. Cooperative MARL for LLM Fine-Tuning

The section introduces cooperative MARL methods for LLM fine-tuning and analyzes their benefits and limitations. Monte Carlo policy updates can be unbiased, but variance reduction through branched sampling increases inference costs; actor-critic methods instead learn value estimates, with decentralized critics facing non-stationarity.

4.1. MA-REINFORCE

  • Monte Carlo policy gradients receive returns only at dialog termination and suffer high variance, limiting scalability to long-horizon online learning.MA-REINFORCE updates each agent's individual policy using Monte Carlo estimates of joint returns.Source: §4.1 · p. 3
  • K-sampling MA-REINFORCE recursively expands a K-ary rollout tree across successor histories, increasing the number of inference calls as the horizon grows.For a full K-ary tree without early termination, the stated call count is nK(K^H−1).Source: §4.1 · pp. 3–4
  • K-sampling MA-REINFORCE's averaged gradient estimator is unbiased, and under independent rollouts its variance scales as σ^2/K^(H−t).The variance reduction assumes independent gradient estimates and no early episode termination; expanding the rollout tree rapidly increases LLM inference calls.Source: §4.1 · pp. 3–4

4.2. Multi-Agent Actor-Critic

  • Actor-critic methods learn a policy actor and a value critic to reduce gradient-estimate variance and improve sample efficiency.The section contrasts decentralized critics conditioned on local histories with a centralized critic conditioned on joint history; the critic is removed at execution, so either approach can execute decentrally.Source: §4.2 · p. 4
  • Decentralized critics can face harder convergence conditions because other agents' changing policies induce non-stationarity in each local critic's learning problem.The unbiased-gradient statement for decentralized and centralized critics assumes critic convergence.Source: §4.2 · p. 5

5. CoLLM-CC

CoLLM-CC uses a centralized critic while keeping agents’ KV caches independent and private for decentralized inference. It treats responses as sequence-level actions and trains actors and the critic from replayed joint transitions.

5. CoLLM-CC

  • CoLLM-CC is presented as a representative MAAC approach with key design features; its decentralized-critic counterpart, CoLLM-DC, is presented in Appendix B.Source: §5 · p. 5

5.1. History Representation

  • Unlike lossy RNN summaries, CoLLM-CC uses Transformer KV caches for recent dialog history, keeps caches private per agent, and can add global information to critic value estimates.Source: §5.1 · p. 5

5.2. Sequences as Actions

  • Because response-level rewards make token actions uninformative and the response space is combinatorially large, teacher-forced passes sum token log-probabilities to obtain sequence probabilities in one pass.The paper applies teacher-forced passes during both rollout and policy updates.Source: §5.2 · p. 5

5.3. Algorithm

  • Each rollout records joint responses, rewards, evolving prompts, global information, and behavior policies; minibatches then update actor policies and the centralized critic.Source: §5.3 · p. 6

6. Experiments

The experiments compare CoLLM-DC and CoLLM-CC with single-model, prompt-based multi-agent, and MARL baselines across writing, coding, and language-based games. CoLLM-CC is strongest in the reported coding and Minecraft comparisons, while results vary by task and method.

6.1. Setup

  • Writing divides long-document work by roles; coding uses decomposable programs over two turns; Minecraft uses four-turn construction, with HouseBuild adding spider defense.Source: §6.1 · pp. 6–7

6.3. Evaluation Results

  • Across tasks, CoLLM-CC matches MAGRPO in writing and outperforms MAGRPO and CoLLM-DC in coding and Minecraft; CoLLM-DC is unstable and MAGRPO needs more samples.Table 1 covers response time, cost, and performance across the evaluated tasks.Source: Table 1 · p. 7, §6.3 · pp. 8–9

6.2. Baselines

  • Baselines include larger single models, prompt-only parallel, sequential, and discussion agents, and MAGRPO; MARL uses matched learning rates and K=4 or K=2 by task.Source: §6.2 · p. 8

6.3. Evaluation Results

  • At n=3, CoLLM-CC has the strongest overall results: Adj ↓0.4%, HP ↑7.8%, and IoU ↑6.9%/↑8.6% on StrBuild/HouseBuild.The n=3 deltas are relative to the n=2 control setting.Source: Table 2 · p. 9, §6.3 · p. 9

7. Conclusion

CoLLM-CC has the best performance, lowest variance, and highest sample efficiency, while MA-REINFORCE and CoLLM-DC struggle in long-horizon, sparse-reward settings. The appendix states the scope of the work’s limitations and proves that the K-sampling gradient estimate is unbiased.

7. Conclusion

  • CoLLM-CC achieves the best performance with the lowest variance and highest sample efficiency, while MA-REINFORCE and CoLLM-DC struggle in long-horizon, sparse-reward settings.The reported challenges are low sample efficiency for MA-REINFORCE and convergence issues for CoLLM-DC.Source: §7 · p. 9
  • The methods are not evaluated on longer-horizon tasks, and the work focuses on concurrent execution without inter-agent communication.The authors identify extending to more complex agentic domains and relaxing communication or coordination constraints as future work.Source: §7 · p. 9

A. Proofs

  • The appendix provides full proofs for the paper’s propositions.Source: A. Proofs · p. 14

A.1. Proof of Proposition 4.1

  • The K-sampling MA-REINFORCE estimate is unbiased for the true gradient term at each history.The proof relates its expected estimate to the K=1 MA-REINFORCE estimate.Source: §A.1 · p. 14

A.2. Proof of Proposition 4.2

Under independent gradient estimates and episodes that are not terminated early, the variance of the K-sampling estimate scales inversely with K. CoLLM-DC trains each agent’s actor and critic independently using a critic conditioned on local history; parameter sharing may introduce gradient interference.

A.2. Proof of Proposition 4.2

  • Under independent gradient estimates and no early episode termination, the variance of the K-sampling gradient estimate scales inversely with K.The independent-gradient assumption may be hard to satisfy in Dec-POMDPs because histories with the same prefixes usually have positively correlated values.Source: §A.2 · p. 14

B. CoLLM-DC

  • CoLLM-DC conditions each decentralized critic on its agent’s local history and updates each agent’s critic and actor independently from minibatches of joint transitions.The rollout and actor updates match CoLLM-CC; the critic-training phase differs.Source: B. CoLLM-DC · p. 15
  • CoLLM-DC agents may share a model because they learn the same latent representation, but their different optimization objectives may cause gradient interference.Source: B. CoLLM-DC · p. 15

C. Experimental Settings

On CoopHE, MARL methods can match or outperform a larger single model, and CoLLM-CC leads most reported pass@k results; CoLLM-DC underperforms due to non-convergence. The section also introduces the experimental settings, datasets, model architectures, and hyperparameters.

C. Experimental Settings

D.1. Pass@k Results on CoopHE

  • GRPO and actor-critic fine-tuning yield marginal gains over the raw model, primarily by increasing the likelihood of correct solutions already within its representational scope.Source: §D.1 · p. 17
  • Prompt-based multi-agent approaches may underperform single-model baselines without timely communication; sequential turn-taking reduces inference speed.Source: §D.1 · p. 17
  • MARL methods can match or outperform a larger single model, while CoLLM-CC leads most pass@k results and CoLLM-DC trails CoLLM-CC and MAGRPO due to non-convergence.The table reports pass@k results for k = 1, 3, 5, and 10, averaged over five runs.Source: §D.1 · p. 17, Table 3 · p. 17
  • CoLLM-CC guides the primary agent to infer the auxiliary agent’s functionality and provide fallback solutions, helping complete implementation components in parallel.Source: §D.1 · p. 17

D.2. Training Overhead

This section covers CoLLM-DC and CoLLM-CC training overhead, including time and GPU memory relative to MAGRPO, sample use, and scaling with agent count. It also describes domain-specific reward designs for writing, coding, and Minecraft.

D.2. Training Overhead

  • CoLLM-DC and CoLLM-CC require substantially more time and GPU memory than MAGRPO, but CoLLM-CC ultimately uses the fewest samples.CoLLM-CC converges faster, and many samples trigger early termination during early training; the overhead does not scale linearly with the number of LLMs.Source: §D.2 · pp. 17–18, Table 4 · p. 17
  • Critic LLMs account for the added time and memory, while lower-variance gradients reduce retained back-propagation samples and a single centralized critic lowers update cost versus independent critics.The centralized critic's update cost is lower than maintaining n independent critics of the same size in CoLLM-DC.Source: §D.2 · p. 17
  • As the number of agents grows, MAGRPO training time rises rapidly and CoLLM-DC faces similar scaling pressure; parameter sharing mitigates CoLLM-DC‡ overhead but decentralized training remains susceptible to non-stationarity.CoLLM-CC provides more stable training without additional inference-time overhead under decentralized execution, although its training cost still increases with MAS size.Source: §D.2 · p. 18, Table 5 · p. 18

F.1. Writing Collaboration

  • Writing rewards compare two paragraphs on structural length and vocabulary coverage, stylistic similarity, and transition-word diversity.Structural wellness uses optimal length and unique-word ratios, while style is measured by capped Jaccard similarity and coherence by diversity across 12 transition-word categories.Source: §F.1 · p. 20

F.2. Coding Collaboration

  • Coding rewards structural integrity, syntax, test success, and substantive cooperation between auxiliary and main functions.Missing main functions, syntax errors, or zero passing tests terminate evaluation; superficial auxiliary calls are penalized.Source: §F.2 · p. 20

F.3. Minecraft

  • Minecraft rewards construction coverage while penalizing excess resources, identical-texture adjacency, HouseBuild redundancy, and spider damage.StrBuild uses coverage and adjacency measures; HouseBuild also rewards coverage and penalizes redundancy and damage inflicted by spiders.Source: §F.3 · pp. 20–21

G. Compute Resources

The appendix lists compute resources used for training and evaluation, including systems for collaboration experiments and inference.

• Coding Collaboration

  • The listed configurations pair Intel Xeon or AMD EPYC CPUs with NVIDIA Hopper H200 or Blackwell B200 GPUs across GPU cluster and GPU cloud-instance setups.The CPU models include Intel Xeon Platinum 8558, Gold 5318Y, Platinum 8592+, Platinum 8568Y+, and AMD EPYC 9755.Source: • Coding Collaboration · p. 21

G.2. Inference Device

  • The inference-device subsection identifies a standalone workstation.Source: §G.2 · p. 21

H. Code and Dataset

The appendix identifies an open-source MARL library, experiment repositories, and the CoopHumanEval dataset.

Related papers

Loading 2601.21972v5…