PaperlayerView exact revision on arXiv

Source-linked AI summary

Belief-Conditioned Multi-Agent Actor Critic for Decentralized LLM Collaboration

Tianle Chen, Shuo Liu, Christopher Amato

arXiv:2610.35398v1cs.AIcs.LGcs.MA

TL;DR

Decentralized LLM collaboration is difficult when agents must infer useful information across partially observed, multi-turn interactions. BCMAAC conditions actors on belief context and uses centralized value estimation during training; it consistently outperforms MAAC across Minecraft, Web, and tool-based tasks.

  • Problem

    Partial observability, natural-language actions, and delayed sparse rewards make it difficult to infer task-relevant state and determine which interactions contributed to success.

    Source: §1 · pp. 1–2

  • Method

    BCMAAC uses a recurrent belief encoder to produce latent context from structured side information, conditioning actors and a centralized training critic on belief context.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    BCMAAC consistently outperforms MAAC across Minecraft, Web, and tool-based collaboration tasks; Stateful Tool improves from 59.2% to 77.5% (+18.3).

    Source: §5.2 · p. 8, §6 · p. 9

  • Takeaways & Limitations

    Explicit belief context is particularly useful in long-horizon, partially observable collaboration, where actions depend on persistent hidden state and delayed or sparse feedback.

    Source: §1 · p. 2

    All six evaluated tasks involve two agents, paired into complementary roles suited to the task designs.

    Source: §6 · p. 9

Abstract

from arXiv · show

Recent work has explored optimizing LLM collaboration through Multi-Agent Reinforcement Learning (MARL), but many approaches still rely on predefined workflows, limiting efficiency and scalability. Decentralized LLM collaboration is challenging because agents execute independently with limited communication, share a joint objective, and must infer useful information from communication traces. In this paper, we propose BCMAAC, a Belief-Conditioned Multi-Agent Actor-Critic method that maintains beliefs over contextual information and uses them to facilitate training. Experiments on Minecraft game-playing, web searching and tool-using tasks show that BCMAAC improves task performance and collaborative behavior compared to naive actor-critic baselines. The improvements are particularly pronounced in multi-turn tasks with richer contextual information, where belief modeling provides a clear advantage.

1 Introduction

Decentralized LLM collaboration makes it difficult to infer task state and other agents’ intentions from long, sparse-feedback interactions. BCMAAC uses belief context to support collaboration, with reported gains over MAAC and benefits from auxiliary supervision.

1 Introduction

  • Partial observability and long-horizon sparse rewards make decisions and learning difficult because rewards are delayed and it is hard to identify which interactions contributed to success.Source: §1 · pp. 1–2
  • BCMAAC encodes structured side information as latent belief context for each actor and uses joint belief context for centralized value estimation during training.Source: §1 · p. 2
  • BCMAAC outperforms MAAC on collaboration tasks, while ablations show that task-belief and partner-action auxiliary signals both help improve performance.Source: §1 · p. 2

2 Related Works

Prior work spans prompt-defined LLM collaboration, reinforcement-learning approaches, and belief or partner modeling in MARL. The paper contrasts these with the challenges of learning collaboration from long textual histories.

LLM-based multi-agent collaboration

RL for LLM collaboration

Partner modeling and belief learning in MARL

3 Background

The background describes decentralized agents acting from local contexts, formalizes their interaction as a cooperative Dec-POMDP, and introduces MAAC as a centralized-training, decentralized-execution baseline.

3.1 Decentralized LLM Collaboration

  • Agents act from local observations, histories, messages, and tools; their responses form joint actions that update the environment, while decentralized execution supports parallel reasoning and reduces individual context pressure.Source: §3.1 · p. 3

3.2 LLM Dec-POMDP Formulation

  • The Dec-POMDP formulation models agents’ local observations and response-sequence actions, environment transitions, shared rewards, and discounted episode returns under decentralized policies.Source: §3.2 · p. 3

3.3 MAAC

  • MAAC uses local-history actors and a centralized critic during training, but raw histories make task-state extraction and explicit modeling of other agents’ knowledge and future actions difficult.Source: §3.3 · p. 3

4 Belief-Conditioned MAAC

BCMAAC encodes execution-time side information into recurrent belief contexts that condition decentralized actors and are aggregated by a centralized training critic. Its theoretical properties and algorithm specify how these contexts support value estimation, auxiliary supervision, and decentralized execution.

4.1 BCMAAC Formulation

  • A recurrent encoder maps each agent’s execution-time side information and recurrent state to a belief context, updated state, task-belief prediction, and partner-action prediction.The belief context conditions the actor and critic; auxiliary predictions use per-turn targets generated from completed trajectories and environment state during training.Source: §4.1 · p. 4

4.2 Theoretical Properties

  • A pre-action baseline β(c_t) does not change the expected policy-gradient estimate when c_t is fixed before the joint action is sampled.The proof uses factorization of the joint policy and shows the baseline term contributes zero in expectation.Source: §4.2 · p. 4
  • When the joint belief context is sufficient for future return, value conditioned on all available information equals value conditioned on that context.This is an ideal-case condition, not a claim that every learned context is sufficient.Source: §4.2 · p. 5
  • Auxiliary per-turn labels give the belief encoder an additional representation-learning signal for state estimation and agent modeling, including when rewards are sparse.Labels are generated from trajectories; the auxiliary losses supervise task-belief and partner-action predictions.Source: §4.2 · p. 5, §4.3 · p. 5

4.3 BCMAAC Algorithm

  • Training collects trajectories, generates auxiliary labels, recomputes contexts, and updates the critic, actors, and encoder with value, policy, and auxiliary losses.The critic uses TD regression, actors use advantage-weighted response log-probabilities, and the encoder receives the combined losses.Source: §4.3 · pp. 5–6
  • At execution, each agent recursively computes its context from local information and samples its action without the centralized critic or auxiliary heads.The resulting joint policy remains decentralized.Source: §4.3 · p. 5

5 Experiments

Across Minecraft, web, and tool tasks, BCMAAC leads the six-task evaluation, with its largest reported gain on Stateful Tool. Its analysis links structured belief context to faster, higher, and more stable learning, while noting that its advantage may be limited in nearly fully observable tasks or when short histories suffice.

5.1 Setup

  • The setup evaluates long-horizon collaboration across Minecraft, web evidence gathering, and tool tasks that divide information access from execution.Minecraft agents face partial views and complementary capabilities; web agents must combine distributed evidence, while tool agents must communicate findings before restricted actions.Source: §5.1 · pp. 6–8

5.2 Results

  • BCMAAC achieves the best performance on all six tasks; trained multi-agent methods excel where tasks decompose, while prompt-only collaboration does not consistently improve performance.The overall evaluation spans Minecraft, Web, and Tool tasks; the comparison also indicates that adding agents alone is insufficient.Source: Table 1 · p. 7, §5.2 · p. 8
  • On Filesystem Wrangling, single-agent AC scores 66.9% versus decentralized MAAC’s 58.9%, but BCMAAC reaches 73.0% by tracking Editor readiness.The Editor’s dependence on paths found and communicated in a previous turn makes this task more sequential than parallel.Source: §5.2 · p. 8
  • Against MAAC, BCMAAC’s largest gain is Stateful Tool (59.2%→77.5%, +18.3), followed by Resource Gathering (78.4%→92.8%) and Filesystem Wrangling (58.9%→73.0%).Web-task gains are Cross-Lingual Gathering (45.1%→53.2%) and Multi-Source Gathering (47.6%→58.5%).Source: §5.2 · pp. 8–9
  • The full BCMAAC performs best overall: partner-action prediction helps most on Bridge Building, while task-belief supervision helps more on resource, web-evidence, and tool tasks.Each single-auxiliary variant performs between MAAC and full BCMAAC on all tasks, supporting the complementarity of the two signals.Source: §5.2 · p. 9

5.3 Analysis

  • Across most tasks, BCMAAC converges faster, earns higher rewards, and is more stable than MAAC; gains are smaller on web tasks and Cross-Lingual Gathering remains noisy.It overtakes MAAC in Filesystem Wrangling, improves smoothly in Resource Gathering, and reaches higher final rewards in Bridge Building and Stateful Tool.Source: §5.3 · p. 9
  • The results suggest that explicit latent collaboration state helps beyond reconstructing it from raw histories, while auxiliary objectives provide dense turn-level supervision during low-feedback communication and exploration.The centralized critic estimates returns from structured joint beliefs.Source: §5.3 · p. 9
  • BCMAAC’s advantage may be limited in nearly fully observable tasks or when short histories suffice; its benefit depends on whether reduced estimation error outweighs information lost through compression.The paper finds this trade-off favorable for the partially observable, long-horizon tasks studied.Source: §5.3 · p. 9

6 Conclusion

BCMAAC maintains structured belief context for decentralized decisions and uses joint belief context for centralized value estimation during training. It outperforms MAAC across Minecraft, Web, and tool-based tasks, with gains strongest when collaboration requires tracking latent state and coordinating delayed actions.

6 Conclusion

  • BCMAAC consistently outperforms MAAC and improves training stability across Minecraft, Web, and tool-based collaboration tasks.Its gains are strongest when agents track latent state, integrate other agents’ information, coordinate delayed actions, and combine evidence from multiple sources.Source: §6 · p. 9
  • The study evaluates six tasks, all involving two agents paired into complementary roles suited to the task designs.For example, Filesystem Wrangling pairs a read-only explorer with a write-capable editor.Source: §6 · p. 9

A.1 Tasks and Datasets

The evaluation covers six decentralized, two-agent tasks across Minecraft, Web, and tool environments. The tasks use structured side information to form belief targets and teammate-action predictions, with auxiliary labels used only in training.

A.1 Tasks and Datasets

  • The task suite spans Minecraft, Web, and deterministic stateful tool environments, with task-native success or answer-match metrics.Table 2 presents dataset splits and primary metrics; the supplied passage does not report its cell values.Source: §A.1 · p. 12, Table 2 · p. 12
  • Resource Gathering requires agents with asymmetric harvesting tools to collect resources, while Bridge Building requires discovering true supports and connecting anchor regions.Agents can communicate observations or probing results to coordinate their actions.Source: §A.1 · p. 12
  • Cross-Lingual Gathering splits English and Chinese retrieval across agents, while Multi-Source Gathering requires combining passage and table evidence.The former uses a Chinese MKQA subset; the latter uses HybridQA examples.Source: §A.1 · p. 12
  • Stateful Tool separates fact retrieval from task execution, and Filesystem Wrangling separates read-only exploration from write operations.Both require information to be handed off across turns before the executor can act.Source: §A.1 · pp. 12–13

A.2 Structured Context and Auxiliary Targets

  • BCMAAC receives compact structured side information built only from each agent’s execution-time observations, without global answers or hidden training labels.Belief targets identify relevant resource cells, true bridge supports, evidence pieces, or required tool-task progress slots.Source: §A.2 · p. 13
  • A four-class teammate action-type prediction signal is used across tasks, and auxiliary labels are training-only; execution retains the actor and recurrent belief encoder.Bridge Building and Resource Gathering use task-specific action categories.Source: §A.2 · p. 13

A.3 Training Details

Training uses a shared base model and matched environment, reward, rollout, evaluation, actor, and critic settings for MAAC and BCMAAC, with a separate context learning rate for BCMAAC.

A.3 Training Details

  • Agents use Qwen/Qwen3-4B-Instruct-2507, train in bfloat16, and sample actions with temperature 0.5 and top-p = 0.6.Source: §A.3 · p. 13
  • MAAC and BCMAAC share environments, reward definitions, rollout budgets, evaluation protocols, and matched actor and critic learning rates.BCMAAC additionally trains its belief encoder with a separate context learning rate.Source: §A.3 · p. 13
  • The belief encoder uses a hidden dimension of 128, while the structured value head uses 256; BCMAAC additionally trains the belief encoder with a separate context learning rate.Table 3 lists the main training hyperparameters.Source: §A.3 · p. 13

A.4 Reward Designs

The reward designs use deterministic environment rules and dense shaping to expose intermediate progress, while evaluation retains task-native success or answer-match scores. Prompt skeletons specify structured JSON actions for task setup, communication, and tools.

A.4 Reward Designs

  • Evaluation uses task-native success or answer-match scores, while dense shaping exposes intermediate progress.Rewards are computed by deterministic environment rules after parsing agents’ JSON actions.Source: §A.4 · p. 14
  • Resource Gathering rewards collection progress, goal completion, useful communication, movement toward zones, and penalizes wasted extraction attempts.The reward also includes a first-zone-entry indicator.Source: §A.4 · p. 14
  • Bridge Building rewards reducing the connection gap, merging connected components, connecting true supports, and terminal connectivity, with a late-turn quality term for cleaner, compact bridges.The design also includes a penalty signal for newly unsafe adjacencies to fake supports.Source: §A.4 · p. 14
  • Web Gathering rewards answer match, support coverage, useful and cross-source communication, informed search, support-chain completion, and answer readiness.Substring matches receive partial credit; wrong, premature, conflicting, or unsupported-answer penalties are set to zero in reported runs.Source: §A.4 · p. 14
  • Stateful Tool and Filesystem Wrangling reward progress, success, useful communication, and valid calls; filesystem also penalizes errors, premature answers, and terminal failure.Its largest penalty applies only when prerequisites are satisfied but the irreversible action was never executed.Source: §A.4 · p. 14

A.5 Prompt Details

  • Prompt skeletons provide task setup, local state or evidence, teammate messages, and structured JSON fields for communication and actions or tools.Web prompts include search and answer fields; tool prompts include role, allowed tools, known facts, and tool arguments.Source: §A.5 · pp. 14–15

A.6 Evaluation Protocol and Metrics

The appendix defines task-native evaluation and controlled baseline comparisons, then describes execution-time structured inputs and auxiliary supervision for BCMAAC.

A.6 Evaluation Protocol and Metrics

  • Minecraft and Tool tasks use binary success metrics, while Web tasks use match scores; results average late-training checkpoints across five runs.Bridge Building requires connectivity between anchor regions through traversable cells, and Resource Gathering requires meeting all resource targets.Source: §A.6 · pp. 15–16

A.7 Baseline Details

  • All methods share task instances, transitions, rewards, parsing rules, and task-native metrics; ablations change only auxiliary supervision.MAAC and BCMAAC also share rollout budgets, base LLM initialization, and actor/critic optimization settings.Source: §A.7 · p. 16

A.8 Structured Side Information

  • Execution-time side information is limited to information available to each agent, and Table 5 summarizes task-specific tensor dimensions and belief targets.The inputs are constructed from locally available information, including received communication.Source: §A.8 · pp. 16, 18, Table 5 · p. 17

A.9 Auxiliary Label Generation

  • After each rollout, Γξ labels trajectories using environment state; masked binary cross-entropy supervises unresolved or required task-belief entries.Answer readiness is grouped with task belief, while action preference is grouped with partner-action supervision.Source: §A.9 · p. 18

B Proofs

The proof of Proposition 4.2 uses the fact that conditioning on F_t determines c_t.

B Proofs

  • Because c_t is determined by information in F_t, conditioning on F_t also determines c_t.This is the stated step in the proof of Proposition 4.2.Source: B Proofs · p. 18
Loading 2610.35398v1…