PaperlayerView exact revision on arXiv

Source-linked AI summary

MASBench: Benchmarking LLM-based Multi-Agent Collaboration under Partial Observability

Qizhi Chu, Zekai Yu, Sijie Wen, Yang Liu, Chen Qian, Cheng Yang, Chuan Shi, Zhiyuan Liu

arXiv:2610.04672v1cs.AI

TL;DR

Many multi-agent benchmarks assume global observability and offer limited support for systematic evaluation of collaboration mechanisms. MASBench introduces a partially observable benchmark with progressive tasks and deterministic metrics. Experiments report backbone and mechanism comparisons, including consistent Qwen-Plus Score leadership across the three task categories.

  • Problem

    Many existing multi-agent benchmarks assume global observability and provide limited support for mechanism-level analysis, while some partially observable benchmarks emphasize end-to-end performance.

    Source: §1 · p. 2

  • Method

    MASBench organizes partially observable collaboration into progressive Reasoning, Scheduling, and Game tasks and evaluates performance, communication cost, and communication efficiency deterministically.

    Source: §1 · p. 2, §4.3 · p. 6

  • Results

    Across all three task categories, Qwen-Plus achieves the highest Score; in Game, GATE outperforms RANDOM by 2.35 to 6.57 Score points and reduces Tok by 28.0% to 35.6% across backbones.

    Source: §5.2 · pp. 7–8

  • Takeaways & Limitations

    Experiments across diverse LLM backbones and collaboration mechanisms provide empirical guidance for designing more efficient and effective multi-agent systems.

    Source: §1 · p. 2, §6 · p. 10

    Agents have distinct observation functions that provide only a local view of the global state, alongside messages sent to them.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Large language models (LLMs) have progressively evolved into the core of autonomous agents. Building on this progress, LLM-based multi-agent systems (MAS) coordinate multiple agents into a synergistic team to accomplish complex tasks that exceed the capabilities of individual agents. The effectiveness of such systems depends not only on the agents themselves, but also on how collaboration mechanisms are designed and organized. Note that real-world collaboration is typically partially observable, where each agent can only access partial information about the environment due to physical or privacy-related constraints. However, many existing multi-agent benchmarks assume global observability, and leave limited support for systematically evaluating collaboration mechanisms. To bridge this gap, we introduce MASBench, a multi-agent collaboration benchmark designed under partially observable constraints. It is organized into three progressive task categories: Reasoning, Scheduling, and Game. Through this structure, we progressively evaluate three representative collaboration mechanisms: Protocol, Memory, and Routing. MASBench further provides deterministic evaluation metrics, including performance score, communication cost, and cost effectiveness, to characterize both collaboration outcomes and communication overhead. Experiments across diverse LLM backbones and mechanism configurations offer empirical guidance for effective MAS design. Code is available at: https://github.com/BUPT-GAMMA/MASBench

1 INTRODUCTION

MASBench addresses the limited support for evaluating collaboration mechanisms in realistic partially observable settings. It organizes three task categories around different mechanisms and provides deterministic collaboration metrics.

1 INTRODUCTION

  • MASBench benchmarks multi-agent collaboration under partial observability, where agents lack direct access to the global state.Existing benchmarks often assume global observability, while recent partially observable benchmarks tend to emphasize end-to-end performance rather than mechanism-level analysis.Source: §1 · p. 2
  • Reasoning distributes evidence between two agents; Scheduling requires negotiation over private calendars; Game requires resource coordination under local observation.These tasks focus respectively on Protocol; Memory and Protocol; and Memory, Routing, and Protocol.Source: §1 · p. 2
  • The task progression targets specific collaboration mechanisms through scenario design, rather than merely stacking task difficulty.The introduction identifies memory, routing, and protocol as representative mechanisms governing collaboration.Source: §1 · pp. 1–2
  • MASBench reports performance score, communication cost, and communication efficiency without relying on LLM-as-a-judge.The authors state that these metrics quantify collaboration and aim to ensure reproducibility and fairness.Source: §1 · p. 2
  • Experiments across diverse LLM backbones and collaboration mechanisms provide empirical guidance for designing more efficient and effective multi-agent systems.The authors frame collaboration effectiveness as depending on the design and organization of mechanisms, not simply on increasing agent numbers.Source: §1 · pp. 1–2

2 RELATED WORK

Prior work studies LLM-based collaboration mechanisms and increasingly uses collaborative or partially observable benchmarks. MASBench instead focuses on controlled mechanism-level evaluation with deterministic metrics.

2.1 LLM-BASED MAS

  • Memory maintains state across turns, routing shapes who communicates with whom, and protocols structure interactions through stages or message schemas.Prior examples include memory streams in Generative Agents, context-sensitive interaction graphs in DyLAN, and communication optimization in OPTIMA.Source: §2.1 · p. 3

2.2 BENCHMARKS FOR LLM-BASED MAS

  • Existing benchmarks span general-purpose agent tasks, collaborative problem solving, simulations, and newer partially observable settings.The passage cites GAIA and AgentBench as general-purpose examples and describes an expansion toward collaborative tasks and simulation.Source: §2.2 · p. 3
  • Existing benchmarks primarily evaluate end-to-end capability rather than controlled contributions of collaboration mechanisms under partial observability.Many also rely on LLM-as-a-judge, which the passage describes as prompt-sensitive and vulnerable to systematic biases that undermine reproducibility.Source: §2.2 · p. 3
  • MASBench progressively stresses protocol, memory, and routing while reporting deterministic task-grounded performance, communication cost, and communication efficiency.Domain-specific benchmarks are described as complementary, often prioritizing individual-agent adaptation to particular environments.Source: §2.2 · p. 3

3 PRELIMINARY

The paper models collaboration as LLM-based agents acting in a shared environment through repeated observation, memory, decision, and communication processes. It distinguishes memory, routing, and protocol as mechanisms shaping that collaboration.

3.1 PROBLEM FORMULATION

  • A multi-agent system is formalized as an overall goal, a set of LLM-based agents, and a shared environment.Each agent is further represented through observation, memory, memory management, decision, and communication components.Source: §3.1 · p. 3
  • Under partial observability, each agent perceives only a local view of the global state and receives messages sent to it.The system-level output is usually derived from the last global state to match the overall goal.Source: §3.1 · pp. 3–4

3.2 KEY COLLABORATION MECHANISMS

  • Memory records and summarizes past observations and interactions to maintain historical information for an agent's decisions.It is framed as supporting consistency in long-term collaboration.Source: §3.2 · p. 4
  • Routing determines who communicates with whom and when, while protocol specifies message schemas and interaction rules.The paper says routing and protocol primarily regulate communication, while task-specific rules may also constrain or inform decisions.Source: §3.2 · p. 4

4 MASBENCH

MASBench uses three tasks to progressively increase collaboration-mechanism demands under partial observability. Its design distinguishes task-specific mechanisms and evaluates performance alongside communication cost and efficiency.

4.1 BENCHMARK OVERVIEW

  • MASBench uses Reasoning, Scheduling, and Game to progressively increase collaboration-mechanism demands under partial observability.The framework evaluates how agents use memory, protocols, and routing to bridge information gaps.Source: §4.1 · p. 4

4.2 BENCHMARK DESIGN

  • Reasoning splits evidence from multi-hop HotpotQA instances between two agents, so neither can solve each problem independently.Agents exchange clues in multi-turn dialogue; Simple and Complex subsets vary by supporting-fact count, context length, and answer length.Source: §4.2 · p. 5
  • Scheduling requires agents to negotiate a mutually feasible meeting time using private calendars and longer, multi-round interactions.Broadcast-style chat excludes routing; calendars use 10-minute granularity, and difficulty depends on the number of feasible continuous 20-minute windows.Source: §4.2 · p. 5
  • Game is a cooperative grid-world task requiring exploration, resource collection, and delivery under local observation and limited rounds.Point-to-point messaging and local windows make memory, protocol, and routing relevant; map scale and resource demands distinguish difficulty levels.Source: §4.2 · p. 5

4.3 EVALUATION METRICS

  • Performance Score averages normalized task scores on a 0–100 scale, with task-specific scoring for reasoning, scheduling, and game.The metrics also include inter-agent message tokens as communication cost, excluding local reasoning tokens, and performance score per 1,000 communication tokens as communication efficiency.Source: §4.3 · pp. 5–6
  • Tables 2 and 3 report Score, Tok, and Eff for Reasoning and Scheduling, respectively, with best results marked per backbone.Table 3 also marks second-best results per backbone.Source: Table 2 · p. 6, Table 3 · p. 6

5 EXPERIMENTS

Experiments compare LLM backbones and task-relevant collaboration mechanisms across six settings, with further analyses tracking performance and communication over rounds. Results show backbone and mechanism differences, including trade-offs between task score and communication efficiency.

5.1 EXPERIMENTAL SETTINGS

  • The evaluation uses five LLM backbones and task-specific subsets of MEMORY, PROTOCOL, and ROUTING; Llama-3.1-8B is included only for Reasoning because of high failure rates in Scheduling and Game.The benchmark compares task-relevant mechanisms and reports performance and communication metrics.Source: §5.1 · pp. 6–7

5.2 MAIN RESULTS

  • In Game, GATE improves Score by 2.35–6.57 points and reduces Tok by 28.0%–35.6% versus RANDOM, while MEMORY+GATE+PROTOCOL yields 1.12×–1.91× higher Eff than routing-only variants.MEMORY+RANDOM+PROTOCOL still underperforms MEMORY+GATE in Score, highlighting routing quality.Source: Table 4 · p. 7, §5.2 · p. 8
  • Qwen-Plus achieves the highest Score across all three task categories, while it is frequently surpassed by other backbones on Eff.Higher utility may come with disproportionate communication overhead.Source: §5.2 · p. 7
  • Across backbones, task-relevant mechanisms generally improve collaboration effectiveness; in Reasoning, PROTOCOL raises Score by 1.5%–28.4%, reduces Tok by 45.0%–72.7%, and yields a 2.0×–4.2× gain in Eff.In Scheduling, MEMORY improves Score over BASE, and PROTOCOL further improves success rate and efficiency.Source: §5.2 · p. 7

5.3 FRAMEWORK-LEVEL EVALUATION

  • AgentVerse generally communicates more compactly and performs better in most Scheduling settings, whereas CAMEL achieves stronger performance in most Game settings.Performance gaps between the frameworks are generally modest in Reasoning but larger in Scheduling and several Game settings.Source: Table 5 · p. 8, §5.3 · p. 8

5.4 FURTHER ANALYSIS ON REASONING CATEGORY

  • On Reasoning, PROTOCOL reaches a higher score plateau than BASE, especially on the Complex split, and produces shorter, more stable messages across backbones.The analysis attributes reduced redundancy to structured clue requests and evidence sharing.Source: Figure 3 · p. 8, §5.4 · p. 9

5.5 FURTHER ANALYSIS ON SCHEDULING CATEGORY

  • On Scheduling, MEMORY+PROTOCOL improves scores fastest early and reaches the highest plateau on both splits, while constraining negotiation reduces message length and variance.MEMORY alone does not consistently shorten messages; it supports long-horizon constraint consistency.Source: Figure 4 · p. 9, §5.5 · p. 9

5.6 FURTHER ANALYSIS ON GAME CATEGORY

  • On Game, memory-enabled variants outperform BASE, and MEMORY+GATE+PROTOCOL achieves the best performance; adding PROTOCOL shortens messages and reduces their variance under both routing modes.Routing-only communication can still involve relatively long and variable messages.Source: Figure 5 · p. 9, §5.6 · p. 9

6 CONCLUSION

The conclusion presents MASBench as a benchmark for collaboration under partial observability, with experiments intended to inform multi-agent system design. The appendices describe related framework categories and a sampled failure-analysis procedure.

6 CONCLUSION

  • MASBench evaluates multi-agent collaboration under partially observable constraints through three progressive task categories and experiments across diverse LLM backbones.The authors present the benchmark as a source of empirical guidance for designing more efficient and effective multi-agent systems.Source: §6 · p. 10

A EXTENDED RELATED WORK ON LLM-BASED MULTI-AGENT SYSTEMS

  • Existing multi-agent systems are grouped into general orchestration frameworks with reusable abstractions and domain-specific frameworks tailored to particular workflows.Examples include AgentVerse for general-purpose task solving and MetaGPT and ChatDev for phased software-development tasks.Source: A EXTENDED RELATED WORK ON LLM-BASED… · p. 14

B.1 ERROR ANALYSIS

  • The error analysis characterizes recurring collaboration problems using sampled failures and qualitative trajectory inspection, not benchmark-wide failure rates.Each selected trajectory receives one primary task-specific category based on its dominant observed failure mode.Source: §B.1 · p. 14, §B.1.1 · p. 14

B.1.1 SAMPLING AND ANNOTATION

  • For each task, the analysis pools the bottom 30% of runs within each backbone, mechanism setting, and difficulty subset, then randomly samples 40 cases for manual review.Runs are ranked separately within Simple and Complex subsets.Source: §B.1.1 · p. 14

B.1.2 ERROR DISTRIBUTIONS

The error analysis reports the distribution of primary categories among 40 sampled failure cases per task, rather than failure rates across the full benchmark.

Reasoning

  • 40.0% of sampled Reasoning failures were Incomplete Answer (16/40), the most frequent category; Incorrect Answer and Format Violation each accounted for 27.5% (11/40).Incomplete answers may lose recall under word-level F1, but do not by themselves show whether information was unavailable, poorly exchanged, or omitted during answer construction.Source: §B.1.2 · pp. 14–15, Figure 6 · p. 14

Scheduling

  • Exploration Failure (42.5%, 17/40) and Misjudgment (37.5%, 15/40) together made up 80.0% of sampled Scheduling failures.The categories distinguish failure to surface feasible common windows from incorrect evaluation of proposed candidates.Source: Figure 6 · p. 14, §B.1.2 · p. 15

Game

  • Low Efficiency accounted for 47.5% (19/40) of sampled Game failures, followed by Low Exploration at 27.5% (11/40) and Navigation Failure at 20.0% (8/40).Efficiency and exploration issues together represented 75.0% of the sample; the reported problems include redundant or unproductive actions and insufficient map coverage.Source: Figure 6 · p. 14, §B.1.2 · p. 15

B.1.3 QUALITATIVE CASE STUDIES

The case studies trace how agents handle missing evidence, negotiation state, and local discoveries under partial observability. They illustrate interaction patterns, not population-level estimates of mechanism effects.

Reasoning

  • For the question comparing Elara Vance’s and Rhys Calder’s combined Nebula Quill and Crimson Scroll wins, Base answered “Elara Vance” (word-level F1 = 0), while Protocol answered reference “Rhys Calder” (F1 = 1).Protocol obtained the missing award counts before committing to an answer.Source: §B.1.3 · p. 15

Scheduling

  • On the same calendars, BASE and MEMORY find no slot, while MEMORY+PROTOCOL finalizes the feasible Day 4 08:00–08:20 window in round 3.Memory retained the candidate, but an unstructured reply was not converted into agreement; explicit voting resolved this in the recorded run.Source: §B.1.3 · pp. 15–16

Game: successful communication

  • In a successful GAME case, Agent 0 shares an unseen warehouse location, after which Agent 1 deposits four sand units and the team score rises from 8 to 12.The example links a local discovery shared by message to a task outcome, rather than message volume alone.Source: §B.1.3 · p. 16

Game: failed deposit

  • In a separate GAME failure, Agent 3 follows earlier teammate reports over a current observation showing zero sand capacity, and the attempted deposit yields no score gain.The records localize the failure to reconciling conflicting evidence at action selection, but do not identify whether memory updating or message interpretation caused the discrepancy.Source: §B.1.3 · p. 17

Cross-task implications

  • The selected cases and error distributions motivate checking evidence requests, proposal and feasibility coordination, and whether received information becomes valid, productive actions.The cases illustrate interaction patterns rather than estimate their frequency.Source: §B.1.3 · pp. 15, 17

B.2 CONTRIBUTION ANALYSIS ON GAME CATEGORY

The GAME contribution analysis uses normalized Contribution Entropy to assess how evenly agents contribute. Results show higher, tighter entropy distributions when Protocol is added.

Metric

  • Contribution Entropy measures the distribution of agents’ final cumulative contributions from resources successfully deposited in warehouses.Contributions are normalized before entropy is computed.Source: §B.2 · pp. 17–18
  • Runs with no positive contribution are excluded; zero-contribution agents remain in normalization, and a run with one positive contributor receives entropy 0.The normalized statistic ranges from 0 to 1, with higher values indicating more evenly distributed contributions.Source: §B.2 · p. 18

Results

  • Adding Protocol yields higher normalized entropy and tighter distributions across backbones, while Memory or Routing alone does not guarantee equitable participation.Gate routing improves stability over Random, but both still show visible variance.Source: §B.2 · p. 18

B.3 ADDITIONAL RESULTS OF ROUND-WISE SCORE EVOLUTION

Additional backbones show score gains and faster convergence when task-relevant collaboration mechanisms are incorporated. Convergence differs by task: gains appear earlier in Reasoning and Scheduling than in Game.

Round-wise score evolution

  • Across the remaining evaluated backbones, task-relevant collaboration mechanisms contribute to overall performance and faster multi-round convergence.Figure 8 presents round-wise score evolution for REASONING.Source: §B.3 · p. 18, Figure 8 · p. 18

Task-dependent convergence

  • In REASONING and SCHEDULING, most score gains emerge in early rounds as agents exchange missing clues or eliminate infeasible meeting slots.These task-specific patterns are reported across the round-wise curves.Source: §B.3 · p. 18
  • GAME score growth is more gradual because agents first explore, accumulate spatial knowledge, and coordinate resource delivery over longer horizons.The authors connect this pattern to the progressive task design.Source: §B.3 · p. 18

B.4 ADDITIONAL GAME RESULTS WITH FULL MESSAGE-PHASE TOKEN COST

The alternative GAME accounting includes all completion tokens generated during the message phase, not only transmitted message tokens. Under this accounting, transmitted volume does not always track generation cost, while adding Protocol to Gate improves message-phase efficiency across backbones.

Full message-phase accounting

  • Under Table 6’s alternative GAME accounting, Tok includes all completion tokens generated during the message phase, making communication overhead more conservative.The accounting is especially relevant to routing methods with additional send and recipient decisions.Source: §B.4 · pp. 18–19, Table 6 · p. 19

Cost comparison

  • Fewer transmitted tokens do not always mean lower message-phase generation cost: MEMORY+GATE costs more than MEMORY+RANDOM for GPT-5-nano and slightly more for DeepSeek-v3.2.MEMORY+GATE uses fewer transmitted tokens than MEMORY+RANDOM for every backbone.Source: §B.4 · p. 19
  • Adding PROTOCOL to GATE improves overall message-phase efficiency across backbones, though the reduction in generation cost varies substantially by model.The results distinguish communication volume from the generation overhead of deciding when and where to communicate.Source: §B.4 · p. 19

B.5 CROSS-ROUND MEMORY ABLATION ON SCHEDULING

The Scheduling ablation finds that retaining negotiation state across rounds improves Score over BASE, while full MEMORY usually adds further gains with additional communication. The framework-level evaluation applies the benchmark’s task boundaries across two agent frameworks, with task-specific integration and interaction settings.

B.5 CROSS-ROUND MEMORY ABLATION ON SCHEDULING

  • CROSS-ROUND MEMORY raises Overall Score over BASE for all four backbones, and Table 7 reports the ablation across backbones and splits.Overall Score rises from 1.0 to 2.0 for GPT-5-nano, 17.0 to 27.0 for Gemini-2.5-Flash-Lite, 22.5 to 36.5 for DeepSeek-v3.2, and 13.5 to 39.5 for Qwen-Plus.Source: Table 7 · p. 20, §B.5 · p. 20
  • Full MEMORY outperforms CROSS-ROUND MEMORY in nine of 12 backbone–split Score comparisons, ties in one, and scores lower in two.At Overall, the variants tie for GPT-5-nano; full MEMORY raises Score from 27.0 to 36.5 for Gemini, 36.5 to 38.0 for DeepSeek, and 39.5 to 40.0 for Qwen.Source: §B.5 · p. 20
  • CROSS-ROUND MEMORY uses more communication tokens than BASE in all 12 comparisons but has higher communication efficiency in 11, with DeepSeek-v3.2 Simple the exception.It uses fewer tokens than full MEMORY in 10 of 12 comparisons; the full configuration usually adds task-performance gains at the cost of additional communication.Source: §B.5 · p. 20

C FRAMEWORK-LEVEL EVALUATION DETAILS

  • The framework-level evaluation tests CAMEL and AgentVerse on all three MASBench tasks, using task adapters for observation boundaries and interaction rules.Frameworks provide persistent agents and conversation management; the reference uses PROTOCOL for REASONING, MEMORY+PROTOCOL for SCHEDULING, and MEMORY+GATE+PROTOCOL for GAME.Source: C FRAMEWORK-LEVEL EVALUATION DETAILS · p. 20

C.1 EVALUATION SUBSETS AND INFORMATION BOUNDARIES

  • Evaluation uses 30% of instances sampled within task difficulty strata with seed 42, and preserves each task’s private-input boundaries.The subsets contain 90 REASONING instances, 60 SCHEDULING instances, and 30 GAME maps; the same manifests are used across frameworks and backbones.Source: §C.1 · p. 21

C.2 FRAMEWORK INTEGRATION

  • Framework adapters deliver public utterances through native memory interfaces, while GAME uses restricted point-to-point messages and disables the reference scaffold’s extra memories.CAMEL retains a 32-entry window and AgentVerse the most recent 16 model outputs, approximately eight action–message rounds under their respective formats.Source: §C.2 · p. 21

C.3 TASK-SPECIFIC INTERACTION AND TERMINATION

  • REASONING alternates agents for at most ten individual turns, while SCHEDULING uses sequential member utterances and ends after checker-confirmed agreement or five rounds.Scheduling proposals are deterministically checked against all private schedules, and infeasible proposals fail even if consensus is reported.Source: §C.3 · pp. 21–22
  • GAME runs for 50 rounds without early stopping, and its results are averaged over five executions without recoverable standard deviations or significance claims.The runner uses environment and model seed 42, temperature zero, and top-p one.Source: §C.3 · p. 22

C.4 EVALUATION METRICS AND AGGREGATION

The evaluation defines task-specific performance scores and communication cost, then aggregates results over instances. Comparisons concern complete system configurations, so they do not isolate the causal effect of changing only the framework library.

C.4 EVALUATION METRICS AND AGGREGATION

  • Performance uses word-level F1 for REASONING, feasible-window success for SCHEDULING, and reference-normalized resource scores for GAME, scaled by 100.Overall performance weights instances equally, and GAME averages repetitions per map before aggregation.Source: §C.4 · p. 22
  • Tok measures average inter-agent message tokens, excluding prompts, private reasoning, private actions, and non-transmitted control outputs.It measures communication volume, not end-to-end inference cost, so differences do not directly indicate total generated tokens, API expenditure, or execution time.Source: §C.4 · p. 22
  • Task-specific Tok accounting includes public utterances or shared-conversation messages, and in GAME includes transmitted STATE, OBS, and MSG fields.The accounting scope is applied consistently to reference and framework implementations.Source: §C.4 · p. 22

C.5 SCOPE OF INTERPRETATION

  • Framework comparisons cover complete system configurations, so observed differences may reflect several implementation choices rather than only the framework library.The results distinguish task-dependent performance and communication behavior but do not isolate the causal effect of replacing only the underlying framework.Source: §C.5 · p. 22

D BENCHMARK DESIGN DETAILS

This section specifies the task workflows and collaboration mechanisms for partially observable Reasoning, Scheduling, and Game tasks, including communication constraints, information and memory handling, routing, voting, and prompt templates.

D.1.2 QUALITY ASSURANCE AND SCORING

  • Reasoning instances split multi-hop evidence between two agents, with quality checks and retry selection filtering generations before difficulty stratification.Validation checks format and supporting-fact coverage, question unanswerability without context, sufficiency of supporting facts, and absence of answer shortcuts; generation retries up to five times until s >= 3.5. A heuristic score divides instances into Simple and Complex subsets, but is not used for evaluation.Source: §D.1.2 · p. 23, §D.1.1 · p. 23, §D.1.3 · p. 23

D.2.1 CALENDAR GENERATION

  • Scheduling calendars span five days in 10-minute units, with 20-minute meetings; stochastic busy blocks and fixed rest periods shape agents’ availability.The generator samples each agent’s busy ratio from 0.3–0.6, inserts activities lasting 10–60 minutes, and regenerates schedules with too few common feasible windows.Source: §D.2.1 · pp. 24–25

D.2.2 CANDIDATE SOLUTION PRUNING

  • Candidate pruning preserves exactly K feasible meeting windows, and difficulty stratification defines Simple as K ∈{4, 5} and Complex as K ∈{2, 3}.The algorithm protects K sampled solutions and adds minimum-unit BUSY blocks to eliminate non-protected solutions; retained instances remain solvable. Agents propose a normalized time interval, evaluated by strict Exact Match, with partial overlaps counted as failures.Source: §D.2.2 · p. 25, §D.2.3 · p. 25, §D.2.4 · p. 25

D.3.1 ENVIRONMENT

  • The Game task is a partially observable 2D resource-collection game in which agents must deliver resources to compatible warehouses within 50 rounds.Agents observe a local square window; maps grow from 15 × 15 in Simple instances to 18 × 18 in Complex instances, which also use more agents and resources. Deposits count only within total and per-resource warehouse capacity limits.Source: §D.3.1 · pp. 26–27, §D.3.2 · p. 27

D.3.2 OBJECTIVES AND SCORING

  • Game scores normalize correctly deposited resources against a map-specific heuristic reference, which is not a guaranteed optimum or strict upper bound.The reference planner uses full-map information and heuristic task-allocation plans; the reported score averages normalized scores across evaluated maps. No reported run exceeded the reference score, so clipping did not affect results.Source: §D.3.2 · p. 27

E.1.1 BASE WORKFLOW

  • The Reasoning task’s Base workflow has two agents exchange private evidence through turn-based natural-language dialogue visible to both participants.Each agent sees the question, its private context, and the full dialogue history; neither has enough information to solve the task independently, and messages need not follow a predefined schema.Source: §E.1.1 · pp. 30–31

E.1.2 PROTOCOL

  • Protocol structures reasoning-task collaboration through public MESSAGE blocks and Q/R/A tags, ending when an answer is proposed or after 10 rounds without one.Agents are instructed to seek missing information from partners before committing to a complete answer; only the public message block is transmitted.Source: §E.1.2 · p. 31, §E.1.3 · p. 31

E.1.3 PROMPT TEMPLATES

  • The reasoning-task prompts make evidence incomplete by design and instruct agents to use a partner’s information before returning a complete answer.The Base workflow accepts answers from the last <A>answer</A> occurrence or the first \boxed{answer} occurrence.Source: §E.1.3 · p. 31, §E.1.1 · p. 31

E.2.4 PROMPT TEMPLATES

  • The scheduling task asks privacy-constrained participants to find one common slot of the specified duration within a turn limit, without revealing full schedules.The prompt also specifies shared unavailable periods and directs an adjudicator to classify proposals and identify consensus.Source: §E.2 · p. 32, §E.2.4 · pp. 34–35

E.2.1 BASE WORKFLOW

  • The scheduling Base workflow has agents negotiate in natural language using private calendars and shared conversation history, with a neutral detector checking for consensus.Each agent infers free time from gaps between busy intervals; the detector has no access to private schedules or hidden states.Source: §E.2.1 · pp. 32–33

E.2.2 MEMORY

  • Scheduling Memory precomputes each agent’s feasible availability and tracks proposed and rejected slots across rounds to reduce repeated reasoning.It extracts free intervals, filters out those shorter than the meeting duration, stores the verified list locally, and updates shared team memory from the conversation log.Source: §E.2.2 · p. 33

E.2.3 PROTOCOL

  • Scheduling Protocol replaces open-ended discussion with three phases—broad availability scan, concrete proposals, and votes—and phase-specific JSON outputs.This structure reduces ambiguity in collecting candidates and voting, making final consensus easier to verify.Source: §E.2.3 · p. 33

Memory

  • Agents check each live proposal against their private availability, accept only slots fully inside a free interval, and share decisions rather than their full agendas.The negotiation instructions also require agents to address live slots before proposing alternatives.Source: Memory · p. 38
  • A proposed slot remains a candidate only while no agent has rejected it; any rejection makes it dead, and rejected slots take precedence in shared memory.The scribe classifies proposed, not-yet-rejected slots as candidates and merges new rejections while removing rejected slots from the candidate set.Source: Memory · p. 38
  • Memory+Protocol first aggregates votes for day-and-time blocks, selects the top two priorities, and uses protocol state transitions to move between collection, voting, reset, and finalization.The state machine stores broad blocks, candidates, rejections, and the current phase; a failed vote resets candidates into rejected slots.Source: Memory · pp. 40–41
  • During slot collection, agents propose exact-duration slots in leader-prioritized zones, favoring underrepresented hours while preserving availability privacy.The protocol coordinator removes wrong-duration and blacklisted slots, deduplicates exact matches, retains overlapping slots, and sorts the results by day and start time.Source: Memory · pp. 41–42
  • A candidate slot is valid only if it fits wholly within an agent’s free interval, and adjudicators accept it only with 5/5 votes; otherwise collection resumes.Agents must verify interval boundaries without revealing their full availability.Source: Memory · pp. 43–44

E.3 GAME

  • The Game task has agents cooperatively collect resources and deposit them on warehouse tiles within a fixed number of rounds, with Memory, Routing, and Protocol extending the base workflow.Agents operate under partial observability.Source: §E.3 · p. 44

E.3.1 BASE WORKFLOW

  • Agents act from local observations each round; the environment synchronously resolves movement, collisions, pickups, and deposits, while Base disables communication.Only compatible warehouse deposits count toward score, subject to resource quotas and action-capacity constraints.Source: §E.3.1 · p. 44

E.3.2 MEMORY

  • Memory maintains a bounded known map and communication buffers, merges new observations with message-derived facts, and injects pruned relevant hints as advisory context for action selection.Direct observations override older message-derived facts; simulator rules still determine valid movement, pickup, and deposit actions.Source: §E.3.2 · pp. 44–45

E.3.3 ROUTING

  • Routing controls whether to message and which in-range teammate receives it, while environment actions remain in the Base workflow; Random selects recipients deterministically and Gate-LLM uses SEND and TO decisions.Messages are limited to one per communication round and are delivered next round if the recipient remains in range.Source: §E.3.3 · p. 45

E.3.4 PROTOCOL

  • Protocol replaces free-form payloads with structured STATE, OBS, and MSG components and runtime guards against unsupported or contradictory facts, without changing action rules.Routing determines whether and to whom to send; Protocol governs message construction and validation.Source: §E.3.4 · p. 45

E.3.5 PROMPT TEMPLATES System Prompt

  • Prompt templates share the common action prompt, add memory hints when enabled, and give Routing a separate message phase with distinct Gate and Random prompts.Gate prompts separate the send decision, recipient choice, and message generation; the common action prompt requires an action and applicable quantity or type fields.Source: §E.3.5 · pp. 46–48
  • Recipients are selected from communication-range candidates, and Protocol reuses Routing’s recipient while constraining message content.Protocol prompts request one useful message and attach supporting STATE/OBS evidence.Source: §E.3.5 · p. 48
  • Routing requires a two-line LANE/MSG update, with a short grounded message that avoids inventing coordinates, quantities, capacities, or carried items.The template supplies recipient context, routing context, content candidates, and evidence; resource and warehouse updates must include type and coordinate.Source: §E.3.5 · pp. 48–49
Loading 2610.04672v1…