Source-linked AI summary
CoCoBench: A Cooperative Coordination Benchmark for Embodied Multi-Agent Task Planning
Yang Chen, Ye-Xin Xie, Lirong Che, Danyang Peng, Yuzhe Yang, Peiwen Lin, Xu Cao, Chuang Wang, Lei Yuan, Jian Su, Lan-Zhe Guo
TL;DR
Existing embodied-agent benchmarks provide limited fine-grained diagnosis of multi-agent coordination beyond task completion. CoCoBench introduces an 897-instance benchmark organized around four coordination constructs and evaluates 11 MLLMs with construct-level scores. Results show coordination is highly structured, with centralized state aggregation remaining important and larger teams increasing difficulty.
Problem
Existing embodied-agent benchmarks often summarize multi-agent behavior with aggregate success or efficiency metrics, limiting fine-grained diagnosis of coordination.
Method
CoCoBench organizes 897 executable household-task instances around four coordination constructs and evaluates trajectories with construct-level scores alongside task success.
Results
Across 11 leading MLLMs, coordination ability varies by construct; centralized state aggregation remains important, and increasing team size amplifies coordination difficulty.
Takeaways & Limitations
CoCoBench separates goal completion from coordination quality and enables construct-level diagnosis of cooperative execution.
Takeaways & Limitations
Trajectory-level construct scores require sufficient evidence in the recorded trajectory and may differ from final task success.
Abstract
from arXiv · showhide
Agent systems powered by multimodal large language models (MLLMs) have advanced rapidly in recent years, yet existing embodied-agent benchmarks still lack fine-grained diagnostics for multi-agent coordination. Most benchmarks either focus on single-agent task completion or summarize multi-agent behavior with overall task success rates, which can obscure coordination failures such as duplicated work, violations of ordering constraints, resource contention, and desynchronized handoffs. In this paper, we introduce CoCoBench, a construct-level benchmark for evaluating multi-agent embodied coordination in executable household tasks. CoCoBench contains 897 oracle-validated instances organized around four recurring coordination constructs: task allocation, sequential ordering, mutual exclusion, and handoff coordination. In addition to task success rate, CoCoBench provides construct-level scores that measure whether agents coordinate effectively. We evaluate 11 leading MLLMs across different coordination modes, observation inputs, and numbers of agents. The results show that coordination ability is highly construct-specific: strong overall performance does not imply balanced competence across different coordination types. These findings point to new directions for designing targeted model architectures and improving multi-agent coordination ability.
Introduction
CoCoBench addresses the limited diagnostic value of aggregate task success by evaluating how embodied agents coordinate across recurring collaboration demands. It introduces construct-level task designs and scores, then shows coordination varies across constructs and experimental settings.
- Aggregate success rates can obscure duplicated work, ordering violations, resource contention, handoff desynchronization, and low-level execution errors.
- CoCoBench contains 897 instances organized around task allocation, sequential ordering, mutual exclusion, and handoff coordination.
- Its construct-level scores measure allocation efficiency, precondition compliance, conflict-free resource scheduling, and handoff rhythm beyond final task success.
- The benchmark uses configurable high-level skills, including navigation, pickup, placement, and slicing, while grounding instances in executable household activities.
- Across 11 leading MLLMs, strong aggregate performance does not imply balanced competence across coordination constructs.
- Removing visual observations does not substantially degrade performance under the high-level skill interface, while larger teams increase coordination difficulty.
- Successful episodes may still diverge from legal coordination, making task success alone insufficient for diagnosis.
Related Work
Prior embodied-agent benchmarks span household collaboration, robotic manipulation, and open-world environments, but their coordination taxonomies often support limited mechanism-specific diagnosis. CoCoBench instead decomposes dependencies into explicit coordination mechanisms and construct-level scores.
- Prior embodied-agent research evaluates planning, navigation, and open-world exploration, with household environments serving as a major application domain.
- Existing embodied multi-agent benchmarks cover collaborative household tasks, robotic manipulation, and open-world environments.
- MECoBench varies team sizes, coordination modes, communication settings, and parallel or sequential task structures.
- Existing task taxonomies often cover limited high-level coordination structures or conflate multiple dependency mechanisms within one task.
- CoCoBench decomposes task-level dependencies into explicit coordination mechanisms and provides construct-level scores for each mechanism.
Problem Formulation
CoCoBench formulates embodied coordination as high-level joint decision making in a shared environment, where one agent’s actions can alter another’s feasibility, usefulness, or efficiency. It evaluates resulting trajectories with separate task-success and construct-specific coordination measures.
- CoCoBench models coordination as joint high-level decision making by multiple agents in a shared interactive environment.
- Agents receive observations and select executable high-level skills that a shared executor applies to the environment.
- The executor records states, observations, actions, and execution events into a trajectory for evaluation.
- Coordination coupling occurs when one agent’s action changes another agent’s action feasibility, goal contribution, or execution cost.
- CoCoBench evaluates each trajectory along task completion and construct-specific coordination quality rather than a single blended reward.
Benchmark Design
CoCoBench grounds four coordination constructs in executable household activities and scores coordination quality separately from final task success. Its fixed interface, deterministic execution, and oracle-based filtering support mechanism-specific evaluation across varied task families and team sizes.
- Coordination Constructs and Scores: The benchmark defines four constructs: task allocation, sequential ordering, mutual exclusion, and handoff coordination.These represent parallel work division, causal precedence, exclusive-resource scheduling, and paced producer-consumer transfer.
- Coordination Constructs and Scores: Construct scores assess allocation balance, ordering compliance, contention-free resource use, and producer-consumer pacing independently of task completion.A high handoff score indicates that producers and consumers maintain a compatible pace around the buffer.
- Benchmark Design: 897 validated instances span ten task families, four room categories, and team sizes N ∈{2, 3, 4}.
- Agent Evaluation Design: A fixed executable skill interface and deterministic executor expose construct-specific events while isolating high-level team coordination.The recorder logs states, observations, selected skills, outcomes, and failures for deterministic metric computation.
- Benchmark Design: Instances pass placement, oracle, and metric quality-control gates before inclusion.The gates verify feasibility, solvability, and near-ceiling oracle performance on the target construct.
- Agent Evaluation Design: Figure 4 provides qualitative examples across the benchmark’s four coordination dimensions.
Experiments
Experiments evaluate 11 MLLMs on CoCoBench using SR and CS across centralized and decentralized policies, observation settings, team sizes, and failure diagnostics. Results show construct-specific performance, benefits from centralized state aggregation, increasing coordination difficulty with team size, and divergence between task success and legal coordination.
- Experimental setup: The evaluation uses a full 897-instance set and a balanced 240-instance subset, reporting task success rate (SR) and construct score (CS).The subset contains 60 instances per construct and balances construct-by-team-size comparisons.
- Metric validation: The four coordination constructs are largely non-redundant, supporting construct-level analysis beyond binary task success.Cross-applying all four scoring functions to GPT-5.6-sol trajectories produced a four-dimensional score vector for each trajectory.
- Main results: GPT-5.6-sol achieves the highest overall SR (84.8%) and CS (0.90), but leading performance varies across constructs and does not necessarily transfer between them.Qwen3-VL-8B leads D1, Claude Opus 4.8 leads D2, and GPT-5.6-sol leads D3 and D4.
- Architecture effects: 48.0% SR and 0.69 CS under centralized planning exceed decentralized planning with communication (39.6% SR, 0.65 CS) and without communication (36.7% SR, 0.63 CS).Communication benefits decentralized coordination but only partially compensates for losing centralized state aggregation.
- Observation ablation: Removing images does not reduce aggregate performance: blind evaluation reaches 42.3% SR and 0.660 CS versus 40.6% SR and 0.655 CS with images.Under the high-level skill interface, the benchmark identifies symbolic multi-agent coordination rather than raw visual recognition as the main bottleneck.
- Team-size scaling: Average SR declines from 48.8% with two agents to 45.8% with three and 43.1% with four, while CS declines from 0.68 to 0.64 and 0.63.InternVL3.5-8B drops from 16.9% SR at two agents to 7.7% at four agents, illustrating stronger degradation in a weaker model.
- Failure analysis: Open-weight models average 17.5% protocol failures, and budget-plus-protocol terminations comprise 54.3% of their episodes.Budget exhaustion reaches 61% for InternVL3.5-8B and 52% for RoboBrain2.5-8B.
- Legality analysis: Success and legality diverge: Claude Opus 4.8 is legal on 96.3% of episodes but succeeds on 78.5%, whereas Claude Haiku 4.5 succeeds on 63.1% but is legal on 47.2%.Legality and CS therefore distinguish correct coordination from reaching the goal through an illegal shortcut.
Conclusion
CoCoBench evaluates multi-agent embodied coordination through four dimensions, separating goal completion from coordination quality. Results across 11 MLLMs show structured coordination ability, the importance of centralized state aggregation, and greater difficulty with larger teams.
- Conclusion: CoCoBench separates goal completion from coordination quality across four coordination dimensions.It enables judge-free, construct-level diagnosis in executable household environments.
- Conclusion: 11 leading MLLMs show highly structured coordination ability rather than balanced competence across constructs.Strong overall performance does not imply equal ability across coordination dimensions.
- Conclusion: Centralized state aggregation remains important for multi-agent coordination.The conclusion identifies this as a recurring result of the evaluation.
- Conclusion: Increasing team size amplifies coordination difficulty.
Appendix Benchmark Details
This section details CoCoBench’s benchmark-side design, including task realization, symbolic-template grounding, validation, composition, agent inputs, and successful trajectories.
- Appendix Benchmark Details: The benchmark details cover task realization, symbolic-template grounding, AI2-THOR validation, composition, agent inputs, and successful trajectories.
Task Definition
CoCoBench organizes ten household task types around four coordination dimensions: allocation, ordering, mutual exclusion, and handoff coordination. These dimensions test complementary requirements such as division of labor, causal progress, resource scheduling, and role-aware transfers.
- Task Definition: Ten task types are tied to one coordination dimension and instantiated with scene-specific objects, receptacles, and agent counts.
- Task Allocation: Task allocation tests decomposing conjunctive goals into independent work units without unnecessary duplication.Examples include concurrently transporting different groceries or completing multiple object–destination pairs in parallel.
- Sequential Ordering: Sequential ordering tests whether agents preserve causal dependencies and delay terminal actions until teammates finish pending placements.Prematurely closing a receptacle can invalidate or block another agent’s work.
- Mutual Exclusion: Mutual exclusion tests contention-free scheduling when agents share an exclusive resource such as one Knife.Agents must acquire, use, and release or transfer access without simultaneous acquisition or unnecessary retention.
- Handoff Coordination: Handoff coordination tests producer–consumer pacing as objects cross an intermediate buffer or spatial boundary.Producers must avoid overflow or contention, while consumers must avoid waiting at an empty buffer.
Benchmark Construction Details
CoCoBench grounds symbolic coordination templates in executable AI2-THOR scenes and uses a fixed high-level skill interface across instances. The construction process specifies scene, goal, team, role, and coordination constraints before simulator execution records outcomes.
- Benchmark Construction Details: Each instance is generated by grounding a symbolic coordination template in a compatible executable AI2-THOR scene.Templates specify room category, object classes, goal predicates, team size, target construct, and construct-specific constraints.
- Benchmark Construction Details: The generator samples floorplans, concrete objects, legal placements, applicable agent roles, and a random seed before rendering natural-language instructions.
- Benchmark Construction Details: The executable interface remains fixed, providing parameterized high-level skills for navigation, waiting, pickup, placement, and related interactions.D1–D3 use homogeneous menus, while D4 uses role-conditioned producer and consumer menus.
- Benchmark Construction Details: Table 2 defines and illustrates ten coordination-oriented task types in CoCoBench.
Coordination
Coordination tasks can require agents to divide independent household subtasks without repeating one another’s actions.
- Task Allocation: Agents split independent household objectives, such as storing food and switching off a light, while avoiding repeated actions.The example assigns potato and lettuce placement alongside turning off the LightSwitch.
D2 Sequential
Sequential tasks require agents to complete a loading sequence before performing the container’s final close action.
- Sequential Ordering: Agents load several objects into a container and close it only after the loading sequence is complete.The example requires placing the KeyChain, Watch, CreditCard, and RemoteControl in the Box before closing it.
Ordering
CoCoBench evaluates multi-agent embodied coordination through explicit constructs, executable task instances, and a high-level execution loop. Its examples show that allocation, ordering, resource sharing, and handoffs require distinct coordination behaviors, while diagnostics expose model-specific failure patterns and oracle limitations.
- Benchmark Design: CoCoBench contains 897 oracle-validated instances spanning four coordination dimensions, six household task families, four room categories, and three team sizes.The dimensions include task allocation, sequential ordering, mutual exclusion, and handoff coordination.
- Failure Diagnostics: Dependency violations account for 11–95% of failures, whereas occupancy conflicts account for 1–7%, affordance failures 0–3%, and illegal-skill use 0% across models.The action menu blocks illegal attempts before they propagate into episode failures.
- Execution Loop: The execution loop grounds model-selected action IDs into concrete skills, executes them in AI2-THOR, and feeds updated progress and feedback into subsequent prompts.Prompts can also include inventories and teammate broadcasts for D1 coordination.
- Task Allocation: D1 succeeds when agents commit to different pending subgoals and avoid redundantly pursuing objects already handled by teammates.The Store Groceries example combines separable food-placement and light-switch objectives.
- Sequential Ordering: D2 requires delaying a terminal Close action until live progress confirms that all prerequisite placements have succeeded.Agents may retrieve items concurrently, but the shared Cabinet must remain open until both placements are complete.
- Mutual Exclusion: D3 serializes ownership of an exclusive Knife while allowing other preparation actions to proceed.The current holder yields access after completing its assigned slice, preventing conflicting acquisition attempts.
- Handoff Coordination: D4 uses synchronized producer–consumer handoffs, with source agents filling a transfer buffer and destination agents moving its contents onward.This pattern prevents premature downstream searches and excessive accumulation at the handoff point.