AgentWorld: Benchmarking Long-Horizon Collaboration of Multi-agent LLMs

1 Introduction

• Existing benchmarks often use horizons of 10 steps or fewer, mix collaboration with low-level control, or measure emergent social behavior rather than concrete tasks.The paper frames systematic evaluation of collaboration as important because many real-world tasks require specialized agents to coordinate toward shared goals.Source: §1 · p. 2
• 52.0% task success and CCE = 0.320 are the best model’s results, with less than a third of its actions contributing causally to task completion.The reported failure patterns include communication breakdowns, role confusion, and difficulty maintaining shared plans across rounds.Source: §1 · pp. 2–3
• AgentWorld contains 100 human-annotated tasks and 100 augmented variants requiring 3–20 agents with asymmetric roles to coordinate over 25–55 rounds.Its sandbox abstracts low-level control, while CCE traces causal dependencies between actions to measure the fraction of team effort contributing to outcomes.Source: §1 · pp. 2–3

2 Related Work

• Many multi-agent LLM benchmarks have horizons under 20 steps, lack asymmetric roles, or do not enforce blackbox interaction.Single-agent benchmarks evaluate agents without addressing multi-agent collaboration.Source: §2 · p. 3
• Social simulations explore emergent dynamics but may lack concrete tasks with measurable outcomes, while cited RL platforms target RL agents rather than LLM systems.The passage contrasts those platforms with LLM agent frameworks and simulations.Source: §2 · p. 4
• AgentWorld combines blackbox interaction, asymmetric roles, long horizons, and a world-level MMORPG sandbox.The comparison table covers simulator capabilities and benchmark characteristics.Source: §2 · p. 4, Table 1 · p. 4

3 AgentWorld Benchmark

• AgentWorld’s curated tasks cannot be solved by a single agent and require sustained coordination across multiple rounds.The simulator abstracts low-level control so benchmark performance reflects collaboration rather than fine-grained action proficiency.Source: §3 · p. 4

3.1 Simulation Environment

• The simulator balances world complexity with high-level tools that abstract low-level actions, leaving agents to decide what to do and whom to coordinate with.The Kaetram world includes 380+ items, 144 mob types, 70+ NPCs, and 1,531 harvestable resource nodes; agents act in round-robin order without access to others’ internal states.Source: §3.1 · pp. 4–5

3.2 Task Design

• Each task specifies a collaborative objective, asymmetric agent configurations, a round budget, agent-specific context, and a Python success verifier.Tasks distribute necessary skills and resources across agents, as in Resource Caravan, where teams gather materials in separate biomes for a crafter.Source: §3.2 · p. 5

3.3 Task Annotation

• Five annotators define complementary roles, objectives, checkpoints, verifiers, and agent-specific documentation, then pilot tasks and revise them until only agent errors remain.Round budgets are calibrated from pilot runs; the dataset also includes 100 LLM-augmented variants.Source: §3.3 · pp. 5–6

3.4 Dataset Statistics

• The 100 main tasks span eight categories, use 3–20 agents, and allow 25–55 rounds; survival has 67–100% SR, while coordination has 12% and construction 20%.Most tasks require three agents, and the average round budget is 38.Source: §3.4 · p. 6, Figure 4 · p. 6

3.5 Evaluation Metrics

• Task success rate uses task-specific Python verifiers, while partial success rate averages the fraction of verifier checkpoints achieved across tasks.Successful tasks receive PSR = 100%.Source: §3.5 · p. 7
• CCE is the ratio of actions on causal paths to success to all actions; failed tasks receive CCE = 0.It uses iterative backward tracing with binary causal judgments rather than holistic collaboration ratings; the illustrated Task 01 score is 12/27 = 44.4%.Source: Figure 5 · p. 6, §3.5 · p. 7

4.2 Main Results

• Gemini 3 Flash led the main set at 52.0% SR, while all four models’ SR fell substantially on the augmented set.The main-set ranking was Gemini 3 Flash (52.0%), Claude Haiku 4.5 (45.0%), GPT-5 Mini (36.0%), and DeepSeek R1-70B (20.0%).Source: §4.2 · pp. 7–8, Table 2 · p. 8
• GPT-5 Mini sent the most messages (44.1) but ranked third in SR, while DeepSeek delayed its first chat until round 5 on average.On failed tasks, GPT-5 Mini devoted 26% of its actions to chatting; the authors suggest communication should coordinate without displacing productive action.Source: §4.2 · p. 8
• Survival was easiest (50–100% SR across models), while coordination, construction, and trading were harder; augmentation lowered every model’s SR.The reported augmented-set drops were Gemini 52→24%, Claude 45→26%, GPT-5 36→21%, and DeepSeek 20→10%.Source: §4.2 · p. 8

4.3 Causal Collaboration Effectiveness

• Even Gemini’s CCE was 0.320, meaning fewer than a third of its actions lay on a causal path to task completion.Claude had the highest per-task collaboration efficiency on successful tasks (0.653 versus Gemini’s 0.609).Source: §4.3 · pp. 8–9
• Human annotators agreed with the GPT-4.1 causal judge on 82% of sampled judgments, and three judge models preserved the same CCE ranking.The human comparison yielded Cohen’s κ of 0.64; switching judges shifted absolute CCE values by at most about 0.05.Source: §4.3 · p. 9

4.4 Baselines, Ablations, and Robustness

• Random actions solved 5.7% of tasks and the single-agent baseline reached 28.6%; oracle communication raised vanilla success from 52.0% to 60.0%.A human shared plan without communication slightly hurt success (22.9% versus 17.6%); even oracle communication left a 40% failure rate.Source: §4.4 · pp. 9–10, Table 3 · p. 9
• Removing task documentation or shortening the horizon produced the largest ablation drops, while randomizing spawn locations had the smallest effect.On the 35-task subset, documentation removal changed success from 54.3% to 29.4%, horizon shortening to 10 rounds changed it to 37.1%, and spawn randomization changed it to 51.4%.Source: §4.4 · p. 10, Table 4 · p. 10

4.5 Analysis on Agent-to-Agent Communication

• Across 100 Gemini task traces, stale and redundant messages accounted for 37.7% of classified communication errors.The authors suggest these errors likely stem from incorrect understanding or prediction of other agents’ states.Source: §4.5 · p. 10

5 Conclusion

• AgentWorld provides a benchmark for evaluating long-horizon multi-agent collaboration in an MMORPG sandbox with 100 human-annotated tasks and 100 augmented variants.Tasks require 3–20 agents with asymmetric roles to coordinate through natural language under blackbox interaction.Source: §5 · p. 10
• CCE uses backward tracing to build causal action graphs and measure the fraction of team effort that contributed to the outcome.Source: §5 · pp. 10–11
• Across four frontier models, the best achieved 52.0% task success, and less than a third of its actions contributed causally to success.The conclusion also reports a persistent partial-to-full-success gap and that communication quantity did not predict success.Source: §5 · p. 11

• AgentWorld uses simulated sandbox scenarios to test collaboration, not negative behaviors such as attacks.Source: Ethics Statement · p. 11
• Annotators came from diverse backgrounds and were compensated in accordance with local standards.Source: Ethics Statement · p. 11

• Table 5 reports a communication failure-mode analysis, referenced in Section 4.5.Source: A Additional Experimental Results · p. 14, Table 5 · p. 14
• Table 6 reports CCE under three judge models, with the agent-model ranking preserved across all three.Source: Table 6 · p. 14
• Absolute CCE values shift by at most approximately 0.05 across the three judges.Source: Table 6 · p. 14

• Task 01 is a crafting example involving three agents.Source: B Example Task Definitions · p. 15
• Task 76 is a combat example involving ten agents.Source: B Example Task Definitions · p. 15

B.1 Task 01: Cooperative Magic Staff Crafting

• Task 01 prepares a wizard with a crafted magic staff under a 25-step maximum; the lumberjack agent has lumberjacking 25, while the woodworker has fletching 15 and crafting 10.Source: §B.1 · p. 15
• The staff-crafting example includes a wizard with a bead, a fletcher with fletching skill, and a recipe that turns logs into sticks.Source: §B.1 · p. 15

B.2 Task 76: Elemental Bosses (10 agents)

• Task 76 sets a primary objective to defeat the Ice Guardian, Iron Ogre, and Water Guardian, with a maximum of 48 action steps.Source: §B.2 · p. 15
• The ten-agent combat setup includes an Ice team leader, DPS agents, and support, with seven more agents across mountain and water regions.Source: §B.2 · p. 16

C Task Annotation Platform

• The platform provides controls for configuring spawn locations, skill levels, inventories, and equipment.Annotators select map coordinates, adjust eight skills, browse 133 items, and configure equipment slots with a character preview.Source: C Task Annotation Platform · p. 16
• Annotators choose agent spawn locations by selecting coordinates on a world map.Source: C Task Annotation Platform · p. 16
• The searchable inventory browser offers 133 items, while equipment slots include a character preview.Source: C Task Annotation Platform · p. 16

D Augmented Task

• All sampled augmented variants preserved collaboration needs, objectives, and role asymmetry; 90% were judged appropriately difficult.The remaining difficulty judgments depended on whether a substituted enemy was weaker than the original.Source: D Augmented Task · p. 16
• Variants change task demands, spawn distances, or starting items without changing the original execution order.Examples include heavier crafting objectives, more distant spawns, and fewer initial items.Source: D Augmented Task · pp. 16–17

E.1 System Prompt

• All four models receive identical system and user prompts, with the system prompt requiring exactly one tool call per response.The system instructions also encourage action-oriented tool use and describe movement, resource gathering, communication, combat, and purposeful waiting.Source: E Agent Prompt · p. 17, §E.1 · pp. 17–18

E.2 User Prompt (Per-Task, Per-Agent)

• Each agent's user prompt supplies task objectives, relevant context, and team information through task-specific template variables.Agents receive their own username, teammate names, and game context relevant to their role.Source: §E.2 · p. 18

F.1.1 Background

• A trajectory contains actions by N agents across R rounds, including tool uses and chat messages, and identifies success actions as those directly achieving task criteria.Source: §F.1.1 · pp. 18–19

F.1.2 Causal Action Graph (CAG)

• The Causal Action Graph links actions when one causally enables another, with influence flowing forward in time.The graph is a directed acyclic graph because actions can only enable actions in the same or later rounds.Source: §F.1.2 · p. 19

F.1.3 Contributing Set

• The contributing set contains success actions and every action with a directed path to a success action.Actions outside this set are classified as non-contributing.Source: §F.1.3 · p. 19

F.1.4 Metrics

• CCE measures the fraction of all actions that contributed to task success, with failed tasks assigned zero.The metric treats every action equally; CCE near 1 indicates all actions contributed, while values near 1/N indicate only one agent contributed.Source: §F.1.4 · p. 19
• Per-Agent Contribution measures each agent's useful-action fraction, revealing whether success depends disproportionately on a subset of agents.High CCE alongside skewed PAC values can indicate that only some agents drove success.Source: §F.1.4 · p. 19

F.2 Backward Causal Tracing Algorithm

• Backward tracing starts from success actions, asks which prior actions enabled them, and computes CCE and PAC from the resulting contributing set.The procedure prunes candidate action pairs by relevance, batches queries, and applies a temporal window; each tool use or chat message counts as one equally weighted action.Source: §F.2 · p. 20

F.3 CCE Judge Prompts

• The judge prompts classify actions as contributing when they enable useful later actions through resources, movement, communication, or prerequisite steps.The prompt uses an inclusive rule: even somewhat helpful actions count as contributing.Source: §F.3 · p. 21

G Example Trajectories

• Example trajectories include a 20-agent, six-round Maritime Trading success and a 12-agent, nine-round Trade Network success with 19 transfers.Trajectory visualizations mark action types with distinct shapes, including chat, transfers, crafting, attacks, and harvest/move/sleep.Source: Figure 7 · p. 21, G Example Trajectories · p. 21, Figure 8 · p. 22