PaperlayerView exact revision on arXiv

Source-linked AI summary

The Collaboration Gap: Exploration and Benchmarking of Open-World Agentic Cooperation

Tim R. Davidson, Adam Fourney, Saleema Amershi, Robert West, Eric Horvitz, Ece Kamar

arXiv:2511.02687v2cs.AIcs.CL

TL;DR

Language-model agents may need to collaborate despite having different information, capabilities, and tools, yet evidence on unconstrained agent collaboration is limited. The paper evaluates collaboration with a maze benchmark across 32 models and finds that strong solo performance can degrade in teamwork. It also reports that relay inference—having a stronger model lead before handing off—can improve joint outcomes.

  • Problem

    Evidence is limited on how solo performance relates to ad hoc collaboration in partially observed, multi-agent tasks, despite the importance of collaboration among agents with differing information and capabilities.

    Source: §1 · p. 1

  • Method

    The authors evaluate 32 language models with a maze benchmark that isolates collaboration and permits unconstrained dialogue, using transcript grading to assess outcomes.

    Source: §1 · p. 2

  • Results

    The paper reports a collaboration gap: strong solo models can show a performance collapse in teamwork, while relay inference can reliably improve joint outcomes.

    Source: §5 · p. 9, §6 · p. 10

  • Takeaways & Limitations

    The authors conclude that collaboration is a distinct capability that current training strategies insufficiently capture and that interaction order and priming can matter.

    Source: §1 · p. 3, §5 · p. 9

    The unstructured rollouts require language-model grading, which may introduce nondeterministic noise or model-specific extraction bias.

    Source: Reproducibility statement · p. 11

Abstract

from arXiv · show

The trajectory of AI development suggests that we will increasingly rely on agent-based systems powered by language models, composed of independently developed agents with different information, privileges, and tools. The success of these systems will depend critically on effective collaboration among these heterogeneous agents, even under partial observability. Despite intense interest, the literature lacks empirical studies evaluating agentic collaboration without relying on fixed communication protocols, limiting insights for open-world deployments. We propose an illustrative collaborative maze-solving benchmark that (i) isolates collaborative capabilities, (ii) modulates problem complexity, (iii) enables scalable automated grading, and (iv) imposes no output-format constraints, benchmarking unguided, natural communication. Using this benchmark, we evaluate 32 leading open- and closed-source models in solo, homogeneous, and heterogeneous pairings. Our results reveal a surprising collaboration gap: models that perform well solo often degrade substantially when required to collaborate. We identify mitigations that show remarkable influence. For example, a small nudge via a relay inference approach, where a stronger agent leads before handing off to a weaker one, closes much of the gap. Our findings argue for (1) collaboration-aware evaluation, (2) training strategies to enhance collaborative capabilities, and (3) deliberate interaction design to elicit agents' collaboration skills, principles relevant to AI-AI and human-AI settings where agents must establish common ground.

1 Introduction

The paper asks whether language-model agents can collaborate effectively in open-world settings, where agents may have different information, capabilities, tools, and privileges. It introduces a maze benchmark and reports a collaboration gap, effects of interaction order, and relay inference as a mitigation.

1 Introduction

  • Fixed protocols and centralized orchestration constrain open-world integration because protocols cannot feasibly cover every situation or be enforced across independently controlled agents.The authors argue that successful open-world deployment therefore relies on flexible, on-the-fly communication.Source: §1 · p. 1
  • Most benchmarks emphasize static reasoning rather than long-horizon, multi-agent collaboration in partially observed environments, leaving the relation between solo performance and ad hoc collaboration uncertain.The paper frames targeted evaluation as necessary to examine that question.Source: §1 · p. 2
  • Across 32 leading open- and closed-source LMs, highly capable solo models show a statistically reliable performance drop when collaborating with an identical copy, especially distilled models.The paper terms this phenomenon the “collaboration gap.”Source: §1 · p. 2
  • Task performance in heterogeneous collaboration is heavily influenced by which agent initiates the dialogue.This motivates studying interaction order as part of collaboration rather than treating pairings alone as sufficient.Source: §1 · pp. 2–3
  • The benchmark uses maze copies with cells hidden from each agent, followed by dialogue and transcript grading against the ground-truth maze.The system diagram describes the sequence from maze generation to outcome checking.Source: Figure 2.1 · p. 3
  • Relay inference has a stronger model seed a task before handing it to a weaker model, substantially boosting the weaker model’s collaborative performance.The authors present the strategy as a way to prime or ground the initial steps of a task.Source: §1 · p. 3

2 Measuring agents’ collaborative capabilities

The benchmark uses partially observed mazes to separate maze-solving ability from collaboration, while allowing agents to communicate without a prescribed protocol. Its task also surfaces grounding, conflict resolution, and turn-taking as challenges for collaborative agents.

2.1 Collaborative maze solving

  • Mazes support comparison of solo and team performance, allow correctness to be verified, and can scale in complexity through grid size and wall density.The same maze can be described in different ways, requiring agents to align representations.Source: §2.1 · p. 3
  • Each agent receives a maze copy with roughly half the cells obfuscated, while the two copies together recover the full map.A solo agent can also be given both distributed maps to measure performance with distributed information absent collaboration.Source: §2.1 · p. 3
  • Agents exchange natural-language dialogue under two rules—agree before each move and execute one move at a time—with no other required communication protocol or output format.They see only their own maze copy, the rules, and the dialogue so far; a grader extracts agreed moves from the transcript and checks possible maze schemes against the original maze.Source: §2.1 · pp. 3–4
  • Automated grading makes large-scale experiments feasible, and sensitivity analyses found limited variance across graders’ average outcomes and virtually no evidence of model-specific bias.The study contrasts this scale with human collaboration studies to date.Source: §2.1 · p. 4

2.2 Communication challenges

  • Agents must ground their partial views and agree on coordinate and action meanings; mismatched row-column conventions can make the same coordinates mean different cells.They also need to resolve perceptual conflicts about cells or current location and determine who proposes the next move, potentially deferring according to capability.Source: §2.2 · pp. 4–5

3 Experimental setup

The experiments compare solo, homogeneous, heterogeneous, and relay-inference settings using maze tasks with controlled difficulty. The setup varies agent pairing and turn order, and reports both binary success and a weighted outcome.

Experimental settings

  • Main experiments use 6×6 mazes with 30% wall density and average solution paths of 7 to 9 steps, chosen so most models can solve them solo.The setup treats this as a meaningful baseline for collaboration.Source: §3 · p. 5
  • Solo baselines test a fully visible map and distributed information, while homogeneous collaboration pairs an agent with an independent copy of itself.Solo agents may review a proposed solution in a critic step; homogeneous runs use 100 rollouts per agent and a 50-turn maximum.Source: §3 · p. 5
  • Heterogeneous pairings examine ordering effects across models from selected providers, with at least 50 rollouts per pairing and a 50-turn maximum.The study includes in-family and selected cross-family pairings.Source: §3 · p. 5

Relay inference

  • Relay inference freezes the first 2, 4, 6, or 8 turns of AB or BB rollouts, then swaps in the weaker or stronger model to complete the task.At least 100 rollouts are run per model pairing and relay point.Source: §3 · p. 5

Outcome metrics

  • The study reports binary success for a correct start-to-goal path and a weighted outcome based on optimal path length relative to distance from the last valid position to the goal.The weighted outcome can be negative and is used as a continuous evaluation signal.Source: §3 · p. 5

4 Results

Across solo and collaborative 6×6 maze settings, results show broad solo capability but a reliable collaboration drop. Pairings also vary with order, model family, and how interactions begin.

4 Results

  • Virtually all studied models show a statistically reliable performance drop in homogeneous collaboration versus solo solving.The paper reports that context or turn limits are unlikely to explain the degradation.Source: Figure 4.1 · p. 6, §4.1 · p. 6

4.1 Solo baselines versus homogeneous collaboration

  • Half the models reach near-perfect solve rates with full visibility, while distributed information significantly reduces performance for about a third.Distilled models appear disproportionately affected by the solo-to-collaboration drop.Source: §4.1 · p. 6

4.2.1 Same model family: working with models of different strength

  • For gpt-4.1-mini paired with o3, performance is markedly better when o3 starts than when gpt-4.1-mini starts (0.77 ± 0.04 vs. 0.62 ± 0.06).When gpt-4.1-mini starts and o3 moves second, o3 partially imitates its communication format and opts for a greedy local strategy.Source: §4.2.1 · p. 7

4.2 Heterogeneous collaboration

  • Heterogeneous pairings show significant ordering and cross-family effects; weaker models can perform worse with a stronger partner than with same-family copies.For example, gemini-2.5-flash-lite and gpt-4.1-mini each perform better with copies of themselves than when combined.Source: Figure 4.3 · p. 7, §4.2.2 · p. 8

4.2.2 Different model family: working with models of varying strength

  • In everyday flagship pairings, grok-3-mini keeps performance close to its high homogeneous baseline and challenges incorrect moves rather than deferring.In the described exchange, it recalibrates shared map information and seeks agreement after gemini-2.5-flash disputes a proposed move.Source: §4.2.2 · p. 8
  • Homogeneous pairs encounter lower odds of conflict than heterogeneous pairs (OR = 0.60), but conditional on conflict, strategy or grounding disputes are similarly likely (OR = 0.97).Strong models are more likely than medium-strength reference models to trigger non-factual disputes over strategy or grounding.Source: §4.2.2 · p. 8

4.3 Relay inference

  • A single opening message from o3 can substantially boost weaker models, and strong-model priming appears more effective than strong-model recovery after weak-model exchanges.o3 sees only an incomplete map, so its opening grounds the coordinate schema and reference frame rather than proposing a full solution; late intervention has diminishing returns.Source: §4.3 · pp. 8–9, Figure 4.6 · p. 9

5 Discussion

The discussion argues that agent collaboration must be treated as a distinct capability, especially as agents become more specialized. It highlights interaction design and training while cautioning that maze findings do not establish the size of the gap in real-world settings.

5 Discussion

  • Specialization increases the need for agents to collaborate across capability gaps, while naively dividing work among current models could introduce collaborative slippage.The authors argue that collaborative capabilities should be explicitly trained rather than assumed to emerge from specialization.Source: §5 · p. 9
  • Priming may improve collaboration in weaker models, whereas performance is subdued when weaker models lead.The authors connect this interaction-order finding to human–AI settings, where humans typically lead.Source: §5 · p. 9
  • The discussion connects collaboration to people’s competence in interacting with AI and to an iterative process of identifying users’ needs before solving them.It frames information queries as dynamic processes rather than single events.Source: §5 · p. 10
  • Maze environments capture important aspects of collaboration but are not a proxy for all collaboration; a wider real-world gap is conjectured, not established.The authors point to split-context software engineering as a possible future task for their distributed-information approach.Source: §5 · p. 10

6 Conclusion

The conclusion presents the collaboration gap as a roadblock for language-model agents and argues that collaboration should be treated as a capability to design for. It reports that interaction order matters and that relay inference can improve joint outcomes.

6 Conclusion

  • High-solo-capability agents show a sharp performance collapse in simple teamwork, a collaboration gap demonstrated across leading language models.The conclusion describes the finding as a blind spot in current training paradigms, observed even in a simple testbed.Source: §6 · p. 10
  • Strategically priming interactions through relay inference can reliably improve joint outcomes, and interaction order is a critical factor.The conclusion characterizes relay inference as a promising insight.Source: §6 · p. 10
  • The authors call for collaborative intelligence to be treated as a core objective to design for, rather than an emergent property to expect.They frame the work as an opportunity to invest in addressing collaboration gaps and boosting AI capabilities.Source: §6 · p. 10

Reproducibility statement

The authors share experiment code and prompts and describe their implementation, while reporting evaluation and replication constraints. They also test grading robustness and document steps taken to address rollout variability.

Reproducibility statement

  • They share the experiment code and prompts in Appendix B, alongside implementation considerations.The code is available at the cited GitHub repository.Source: Reproducibility statement · p. 11
  • Closed-model opacity leaves open whether prompts clashed with developer instructions and disadvantaged some models; Cohere’s command-a sometimes prematurely signaled success, and instructions could not be revised at scale.The authors report making best efforts to ensure models understood the instructions and followed the message protocol.Source: Reproducibility statement · p. 11
  • For non-deterministic APIs, they collected at least 100 homogeneous and 50 heterogeneous rollouts, reported 95% confidence intervals, and fixed maze-generation seeds.The authors say this captures average tendencies but does not guarantee metrics represent every possible rollout.Source: Reproducibility statement · p. 11
  • They used gpt-4.1 to grade unstructured rollouts; ablations with repeated grading, o3, and gemini-2.5-flash found no statistically significant grading noise or model bias.The authors note that grader non-determinism and potential provider-related bias were possible concerns.Source: Reproducibility statement · p. 11
  • At least one maze schema escaped solution normalization, although manual author checks did not suggest the issue was widespread.Source: Reproducibility statement · p. 11
  • Replicating the study may be prohibitively costly, due to frontier-model prices and rising input context across turns; average context length is K · (T −1)/2 for message length K and maximum turns T.Source: Reproducibility statement · p. 11

Appendices

The appendices situate the benchmark against collaboration approaches built around fixed protocols, co-optimized agents, and ad hoc teamwork. They also document the model set and the prompts and message-handling choices used to run solo and collaborative experiments.

Rational agents and system-wide protocols

  • Earlier protocol-based approaches rely on designer-specified communication rules, while the benchmark targets collaboration without a pre-agreed protocol.The related-work review traces rigid communication languages and system-wide protocols, noting their transfer limits across heterogeneous agents and unanticipated environments.Source: Rational agents and system-wide… · p. 20

Reinforcement learning agents

  • RL approaches co-optimize communication for particular environments, whereas this evaluation tests zero-shot collaboration with heterogeneous models using flexible natural language.The authors distinguish this setting from co-optimized, task-specific communication vectors and protocols.Source: Reinforcement learning agents · p. 21

Ad hoc teamwork

  • Ad hoc teamwork studies collaboration with unknown partners, and this benchmark extends that setting to rich, unstructured natural-language interaction without shared training or prior coordination.The authors frame the benchmark as an empirical test of whether modern LMs have acquired ad hoc teamwork as an emergent capability.Source: Ad hoc teamwork · p. 21

Language model agents

  • Existing LM collaboration benchmarks commonly impose structured protocols, which can conflate communication overhead with task difficulty; this benchmark removes format constraints and varies maze complexity.The authors report evaluations across 32 leading open- and closed-source models.Source: Language model agents · p. 22

B.1 Models

  • The model comparison table records builder, public weight availability, distillation status, and system-prompt flexibility.The appendix defines “Open” as weights being publicly available for private deployment.Source: Table 1 · p. 23

B.2 Facilitating agent-to-agent communication

  • To support dialogue across API providers, the study uses explicit prefixes to distinguish the user-intermediary from the other agent in the conversation.Messages are converted between User and Assistant roles after the first message, which remains a User message.Source: §B.2 · p. 24

B.3 Solo prompts; B.4 Collaboration prompt; B.5 Verification prompts

  • Solo prompts include distributed map information and a critic step, while collaboration prompts specify interaction rules and the completion phrase “ACTI!.”The appendix also provides separate verification prompts for solo and collaboration experiments.Source: Figure B.2 · p. 25, Figure B.3 · p. 25, Figure B.4 · p. 26

C Maze hyperparameter ablations

The appendices report maze-parameter ablations, model-specific observations, and an exploratory analysis of behaviors associated with the collaboration gap. They also describe limits on the scope and interpretation of these additional results.

C.1 Varying maze size

  • The maze-size ablation is presented as weighted outcomes, but the supplied figure caption gives no numerical comparisons.Figure C.1 identifies the ablation scope as maze size.Source: Figure C.1 · p. 29

C.2 Varying wall density

  • Wall-density ablations show declining performance for all three models in the distributed setting as density increases.For solo full mazes, gpt-4.1 and gpt-4.1-mini vary most at densities 0.30–0.45, while gpt-4.1-nano declines monotonically as density increases.Source: Figure C.2 · p. 30

D Additional results

  • The appendix defines collaborative efficiency as median tokens per message multiplied by messages per solution, but leaves open-weight heterogeneous pairings for future work.Because “thinking” tokens are not always exposed, the metric reflects information-sharing rather than computational efficiency.Source: D Additional results · p. 31

D.1 Qualitative observations on specific models

  • The qualitative observations identify model-specific behaviors, including command-a/r premature completion claims, gpt-5 strength at larger mazes, and small models’ limited map sharing.The authors caution that command-a/r results may be a lower bound; gpt-5 solved full and distributed mazes up to 12 × 12 perfectly, while many smaller models explored locally instead of sharing maps.Source: §D.1 · pp. 31–32

D.2 Quantifying collaboration mechanisms

  • The mechanism analysis annotates 50 rollouts for each of 53 interaction pairs using two language models, covering 13 models across four providers.The analysis is described as a starting point rather than an exhaustive account; LM annotation may introduce correlated measurement noise, and the analyses are observational.Source: §D.2 · p. 32

D.2.1 Grounding cascades predict outcome success

The mechanism analysis associates grounding with global planning and better weighted outcomes, while finding that the strategy’s benefit depends on grounding. It also examines disagreement patterns and communication efficiency, with observational and measurement caveats.

D.2.1 Grounding cascades predict outcome success

  • 0.75 ± 0.01 is the average weighted outcome for global strategies, compared with 0.47 ± 0.01 for local strategies.Global strategies were also more likely when stronger models participated, and require at least one agent to have access to both maps.Source: §D.2.1 · p. 32
  • 38.4% of the total grounding effect on weighted outcomes is estimated to be mediated through global strategy in the additive model.Grounding score is associated with global strategy (a ≈ 0.69, p < 0.001), and global strategy with weighted outcome (b ≈ 0.17, p < 0.001); the indirect effect is ≈ 0.12 (CI: [0.07, 0.16]).Source: §D.2.1 · p. 33, Table 2 · p. 34
  • At perfect grounding, the estimated global-strategy boost reaches ≈ 0.24, while planning without grounding provides no performance benefit.The synergy coefficient is ≈ 0.28 (p < 0.001); the authors caution that these results are observational and should be read as predictive patterns, not causal mechanisms.Source: §D.2.1 · pp. 33–35

D.2.2 When and what do agents disagree on?

  • In clustered bootstrapping (B = 1000), homogeneous collaborations had lower odds of any conflict (OR = 0.60, p < 0.05) and factual disagreement (OR = 0.66); Google starters nearly doubled factual-dispute odds (OR = 1.91).Precise odds ratios for non-factual conflicts should be interpreted with caution.Source: §D.2.2 · p. 36
  • Strong followers are more likely than medium followers to trigger non-factual disputes (OR = 1.61), while homogeneous pairs are no less likely to have such disputes once a conflict occurs (OR = 0.97).The authors report that shared model identity primarily reduces how often conflicts arise, not what they concern.Source: §D.2.2 · p. 36

D.3 Weighted outcome; D.4 Binary success rate

  • The weighted-outcome and binary-success tables report mean collaborative performance with 95% confidence intervals, without numerical results in the supplied passages.The captions identify these as collaborative performance comparisons.Source: Table 4 · p. 37, Table 5 · p. 38

D.5 Communication efficiency

  • Collaborations typically finish within 20–40 messages, with median message lengths under 100 tokens, and failures rarely hit the 50-turn timeout.The authors argue that dialogue context lengths remain within modern models’ capacity, weakening increased context length as a potential confounder.Source: §D.5 · p. 39

Median token usage

  • The token-efficiency figure reports means with 95% confidence intervals, but the supplied passage gives no values to compare.The caption identifies the metric as median token efficiency.Source: Figure D.4 · p. 39

E Grading ablation

The grading ablations assess consistency within and across grader models on solo and homogeneous-collaboration outcomes. Graders show high agreement overall, though repeated scoring modestly compresses weighted outcomes and some disagreement remains.

E Grading ablation

  • The ablation samples solo and homogeneous-collaboration outcomes, collects multiple independent grader evaluations, and assesses agreement, correlations, voting, conditional disagreement, and model-specific effects.Solo sampling stratifies by maze visibility and success; homogeneous-collaboration sampling stratifies by success, with sampling without replacement.Source: E Grading ablation · p. 41

E.1 Intra-model consistency

  • ICC≃0.99 and Fleiss’ Kappa≃0.99 across independent gpt-4.1 grader runs support its consistency for solo grading.Repeated runs show a small, symmetric regression-to-the-mean bias in weighted outcomes conditioned on binary outcome.Source: Solo · pp. 41–42, Table 6 · p. 42

Homogeneous collaboration

  • ICC≃0.89 and Fleiss’ Kappa≃0.87 indicate high grader agreement on homogeneous-collaboration outcomes.Repeated scoring compresses weighted scores toward the mean; agreement is high for most models, with slightly more disagreement for command-a and gpt-5.Source: Homogeneous collaboration · p. 42, Table 7 · p. 43

E.2 Inter-model consistency

  • ICC≃0.88 and Fleiss’ Kappa≃0.91 across three graders suggest changing the grader is unlikely to significantly change average solo outcomes.Success cases had 9% disagreement versus 3% for failures, a difference not significant after FDR correction (p = 0.02 →0.06).Source: Solo · p. 42, Table 8 · p. 43
  • ICC≃0.84 and Fleiss’ Kappa≃0.77; heterogeneous graders’ majority votes align almost perfectly with the reference, with statistically significant disagreement but a small effect (d≃0.06).Disagreement is concentrated in success cases; weighted-outcome differences are not significant, and manual inspection found the flagged gpt-5 binary failures were incorrect.Source: Homogeneous collaboration · pp. 42–43, Table 9 · p. 44

F.1 Weak mirroring strong

  • gpt-4.1-mini adopts o3’s row-column coordinates and confirms the symbols’ meaning, illustrating weak-model imitation of a stronger model.Source: Figure F.1 · p. 45

F.2 Conflict resolution

  • In one exchange, gemini-2.5-flash misreads its map, and gpt-4.1 accepts its incorrect correction while still proposing caution.Source: Figure F.2 · p. 46
  • In another exchange, a gemini-2.5-flash agent correctly identifies an illegal jump in its partner’s proposed path.Source: Figure F.3 · p. 46

F.3 Meta analysis

The appendix illustrates agents recognizing fatal errors and, in one case, backtracking to resume progress toward the goal.

F.3 Meta analysis

  • Two gemini-2.5-pro agents recognize that they made a fatal error.Source: Figure F.4 · p. 47
  • Two claude-sonnet-4 agents recognize that one of them made a fatal error.Source: Figure F.5 · p. 47

F.4 Backtracking

  • Two claude-opus-4.1 models realize they are stuck and backtrack to an earlier position to continue toward the goal.Source: Figure F.6 · p. 47

G Error analysis

The error analysis reports that unanticipated valid strategies are rare and do not drive the macro-level collaboration gap, while parser assumptions can misgrade some exchanges.

G Error analysis

  • Unanticipated valid strategies occur, but are rare and do not drive the macro-level collaboration gap.Source: G Error analysis · p. 48

G.1 Unanticipated maze schema

G.2 Agents decide to split up

  • The parser grades claude-sonnet-4 agents’ decision to split up as a failure, although splitting up is not forbidden and offers no practical benefit in this game.Source: Figure G.2 · p. 48, §G.2 · p. 48
Loading 2511.02687v2…