Source-linked AI summary
CooperBench: Why Coding Agents Cannot be Your Teammates Yet
Arpandeep Khatua, Hao Zhu, Peter Tran, Arya Prabhudesai, Frederic Sadrieh, Johann K. Lieberwirth, Xinkai Yu, Yicheng Fu, Michael J. Ryan, Jiaxin Pei, Diyi Yang
TL;DR
Current AI agents may possess coding competence without the coordination abilities needed for effective teamwork. The paper introduces CooperBench, a realistic benchmark of collaborative coding tasks, and finds a major cooperation deficit while identifying both recurring capability gaps and rare successful coordination patterns.
Problem
The paper asks whether strong individual coding capabilities translate into effective cooperation, a question that matters for deploying AI agents alongside humans or other agents.
Method
CooperBench evaluates agents on 652 tasks from 12 open-source libraries in four programming languages, pairing potentially conflicting features with expert-written tests and ground-truth solutions.
Results
25% success with two-agent cooperation is roughly half the success rate of a single agent performing the same workload, with failures involving communication, commitments, and expectations.
Takeaways & Limitations
Rare role division, resource division, and negotiation behaviors suggest coordination capability exists but must become more reliable.
Takeaways & Limitations
CooperBench does not compare different agent frameworks or communication tools because it focuses on foundation models’ intrinsic cooperation capability.
Abstract
from arXiv · showhide
Resolving team conflicts requires not only task-specific competence, but also social intelligence to find common ground and build consensus. As AI agents increasingly collaborate on complex work, they must develop coordination capabilities to function as effective teammates. Yet we hypothesize that current agents lack these capabilities. To test this, we introduce CooperBench, a benchmark of over 600 collaborative coding tasks across 12 libraries in 4 programming languages. Each task assigns two agents different features that can be implemented independently but may conflict without proper coordination. Tasks are grounded in real open-source repositories with expert-written tests. Evaluating state-of-the-art coding agents, we observe the curse of coordination: agents achieve on average 30% lower success rates when working together compared to performing both tasks individually. This contrasts sharply with human teams, where adding teammates typically improves productivity. Our analysis reveals three key issues: (1) communication channels become jammed with vague, ill-timed, and inaccurate messages; (2) even with effective communication, agents deviate from their commitments; and (3) agents often hold incorrect expectations about others' plans and communication. Through large-scale simulation, we also observe rare but interesting emergent coordination behavior including role division, resource division, and negotiation. Our research presents a novel benchmark for collaborative coding and calls for a shift from pursuing individual agent capability to developing social intelligence.
1. Introduction
CooperBench examines whether current AI agents can coordinate effectively on collaborative coding tasks. The benchmark finds substantially worse cooperative performance and identifies communication, commitment, and expectation failures alongside rare successful coordination behaviors.
- Current AI cooperation remains an open question because effective teamwork requires communicating intentions, understanding goals, and negotiating compatible solutions.
- CooperBench contains 652 collaborative coding tasks from 12 open-source libraries spanning Python, TypeScript, Go, and Rust.Tasks were designed by authors with software-engineering backgrounds using new features, unit tests, and ground-truth code.
- 25% success with two-agent cooperation is around 50% lower than the “Solo” baseline implementing both features.This result was observed for GPT-5- and Claude Sonnet 4.5-based agents on CooperBench.
- Agents exhibit three coordination failures: vague or mistimed communication, deviation from commitments, and incorrect expectations about partners’ plans and changes.
- Rare emergent behaviors that often accompany successful tasks include role division, resource division, and negotiation.The authors identify these categories through large-scale simulation.
- CooperBench provides an open benchmark for evaluating and improving cooperative coding agents.
2. CooperBench Benchmark
CooperBench constructs realistic, potentially conflicting coding tasks from open-source repositories and evaluates whether merged agent solutions remain compatible and correct. Its design emphasizes expert-grounded task construction, deterministic evaluation, and individually tractable features.
- CooperBench measures cooperation through realistic, potentially conflicting tasks with deterministic, interpretable evaluation.Its desiderata are realism, conflict potential, and verifiability.
- Task space: Each task pairs two features from a compatible feature pool, merges the agents’ patches, and evaluates both assigned features.Agents can only view their own features, creating opportunities for coordination failures.
- Features: Features represent missing functionality, bug fixes, or both, with manually written tests and ground-truth solutions withheld to prevent test leakage.
- Task composition: 77.3% of tasks have conflicting ground-truth solutions despite jointly implementable features.Compatibility is verified with a joint ground-truth solution that passes all individual unit tests.
- Action space: Agents use open-ended communication and local file and terminal operations within the controlled action space.GUI and browser-based actions are left for future extensions.
- Evaluation: Evaluation requires both compatible patches and implementation correctness.Semantic merge handling reduces the impact of superficial formatting conflicts, while merged unit tests assess correctness.
- Dataset Construction: The benchmark is built in three stages: selecting repositories and pull requests, creating adjacent feature pools, and validating all feature combinations.The pipeline uses 12 actively maintained repositories and controlled execution environments.
- CooperBench features are individually tractable and realistic, making coordination of partially overlapping implementations the primary challenge.
3. Experiment Settings
The experiments evaluate cooperation among coding agents using multiple models, frameworks, and communication settings. CooperBench is treated as a model-focused benchmark rather than a comparison of agent frameworks or communication tools.
- The study asks how state-of-the-art foundation models cooperate, use communication, and fail or succeed in coding tasks.
- The evaluation framework uses OpenHands v0.54 with two agents working independently in Docker-based containers and communicating through an SQL-backed tool.
- CooperBench does not bind evaluation to a particular agent framework or communication tool because the paper focuses on foundation models’ intrinsic cooperation capability.
- Five language models are evaluated: GPT-5, Claude 4.5 Sonnet, MiniMax-M2, Qwen3-Coder-30B-A3B-Instruct, and Qwen3-30B-A3B-Instruct-2507.
- The benchmark compares cooperative performance across models and task difficulty, including the Solo-Coop gap shown in Figure 4.
4. How well are agents able to cooperate with each other?
Agents perform substantially worse when coordinating on two features than when one agent handles both, revealing a coordination gap that is especially pronounced on moderately difficult tasks. Communication reduces merge conflicts but does not translate into higher cooperation success.
- Two-agent cooperation achieves success rates consistently below the Solo setting across models, with gaps reaching 50% for GPT-5, Claude Sonnet 4.5, and MiniMax M2.
- The coordination gap is larger and more significant for middle-level technical difficulties than for tasks that are extremely easy or difficult.
- Communication does not significantly improve cooperation success, although it substantially reduces naive merge conflicts across models.
- Scaling concurrent cooperation from two to four agents is proposed as a direct test of whether coordination overhead increases with team size.
5. What is the role of communication in agent-agent cooperation?
Agents use communication heavily, but communication improves spatial conflict avoidance more reliably than end-to-end cooperation. Effective patterns include early planning and specific references, while semantic misunderstandings and flawed messages continue to undermine compatible implementations.
- Communication is the only coordination channel, and the study compares ordinary communication with a no-communication baseline.
- Communication does not significantly improve cooperation success, even though agents actively use the communication tool.
- Communication significantly reduces merge conflicts for Claude Sonnet 4.5, GPT-5, MiniMax M2, and Qwen Instruct, while consuming up to 20% of action steps.
- Conflict-free trajectories have a Plan:Question ratio of 2.04 versus 1.31, indicating that questioning accompanies rather than resolves coordination problems.
- A first-turn Plan message nearly halves conflict rates from 51.5% to 29.4%, with a 39% reduction at the highest difficulty.
- Successful trajectories contain more concrete line-number and file-path references, helping agents avoid overlapping edits.
- Spatial coordination can prevent overlapping edits, but semantic coordination is still required to align implementation parameter values.
- Repetition, unresponsiveness, and hallucination make messages less actionable and jam the communication channel.
6. What are the coordination failures that the agents exhibit?
Agents exhibit coordination failures across communication, commitment, and expectation: they may exchange unusable information, break promises, or fail to model partners’ actions. Successful but rare behaviors instead use explicit role or resource boundaries and negotiation to make plans verifiable under partial observability.
- Expectation: Expectation failures occur when agents communicate plans but fail to integrate them into an accurate model of partner state.One agent announced changes depending on the other’s planned registry, yet proceeded as if that code would not exist.
- Commitment: Commitment failures occur when agents make unverifiable claims, ignore agreed integration points, or break explicit promises.A promised bypass check was absent after merging because the partner had no accessible signature or diff to verify the claim.
- Communication: Communication failures include unanswered questions, late messages, repetitive updates, and inaccurate information that prevent agents from establishing shared context.In one example, silence after a direct question left both agents proceeding under potentially incompatible assumptions.
- Underlying mechanism: Partial observability makes coordination difficult because agents act with uncertain models of partners’ states, edits, and commitments.Even a conflict-free merge can preserve incompatible assumptions, requiring checkable commitments and cross-branch integration reasoning.
- Emergent coordination behavior: Successful coordination rarely emerges through role division, resource division, and negotiation that create explicit, mutually understood boundaries.Effective negotiation proposes fully specified alternatives, while line-level resource boundaries create safe zones where conflicts are impossible.
- Emergent coordination behavior: These coordination patterns are rare, but successful traces suggest agents may possess the underlying capabilities without using them reliably.The paper frames the challenge as improving reliability rather than teaching entirely new coordination skills.
7. Related Work
Prior work studies multi-agent systems, coding agents, coordination benchmarks, and Theory of Mind, but often relies on engineered interaction structures or evaluates individual success. CooperBench instead isolates free-form coordination among coding agents under partial information and measures failures in integrating changes.
- Multi-agent LLM systems: Multi-agent frameworks commonly improve performance through conversation programming, simulated organizations, or explicit orchestration, while still exhibiting high failure rates.Examples include CAMEL, AutoGen, MetaGPT, ChatDev, Magentic-One, MAGIS, and AgileCoder.
- Tool-using coding agents: Tool-using coding-agent benchmarks such as SWE-bench primarily measure single-agent task success rather than peer integration under partial information.This leaves coordination between multiple coding agents insufficiently evaluated.
- Coordination benchmarks: Existing coordination benchmarks cover games, embodied tasks, reasoning, multi-agent collaboration, robotics, and software settings with varied coordination demands.The paper positions CooperBench within this broader benchmark landscape while targeting free-form coding coordination.
- Theory of Mind evaluation: Theory of Mind benchmarks evaluate whether language models represent partners’ beliefs and intentions, with studies reporting substantial gaps versus human performance.CooperBench connects this concern to coding-agent coordination without reducing coordination evaluation to Theory of Mind alone.
- CooperBench: CooperBench isolates free-form coordination by assigning partially overlapping features, isolating workspaces, restricting interaction to natural language, and evaluating integration failures.Its controlled setting targets redundancy, inconsistent assumptions, semantic breakage, and the broader social-intelligence bottleneck.
8. Conclusion and Future Work
CooperBench shows that coordination, rather than raw coding ability alone, remains a central bottleneck for agents working as teammates. Rare successful patterns suggest that training and verifiable coordination protocols may improve reliability, although the evidence is grounded in software development.
- 25% success with two-agent cooperation is roughly half the success rate of one agent performing the same workload.
- Three capability gaps underlie the coordination curse: ineffective communication, commitment deviations, and incorrect expectations about partners.
- Rare successful traces exhibit role division, resource division, and negotiation that convert vague intentions into verifiable commitments.
- The paper proposes coordination-focused training objectives, lightweight commitment protocols, and richer communication channels as future directions.
- The findings are presented as potentially generalizing beyond software development to domains with role and resource conflicts under partial observability.
A. Dataset Details
CooperBench is built from real repositories and compact, conflict-prone feature pairs designed for deterministic evaluation. Its construction emphasizes realistic teamwork, measurable compatibility, and implementation correctness while limiting raw feature complexity.
- The benchmark spans source repositories whose feature counts and task pairs are aggregated across base commits.
- 652 evaluated feature pairs comprise 199 individual features grouped into 52 task sets.
- Features are intentionally compact and comparable in difficulty so multi-agent failures primarily reflect coordination limitations rather than disproportionate complexity.
- Feature complexity is measured by changed lines, modified functions and hunks, and affected files.
- Merged-code evaluation uses a local learned resolver after standard merging to remove trivial conflict markers while preserving both sides’ intent.
C. Difficulty-stratified evaluation
The difficulty-stratified evaluation compares Solo and Coop success across normalized task difficulty, using confidence intervals and AUC-based retention to quantify coordination cost. Coordination losses are especially pronounced at mid difficulty and vary independently from coding strength.
- Tasks are partitioned into 10 equal-width buckets over normalized difficulty [0, 1], with Solo and Coop success rates computed at each midpoint.
- 95% Wilson confidence intervals accompany each bucketed success rate, including near the boundaries of the success-rate range.
- Retention = AUCCoop/AUCSolo normalizes coordination performance for baseline capability, while ΔAUC = AUCSolo − AUCCoop measures the absolute gap.
- The gap between Solo and Coop success is largest at mid-difficulty rather than on the easiest or hardest tasks.
- 41% of Solo capability is lost during coordination, corresponding to pooled retention of 0.59.
- MiniMax has the lowest retention at 0.46, whereas Qwen has the highest at 0.68 despite being the weakest coder.
- The evaluation prompt explicitly targets context misunderstanding, overlapping edits, and missing final status updates through mandatory coordination guidance.
E. Communication ablation
The communication ablation finds that communication slightly reduces raw conflicts for naive merging, but this advantage disappears after union and learned resolution. Its final effect on cooperation success is near zero or slightly negative.
- Communication does not improve final cooperation success after all merge-resolution steps.
- Communication slightly improves Naive merge rates by reducing raw conflicts, but the benefit disappears after Union and LLM resolution.
- The final communication effect is near zero or slightly negative across all models.
F. Communication error detection
The paper defines a precision-first process for detecting communication failures in multi-agent traces, using fine-grained categories and quoted conversational evidence. Case analyses show that vague, unanswered, repetitive, or unverifiable communication can produce incompatible code and failed tests.
- Evidence requirements: The detector requires strong evidence, including exact quotes, and omits an issue label when the conversation does not support one clearly.Questions require evidence of missing or irrelevant responses, while repetition requires two quotes showing duplication.
- Taxonomy: The taxonomy separates unresponsiveness, hallucinated code-state claims, and repetitive messages into verifiable subcategories.It distinguishes corrected from uncorrected false claims and repetition that consumes communication budget without adding information.
- Detection process: The six-stage process links failures in merged artifacts to interactions, develops symptom categories through qualitative coding, calibrates an LLM annotator, and reports the resulting analysis.The process includes collecting collaborative traces, identifying failures, resolving annotation disagreements, and finalizing the symptom set.
- Failure examples: Despite explicit boundaries, duplicate edits can preserve an incorrect version and cause failing GUID tests after merging.One agent added a section containing another agent’s work, and the merged regex omitted the required curly braces.
- Failure examples: A clean merge can still silently erase one agent’s additions, producing a missing exported API and an ImportError.The agents expected their changes to combine cleanly, but a class rewrite overwrote the region containing register_filter and unregister_filter.
- Failure analysis: Communication reduces merge conflicts but does not improve task success because agents prioritize spatial coordination while neglecting semantic coordination.The paper contrasts agreement about edit locations with agreement about values and behaviors, illustrated by failures involving repeated updates, unanswered decisions, unverifiable claims, and incompatible interfaces.
I.1. Task Setup
This section constructs collaborative coding tasks in which two agents extend the same function with parameters that are independently implementable but jointly require coordination. The case study shows that agents can coordinate edit locations while failing to coordinate semantic details, producing an incorrect implementation.
- Task Setup: Each task gives two agents features to implement in the same repository, with both features requiring changes to a shared function signature.The case study adds case_sensitive and reverse parameters to Jinja2’s groupby filter.
- Task Setup: The agents exchanged 10 messages totaling over 3,000 words, focusing almost entirely on line numbers, file paths, and edit ranges.Their planning identified the shared code region and established who would edit which portion.
- Task Setup: Spatial coordination succeeded, but semantic coordination failed because the agents never confirmed the case_sensitive default or how the parameters should interact.They independently added both parameters to avoid a Git conflict, while leaving correctness-relevant meanings unresolved.
- Task Setup: Agent 2 reported case_sensitive=True even though the correct default was False, and neither agent verified the other’s implementation.The intended False default matched the sort filter’s case-insensitive behavior, but status messages were treated as sufficient evidence.
- Task Setup: A correct merged signature would include case_sensitive: bool = False and reverse: bool = False alongside the existing parameters.The gold patches specify the False default for case_sensitive and the reverse parameter in the combined signature.
- Task Setup: One clarifying message specifying the False default and its rationale could have prevented the failure entirely.The message would have instructed the partner to use exactly case_sensitive: bool = False when including the parameter in the patch.