Source-linked AI summary
Effective Strategies for Asynchronous Software Engineering Agents
Jiayi Geng, Graham Neubig
TL;DR
Long-horizon software engineering tasks remain difficult for single agents and for multi-agent systems sharing artifacts. CAID uses dependency-aware centralized delegation, asynchronous execution in isolated workspaces, and test-verified branch-and-merge integration. It consistently improves performance over single-agent baselines across Commit0 and PaperBench, while coordination adds cost and runtime and task decomposition remains a limitation.
Problem
Long-horizon tasks with interdependent subtasks challenge single-agent accuracy and completion time, while asynchronous collaboration struggles with interference, dependency synchronization, and coherent integration.
Method
CAID centrally models dependencies, delegates executable subtasks asynchronously to isolated git worktrees, and integrates verified changes through commits and merges.
Results
CAID consistently improves performance over single-agent baselines across Commit0 and PaperBench, including across multiple underlying models.
Takeaways & Limitations
Branch-and-merge coordination is a central mechanism for reliable multi-agent software engineering over shared artifacts.
Takeaways & Limitations
Multi-agent execution incurs higher API cost and does not substantially reduce wall-clock runtime because integration remains sequential and test-gated.
Abstract
from arXiv · showhide
AI agents have become increasingly capable at isolated software engineering (SWE) tasks such as resolving issues on Github. Yet long-horizon tasks involving multiple interdependent subtasks still pose challenges both with respect to accuracy, and with respect to timely completion. A natural approach to solving these long-horizon tasks in a timely manner is asynchronous multi-agent collaboration, where multiple agents work on different parts of the task at the same time. But effective application of multi-agent systems has proven surprisingly difficult: concurrent edits by multiple agents interfere with each other, dependencies are difficult to synchronize, and combining partial progress into a coherent whole is challenging. On the other hand, human developers have long relied on mature collaboration infrastructure to manage these challenges in large software projects. Inspired by these collaboration primitives, we introduce Centralized Asynchronous Isolated Delegation (CAID), a structured multi-agent coordination paradigm grounded in three core SWE primitives: centralized task delegation, asynchronous execution, and isolated workspaces. CAID constructs dependency-aware task plans through a central manager, executes subtasks concurrently in isolated workspaces, and consolidates progress via structured integration with executable test-based verification. In empirical evaluation, we find that CAID improves accuracy over single-agent baselines by 25.6% absolute on paper reproduction tasks (PaperBench) and 14.7% on Python library development tasks (Commit0). Through systematic analysis, we find that branch-and-merge is a central coordination mechanism for multi-agent collaboration, and that SWE primitives such as git worktree, git commit, and git merge enable it to be realized in a reliable and executable manner.
1 Introduction
Long-horizon software engineering tasks exceed the reliable scope and timely completion of single agents, while asynchronous collaboration introduces shared-artifact coordination problems. CAID addresses these problems by combining centralized delegation, isolated concurrent work, and structured integration with verification.
- Single agents remain limited on large tasks and require significant wall-clock time despite growing task-completion horizons.
- Asynchronous multi-agent systems must coordinate locally reasonable edits, dependencies, and partial progress over shared artifacts.
- Human software teams mitigate interference through isolated repository copies, explicit merge workflows, and automated test suites.
- CAID uses a manager to delegate tasks dynamically to engineers working concurrently in isolated git worktrees, integrating changes through git merge.Engineers also perform executable self-verification and conflict resolution at commit time, while structured JSON and commits replace free-form inter-agent dialogue.
- CAID is evaluated on Commit0 and PaperBench, where it consistently improves performance across multiple models.Commit0 requires Python-library implementation, while PaperBench evaluates conference-paper reproduction.
2 Branch-and-Merge Multi-Agent Coordination with SWE Primitives
CAID operationalizes multi-agent coordination with dependency modeling, centralized delegation, isolated workspaces, asynchronous execution, structured communication, and test-gated integration. Its dependency graph determines which work units become executable and when.
- CAID maps task decomposition, isolated development, integration, and verification to SWE primitives including dependency graphs, git worktrees, git merge, and test suites.
- Task Specification and Dependency Graph: The manager represents repository work as a dependency graph and delegates a unit only when all predecessor units are completed and integrated.Ready units satisfy Ready_t(v_j) ⇔ ∀(v_i, v_j) ∈ E, v_i ∈ C_t.
- Dependency-Aware Task Delegation: The manager converts dependencies into at most N executable task groups, activates only engineers whose dependencies are satisfied, and groups strongly dependent files together.
- Dependency-Aware Task Delegation: Task priority favors earlier test execution, stronger evaluation signals, and upstream dependency position, with dependency state updated after each merge.A round comprises delegation, implementation, and dependency update, continuing until no executable groups remain or limits are reached.
- Workspace Isolation and Integration: Each engineer works in a git worktree derived from a consistent main-branch state, while restricted shared files receive special handling.
- Structured Communication: Structured JSON specifies assignments, paths, target functions, and dependencies so task boundaries and outputs can be programmatically validated.
- Asynchronous Execution: An asynchronous manager-controlled event loop runs engineers concurrently, reassigns newly executable work, and preserves dependency state and unresolved errors in compressed history.
- Self-Verification and Termination Control: Engineers must run relevant tests or a fallback executable check, resolve failures, and submit only verified commits for integration.
3 Main Results
CAID is evaluated on long-horizon Python-library development and paper-reproduction benchmarks against controlled single-agent baselines. Branch-and-merge coordination improves performance across models, whereas simply extending iteration budgets is less effective and fallback execution adds cost and runtime.
- CAID is evaluated on two long-horizon software engineering benchmarks: Commit0 for Python-library implementation and PaperBench for conference-paper reproduction.Commit0-Lite is used as the primary Commit0 evaluation set, while PaperBench uses the Code-Dev protocol.
- The evaluation uses GLM 4.7, MiniMax 2.5, and Claude 4.5, with a same-framework single-agent baseline controlling for framework-level differences.
- CAID consistently outperforms single-agent execution across both benchmarks and all three evaluated models.The comparison uses branch-and-merge coordination under matched underlying models and fixed per-configuration iteration budgets.
- Branch-and-Merge Results: 36.1% versus 10.5%: MiniMax 2.5’s PaperBench score rises under multi-agent execution compared with its single-agent score.Claude 4.5 also rises from 57.2% to 63.3%, while Commit0-Lite Claude 4.5 rises from 53.1% to 59.1%.
- Strategy Implications: A single-agent fallback followed by multi-agent execution raises PaperBench Claude Sonnet 4.5 from 63.3% to 66.8%, but runtime rises from 2080.4s to 3883.9s and cost from 6.5 to 9.7.
- Iteration Budgets: Doubling a single agent’s iteration limit yields marginal or negative changes, while CAID’s PaperBench gain for MiniMax 2.5 exceeds 25 percentage points.
4 Analysis
The analysis shows that reliable multi-agent performance depends on isolation, task structure, and delegation quality rather than simply adding engineers. Coordination prompts also expose a trade-off between verification robustness and execution efficiency.
- 4.1 Git worktree Isolation: 59.1% on Commit0-Lite with worktree isolation exceeds 56.1% with soft isolation and 53.1% for the single-agent baseline.On PaperBench, worktree isolation reaches 63.3%, compared with 55.5% for soft isolation and 57.2% for single-agent execution.
- 4.2 Choosing the Degree of Parallel Execution: Increasing engineers from 2 to 4 improves Commit0-Lite performance, but expanding to 8 decreases it as integration and conflict-resolution costs rise.On PaperBench, adding engineers beyond 2 yields minimal score gains while runtime and cost increase steadily.
- 4.3 Delegation Shapes Execution Trajectory: Figure 4 compares file-level implementation intervals and manager phases across one single-agent run and two CAID runs, whose module assignments produce distinct trajectories and pass rates.The comparison links execution timelines to differences in delegated modules rather than merely agent count.
- 4.3 Delegation Shapes Execution Trajectory: 34.3% pass rate in CAID Run 2 exceeds 8.7% in Run 1 because the manager assigns sustained work to the critical autodiff.py dependency.The single-agent run reaches 17.4% after touching autodiff.py without completing it.
- 4.4 Scaling Asynchronous Parallelism: Figure 5 compares runtime and pass rate under round-manager review, engineer self-verification, and efficiency-prioritized coordination prompts.Efficiency emphasis can shorten runtime but may reduce integration robustness, whereas stricter review improves stability at additional computational cost.
5 Related Work
Related work spans role-based, hierarchical, conversational, and adaptive multi-agent coordination, but software engineering adds shared-artifact conflicts that require explicit artifact-level control. SWE benchmarks provide the evaluation context for autonomous coding agents.
- Multi-Agent Coordination: Prior multi-agent systems explore static roles, hierarchical delegation, verification, communication topologies, and task-adaptive orchestration.Examples include CAMEL, Generative Agents, ChatDev, EvoMAC, and other architectural approaches.
- Multi-Agent Coordination: Many systems rely on standardized procedures, conversational interaction, implicit co-player inference, or consensus-based evaluation to coordinate agents.These approaches primarily organize information flow and interaction rather than explicit control of shared software artifacts.
- Software Engineering Coordination: Software engineering contributes branching, merging, dependency management, continuous integration, and code review as explicit mechanisms for coordinating parallel work over versioned artifacts.These mechanisms have also influenced recent multi-agent frameworks.
- SWE Benchmarks: SWE-bench established autonomous issue resolution as a benchmark, while SWE-bench Verified refined evaluation fidelity and robustness for coding agents.SWE benchmarks evaluate autonomous development activities across complex codebases.
6 Limitations and Future Directions
The paper identifies cost, runtime, delegation quality, and domain generalization as limitations of structured multi-agent execution. Future work targets more efficient coordination, stronger delegation, and validation beyond software engineering.
- Cost and Runtime: Multi-agent execution consistently costs more than single-agent baselines, while parallelism does not substantially reduce wall-clock runtime.Sequential integration and test-gated verification limit end-to-end acceleration despite concurrent engineering.
- Cost and Runtime: Future work should improve scheduling efficiency, reduce redundant verification cycles, and learn when to merge or prune intermediate states.The paper frames the cost–performance frontier of structured multi-agent execution as an open problem.
- Isolated Task Delegation Capabilities of Agents: Task assignment currently relies primarily on prompt-engineering heuristics, so coarse or dependency-misaligned decomposition can produce locally correct outputs that integrate inefficiently.Proposed directions include learned delegation policies, dependency-aware planning, and adaptive subtask refinement.
- Generalization Beyond Software Engineering Tasks: CAID is evaluated on software engineering benchmarks whose explicit workspaces, version control, and executable tests may not exist in other long-horizon shared-artifact domains.Applying CAID to document synthesis, research planning, or multimodal artifacts requires adapted isolation, integration, and validation mechanisms.
7 Conclusion
CAID structures long-horizon software engineering through dependency-aware delegation, isolated branches and worktrees, controlled git integration, and executable tests. Across Commit0 and PaperBench, it improves over single-agent baselines, while iteration increases and fallback switching do not reliably help.
- CAID uses a manager, dependency-aware task units, isolated engineer worktrees, git commit and merge, and tests to coordinate long-horizon tasks.
- CAID consistently improves over single-agent baselines across Commit0 and PaperBench, even with the same underlying model.
- Increasing the single-agent iteration budget does not reliably improve outcomes, while switching from a single agent to multi-agent execution mainly wastes runtime and cost.
- Branch-and-merge and SWE primitives provide a practical structure for parallel, coordinated development on complex dependency-aware tasks.
A Prompt Engineering for Multi-Agent Task Delegation
The prompts operationalize centralized delegation for Commit0 and PaperBench by assigning dependency-aware work, preserving repository structure, verifying changes, and prioritizing core reproduction results under limited time.
- A.1 Commit0 Prompts user instruction: The manager delegates implementation work through structured JSON assignments specifying agents, task identifiers, files, functions, complexity, instructions, and dependencies.
- A.1 Commit0 Prompts user instruction: Engineers run the specified unit tests, verify implementation quality, and submit local git commits for integration after completing their assignments.
- A.1 Commit0 Prompts user instruction: After completed work, the manager assigns the next highest-priority file while checking missing functions, dependencies, and overall implementation progress.
- A.1 Commit0 Prompts user instruction: Delegation divides work into balanced major tasks, keeps highly interdependent files together, and prefers file-level splits with non-overlapping assignments.
- A.1 Commit0 Prompts user instruction: Engineers implement functions with pass statements, preserve existing names and formatting, and avoid changing or commenting out existing code.
- A.2 PaperBench Prompts user instruction: For PaperBench, the manager delegates reproduction tasks, summarizes the paper for engineers, and prioritizes core contributions and results within limited time.
- A.2 PaperBench Prompts user instruction: PaperBench reproduction receives partial credit for correctly implemented parts, weighted by their importance to the paper’s main contributions and results.
- A.2 PaperBench Prompts user instruction: Appendix-only experiments are excluded from the core contributions considered in the reproduction task.
B Full Results
The full-results section reports Commit0-Lite and PaperBench outcomes for each repository or paper across three language models, with configuration-specific results presented in tables.
- Commit0-Lite and PaperBench are evaluated across three LLMs, with full results reported for every repository and paper.
- Table 4 presents Claude 4.5 Sonnet results on Commit0-Lite across different configurations.
C One-sided t-test
One-sided paired t-tests evaluate whether CAID exceeds the single-agent baseline across repositories or papers for each model. Results are significant for all three PaperBench models, but Commit0-Lite significance varies by model and repository variance.
- One-sided paired t-tests test whether CAID exceeds Single-Agent across all repositories or papers for each model.
- On Commit0-Lite, Claude Sonnet 4.5 and MiniMax 2.5 show significant improvements, while GLM 4.7’s 3.6-point mean gain is not significant.
- On PaperBench, CAID’s improvements are significant for Claude Sonnet 4.5, GLM 4.7, and MiniMax 2.5.
- Commit0-Lite results vary across repositories, including a +30.7 gain on simpy and a −10.5 regression on tinydb for CAID.
- CAID’s effectiveness depends on accurate dependency graphs and delegation, while weaker models produce less reliable decomposition on open-ended PaperBench tasks.
D Failure on Scaling the Parallel Execution
Scaling parallel execution can reduce performance when delegation becomes less disciplined. Clean, non-overlapping assignments support stable integration, whereas overlapping work within shared files increases integration risk.
- Scaling comparison: The simpy execution timelines compare CAID runs using N = 2, 4, and 8 engineers.The comparison illustrates why increasing the number of engineers does not always improve execution.
- N = 4: 92.1% pass rate is reached when four engineers receive distinct, largely non-overlapping files.The manager avoids assigning closely coupled modules simultaneously, and no two engineers work on the same file.
- N = 8: 44.3% performance results when eight engineers are assigned overlapping functions within the same file.Competing updates increase merge-conflict and inconsistent-state risks, limiting clean consolidation.
- N = 8: The N = 8 degradation reflects delegation that ignores file-level ownership boundaries, not excessive parallelism alone.More parallel activity does not translate into stable global progress when task partitioning is incoherent.