PaperlayerView exact revision on arXiv

Source-linked AI summary

AsynCodeBench: Benchmarking Collaboration of Asynchronous Multi-Agent Systems in Software Engineering

Kaituo Zhang, Zhen Xiong, Zhimeng Jiang, Mingyu Zhong, Zhouyuan Yuan, Zhecheng Li, Bowen Lin, Chia-Yuan Chang, Mingzhi Hu, Huazheng Wang, Ying Lin

arXiv:2609.32662v1cs.SEcs.AIcs.MA

TL;DR

Multi-agent software engineering lacks direct measures of collaboration, while task-level outcomes can conflate coding ability with cross-agent coordination. AsynCodeBench evaluates asynchronous collaboration by tracking dependencies and their resolution over execution. Its experiments show that coding gains do not necessarily bring stronger collaboration, and task-level metrics can diverge from dependency-level measures.

  • Problem

    Multi-agent evaluations largely use task-level outcomes that conflate individual coding ability with cross-agent coordination and do not directly measure collaboration.

    Source: §1 · p. 2

  • Method

    AsynCodeBench tracks cross-component dependency resolution during asynchronous execution, measuring outcomes with ADPR and timing with DRS.

    Source: §1 · p. 2

  • Results

    Coding improvements do not necessarily improve collaboration, and task-level outcomes can diverge substantially from dependency-level measures.

    Source: §1 · p. 2, §5.2 · p. 6

  • Takeaways & Limitations

    Dependency-aware evaluation provides a complementary view of multi-agent software engineering beyond final task success.

    Source: §6 · p. 10

Abstract

from arXiv · show

Multi-agent coding has emerged as an increasingly active direction in software engineering, where complex development tasks are decomposed across multiple specialized agents working on different parts of the problem. Despite the shift from individual problem solving to distributed collaboration, multi-agent systems still lack a direct measure of collaboration and are largely evaluated through task-level outcomes inherited from single-agent coding, conflating individual coding capability with cross-agent coordination. We introduce AsynCodeBench, a dependency-centric benchmark for asynchronous multi-agent software engineering that represents each task with an explicit dependency graph and executable Dependency Checkers. Through this dependency-tracking process, we propose two complementary measures: Asynchronous Dependency Pass Rate (ADPR), which measures how many cross-agent dependencies are ultimately satisfied, and Dependency Resolution Step (DRS), which measures when each dependency first becomes satisfied during execution. AsynCodeBench comprises 19 tasks from real-world repositories, exposing 52 directed dependencies as explicit units for evaluating cross-agent collaboration. Experiments across model families, scales, and generations reveal a clear gap between coding and collaboration capability: improvements in coding performance do not necessarily translate into stronger collaboration, and task-level metrics can diverge substantially from dependency-level collaboration measures. Dependency-trajectory analysis further reveals that successful coordination often emerges not gradually, but through concentrated bursts in which many dependencies become resolved over a short portion of the execution trajectory, a pattern we term a hopping window.

1 INTRODUCTION

Multi-agent coding requires coordination across independently produced components, but evaluations based on final task outcomes do not directly measure that collaboration. AsynCodeBench tracks dependency resolution and timing, revealing that coding performance and collaboration can diverge.

1 INTRODUCTION

  • Task-level evaluations can conflate coding ability with coordination and hide unresolved dependencies across independently implemented components.Existing repository benchmarks and multi-agent systems commonly report final outcomes or gains over single-agent baselines.Source: §1 · p. 2
  • AsynCodeBench measures collaboration through cross-component dependencies, using ADPR for resolution outcomes and DRS for resolution timing.This makes dependency resolution observable throughout execution.Source: §1 · p. 2
  • Coding improvements do not uniformly yield collaboration gains: within the evaluated Qwen family, stronger single-agent coding does not improve every aspect of collaboration.Multi-agent execution is also not uniformly stronger than single-agent execution; dependency-aware evaluation can produce different quality–cost assessments from test-based metrics.Source: §1 · p. 2
  • Successful coordination often appears in concentrated phases, with the timing of these dependency-resolution bursts varying across models and execution protocols.The paper terms this recurring pattern a hopping phenomenon.Source: §1 · p. 2

2 RELATED WORK

Repository-level benchmarks preserve real-codebase challenges but still largely reduce execution to final task completion. Multi-agent software engineering research has expanded its workflows, while evaluation continues to rely mainly on end-task performance rather than direct measures of collaboration.

2.1 REPOSITORY-LEVEL CODING AGENTS AND BENCHMARKS

  • Repository-level coding benchmarks use real repositories and executable validation to retain challenges such as cross-file reasoning, dependencies, feedback, and consistency across interacting changes.Tasks have broadened from feature and library implementation to software evolution and repository-scale transformation, while intermediate progress and failure dynamics remain comparatively under-characterized.Source: §2.1 · p. 2

2.2 MULTI-AGENT SOFTWARE ENGINEERING

  • Multi-agent software engineering has moved toward repository-scale specialization, delegation, integration, and concurrent execution, but evaluation still largely measures end-task performance rather than collaboration itself.AsynCodeBench traces cross-component dependencies to expose failed handoffs and coordination dynamics that final task performance can hide.Source: §2.2 · p. 3

3 BENCHMARK CONSTRUCTION

AsynCodeBench models asynchronous collaboration through cross-agent dependencies and executable checkers, then tracks whether and when dependencies are resolved. It pairs dependency-level measures with repository-task curation and human validation.

3.1 PROBLEM DEFINITION

  • Asynchronous collaboration lets agents work independently without necessarily waiting for dependent agents to finish or propagate updates; a dependency links correctness across their implementations.The benchmark makes these dependencies explicit and tracks their resolution across integration checkpoints.Source: §3.1 · p. 3

3.2 DEPENDENCY GRAPH AND EXECUTABLE CHECKERS

  • Each task uses a directed graph linking upstream producers to downstream consumers, with pre-registered executable checkers for producer, consumer, and integrated dependency behavior.The integrated checker determines whether the dependency holds after integration; the other checker groups diagnose its two sides.Source: §3.2 · p. 4
  • Binary dependency states across ordered integration checkpoints form trajectories that distinguish unresolved, delayed, and later-regressing dependencies.The trajectory retains information that a final-workspace evaluation loses.Source: §3.2 · p. 4

3.3 EVALUATION METRICS

  • Test Pass Rate gives proportional credit for passed repository test points but does not directly assess dependencies; ADPR measures the fraction of benchmark-defined dependencies satisfied after integration.ADPR supplies a dependency-level counterpart to repository-level testing.Source: §3.3 · p. 4
  • DRS records the first checkpoint when a dependency passes, normalizes for runs with different checkpoint counts, and assigns 1 to non-resolution.Smaller values mean earlier resolution; DRS uses logical checkpoints, not wall-clock time, and SINGLE is excluded because it exposes only the final workspace.Source: §3.3 · pp. 4–5

3.4 TASK CURATION AND HUMAN VALIDATION

  • The benchmark curates 19 tasks from 16 repositories across five engineering areas, packaging each with a fixed revision, scoped paths, task instruction, and pytest evaluator.All execution protocols use the same task package, OpenHands harness, and task-specific Docker environment.Source: §3.4 · p. 5
  • Three independent reviewers validate each dependency's validity, direction, and executable evidence, retaining only validated dependencies.Source: §3.4 · p. 5

4 EVALUATION PROTOCOLS

The five protocols progress from independent task solving to increasingly structured collaboration. Their contrasts vary specialist handoffs, asynchronous access, and the manager’s ability to intervene.

4 EVALUATION PROTOCOLS

  • The five protocols span independent full-task solving, specialist handoffs, private concurrent work, and manager-mediated coordination with or without bounded repair.SINGLE assigns the whole task to one agent; the manager variants add coordination first without write access and then with bounded repair.Source: §4 · p. 5
  • ASYNC-PRIVATE isolates asynchronous execution by letting specialists work concurrently from the same initial repository state without seeing one another’s in-flight changes.SERIAL instead starts each downstream specialist only after the corresponding upstream artifact is integrated.Source: §4 · p. 5
  • ASYNC-MANAGER-RO coordinates integration through artifact inspection and feedback, while ASYNC-MANAGER additionally permits bounded repair after specialist integration.The contrast separates information exchange from direct code intervention.Source: §4 · p. 5

5 EXPERIMENTS

The experiments assess collaboration across ten open-weight models and four research questions, comparing task performance, dependency resolution, resolution timing, and token-cost trade-offs.

5.2 COLLABORATION GAINS AND HIDDEN DEPENDENCY FAILURES

  • Six of ten models improve under their best multi-agent protocol by 7.8–22.7%, while four remain below SINGLE.For Qwen3.8-27B and DeepSeek-V4, Test Pass Rate falls from 93.6% under SINGLE to 86.7% and 89.9%, respectively, under their best multi-agent protocol.Source: §5.2 · p. 6
  • Test Pass Rate exceeds ADPR in 136 of 190 outcomes (71.6%); 32 outcomes have at least 80% Test Pass Rate but below 50% ADPR.Of those 32 outcomes, 26 have zero ADPR; the mean Test Pass Rate is 48.0%, compared with 18.8% mean ADPR.Source: §5.2 · p. 6, Figure 2 · p. 7

5.3 MODEL CAPABILITY: QWEN SCALE AND GENERATION

  • Scaling Qwen3.5 from 9B to 27B raises Test Pass Rate under every protocol, but ADPR changes depend strongly on the protocol.Test Pass Rate rises from 24.4% to 42.1% under SINGLE and from 36.9% to 59.2% under ASYNC-MANAGER; ADPR shifts from 14.2% to 12.3% and from 2.8% to 31.6%, respectively.Source: §5.3 · p. 6, Figure 3 · p. 7
  • At 27B, Qwen3.5-to-Qwen3.8 gains in SINGLE ADPR are not matched under ASYNC-MANAGER, and similar final ADPR can mask different resolution timing.Under ASYNC-MANAGER, ADPR rises from 31.6% for Qwen3.5 to 63.2% for Qwen3.6 and 64.9% for Qwen3.8; normalized DRS is 0.641 for Qwen3.6 versus 0.719 for Qwen3.8.Source: §5.3 · pp. 6–7, Figure 3 · p. 7

5.4 PROTOCOL EFFECTS AND DEPENDENCY CLOSURE DYNAMICS

  • No collaborative protocol consistently improves ADPR across models; Qwen3.6-27B ranges from a 14.9% decrease to a 26.3% improvement, a 41.2% spread.The comparison is each collaborative setting against SINGLE under the same model.Source: §5.4 · p. 8, Figure 4 · p. 9
  • Across collaborative protocols, closure often concentrates in 0.50 < x ≤0.75, but its onset and duration vary by model and protocol.This interval contains 34.9%–67.8% of eventual closure mass across protocols. Of 40 model–protocol configurations, 28 show closure growth in this interval, including all 16 reaching at least 20% final ADPR; Qwen3-Coder-Next has an earlier burst around x ≈0.4.Source: §5.4 · p. 8, Figure 4 · p. 9, Figure 5 · p. 9

5.5 QUALITY–COST TRADE-OFFS

  • Across ten models, five have different within-model Pareto frontiers under ADPR and Test Pass Rate, so quality–cost comparisons depend on the quality metric.Nemotron-3.5’s Async-Private improves Test Pass Rate from 28.99% to 32.94% over Single at the same 1.05% ADPR, while Async-Manager-RO reaches 2.81% ADPR.Source: §5.5 · pp. 8–9, Figure 6 · p. 9

6 CONCLUSION

AsynCodeBench makes cross-agent collaboration and its evolution measurable through repository-grounded dependencies, revealing a gap between standalone coding capability and collaborative performance.

6 CONCLUSION

  • Task-level outcomes can hide unresolved coordination failures, so dependency dynamics provide a complementary view of multi-agent software engineering.The benchmark aims to support coding agents effective both individually and collaboratively.Source: §6 · p. 10

AI USE STATEMENT

The authors used ChatGPT for language, literature, ideation, methodology feedback, and code assistance, while retaining responsibility for the paper.

AI USE STATEMENT

  • ChatGPT assisted with manuscript polishing, literature retrieval, idea refinement, methodology feedback, and code development and debugging.The authors independently reviewed and validated methodological choices, benchmark construction, configurations, results, and conclusions, and take full responsibility for the paper.Source: AI USE STATEMENT · p. 10

A.1 TASK-LEVEL DEPENDENCY GRAPHS

The appendix details how AsynCodeBench’s dependency graphs, tasks, review process, execution protocols, checkpoints, and model set are constructed and controlled.

A.1 TASK-LEVEL DEPENDENCY GRAPHS

  • Dependency graphs link repository-grounded producer–consumer implementation units only when consumer correctness depends on producer behavior, state, interfaces, or artifacts.Edges are verified against repository evidence, manually reviewed, and frozen before model evaluation; the benchmark identifies nine dependency-relation types.Source: §A.1 · p. 15, Figure 7 · p. 15

A.2 TASK INVENTORY AND PROVENANCE

  • The benchmark contains 19 tasks from 16 repositories, with 64 implementation units, 52 directed edges, and 95 task–protocol scenarios.Its 52 edges use 156 checker groups and 508 selector references, which deduplicate within tasks to 274 unique selectors.Source: §A.2 · pp. 15–16

A.3 HUMAN REVIEW DETAILS

  • Three reviewers independently assess task inclusion, parallelizability, and rationale, and verify each dependency edge’s validity, direction, and repository evidence.Disagreements are discussed, discrepancies with the dependency graph are recorded, and Table 2 illustrates the review sheet.Source: §A.3 · pp. 16–17, Table 2 · p. 17

B PROTOCOL DETAILS

  • The five protocols hold the repository, instruction, model configuration, evaluator, and checkers fixed while varying scheduling, visibility, management, and manager patch authority.They range from single-agent ownership and sequential specialists to concurrent private worktrees, read-only management, and bounded manager interventions; these are controlled contrasts, not a presumed capability ranking.Source: B PROTOCOL DETAILS · pp. 17–18

C INTEGRATED-WORKSPACE CHECKPOINT CONSTRUCTION

  • Dependency trajectories retain integrated-workspace observations after integrations, accepted manager interventions, and final evaluation, excluding private artifact checkpoints.Checkpoint indices mark run-specific logical progress, not turns, tool calls, wall-clock time, or fixed computation; SINGLE is excluded from temporal DRS analysis.Source: C INTEGRATED-WORKSPACE CHECKPOINT… · p. 18

D EVALUATED MODEL CHECKPOINTS

  • The evaluated model set reports total and per-token activated parameters separately because MoE models may activate substantially fewer parameters than their total size.For dense models, activated parameters equal total language-model parameters; FP8 and BF16 weight formats do not change parameter counts.Source: D EVALUATED MODEL CHECKPOINTS · p. 19, Table 4 · p. 19

E TEST PROGRESS AND DEPENDENCY RESOLUTION ACROSS PROTOCOLS

Across protocols, task-level test progress often exceeds dependency closure, and this mismatch persists in both categorical and continuous comparisons.

E TEST PROGRESS AND DEPENDENCY RESOLUTION ACROSS PROTOCOLS

  • The comparison covers four protocols in Figure 10, with 190 model–task outcomes per panel from ten models and 19 tasks.Panels use identical axes and visual encodings for direct comparison.Source: E TEST PROGRESS AND DEPENDENCY… · p. 19
  • 22.1–27.4% of outcomes are high-test, low-ADPR under every protocol, while low-test, high-ADPR outcomes are absent in four protocols and number only three under ASYNC-MANAGER.The 50% threshold is a descriptive visual partition.Source: E TEST PROGRESS AND DEPENDENCY… · p. 19, Table 5 · p. 20
  • 62.6–77.9% of outcomes have Test Pass Rate above ADPR, with mean signed gaps of 23.7–29.2% across protocols.Test Pass Rate and ADPR remain positively associated, with Pearson correlations of 0.599–0.716 and Spearman correlations of 0.559–0.737; 12.6–16.8% of outcomes reach at least 80% Test Pass Rate while resolving fewer than half of dependencies.Source: E TEST PROGRESS AND DEPENDENCY… · p. 20, Table 6 · p. 20

F.1 AGGREGATE CLOSURE CONCENTRATION AND NORMALIZATION

Aggregate dependency closures cluster within a relatively short segment of execution, but normalization sharpens rather than creates the pattern. Model-level trajectories show that the timing and magnitude of concentrated closure vary across models and protocols.

F.1 AGGREGATE CLOSURE CONCENTRATION AND NORMALIZATION

  • 34.9%–67.8% of eventual closure mass falls within 0.50 < x ≤ 0.75 across the four protocols, with SERIAL reaching 67.8%.For SERIAL, cumulative closure rises from 4.9% at x = 0.50 to 20.3% at x = 0.75.Source: §F.1 · p. 20, Table 7 · p. 21
  • 38.4%–60.2% of eventual closures still occur at checkpoint k = 2 across protocols, although normalization concentrates 50.0%–73.9% of regional first closures at k = 2, T = 3.The shared ratio can sharpen the aggregate peak, but the raw-checkpoint concentration remains.Source: §F.1 · p. 21, Table 8 · p. 21
  • The interval 0.50 < x ≤ 0.75 is a representative aggregate concentration region, not a fixed timing boundary for every model–protocol configuration.The normalized position is x = k/T, where k is the first integrated checkpoint of closure and T is the run’s total integrated checkpoints.Source: §F.1 · p. 21

F.2 MODEL-LEVEL VARIATION IN HOPPING DYNAMICS

  • Low-closure configurations can remain largely flat, while substantial-closure configurations show bursts whose onset and magnitude vary across models and protocols.ASYNC-MANAGER begins accumulating dependencies earlier for Qwen3.5-27B and Qwen3.6-27B, while Qwen3.8-27B concentrates closure later.Source: §F.2 · pp. 21–22

G ADDITIONAL QUALITY–COST ANALYSIS

Quality–cost frontiers can differ depending on whether quality is measured by ADPR or Test Pass Rate. A case study shows how a high test score can coexist with unresolved dependencies and a shared coordination bottleneck.

G ADDITIONAL QUALITY–COST ANALYSIS

  • Five of ten models have different ADPR and Test Pass Rate frontiers, with two ADPR-only and four Test-Pass-only configurations.The comparison holds protocols and token costs fixed while changing the quality metric.Source: G ADDITIONAL QUALITY–COST ANALYSIS · p. 22
  • For Qwen3.5-9B, Async-Manager enters the Test Pass Rate frontier while collaborative protocols are dominated under ADPR; GLM-4.7-Flash shows the opposite pattern for Serial.Gemma-4-26B and Muse-Glimmer-30B also have Serial on the Test Pass Rate frontier but not the ADPR frontier.Source: G ADDITIONAL QUALITY–COST ANALYSIS · p. 22
  • Metric-dependent frontiers show that test-performance gains do not necessarily translate into dependency-resolution gains, and vice versa.The same protocols can be non-dominated under one quality metric but not the other.Source: G ADDITIONAL QUALITY–COST ANALYSIS · p. 22

H CASE STUDY: WHEN HIGH TEST PASS RATE MASKS UNRESOLVED DEPENDENCIES

  • DeepSeek-V4 under ASYNC-MANAGER on TVM-20073 passes 24 of 28 evaluator items (85.7%) but resolves 0/2 dependencies (ADPR 0%).Both upstream requirements become satisfied, but the downstream span-propagation logic remains unsatisfied after two attempts.Source: H CASE STUDY: WHEN HIGH TEST PASS RATE… · pp. 22–23
  • Four failed evaluator items trace to one missing span-propagation behavior, while ADPR identifies the two unresolved dependencies behind those failures.The required upstream capabilities were available but were not successfully composed in the downstream implementation.Source: H CASE STUDY: WHEN HIGH TEST PASS RATE… · p. 23
  • Dependency trajectories expose whether cross-agent dependencies are established and when, information that the final test outcome alone does not show.Test Pass Rate summarizes repository correctness over evaluator items; dependency-aware evaluation surfaces coordination structure.Source: H CASE STUDY: WHEN HIGH TEST PASS RATE… · p. 23

I DETAILED RESULTS OF ALL MODELS

The detailed results report multiple task, dependency, timing, and cost measures across models and protocols. Additional figures compare metric relationships, closure trajectories, Pareto frontiers, and a case where test performance masks unresolved dependencies.

Detailed results

  • Tables 10 and 11 report Test Pass Rate, ADPR, normalized DRS, and mean token consumption across ten models, five protocols, and the same 19 tasks.Lower DRS indicates earlier dependency resolution; DRS is not reported for Single, which has no intermediate integration checkpoints.Source: I DETAILED RESULTS OF ALL MODELS · p. 23, Table 10 · p. 24, Table 11 · p. 29
  • Table 12 counts successful tasks out of 19, requiring both a complete task pass and full dependency resolution (ADPR = 1).The table reports counts with corresponding percentages.Source: I DETAILED RESULTS OF ALL MODELS · p. 23, Table 12 · p. 30
  • Only ordered integrated-workspace outcomes at trajectory checkpoints form the dependency trajectory used by ADPR and DRS.Checkpoints follow integration attempts, accepted state-changing manager interventions when applicable, and final evaluation; artifact-checker outcomes are for local diagnostics.Source: Figure 9 · p. 25
  • Figure 10 compares Test Pass Rate with ADPR, while Figure 11 encodes each model–protocol cell’s share of eventual first closures in the hopping window.Figure 10 uses a dashed equality line and 50% boundaries; Figure 11 marks no observed closure as N/A.Source: Figure 10 · p. 26, Figure 11 · p. 27
  • Figure 12 plots cumulative dependencies resolved over normalized execution for seven models, with crosses marking final ADPR and shading the representative concentration region.The caption notes that the onset and magnitude of concentrated closure vary across models and protocols.Source: Figure 12 · p. 27
  • Figure 13 compares within-model ADPR and Test Pass Rate Pareto frontiers, distinguishing protocols by marker shape and retaining dominated configurations.Its axes are independently scaled across models.Source: Figure 13 · p. 28
  • DeepSeek-V4 under ASYNC-MANAGER on TVM-20073 reaches an 85.7% Test Pass Rate while ADPR remains 0%.The shared downstream span-propagation logic remains unsatisfied after two attempts, leaving both dependencies unresolved.Source: Figure 14 · p. 28

Related papers

Loading 2609.32662v1…