PaperlayerView exact revision on arXiv

Source-linked AI summary

When Sub-Agents Work in Parallel: The Promises and Pitfalls of Dynamic Concurrency in Long-Horizon Coding Tasks

Han Li, HanHaoNing Li, Ziqian Jiang, Yiling Lou

arXiv:2610.10263v1cs.SE

TL;DR

Existing evaluations provide limited evidence about dynamic concurrency in frontier coding agents on long-horizon tasks. This study compares native concurrent and sequential execution and analyzes trajectories; concurrency has mixed effects on task success and resource use, with gains concentrated in long-horizon settings and benefits under specific coordination conditions.

  • Problem

    Existing work focuses largely on single-agent systems, predefined collaboration protocols, or bounded tasks, leaving dynamic concurrency in frontier agents on long-horizon coding tasks insufficiently evaluated.

    Source: §1 · pp. 1–2

  • Method

    The study compares native dynamic-concurrency and sequential executions in frontier coding agents and analyzes their outcomes, coordination behavior, and trajectories.

    Source: §1 · p. 3

  • Results

    Resolution effects range from an 11.9 percentage point decrease to a 2.3 point increase, and mean token use reaches 1.41 to 3.31 times sequential mode; gains occur on long-horizon benchmarks, especially LoopsBench, but not on SWE-bench Verified.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Benefits arise when sub-agents complete complementary work, explore alternatives, or independently validate and correct the main agent’s implementation.

    Source: §1 · p. 3

    Findings are scoped to three native concurrency mechanisms across five benchmarks, and evaluation cost limits large-scale repeated runs despite stochastic execution.

    Source: §8.2 · p. 18

Abstract

from arXiv · show

As coding agents advance from bounded software engineering tasks toward long horizon development, dynamic concurrency offers a promising way to scale complex development tasks. Under this policy, agents decide during execution whether and how to spawn concurrent sub-agents. Model capability largely determines outcomes on shorter tasks, whereas long horizon development makes orchestration central to task completion. Existing work, focused on coding agent failures on shorter tasks or collaboration in predefined multiagent workflows, offers little insight into dynamic concurrency in frontier agents across task complexity. We study dynamic concurrency as an execution policy through controlled comparisons of matched Codex, Claude Code, and Kimi Code executions with the policy enabled or disabled. Across 354 tasks and 2,124 executions spanning a range of task complexities and execution horizons, we evaluate its end to end effects and scheduling behavior, and analyze matched trajectories to characterize 13 concurrency specific failure modes, 28 observable patterns, and the conditions under which it provides an advantage.

1 Introduction

Dynamic concurrency is presented as a flexible way to scale coding agents for long-horizon work, where tasks evolve and fixed workflows may not fit. The study compares native concurrency in frontier agents and finds mixed success-rate effects alongside identifiable coordination benefits and failure patterns.

Motivation

  • Long-horizon software development involves extensive exploration, implementation, testing, and verification, while a single agent faces context, reasoning, and execution limits.Multi-agent delegation may increase execution capacity, enable parallel exploration, and reduce end-to-end latency.Source: §1 · pp. 1–2

Dynamic concurrency

  • Dynamic concurrency lets the main agent decide during execution whether, when, and how to spawn sub-agents, then integrate and verify their work.Codex supports direct and nested delegation, Claude Code schedules parallel stages with dependencies, and Kimi Code launches task-specific batches.Source: §1 · p. 2

Research gap

  • Prior evaluations largely cover single agents, predefined collaboration protocols, or bounded tasks, leaving dynamic concurrency on long-horizon coding tasks unclear.Such tasks may involve generating tens of thousands of lines of code.Source: §1 · p. 2

Study findings

  • Resolution effects range from an 11.9 percentage point decrease to a 2.3 point increase, while mean token use reaches 1.41 to 3.31 times sequential mode.Pass-rate gains occur on long-horizon benchmarks, especially LoopsBench, but not on SWE-bench Verified; the taxonomy includes four categories, 13 subcategories, and 28 failure patterns.Source: §1 · p. 3

Contributions

  • The study contributes a systematic evaluation, trajectory-based failure analysis, and release of 2,124 trajectories, annotations, and analysis scripts.The analysis identifies benefits when sub-agents do complementary work, explore alternatives, or independently validate and correct the main agent’s implementation.Source: §1 · pp. 3–4

2 Background and Related Work

This section contrasts predefined multi-agent workflows with dynamic concurrency and situates the study among coding-agent evaluations. A seqtk reconstruction example illustrates how delegation can proceed in parallel yet stall before integration.

Concurrency in coding agents

  • Earlier systems coordinate agents through predefined roles, workflows, task graphs, discussion rounds, or layered aggregation.Examples include ChatDev and MetaGPT [22] [23] [50], CodeR [10], Multiagent Debate and SWE-Debate [18] [32] [52], and Mixture-of-Agents [56].Source: §2 · p. 4

Dynamic concurrency

  • Dynamic concurrency lets the main coding agent decide during execution whether to delegate, what to delegate, and how to coordinate and integrate results.The study examines Codex Multiagent Version 2, Claude Code Dynamic Workflow, and Kimi Code AgentSwarm.Source: §2 · p. 4

Empirical studies and example

  • Prior work analyzes coding-agent coordination and failures, but this study focuses on dynamic concurrency across challenging long-horizon coding tasks.The seqtk example concerns rebuilding a command-line tool from a reference executable and documentation; its graph represents sub-agents as nodes and the main agent as the root.Source: §2 · p. 4

Illustrative example

  • In the seqtk task, Kimi Code assigned 22 subcommands to 20 sub-agents, but waited for the full batch and never reached integration or final tests.The program passed 92 of 440 tests; the seq command was missing, and 18 agents were aborted at the deadline.Source: §2 · pp. 4–5

3 Methodology

The methodology frames four research questions and evaluates three native agent systems across sampled long-horizon and bounded coding tasks. Matched concurrent and sequential runs hold key experimental conditions constant and collect trajectories, artifacts, and test results.

3.1 Research Questions

  • RQ1–RQ4 ask about outcomes and resource use, delegation and coordination, concurrency failure patterns, and conditions for effectiveness or efficiency benefits.Source: §3.1 · p. 5

3.2 Studied Agents

  • The study pairs Codex with GPT-5.4, Claude Code with Claude Opus 5, and Kimi Code with Kimi K3, using each agent’s native scaffold and highest reasoning effort.The agent–model configurations were selected in July 2026.Source: §3.2 · p. 5

3.3 Benchmarks

  • Four long-horizon benchmarks cover package or interface reconstruction, program rebuilding, specification-to-repository construction, and sustained development; SWE-bench Verified represents bounded tasks.Stratified random sampling selects 25% of tasks from each long-horizon benchmark and 20% from SWE-bench Verified. Figure 3 shows implementation sizes across benchmarks and difficulty groups; LoopsBench includes tasks involving 26.8K lines of code.Source: §3.3 · pp. 5–6, Table 1 · p. 6, Figure 3 · p. 6

3.4 Experimental Process

  • Each task runs in concurrent and sequential modes with the backbone model, prompt, environment, and time budget controlled between modes.Concurrency uses each agent’s native interface and default settings; the researchers collect complete main-agent and sub-agent trajectories, submitted artifacts, and official test results.Source: §3.4 · pp. 6–7

4 Effectiveness and Efficiency (RQ1)

Across three agents and five benchmarks, dynamic concurrency has mixed effectiveness: average test-pass advantage grows with benchmark implementation size, while resource use and runtime generally rise. Faster completion occurs in a subset of tasks solved in both modes.

4.1 Effectiveness

  • LoopsBench shows the clearest gains: task pass rates rise up to 14.3 percentage points for Codex and Claude Code, while test pass rates improve up to 6.0 points for all agents.Source: §4.1 · p. 8
  • Across pooled task groups, Claude Code’s overall decline (p=5.08348 × 10^-7) and Kimi Code’s (p=0.0132176) are significant, while Codex’s improvement (p=0.229481) is not.Source: §4.1 · p. 8
  • Average test-pass difference (concurrent minus sequential) rises with benchmark implementation size: −8.29 percentage points on RepoZero, −0.04 on NL2Repo, +1.24 on ProgramBench, and +4.31 on LoopsBench.Source: §4.1 · p. 8
  • Overlap analysis counts ProgramBench as successful above zero test pass, but requires full resolution elsewhere; concurrent execution also yields unique successes across agents and benchmarks.Claude Code uniquely solves four LoopsBench tasks under concurrency; on SWE-bench Verified, it uniquely solves five despite a lower overall pass rate, while Codex has nine unique tasks in each mode.Source: §4.1 · p. 8, Figure 4 · p. 9

4.2 Resource Efficiency

  • Token consumption reaches 1.41–3.31× sequential levels, and mean runtime increases for all three agents and 14 of 15 agent-benchmark combinations.Codex uses 24.12 million recorded tokens per solved task under concurrency versus 7.69 million sequentially; Claude Code takes 27.8 versus 17.5 minutes per SWE-bench Verified task.Source: §4.2 · pp. 9–10
  • Among 366 tasks solved in both modes, concurrency finishes faster in 102 cases (27.9%) and at least 20% faster in 46 cases (12.6%).Source: §4.2 · p. 10

5 Delegation and Coordination Behavior (RQ2)

Across long-horizon tasks, agents generally delegate more as difficulty rises, but their delegation scale and timing differ. Codex starts early but sustains limited overlap, whereas Claude Code sustains the most overlap and Kimi Code the least.

5.1 Delegation Scale

  • Across equally weighted pooled executions, mean created sub-agents rose from 6.69 on easy tasks to 7.53 on medium and 15.84 on hard tasks, while mean concurrent peak rose from 3.66 to 4.61.Source: §5.1 · p. 10
  • Delegation does not rise uniformly: LoopsBench shows the largest easy-to-hard increase, while bounded SWE-bench tasks and some RepoZero settings are exceptions.Task structure and coordination can constrain independent subtasks even as total work grows; significant difficulty associations occur only in some tested settings.Source: §5.1 · p. 11
  • Codex generally keeps groups smaller, while on hard LoopsBench tasks Claude Code creates more agents overall and Kimi Code reaches a higher peak.Codex mean peaks range from 1.21 to 3.00 across task groups; on hard LoopsBench, Claude Code averages 133.69 created agents and a 25.46 peak, versus Kimi Code's 67.00 and 37.46.Source: §5.1 · p. 11

5.2 Temporal Dynamics and Context Overlap

  • Codex starts delegation earliest, at a median 4.0% of the trace, versus 22.1% for Claude Code and 26.0% for Kimi Code.The activity profiles show differences in delegation onset, sustained activity, and successive waves; their bands represent mean active sub-agents, with tasks equally weighted in the overall panel.Source: §5.2 · pp. 11–12, Figure 6 · p. 12
  • Claude Code sustains the most overlap, with a 52.7% median overlap fraction, compared with 23.9% for Codex and 15.6% for Kimi Code.Codex delegates early but often does not overlap sub-agents; its 24.3% of executions with sub-agents but no overlap contrasts with 1.7% for Claude Code and 2.8% for Kimi Code. Median last sub-agent endpoints are 83.3%, 100.0%, and 93.6% of the trace, respectively.Source: §5.2 · pp. 12–13

6 Taxonomy of Concurrency Failure Patterns (RQ3)

The authors manually compare concurrent trajectories with matched sequential runs, then organize observed coordination failures into a multi-level taxonomy. The resulting taxonomy contains four main categories, 13 subcategories, and 28 failure patterns.

6.1 Methodology

  • Annotators reviewed 1,062 concurrent trajectories alongside the same agent's sequential trajectory for the same task, outcomes, and official test results.Reviews included the main agent, nested sub-agents, workflow and status records, and generated artifacts.Source: §6.1 · p. 13
  • Two annotators developed an initial codebook from 50 trajectories, and three annotators independently double-coded cases using the resulting taxonomy.Independent failure-pattern annotations achieved Cohen's κ=0.77; disagreements were discussed and unresolved cases adjudicated by a senior author.Source: §6.1 · p. 13

6.2 Taxonomy

  • The taxonomy comprises four main categories, 13 subcategories, and 28 failure patterns, observed in 650 reviewed trajectories as 804 pattern instances.A trajectory may contain multiple patterns, so counts and percentages are computed over pattern instances; Table 3 presents the taxonomy.Source: §6.2 · p. 13, Table 3 · p. 14

C. Global Context Management Problems

Global context management failures arise when agents lack or lose the context needed to connect their work. The examples span omitted requirements, incompatible interfaces, overloaded handoffs, unchecked completion, and retries that fail to preserve progress.

C. Global Context Management Problems

  • Global context management accounts for 75 of 804 pattern instances, or 9.3%.The category includes failures involving requirements, interfaces, handoffs, completion checks, and retries.Source: D. Shared State and Merge Problems · p. 16
  • Missing task requirements leave delegated work incomplete: a package sub-agent returned an unchanged export list despite a required wildcard import.The artifact passed all tests, but the delegation and recorded checks did not show that the requirement had been explicitly handled.Source: D. Shared State and Merge Problems · p. 16
  • Missing interface contracts create incompatible components, as when sub-agents disagreed on return types and function signatures, producing nine initial test failures.The main agent had to reconcile the mismatches after integration.Source: D. Shared State and Merge Problems · p. 16
  • Overloaded handoffs can prompt repeated investigation rather than implementation: an 80K-character brief led a sub-agent to repeat environment and source probes before interruption.The required output file was not produced.Source: D. Shared State and Merge Problems · p. 16
  • Global completion can be declared with known requirements unchecked, while unchanged retries can repeat work without carrying forward useful progress.In one example the main agent ignored stderr and a known argument-parsing deviation; in another, four replacement attempts repeated probes without prior findings or a narrower next step.Source: D. Shared State and Merge Problems · p. 16

7 Benefit Analysis of Dynamic Concurrency (RQ4)

The matched trajectory analysis identifies three situations in which concurrency brings benefits: complementary parallel work, alternative attempts that can be selected or combined, and independent validation that corrects assumptions. In each, the main agent uses sub-agent output to improve progress or the final artifact.

7 Benefit Analysis of Dynamic Concurrency (RQ4)

  • 734.116 to 467.997 seconds: complementary work reduced wall time when sub-agents handled delivery files and checks while the main agent implemented the core algorithm.Both concurrent and sequential executions passed all tests, and the main agent integrated the eight sub-agent results before final verification.Source: §7 · p. 17
  • 292/391 tests passed with concurrency versus 0/391 sequentially in Halite, where the main agent combined a sub-agent's map behavior with its own protocol handling.In another example, the main agent compared concurrent implementations and adopted the 1.4.0 tables for NL2Repo unidecode.Source: §7 · p. 17
  • Independent validation can provide corrective evidence: a crypto sub-agent's P-256 curve-order finding helped the main agent generalize its ECDSA nonce-reuse recovery algorithm.The concurrent artifact passed all checks while the sequential mode failed; other examples describe sub-agent probes correcting assumptions about CSV headers.Source: §7 · p. 17

8 Discussion

The discussion recommends selective use of dynamic concurrency based on task characteristics and resource budgets, while noting that the evidence is scoped to the tested agents and benchmarks and limited by execution cost.

8.1 Implications

  • Enable dynamic concurrency selectively: it is more promising for difficult, long-horizon tasks, while simpler tasks may see limited or negative benefits.Potentially favorable settings include clearly decomposable work, parallel attempts, or independent validation, weighed against increased token use, tool calls, and runtime.Source: §8.1 · p. 18
  • Reliable concurrent coding calls for better task decomposition and load balancing, runtime monitoring and recovery, global context sharing, and shared-state coordination.These research directions address orchestration, execution governance, global context management, and shared-state coordination.Source: §8.1 · p. 18

8.2 Threats to Validity

  • The findings are scoped to three agents and five benchmarks, and evaluation cost limits large-scale repeated runs despite evidence that randomness is unlikely to alter overall trends.Manual failure analysis may also involve subjective judgment; the authors use a pilot codebook, independent annotations, discussion, and senior-author adjudication.Source: §8.2 · p. 18

9 Conclusion

Across five benchmarks, enabling concurrency increases token use while task-success effects vary by tool and task. The conclusion reports a taxonomy of failure patterns, mechanisms associated with observed advantages, and an artifact for reproducing reported figures and tables.

9 Conclusion

  • Across five benchmarks and 2,124 executions, concurrency increases token use while task-success effects vary across tools and tasks.The executions compare concurrent and sequential settings across three frontier coding agents.Source: §9 · p. 18
  • The failure taxonomy covers orchestration, execution governance, global context management, and shared state and merge; observed advantages involve complementary work, alternative attempts and selection, or independent validation.The main agent turns concurrent activity into progress by integrating components, selecting candidates, or applying corrective evidence.Source: §9 · p. 19

Data Availability

  • The artifact includes the execution manifest, evaluator summaries, trajectory records, qualitative codebook, review registers, and evidence for reported cases.Scripts regenerate numeric tables and figures from frozen machine-readable inputs; benchmark data and upstream repositories are referenced rather than redistributed where licenses require it.Source: Data Availability · p. 19
Loading 2610.10263v1…