PaperlayerView exact revision on arXiv

Source-linked AI summary

Passes Alone, Fails Together: Benchmarking Semantic Coordination in Parallel LLM-Agent Development

Haocheng Xia, Eugene Wu, Yongjoo Park

arXiv:2609.25396v1cs.CLcs.AIcs.SE

TL;DR

Parallel coding agents can produce patches that pass alone but fail together when one changes an interface or rule another relies on. The paper benchmarks these interaction failures using the same tests on patches alone and in combination, across synthetic tasks, mined pull requests, and constructed Django tasks. Constructed tasks show frequent interference and substantial recovery with a message, while the paper cautions that these rates do not estimate real-world prevalence.

  • Problem

    Parallel patches can pass individually yet fail together when one changes an interface or rule another relies on, making semantic coordination across concurrent changes a research problem.

    Source: §1 · p. 1

  • Method

    stale tests the same patches alone and together with a common test set, comparing blind work with agents that see a completed concurrent change across three benchmark tiers.

    Source: §1 · p. 2

  • Results

    Constructed Django tasks showed 97% interference and 82% recovery with an oracle message, while mined reviewed pull requests rarely retained unresolved interactions.

    Source: §6 · p. 4

  • Takeaways & Limitations

    The constructed tasks demonstrate the failure mechanism on real Django helpers, but their rates do not establish how often it occurs in everyday parallel development.

    Source: §6 · p. 4

    The agent experiments are limited to Django and do not estimate the frequency or severity of failures in everyday parallel development.

    Source: §4 · p. 4

Abstract

from arXiv · show

Parallel coding agents can produce patches that work alone but fail when merged. This happens when one agent changes an interface or rule that another agent still relies on. We study these failures with stale, a benchmark for semantic coordination. Our evaluation runs the same tests on each patch alone and on their combination, counting only failures introduced by combining the patches. We use three tiers: synthetic tasks with controlled interface changes, pairs of merged pull requests, and constructed tasks that use real Django helpers. Among 834 runs on 417 mined Django pairs, only one showed interference after correcting the grading procedure. On constructed tasks using 12 Django helpers, interference occurred in 97% of runs. A message describing the completed concurrent change recovered 82% of runs. Reviewed pull requests may contain few unresolved parallel changes, even when agents fail on controlled tasks using real code. The constructed failure rates do not estimate how often these problems occur in practice.

1 Introduction

Parallel patches can each work alone yet fail together when one changes an interface or invariant another relies on. The paper defines this consistency problem as semantic coordination and introduces a benchmark to measure interaction failures and the benefit of seeing a concurrent change.

1 Introduction

  • Semantic coordination is maintaining consistency across concurrent, coupled changes, where individually passing patches can fail in combination.The benchmark counts only tests that pass with each patch alone and fail with both patches together.Source: §1 · pp. 1–2
  • A caching change can make a separate bulk-update path return a stale price when that path bypasses cache invalidation.The example illustrates a changed invariant: every price update must invalidate the corresponding cache.Source: §1 · p. 1
  • A clean merge does not ensure behavioral correctness: renamed helpers, new required arguments, or changed rules can break combined code.Diagnosing such failures may require understanding both patches.Source: §1 · p. 1
  • The benchmark compares blind agents with an oracle setting in which an agent sees the other worker’s completed edit.This estimates coordination headroom under ideal visibility; the same test set is used to distinguish interaction failures from grading differences.Source: §1 · p. 2
  • The study combines synthetic interface changes, mined pull-request pairs, and constructed changes to real Django helpers.Mined pull requests may have already resolved conflicts, while controlled tasks expose the failure mechanism.Source: §1 · p. 2

2 Benchmark Design

The benchmark isolates semantic interference by comparing fixed patches on the same tests alone and merged, across controlled and real-code tiers.

Evaluation settings

  • Agents solve tasks alone, independently on a shared base before merging, or sequentially with visibility into earlier edits.The sequential informed condition serves as an ideal-visibility baseline.Source: §2 · p. 2

Grading

  • The evaluation applies the same combined hidden-test suite to patches alone and together, reusing solo patches in the blind merge.Agent test-file edits are discarded and human gold test patches are applied before grading.Source: §2 · p. 2

Interference metric

  • Δblind counts tests that pass with every patch alone but fail after merging, while Iblind indicates whether that count is greater than zero.The count depends on test coverage and test-set composition; eligible pairs must pass each task’s tests during combined-suite grading.Source: §2 · p. 2

Benchmark tiers

  • The three tiers test interference through controlled interface changes, reviewed pull-request pairs, and scripted changes to real Django helpers.The tiers respectively test effects of stale information, unresolved interference in repository history, and whether the mechanism occurs in existing code.Source: §2 · p. 2

Communication

  • The communication condition gives an agent a templated oracle message describing the other worker’s completed change, not a message generated during ongoing work.The benchmark summary covers two model families and more than 4,000 graded task solves.Source: §2 · p. 2, Table 1 · p. 3

3 Dataset Construction

The dataset pipeline mines related pull requests, checks whether their patches can be evaluated from a valid shared base, and retains a subset as coupled instances.

Repositories

  • The deterministic construction procedure is used primarily with Django and also with SymPy, xarray, and seaborn.Django was selected for its established tools for reproducible agent evaluation.Source: §3 · p. 3

Mining

  • Candidate pairs share at least one runtime source file, and the mined pull requests change 1–12 such files.Documentation, CI, and formatting files are excluded; shared-file edits identify potentially related changes but do not guarantee interference.Source: §3 · p. 3

Checking the shared base

  • Each candidate pair starts from the merge base of the two pull-request bases and must pass clean-merge and base-validity checks.For each pull request, its gold tests must fail on the shared base and pass after its code patch is applied.Source: §3 · p. 3
  • Shared-base validation prevents prior changes in one pull request’s base from hiding an interaction; invalid bases accounted for about one third of rejections in a representative batch.One HTTP-header pair was rejected because the later base already contained a helper from the earlier pull request.Source: §3 · p. 3

Yield

  • 417 of 4,262 candidate Django pull-request pairs were accepted as valid coupled instances after deterministic validation.In a representative batch, about 36% failed from textual conflicts, 33% from base validity, 17% from failure to apply on a common base, and 14% were accepted.Source: §3 · p. 3

4 Preliminary Study

The study tests semantic coordination across synthetic interface changes, reviewed Django pull-request pairs, and constructed tasks using real Django helpers. Corrected grading found almost no interference in mined pairs, while constructed tasks produced frequent failures despite clean textual merges.

Synthetic Tier

  • For synthetic renames and required-argument changes, blind patches failed on all d changed helpers (Δblind = d), while informed patches had no failures.For return-type changes, informed patches still averaged 0.13 failures per run.Source: §4 · p. 3

Synthetic Tier with an Oracle Message

  • A roughly 130-token oracle message reduced mean synthetic interference from 2.50 to 0.04, a 98% reduction.The message was generated from the scripted transformation and gave the blind agent a complete, accurate description.Source: §4 · p. 3, Figure 3 · p. 3

Mined PR Tier

  • The mined-PR evaluation used GPT-5-class coding models in solo and blind conditions, then corrected grading to apply the same combined test set in every condition.The initial procedure could falsely inflate interference by using different tests for solo and merged patches.Source: §4 · pp. 3–4, Figure 4 · p. 4

Mined PR Tier; Constructed Tasks on Real Django Helpers

  • Only 1 of 834 runs on 417 validated Django pairs showed interference after corrected grading, while constructed tasks produced interference in 105/108 runs (97%).The constructed-task failures occurred across all 12 helpers despite clean textual merges.Source: §4 · p. 4

Constructed Tasks on Real Django Helpers

  • The real-derived tier applied three interface transformations to 12 Django helpers, creating 36 instances with one scripted change and one agent-written patch.More broadly, the study evaluated more than 4,000 graded task solves across two model families.Source: §4 · pp. 3–4

Threats to Validity

  • The constructed-task rates demonstrate the failure mechanism on real helpers but do not estimate its frequency or severity in everyday parallel development.The tasks constrain agents to call selected helpers, and agent experiments are limited to Django.Source: §4 · p. 4

5 Ongoing Work and Open Problems

The paper leaves open how often interference occurs before review, how agents should communicate during parallel work, and how findings extend beyond pairs of tasks.

5 Ongoing Work and Open Problems

  • Pre-review prevalence requires execution records that preserve a shared starting commit, independent patches, and intermediate states.Such records would enable comparisons across isolated workers, asynchronous updates, explicit orchestration, and shared state [7].Source: §5 · p. 4
  • The 98% synthetic interference reduction from oracle messages does not establish how plans, unfinished patches, or generated summaries compare under real-time constraints.Message options differ in completeness, delay, and privacy; comparisons should control token budgets.Source: §5 · p. 4
  • Results for pairs may not predict larger groups, where failures can require three or more changes together and dependencies multiply.Extending the benchmark could test whether pairwise results predict larger-group reliability and estimate coordination headroom before work begins.Source: §5 · p. 4

6 Conclusion

The conclusion presents stale as a test of whether individually passing patches fail when combined and uses visibility of completed concurrent changes to estimate coordination headroom. Results differ across benchmark tiers, and constructed-task rates do not estimate real-world prevalence.

6 Conclusion

  • stale tests the same patches alone and together to identify failures introduced by combining them.Comparing blind agents with agents who see the completed concurrent change estimates how much coordination could help.Source: §6 · p. 4
  • On constructed tasks reproducing stale-interface failures across 12 Django helpers, interference occurred in 97% of runs, and an oracle message enabled 82% recovery; these rates do not measure real-world prevalence.Source: §6 · p. 4

A Related Work

The related work situates stale alongside benchmarks and tools for coding agents, multi-agent coordination systems, and research on software merge conflicts. It applies hidden tests to detect behavioral failures after clean textual merges.

Agent Benchmarks and Scaffolds

  • SWE-bench [6] and SWE-agent support reproducible, single-task coding-agent evaluation; stale extends test-based evaluation to changes combined in parallel.Source: A Related Work · p. 5

Multi-Agent Software Development

  • AutoGen [14], MetaGPT [5], and ChatDev [11] coordinate agents through roles or communication protocols, while stale tests failures from stale information.Evaluating messages based on plans or unfinished work remains open.Source: A Related Work · p. 5

Merging and Collaboration Conflicts

  • Prior work studies textual and semantic conflicts, merge-time detection, and transactional isolation [3]; stale tests behavioral failures after clean textual merges.Source: A Related Work · p. 5

B A Minimal Worked Instance

The worked instance shows how independently correct changes can become incompatible when one task renames helpers another task calls. Seeing the completed rename lets the informed agent use the new helper name.

B A Minimal Worked Instance

C A Real-Derived Staleness Tier

This tier tests scripted interface changes on consumers of real Django helpers. Interference was common in these constructed tasks, while an oracle message often enabled recovery; these results do not estimate how often unresolved dependencies occur in practice.

C A Real-Derived Staleness Tier

  • The tier links 12 unit-testable Django helpers to validated PR pairs, then constructs breaking helper changes and consumers that call them.The historical pairs supply examples of dependencies; the constructed tasks deliberately make the helper change breaking.Source: C A Real-Derived Staleness Tier · p. 5
  • Three scripted transformations—rename, required keyword-only argument, and return-type change—produce 36 task instances.The return-type transformation returns an object whose .text field contains the original result.Source: C A Real-Derived Staleness Tier · p. 5
  • In the phone2numeric example, the consumer calls the old name after a rename, so the combined code raises a NameError despite passing alone and merging cleanly.The consumer strips spaces and hyphens before calling the helper.Source: C A Real-Derived Staleness Tier · p. 5
  • 89/108 runs (82%) recovered with an oracle message, rising to 93% among message-conditioned patches that merged without textual conflicts.The message described the finalized scripted edit.Source: C A Real-Derived Staleness Tier · p. 5, Table 2 · p. 6
  • Because the breaking changes are scripted and communication describes them completely, estimating real-world prevalence requires records of parallel agent work before review.The constructed failure rates are not estimates of how often unresolved dependencies occur in practice.Source: C A Real-Derived Staleness Tier · p. 5
  • 105 of 108 blind runs (97%) showed interference across the 12 Django helpers.Each counted run passed with the consumer patch alone and merged without textual conflicts, but failed hidden tests after the scripted transformation.Source: C A Real-Derived Staleness Tier · p. 5, Table 2 · p. 6

Related papers

Loading 2609.25396v1…