PaperlayerView exact revision on arXiv

Source-linked AI summary

WideSWE: Can Coding Agents Coordinate Changes Across Repositories?

Baoyi Wang, Xingliang Wang, Jinyang Wu, Keming Wu, Chen Zhi, Jianwei Yin

arXiv:2609.33382v1cs.SE

TL;DR

Coding-agent evaluations have broadened beyond code generation, but remain centered on single-repository tasks even though ecosystem changes may require coordinated edits across repositories. WideSWE evaluates this setting with real-world tasks and requirement-aligned hidden tests. Across seven configurations, task success peaks at 42.50%, and joint execution can use related-repository information to guide implementation and verification.

  • Problem

    Coding-agent evaluations have broadened beyond code generation but remain centered on single-repository tasks, despite ecosystem changes that may require coordinated edits across repositories.

    Source: §1 · p. 1

  • Method

    WideSWE groups linked multi-repository changes into 120 real-world tasks, builds prompts from issues and PRs, and revises hidden tests to preserve requirements while allowing alternative correct implementations.

    Source: §1 · p. 2

  • Results

    Across seven configurations, task success peaks at 42.50%; among 89 paired cases, joint execution solved 36 versus independent execution’s 32, with 20 outcome reversals.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Joint execution can use information from related repositories to guide implementation and verification, while independent execution can recover omitted work.

    Source: §1 · p. 2, §6 · p. 10

    The benchmark covers Linux tasks involving two or three target repositories, so results do not establish performance on tasks with more repositories or other platforms.

    Source: D BENCHMARK SCOPE AND LIMITATIONS · p. 30

Abstract

from arXiv · show

Coding-agent evaluation has progressed from resolving individual issues to carrying out long-horizon development, yet task completion is still largely assessed within a single codebase. In software ecosystems, many features and bug fixes require coordinated changes across multiple repositories. We introduce WideSWE to evaluate coding agents on such cross-repository tasks. Mining and reviewing changes across 103 software ecosystems yields 120 real-world tasks, balanced between 60 bug fixes and 60 features. We derive prompts from related issues and pull requests. We systematically review and adapt hidden tests to support diverse correct implementations while preserving required behavior and regression checks. Across seven agent configurations, full task success ranges from 10.83% to 42.50%, with the configuration pairing Codex CLI with GPT-5.6-sol achieving the highest rate. Trajectories show agents failing to identify necessary changes, recognizing changes but leaving them unfinished, or modifying the required repositories without fully satisfying the request. To examine whether working on one repository at a time can alleviate these difficulties, we compare it with joint execution under identical prompts. Independent execution mainly recovers omitted work and is less effective at correcting previously attempted but unsuccessful implementations. Joint execution can use information from related repositories to guide implementation and verification. Code is available at https://github.com/ZJU-ACES-ISE/WideSWE.

1 INTRODUCTION

Coding-agent benchmarks have broadened toward long-horizon work but remain centered on individual repositories, although real software changes can span an ecosystem. WideSWE studies whether agents can coordinate those changes and examines their failure patterns and execution modes.

1 INTRODUCTION

  • Across 103 ecosystems, 109,233 of 1,729,171 analyzed PRs explicitly referenced another repository, motivating evaluation at the ecosystem level.Source: §1 · p. 1
  • Observed failures span missing affected repositories, recognized but unfinished work, and incorrect edits despite modifying all required repositories.Source: Figure 1 · p. 2, §1 · p. 2
  • WideSWE groups linked multi-repository changes into 120 tasks, balanced between 60 bug fixes and 60 features, and revises hidden tests to support alternative correct implementations.Source: §1 · p. 2
  • Codex CLI–GPT-5.6-sol solved 42.50% of cases; among 89 paired cases, joint execution solved 36 versus independent execution’s 32, with 20 outcome reversals.Source: §1 · p. 2
  • The empirical study evaluates seven agent configurations and combines joint-versus-independent comparisons with trajectory analysis.Source: §1 · p. 3

2 THE WIDESWE BENCHMARK

The benchmark formulates tasks around shared changes across repositories, constructs cases from linked real-world development records, and scores completion across every target repository.

2.1 TASK FORMULATION

  • Each task targets at least two repositories needing substantive changes, while other workspace repositories provide context; completion must preserve behavior and cross-repository consistency.Source: §2.1 · p. 3

2.2 BENCHMARK CONSTRUCTION

  • Mining linked PRs across 103 ecosystems yields 120 validated tasks—60 bug fixes and 60 features—spanning 41 ecosystems.Source: §2.2 · pp. 3–4
  • Prompts combine original issue and PR requirements; hidden tests relax implementation-specific constraints and remove checks for unrequested functionality while retaining required behavior and regressions.Source: §2.2 · p. 4

2.3 EVALUATION

  • Task scoring uses hidden F2P and P2P tests, and a task succeeds only when every target repository passes all applicable tests; repository- and test-level results diagnose partial progress.Source: §2.3 · p. 5

3 EXPERIMENTAL SETUP

The evaluation uses seven configurations under shared run limits, measures success across bug fixes, features, and all tasks, and compares joint with independent execution on a prompt-matched subset.

3 EXPERIMENTAL SETUP

  • Seven configurations pair Codex CLI or Claude Code with language models and reasoning-effort settings under the same permissions, timeout policy, and per-run resource limits.Codex CLI is paired with GPT-5.6-sol; Claude Code is paired with GPT-5.6-sol, Claude Opus 5, Gemini 3.8 Flash, DeepSeek V4 Pro, Qwen 3.8 Max, and GLM 5.3.Source: §3 · p. 5
  • Case-level success requires every target repository to pass all fail-to-pass and pass-to-pass tests; repository-level success applies that criterion separately to each target.The evaluation also reports F2P-complete, P2P-preserved, and mean API requests per task.Source: §3 · p. 5
  • The experiments ask how well agents complete cross-repository tasks, how success varies with task type and scope, and how joint execution compares with independent runs.The setup includes Table 2’s failure categories and Table 3’s joint-versus-independent comparison.Source: §3 · pp. 5–6, Table 2 · p. 6, Table 3 · p. 6
  • RQ3 compares joint and independent execution on 89 cases with identical prompts; 31 cases needing repository-specific prompt adaptations are excluded.The subset contains 29 bug fixes and 60 features.Source: §3 · p. 6

4 RESULTS

Across configurations, complete cross-repository task success remains limited, and the results vary by task type and execution mode. Joint execution can use related-repository context, while independent runs can recover some omitted work.

4.1 RQ1: PERFORMANCE ON CROSS-REPOSITORY TASKS

  • 42.50% of tasks are fully solved by Codex CLI–GPT-5.6-sol, the best configuration; the other configurations achieve 10.83%–37.50%.The best configuration solves at least one target repository in 83.33% of tasks, showing a gap between repository-level progress and complete task resolution.Source: §4.1 · p. 6
  • For every configuration except Gemini, most unresolved tasks (52.08%–69.57%) have at least one repository passing all F2P tests, while failures solely on P2P tests are uncommon.Gemini has no repository passing all F2P tests in 85.98% of unresolved tasks.Source: §4.1 · p. 6
  • 37.68%, 2.90%, and 59.42% of Codex CLI–GPT-5.6-sol unresolved runs are classified as incomplete scope identification, recognized work without delivery, and post-edit failure, respectively.Post-edit failures include changes that do not satisfy required behavior or regression checks.Source: §4.1 · p. 6
  • Gemini’s unresolved runs most often leave recognized work undelivered (52.34%), while Opus and Qwen most often fail after editing (67.95% and 73.33%, respectively).DeepSeek, Qwen, and GLM have lower recognized-work-without-delivery shares (7.95%, 5.33%, and 9.38%). DeepSeek and Gemini examples show agents continuing on one repository while leaving another unchanged.Source: §4.1 · p. 7

4.2 RQ2: TASK INTENT, REPOSITORY SCOPE, AND LANGUAGE DIVERSITY

  • All seven configurations do better on multiple-family than single-family bug fixes, but feature success does not follow that pattern; Qwen falls from 37.14% to 12.00% across feature groups.For multiple-family features, GPT and Opus each reach 28.00%, while Opus is comparatively stable across the two feature groups.Source: §4.2 · pp. 7–8, Figure 4 · p. 7
  • All configurations have lower task success on three-repository tasks; Codex CLI–GPT-5.6-sol falls from 42.99% on two-repository tasks to 38.46% on three-repository tasks.For that configuration, repository success rises from 63.08% to 66.67%; 87.50% of failed three-repository tasks modify all targets, and 62.50% solve two of them.Source: Figure 4 · p. 7, §4.2 · p. 8

4.3 RQ3: JOINT VERSUS INDEPENDENT EXECUTION

  • Joint execution solves 40.45% of the 89 paired tasks, versus 35.96% for independent execution, which uses 296.6 API requests per task compared with 94.7 for Joint.The two settings change outcomes in either direction for 22.47% of tasks.Source: §4.3 · p. 8, Figure 5 · p. 8
  • Among repositories failing in Joint execution, 60.00% of those left unmodified succeed independently, versus 9.43% of those already modified; joint context can also guide implementation and verification.In Ansible, one repository’s implementation guides compatible changes in another; in Symfony, comparing implementations helps resolve a behavioral difference so both repositories pass.Source: Figure 5 · p. 8, §4.3 · p. 9

5 RELATED WORK

Related work studies software dependencies, coordination, repository context, and issue resolution, while WIDESWE evaluates whether agents can deliver one shared change across multiple target repositories.

5 RELATED WORK

  • Prior studies examine project dependencies and human coordination, including tracing bug causes across projects and coordinating upstream and downstream repairs.WIDESWE turns real cross-repository changes into executable tasks that test whether agents identify the required scope and fulfill a shared request.Source: §5 · p. 9
  • Repository-context benchmarks evaluate code completion, while issue-resolution and newer benchmarks extend evaluation to real repositories, languages, and long-horizon engineering.These extensions broaden agents’ work within a project but do not themselves assess coordinated changes across repositories.Source: §5 · p. 9
  • BeyondSWE’s CrossRepo tasks use external-repository code and solutions to resolve an issue in one target, whereas WIDESWE scores a shared change across multiple target repositories.WIDESWE task success requires all target repositories to pass their required tests.Source: §5 · p. 9

6 CONCLUSION

WIDESWE evaluates shared feature and bug-fix tasks across repositories; its construction screens linked changes, reviews requirements and tests, and selects a balanced task set.

6 CONCLUSION

  • WIDESWE contains 120 real-world tasks requiring coordinated changes across repositories, and the highest task success across seven evaluated configurations is 42.50%.Agents may miss necessary changes, leave recognized work unfinished, or modify required repositories without fully satisfying the request; independent execution can recover omitted work but does not improve overall success in the paired comparison.Source: §6 · p. 10

A.1 CANDIDATE DISCOVERY AND TIME SCOPE

  • Candidate discovery screens linked changes in active ecosystems and excludes organizations lacking two non-archived top-ranked repositories or recent activity.The review cutoff uses the latest merge timestamp among involved pull requests; cross-referencing records count linked activity, not distinct tasks, and link direction does not establish dependency direction.Source: §A.1 · p. 13

A.2 SEMANTIC REVIEW, CASE ASSEMBLY, AND SELECTION

  • Reviewers verify that linked records address one feature or fix and require substantive changes in multiple repositories; shared change chains are consolidated only when they meet a common requirement.The gRPC timeout interoperability case combines Java and Go pull requests, then regenerates reference and test patches and reruns validation.Source: §A.2 · p. 13
  • The release balances 60 bug fixes and 60 features, retaining all bug fixes and selecting 60 features from 132 eligible features.Selection includes all 13 three-repository feature cases and 47 two-repository cases, favoring substantive runtime, protocol, schema, or SDK-integration changes.Source: §A.2 · p. 13

A.3 DATASET DISTRIBUTIONS

The dataset covers two cross-repository coordination patterns, with descriptive statistics on workspace context, patch size, and tests. The appendices also document prompt construction, historical snapshots, and evaluator test isolation.

A.3 DATASET DISTRIBUTIONS

  • 96 tasks (80.0%) involve producer–consumer dependencies, while 24 (20.0%) propagate behavior or fixes in parallel.Producer–consumer tasks include capabilities, interfaces, schemas, service behavior, or build artifacts that other repositories consume or adapt to.Source: §A.3 · p. 13, Table 4 · p. 14
  • Feature patches are generally larger than bug-fix patches; across all tasks, median patch size is 13 files and 416 lines.Patch counts include production code, tests, generated files, documentation, and configuration; size describes changes, not task difficulty.Source: §A.3 · p. 14, Figure 7 · p. 15
  • There are 2,815 F2P and 22,139 P2P tests across 253 target-repository instances.The per-task curves aggregate tests across repositories and do not represent agent success rates.Source: §A.3 · p. 14, Figure 8 · p. 15

A.4 PROMPT PROVENANCE AND LANGUAGE LABELS

  • Prompts retain observable requirements from source issues and PRs without adding implementation-specific instructions, and two authors independently review each prompt.Test-required behavior must be supported by public demand and described sufficiently without imposing undisclosed naming or representation choices.Source: §A.4 · p. 14, §A.5 · p. 16

A.5 PROMPT CONSTRUCTION EXAMPLE

  • The Expo logout prompt consolidates overlapping requirements from eas-cli#3555 and expo#45802, including failure handling and parity across repositories.The prompt omits implementation specifics so agents must determine how to realize the behavior in each codebase.Source: §A.5 · p. 16

A.6 HISTORICAL WORKSPACE SNAPSHOTS

  • Target repositories use their PR base commits, while context repositories are snapshots no later than the earliest target base; agents do not receive Gold patches or hidden tests.Source: §A.6 · p. 16

A.7 EVALUATOR TEST ISOLATION AND PATCH CONFLICTS

  • Evaluator test overlays isolate hidden-test patches from agent edits and handle relocated or duplicate tests, but do not change behavioral acceptance criteria.Patch-injection failures are recorded and cannot yield successful evaluations.Source: §A.7 · p. 16

A.8.1 RELAX IMPLEMENTATION-SPECIFIC CONSTRAINTS

The test audit relaxes implementation-specific constraints and removes checks for unrequested functionality while retaining required behavior. Across the benchmark, revisions affected 88 of 120 tasks.

A.8.1 RELAX IMPLEMENTATION-SPECIFIC CONSTRAINTS

  • For TensorDict/TorchRL, tests require preserving behavior when generator=None and raising TypeError for unsupported values, but not a particular error message.The audited test checks accepted and unsupported generator values while removing the inherited message match.Source: §A.8.1 · pp. 16, 18

A.8.2 REMOVE UNREQUESTED FUNCTIONALITY CHECKS

  • For Nextcloud, tests retain separate timed-event and all-day reminder settings, propagation, validation, and event-specific behavior, but not three unrequested preset choices.The hidden patch removes preset-expansion checks while retaining existing preset coverage and support for user-configured offsets.Source: §A.8.2 · pp. 18–19
  • The revised TensorDict/TorchRL and Nextcloud cases retain failing Base behavior and pass with Gold, with 21 F2P and 580 P2P checks, and 53 F2P and 223 P2P checks, respectively.Source: §A.8.2 · p. 19
  • Two independent reviewers assessed each proposed test revision against the task requirements and source issues or PRs, resolving disagreements through discussion.Revisions were accepted only when both reviewers judged that they removed unrequested or implementation-specific constraints without weakening required behavior.Source: §A.8.2 · p. 19
  • 88 of 120 tasks received requirement-aligned test revisions: 87 relaxed implementation-specific constraints, 16 removed unrequested checks, and 15 involved both.These are task-level counts, not counts of tests or assertions.Source: §A.8.2 · p. 19

B SUPPLEMENTARY EVALUATION RESULTS

The supplementary analyses define how joint and independent runs are compared, how partial test progress is measured, and how unresolved runs are classified. The paired task outcomes differ by task intent, while recovery comparisons are descriptive rather than causal.

B.1 JOINT AND INDEPENDENT REPOSITORY COMPARISON

  • The Joint–Independent comparison matches per-run limits, not total task budgets: Independent receives one full run for each target repository.Joint exposes the complete ecosystem workspace to one run; the comparison examines whether separate solving alleviates cross-repository difficulties, not equal-total-budget execution scope.Source: §B.1 · pp. 19–20
  • Recovery among the 73 Joint-failed repositories is grouped by whether Joint modified them, but the difference in recovery rates is descriptive, not causal.A recorded modification does not imply correctness; Independent recovery requires all F2P and P2P tests to pass.Source: §B.1 · p. 20, Table 6 · p. 20
  • Among 29 bug fixes, nine cases succeed in both settings, one only in Joint, four only in Independent-all, and 15 in neither; among 60 features, the counts are 15, 11, four, and 30.The authors caution that the opposite net directions should not be generalized beyond this sample.Source: §B.1 · p. 20, Table 5 · p. 20

B.2 TEST-LEVEL PASS RATES

  • Test-level pass rates divide total passed tests by total tests, capturing partial progress and weighting repositories by their test counts.Table 8 uses the same 2,532 F2P and 18,051 P2P tests in each condition for the 89 paired RQ3 tasks.Source: §B.2 · p. 20, Table 8 · p. 21

B.3 FAILURE ANNOTATION PROTOCOL

  • Unresolved runs are classified as Post-edit failure, Incomplete scope identification, or Recognized work without delivery, with 91.2% annotation agreement (Cohen’s κ = 0.86).Two authors annotated independently and resolved disagreements through discussion.Source: §B.3 · p. 21

C.1 CROSS-REPOSITORY BEHAVIOR IN JOINT RUNS

Joint-run cases show that repository coverage alone does not ensure a correct coordinated change: agents may miss targets, leave paths incomplete, or violate interface contracts. Related repositories can also provide implementation and behavioral references, while the comparison between configurations is bounded in what it isolates.

C.1 CROSS-REPOSITORY BEHAVIOR IN JOINT RUNS

  • The examples show scope and completion failures: Sentry's Go SDK passed 14/14 F2P checks while Python and Ruby passed 0/22 and 0/13; Godot stayed at 0/1 despite Native passing 11/11.The Kubernetes consumer migration remained incomplete despite edits to both repositories.Source: §C.1 · p. 21
  • Some runs modify the wrong location or only part of the scope: Laravel AI passes while MCP remains unchanged, and Swagger’s library path remains unfixed.In Sentry CLI/Dart, the agent identifies the CLI defect but not the Dart adaptation, leaving both repositories unmodified.Source: §C.1 · p. 21
  • Changes can reach every target yet remain incomplete within a repository or encode a shared but mistaken interpretation.The Python Celery integration still overwrites baggage; in Sentry/Symbolicator, both endpoints use an exception-name string where a structured exception is required.Source: §C.1 · pp. 21–22
  • Across examples, editing all required repositories is insufficient when adapters or shared representations violate their contracts; complementary endpoint rules can instead succeed.The React Router and Symfony examples fail to preserve input or error behavior, while gRPC succeeds with positive Java timeouts and Go acceptance of legacy zero-valued input.Source: §C.1 · p. 22, Figure 14 · p. 25

C.2 JOINT VERSUS INDEPENDENT CODEX RUNS

  • Independent runs recover omitted Sentry SDK work; joint access to related repositories also supplies implementation facts in Ansible and behavioral references in Symfony.The Ansible joint run inspects the service model and submits a query consistent with its tables and field names, unlike the independent utility run.Source: §C.2 · p. 22, Figure 16 · p. 27, Figure 18 · p. 29

C.3 CODEX CLI VERSUS CLAUDE CODE WITH GPT-5.6-SOL

  • In the Rails/Propshaft pair, the model and prompt are fixed, so the comparison illustrates scope-checking differences rather than isolating a framework component.Source: §C.3 · p. 30

C.4 MODEL COMPARISONS WITHIN CLAUDE CODE

With Claude Code and prompts held fixed, the selected runs illustrate Delivery and Post-edit failures, not model-wide rankings.

Fleet: carrying a plan through to implementation

  • Opus implements Elastic client compression and server decoding, verifies equivalent request behavior, and passes both repositories, while Gemini only plans the changes.Gemini passes 0/5 and 0/2 F2P checks; Opus passes 5/5 and 2/2.Source: §C.4 · p. 30

Rust/Cargo: covering the consumer’s entry points

  • In Rust/Cargo, GPT and DeepSeek include bootstrap.py in the Cargo migration and pass both repositories, while Opus leaves it unchanged and fails Rust’s F2P check.Opus modifies both target repositories, so the failure is incomplete implementation within an edited repository.Source: §C.4 · p. 30

PyPA: checking exchanged metadata

  • In PyPA, Opus checks absent and explicitly empty Import-Name cases through the producer-parser exchange, while DeepSeek drops the empty field and remains F2P-incomplete.DeepSeek reports passing local suites despite the missing field.Source: §C.4 · p. 30

TensorDict/TorchRL: checking downstream state updates

  • In TensorDict/TorchRL, Qwen probes seed updates in both layers and passes, while DeepSeek’s integer-seed write-back error affects both layers.Both DeepSeek repositories remain F2P-incomplete.Source: §C.4 · p. 30

D BENCHMARK SCOPE AND LIMITATIONS

WIDESWE covers real-world cross-repository tasks involving two or three repositories in Linux environments; its results do not establish performance beyond that scope.

D BENCHMARK SCOPE AND LIMITATIONS

  • WIDESWE tasks involve two or three target repositories and run in Linux; the results do not establish performance on tasks involving more repositories or other platforms.Examples span shared middleware changes, client-server behavior, consumer entry points, exchanged metadata, and downstream state updates.Source: D BENCHMARK SCOPE AND LIMITATIONS · p. 30
  • In Rails/Propshaft, Codex extends the change to Rails’ static-file middleware after Propshaft tests pass, while Claude Code modifies only Propshaft.The two runs use the same GPT model.Source: Figure 19 · p. 31
  • In Elastic, Opus implements client compression and server decoding, checks equivalent request behavior, and passes both repositories; Gemini remains in investigation after planning both changes.Source: Figure 20 · p. 32
  • In Rust, both runs complete the Cargo change, but only GPT includes the Python bootstrap entry in the migration and passes both repositories.Opus leaves the bootstrap entry unchanged.Source: Figure 21 · p. 33
  • In PyPA, Opus checks whether emitted metadata retains its meaning when parsed downstream, while DeepSeek drops explicitly empty import-name fields.Source: Figure 22 · p. 34
  • In TensorDict/TorchRL, Qwen probes seed updates in the library and downstream actor, while DeepSeek’s integer-seed write-back error affects both layers.Source: Figure 23 · p. 35

Related papers

Loading 2609.33382v1…