PaperlayerView exact revision on arXiv

Source-linked AI summary

CrossSWE: Benchmarking Cross-Repository Joint Development for Coding Agents

Silin Chen, Yuling Shi, Heng Lian, Zihan Zhang, Xiaodong Gu, Haotian Lin, Zhensu Sun

arXiv:2611.01484v1cs.SE

TL;DR

Coding-agent benchmarks have expanded task complexity, but most do not require coordinated changes across interdependent repositories. CrossSWE constructs and verifies real-world tasks requiring jointly necessary repository modifications; its evaluation finds substantial difficulty, with the strongest model achieving 58.48% Pass@1.

  • Problem

    Existing benchmarks largely increase task complexity within single repositories, while real-world development can require coordinated changes across interdependent repositories.

    Source: §1 · pp. 1–2

  • Method

    CrossSWE builds real-world multi-repository tasks and verifies joint necessity by testing whether partial repository patches fail to achieve the target behavior.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    58.48% Pass@1 was achieved by the strongest evaluated model, GPT-6.1 Sol, and cross-repository inconsistencies were the dominant failure mode.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    CrossSWE provides a benchmark for evaluating coding agents on jointly necessary changes across repositories, including their ability to maintain cross-repository consistency.

    Source: §6 · pp. 8–9

    Coverage of large-scale multi-repository development is limited: only six of 171 instances involve three or more repositories.

    Source: Limitations · p. 9

Abstract

from arXiv · show

As coding agents achieve increasingly high success rates on software engineering benchmarks, recent efforts have sought to increase task complexity through longer horizons and more extensive code modifications. However, real-world software development often spans multiple interdependent repositories, introducing additional challenges in dependency reasoning, change propagation, and interface consistency. Rather than further increasing complexity within individual repositories, we introduce CrossSWE, a benchmark for cross-repository joint development comprising 171 real-world instances spanning 165 repositories, 20 technical domains, and 13 programming languages. CrossSWE verifies the necessity of coordinated changes by testing whether partial repository modifications are insufficient to achieve the intended behavior. Our evaluation of five LLMs reveals substantial challenges, with the strongest model, GPT-6.1 Sol, achieving only 58.48% Pass@1. Furthermore, cross-repository inconsistencies emerge as the most prevalent failure mode (44.2%), highlighting the difficulty of maintaining consistency across interdependent repositories.

1 Introduction

As coding-agent benchmarks grow more complex, most still keep changes and tests within one repository, leaving coordinated cross-repository development underexplored. CrossSWE benchmarks tasks requiring jointly necessary changes across repositories, targeting challenges in dependency reasoning, change propagation, and consistency.

1 Introduction

  • Existing efforts mainly increase complexity within individual repositories, while other multi-repository benchmarks keep patches and tests confined to one repository.In those settings, additional repositories provide context rather than requiring coordinated modifications.Source: §1 · pp. 1–2
  • Cross-repository tasks require coordinated changes because modifying only one repository may leave dependent behavior or shared interfaces inconsistent.The oauth2-proxy and Higress example requires both repositories to handle multiple cookies correctly during session refresh.Source: §1 · p. 2
  • Cross-repository development adds challenges in dependency reasoning, change propagation, and maintaining consistency across repositories.Individually plausible patches may still fail to resolve the task when they are inconsistent with one another.Source: §1 · p. 2
  • CrossSWE contains 171 real-world tasks across 165 repositories, 20 technical domains, and 13 programming languages.The benchmark targets coordinated development across repository boundaries.Source: §1 · p. 2

2 Related Work

Prior benchmarks have broadened repository-level task scope, while multi-repository settings often use other repositories as context rather than requiring joint changes. CrossSWE adds the complementary dimension of tasks whose solutions require compatible modifications across interdependent repositories.

2.1 Repository-Level Software Engineering Benchmarks

  • CrossSWE explores cross-repository coordination as a complementary dimension to benchmarks that increase complexity within individual repositories.The benchmark focuses on real-world tasks requiring jointly necessary changes across interdependent repositories.Source: §2.1 · p. 3

2.2 Multi-Repository Software Engineering

  • Unlike settings that use cross-repository information while keeping the patch and validation in one repository, CrossSWE verifies that multiple repositories must be modified together.It tests whether agents can implement mutually compatible changes, not merely reason about dependencies.Source: §2.2 · p. 3

3 CrossSWE

CrossSWE formulates tasks as coordinated changes across multiple repositories, then constructs and verifies instances so partial patches cannot satisfy the target behavior. Its 171 instances span varied task intents, technical domains, languages, and reference-patch sizes.

3.1 Task Formulation

  • An instance is resolved only when the joint repository state passes the complete evaluation suite; changes to any one repository alone must be insufficient.Each instance includes participating repositories, a spanning task statement, tests for target and existing behavior, and repository-specific gold patches.Source: §3.1 · p. 4

3.2 Benchmark Collection

  • The collection pipeline mines candidate changes, screens for substantive related implementation in at least two repositories, and constructs reproducible environments.Candidates with only documentation or other nonimplementation changes, or with insufficient evidence, are excluded.Source: §3.2 · pp. 4–5
  • Joint necessity is checked by testing mixed configurations of repository-specific gold patches; only the complete coordinated patch set should pass.For two repositories, the all-base and either single-patch configuration must fail, while the combined-patch configuration must pass.Source: §3.2 · p. 5

3.3 Benchmark Composition

  • 56.1% of CrossSWE instances modify more than ten files, versus 6% in SWE-bench Pro and 30% in SWE-Bench ProMax.The comparison describes reference-patch sizes across the benchmarks.Source: §3.3 · p. 6, Table 1 · p. 6
  • 66.7% of CrossSWE instances change over 200 lines, versus 13% in SWE-bench Pro and 32% in SWE-Bench ProMax.The broader aggregate solution scope reflects changes spanning multiple repositories.Source: §3.3 · p. 6
  • CrossSWE instances cover eight task-intent categories, 20 technical domains, and 13 programming languages.The categories include feature development, API or protocol evolution, security and observability coordination, migration, and compatibility maintenance.Source: §3.3 · pp. 5–6

4 Experimental Setup

The experiments use a shared agent scaffold and fixed configurations to compare five models on CrossSWE. Pass@1 measures whether the final repository state satisfies the full evaluation suite, alongside resource and interaction measures.

4 Experimental Setup

  • All experiments use MINI-SWE-AGENT v2.4.6 as a common scaffold for iterative shell-based code inspection, modification, and testing.The scaffold configuration is held fixed across models.Source: §4 · p. 6
  • The evaluation compares five frontier models at high reasoning effort, with each run capped at 300 interaction steps and $10.The models are GPT-6.1 Sol, Claude Sonnet 5.5, DeepSeek V4.1 Flash, GLM-5.3 Flash, and Qwen3.8-Flash-Next.Source: §4 · p. 6
  • Pass@1 counts an instance as resolved only when the final repository state satisfies the complete evaluation suite.Input tokens, output tokens, steps, and cost are also reported to characterize resource and interaction costs.Source: §4 · p. 6

5 Results

CrossSWE remains challenging for evaluated agents, and jointly changing interdependent repositories is harder than modifying only one side. Agents use different interaction strategies, while cross-repository inconsistency is the most prevalent failure category.

5.1 Main Results

  • 58.48% Pass@1 for GPT-6.1 Sol is below its reported 75.2% on Deep-SWE v1.1; it leads on single-language tasks with 60.34% versus 54.55% on multi-language tasks.DeepSeek V4.1 Flash achieves 52.73% Pass@1 on multi-language tasks. GPT-6.1 Sol averages $0.3496 per instance, close to GLM-5.3 Flash at $0.3411, while requiring 32.9 interaction steps versus 152.7 for DeepSeek V4.1 Flash.Source: §5.1 · pp. 6–7

5.2 Joint vs. Independent Development

  • GPT-6.1 Sol improves from 58.48% to 64.55% Pass@1 on single-side tasks, while DeepSeek V4.1 Flash improves from 52.63% to 61.81%.Average steps also decrease from 32.90 to 18.54 for GPT-6.1 Sol and from 152.70 to 126.94 for DeepSeek V4.1 Flash; both single-side tasks retain both repositories but require modifications to only one.Source: §5.2 · p. 7, Table 2 · p. 7

5.3 Agent Behaviour Analysis

  • GPT-6.1 Sol and Claude Sonnet 5.5 tend to make earlier, more concentrated edits, while DeepSeek V4.1 Flash and GLM-5.3 Flash spend more interactions exploring and alternating edits with tests.Across agents, interaction counts do not scale proportionally with final source lines changed; Figure 6 presents distributions of token usage, code changes, and interaction steps.Source: §5.3 · pp. 7–8, Figure 6 · p. 7

5.4 Failure Patterns Analysis

  • Cross-repository inconsistency is the most prevalent failure category across all five models, accounting for 44.2% of GPT’s unsuccessful trajectories and 41.2% of Claude’s.Incomplete implementation is also common for Claude (29.4%), DeepSeek (25.0%), and GLM (25.4%); the analysis classifies failures into five categories, including compatibility regression, logic/state errors, and boundary handling.Source: §5.4 · p. 8, Figure 7 · p. 8

6 Conclusion

CrossSWE benchmarks real-world tasks requiring jointly necessary changes across repositories. Evaluation of five LLMs finds substantial challenges, with cross-repository inconsistency the dominant failure mode.

6 Conclusion

  • CrossSWE contains 171 real-world instances across 165 repositories, 20 technical domains, and 13 programming languages, with executable verification of jointly necessary repository changes.The strongest evaluated model achieves only 58.48% Pass@1; single-side modification tasks have higher success rates, and cross-repository inconsistency is the dominant failure mode.Source: §6 · pp. 8–9

Limitations

CrossSWE’s evaluation is bounded by limited coverage of larger repository networks, supplied repository selection, and a shared agent scaffold with interaction and cost limits.

Limitations

  • Only six of 171 instances involve three or more repositories, limiting coverage of large-scale multi-repository development.The authors suggest extending the benchmark to larger networks of interdependent repositories.Source: Limitations · p. 9
  • Because participating repositories are supplied, CrossSWE focuses on dependency reasoning and coordinated modifications rather than repository discovery.Real-world development may require identifying relevant repositories from a broader software ecosystem.Source: Limitations · p. 9
  • Evaluation uses a unified mini-SWE-agent scaffold and is capped at 300 interaction steps and $10 per instance.The authors say findings may provide insights into other shell-based frameworks, but direct validation remains necessary.Source: Limitations · p. 9

Ethical Considerations

The benchmark uses public GitHub repositories and development histories, which may contain sensitive or offensive material; the authors reviewed artifacts and note evaluation costs.

Ethical Considerations

  • Public GitHub repositories and development histories may contain personal information, sensitive data, or offensive content.Evaluating coding agents may also incur substantial computational and monetary costs.Source: Ethical Considerations · p. 9
  • Before release, the authors reviewed benchmark artifacts for personal identifiers, sensitive information, and potentially offensive or objectionable content.Source: Ethical Considerations · p. 9
  • The benchmark focuses on technical information needed for task reconstruction and evaluation rather than developers’ personal identities or unrelated content.The authors state this minimizes unnecessary exposure of such information.Source: Ethical Considerations · p. 9

A.2 Task-Intent Taxonomy

The taxonomy organizes the 171-instance formal evaluation subset into mutually exclusive high-level task-intent categories.

A.2 Task-Intent Taxonomy

  • The high-level task-intent categories are mutually exclusive and cover all 171 instances in the formal evaluation subset.Source: §A.2 · p. 12
  • Table 3 defines each category by the primary cross-repository behavior it represents.Source: §A.2 · p. 12

C Use of AI Assistants

AI assistants helped draft basic code and refine manuscript language; the authors report conducting and verifying the research themselves.

C Use of AI Assistants

Loading 2611.01484v1…