PaperlayerView exact revision on arXiv

Source-linked AI summary

BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing?

Guoxin Chen, Fanzhe Meng, Jiale Zhao, Minghao Li, Daixuan Cheng, Huatong Song, Jie Chen, Yuzhi Lin, Hui Chen, Xin Zhao, Ruihua Song, Chang Liu, Cheng Chen, Kai Jia, Ji-Rong Wen

arXiv:2603.03194v2cs.CLcs.SE

TL;DR

Existing code-agent benchmarks largely test localized fixes within one repository, leaving tasks requiring external knowledge or broader repository changes under-tested. BeyondSWE addresses this gap with a 500-instance benchmark across four broader settings and uses SearchSWE to diagnose search-augmented coding. The benchmark remains far from saturated, while search improves many models unevenly because agents still struggle to ground retrieved evidence in repository-local code and dependency versions.

  • Problem

    Existing benchmarks under-test software-engineering tasks requiring external knowledge or broader repository-level changes beyond localized single-repository bug fixing.

  • Method

    BeyondSWE evaluates four settings across broader knowledge and resolution scopes, while SearchSWE adds controlled web search and fetch access to a coding workflow.

  • Results

    BeyondSWE remains far from saturated; search improves most models, including Codex with GPT-5.4 (xhigh) from 48.48 to 56.65 under an explicit search-aware prompt.

  • Takeaways & Limitations

    Deep search for coding remains an open problem because agents must combine external evidence with repository-local reasoning and execution-based verification.

  • Takeaways & Limitations

    Search gains remain limited and uneven because retrieved evidence may be poorly grounded in local dependencies, versions, or task context.

Abstract

from arXiv · show

Current code-agent benchmarks primarily evaluate localized issue resolution within a single target repository, leaving under-tested many software engineering tasks that require external knowledge or broader repository-level changes. We introduce BeyondSWE, a 500-instance benchmark drawn from 246 real-world GitHub repositories to evaluate code agents beyond single-repository bug fixing. BeyondSWE covers four representative settings: cross-repository issue resolution, domain-specific issue resolution, dependency-driven migration, and document-to-repository generation, spanning both broader knowledge scope and broader resolution scope. Our evaluation shows that BeyondSWE remains far from saturated: the best OpenHands-based agent reaches 46.12 average score, while the strongest Codex harness with GPT-5.4 (xhigh) reaches 56.65 under a search-aware prompt. To study whether external information access closes this gap, we use SearchSWE as a controlled diagnostic baseline for search-augmented coding. Search access improves most models and substantially helps some tasks, but the gains remain limited and uneven, showing that current agents still struggle to convert retrieved information into precise, version-compatible, and locally actionable code changes. These results suggest that deep search for coding remains an open problem: progress requires agents that can reliably combine external evidence with repository-local reasoning and execution-based verification.

1. Introduction

BeyondSWE extends code-agent evaluation beyond localized fixes in one repository by testing broader knowledge and resolution scopes. Its benchmark remains challenging, while controlled search helps unevenly.

  • Motivation: Existing benchmarks under-test tasks requiring external software, documentation, domain knowledge, repository-wide coordination, or full repository construction.Most conventional evaluations assume the issue and relevant context are recoverable from one target repository.
  • Benchmark design: BeyondSWE organizes evaluation around knowledge scope and resolution scope while retaining executable, test-based evaluation.Knowledge scope covers repository-local versus external information; resolution scope covers localized fixes, repository-wide transformations, and repository construction.
  • Benchmark design: The benchmark defines CrossRepo, DomainFix, DepMigrate, and Doc2Repo as four stress-test settings spanning external knowledge and broader repository changes.These settings cover external repositories, specialized scientific or engineering knowledge, breaking dependency changes, and specification-based repository generation.
  • Benchmark scale: 500 instances drawn from 246 real-world GitHub repositories require solutions affecting an average of 10.9 files and 1039 lines per instance.The modification scale substantially exceeds that of existing SWE-bench-style benchmarks.
  • Results: 46.12 average score is achieved by the best OpenHands-based agent, indicating that BeyondSWE remains far from saturated.The benchmark exposes task-specific weaknesses in external knowledge, domain reasoning, coordinated edits, and complete repository generation.
  • Search diagnostic: 48.48 to 56.65 is the improvement for the Codex harness using GPT-5.4 (xhigh) when explicitly prompted to integrate search.Search improves most models, but gains remain limited and uneven because retrieved information is not always converted into precise, version-compatible, locally actionable changes.

2. Related Work

Related benchmarks broaden single-repository issue resolution in several directions, while BeyondSWE focuses on tasks requiring external knowledge or broader repository changes. SearchSWE isolates the effect of adding external information access to coding workflows.

  • SWE benchmarks: Follow-up SWE benchmarks add live updates, broader repository coverage, multilingual settings, decontamination, and harder issue instances.Most retain the assumption that the target repository contains the main problem-solving context.
  • Code agents and search: SearchSWE provides a controlled baseline because production coding systems combine search with proprietary prompts, tool policies, models, and scaffolding.The baseline enables comparisons of when search helps, when it remains insufficient, and why.

3. BeyondSWE

BeyondSWE is a 500-instance, executable benchmark spanning broader knowledge and resolution scopes across four task settings. Its construction emphasizes reproducible environments, stable tests, integrity safeguards, and task-appropriate human verification.

  • 3. BeyondSWE: BeyondSWE evaluates tasks requiring broader knowledge or resolution scope than single-repository bug fixing across 500 instances from 246 GitHub repositories.Target solutions affect an average of 10.9 files and 1039.6 lines per instance.
  • Task formulation: Each instance combines a problem statement, a reproducible Docker environment, and executable tests defining success criteria.Issue-resolution tasks use pass-to-pass and fail-to-pass tests, while Doc2Repo evaluates the generated repository with a complete test suite.
  • Benchmark tasks: CrossRepo uses external repositories or linked artifacts, while DomainFix combines code reasoning with specialized scientific and engineering knowledge.CrossRepo contains 200 issues across 67 repositories with an average of 1.3 external links per issue; DomainFix contains 72 issues across 12 repositories.
  • Benchmark tasks: DepMigrate requires coordinated repository-level edits under breaking upstream dependency changes, whereas Doc2Repo requires constructing a complete repository from a natural-language specification.These settings move beyond localized bug repair toward migration and repository generation.
  • Environment construction: Agent-assisted Docker construction starts from a base Ubuntu container, checks out the pre-PR commit, and resolves setup failures until tests run.Environments are repeatedly tested for stable fail-to-pass and pass-to-pass behavior, and invalid or flaky instances are discarded.
  • Evaluation safeguards: Final verification applies extracted changes in a fresh container, preventing workspace artifacts, cached state, or local configuration from affecting scores.Test files are restored before evaluation to prevent test manipulation.
  • Quality control: BeyondSWE combines automated filtering with task-specific human verification by domain experts, software-engineering experts, senior engineers, and PhD researchers.Review removes setup artifacts and checks that retained tasks represent genuine, sufficiently specified software-engineering challenges.

4. Search-Augmented Diagnostic Baseline

SearchSWE tests whether agents can convert external information into correct repository-local changes. It adds general web search and browsing while blocking direct access to solution-revealing artifacts.

  • Motivation: SearchSWE asks whether external information access reliably becomes correct code changes for tasks whose needed knowledge may lie outside the target repository.The diagnostic targets CrossRepo, DomainFix, and DepMigrate-style needs such as related implementations, domain knowledge, and migration documentation.
  • Controlled baseline: SearchSWE minimally adds web search and webpage fetching to a standard code-agent workflow, enabling otherwise similar comparisons with and without search.The agent chooses when to search, which queries to issue, which results to inspect, and how to integrate evidence into implementation and verification.
  • Controlled access: SearchSWE blocks target-repository URLs, shell operations, and solution-revealing artifacts so agents must rely on indirect external evidence.The safeguards cover GitHub, GitLab, raw-content and API endpoints, and direct git operations.

5. Experiments

BeyondSWE remains challenging for current code agents across broader knowledge and repository-level tasks. Search generally helps, but improvements are uneven because agents struggle to ground retrieved evidence in local code, versions, and task context.

  • Overall Performance: 46.12 average score is reached by the best OpenHands model, while Codex with GPT-5.4 (xhigh) reaches 48.48 by default and 56.65 with a SearchSWE-style prompt.The benchmark remains far from solved across all four task families.
  • Overall Performance: The four task families expose distinct challenges: external software knowledge, domain-specific reasoning, coordinated dependency migration, and complete repository construction.Doc2Repo is especially demanding: the best configuration produces only 2 fully correct repositories out of 50.
  • Search Helps, but Integration Matters: SearchSWE improves seven of eight models, but only 20 of 32 paired task comparisons improve and 31.2% regress.DomainFix benefits most, while CrossRepo, DepMigrate, and Doc2Repo retain substantial regressions.
  • Search Helps, but Integration Matters: The SearchSWE-style Codex prompt raises average score from 48.48 to 56.65, with gains of +19.4 on DomainFix and +8.9 on CrossRepo.It increases average web-search calls by about 2.8× while preserving a similar local coding-tool profile.
  • Search Helps, but Integration Matters: More search calls do not monotonically produce larger gains within either evaluated task.This indicates that search frequency alone is insufficient without relevant retrieval and local grounding.
  • Search Helps, but Integration Matters: Search failures arise from wrong-granularity evidence, version-mismatched knowledge, and semantically unrelated search results contaminating coding context.Reliable search-augmented coding therefore requires source discrimination, version grounding, and local verification.
  • Efficiency: Higher cost and longer trajectories do not reliably yield better performance, as several high-cost or high-token configurations fall below the performance frontier.The token trend rises at low budgets but flattens or declines at higher budgets.

6. Conclusion

BeyondSWE evaluates code agents on software-engineering tasks requiring broader knowledge or broader repository changes than conventional single-repository bug fixing. Results show that agents remain far from saturation, while search is useful but uneven and still requires reliable evidence grounding and code conversion.

  • 6. Conclusion: BeyondSWE is a 500-instance benchmark spanning cross-repository reasoning, domain-specific repair, dependency migration, and repository construction from specifications.These settings vary in both knowledge scope and resolution scope.
  • 6. Conclusion: Current agents remain far from saturating tasks that require broader knowledge or broader resolution scope.Stronger models and harnesses improve performance, but reliable end-to-end solutions remain difficult.
  • 6. Conclusion: External search is useful, especially when missing knowledge is externally retrievable, but its gains depend on grounding evidence in the repository, dependency version, and task specification.Search augmentation does not by itself ensure precise executable changes.
  • 6. Conclusion: Deep search for coding remains an open problem requiring agents to judge evidence, ignore unreliable information, and convert trusted results into executable code changes.This complements prior work that largely targets repository-local issue resolution.

B. Qualitative Analysis of Search-Augmented Coding Failures

Search-augmented coding breaks between retrieval and implementation: relevant evidence may be poorly ranked, version-mismatched, or semantically irrelevant. These failures explain why search helps but remains insufficient on BeyondSWE.

  • Search-augmented coding fails when retrieved evidence is not relevant, not grounded in the local repository state, or not semantically filtered.Agents may receive user-facing documentation instead of implementation logic, newer-version patterns for pinned environments, or authoritative but unrelated sources.
  • These three failure points explain why external search is useful but insufficient for BeyondSWE coding tasks.

B.1. Failure Mode I: Source-Level Evidence Is Hard to Retrieve

Search engines may return conceptually relevant documentation instead of the low-level artifacts needed for robust implementation. In the illustrated case, this mismatch produces a partial patch that fails comprehensive testing.

  • Web search often prioritizes curated documentation over source files, commit diffs, or backend logic needed for precise code changes.
  • In unidata_siphon_pr234, implementing all-station retrieval required understanding parameter handling in an external Iowa Environmental Mesonet backend.
  • The search returned an ambiguous API help page rather than backend source code, leaving key station-parameter behavior unspecified.The documentation did not clarify whether to omit the station parameter, set it to None, use a wildcard, or implement backend-specific error handling.
  • The agent omitted the station parameter in selected cases, but the resulting patch missed source-level edge handling and failed comprehensive tests.
  • Agents must judge whether retrieved evidence has sufficient granularity for source-level changes, rather than treating any relevant result as implementation-ready.

B.2. Failure Mode II: Retrieved Knowledge Must Be Version-Grounded

External coding knowledge must be reconciled with the repository’s dependency versions and inherited architecture. Ignoring those local constraints can turn a plausible modern pattern into a failing change.

  • The correct maintenance solution must match installed dependency versions and the repository’s existing architecture, not necessarily the newest API pattern.
  • In behave_behave-django_pr162, the fixture-loading fix was constrained by legacy Django versions and locally available configuration and packages.
  • The agent assumed a newer Django setting and adopted a class-method lifecycle pattern incompatible with the repository’s legacy instance-method implementation.
  • Changing _pre_setup to a classmethod broke instance-level state and the inherited test lifecycle, causing the test suite to fail.
  • Agents should use local versions, installed APIs, and inherited structure to filter external knowledge before editing code.

B.3. Failure Mode III: Keyword Matches Can Contaminate Context

Keyword overlap can contaminate coding context when overloaded terms retrieve authoritative but unrelated sources. In the example, this noise encourages a plausible plugin pattern that conflicts with local tests.

  • Overloaded technical terms can produce high-ranking results from unrelated domains, contaminating the context for niche-library coding tasks.
  • For abravalheri_validate-pyproject_pr105, a query about repo-review check families drifted toward Autodesk Revit and RelativityOne results.
  • The agent failed to isolate the relevant source and recover a precise integration pattern for the target library.
  • A generic entry_points registration fallback conflicted with fixture-based registration, registering the plugin twice and producing an assertion failure.
  • Agents must reject out-of-domain results and verify that suggested patterns are locally actionable before modifying code.

C.3. BeyondSWE Dataset Details

BeyondSWE combines diverse, human-verified repository tasks with substantial generation demands and reproducible test-based evaluation. Its dataset spans specialized scientific domains, non-trivial repository sizes, and a documented collection and verification pipeline.

  • Dataset scope: DomainFix covers 11 scientific fields, including astronomy, quantum physics, bioinformatics, and materials science, requiring specialized domain knowledge.
  • Dataset scope: Doc2Repo includes 50 repositories ranging from approximately 400 to over 13,000 lines of code, with 31 exceeding 1,500 lines.The distribution is intended to test generation of non-trivial repositories rather than toy examples.
  • Data format: Each instance is represented as JSON, while Doc2Repo uses a repository specification and the complete adapted test suite instead of a P2P/F2P split.
  • Construction pipeline: Candidates come from merged pull requests or specifications and are filtered for executability, stability, task alignment, and reproducible environments.Issue-resolution instances are rewritten into issue-style statements that preserve observable requirements while removing solution-specific details.
  • Construction pipeline: Retention requires tests to distinguish buggy from fixed states for issue-resolution tasks, while Doc2Repo specifications and adapted tests are checked against reference implementations.
  • Verification: Human verification checks capability relevance, task fairness, environment construction, data cleaning, and category fit, using domain or software-engineering experts where appropriate.Reviewers assess whether tasks provide enough information to identify required behavior without revealing the gold solution.

D. Representative Task Examples

The representative examples show four task formats that vary the information available to agents and the capabilities required. Prompts and evaluation controls make external investigation, domain verification, migration compatibility, and specification-driven generation explicit.

  • Prompt design: Benchmark construction rewrites pull requests into issue-style statements that preserve observable context while removing solution-specific details.
  • SearchSWE: SearchSWE uses a shared prompt and search policy that directs agents to search strategically for missing version-specific APIs, obscure errors, or absent implementation details.The workflow combines external evidence with local coding and verification rather than treating search as a substitute for repository reasoning.
  • CrossRepo: CrossRepo asks agents to fix repository issues using related external repositories or linked artifacts when local context is insufficient.The example concerns honoring explicit host settings and an environment variable rather than binding only to localhost.
  • DomainFix: DomainFix requires combining code reasoning with specialized scientific knowledge, exemplified by sparse Cholesky support for positive definite sparse matrices.The example requires understanding the permutation relation P A P^T = L L^T and compatibility with sparse representations.
  • DepMigrate: DepMigrate tests repository-wide migration reasoning under breaking dependency changes, including compatibility with upgraded package versions and Azure-specific behavior.The example updates an LM client for openai>=1.0 while handling changed model, engine, and deployment_id semantics.
  • Doc2Repo: Doc2Repo starts from an empty workspace and asks agents to infer repository structure, APIs, dependencies, and behavior from a natural-language specification.The example constructs a document-processing library that converts PDFs, Office files, images, and HTML into Markdown.
Loading 2603.03194v2…