PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios

Tue Le, Minh V. T. Thai, Dung Nguyen Manh, Huy Phan Nhat, Nghi D. Q. Bui

arXiv:2512.18470v6cs.SEcs.AIcs.MA

TL;DR

Existing benchmarks largely assess isolated software issues, leaving sustained evolution from high-level requirements insufficiently evaluated. SWE-EVO constructs release-level, multi-file tasks from mature Python repositories and measures agents against comprehensive tests. The benchmark reveals a large capability gap between SWE-Bench Verified and software-evolution performance, while highlighting scope and statistical limitations that constrain interpretation.

  • Problem

    Existing benchmarks emphasize isolated issue resolution, whereas real software evolution requires coordinated changes across modules, versions, and specifications.

  • Method

    SWE-EVO uses release notes and version histories from mature open-source Python repositories to create release-level tasks requiring multi-step codebase evolution and validation.

  • Results

    25% is the best reported SWE-EVO resolution rate for gpt-5.4, while gpt-5.2 falls from 72.80% on SWE-Bench Verified to 22.92% on SWE-EVO.

  • Takeaways & Limitations

    SWE-EVO complements existing benchmarks by targeting long-horizon software evolution rather than isolated bug fixing.

  • Takeaways & Limitations

    SWE-EVO currently covers Python library projects, release-note-centered specifications, and 48 curated instances with imbalanced repository distribution.

Abstract

from arXiv · show

Existing benchmarks for AI coding agents focus on isolated, single-issue tasks such as fixing a bug or adding a small feature. However, real-world software engineering is a long-horizon endeavor: developers interpret high-level requirements, coordinate changes across many files, and evolve codebases over multiple iterations while preserving functionality. We introduce SWE-EVO, a benchmark for this long-horizon software evolution challenge. Constructed from release notes of seven mature open-source Python projects, SWE-EVO comprises 48 tasks requiring multi-step modifications spanning an average of 21 files, validated against test suites averaging 874 tests per instance. Experiments reveal a striking capability gap: GPT-5.4 with OpenHands achieves only 25% on SWE-EVO versus 72.80% achieved by GPT-5.2 on SWE-Bench Verified, showing that current agents struggle with sustained, multi-file reasoning. We also propose Fix Rate, a metric capturing partial progress on these complex, long-horizon tasks.

1. Introduction

Existing coding benchmarks emphasize isolated issue resolution, while SWE-EVO evaluates sustained software evolution from release-level requirements. Its experiments expose a substantial gap between strong performance on SWE-Bench Verified and performance on multi-file evolution tasks.

  • Motivation and benchmark proposal: SWE-EVO evaluates release-level software evolution, requiring agents to interpret requirements and coordinate changes across an existing codebase.Tasks use release notes and versioned snapshots rather than single-issue repair.
  • Benchmark design: SWE-EVO’s pipeline combines release notes and a start-version codebase with multi-file patch generation and comprehensive regression-aware testing.Tests include FAIL→PASS checks for requested changes and PASS→PASS checks for preserved behavior.
  • Motivation and benchmark proposal: SWE-Bench focuses on discrete GitHub issues, whereas SWE-EVO combines multiple fixes, features, and maintenance changes into one evolution task.The benchmark targets continuous system evolution, which existing isolated-task evaluations do not capture.
  • Results: 25% is the best reported SWE-EVO resolution rate for gpt-5.4, while gpt-5.2 falls from 72.80% on SWE-Bench Verified to 22.92% on SWE-EVO.The evaluation covers OpenHands, SWE-agent, and 18 models.

2. Related Work

Related work has progressed from function-level code generation and isolated repository issues toward interactive, autonomous, and multi-agent software engineering systems. Long-horizon agents additionally depend on context management across large codebases and many turns.

  • Repository-level evaluation: Repository-level benchmarks extended code-generation evaluation beyond earlier function-level and largely single-file tasks.SWE-Bench introduced verifiable patches for real GitHub issues, while newer benchmarks broaden code understanding and documentation evaluation.
  • Coding agents: Interactive and autonomous coding systems explored bug fixing, agent–computer interfaces, AST-based search, localization-repair pipelines, and multi-agent role specialization.These systems established several architectures for practical software-engineering agents.
  • Long-horizon context management: Long-horizon agents require context management spanning retrieval, processing, compression, structured retrieval, and memory systems.This requirement is central when tasks unfold across large codebases and many turns.

3. SWE-EVO Dataset

SWE-EVO is a release-level benchmark that evaluates whether agents can interpret requirements, coordinate multi-file changes, and preserve behavior across software evolution. It constructs 48 validated tasks from mature open-source Python repositories and introduces Fix Rate to capture partial progress.

  • 3.1. Benchmark Construction: SWE-EVO constructs release-level evolution tasks from release notes and version histories, requiring agents to implement behavioral changes across codebases rather than resolve isolated issues.Tasks are defined by the release-note delta between tagged repository versions, without an oracle decomposition into separate pull requests.
  • 3.1. Benchmark Construction: 48 high-confidence instances span 7 repositories and are manually checked for alignment among release notes, linked context, gold patches, and executable tests.The benchmark retains candidates after execution-based filtering and quality-focused human verification.
  • 3.1. Benchmark Construction: SWE-EVO is consistently larger than SWE-Bench in specification length, edited scope, and test-suite size across both average and worst-case instances.The benchmark therefore requires broader requirement interpretation, coordinated changes across more code, and heavier verification.
  • 3.2. Task Formulation: The benchmark supplies a release-note-centered specification and a complete pre-release codebase, then evaluates the generated patch with Resolved Rate, Patch Apply Rate, and Fix Rate.Resolved Rate is binary, while Fix Rate measures the fraction of FAIL_TO_PASS tests fixed subject to preserving PASS_TO_PASS behavior.
  • 3.2. Task Formulation: Fix Rate captures partial progress while assigning zero when any PASS_TO_PASS regression occurs, aligning soft progress measurement with behavior preservation.It remains consistent with Resolved Rate, where an instance is resolved when Fix Rate equals 1.
  • 3.3. Features of SWE-EVO: SWE-EVO instances may aggregate multiple pull requests, and 81% contain two or more, with an additional mean of 793 PASS_TO_PASS regression tests per instance.This broadens task scope beyond SWE-Bench’s single-pull-request framing and strengthens regression checking.

4. Experiments

SWE-EVO evaluates coding agents on release-scale software evolution using multiple agent scaffolds and models, revealing substantial performance gaps, failure patterns, and difficulty-dependent behavior.

  • Experimental Setup: The evaluation uses OpenHands and SWE-agent with 18 models spanning proprietary and open-weight systems, under bounded iteration or call budgets.OpenHands uses CodeActAgent with up to 100 iterations, while SWE-agent allows up to 100 LLM calls.
  • Performance on SWE-EVO: 25% is the best reported SWE-EVO resolution rate, while gpt-5.2 falls from 72.80% on SWE-Bench Verified to 22.92% on SWE-EVO.The comparison indicates a large capability gap between isolated issue repair and long-horizon evolution tasks.
  • Context Comparison Results: Release-note-only evaluation causes a modest performance drop while preserving overall trends, indicating that SWE-EVO remains challenging with less specification context.The default setting provides release notes with linked PR or issue text, whereas the harder setting removes that additional context.
  • Fine-Grained Analysis with Fix Rate: Fix Rate reveals partial progress hidden by Resolved Rate: gpt-4.1 and gpt-oss-120b both resolve 2.08%, but their Fix Rates are 4.65% and 2.08%.The soft metric measures the fraction of FAIL_TO_PASS tests fixed, distinguishing repair progress on large test suites.
  • Failure Analysis: Over 60% of gpt-5 failures are attributed to Instruction Following, whereas smaller models show more implementation, tool-use, and syntax failures.Older models more often exhibit looping and early-termination issues, while the taxonomy separates semantic from low-level execution failures.
  • Difficulty Analysis: Harder instances average 14.84 linked PRs versus 1.67 for the easiest group, and stronger models spend more turns on harder instances.The analysis uses resolution-based difficulty groups and finds less adaptive turn allocation among weaker models.

5. Conclusion

SWE-EVO benchmarks realistic software evolution rather than isolated bug fixing, requiring release-note interpretation, coordinated multi-file changes, and comprehensive validation. Experiments show that current agents achieve limited success on these long-horizon tasks.

  • Conclusion: SWE-EVO complements existing benchmarks by focusing evaluation on long-horizon software evolution rather than isolated issue resolution.The benchmark uses OpenHands and SWE-agent across 18 models and exposes distinct failure patterns across model families.
  • Conclusion: 25% is the best reported SWE-EVO task resolution rate, while matched gpt-5.2 performance drops from 72.80% on SWE-Bench Verified to 22.92% on SWE-EVO.The benchmark targets release-note-centered specifications, multi-file changes, and functionality preservation under comprehensive tests.

6. Limitations

The benchmark’s scope and statistical power are constrained by its Python-only, release-note-centered design, limited instance count, repository imbalance, and narrow Fix Rate measurement.

  • Limitations: SWE-EVO currently covers only Python library projects, leaving multilingual and downstream-application evolution for future work.The authors chose this scope for reproducibility and compatibility with SWE-Bench and SWE-Gym environments.
  • Limitations: The 48 curated instances and imbalanced repository distribution limit statistical power for fine-grained comparisons.The benchmark prioritizes large, execution-validated release transitions over many shallow tasks.
  • Limitations: Release-note-centered specifications omit evolution drivers such as security advisories, dependency updates, and design-document-driven refactors.Tasks may optionally include linked PR or issue text but do not represent all sources of software evolution requirements.
  • Limitations: Fix Rate weights tests equally and does not measure code quality, maintainability, or patch minimality.Although execution-verifiable, the metric does not capture these broader properties of software changes.

E. Context Comparison Results

Providing linked PR or issue context generally improves resolved rates while preserving model rankings and relative trends.

  • Context Comparison Results: Adding PR or issue context generally improves resolved rates across most models while preserving the overall ranking and relative trends.The comparison contrasts release-note-only evaluation with release notes augmented by linked PR or issue text.

F. Pull Request Distribution Analysis

SWE-EVO instances can include multiple linked pull requests, unlike SWE-Bench’s single-PR instances, reflecting multi-step development efforts and varied difficulty.

  • F. Pull Request Distribution Analysis: SWE-EVO instances may associate multiple pull requests with one release-note change, collectively implementing or refining it.This contrasts with SWE-Bench, where each instance corresponds to a single pull request.
  • F. Pull Request Distribution Analysis: The number of pull requests per instance varies widely, indicating a broad range of difficulty levels.

G. Difficulty Analysis

The appendix points readers to the main paper’s difficulty analysis, which reports results in Figure 6 and Section 4.3.2.

  • G. Difficulty Analysis: The main paper reports SWE-EVO’s difficulty analysis in Figure 6 and Section 4.3.2.This appendix section serves as a pointer rather than duplicating the figure.

H. Repository Composition and Bias Checks

Repository-level evaluation is constrained by an uneven repository composition, despite strict collection procedures designed to improve benchmark validity.

  • H. Repository Composition and Bias Checks: 26 of 48 SWE-EVO instances come from dvc, creating quantity imbalance across repositories.The passage identifies this imbalance as a consequence of requiring stable release tags, reproducible environments, and executable tests.
  • H. Repository Composition and Bias Checks: The collection process requires stable release tags, reproducible environments, and executable tests to improve validity.

I. Fix Rate Results

Fix Rate complements binary resolution by measuring partial progress on SWE-EVO’s large, multi-test tasks, revealing differences that Resolved Rate can obscure.

  • I. Fix Rate Results: Fix Rate distinguishes models with identical 2.08% Resolved Rates: gpt-4.1 reaches 4.65% versus 2.08% for gpt-oss-120b under OpenHands.The difference indicates that gpt-4.1 fixes more failing tests per instance despite equal binary resolution.
  • I. Fix Rate Results: Fix Rate and Resolved Rate differ consistently across models, showing that many trajectories make partial but incomplete progress.This finer granularity is especially relevant because SWE-EVO tasks involve many tests.
  • I. Fix Rate Results: The PR-count distribution spans instances with different numbers of linked pull requests, reflecting varied task complexity.
  • I. Fix Rate Results: FAIL_TO_PASS and PASS_TO_PASS rates separate implementing the requested change from preserving tests that already passed.Across most models, higher PASS_TO_PASS rates tend to coincide with lower FAIL_TO_PASS rates, suggesting a trade-off.

M. Dataset Fields

SWE-EVO task instances are represented with release-note-centered specifications, optional linked PR/issue context, a complete pre-release codebase, and structured dataset fields. The benchmark also records executable tests and model-generated patches for evaluating repository-wide changes.

  • SWE-EVO provides a release-note-centered specification, linked PR/issue context when available, and the complete codebase at the pre-release commit.The model must implement the described change through edits represented as a patch across files.
  • Each task defines its requested evolution from official release notes describing changes between a start version and an end version.The release note is presented as the primary natural-language specification.
  • Linked pull requests and issues are fetched and appended as an unrewritten PR / Issue Context block beneath the release note.This preserves the original artifact text while making references actionable for the agent.
  • Table 9 specifies the required fields for a typical task instance, including fields newly added relative to SWE-Bench.The dataset documentation describes how these fields are obtained during curation.
  • The dataset examples pair release-transition problem statements with concrete release-note items and linked issue or pull-request content.Examples cover changes such as preventing silent remote overwrites, handling Azure access, and fixing Dask behavior.
Loading 2512.18470v6…