PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?

Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Vijay Bharadwaj, Jeff Holm, Raja Aluri, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, Brad Kenstler

arXiv:2509.16941v2cs.SEcs.CL

TL;DR

Existing coding benchmarks face contamination risks and may not capture the complexity of enterprise software engineering. SWE-Bench Pro curates diverse, substantial multi-file tasks from GPL-licensed repositories and private startup codebases. Top-tier models perform substantially worse on this benchmark: Opus 4.1 and GPT-5 achieve 23% success, compared with over 70% on SWE-Bench Verified.

  • Problem

    Existing coding benchmarks face contamination risks and may inadequately capture the substantial, multi-file changes common in enterprise software engineering.

    Source: §1 · pp. 1–2

  • Method

    SWE-Bench Pro curates diverse, substantial multi-file tasks from GPL-licensed repositories and private startup codebases, using licensing and restricted access to mitigate contamination.

    Source: §1 · p. 2

  • Results

    Top-tier models perform substantially worse on SWE-Bench Pro: Opus 4.1 and GPT-5 achieve 23% success, compared with over 70% on SWE-Bench Verified.

    Source: §8 · p. 9

  • Takeaways & Limitations

    SWE-Bench Pro provides a baseline for measuring progress and guiding research toward autonomous software engineering agents.

    Source: §8 · p. 10

    Test-suite verification may reject valid alternative solutions that do not pass the task’s original tests.

    Source: §7.1 · p. 9

Abstract

from arXiv · show

We introduce SWE-Bench Pro, a substantially more challenging benchmark that builds upon the best practices of SWE-BENCH [25], but is explicitly designed to capture realistic, complex, enterprise-level problems beyond the scope of SWE-BENCH. SWE-BENCH PRO contains 1,865 problems sourced from a diverse set of 41 actively maintained repositories spanning business applications, B2B services, and developer tools. The benchmark is partitioned into a public set with open access to problems sourced from 11 repositories, a held-out set of 12 repositories and a commercial set of 18 proprietary repositories where we have formal partnership agreements with early-stage startups. Problems in the held-out and the commercial set are not publicly accessible, but we release results on the commercial set. Our benchmark features long-horizon tasks that may require hours to days for a professional software engineer to complete, often involving patches across multiple files and substantial code modifications. All tasks are human-verified and augmented with sufficient context to ensure resolvability. To better understand these limitations, we cluster the failure modes observed in the collected agent trajectories for a clearer characterization of the error patterns exhibited by current models. Overall, SWE-BENCH PRO provides a contamination-resistant testbed that more faithfully captures the complexity and diversity of real-world software development, advancing the pursuit of truly autonomous software engineering agents at a professional level.

1. Introduction

SWE-BENCH PRO targets contamination risk and unrealistic task simplicity through protected data sources, substantial multi-file tasks, and human verification. It combines differentiated data-access policies with standardized evaluation and trajectory-level failure analysis.

1. Introduction

  • 161 out of 500 SWE-Bench Verified problems require only one- to two-line modifications, contrasting with enterprise changes spanning hundreds of lines across files.Existing benchmarks also face contamination risks because public repositories may already be accessible as training data.Source: §1 · pp. 1–2
  • SWE-BENCH PRO mitigates contamination through GPL-licensed repositories and private startup codebases, with different access policies across three subsets.The public set contains 11 repositories and releases problems and results; the 12-repository held-out set remains private for future overfitting checks. The commercial set contains 18 repositories and releases results while keeping codebases private.Source: §1 · p. 2
  • 107.4 lines across 4.1 files characterize the average reference solution, reflecting the benchmark’s emphasis on substantial, multi-file modifications.Every problem requires at least 10 lines of change, and over 100 tasks require more than 100 lines of modification.Source: §1 · p. 2
  • 50–100 instances per repository balance representation across actively maintained consumer applications, B2B services, and developer tools, limiting concentration in any single repository.A strict cap of 100 instances per repository is intended to reduce repository-specific overfitting.Source: §1 · p. 2
  • A three-stage human-in-the-loop workflow clarifies ambiguous tasks, adds missing context, and makes unit tests robust verifiers while preserving implementation flexibility.The workflow preserves core technical challenges and constrains solution spaces to avoid false negatives.Source: §1 · p. 2
  • Standardized evaluation protocols and trajectory-level failure analyses complement transparent curation to assess coding-agent progress beyond SWE-Bench Verified saturation.The held-out set is reserved to monitor potential overfitting to the public set.Source: §1 · p. 2

2. Related Work

Code evaluation has evolved from function-level problems to repository-level challenges, while software engineering agents have emphasized interface design and AST-based code search.

2.1 Code and Software Engineering Benchmarks

  • 164 handwritten problems in HumanEval [4], approximately 1,000 crowd-sourced Python problems in MBPP [2], and 10,000 programming problems in APPS [9] characterize earlier code benchmarks.HumanEval established a standard for functional-correctness evaluation, MBPP targets entry-level programmers, and APPS spans simple to complex algorithmic challenges.Source: §2.1 · p. 2
  • 2,294 GitHub issues from 12 Python repositories underpin SWE-bench, while Multi-SWE-bench [18] extends repository-level evaluation with 1,632 expert-curated instances across seven languages.SWE-bench revealed that state-of-the-art models resolved only the simplest issues. Multi-SWE-bench covers Java, TypeScript, JavaScript, Go, Rust, C, and C++; Da et al. [6] show these instances also support RL training.Source: §2.1 · p. 2

2.2 Software Engineering Agents

  • 12.5% resolution rate on SWE-bench was achieved by SWE-agent [16], while AutoCodeRover [21] achieved 19% on SWE-bench-lite.SWE-agent emphasizes agent-computer interfaces, showing that interface design can matter as much as model capabilities. AutoCodeRover combines LLMs with AST-based code search while maintaining low operational costs.Source: §2.2 · p. 3

3. Dataset Overview

SWE-BENCH PRO combines diverse professional repositories and substantial, multi-file tasks with human-augmented specifications and human-reviewed tests. Its 1865 problems are divided into public, commercial, and held-out sets.

3.1 Characteristics of SWE-BENCH PRO

  • 50-100+ instances per repository aim to discourage repository specialization, while tasks require substantial, multi-file changes across diverse professional projects.Actively maintained repositories have substantial user bases, comprehensive documentation, and established development practices. They span consumer applications with complex interfaces, B2B platforms with intricate business rules, and developer tools with sophisticated APIs.Source: §3.1 · p. 3
  • Copyleft licensing and private startup repositories underpin the benchmark’s contamination-resistant design.The paper argues that GPL and other copyleft licenses create legal barriers to inclusion in proprietary model training corpora; purchased startup repositories are private.Source: §3.1 · p. 3

3.2 Task Specification

  • Human-augmented task descriptions combine rewritten problem statements, test-grounded requirements, and explicit interfaces to support patch generation without missing specifications.Statements draw on commits, pull requests, and issues; requirements clarify ambiguity, while expected class and function names prevent interface-related false negatives when relevant. Human verification checks that descriptions contain critical information and tests are well specified.Source: §3.2 · p. 3, §3.1 · p. 3
  • Human-reviewed fail2pass tests verify issue resolution, while pass2pass tests check preservation of existing functionality in containerized, language-specific environments.Environments provide full dependency resolution using isolated Python virtual environments, Node.js with npm/yarn for JavaScript/TypeScript, and module-aware Go environments with GOPATH configuration. Pre-built Docker images are planned for release to ensure reproducibility.Source: §3.2 · p. 4

3.3 Public, Commercial, and Held-Out SWE-BENCH PRO

  • 1865 human-verified, augmented problems comprise 731 public, 276 commercial, and 858 held-out instances.Public instances are released on HuggingFace; commercial problems remain private for legal reasons, but results are reported publicly with leaderboard updates planned. Held-out problems mirror the public set using separate repositories and remain private for future overfitting checks.Source: §3.3 · p. 4

4. Dataset Creation

Each SWE-BENCH PRO problem combines a task description, relevant tests, and a working code environment. Dataset creation uses commit-history sourcing, human augmentation of task descriptions, and automated and human verification of environments and tests.

4.1 Sourcing Problems

  • Problems come from consecutive commits, with test-file changes forming the test patch and all remaining changes forming the gold patch.The older commit is the base and the newer commit the instance. Public-set tasks involve multiple files and include feature requests and bug fixes across optimization, security, UI/UX, and backend changes.Source: §4.1 · pp. 4–5, Figure 2 · p. 5

4.2 Creating Task Descriptions

  • Human augmentation supplies source-grounded problem statements, behavioral requirements, and optional interfaces, providing sufficient context without prescribing implementation.Annotators organize issue discussions, commit messages, and pull requests, grounding requirements in the gold patch and test expectations when applicable. This enables problem construction beyond existing GitHub issues or pull requests.Source: §4.2 · p. 5

4.3 Creating Environments

  • Expert-built Docker environments undergo repeated automated verification and human test review, excluding unreliable problems and tests that are irrelevant or too broad.Engineers incorporate system packages, repository documentation, build tools, and dependencies into Dockerfiles so agents can run code and tests out of the box. Gold tests must pass consistently; problems are also dropped if no relevant, sufficiently focused tests remain.Source: §4.3 · pp. 5–6

5. Results

Leading SWE-BENCH PRO resolve rates reach 43.6%, while the best models score below 20% on the commercial set. Evaluations use SWE-Agent [16] and detailed repair specifications.

Scaffold

  • Results focus on SWE-Agent [16] because Agentless [14] struggles with multi-file editing and consequently produces low evaluation scores.The evaluated suite includes frontier models, open-weight models, and models fine-tuned on SWE-bench-like trajectories.Source: §5 · p. 6

Evaluation settings

  • Models share the default prompt from [16], a maximum of 50 turns, and versions current as of September 18th, 2025.Open-source models run through vllm on one node with eight NVIDIA H100 GPUs; tool use is enabled where possible, with syntax parsing for open-weight models. Opus 4.1 runs without extended thinking.Source: §5 · p. 6

Issue Ambiguity

  • Agents receive the problem statement, requirements, and interface specification, testing implementation of specified repairs rather than ambiguity resolution.The evaluation explicitly uses a setting intended to eliminate issue ambiguity by supplying significant repair details.Source: §5 · p. 6

Results

  • 43.6% and 42.7% Pass@1 resolve rates place CLAUDE SONNET 4.5 and CLAUDE SONNET 4 highest, respectively, followed by OPENAI GPT-5 (HIGH) at 41.8%.CLAUDE HAIKU 4.5 achieves 39.5%. Earlier-generation KIMI K2 INSTRUCT and OPENAI GPT-OSS 120B achieve 27.7% and 16.2%, respectively.Source: §5 · p. 6

Evaluation sets

  • Below 20% resolve rates for the best models on the commercial set reveal a substantial performance gap from the public set.Evaluations cover public and commercial sets, but all analysis uses the public set to avoid potential commercial-set leakage. The private set remains held out for future analysis.Source: §5 · p. 6

6. Analysis

Under a 50-turn, $2 limit, SWE-BENCH PRO analysis finds that model performance varies across languages, repositories, and task complexity, with distinct failure patterns across models. Human-provided requirements and interfaces improve task resolvability by reducing verifier false negatives.

6.1 Analysis on Model Performance on SWE-BENCH PRO

  • Above 10% resolve rates persist for Claude Opus 4.1 and OpenAI GPT-5 on 10+ file problems, while open-source alternatives approach zero.Performance declines as file counts increase, and the gap between frontier and smaller models widens beyond three files. Categories involving 10+ files contain about 20–30 examples and therefore have higher variance.Source: Figure 3 · p. 7, §6.1 · p. 7
  • Above 30% resolve rates occur for some models in Go and Python; JavaScript and TypeScript range from near 0% to over 30%, depending on model.Go and Python generally have higher resolve rates across most models.Source: §6.1 · p. 7
  • Below 10% resolve rates occur across all models on some repositories, while other repositories allow certain models to exceed 50%.Claude Opus 4.1 and OpenAI GPT-5 perform relatively consistently across repositories and languages; smaller models show more erratic patterns, including near-zero resolve rates on certain repositories.Source: §6.1 · pp. 7–8

6.2 Ablation: Removing Human Augmentations

  • Removing requirements and interface augmentations significantly degrades GPT-5 and CLAUDE OPUS 4.1 performance compared with providing both alongside the problem statement.Without these constraints, agents can produce diverse solutions that fail tests expecting specific APIs, signatures, or behavior. Human augmentations narrow the solution space and mitigate these verifier false negatives.Source: §6.2 · p. 8, Table 3 · p. 8

6.3 Trajectory Failure Modes

  • 35.6% of SONNET 4 failures involve context overflow, while 17.0% involve endless file reading.GEMINI 2.5 failures include tool errors (38.8%), syntax errors (30.5%), and wrong solutions (18.0%). QWEN3 32B exhibits the highest tool error rate (42.0%).Source: §6.3 · p. 8
  • 35.9% of OPUS 4.1 failures are wrong solutions and 24.2% are syntax errors; GPT-5 shows fewer wrong solutions but potential tool-use differences.Using hand-curated failure categories, GPT-5 judges unresolved trajectories from their last 20 turns, producing a reasoning paragraph before selecting one category. Twenty turns showed better correspondence with human validation than 10 or 40 turns.Source: §6.3 · p. 8, Table 4 · p. 9

7. Limitations and Future Work

SWE-BENCH PRO has limitations in language coverage and test-based verification. Future work proposes broader evaluation approaches and collaborative development scenarios.

7.1 Limitations

  • Uneven language coverage underrepresents Java, C++, and Rust, which may limit assessment across the full spectrum of modern software development.The benchmark includes Python, JavaScript, TypeScript, and Go, but their distribution is not uniform.Source: §7.1 · p. 9
  • Reliance on fail2pass and pass2pass tests may reject valid alternative solutions, motivating verifiers capable of accepting any valid solution.Correct solutions may not pass the original tests specified for a task.Source: §7.1 · p. 9

7.2 Future Work

  • Future evaluation could extend beyond tests to rubrics, code quality, security, performance, and adherence to software engineering best practices.Human evaluation could assess code maintainability, readability, and architectural soundness.Source: §7.2 · p. 9
  • Proposed collaborative tasks would require multiple agents or human-agent teams to coordinate through code reviews, merge conflict resolution, and distributed development workflows.These scenarios would reflect modern team-based software development practices.Source: §7.2 · p. 9

8. Conclusion · Appendix · A. Failure Mode Category Descriptions

SWE-BENCH PRO emphasizes diverse real-world tasks, multi-file changes, and contamination prevention; Opus 4.1 and GPT-5 achieve 23% success versus over 70% on benchmarks like SWE-Bench Verified. The appendix provides dataset-instance details and defines categories of agent failure.

8. Conclusion

  • 23% success for Opus 4.1 and GPT-5 on SWE-BENCH PRO, versus over 70% on benchmarks like SWE-Bench Verified, highlights a gap from real-world development demands.The conclusion presents this baseline as a more accurate measure of progress and a way to identify limitations requiring further research.Source: §8 · pp. 9–10
  • SWE-BENCH PRO combines diverse real-world tasks, challenging multi-file changes, and strict contamination prevention to better reflect professional software engineering.The conclusion positions the contamination-resistant testbed as a guide for research toward autonomous software engineering agents.Source: §8 · pp. 9–10

Appendix

  • The appendix provides additional details about example dataset instances.Source: Appendix · p. 13

A. Failure Mode Category Descriptions

  • Wrong solution covers syntactically valid but functionally incorrect, incomplete, or irrelevant patches; Syntax error covers edits that make the codebase uncompilable or unrunnable.These categories distinguish failures of patch functionality from failures of syntactic validity.Source: A. Failure Mode Category Descriptions · p. 13
  • Tool-Use failures prevent information gathering or correct changes through tool misuse; Incorrect file failures mislocate the modification target despite understanding the high-level goal.The latter category includes choosing the wrong source file or function.Source: A. Failure Mode Category Descriptions · p. 13
  • File Reading failures involve repetitive exploration without implementation: agents gather information but never translate it into code changes, eventually timing out or failing through inaction.Exploratory loops include repeatedly reading files, searching keywords, and viewing code snippets.Source: A. Failure Mode Category Descriptions · p. 13
  • Task-objective misinterpretation leads agents to pursue tangential or incorrect goals, producing coherent actions that nevertheless fail to address the actual issue.One example is constructing a complex runtime reproduction environment for a simple refactoring task.Source: A. Failure Mode Category Descriptions · p. 13
  • Other covers uncategorized failures, including inefficient workflows that exceed computational or time limits and combinations of minor, compounding issues.Excessively verbose testing can exhaust cost limits without a single clear technical mistake.Source: A. Failure Mode Category Descriptions · p. 13

B. Example Task Instance · B.1 Problem Statement

SWE-BENCH PRO problem statements define agents’ codebase tasks in GitHub Issue-style form. They are curated from existing development artifacts and rewritten to be well-specified, while aiming to preserve real-world repository conventions.

B. Example Task Instance

B.1 Problem Statement

  • The problem statement describes the task an agent must complete in the codebase.Source: §B.1 · p. 14
  • Problem statements follow the Markdown formatting and conventions found in common open-source repositories.Source: §B.1 · p. 14
  • Problem statements use the repository’s default issue templates for the specific task.Source: §B.1 · p. 14
  • Problem-statement creation aims to stay as close as possible to the real-world distribution.Source: §B.1 · p. 14
  • Problem statements are curated from existing commits, issues, and PRs, then rewritten to be well-specified.Source: §B.1 · p. 14

B.1.1 Example … B.3 Interface

SWE-BENCH PRO provides human-authored requirements and optional public-interface specifications to clarify unit-test expectations without leaking solutions. An Open Library example illustrates the behavior and integration requirements for adding Google Books as a fallback metadata source.

B.1.1 Example

  • The Open Library example proposes Google Books fallback metadata to address incomplete imports caused by missing or inadequate Amazon and ISBNdb metadata.The problem particularly affects ISBN-13-only records and less common or international titles. Expected benefits include higher import success rates and fewer placeholder entries, rather than reported experimental improvements.Source: §B.1.1 · p. 14
  • The feature requires updated metadata parsing, with automated tests checking field mapping, incomplete responses, and rejection of zero or multiple Google Books matches.Success also requires fetching and staging Google Books metadata using ISBN-13.Source: §B.1.1 · p. 14

B.2 Requirements

  • Human-authored requirements specify test-verifiable behavior and otherwise unavailable details without prescribing code implementations or leaking solutions.Requirements can specify exact error-log strings when unit tests explicitly check them.Source: §B.2 · pp. 14–15

B.2.1 Example

  • Integration requirements register google_books as a staged source, prescribe the staging URL, and preserve existing source identifiers when supplementing records.The URL is http://{affiliate_server_url}/isbn/{identifier}?high_priority=true&stage_import=true, using the affiliate server configured in openlibrary/core/vendors.py and accepting ISBN-10, ISBN-13, or B*ASIN. Existing source_records must be extended rather than replaced.Source: §B.2.1 · p. 15
  • Google Books staging must persist successful metadata through Batch.add_items, match Open Library’s import structure, and replace Amazon-only promise-item enrichment.Required fields include isbn_10, isbn_13, title, subtitle, authors, source_records, publishers, publish_date, number_of_pages, and description. Promise-batch enrichment must use stage_bookworm_metadata.Source: §B.2.1 · p. 15
  • The example requires Google Books fallback after unsuccessful Amazon ISBN-13 lookups only when both high_priority=true and stage_import=true are set.The broader proposal updates source handling and stages Google Books API metadata for import. Multiple Google Books matches must trigger a warning and skip staging.Source: §B.1.1 · p. 14, §B.2.1 · p. 15

B.3 Interface

  • Optional interface specifications identify modified or new public classes and functions, their signatures, and file paths to mitigate unit-test false negatives.Because benchmark tests remain unmodified, these specifications help prevent viable implementations from failing through unexpected class names or module paths.Source: §B.3 · p. 15

B.3.1 Example

This Open Library example documents interfaces for fetching, normalizing, and staging Google Books metadata, alongside import-batch management and threaded API lookup workers.

B.3.1 Example

  • fetch_google_book retrieves Google Books metadata for an ISBN-13 identifier, returning raw JSON for HTTP 200 responses and None otherwise.The returned dictionary contains the Google Books API response.Source: §B.3.1 · p. 16
  • process_google_book converts Google Books JSON into a dictionary of normalized Open Library edition fields, returning None if processing fails.Source: §B.3.1 · p. 16
  • stage_from_google_books accepts ISBN-10 or ISBN-13 identifiers, fetches Google Books metadata, and adds available metadata to an import batch.Located in scripts/affiliate_server.py, it returns True when staging succeeds and False otherwise.Source: §B.3.1 · p. 16
  • get_current_batch retrieves or creates named staging batches, while threaded lookup workers process queued items and batch Amazon requests under API timing constraints.BaseLookupWorker.run processes queued items in a loop; its subclass AmazonLookupWorker batches up to 10 Amazon identifiers.Source: §B.3.1 · p. 16

C. Trajectory Failure Mode Analysis · C.1 LLM-as-a-judge Prompt

The LLM-as-a-judge prompt requests an explanation of a failed software-engineering trajectory and exactly one predefined error category. It supplies trajectory context and tool specifications, with explicit output-format requirements.

C.1 LLM-as-a-judge Prompt

  • Tool specifications warn that shell listings and directory views can consume substantial context, while long editor outputs are truncated.The editor marks truncated output with <response clipped>.Source: §C.1 · p. 18
  • The tool descriptions cover file navigation, text and filename searches, file creation, and submission.Navigation supports opening files, jumping to lines, and scrolling; searches operate within files or directories. Submission can indicate either perceived completion or inability to proceed, not necessarily success.Source: §C.1 · pp. 18–19
  • Editing specifications require exact, unique replacement matches and reject Python block edits that fail syntax checks.The persistent editor also supports undoing the last edit and prevents creating a file at an existing file path.Source: §C.1 · pp. 18–19
  • The prompt supplies the problem statement, recent actions and observations, total steps, and failed final state, directing attention to the termination condition.The judge is instructed to consider how the final actions contributed to the specific exit condition.Source: §C.1 · pp. 19–20
  • The judge must explain why the trajectory failed and assign exactly one predefined error category, using “other” when none fits.The instructions prohibit inventing or proposing new categories.Source: §C.1 · p. 20
  • The taxonomy distinguishes incorrect file identification, missed edge cases, problem misunderstanding, wrong solutions, tool errors, looping, repeated file reading, listing-induced context overflow, syntax errors, and other failures.Endless file reading means repeatedly reading the same file without changes; listing-induced context overflow specifically concerns operations such as ls and find.Source: §C.1 · p. 20
  • The output must wrap the explanation in description tags and end with only the lowercase category name, separated by two newlines.The prescribed explanation delimiters are <description> and </description>.Source: §C.1 · p. 20

Related papers

Loading 2509.16941v2…