Source-linked AI summary
SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Vijay Bharadwaj, Jeff Holm, Raja Aluri, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, Brad Kenstler
TL;DR
SWE-BENCH PRO targets the lack of contamination-resistant, realistic evidence about AI agents solving complex software-engineering tasks. It constructs a large, diverse benchmark with human augmentation and verification, finding a substantial gap between performance on SWE-BENCH PRO and easier benchmarks while exposing complexity-related degradation. The benchmark’s coverage is limited by uneven language representation and reliance on test-suite verification.
Problem
Existing coding benchmarks face contamination risks and may underrepresent enterprise software engineering, where tasks often require substantial multi-file modifications.
Method
SWE-BENCH PRO combines copyleft and private startup repositories with commit-based problem sourcing, human augmentation, verified tests, and reproducible environments.
Results
Top-tier models achieve 23% success on SWE-BENCH PRO compared with over 70% on SWE-Bench Verified, while performance declines sharply as task file count increases.
Takeaways & Limitations
SWE-BENCH PRO provides a contamination-resistant baseline for measuring progress toward agents capable of professional, real-world software engineering.
Takeaways & Limitations
The benchmark underrepresents Java, C++, and Rust and relies on test suites that may not recognize all valid solutions.
Abstract
from arXiv · showhide
We introduce SWE-Bench Pro, a substantially more challenging benchmark that builds upon the best practices of SWE-BENCH [25], but is explicitly designed to capture realistic, complex, enterprise-level problems beyond the scope of SWE-BENCH. SWE-BENCH PRO contains 1,865 problems sourced from a diverse set of 41 actively maintained repositories spanning business applications, B2B services, and developer tools. The benchmark is partitioned into a public set with open access to problems sourced from 11 repositories, a held-out set of 12 repositories and a commercial set of 18 proprietary repositories where we have formal partnership agreements with early-stage startups. Problems in the held-out and the commercial set are not publicly accessible, but we release results on the commercial set. Our benchmark features long-horizon tasks that may require hours to days for a professional software engineer to complete, often involving patches across multiple files and substantial code modifications. All tasks are human-verified and augmented with sufficient context to ensure resolvability. To better understand these limitations, we cluster the failure modes observed in the collected agent trajectories for a clearer characterization of the error patterns exhibited by current models. Overall, SWE-BENCH PRO provides a contamination-resistant testbed that more faithfully captures the complexity and diversity of real-world software development, advancing the pursuit of truly autonomous software engineering agents at a professional level.
1. Introduction
SWE-BENCH PRO addresses contamination and realism gaps in coding benchmarks by combining restricted data sources with challenging, diverse, human-verified tasks.
- Existing benchmarks risk contamination because permissively licensed repositories may appear in web-crawled pretraining data.
- 161 of 500 SWE-Bench Verified problems require only one- to two-line modifications, unlike industrial tasks often requiring multi-file changes spanning hundreds of lines.
- SWE-BENCH PRO uses GPL repositories for public and held-out sets plus private startup codebases for its commercial set to mitigate contamination.
- Reference solutions average 107.4 lines across 4.1 files, with every problem requiring at least 10 changed lines and over 100 tasks requiring more than 100.
- Repositories span consumer applications, B2B services, and developer tooling, while capping each repository at 100 instances to reduce overfitting.
- A three-stage human-in-the-loop workflow clarifies ambiguity, adds missing context, and recovers unit tests as robust verifiers while preserving implementation flexibility.
2. Related Work
Software-engineering evaluation progressed from function-level benchmarks to repository-level issue resolution, exposing limitations in models’ ability to handle complete codebases.
- HumanEval introduced 164 handwritten programming problems as a foundational functional-correctness benchmark.
- MBPP added approximately 1,000 crowd-sourced Python problems designed for entry-level programmers.
- APPS expanded algorithmic evaluation to 10,000 programming problems ranging from simple to complex challenges.
- SWE-bench pioneered repository-level evaluation with 2,294 real GitHub issues from 12 Python repositories, revealing weak performance on all but the simplest issues.
- Multi-SWE-bench extended repository-level evaluation across seven programming languages with 1,632 expert-curated instances.
- SWE-agent and AutoCodeRover illustrate complementary advances through agent-computer interfaces and AST-based code search.
3. Dataset Overview
SWE-BENCH PRO is an industrially oriented benchmark of human-verified, contamination-resistant repository tasks, with explicit task context, tests, environments, and multiple access subsets.
- Repositories are actively maintained professional projects spanning consumer applications, B2B platforms, and developer tools, including private startup repositories.
- GPL and other copyleft repositories, together with private commercial repositories, are used to reduce the likelihood of benchmark content entering proprietary training corpora.
- Each task includes a human-augmented problem statement, requirements, interface, relevant tests, and a working environment for patch evaluation.
- The benchmark evaluates generated patches using human-reviewed fail2pass tests for issue resolution and pass2pass tests for preserving existing functionality.
- SWE-BENCH PRO contains 1,865 human-verified and augmented problems divided into public, commercial, and held-out subsets.
- The commercial subset contains 276 private startup-repository problems, with results reported publicly despite legal restrictions on releasing the code.
- The held-out subset contains 858 problems from separate repositories and is retained for future overfitting checks.
4. Dataset Creation
Problems are sourced from commit histories and then augmented, tested, and packaged in reproducible environments before evaluation on public and commercial subsets.
- Problem collection identifies consecutive commits whose evolution captures issue resolution.
- Figure 2 characterizes public tasks as complex, long-horizon problems spanning several files and bug fixes or feature requests across multiple domains.
- The test patch contains test-file changes between commits, while the remaining non-test diff is designated the gold patch.
- Human-driven augmentation adds sufficient context because scraped commit messages may be incomplete, unstructured, or missing.
- Experts manually construct Docker environments, automated checks test consistency and flakiness, and humans remove irrelevant or overly broad tests.
- Evaluation reports model performance separately for the public and commercial subsets, including a public-set sample size of 731 and a commercial-set sample size of 276.
5. Results
The evaluation uses SWE-Agent to test models under standardized, low-ambiguity conditions on public and commercial SWE-BENCH PRO tasks. Claude Sonnet 4.5 achieves the highest public resolve rate, while commercial performance remains below 20%.
- Evaluation settings: SWE-Agent is the primary scaffold, while Agentless is excluded from the main results because it struggles with multi-file editing.
- Evaluation settings: Models receive the problem statement, requirements, and interface specification, so evaluation focuses on implementing specified repairs rather than resolving ambiguity.
- Evaluation sets: Evaluations cover the public and commercial sets, while the held-out private set is retained for future analysis.
- Results: 43.6% is Claude Sonnet 4.5’s Pass@1 resolve rate, the highest reported result, followed by Claude Sonnet 4 at 42.7%.
- Results: Less than 20% is the best-model resolve rate on the commercial set, compared with substantially higher results for leading models on the public set.
6. Analysis
Performance varies substantially by language, repository, and task complexity, with multi-file problems exposing sharp weaknesses in smaller models. Failure analyses further distinguish semantic, contextual, syntactic, and tool-use limitations across models.
- Model performance: Go and Python generally produce higher resolve rates, whereas JavaScript and TypeScript range from near 0% to over 30% depending on the model.
- Model performance: Some repositories remain below 10% resolve across models, while others exceed 50% for certain models, indicating strong repository-level variation.
- Model performance: Resolve rates decline sharply as file count increases; frontier models remain above 10% on 10+ file problems while open-source alternatives approach zero.
- Model performance: Frontier models maintain more consistent performance across repositories and languages, while smaller models show erratic, sometimes near-zero results.
- Human augmentations: Without human augmentations, GPT-5 and Claude Opus 4.1 show significantly degraded performance because test verifiers can otherwise produce false negatives.
- Trajectory failure modes: Failure patterns differ by model: Opus 4.1 most often produces wrong solutions, Sonnet 4 encounters context overflow, and Qwen3 32B has a 42.0% tool error rate.
7. Limitations and Future Work
The benchmark has limited coverage of some major programming languages and relies on test-suite verification, while future work proposes broader evaluation and collaborative development scenarios.
- Limitations: Java, C++, and Rust are underrepresented, limiting assessment across the full spectrum of modern software development.
- Limitations: The benchmark relies on fail2pass and pass2pass tests, although valid solutions may exist that do not pass the original task tests.
- Future work: Future evaluation could add rubrics, code-quality, security, performance, maintainability, readability, and architectural-soundness assessments.
- Future work: Future benchmarks could include multi-agent coordination and human-agent collaboration, including code reviews, merge conflicts, and distributed workflows.
8. Conclusion
SWE-BENCH PRO is presented as a more rigorous and realistic benchmark for AI coding agents, combining challenging tasks, contamination prevention, and diagnostic analyses. Top-tier models achieve 23% success, revealing a gap between current agent capabilities and professional software engineering demands.
- SWE-BENCH PRO provides a more accurate measure of progress by reflecting the complexity of professional software engineering.
- The benchmark combines diverse real-world tasks, challenging multi-file changes, and strict contamination prevention.
- Trajectory-level failure analyses characterize specific limitations exhibited by current models.
B. Example Task Instance
The example task instance section explains how SWE-BENCH PRO represents tasks through realistic, rewritten problem statements. These statements preserve repository conventions while improving specification.
- Problem statements describe the task an agent must complete in a codebase and follow GitHub Issue-like markdown conventions.
- Problem statements retain repository-specific default issue templates to approximate real-world task distributions.
- Statements are curated from existing commits, issues, and pull requests, then rewritten to be well-specified.
B.1.1 Example
The Open Library example illustrates how SWE-BENCH PRO specifies a realistic feature request, its motivation, requirements, and interfaces. The task adds Google Books as a fallback metadata source while defining testable behavior across the import pipeline.
- Problem / Opportunity: The feature addresses incomplete metadata that can reduce import success and leave poor-quality Open Library records.
- Justify: Why should we work on this and what is the measurable impact?: The proposed integration is expected to improve imported-book completeness, reduce sparse-metadata failures, and reduce placeholder entries.
- Requirements: Google Books responses must be normalized into Open Library fields, while ambiguous multiple-result responses are logged and skipped.
- Example: Google Books is added as a fallback metadata provider when Amazon lookup fails or only an ISBN-13 is available.
- Define Success: How will we know when the problem is solved?: Requirements specify observable behavior without prescribing implementation, including exact tested error messages when necessary.
- Requirements: The import pipeline must recognize google_books as a valid staged source and use the specified staging URL and batch workflow.
- Interface: The interface field helps prevent false negatives when tests expect particular public class and function names or signatures.
B.3.1 Example
The interface example lists the public classes and functions associated with the Open Library feature, including Google Books fetching, processing, staging, batching, and worker execution. These entries connect expected interfaces to the feature’s implementation and tests.
- B.3.1 Example: The example identifies Google Books functions for fetching raw API data, normalizing edition fields, and staging successful metadata.
- B.3.1 Example: get_current_batch retrieves or creates the batch object used for staging import items.
- B.3.1 Example: BaseLookupWorker and AmazonLookupWorker define queue processing, batching, and API-timing behavior for lookup workers.
C. Trajectory Failure Mode Analysis
The trajectory failure analysis examines why an agent terminated, using recorded actions, observations, exit conditions, and a predefined error taxonomy. The interaction includes tools for executing commands, viewing and editing files, searching the workspace, and submitting a result.
- C. Trajectory Failure Mode Analysis: Agents can use bash, file viewing, workspace search, and persistent editing tools to inspect and modify repository files.Editing supports replacement, insertion, block editing, file creation, and undo operations.
- C. Trajectory Failure Mode Analysis: The trajectory summary records total steps and whether a successful patch was generated.A failed final state is explicitly represented as no successful patch generated.
- C. Trajectory Failure Mode Analysis: The analysis asks for an explanation of the issue and why the trajectory failed, followed by exactly one error category.The category is placed after the explanation and selected from a predefined set.
- C. Trajectory Failure Mode Analysis: The failure taxonomy includes incorrect file identification, missed edge cases, misunderstood requirements, wrong solutions, tool errors, infinite loops, and endless file reading.The taxonomy is intended to classify the reason an agent failed rather than merely record termination.