Source-linked AI summary
SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring
Yuling Shi, Jinghan Xu, Kelin Fu, Wenhao Zeng, Shilin He, Lei Zhang, Yue Liu, Zelin Zhao, Terry Yue Zhuo, Jialun Cao, Siyu Ye, Tianyu Liu, Kai Cai, Shing-Chi Cheung, Xiaodong Gu
TL;DR
Existing coding benchmarks face saturation, flawed tests, and limited refactoring coverage. SWE-Bench ProMax addresses these gaps with an expert-curated, multilingual benchmark of large-scale refactoring tasks, and GPT-5.2 resolves only 41.2% of instances.
Problem
Existing benchmarks increasingly saturate, contain flawed tests, and underrepresent multilingual, large-scale code refactoring.
Method
SWE-Bench ProMax uses expert-curated refactoring instances from real commits, with rewritten specifications, reviewed tests, complexity filtering, and evaluation across two agent scaffolds.
Results
41.2% resolve rate is achieved by GPT-5.2, the best-performing model across the 170-instance benchmark.
Takeaways & Limitations
SWE-Bench ProMax remains a meaningful, unsaturated challenge for current AI coding agents.
Takeaways & Limitations
The dataset consists solely of code, commit metadata, and test cases from publicly available repositories under approved open-source licenses.
Abstract
from arXiv · showhide
As AI coding agents take on increasingly complex, long-horizon software engineering tasks, existing benchmarks are rapidly saturating and their evaluation quality has come under serious scrutiny: a recent audit found that nearly 60% of unsolved SWE-bench Verified instances contain flawed tests -- either overly narrow tests that reject correct solutions or overly broad tests that check unstated requirements -- and that frontier models can verbatim reproduce gold patches from training data. Code refactoring, which requires coordinated, behavior-preserving changes across many files, offers a substantially harder and more realistic test of agent capability, yet remains underserved by current benchmarks. We introduce SWE-Bench ProMax, an expert-curated, multilingual code refactoring benchmark of 170 instances drawn from real commits across seven programming languages (Python, Java, TypeScript, Go, C, C++, and Rust). Every instance undergoes rigorous, multi-stage curation that directly addresses the quality problems identified in prior benchmarks: issue descriptions are rewritten from scratch to provide precise, unambiguous specifications, and test suites are manually reviewed to remove overly narrow and overly broad tests. Tasks with insufficient complexity or limited cross-file scope are filtered out, yielding a benchmark of challenging, large-scale refactoring tasks that average 11.4 modified files and 261.6 lines of code per instance, substantially exceeding the scale of existing benchmarks. Experiments with frontier models under two agent scaffolds show that the best model achieves only 41.2% resolve rate, confirming that SWE-Bench ProMax presents a meaningful and unsaturated challenge for current AI coding agents. Our benchmark is available at https://huggingface.co/datasets/swe-bench-promax/SWE-Bench-ProMax.
1 Introduction
SWE-Bench ProMax targets whether benchmark scores reflect genuine software-engineering ability by introducing large-scale, multilingual refactoring tasks with carefully reviewed specifications and tests.
- Refactoring restructures existing code without changing external behavior and remains underexplored in benchmarks focused mainly on bug fixes and feature implementations.
- Nearly 60% of unsolved SWE-bench Verified instances contain flawed tests, including 35.5% with narrow tests and 18.8% with broad tests.
- 170 instances span Python, Java, TypeScript, Go, C, C++, and Rust, drawn from real commits in actively maintained repositories.
- 30% of ProMax instances modify more than 10 files and 32% change over 200 lines, whereas 86% of SWE-bench Verified instances modify one file.
- Expert curation rewrites issue descriptions, manually reviews tests, and filters out tasks lacking complexity or substantial cross-file scope.
- The remaining benchmark averages 11.4 modified files and 261.6 lines of code, while the best model achieves only 41.2% resolve rate.
2 Related work
Related benchmarks have expanded from isolated code generation toward repository-level and multilingual tasks, but long-horizon refactoring remains difficult and existing coverage is uneven.
- From function-level to long-horizon evaluation: Benchmarking progressed from function-level datasets to repository-level challenges and extensions targeting language coverage, task difficulty, contamination, infrastructure, and verification.
- From function-level to long-horizon evaluation: Frontier agents exceed 75% on SWE-bench Verified, while state-of-the-art agents achieve less than 20% on long-horizon tasks.
- Refactoring benchmarks: RefactorBench provides 100 handcrafted tasks across nine Python repositories, averaging 4.3 modified files, while SWE-Refactor provides 1,099 Java instances without human test-quality verification.
- Expert curation: Human curation ranges from fully automated collection to extensive annotation and review, yet audits still find material defects in established benchmarks.
- Refactoring benchmarks: Existing refactoring tools primarily detect refactorings rather than generate them, and empirical studies report broad unsupported coverage among identified refactoring types.
3 SWE-Bench ProMax
SWE-Bench ProMax is constructed as an expert-curated benchmark of validated, large-scale refactoring tasks spanning seven languages and diverse repositories.
- 3.1 Task formulation: Each instance contains a pre-refactoring Docker environment, precise issue description, validating test suite, and gold patch; resolution requires passing every test.
- 3.2 Dataset construction and curation: Only 170 of 29,782 initial candidates remain after automated collection, environment validation, filtering, rewriting, and expert review.
- 3.2 Dataset construction and curation: Candidate commits come from repositories with at least 500 stars, approved licenses, and one of seven target languages comprising at least 80% of the codebase.
- 3.2 Dataset construction and curation: Experts filter single-file, low-line-count, and overly simple tasks, then review tests to remove narrow implementation checks and broad unstated requirements.
- 3.2 Dataset construction and curation: Issue descriptions are rewritten into precise, self-contained specifications covering changed components, transformations, and preserved behavioral invariants.
- 3.3 Dataset composition: The benchmark spans dynamically typed, garbage-collected, gradually typed, manually managed, and ownership-based languages across 70 repositories.
- 3.3 Dataset composition: Gold patches average 11.4 source files and 261.6 lines of code, while total changes average 15.9 files per instance.
4 Experiments
Experiments evaluate six proprietary and open-weight models under two agent scaffolds using test-based resolution across all 170 instances.
- Agent scaffolds: The study evaluates mini-swe-agent and OpenHands, which provide iterative agent interaction with repository tools and execution environments.
- Execution environment: Each instance runs in a pre-built isolated Docker container containing the pre-refactoring repository and installed dependencies.
- Evaluation protocol: Resolution requires passing every test, with Pass@1 resolve rate reported overall and by language across all 170 instances.
- Models: Six frontier models include three proprietary systems—Gemini-3-Pro, Claude Sonnet 4.6, and GPT-5.2—and three open-weight systems—GLM-5, Kimi-K2.5, and Qwen3.5.
- Models: Model selection covers diverse architectures, scales, and training paradigms, enabling analysis of their relationship to refactoring performance.
5 Results and analysis
SWE-Bench ProMax remains difficult for frontier agents, with performance varying by model, scaffold, language, cost, and agent behavior. Failures frequently involve incomplete cross-file refactoring and unproductive exploration rather than simply locating core files.
- Main results: 41.2% resolve rate is achieved by GPT-5.2, far below the 75%+ reported for frontier agents on SWE-bench Verified.The result indicates that multi-file refactoring with cross-file coordination and behavioral preservation remains challenging.
- Language analysis: Resolve rates vary across languages: Claude Sonnet 4.6 leads TypeScript and Rust, GPT-5.2 leads Python and C, and other models lead Java, Go, and C++.The reported leaders are Claude Sonnet 4.6 on TypeScript (53.6%) and Rust (63.6%), GPT-5.2 on Python (48.3%) and C (75.0%), GLM-5 on Java (34.6%), Kimi-K2.5 on Go (43.5%), and Qwen3.5 on C++ (54.5%).
- Agent behavior analysis: Agents modify fewer files than gold patches require, with the discrepancy widening for larger refactorings.For small changes, agents track the gold distribution more closely; for larger patches, they often fail to propagate transformations to peripheral call sites, documentation, configuration, and tests.
- Agent behavior analysis: Resolved instances use markedly fewer interaction rounds than unresolved instances, whose trajectories often enter repetitive edit–revert cycles.These cycles consume steps without expanding the scope of modifications.
6 Conclusion
SWE-Bench ProMax is an expert-curated, multilingual benchmark designed to address saturation, evaluation-quality problems, and the limited representation of refactoring in AI coding evaluation. Its results show that large-scale refactoring remains unsaturated and that model strengths and cost efficiency differ substantially.
- Benchmark scope: SWE-Bench ProMax contains 170 instances from real commits across seven programming languages and 70 repositories.The languages are Python, Java, TypeScript, Go, C, C++, and Rust.
- Benchmark quality: Expert curation rewrites issue descriptions and manually reviews tests to remove ambiguity, answer leakage, overly narrow tests, and overly broad tests.The benchmark was selected from 29,782 initial candidates through a rigorous three-stage pipeline.
- Findings: 41.2% is the best resolve rate achieved by GPT-5.2, and no model dominates across all languages.These results indicate that SWE-Bench ProMax remains far from saturated.
- Findings: Open-weight models are highly competitive with proprietary models on large-scale refactoring at a fraction of the cost.GLM-5 and Qwen3.5 come within five points of the best proprietary model, while trajectory inspection associates higher spending with unproductive exploration cycles.
Ethics statement
The benchmark uses publicly available, openly licensed repository materials and collects no personal or sensitive data.
- Data and licensing: All repositories are publicly available on GitHub and use approved open-source licenses such as MIT, Apache 2.0, or BSD.The authors state that distribution respects each project’s licensing terms.
- Data and licensing: The dataset consists solely of code, commit metadata, and test cases derived from public repositories.No personal or sensitive data is collected.
- Intended use: The benchmark is intended exclusively for evaluating and improving AI coding agents.The authors encourage responsible use aligned with the stated purpose.
A Additional dataset details
SWE-Bench ProMax covers 70 repositories across seven programming languages, with repository and license information organized by language.
- Repositories and licenses: 70 repositories across 7 languages are listed in the dataset details.The repositories are grouped by programming language and accompanied by license information.
B TASK CATEGORY ANALYSIS
SWE-Bench ProMax draws instances from 70 repositories across seven languages, with repository coverage varying substantially by language and patch complexity differing across language ecosystems.
- 70 repositories span predominantly Apache-2.0 and MIT licenses, with additional GPL, BSD, and AGPL representation.
- Go contributes 23 instances from 16 repositories, whereas TypeScript contributes 28 instances from only 2 repositories, including 25 from Angular.
- C has the highest average LOC at 424.1, while Java and C++ average 20.8 and 21.4 modified files per instance, respectively.
- Python, Rust, and TypeScript average 10.6–14.5 modified files per instance while still requiring multi-file coordination.
B Task category analysis
SWE-Bench ProMax tasks combine refactoring with interface changes and other software-engineering work, requiring broad reasoning across files, APIs, patterns, and often multiple task categories.
- Task categories: The benchmark classifies instances across ten categories, including API changes, cleanup, bug fixes, new features, documentation, and security patches.
- Task categories: Refactoring Cleanup appears in 66.5% of instances and API Interface Change in 65.3%, establishing the benchmark’s dominant task categories.
- Task categories: 46.5% of instances span three or more categories, and no instance involves fewer than two categories.
- Task categories: Bug Fix occurs in 41.2% of instances, reflecting the documented coupling between refactoring and defects exposed during restructuring.
- Required skills: Cross-file reasoning is required in 99.4% of instances and API semantics understanding in 98.8%, with pattern matching required in 91.8%.
C.2 Java: plantuml/plantuml (94 files, 1,629 LOC)
Representative instances illustrate SWE-Bench ProMax’s large cross-file scope and the varied maintenance problems it targets, from temporal modeling and misleading APIs to type and I/O abstractions.
- C.2 Java: plantuml/plantuml: PlantUML’s Gantt engine is constrained to day-level scheduling, limiting support for partial-day tasks, intra-day availability, and hour-level dependencies.
- Betaflight’s motor configuration fields use PWM-specific names for settings that also apply to digital protocols, creating confusion during configuration and review.
- Burn’s scalar tensor-operation arguments lack a common runtime representation, requiring conversion boilerplate across autodiff, fusion, router, and backend layers.
- OpenList’s file-upload reliability issues span stream handling, caching behavior, and buffer management across multiple storage drivers.
- LangExtract distributes output-format decisions across prompts, configuration, resolver parsing, and provider-specific implementations, complicating integration and maintenance.