Source-linked AI summary
Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving
Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, Rui Long, Kai Shen, Liang Xiang
TL;DR
Existing issue-resolving benchmarks focus primarily on Python, limiting evaluation across diverse software ecosystems. The paper introduces a multilingual benchmark and evaluates representative agents and models, while also launching an open-source RL community. It reports diagnostic capability boundaries and releases multilingual, reproducible data for further RL research.
Problem
Existing benchmarks such as SWE-bench focus almost exclusively on Python, leaving generalization to other widely used programming languages underexplored.
Method
The paper constructs Multi-SWE-bench across seven languages, evaluates nine models with three agent methods, and launches the Multi-SWE-RL community.
Results
The evaluation finds weaknesses in long-context retention and multi-file reasoning, while the project releases 4,723 containerized instances for RL research.
Takeaways & Limitations
Multi-SWE-bench and Multi-SWE-RL provide multilingual evaluation and reproducible real-world software environments for issue-resolving agents and RL research.
Takeaways & Limitations
Current agents struggle with semantically demanding tasks requiring deeper intent understanding, multi-component reasoning, and cross-file context aggregation.
Abstract
from arXiv · showhide
The task of issue resolving is to modify a codebase to generate a patch that addresses a given issue. However, existing benchmarks, such as SWE-bench, focus almost exclusively on Python, making them insufficient for evaluating Large Language Models (LLMs) across diverse software ecosystems. To address this, we introduce a multilingual issue-resolving benchmark, called Multi-SWE-bench, covering Java, TypeScript, JavaScript, Go, Rust, C, and C++. It includes a total of 1,632 high-quality instances, which were carefully annotated from 2,456 candidates by 68 expert annotators, ensuring that the benchmark can provide an accurate and reliable evaluation. Based on Multi-SWE-bench, we evaluate a series of state-of-the-art models using three representative methods (Agentless, SWE-agent, and OpenHands) and present a comprehensive analysis with key empirical insights. In addition, we launch a Multi-SWE-RL open-source community, aimed at building large-scale reinforcement learning (RL) training datasets for issue-resolving tasks. As an initial contribution, we release a set of 4,723 well-structured instances spanning seven programming languages, laying a solid foundation for RL research in this domain. More importantly, we open-source our entire data production pipeline, along with detailed tutorials, encouraging the open-source community to continuously contribute and expand the dataset. We envision our Multi-SWE-bench and the ever-growing Multi-SWE-RL community as catalysts for advancing RL toward its full potential, bringing us one step closer to the dawn of AGI.
1. Introduction
Existing issue-resolving benchmarks center on Python, leaving cross-language generalization underexplored. Multi-SWE-bench addresses this gap with a multilingual benchmark, broad agent evaluation, and the Multi-SWE-RL community.
- Motivation: SWE-bench contains 2,294 issues from 12 Python libraries, limiting evidence about issue resolving across other programming languages.The benchmark evaluates patches generated from issue descriptions and buggy repositories.
- Motivation: Programming languages differ in paradigms, idioms, and runtime behavior, raising questions about whether Python performance generalizes to seven other languages.The target languages are Java, TypeScript, JavaScript, Go, Rust, C, and C++.
- Benchmark and evaluation: Multi-SWE-bench contains 1,632 issues across seven languages and uses a systematic five-phase construction pipeline.The pipeline begins with repository selection, PR collection, Dockerized environments, and validation.
- Benchmark and evaluation: The benchmark evaluates Agentless, SWE-agent, and OpenHands with nine frontier models to study multilingual issue-resolving generalization.Reported analyses examine how issue descriptions, patch length, and file span relate to resolution.
- Findings: The evaluation identifies weaknesses in long-context retention and multi-file reasoning, delineating current capability boundaries for software-engineering agents.Resolution rates drop sharply when patches exceed 600 tokens or touch more than one file.
- Multi-SWE-RL: Multi-SWE-RL launches an open-source effort for scalable reinforcement-learning data and environments, initially releasing 4,723 containerized instances across seven languages.The instances provide reproducible execution environments for realistic software contexts.
2. Related Work
Prior code benchmarks evolved toward broader real-world software-development evaluation, with SWE-bench introducing repository-level issue resolving. The supplied passages do not provide further supported detail about multilingual benchmark construction.
- Benchmark evolution: Early code benchmarks primarily evaluated models on monolingual, program-level tasks.The cited passage places these efforts within the broader evolution of code-related benchmarks.
- Repository-level issue resolving: SWE-bench broadened evaluation to repository-level issues including bug fixes, feature requests, and optimization.It evaluates issue resolving rather than isolating code subtasks into separate datasets.
3. Multi-SWE-bench
Multi-SWE-bench is constructed through language selection, repository and PR curation, reproducible containerization, semantic validation, and manual verification. The resulting benchmark emphasizes realistic difficulty and reliable test-based evaluation.
- Overview: Multi-SWE-bench spans seven languages and is finalized through five construction phases, with manual verification as the fifth phase.The first four phases create candidate pools; manual verification finalizes the benchmark.
- Phase 1: Repository Selection: Repositories are selected for popularity, maintenance, CI/CD support, and build viability to establish a practical foundation.The filtering targets repositories with more than 500 stars, active maintenance, automated testing, and successful clean-environment builds.
- Phase 2: PR Collection: PR candidates must link to an issue, modify test files, and merge into the main branch before metadata is collected.Collected metadata includes the issue description, base commit, fix.patch, and test.patch.
- Phase 3: Environment Construction: Each PR receives a tailored Docker environment whose dependencies are extracted from CI/CD artifacts and documentation, then built and validated.Unfixable build or launch failures cause the PR to be discarded.
- Phase 4: Semantic Validation: Semantic validation runs the full test suite on the base commit, with test.patch, and with both test.patch and fix.patch.Test-case status transitions include PASSED, FAILED, NONE, and SKIPPED.
- Phase 4: Semantic Validation: 2,456 candidates were reduced to eligible instances by removing regressions, ineffective fixes, and abnormal test transitions.Retained instances include test cases with Any→FAILED/PASSED/SKIPPED/NONE→PASSED transitions.
- Phase 5: Manual Verification: Manual verification uses 68 annotators, independent double labeling, cross-review, and an internal quality team.The quality team verifies that outsourced annotations reach an 80% minimum accuracy threshold.
- Features and Difficulty: Time-based difficulty categories provide a more realistic complexity measure than token count or file span alone.Some easy Rust instances have large repetitive edits, showing why superficial size metrics can mislead.
4. Multi-SWE-RL Open-Source Community
Multi-SWE-RL is an open-source community for building scalable reinforcement-learning datasets and environments from real-world software tasks. Its initial release provides reproducible multilingual issue-resolving data and invites continued community expansion.
- Community Motivation: Multi-SWE-RL responds to the difficulty of creating interactive environments and trajectories for software-engineering reinforcement learning.Producing the 1,632 high-quality Multi-SWE-bench instances took about one year.
- Community Motivation: The community harnesses open-source collaboration to build diverse reinforcement-learning environments.Its purpose is to address the bottleneck of scalable, high-quality RL infrastructure.
- Community Initialization: The initial dataset contains 4,723 issue-resolving instances from 76 repositories across seven programming languages.Each instance has a fully containerized execution environment.
- Community Initialization: The release uses the Multi-SWE-bench construction pipeline but excludes its manual verification phase.The environments are designed for reproducibility and ease of integration.
- Contribution and Recognition: Community updates may add benchmarks, RL data, trial results, and open-source RL models with new contributors.The project maintains a rolling update schedule through arXiv updates or technical reports.
5. Experimental Setups
The study adapts three issue-resolving methods and nine LLMs to evaluate multilingual software issue resolution. It reports resolved rate as the primary metric, supplemented by file-level fault-localization accuracy and average cost.
- Evaluated Methods: Three representative methods—Agentless, SWE-agent, and OpenHands + CodeAct v2.1—were extended from Python-focused implementations to support Multi-SWE-bench’s multilingual setting.The adapted systems are named MagentLess, MSWE-agent, and MopenHands.
- Evaluated Methods: MagentLess retains fault localization and code repair but removes candidate patch selection because cross-language regression and reproduction testing is cumbersome.Its multilingual adaptation also replaces file skeletons with full file contents, uses Tree-sitter extraction, and prunes repository structures to reduce context size.
- Evaluated Methods: MSWE-agent supports multilingual issue resolution through multi-turn interaction with a predefined agent-computer interface.Its adaptation revises prompts, truncates long observations, excludes compiled artifacts, and fixes language-specific execution commands.
- Evaluated Methods: MopenHands was adapted for multilingual issue resolution by revising prompts, excluding compiled artifacts, and fixing patch-rendering behavior.Redirecting diffs to files and reading them through FileReadAction was especially important for Go.
- Evaluated LLMs: The evaluation covers nine LLMs, including GPT-4o, OpenAI-o1, OpenAI-o3-mini-high, Claude-3.5-Sonnet, Claude-3.7-Sonnet, DeepSeek-V3, DeepSeek-R1, Qwen2.5-72B-Instruct, and Doubao-1.5-pro.These models are evaluated across the three adapted methods.
- Metrics: Resolved Rate (%) measures the percentage of issues resolved, while Success Location (%) measures file-level fault-localization accuracy and Average Cost ($) measures cost per issue.Resolved Rate is the primary evaluation metric.
6. Experimental Results
Experiments evaluate language, model, method, issue-difficulty, and repository effects on multilingual issue resolution. Performance is strongest on Python and generally declines with language and repository complexity, while localization and flexible workflows remain important bottlenecks.
- Experimental Scope: The experiments compare language-specific performance, LLMs and agents, and repository-level effects using overall and difficulty-stratified resolved rates.Tables 4 and 5 report resolved rate across programming languages and difficulty levels.
- Performance across Programming Languages: 77.1% of Multi-SWE-bench issues are medium or hard, compared with 61.2% in SWE-Bench-Verified, contributing to limited generalization beyond Python.The evaluated methods were initially optimized for Python, while other languages introduce distinct runtime, memory, build, and type-system challenges.
- Performance across Programming Languages: Resolved rates generally follow a hierarchy in which Python and Java outperform systems languages, while TypeScript and JavaScript perform worst.Go and Rust usually outperform TypeScript and JavaScript, whereas C and C++ vary substantially across models.
- Performance across Programming Languages: Resolved rates decrease sharply as human-labeled issue difficulty rises from easy to hard, with hard issues often approaching zero.The agents are primarily effective on issues human developers can resolve in under 15 minutes.
- LLMs and Agents: OpenAI-o1, OpenAI-o3-mini-high, Claude-3.5-Sonnet, and Claude-3.7-Sonnet generally outperform Qwen2.5-72B-Instruct and Doubao-1.5-pro across the evaluated settings.Performance still varies substantially by programming language, with strong models struggling more on C and C++.
- LLMs and Agents: MopenHands achieves the highest resolved rate in five of seven languages, while MSWE-agent wins twice and MagentLess once.MopenHands and MSWE-agent generally benefit from more flexible workflows, although DeepSeek-R1 and Qwen2.5-72B-Instruct often favor MagentLess outside C and C++.
- Issue Flow: All three methods generally fail to locate issues more often than they succeed, making accurate localization a prerequisite for effective code editing.MagentLess localizes more accurately than MopenHands but edits less effectively, producing a lower overall resolved rate in some cases.
- Repository Performance: Repository quality metrics show a general positive relationship between resolved rate and repository stars, forks, issues, and pull requests.Figures 6 and 7 examine these relationships across the three methods and the evaluated LLMs.
6.2. Influencing Factors of Performance
Issue type, patch complexity, and cross-file scope strongly shape issue-resolving performance, while description length shows no consistent effect across languages.
- Issue Type: Bug fixes achieve the highest resolved rates, followed by new features, while feature optimization is most challenging across methods and languages.For example, MSWE-agent reaches 17.97% on Java bug fixes, versus 3.91% for new features and 1.56% for optimizations.
- Issue Type: Current agents handle localized, symptom-driven repairs better than semantically demanding tasks requiring intent understanding and cross-file reasoning.New functionality and behavior refinement require capabilities that remain underdeveloped in current LLM-based agents.
- Issue Description Characteristics: Longer issue descriptions have no consistent relationship with resolved rate across languages.Longer descriptions correlate with lower rates in Python but higher rates in Go, potentially reflecting either precise guidance or greater task complexity.
- Fix Patch Characteristics: Resolved rates decline significantly as fix patches modify more files across all three methods.For single-file issues, MagentLess outperforms MSWE-agent and MopenHands in five of seven languages.
6.3. Case Study
The case study identifies interaction-budget exhaustion, inaccurate fault localization, rigid action handling, and language-specific tooling failures as recurring obstacles to successful repair.
- Language-General Case: MSWE-agent and MopenHands often exhaust the 50-round interaction limit, sometimes without triggering submission.The pattern appears in representative failures across repositories and indicates difficulty solving complex tasks within bounded interaction budgets.
- Language-General Case: Incorrect fault localization causes many failures across all three agent methods.The analysis points to software engineering techniques such as SBFL as a possible future integration.
- Language-General Case: In some cases, hardcoded execution of only the last action causes premature submission despite multiple valid model actions.This exposes a structural bottleneck in rigid agent control logic.
- Language-Specific Case: TypeScript repository structures can exceed context limits, preventing MagentLess from performing fault localization.JavaScript syntax involving loosely bound arrow functions can also prevent Tree-sitter from constructing contextual edit windows.
- Language-Specific Case: JavaScript agents may fail when launching long-lived development servers because current frameworks lack persistent interactive process support.Premature termination or container crashes can result when agents invoke pnpm during repair.
6.4. Resource Consumption
Resource use varies by language, model, and method: token consumption reflects representation and workflow differences, while LLM-directed workflows generally cost more than fixed workflows.
- Average Token Consumption per Issue: Go has relatively low input and output token consumption in both MagentLess and MSWE-agent.The paper attributes this pattern to Go’s minimalistic syntax and clear conventions.
- Average Token Consumption per Issue: TS is the most token-intensive language for MagentLess, whereas Python is most token-intensive for MSWE-agent.Python usage is further increased because the original SWE-agent implementation lacks the truncation mechanism used for other languages.
- Average Cost per Issue: DeepSeek-V3, DeepSeek-R1, and Qwen2.5-72B-Instruct cost below $0.03 per resolved issue.Their low costs benefit from input-token pricing below $0.14 per million tokens.
- Average Cost per Issue: MagentLess has lower costs than MSWE-agent because its fixed workflow requires fewer environment interactions.MSWE-agent and MopenHands use LLM-determined workflows with more interaction turns and higher costs.
6.5. Troubleshooting
The benchmark pipeline encountered reproducibility and evaluation issues involving test logs, builds, binary artifacts, and nondeterministic tests, requiring targeted mitigations.
- Data and Evaluation Issues: Test.log and Fix.log can contain different numbers of test cases when fixes alter control flow, remove redundant coverage, or merge test paths.This inconsistency is commonly observed in repositories such as preactjs/preact.
- Data and Evaluation Issues: Some repositories fail to build or execute tests before applying the fix because test.patch introduces symbols defined only by the fix.These pre-fix build failures arise when newly introduced functions or variables are undefined without the fix.
- Data and Evaluation Issues: Compiled binary artifacts in C and C++ can block git apply, so the pipeline removes them through hard-coded filtering.The paper notes that more robust handling remains necessary.
- Data and Evaluation Issues: Java and C evaluations can be nondeterministic because excessive thread concurrency produces inconsistent run.log outcomes.The benchmark mitigates this issue by reducing parallelism during evaluation.
7. Conclusions and Future Works
The paper introduces a multilingual benchmark and an open-source RL community for issue-resolving research, releasing validated instances and the complete data-construction pipeline. It plans to expand coverage across instances, languages, modalities, and broader software-engineering tasks.
- Multi-SWE-bench contains 1,632 human-validated GitHub instances spanning seven programming languages.
- The authors evaluate nine popular models with three agent methods and analyze the resulting issue-resolving performance.
- Multi-SWE-RL releases 4,723 validated instances and the complete data-construction pipeline to support community expansion of issue-resolving datasets.
- Future work will scale the benchmark and RL community to more instances, languages, and modalities, while adding broader software-engineering tasks.
Contributions
The contribution section credits the project leadership, core code-development contributors, paper-writing contributors, data-construction contributors, and agent-integration and evaluation contributors.
- Daoguang Zan is listed as project lead.
- Zhirong Huang is listed for core code development.
- Wei Liu and Daoguang Zan are listed for paper writing.
- Zhirong Huang, Hanwu Chen, Linhao Zhang, and Daoguang Zan are listed for data construction.
- Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, and Daoguang Zan are listed for agent integration and evaluation.
- Names marked with ★ denote equal contribution.