PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving

Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, Rui Long, Kai Shen, Liang Xiang

arXiv:2504.02605v1cs.SEcs.AIcs.CL

TL;DR

Existing issue-resolving evaluations focus largely on Python, leaving unclear whether agents generalize to other widely used languages. Multi-SWE-bench evaluates agents across seven languages, finding better performance on longer-described issues and sharply lower resolution rates for large or multi-file fixes.

  • Problem

    Existing issue-resolving evaluations focus largely on Python, leaving unclear whether agents generalize to other widely used languages with distinct paradigms, idioms, and runtime behavior.

    Source: §1 · p. 3

  • Method

    Multi-SWE-bench provides 1,632 human-validated issues across seven languages, constructed through a systematic five-phase pipeline.

    Source: §1 · pp. 3–4

  • Results

    Agent performance is better with longer issue descriptions, while resolved rates drop sharply for fixes exceeding 600 tokens or touching multiple files.

    Source: §1 · pp. 3–4

  • Takeaways & Limitations

    These findings delineate current LLM software-engineering capability boundaries and obstacles to real-world deployment.

    Source: §1 · p. 4

    Current agents are more effective at localized, symptom-driven repairs but struggle with semantically demanding changes requiring deeper intent understanding and cross-file reasoning.

    Source: §6.2.1 · p. 21

Abstract

from arXiv · show

The task of issue resolving is to modify a codebase to generate a patch that addresses a given issue. However, existing benchmarks, such as SWE-bench, focus almost exclusively on Python, making them insufficient for evaluating Large Language Models (LLMs) across diverse software ecosystems. To address this, we introduce a multilingual issue-resolving benchmark, called Multi-SWE-bench, covering Java, TypeScript, JavaScript, Go, Rust, C, and C++. It includes a total of 1,632 high-quality instances, which were carefully annotated from 2,456 candidates by 68 expert annotators, ensuring that the benchmark can provide an accurate and reliable evaluation. Based on Multi-SWE-bench, we evaluate a series of state-of-the-art models using three representative methods (Agentless, SWE-agent, and OpenHands) and present a comprehensive analysis with key empirical insights. In addition, we launch a Multi-SWE-RL open-source community, aimed at building large-scale reinforcement learning (RL) training datasets for issue-resolving tasks. As an initial contribution, we release a set of 4,723 well-structured instances spanning seven programming languages, laying a solid foundation for RL research in this domain. More importantly, we open-source our entire data production pipeline, along with detailed tutorials, encouraging the open-source community to continuously contribute and expand the dataset. We envision our Multi-SWE-bench and the ever-growing Multi-SWE-RL community as catalysts for advancing RL toward its full potential, bringing us one step closer to the dawn of AGI.

1. Introduction

Multi-SWE-bench introduces a human-validated issue-resolving benchmark spanning seven programming languages, and Multi-SWE-RL releases containerized training instances to support scalable reinforcement learning in real-world software environments.

Motivation

  • SWE-bench [Jimenez et al., 2023] centers on Python issues, leaving open whether agents’ strong Python performance generalizes to languages with distinct paradigms, idioms, and runtime behaviors.SWE-bench contains 2,294 issues from 12 Python libraries, while SWE-bench Verified comprises 500 human-validated issues.Source: §1 · p. 3

Benchmark contributions

  • 1,632 human-validated issues across Java, TypeScript, JavaScript, Go, Rust, C, and C++ form Multi-SWE-bench, built through a five-phase pipeline for reliable evaluation.The pipeline selects runnable, popular repositories, collects issue-related pull requests, builds Dockerized environments, validates bug-fixing effects, and uses dual annotation and cross-review.Source: §1 · pp. 3–4

Evaluation findings

  • Evaluation findings indicate better model performance on longer issue descriptions, but sharply lower resolution rates for fixes exceeding 600 tokens or touching multiple files.These patterns point to weaknesses in contextual grounding, long-context retention, and multi-file reasoning.Source: §1 · pp. 3–4

Evaluation scope

  • The benchmark evaluates Agentless, SWE-agent, and OpenHands with nine frontier models across seven languages and Python, analyzing cross-language performance and language-specific failure cases.The models include GPT-4o, OpenAI-o1, OpenAI-o3-mini-high, Claude-3.5-Sonnet, Claude-3.7-Sonnet, DeepSeek-V3, DeepSeek-R1, Qwen2.5-72B-Instruct, and Doubao-1.5-Pro.Source: §1 · p. 3

RL community

  • 4,723 containerized issue-resolving instances across seven languages launch Multi-SWE-RL, a community effort to build scalable reinforcement-learning data and environments.Each instance includes a reproducible execution environment for plug-and-play RL-agent training.Source: §1 · p. 4

2. Related Work

Code benchmarks have expanded from monolingual program-level evaluation toward multilingual and repository-level tasks, while issue resolving across widely used programming languages remains underexplored. Multi-SWE-bench addresses that gap with a multilingual issue-resolving benchmark.

2. Related Work

  • Code benchmarks evolved from monolingual program-level evaluations toward multilingual and repository-level tasks, including code generation, completion, and bug fixing.These shifts aim to better reflect real-world software development environments.Source: §2 · pp. 4–5
  • Because LLM performance on other widely used languages remains underexplored, Multi-SWE-bench provides 1,632 human-validated GitHub issues across seven languages.SWE-bench includes bug fixes, feature requests, and optimization tasks.Source: §2 · p. 5
  • SWE-bench [Jimenez et al., 2023] covers repository issue resolving; multimodal and visual variants target visual apps, while SWE-Lancer [Miserendino et al., 2025] covers JS/TS freelance tasks.SWE-Lancer includes both technical and managerial tasks.Source: §2 · p. 5

3. Multi-SWE-bench

Multi-SWE-bench contains 1,632 issue-resolving tasks across seven programming languages, built through a five-phase process that ends in manual verification.

3.1. Benchmark Construction

  • The five-phase construction process first builds candidate pools for each language, then finalizes the benchmark through manual verification.The first four phases gather and filter candidate data; the fifth verifies the retained instances.Source: §3.1 · p. 5

3.1.1. Phase 1: Repository Selection

  • Repositories had to be popular, actively maintained, CI/CD-enabled, and buildable in a clean environment, yielding a foundation representative of production-level codebases.Selection required over 500 GitHub stars and at least six months of active maintenance.Source: §3.1.1 · p. 6

3.1.2. Phase 2: Pull Request Crawling

  • Candidate pull requests had to link to an issue, modify test files, and be merged into the main branch.The collected PR metadata included issue descriptions, base commits, fix patches, and test patches.Source: §3.1.2 · p. 6

3.1.3. Phase 3: Environment Determination

  • For each PR, the team identified repo-common and PR-specific dependencies, built a Docker image, and verified the repository launched at the target commit.Unfixable build failures led to discarding the PR; launch failures prompted corrective attempts before validation.Source: §3.1.3 · pp. 6–7

3.1.4. Phase 4: Pull Request Filtering

  • Full-test status transitions filtered PRs: candidates were rejected for regressions, no demonstrated fix, or ambiguous test behavior.Retained instances showed at least one ANY→FAILED→PASSED transition, with test cases recorded across base, test-patch, and fix-patch runs.Source: §3.1.4 · p. 7

3.1.5. Phase 5: Manual Verification

  • 1,632 high-quality instances were retained from 2,456 candidates after manual verification by 68 annotators.Each instance was independently labeled by two annotators; an internal team of 14 engineers checked annotations against reference answers and required at least 80% accuracy.Source: §3.1.5 · pp. 7–8

3.2. Features of Multi-SWE-bench

  • Across 39 repositories and seven languages, issue descriptions and patch complexity varied; time-based difficulty categories tracked longer descriptions and larger patches as difficulty increased.File counts ranged from 24 to 27,632 and code size from 6.7k to 698.6k lines. Easy, medium, and hard correspond to estimated resolution times of ≤15 minutes, 15 minutes–1 hour, and ≥1 hour; some easy Rust instances still involved large edits.Source: §3.2 · p. 9

4. Multi-SWE-RL Open-Source Community

Multi-SWE-RL is an open-source community for building RL training datasets and environments for real-world software engineering. Its initial release and contribution program are designed to support reproducible data creation and sustained collaboration.

Community Introduction

  • The community aims to build high-quality RL datasets and environments for complex software engineering tasks, enabling training of autonomous agents for real-world challenges.The paper says producing such environments and trajectories is difficult; creating Multi-SWE-bench took about one year to produce 1,632 high-quality instances.Source: §4 · p. 10
  • Multi-SWE-RL was launched to harness open-source collaboration in building diverse RL environments.Source: §4 · p. 11

Community Initialization

  • The initial release contains 4,723 issue-resolving instances from 76 repositories across seven programming languages, each with a containerized execution environment.The dataset follows the Multi-SWE-bench pipeline but omits its manual verification process.Source: §4 · p. 11

Contribution Guidelines and Recognition

  • Contributors can follow a detailed demo, while rolling updates every three months recognize new benchmark, dataset, RL-trial, and open-source model contributions.Updates may add languages and authors, contribute Multi-SWE-RL data, report RL performance, or release improved RL models; the community invites sustained participation.Source: §4 · p. 11

5. Experimental Setups

The experimental setup adapts three Python-oriented issue-resolving methods for Multi-SWE-bench and evaluates them with nine LLMs using resolved rate, localization accuracy, and cost metrics.

5.1. Evaluated LLMs and Methods

  • Agentless [Xia et al., 2024], SWE-agent [Yang et al., 2024], and OpenHands + CodeAct v2.1 [Wang et al., 2024b] were extended for Multi-SWE-bench’s multilingual setting.The authors note substantial room to improve language-specific adaptation and robustness.Source: §5.1 · pp. 11–12
  • MagentLess adapts Agentless’s staged issue-resolution workflow, retaining fault localization and repair while removing candidate-patch selection.Its changes include multilingual prompts, full-file inputs, Tree-sitter extraction across languages, and pruning repository structures by file extensions to limit context size.Source: §5.1 · pp. 11–12
  • MSWE-agent adapts SWE-agent’s multi-turn agent-computer interface workflow for stable execution across languages.Changes include multilingual prompts, truncated long observations, excluding compiled artifacts with .gitignore, and fixes to language-specific commands that could crash or hang.Source: §5.1 · p. 12
  • MopenHands adapts OpenHands with multilingual prompts, compiled-artifact exclusions, and implementation fixes for applicable patches.One fix redirected diffs to a file read through FileReadAction because CmdRunAction rendered tabs as spaces, especially affecting Go patches.Source: §5.1 · p. 12
  • The evaluation covered nine popular LLMs across the three methods.The models were GPT-4o, OpenAI-o1, OpenAI-o3-mini-high, Claude-3.5-Sonnet, Claude-3.7-Sonnet, DeepSeek-V3, DeepSeek-R1, Qwen2.5-72B-Instruct, and Doubao-1.5-pro.Source: §5.1 · p. 12

5.2. Evaluation Metrics

  • Resolved Rate (%) was the primary metric, with Success Location (%) and Average Cost ($) also reported.Resolved Rate measures the percentage of issues resolved; Success Location measures file-level fault-localization accuracy, and Average Cost is the average cost per issue.Source: §5.2 · pp. 12–13

6. Experimental Results · 6.1. Performance on Multi-SWE-bench

The evaluation examines performance by programming language, agent and LLM, and repository characteristics. It finds substantial variation across languages and methods, with difficulty and repository properties also shaping resolved rates.

6.1.1. Performance across Programming Languages

  • Resolved rates vary by language: high-level general-purpose languages generally outperform systems and low-level languages, web languages fare worst, and Python leads Java.Go and Rust generally outperform TypeScript and JavaScript, while C and C++ vary across models; the paper links these differences to language-specific challenges and methods initially optimized for Python.Source: Table 4 · p. 13, §6.1.1 · p. 15
  • Resolved rates generally fall as issue difficulty rises; hard issues approach zero, and SWE-agent with Claude-3.5-Sonnet resolves 0% of easy JavaScript issues versus 7.62% of medium issues.The authors say agents mainly handle issues human developers can resolve in under 15 minutes and struggle with tasks requiring over an hour; files requiring modification also affect performance.Source: Table 5 · p. 14, §6.1.1 · p. 15

6.1.2. Performance across Various Methods and LLMs

  • OpenAI-o1, OpenAI-o3-mini-high, Claude-3.5-Sonnet, and Claude-3.7-Sonnet perform relatively strongly, but model performance varies across languages and harder issues.OpenAI-o1 and OpenAI-o3-mini-high do well on Python and Java but struggle on C and C++; the authors suggest differing exposure to high-level and low-level language features may contribute.Source: Table 4 · p. 13, Table 5 · p. 14, §6.1.2 · pp. 15, 17
  • MopenHands achieves the highest resolved rate in five of seven languages, while MSWE-agent wins twice and MagentLess once.The authors attribute the stronger results of MopenHands and MSWE-agent to more flexible workflows; DeepSeek-R1 and Qwen2.5-72B-Instruct are exceptions, generally doing better with MagentLess outside C and C++.Source: Table 4 · p. 13, Table 5 · p. 14, §6.1.2 · p. 17
  • All three methods more often fail than succeed at locating issues; MagentLess locates more accurately than MopenHands but edits less effectively, lowering its overall resolved rate.The difference is particularly evident with Claude-3.7-Sonnet; the authors argue for methods balancing issue identification with effective fix generation.Source: Figure 4 · p. 16, §6.1.2 · p. 17
  • Turn requirements vary by model, language, and method; MopenHands uses fewer turns than MSWE-agent for GPT-4o on Java, while MSWE-agent uses fewer for Python.MopenHands has more dispersed turn counts, particularly with OpenAI-o3-mini-high, and the paper reports no successful issues for MSWE-agent with Qwen2.5-72B-Instruct on C++.Source: §6.1.2 · p. 17, Figure 5 · p. 18

6.1.3. Performance across Different Repositories

  • Repositories with more stars, forks, issues, and PRs tend to have higher resolved rates, especially for MSWE-agent and MopenHands, but these measures do not guarantee effectiveness.MagentLess shows relatively little resolved-rate variation across these repository-quality measures.Source: §6.1.3 · p. 18, Figure 6 · p. 19, Figure 7 · p. 19
  • Resolved rates generally decrease as repository complexity increases; lower language entropy is more clearly associated with higher rates than repository size or file count.Larger Java repositories are an exception, with higher resolved rates; the authors point to factors such as modularity, documentation, and standardized practices as possible contributors.Source: §6.1.3 · pp. 18–19

6.2. Influencing Factors of Performance

The section examines how issue type, issue-description length, fix-patch length, and the number of modified files relate to issue-resolving performance.

6.2.1. Issue Type

  • Bug fixes have the highest resolved rates, followed by new features and feature optimizations; MSWE-agent scores 17.97% on Java bug fixes, 3.91% on new features, and 1.56% on optimizations.The authors attribute the gap to agents’ greater effectiveness on localized, symptom-driven repairs than on tasks requiring deeper intent understanding, multi-component reasoning, and cross-file context aggregation.Source: §6.2.1 · pp. 20–21, Table 6 · p. 21

6.2.2. Characteristics of Issue Description

  • Description length has no consistent relationship with resolved rate: longer descriptions correlate with lower rates in Python but higher rates in Go, while MagentLess generally performs better on longer descriptions.Long descriptions may provide precise issue locations and steps or instead describe more complex issues, with different effects on resolution difficulty; MagentLess’s relative performance is especially notable in Python, Java, and TypeScript.Source: §6.2.2 · p. 21, Figure 10 · p. 22

6.2.3. Characteristics of Fix Patches

  • Resolved rates generally fall as fix patches grow longer; patches over 1,000 tokens reduce performance significantly for all three methods and reach zero in Java.Shorter patches are easier to handle because they require less reasoning and simpler edits; MagentLess peaks in the shorter-patch range for Python, Java, and C.Source: §6.2.3 · pp. 22–23, Figure 13 · p. 23
  • Resolved rates drop as fix patches modify more files; for single-file issues, MagentLess outperforms MSWE-agent and MopenHands in five of seven programming languages.The authors suggest cross-file changes require more intricate handling or coordination across repository components.Source: §6.2.3 · p. 23

6.3. Case Study

The case studies identify general agent failure patterns and language-specific workflow limitations, while highlighting opportunities to improve localization, agent control, and adaptability.

6.3.1 Language-General Case

  • Interaction-limit exhaustion caused MSWE-agent and MopenHands to fail, sometimes without triggering submission, while incorrect fault localization caused many failures across all three methods.The cases suggest completing complex tasks within limited rounds and integrating techniques such as SBFL [Abreu et al., 2007, Jones et al., 2002] as future directions.Source: §6.3.1 · pp. 23–24
  • In some cases, the framework executed only the last of multiple valid actions generated in a turn, causing premature submission and motivating lightweight, model-centric agents.The authors call for delegating full decision autonomy to the LLM.Source: §6.3.1 · p. 24
  • Successful repairs often followed issue reproduction, but Claude-3.5-Sonnet and Claude-3.7-Sonnet also sometimes resolved issues without it, supporting selective reproduction based on traceability, confidence, and cost.Failure to reproduce often left cases unresolved, though it was not always a prerequisite for success.Source: §6.3.1 · p. 24

6.3.2 Language-Specific Case

  • In some TypeScript repositories, extracted repository structures exceeded MagentLess’s context limit, preventing fault localization and exposing fixed workflows’ limited adaptability to language-specific scenarios.The cases motivate greater robustness and adaptability in fixed workflows.Source: §6.3.2 · p. 24
  • In JavaScript repositories with loosely bound syntax such as arrow functions, Tree-sitter sometimes failed to extract code structures, preventing MagentLess from building contextual windows around candidate edits.The cases expose syntax-driven workflow brittleness and motivate tolerance to parsing failures and language-specific irregularities.Source: §6.3.2 · pp. 24–25
  • In some JavaScript projects, agents invoked pnpm to launch development servers, but frameworks lacking long-lived interactive-process support caused premature termination or container crashes.The authors point to persistent shell sessions and interactive service control, as enabled by SWE-ReX, as a future direction.Source: §6.3.2 · p. 25

6.4. Resource Consumption

The section compares token consumption and per-issue costs across languages, models, and methods, highlighting variation by language and workflow as well as differences in model pricing.

Average token consumption per issue

  • Token use varies by language and method: TS is highest in MagentLess, Python in MSWE-agent, while Go uses relatively few input and output tokens in both.MSWE-agent’s Python implementation retains the original SWE-agent without the over-length truncation used for other languages, increasing usage for GPT-4o, OpenAI-o1, OpenAI-o3-mini-high, and Qwen2.5-72B-Instruct.Source: §6.4 · p. 25

Average cost per issue

  • Under $0.03 per resolved issue, DeepSeek-V3, DeepSeek-R1, and Qwen2.5-72B-Instruct have the lowest costs; OpenAI-o1 is costliest at $15 per million input tokens.The low-cost models benefit from input-token pricing below $0.14 per million.Source: §6.4 · p. 26, Table 8 · p. 26
  • MagentLess has lower costs than MSWE-agent, while LLM-directed MSWE-agent and MopenHands workflows require more environment turns and cost more.MagentLess’s fixed workflow is the stated reason for its lower costs.Source: §6.4 · p. 26

6.5. Troubleshooting

The authors document practical challenges encountered while constructing Multi-SWE-bench and Multi-SWE-RL, along with mitigations and exclusions intended to support reproducibility and future contributions.

6.5. Troubleshooting

  • Test.log and Fix.log can report different case counts when fixes change control flow or coverage, and some repositories fail to build before fixes define symbols used by tests.The benchmark lowercases test names to align casing differences between logs.Source: §6.5 · pp. 26–27
  • C and C++ agent runs can produce binary artifacts that block patch application, while Java and C tests can be nondeterministic under high concurrency; binaries are filtered and evaluation parallelism reduced.The authors note that more robust binary-artifact handling is needed.Source: §6.5 · p. 27
  • Instances with dynamically generated test identifiers are excluded, and interleaved concurrent Java test output can make rule-based log parsing infeasible.The interleaving is attributed as likely due to unsynchronized multithreaded logging.Source: §6.5 · p. 27

7. Conclusions and Future Works

Multi-SWE-bench provides a multilingual issue-resolving benchmark and evaluation, while Multi-SWE-RL supports community-built training data. Future work will expand their scale and scope across languages, modalities, and software engineering tasks.

7. Conclusions and Future Works

  • Multi-SWE-bench comprises 1,632 human-validated instances across seven programming languages and evaluates nine models using three agent methods.The authors also report conducting a thorough analysis of the evaluation results.Source: §7 · p. 27
  • Multi-SWE-RL aims to build large-scale issue-resolving training datasets, and the authors release 4,723 validated instances with the complete data construction pipeline.They encourage the open-source community to contribute to and expand the dataset.Source: §7 · p. 27
  • The authors plan to scale both efforts to more instances, languages, and modalities, and extend them to project generation [Zan et al., 2024, Starace et al., 2025], runtime setup [Zala et al., 2024, Hu et al., 2025, Eliseeva et al., 2025], bug reproduction [Wang et al., 2024a, 2025] and localization [Hossain et al., 2024], testing and maintenance [Lemner et al., 2024, Peng et al., 2025].Source: §7 · p. 27
  • The authors envision a scalable, sustainable, data-centric infrastructure to empower future research.Source: §7 · p. 27

Related papers

Loading 2504.02605v1…