PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-SWT-Bench: A Multilingual Benchmark for Reproduction Test Generation

Kazuki Kusama, Sota Nakashima, Haruka Tokumasu, Masanari Kondo, Lingming Zhang, Yasutaka Kamei

arXiv:2609.34752v1cs.SE

TL;DR

Existing reproduction-test benchmarks focus on individual programming languages, leaving a gap in unified evaluation across software ecosystems. The paper introduces MULTI-SWT-BENCH and evaluates four representative methods with state-of-the-art LLMs. Python consistently exceeds the all-language aggregate, while C++ performs particularly poorly; the analysis identifies language-specific and cross-language challenges.

  • Problem

    Existing reproduction-test benchmarks focus on individual languages, leaving unclear whether agent performance generalizes across programming ecosystems.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces MULTI-SWT-BENCH, a 1,963-instance benchmark across eight languages, and evaluates state-of-the-art LLMs using four representative methods.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Python alone exceeds the all-language aggregate across all evaluated configurations, while C++ exhibits particularly low success rates.

    Source: §4.1.1 · pp. 5–6

  • Takeaways & Limitations

    Evaluations should cover diverse programming ecosystems, and agents should account for ecosystem-specific testing conventions and validate behavior after revisions.

    Source: §6 · p. 9

Abstract

from arXiv · show

Reproduction test generation translates a natural-language issue description into executable tests that fail on the original code and pass after the issue is resolved, providing executable evidence for verifying candidate patches. Existing benchmarks are constructed for individual programming languages, preventing a unified evaluation across diverse programming ecosystems. To address this limitation, we introduce MULTI-SWT-BENCH, a multilingual benchmark for reproduction test generation consisting of 1,963 instances across eight programming languages: Python, Java, TypeScript, JavaScript, Go, Rust, C, and C++. Using this benchmark, we conduct an empirical study of state-of-the-art LLMs with four representative methods (MSWE-agent, MOpenHands, Codex, and Claude Code) and perform a failure analysis across programming languages. Our evaluation reveals a systematic language gap. Across every evaluated method and LLM, the success rate on Python exceeds the aggregate success rate across all languages, while C++ exhibits particularly low success rates. Our failure analysis identifies both language-specific challenges arising from repository testing conventions and cross-language challenges in inferring implicit setup requirements and preserving the target behavior through iterative revisions. These findings demonstrate the importance of multilingual evaluation and provide actionable directions for developing reproduction test generation methods that generalize across software ecosystems and reliably capture issue-specific behavior.

1 INTRODUCTION

Existing reproduction-test benchmarks focus on individual languages, leaving unclear whether agent performance generalizes across programming ecosystems. The paper introduces a multilingual benchmark and evaluates agents to reveal language-specific and cross-language challenges.

1 INTRODUCTION

  • Existing benchmarks focus on individual languages, so it remains unclear whether agent capabilities and relative effectiveness generalize across ecosystems with differing test practices.Reproduction tests translate issue descriptions into executable checks that fail before a fix and pass after resolution, helping verify candidate patches.Source: §1 · pp. 1–2
  • Python alone consistently exceeds the all-language aggregate across evaluated configurations, while C++ has particularly low success rates.The introduction identifies a systematic language gap across the evaluated methods and LLMs.Source: §1 · p. 2
  • The failure analysis identifies ecosystem- and repository-convention challenges alongside cross-language difficulties with implicit setup and preserving target behavior through revisions.The paper presents these findings as directions for improving reproduction-test generation across software ecosystems.Source: §1 · p. 2
  • MULTI-SWT-BENCH provides a unified evaluation setting with 1,963 reproduction-test instances across eight programming languages.The benchmark supports systematic performance comparisons across programming ecosystems.Source: §1 · p. 2

2 MULTI-SWT-BENCH

MULTI-SWT-BENCH defines reproduction-test success through fail-to-pass behavior and constructs a multilingual benchmark from validated real-world issue instances. Its 1,963 instances span eight languages and varied repository and test characteristics.

2 MULTI-SWT-BENCH

  • MULTI-SWT-BENCH spans eight programming languages and provides a multilingual setting for reproduction-test generation.The benchmark is introduced alongside an overview of its construction and key features.Source: §2 · p. 2

2.1 NOTATION AND DEFINITIONS

  • A generated test reproduces an issue when it fails on the original codebase and passes after the issue-resolving patch.For a test set, success requires at least one such fail-to-pass test and all generated tests to pass on the patched codebase.Source: §2.1 · p. 2

2.2 BENCHMARK CONSTRUCTION

  • The benchmark converts 2,132 real-world issue-resolution instances into test-generation tasks, validates reference-test oracles, and retains 1,963 instances.Instances without reliably evaluable reference-test outcomes are excluded; the paper reports experiments on the 340-instance MULTI-SWT-BENCH-LITE subset.Source: §2.2 · pp. 2–3

2.3 FEATURES OF MULTI-SWT-BENCH

  • The 1,963 instances cover 51 repositories, with at least 111 instances per language and substantial variation in repository scale and reference-test characteristics.Average repository size ranges from 93K to 994K lines of code; reference tests average 17.3 cases per instance, of which 15.1 show fail-to-pass behavior.Source: §2.3 · p. 3, Table 2 · p. 3

3 EXPERIMENTAL SETUP

The evaluation adapts four representative methods to reproduction test generation and selects LLMs to compare methods and cost–performance trade-offs. It measures successful reproduction, change coverage, and whether generated patches can be applied.

3.1 EVALUATED METHODS AND LLMS

  • Four methods are adapted: issue-resolution agents MSWE-agent and MOpenHands, plus the general-purpose coding agents Codex and Claude Code.MSWE-agent's prompts and execution harness are adapted; MOpenHands's prompt is adapted and browsing and external network access are disabled except for LLM-service connections. Codex and Claude Code run with external access restricted.Source: §3.1 · p. 4
  • LLMs are selected to cover multiple providers and price tiers, with models shared across methods for comparison and additional subscription-plan models included.The stated aim is to cover cost–performance trade-offs within the experimental budget.Source: §3.1 · p. 4

3.2 EVALUATION METRICS

  • Success requires at least one test that fails before the golden patch and passes afterward, with no test failing after the patch.The evaluation also reports Fail-to-Pass, Fail-to-Any, and Pass-to-Pass test proportions.Source: §3.2 · p. 4
  • Change coverage measures executable golden-patch lines covered by generated tests; patch well-formedness measures whether the generated patch applies without errors.Instances with no executable lines modified by the golden patch are excluded from change coverage.Source: §3.2 · pp. 4–5

4 EXPERIMENTAL RESULTS

Across language, method, and LLM comparisons, reproduction-test success varies, and failure analysis identifies language-specific testing conventions and cross-language setup and revision challenges. The results also compare inference cost with success rate.

4.1.1 PERFORMANCE ACROSS PROGRAMMING LANGUAGES

  • 12/12 configurations: Python's success rate exceeds All; every other language fails to exceed All in at least one configuration.The authors note that Python-only evaluations may provide an overly optimistic view of reproduction test generation performance.Source: §4.1.1 · pp. 5–6, Table 3 · p. 5
  • 52.50%: Claude Code + Claude Opus 5's success rate on C++; C++ and Go remain below All across all 12 configurations.C++ has the largest average codebase size (994.3K LoC) and number of changed test-patch lines (294.3); the authors suggest instance properties may also be associated with its low success rate.Source: Table 3 · p. 5, §4.1.1 · p. 6

4.1.2 PERFORMANCE ACROSS VARIOUS METHODS AND LLMS

  • MOpenHands has the highest success rate for each of the four LLMs evaluated with multiple methods, while the best-performing LLM varies by method.GPT-5.6 Luna is highest with MSWE-agent (40.00%), whereas Claude Sonnet 5 is highest with MOpenHands (57.94%).Source: §4.1.2 · p. 6, Table 4 · p. 6
  • For MOpenHands, Codex, and Claude Code, F→P rates are 9.71 to 28.53 percentage points below F→× rates, despite triggering failures in over 80% of instances.With MSWE-agent, triggering a failure ranges from 25.59% to 75.59%; reaching code changed by the golden patch also does not guarantee success, with non-successful-instance change coverage above 0% in every configuration.Source: Table 4 · p. 6, §4.1.2 · p. 7, Figure 1 · p. 7

4.2.1 LANGUAGE-SPECIFIC CHALLENGES

  • Rust CLI tests can require matching command options to expected output; reused Java helpers can impose conditions beyond the issue, and new C++ tests can break existing output checks.In the Rust example, 18 of 45 issues involve CLI output or diagnostics; the C++ example concerns repository-specific validation workflows.Source: §4.2.1 · pp. 7–8

4.2.2 CROSS-LANGUAGE CHALLENGES

  • Reproduction can require implicit environment setup, such as controlling current time and enabling plugins; later lint-driven edits can also erase target behavior or prevent compilation.The examples include a Day.js issue, an Express test revised to lose the reported behavior, and a Ponyc test edit that introduced a duplicate definition.Source: §4.2.2 · p. 8

4.3 RESOURCE CONSUMPTION

  • With MOpenHands, DeepSeek-V4-Flash reaches 55.29% success at $0.0175/instance versus Claude Sonnet 5's 57.94% at $0.6222/instance.Across shared LLMs, MOpenHands has higher success and lower cost in three of four comparisons; subscription-plan methods are reported at API-equivalent costs.Source: §4.3 · p. 8, Table 5 · p. 8

5 RELATED WORK

Prior code benchmarks expanded from single-language, program-level tasks toward multilingual and repository-level evaluation. Reproduction-test benchmarks remained mostly monolingual, motivating a unified multilingual evaluation setting.

RELATED WORK

  • Early code benchmarks mainly evaluated program-level tasks in one language, while later work expanded multilingual and repository-level coverage.Examples include multilingual code-generation benchmarks and repository-level code-completion, bug-fixing, and unit-test-generation benchmarks.Source: §5 · p. 9
  • SWE-BENCH evaluates repository-level issue resolution, and MULTI-SWE-BENCH extends that setting to eight programming languages.The passage describes SWE-BENCH as requiring repository modifications to resolve real-world GitHub issues.Source: §5 · p. 9
  • SWT-BENCH and TDD-Bench-Verified focus on Python, while TDD-Bench-Java adds Java; their independent monolingual settings lack a unified cross-ecosystem view.MULTI-SWT-BENCH applies a common task definition, validation criterion, and evaluation protocol across eight language-specific subsets.Source: §5 · p. 9

6 CONCLUSION AND FUTURE WORK

The conclusion presents MULTI-SWT-BENCH and draws evaluation and development lessons from the empirical study and failure analysis. The supplied material also covers the prompts, factor analyses, and replication package.

6 CONCLUSION AND FUTURE WORK

  • The benchmark contains 1,963 instances across eight programming languages and supports unified comparisons across programming ecosystems.The conclusion frames its empirical results and failure analysis as a basis for lessons and future research.Source: §6 · p. 9
  • Python alone consistently exceeds aggregate success across evaluated configurations, while the authors call for multilingual evaluation and ecosystem-aware, behaviorally validated methods.They recommend inspecting test registration and aggregation, running repository-level checks, inferring implicit setup, and revalidating target behavior after revisions.Source: §6 · p. 9
  • The study provides a replication package containing benchmark data, evaluation scripts, experimental configurations, prompts, and reproduction instructions.The package is available at the DOI stated in the paper.Source: REPRODUCIBILITY STATEMENT · p. 10

A PROMPTS

  • The appendix presents the prompts for MSWE-agent, MOpenHands, Codex, and Claude Code, with task-specific changes highlighted in boldface.Source: A PROMPTS · p. 12

B ADDITIONAL ANALYSES OF FACTORS INFLUENCING PERFORMANCE

  • The additional analyses examine issue type, issue-description characteristics, fix-patch characteristics, and reference-test characteristics as factors associated with performance.Source: B ADDITIONAL ANALYSES OF FACTORS… · p. 12

B.1 ISSUE TYPE

The additional analyses find no consistent performance hierarchy across issue types and no consistent relationship between issue-description length and reproduction-test success. They also describe the categories and length groups used in those analyses.

B.1 ISSUE TYPE

  • The analysis compares bug fixes, new features, and feature optimizations using manually assigned MULTI-SWE-BENCH issue-type labels.Source: §B.1 · p. 12
  • Issue types show no consistent success-rate hierarchy across languages and methods; in Go, MSWE-agent records 21.05% for bug fixes, 40.00% for new features, and 50.00% for feature optimizations.MOpenHands and Codex show the opposite Go trend, with their highest success rates on bug fixes; Table 6 reports rates by issue type and language.Source: §B.1 · p. 12, Table 6 · p. 15

B.2 CHARACTERISTICS OF ISSUE DESCRIPTION

  • Issue-description length has no consistent relationship with success: longer reports may give precise code-location guidance or describe intrinsically complex issues.The study groups descriptions into five token-length intervals, from <100 to >1,000 tokens.Source: §B.2 · p. 15

B.3 CHARACTERISTICS OF GOLDEN FIX PATCHES

The patch-characteristic analyses find lower success for multi-file fixes in most languages, but no consistent relationship between fix-patch length and success. Patch length may not reflect the behavioral complexity a reproduction test must capture.

B.3 CHARACTERISTICS OF GOLDEN FIX PATCHES

  • Fix-patch length has no consistent relationship with success because long patches may be mechanical or complex, while short patches may address subtle behavior.Fix-patch length is measured by tokens in the complete fix diff and analyzed in five intervals.Source: §B.3 · p. 16, Figure 9 · p. 17
  • Success is lower for multi-file than single-file fixes in most languages and for most agents, with only a limited number of exceptions.The authors suggest that issues whose golden fixes span multiple files tend to be more difficult to reproduce.Source: §B.3 · p. 17, Figure 10 · p. 17

B.4 CHARACTERISTICS OF GOLDEN REFERENCE TESTS

Golden reference-test length has no monotonic relationship with reproduction-test success, although the longest group generally has lower success than the shortest. The association between execution scope and success differs by language.

B.4 CHARACTERISTICS OF GOLDEN REFERENCE TESTS

  • Golden reference-test length has no monotonic relationship with success, but tests of ≥901 tokens generally have lower success than those of ≤300 tokens across most languages and agents.Intermediate-length groups sometimes outperform shorter groups, so length alone does not consistently explain performance.Source: §B.4 · p. 18, Figure 13 · p. 18
  • For Python and Java, multi-file test execution has lower success than single-file execution across agents; this pattern is not consistent for TypeScript, and other languages lack sufficient single-file cases for comparison.The relationship between execution scope and performance is language-dependent, with no consistent language-general trend.Source: §B.4 · p. 18, Figure 14 · p. 19

C DETAILED EXAMPLES FROM THE FAILURE ANALYSIS

Repository conventions can make generated tests fail for reasons beyond the reported bug: expected output may encode unenabled options, reuse helpers may assert unrelated conditions, and added tests may disrupt existing checks. These examples show that test-suite interactions can also prevent otherwise successful reproductions from passing overall.

C.1.1 TESTING CONVENTIONS

  • The ripgrep test expected line-number prefixes without enabling line numbering, so it failed both before and after the patch despite output shrinking to three lines.Before the patch, the command produced too many lines; afterward it produced three lines, but without the expected prefixes.Source: §C.1.1 · pp. 19–20
  • The Jackson test reused a security-exception helper, so it still failed after the patch removed the null-pointer exception and produced an ordinary instantiation error.The helper required security-specific message phrases that the ordinary error did not contain.Source: §C.1.1 · p. 20

C.1.2 EFFECTS ON EXISTING TESTS

  • All four Catch2 tests passed after the patch, but their changed aggregate output caused an existing ApprovalTests check to fail.ApprovalTests compared the aggregate output against a saved expected output.Source: §C.1.2 · p. 21

C.2.1 IMPLICIT SETUP REQUIREMENTS

Reproducing the Day.js issue required more than using the reported timestamp: the test also had to initialize both plugins and control the current time. Runs that omitted an executable test did not reach those conditions.

C.2.1 IMPLICIT SETUP REQUIREMENTS

  • The Day.js issue required initializing RelativeTime and BadMutable and controlling the current time, in addition to using the reported timestamp.The issue's expected output was “4 months ago.”Source: §C.2.1 · p. 21
  • A successful MSWE-agent test initialized both plugins and fixed the time with MockDate, failing before the patch and passing afterward.Before the patch it returned “Infinity years ago” rather than “4 months ago.”Source: §C.2.1 · pp. 21–22
  • Other runs submitted empty test files or no new test, so they never reached an executable reproduction of the required conditions.An empty Jest test suite was rejected both before and after the patch.Source: §C.2.1 · pp. 21–22

C.2.2 LOSS OF INTENDED BEHAVIOR OR EXECUTABILITY AFTER REVISIONS

Iterative revisions can undermine a reproduction test by changing the behavior it exercises or making the test suite unexecutable. The Express and Ponyc examples show both outcomes after initially promising tests.

C.2.2 LOSS OF INTENDED BEHAVIOR OR EXECUTABILITY AFTER REVISIONS

  • For Express, replacing a lint-invalid async throwing handler with a function returning a rejected Promise changed the tested behavior, leaving the revised test failing on both versions.The golden patch handles rejected Promises only when the handler is an AsyncFunction; a controlled rerun showed the initial test failed before and passed after the patch.Source: §C.2.2 · pp. 22–23
  • The initial Ponyc test for the terminal-return diagnostic failed before the patch and passed afterward.Source: §C.2.2 · p. 23
  • A later Ponyc edit duplicated an existing GoogleTest definition, causing compilation to fail before the reproduction test could run on either version.The final test suite therefore failed both before and after the golden patch.Source: §C.2.2 · p. 23

Related papers

Loading 2609.34752v1…