PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents

Muhammad Shihab Rashid, Christian Bock, Yuan Zhuang, Alexander Buchholz, Tim Esler, Simon Valentin, Luca Franceschi, Martin Wistuba, Prabhu Teja Sivaprasad, Woo Jung Kim, Anoop Deoras, Giovanni Zappella, Laurent Callot

arXiv:2504.08703v3cs.SE

TL;DR

Existing coding-agent benchmarks have limited language and task diversity, while pass rates alone provide limited insight into repository navigation and code localization. SWE-PolyBench addresses these gaps with a multi-language, repository-level benchmark and syntax-tree retrieval metrics, finding uneven performance across languages and stronger results in Python than in Java, JavaScript, and TypeScript.

  • Problem

    Existing benchmarks are limited by Python-only repositories, bug-fix-heavy tasks, repository over-representation, and pass rates that do not fully capture repository navigation or code localization.

  • Method

    SWE-PolyBench combines 2110 samples from 21 repositories across four languages and three task categories with file- and Concrete Syntax Tree node-level retrieval metrics.

  • Results

    Agents perform unevenly across languages, with strongest pass rates in Python at 20 % to 24 %, compared with 11 % to 16 % in Java and 5 % to 13 % in TypeScript.

  • Takeaways & Limitations

    SWE-PolyBench provides a stratified SWE-PolyBench500 subset and syntax-tree metrics for comparing coding-agent capabilities across languages, task types, and code-change complexity.

  • Takeaways & Limitations

    The evaluation metrics do not assess code best practices, repository style-guide adherence, maintainability, or potential security flaws.

Abstract

from arXiv · show

Coding agents powered by large language models have shown impressive capabilities in software engineering tasks, but evaluating their performance across diverse programming languages and real-world scenarios remains challenging. We introduce SWE-PolyBench, a new multi-language benchmark for repository-level, execution-based evaluation of coding agents. SWE-PolyBench contains 2110 instances from 21 repositories and includes tasks in Java (165), JavaScript (1017), TypeScript (729) and Python (199), covering bug fixes, feature additions, and code refactoring. We provide a task and repository-stratified subsample (SWE-PolyBench500) and release an evaluation harness allowing for fully automated evaluation. To enable a more comprehensive comparison of coding agents, this work also presents a novel set of metrics rooted in syntax tree analysis. We evaluate leading open source coding agents on SWE-PolyBench, revealing their strengths and limitations across languages, task types, and complexity classes. Our experiments show that current agents exhibit uneven performances across languages and struggle with complex problems while showing higher performance on simpler tasks. SWE-PolyBench aims to drive progress in developing more versatile and robust AI coding assistants for real-world software engineering. Our datasets and code are available at: https://github.com/amazon-science/SWE-PolyBench

1 Introduction

SWE-PolyBench addresses the narrow language and task coverage of prior coding-agent evaluation by offering a diverse repository-level benchmark and syntax-tree-based metrics. It evaluates agents across languages and code-change complexity, exposing uneven performance and difficulties with complex, multi-file modifications.

  • Figure 1 organizes agent pass rates by programming language, syntax-tree complexity, and change type and scope.
  • SWE-Bench is limited to Python repositories, is dominated by bug fixes, and over-represents Django with more than 45% of tasks.
  • SWE-PolyBench spans 21 repositories, four languages, and three task categories to provide a more diverse coding-agent evaluation environment.
  • The benchmark introduces file- and Concrete Syntax Tree node-level retrieval metrics alongside pass rate to assess repository navigation and code localization.
  • Leading open-source agents show varying performance across languages and notable difficulty with complex, multi-file or extensive code changes.

2 Related work

Prior code-generation benchmarks often provide complete prompts or focus narrowly on Python bug fixing, limiting their realism and breadth. Repository-level execution-based benchmarks better reflect software-engineering work, but existing alternatives retain important coverage constraints.

  • Retrieval-free benchmarks provide necessary information in prompts but do not simulate the need to locate relevant code before editing.
  • Execution-based benchmarks assess generated-code correctness with predefined tests, while execution-free approaches use alternative evaluators such as LLM judges.
  • SWE-bench introduced repository-level, issue-driven editing of real-world Python codebases with unit-test-based correctness evaluation.
  • SWE-bench pass rates increased from 0.17% to 33.83% within one and a half years.
  • Most discussed benchmarks focus on Python bug-fixing tasks, while SWE-bench-java contains only 91 instances and is dominated by Jackson-related issues.

3 Building SWE-PolyBench

SWE-PolyBench is constructed from real-world pull requests across four languages and 21 repositories, then filtered through containerized execution tests. The resulting benchmark combines broad task coverage with reproducible runtime environments and explicit testability requirements.

  • SWE-PolyBench contains 2110 samples from 21 repositories across Java, JavaScript, TypeScript, and Python, covering bug reports, feature requests, and refactoring requests.
  • The collection pipeline gathers issue-closing pull requests, applies metadata filtering, creates containerized test environments, and compares outcomes before and after patches.
  • The dataset separates non-test code changes as the ground-truth code patch, test-only changes as the test patch, and identifies the base commit for execution.
  • Language-specific Docker setups accommodate different package managers, installation procedures, and version requirements.
  • Each included pull request must contain at least one fail-to-pass test, while cases with newly created files tested by the patch are excluded as untestable.
  • TypeScript repositories average 8946.0 files, whereas Python repositories average 1928.1 files, including binary and documentation files.

4 SWE-PolyBench Characteristics

SWE-PolyBench broadens repository-level evaluation across languages, task types, and structurally complex changes. Its analyses characterize issue informativeness and show that complexity varies substantially across languages and is higher than SWE-Bench.

  • Dataset comparison: SWE-PolyBench and SWE-PolyBench500 are compared with SWE-Bench and SWE-Bench verified using file statistics and task-category distributions.PB500 is a stratified subset designed to preserve repository coverage and task distributions while enabling manageable experimentation.
  • Dataset comparison: 63% more files need to be edited to solve a SWE-PolyBench task than a SWE-Bench task overall.The benchmark exhibits higher modified-file complexity across all languages.
  • Dataset comparison: Java has the highest average file-modification complexity at 3.6 files per task, while PB500 increases feature requests to 36.80% and refactoring tasks to over 12%.The adjusted PB500 composition supports a more balanced evaluation of software-engineering scenarios.
  • Structural complexity: CST-based analysis counts deepest affected class or function nodes, avoiding double-counting a changed function within its enclosing class.The approach supplements modified-file counts with finer-grained change-location information.
  • Structural complexity: Java reaches 66.06% mixed node changes and 9.81 average node modifications, whereas JavaScript has 84.27% function-only changes and TypeScript has 30.59% non-class/function changes.Python averages 5.76 node modifications and has 37.69% mixed changes; PB500 amplifies complexity, especially for Java.
  • Issue characterization: Issue descriptions are classified by task type and informativeness, including descriptiveness, solution hints, and localization information.The five task categories are Bug Fix, Feature Request, Refactoring, Security, and Testing, while Figure 3 reports the three informativeness dimensions.
  • Issue characterization: Most problem statements provide no exact or complete solution, while sufficient natural-language descriptions and reproducible examples dominate descriptiveness.Solution content is usually absent or partial across the benchmark’s programming languages.

5 Evaluating Open-Source Coding Agents

The evaluation adapts three open-source coding agents to SWE-PolyBench and assesses them using execution-based pass rates alongside file- and CST-node-level retrieval metrics. Results vary substantially by language and change type, with stronger performance on simpler modifications and persistent challenges for complex tasks.

  • Evaluation Setup: Three open-source agents—Aider, SWE-agent, and Agentless—were adapted into Aider-PB, SWE-Agent-PB, and Agentless-PB for SWE-PolyBench.Unless otherwise stated, the modified agents use Claude 3.5 as their foundation model.
  • Technical Challenges: Aider-PB excludes its original regression-validation step because test commands and log parsers would need to support diverse languages and frameworks.The original validation process was tailored to Python and pytest.
  • Metrics: Pass rate counts an instance as solved when both pass-to-pass and fail-to-pass tests succeed on the agent’s updated repository.The evaluation combines the two Boolean test outcomes through conjunction.
  • Metrics: File-level recall and precision are complemented by CST node-level retrieval metrics to measure repository navigation and localization beyond final code-generation success.CST analysis represents structural elements such as functions and classes as labelled nodes and accounts for their tree depth.
  • Pass Rates: 20% to 24% pass rates in Python exceeded Java’s 11% to 16% and TypeScript’s 5% to 13%, while all agents struggled most with mixed changes at 8% to 15%.Class Only and Single Class modifications reached 25% to 40%, whereas Function Only and Single Function changes were around 15%.
  • Pass Rates: Aider-PB achieved the highest average pass rates and used only 19% to 20% of the input tokens per instance compared with the other agents in the same comparison.Performance decreased as tasks required edits to more files; for three or more files, success rates fell below ten percent.

6 Limitations

SWE-PolyBench’s limitations concern benchmark scope, execution-based evaluation, task verifiability, data leakage, and incomplete quality assessment. These constraints bound how its results should be interpreted and motivate broader future evaluations.

  • Task Diversity: The benchmark omits many software-engineering activities beyond code changes, including application deployment and integration testing.Its three task classes target common coding tasks but do not cover the long tail of day-to-day developer work.
  • Evaluation Metrics: The evaluation metrics do not assess coding best practices, repository style adherence, maintainability, or potential security flaws.The authors identify these omissions as important targets for more holistic future evaluations.
  • Limits of execution-based evaluation: Its execution-based evaluation is constrained by tests and may not support major structural changes, such as creating a website from scratch.Unit-test execution is quick and inexpensive, but the benchmark remains tied to repository changes that can be tested automatically.
  • Verifiability: Some tasks lack human verification and may be ambiguous, incomplete, or unsolvable from the provided information alone.The authors recommend balancing verifiability with the varied issue-description quality found in practical software development.
  • LLM-based Classifications: LLM-based annotations of task type and description quality provide complementary evaluation information but carry interpretation risks.The authors caution that these annotations should be interpreted with awareness of known risks associated with LLM-based labeling.
  • Data Leakage: Publicly available source data may have appeared in underlying LLM training sets, limiting the credibility of benchmark results.Opaque training datasets and frequent model releases shorten the window for creating evaluation data unaffected by leakage.

7 Conclusions

The paper introduces SWE-PolyBench as a multi-language, repository-level benchmark and evaluates open-source coding agents across languages and task complexity. Results show language-dependent performance, declining success on more complex edits, and finer-grained repository-navigation analysis from syntax-tree metrics.

  • 7 Conclusions: SWE-PolyBench contains 2110 samples from 21 repositories across Java, JavaScript, TypeScript, and Python, covering bug fixes, features, and refactoring.The paper also provides SWE-PolyBench500 and syntax-tree-based retrieval metrics.
  • 7 Conclusions: Agents perform more strongly on Python tasks than on Java, JavaScript, and TypeScript tasks.The evaluation required substantial agent modifications for multi-language operation, highlighting limited adaptability.
  • 7 Conclusions: Performance consistently declines as task complexity increases, especially for multi-file edits and simultaneous class-and-function modifications.This pattern identifies complex repository changes as a persistent challenge for evaluated agents.
  • 7 Conclusions: Syntax-tree retrieval metrics provide finer-grained evidence about agents’ ability to navigate code repositories.They complement execution-based outcomes with node-level localization information.

A.1 Prompts

The appendix documents prompts used to classify task categories and problem-statement properties. These prompts specify labels, required outputs, and the information available to the annotator.

  • A.1.1 Prompt for classification of tasks: The task-classification prompt assigns each issue to exactly one of Bug Fix, Feature, or Refactoring.The requested output is an XML category tag with no additional text.
  • A.1.1 Prompt for classification of tasks: The task-classification prompt uses the issue description and gold patch to categorize the repository problem.The gold patch supplies the modified, added, or removed files and code lines associated with the solution.
  • A.1.1 Prompt for classification of tasks: All prompt variants require structured XML outputs containing an explanation and the selected label.The task prompt separately requires exactly one category output without additional XML tags.
  • A.1.2 Prompts for classification of problem statements: The problem-statement prompts evaluate detail, solvability, and whether precise code locations are identified.They require a brief explanation followed by an XML label for each assessment.
  • A.1.2 Prompts for classification of problem statements: Solution-information labels distinguish no solution, partial or complete steps, an exact patch, and misleading guidance.The classification uses labels A through E and requires exactly one label per issue description.
  • A.1.2 Prompts for classification of problem statements: Solvability labels range from sufficient natural-language information to insufficient information for solving the issue.The prompt distinguishes reproducible, partially reproducible, and non-solvable descriptions.
  • A.1.2 Prompts for classification of problem statements: The annotation prompts provide the issue description and ground-truth patch but not the codebase itself.This constraint applies to the problem-statement quality assessments.
  • A.1.2 Prompts for classification of problem statements: Location labels distinguish exact natural-language locations, stack-trace locations, searchable keywords, and no location.The prompt asks whether the issue description identifies which code must be modified or fixed.

A.2 Details CST retrieval metrics

The CST retrieval metric identifies the deepest syntax-tree node affected by a code change. Ground-truth and predicted deepest-node sets then support node-level retrieval measurement.

  • A.2 Details CST retrieval metrics: The deepest affected node is the affected CST node that contains no other affected node.This selects the most specific affected structural unit rather than an enclosing class or function.
  • A.2 Details CST retrieval metrics: A CST is modeled with vertices, directed edges, node labels, a root, label mapping, and line spans for nodes.Labels can identify structures such as classes and functions, while each node maps to a line interval.
  • A.2 Details CST retrieval metrics: Node-level retrieval compares deepest nodes modified in the ground-truth patch with deepest nodes affected by predicted changes.The metric construction assumes a CST for each file in which changes occur.

A.3 Task classifications and pass rates

SWE-PolyBench500 classifies issue descriptions by their informativeness and relates these categories to coding-agent pass rates. More informative statements are associated with higher pass rates.

  • Task classifications: Issue descriptions are classified by descriptiveness, solution hints, and issue-localization information.These dimensions are organized as levels of information content in the task-classification scheme.
  • Pass rates: More informative problem statements produce higher pass rates across agents.The pattern holds for descriptiveness, solution hints, and localization information.
  • Pass rates: Pass rates are evaluated for instances grouped by complexity in terms of Concrete Syntax Tree node changes.The complexity breakdown is reported in Table 8.

A.4 Results on SWE-PolyBench500

Results on SWE-PolyBench500 compare coding-agent pass rates across programming languages and code-change complexities. The evaluation also distinguishes structural change types and scopes, including single or multiple classes or functions.

  • Programming languages: Coding-agent pass rates are compared across Java, JavaScript, TypeScript, Python, and overall performance.The language comparison is shown in the left radar chart of Figure 6.
  • Code-change complexity: Figure 6 evaluates code modifications ranging from single-class or single-function changes to changes involving multiple structural elements.The right chart focuses on code-change complexity and structural scope.
  • Issue informativeness: The results include pass-rate comparisons for instances grouped by issue-statement informativeness.Figure 5 organizes pass rates by descriptiveness, solution hints, and issue-localization information.
Loading 2504.08703v3…