PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale

Ibragim Badertdinov, Maksim Nekrashevich, Anton Shevtsov, Alexander Golubev

arXiv:2602.23866v2cs.SEcs.CL

TL;DR

SWE-agent training lacks large, reproducible, multilingual task collections with reliable execution and test signals. SWE-rebench V2 builds such resources through automated environment synthesis and quality filtering, producing 32,079 executable tasks across 20 languages plus 120,000+ PR-derived tasks. The paper also provides diagnostics and ablations to characterize task quality and construction failure modes.

  • Problem

    Large-scale SWE-agent training is constrained by limited multilingual task collections with reproducible environments, reliable tests, and training-oriented artifacts.

    Source: §1 · pp. 1–2

  • Method

    The paper combines interactive repository setup synthesis with automated oracle extraction, issue-clarity filtering, human-calibrated LLM judging, and diagnostic metadata.

    Source: §1 · p. 2, §4.1 · p. 6, §4.2 · p. 7

  • Results

    The pipeline releases 32,079 executable tasks across 20 languages and 3,617 repositories, supplemented by 120,000+ PR-derived tasks and instance-level diagnostics.

    Source: §6 · p. 9

  • Takeaways & Limitations

    SWE-rebench V2 provides multilingual, training-oriented SWE task resources with reproducible environments and metadata for filtering or controlled analysis.

    Source: §1 · p. 2, §6 · p. 9

    The paper does not include end-to-end RL training ablations and instead validates prerequisite properties for training.

    Source: §5 · p. 8

Abstract

from arXiv · show

Software engineering agents (SWE) are improving rapidly, with recent gains largely driven by reinforcement learning (RL). However, RL training is constrained by the scarcity of large-scale task collections with reproducible execution environments and reliable test suites. Although a growing number of benchmarks have emerged, datasets suitable for training remain limited in scale and diversity or often target a limited set of high-resource language ecosystems. We introduce SWE-rebench V2, a language-agnostic automated pipeline for harvesting executable real-world SWE tasks and constructing RL training environments at scale. The pipeline synthesizes repository-specific installation and test procedures via an interactive setup agent, and filters unsound instances using an ensemble of LLM judges, validated against human-verified SWE-bench annotations. Using this pipeline, we construct a dataset of 32,079 tasks spanning 20 languages and 3,617 repositories, with pre-built images for reproducible execution. To further scale training data, we additionally release 120,000+ tasks with installation instructions, fail-to-pass tests and rich metadata, where the problem statement is generated based on the original pull request description. We validate the collected instances through a diagnostic study that covers a subset of tasks in five programming languages across seven popular models, and provide instance-level metadata that flags common confounders such as overly restrictive tests and underspecified descriptions. We release the datasets, the collection and execution code, and associated artifacts to enable large-scale training of SWE agents across diverse languages and repositories.

1. Introduction

SWE-rebench V2 addresses the shortage of scalable, reproducible training environments for repository-level SWE agents by providing a language-agnostic construction pipeline and training-oriented task releases.

  • Executable SWE training environments require correct dependency installation, reproducible tests, and alignment between specifications and test oracles.Source: §1 · p. 1
  • SWE-rebench V2 applies one end-to-end workflow across languages while reusing language-specific base images, runners, and parsers.Source: §1 · p. 2
  • The pipeline combines interactive environment synthesis, automated oracle extraction, and quality filtering without per-instance human verification.Source: §1 · p. 2
  • 32,000+ containerized tasks span 3,600+ repositories and 20 programming languages, with executable environments and pre-built images.Source: §1 · p. 2
  • 120,000+ additional tasks provide installation and test recipes, metadata, and PR-description-based problem statements for larger-scale learning.Source: §1 · p. 2
  • Instance-level diagnostics cover factors such as external dependencies, test brittleness, and underspecification across analyses of hundreds of tasks and thousands of trajectories.Source: §1 · p. 2

2. Related Work

Prior work broadens SWE evaluation and automates task construction, but resources remain limited in language coverage or training-oriented reproducibility. SWE-rebench V2 targets these gaps with a unified executable workflow, larger PR-derived data, and diagnostics.

  • Earlier automated pipelines largely center on Python or a small number of languages, leaving robustness across long-tail toolchains unresolved.Source: §2 · p. 2
  • SWE-rebench V2 adds pre-built environments and instance-level diagnostics for reproducible interactive learning beyond evaluation-first datasets.Source: §2 · pp. 2–3
  • PR-derived problem statements reduce dependence on issue–PR linkage and support a substantially larger recipe-scale learning corpus alongside containerized tasks.Source: §2 · p. 3
  • The pipeline integrates automated quality assessment calibrated against human-verified data and focuses on real issue-resolution histories rather than only synthetic instances.Source: §2 · p. 3

3. Pipeline

The pipeline mines real pull requests, synthesizes reusable repository environments, validates fail-to-pass behavior, filters task quality, and enriches instances with diagnostics. It yields a stable multilingual corpus plus a larger, lower-confidence PR-derived resource.

Pipeline stages

  • The five-stage workflow mines candidate PRs, synthesizes setup procedures, validates environments, filters issue clarity, and adds diagnostic metadata.Source: §3 · p. 3

Preliminary Data Collection

  • Repositories are cloned and filtered by language, activity, licensing, issue resolution, merged PR status, and the presence of modified tests.Source: §3.1 · pp. 3–4

Setup Synthesis

  • The interactive setup agent iteratively inspects repositories, installs dependencies, refines scripts from failures, and verifies reproducible test execution inside Docker environments.Source: §3.2 · p. 4
  • Setup instructions are inferred once per repository and reused across its mined tasks, with language-specific base images and multi-stage builds supporting execution at scale.Source: §3 · p. 3, §3.2 · p. 4, §3.3 · p. 5

PR-based Task Expansion

  • The PR-derived collection generates problem statements from PR descriptions and patches, but is released as a larger, lower-confidence resource with leakage-screening metadata.Source: §3.6 · p. 5

Pipeline Funnel

  • Around 20% of repositories succeed with a single setup attempt, indicating that retries or additional repository states could improve yield.Source: §3.7 · p. 6
  • 32,079 tasks span 20 languages and 2014–2025, with a median task modifying 3 files and 34 lines and coverage across up to 12 PR categories.Source: §3.7 · p. 6

4. Experiments and Details

The experiments evaluate setup synthesis, issue-clarity filtering, and diagnostic metadata across multilingual SWE tasks. Results show interactive setup, repeated attempts, ensemble filtering, and diagnostic labels improve the reliability and interpretability of large-scale training data.

4.1. Setup Synthesis

  • Interactive setup agents consistently outperform non-interactive pipelines, including comparisons against a much larger Qwen3-Coder-480B model.The study varies interactivity, model choice, setup attempts, and context length on 103 tasks spanning ten languages.Source: §4.1 · p. 6
  • 32k-token contexts suffice for most repositories, while longer contexts can increase looping or attention to irrelevant details.Trivial setups are resolved quickly regardless of context length.Source: §4.1 · p. 6
  • Increasing setup attempts substantially improves installation success, with ten runs approaching a twofold improvement over one run in some settings.The main pipeline nevertheless uses one setup run as a throughput-oriented cost-yield trade-off, while allowing more runs when compute is available.Source: §4.1 · p. 6

4.2. Filtering by Issue Clarity

  • Verified+ achieves the best F1 score for issue-clarity filtering, while Verified-E achieves the highest precision and gpt-oss-120b offers the best balanced individual-judge performance.The evaluation uses SWE-bench Verified annotations, whose well-specified labels aggregate three human scores.Source: §4.2 · p. 7
  • Averaging ensemble scores improves robustness and yields the best overall F1, whereas strict consensus is preferable when precision is prioritized.These strategies aggregate scores from gpt-oss-120b, GLM 4.7, and DeepSeek v3.2.Source: §4.2 · p. 7

4.3. Task Analysis

  • Execution-trajectory analysis identifies test-suite coupling, implicit naming requirements, and inaccessible external dependencies as task-formulation or environment pathologies.These findings motivate instance-level metadata enrichment and filtering for training-set curation.Source: §4.3 · pp. 7–8
  • Code A tasks substantially outperform B* tasks across evaluated models, supporting diagnostic metadata as a way to distinguish cleaner learning signals from confounded settings.For example, Gemini reaches 34.0% pass@3 on Code A versus 4.0% on Code B*, while GLM-4.7 reaches 34.0% versus 6.0%.Source: §4.3 · p. 8

5. Discussion and Limitations

SWE-rebench V2 targets the scarcity of diverse executable training data, while acknowledging that automated collection can introduce reward noise and leaves several validation and infrastructure boundaries.

  • The released resources target scarce, diverse SWE training data, especially for long-tail languages and broad repository coverage.Source: §5 · p. 8
  • Automated setup and validation can introduce reward noise through environment-preparation imperfections.Source: §5 · p. 8
  • The paper does not include end-to-end RL training ablations, instead validating prerequisite properties for training.Source: §5 · p. 8
  • Direct comparisons of filtered and unfiltered curricula remain future work.Source: §5 · p. 9
  • Docker images reduce repository-evolution problems but cannot prevent drift in external packages, system packages, or network-hosted resources.Source: §5 · p. 9
  • The single-container design limits coverage of systems requiring multiple services, databases, queues, or distributed infrastructure.Source: §5 · p. 9

6. Conclusion and Future Work

SWE-rebench V2 presents an automated, language-agnostic pipeline and releases large collections of executable and PR-derived SWE tasks. Future work focuses on improving yield, language coverage, task complexity, and reward signals.

  • 32,079 stable issue-linked tasks span 3,617 repositories across 20 languages, supplemented by 120,000+ PR-derived tasks.Source: §6 · p. 9
  • The pipeline automates task construction from pull-request mining through executable-environment synthesis and instance filtering without manual verification.Source: §6 · p. 9
  • Future work will increase setup retries, add curated subsets, and onboard more long-tail languages.Source: §6 · p. 9
  • Future work will extend support to complex, long-horizon tasks in multi-service systems and investigate rewards beyond test-based correctness.Source: §6 · p. 9
  • The released pipeline and resources provide a practical foundation for training and evaluating LLM-based agents on realistic SWE tasks at scale.Source: §6 · p. 9

Impact Statement

The work releases resources intended to support research on LLM-based SWE agents across languages and repositories, alongside safety measures and cautions about inherited dataset biases.

  • The released resources may support research on more capable and robust LLM-based SWE agents across multiple programming languages and real-world repositories.Source: Impact Statement · p. 9
  • Repositories come from publicly available projects with permissive open-source licenses, with automated scans for leaked credentials.Source: Impact Statement · p. 9
  • Tasks derived from open-source code may reflect biases or assumptions present in the underlying repositories.Source: Impact Statement · p. 9

A. Supplementary Prompts and Artifacts

The supplementary materials provide repository-filtering and installation-ablation tables alongside a Julia base Dockerfile example. The Dockerfile configures a reproducible, writable environment with common build dependencies and Julia-specific settings.

A.1. Repo Filtering Shares by Language

  • Table 8 reports the share of tasks retained after repository filtering by language.Source: Table 8 · p. 12
  • Table 9 presents an ablation test set for project installation.Source: Table 9 · p. 12

A.2. Base Dockerfile Example

  • It installs common version-control, archive, compiler, build, and native-library dependencies required by many Julia packages.Source: §A.2 · p. 12
  • The image sets Julia package-server, MPI, and thread-related environment defaults.Source: §A.2 · p. 12
  • A non-root user is added for running projects without root privileges.Source: §A.2 · p. 13
  • Projects use /workspace as the working directory.Source: §A.2 · p. 13
  • The Julia depot is kept inside /workspace so package caches remain writable for the non-root user.Source: §A.2 · p. 13

A.3. Prompt Templates and Examples

The appendix specifies prompts for generating reusable language base images and synthesizing repository-specific installation and testing configurations. These prompts emphasize conservative dependencies, documented commands, non-interactive execution, and verbose, reproducible test outputs.

A.3.1. BASE DOCKERFILE GENERATOR PROMPT

  • The base-image prompt targets a reusable Dockerfile containing the runtime, toolchain, and common system dependencies needed to install, build, and test projects.Source: §A.3.1 · p. 13
  • It avoids repository-specific assumptions and favors official stable images, broadly useful packages, and linux/amd64.Source: §A.3.1 · p. 13
  • The generated Dockerfile should remain short, clean package-manager caches, and contain only Dockerfile contents.Source: §A.3.1 · pp. 13–14
  • The example configures language package and cache directories under /workspace and creates a non-root user.Source: §A.3.1 · pp. 13–14

A.3.2. PROMPT FOR SETUP SYNTHESIS

  • The setup agent discovers documented installation and test procedures, identifies the project root, and uses lockfile-aware dependency commands.Source: §A.3.2 · p. 14
  • It records only necessary successful installation commands, keeps runtime installation out of install_config.json, and sets minimal required environment variables.Source: §A.3.2 · p. 15
  • Before running the full suite, the agent verifies test discovery and then uses general, verbose, no-color commands that expose individual test names and statuses.Source: §A.3.2 · pp. 14–16
  • The interaction is non-interactive and produces install_config.json with ordered commands, test execution settings, and parser-relevant outputs.Source: §A.3.2 · pp. 14–15

B. Additional Plots

The additional plots provide supplementary views of the benchmark corpus. They cover temporal and language distributions, alongside issue-type and patch-size composition.

C.1. Per-language Performance with Confidence Intervals

The per-language plots report pass@1, standard error, 95% confidence intervals, and pass@3 for selected languages and pooled tasks. Their captions specify task counts but not comparative outcomes.

Go

  • The Go panel reports pass@1, SEM, 95% CI, and pass@3 over 60 tasks.Source: §C.1 · p. 24
  • Across panels, the reported metrics are pass@1, SEM, 95% CI, and pass@3.Source: §C.1 · p. 24

Python

  • The Python panel reports pass@1, SEM, 95% CI, and pass@3 over 60 tasks.Source: §C.1 · p. 24

Scala

  • The Scala panel reports pass@1, SEM, 95% CI, and pass@3 over 60 tasks.Source: §C.1 · p. 24

JavaScript

  • The JavaScript panel reports pass@1, SEM, 95% CI, and pass@3 over 60 tasks.Source: §C.1 · p. 24

Rust

  • The Rust panel reports pass@1, SEM, 95% CI, and pass@3 over 60 tasks.Source: §C.1 · p. 24

All

  • The pooled All panel reports pass@1, SEM, 95% CI, and pass@3 over 300 tasks.Source: §C.1 · p. 24

Related papers

Loading 2602.23866v2…