Source-linked AI summary
SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale
Ibragim Badertdinov, Maksim Nekrashevich, Anton Shevtsov, Alexander Golubev
TL;DR
SWE-agent training lacks large, reproducible, multilingual task collections with reliable execution and test signals. SWE-rebench V2 builds such resources through automated environment synthesis and quality filtering, producing 32,079 executable tasks across 20 languages plus 120,000+ PR-derived tasks. The paper also provides diagnostics and ablations to characterize task quality and construction failure modes.
Problem
Large-scale SWE-agent training is constrained by limited multilingual task collections with reproducible environments, reliable tests, and training-oriented artifacts.
Method
The paper combines interactive repository setup synthesis with automated oracle extraction, issue-clarity filtering, human-calibrated LLM judging, and diagnostic metadata.
Results
The pipeline releases 32,079 executable tasks across 20 languages and 3,617 repositories, supplemented by 120,000+ PR-derived tasks and instance-level diagnostics.
Takeaways & Limitations
SWE-rebench V2 provides multilingual, training-oriented SWE task resources with reproducible environments and metadata for filtering or controlled analysis.
Takeaways & Limitations
The paper does not include end-to-end RL training ablations and instead validates prerequisite properties for training.
Abstract
from arXiv · showhide
Software engineering agents (SWE) are improving rapidly, with recent gains largely driven by reinforcement learning (RL). However, RL training is constrained by the scarcity of large-scale task collections with reproducible execution environments and reliable test suites. Although a growing number of benchmarks have emerged, datasets suitable for training remain limited in scale and diversity or often target a limited set of high-resource language ecosystems. We introduce SWE-rebench V2, a language-agnostic automated pipeline for harvesting executable real-world SWE tasks and constructing RL training environments at scale. The pipeline synthesizes repository-specific installation and test procedures via an interactive setup agent, and filters unsound instances using an ensemble of LLM judges, validated against human-verified SWE-bench annotations. Using this pipeline, we construct a dataset of 32,079 tasks spanning 20 languages and 3,617 repositories, with pre-built images for reproducible execution. To further scale training data, we additionally release 120,000+ tasks with installation instructions, fail-to-pass tests and rich metadata, where the problem statement is generated based on the original pull request description. We validate the collected instances through a diagnostic study that covers a subset of tasks in five programming languages across seven popular models, and provide instance-level metadata that flags common confounders such as overly restrictive tests and underspecified descriptions. We release the datasets, the collection and execution code, and associated artifacts to enable large-scale training of SWE agents across diverse languages and repositories.
1. Introduction
SWE-rebench V2 addresses the shortage of scalable, reproducible training environments for repository-level SWE agents by providing a language-agnostic construction pipeline and training-oriented task releases.
- Executable SWE training environments require correct dependency installation, reproducible tests, and alignment between specifications and test oracles.
- SWE-rebench V2 applies one end-to-end workflow across languages while reusing language-specific base images, runners, and parsers.
- The pipeline combines interactive environment synthesis, automated oracle extraction, and quality filtering without per-instance human verification.
- 32,000+ containerized tasks span 3,600+ repositories and 20 programming languages, with executable environments and pre-built images.
- 120,000+ additional tasks provide installation and test recipes, metadata, and PR-description-based problem statements for larger-scale learning.
- Instance-level diagnostics cover factors such as external dependencies, test brittleness, and underspecification across analyses of hundreds of tasks and thousands of trajectories.
2. Related Work
Prior work broadens SWE evaluation and automates task construction, but resources remain limited in language coverage or training-oriented reproducibility. SWE-rebench V2 targets these gaps with a unified executable workflow, larger PR-derived data, and diagnostics.
- Earlier automated pipelines largely center on Python or a small number of languages, leaving robustness across long-tail toolchains unresolved.
- SWE-rebench V2 adds pre-built environments and instance-level diagnostics for reproducible interactive learning beyond evaluation-first datasets.
- PR-derived problem statements reduce dependence on issue–PR linkage and support a substantially larger recipe-scale learning corpus alongside containerized tasks.
- The pipeline integrates automated quality assessment calibrated against human-verified data and focuses on real issue-resolution histories rather than only synthetic instances.
3. Pipeline
The pipeline mines real pull requests, synthesizes reusable repository environments, validates fail-to-pass behavior, filters task quality, and enriches instances with diagnostics. It yields a stable multilingual corpus plus a larger, lower-confidence PR-derived resource.
- Pipeline stages: The five-stage workflow mines candidate PRs, synthesizes setup procedures, validates environments, filters issue clarity, and adds diagnostic metadata.
- Preliminary Data Collection: Repositories are cloned and filtered by language, activity, licensing, issue resolution, merged PR status, and the presence of modified tests.
- Setup Synthesis: The interactive setup agent iteratively inspects repositories, installs dependencies, refines scripts from failures, and verifies reproducible test execution inside Docker environments.
- Setup Synthesis: Setup instructions are inferred once per repository and reused across its mined tasks, with language-specific base images and multi-stage builds supporting execution at scale.
- PR-based Task Expansion: The PR-derived collection generates problem statements from PR descriptions and patches, but is released as a larger, lower-confidence resource with leakage-screening metadata.
- Pipeline Funnel: Around 20% of repositories succeed with a single setup attempt, indicating that retries or additional repository states could improve yield.
- Pipeline Funnel: 32,079 tasks span 20 languages and 2014–2025, with a median task modifying 3 files and 34 lines and coverage across up to 12 PR categories.
4. Experiments and Details
The experiments evaluate setup synthesis, issue-clarity filtering, and diagnostic metadata across multilingual SWE tasks. Results show interactive setup, repeated attempts, ensemble filtering, and diagnostic labels improve the reliability and interpretability of large-scale training data.
- 4.1. Setup Synthesis: Interactive setup agents consistently outperform non-interactive pipelines, including comparisons against a much larger Qwen3-Coder-480B model.The study varies interactivity, model choice, setup attempts, and context length on 103 tasks spanning ten languages.
- 4.1. Setup Synthesis: 32k-token contexts suffice for most repositories, while longer contexts can increase looping or attention to irrelevant details.Trivial setups are resolved quickly regardless of context length.
- 4.1. Setup Synthesis: Increasing setup attempts substantially improves installation success, with ten runs approaching a twofold improvement over one run in some settings.The main pipeline nevertheless uses one setup run as a throughput-oriented cost-yield trade-off, while allowing more runs when compute is available.
- 4.2. Filtering by Issue Clarity: Verified+ achieves the best F1 score for issue-clarity filtering, while Verified-E achieves the highest precision and gpt-oss-120b offers the best balanced individual-judge performance.The evaluation uses SWE-bench Verified annotations, whose well-specified labels aggregate three human scores.
- 4.2. Filtering by Issue Clarity: Averaging ensemble scores improves robustness and yields the best overall F1, whereas strict consensus is preferable when precision is prioritized.These strategies aggregate scores from gpt-oss-120b, GLM 4.7, and DeepSeek v3.2.
- 4.3. Task Analysis: Execution-trajectory analysis identifies test-suite coupling, implicit naming requirements, and inaccessible external dependencies as task-formulation or environment pathologies.These findings motivate instance-level metadata enrichment and filtering for training-set curation.
- 4.3. Task Analysis: Code A tasks substantially outperform B* tasks across evaluated models, supporting diagnostic metadata as a way to distinguish cleaner learning signals from confounded settings.For example, Gemini reaches 34.0% pass@3 on Code A versus 4.0% on Code B*, while GLM-4.7 reaches 34.0% versus 6.0%.
5. Discussion and Limitations
SWE-rebench V2 targets the scarcity of diverse executable training data, while acknowledging that automated collection can introduce reward noise and leaves several validation and infrastructure boundaries.
- The released resources target scarce, diverse SWE training data, especially for long-tail languages and broad repository coverage.
- Automated setup and validation can introduce reward noise through environment-preparation imperfections.
- The paper does not include end-to-end RL training ablations, instead validating prerequisite properties for training.
- Direct comparisons of filtered and unfiltered curricula remain future work.
- Docker images reduce repository-evolution problems but cannot prevent drift in external packages, system packages, or network-hosted resources.
- The single-container design limits coverage of systems requiring multiple services, databases, queues, or distributed infrastructure.
6. Conclusion and Future Work
SWE-rebench V2 presents an automated, language-agnostic pipeline and releases large collections of executable and PR-derived SWE tasks. Future work focuses on improving yield, language coverage, task complexity, and reward signals.
- 32,079 stable issue-linked tasks span 3,617 repositories across 20 languages, supplemented by 120,000+ PR-derived tasks.
- The pipeline automates task construction from pull-request mining through executable-environment synthesis and instance filtering without manual verification.
- Future work will increase setup retries, add curated subsets, and onboard more long-tail languages.
- Future work will extend support to complex, long-horizon tasks in multi-service systems and investigate rewards beyond test-based correctness.
- The released pipeline and resources provide a practical foundation for training and evaluating LLM-based agents on realistic SWE tasks at scale.
Impact Statement
The work releases resources intended to support research on LLM-based SWE agents across languages and repositories, alongside safety measures and cautions about inherited dataset biases.
- The released resources may support research on more capable and robust LLM-based SWE agents across multiple programming languages and real-world repositories.
- Repositories come from publicly available projects with permissive open-source licenses, with automated scans for leaked credentials.
- Tasks derived from open-source code may reflect biases or assumptions present in the underlying repositories.
A. Supplementary Prompts and Artifacts
The supplementary materials provide repository-filtering and installation-ablation tables alongside a Julia base Dockerfile example. The Dockerfile configures a reproducible, writable environment with common build dependencies and Julia-specific settings.
- A.1. Repo Filtering Shares by Language: Table 8 reports the share of tasks retained after repository filtering by language.
- A.1. Repo Filtering Shares by Language: Table 9 presents an ablation test set for project installation.
- A.2. Base Dockerfile Example: It installs common version-control, archive, compiler, build, and native-library dependencies required by many Julia packages.
- A.2. Base Dockerfile Example: The image sets Julia package-server, MPI, and thread-related environment defaults.
- A.2. Base Dockerfile Example: A non-root user is added for running projects without root privileges.
- A.2. Base Dockerfile Example: Projects use /workspace as the working directory.
- A.2. Base Dockerfile Example: The Julia depot is kept inside /workspace so package caches remain writable for the non-root user.
A.3. Prompt Templates and Examples
The appendix specifies prompts for generating reusable language base images and synthesizing repository-specific installation and testing configurations. These prompts emphasize conservative dependencies, documented commands, non-interactive execution, and verbose, reproducible test outputs.
- A.3.1. BASE DOCKERFILE GENERATOR PROMPT: The base-image prompt targets a reusable Dockerfile containing the runtime, toolchain, and common system dependencies needed to install, build, and test projects.
- A.3.1. BASE DOCKERFILE GENERATOR PROMPT: It avoids repository-specific assumptions and favors official stable images, broadly useful packages, and linux/amd64.
- A.3.1. BASE DOCKERFILE GENERATOR PROMPT: The generated Dockerfile should remain short, clean package-manager caches, and contain only Dockerfile contents.
- A.3.1. BASE DOCKERFILE GENERATOR PROMPT: The example configures language package and cache directories under /workspace and creates a non-root user.
- A.3.2. PROMPT FOR SETUP SYNTHESIS: The setup agent discovers documented installation and test procedures, identifies the project root, and uses lockfile-aware dependency commands.
- A.3.2. PROMPT FOR SETUP SYNTHESIS: It records only necessary successful installation commands, keeps runtime installation out of install_config.json, and sets minimal required environment variables.
- A.3.2. PROMPT FOR SETUP SYNTHESIS: Before running the full suite, the agent verifies test discovery and then uses general, verbose, no-color commands that expose individual test names and statuses.
- A.3.2. PROMPT FOR SETUP SYNTHESIS: The interaction is non-interactive and produces install_config.json with ordered commands, test execution settings, and parser-relevant outputs.
B. Additional Plots
The additional plots provide supplementary views of the benchmark corpus. They cover temporal and language distributions, alongside issue-type and patch-size composition.
- These plots complement the main dataset summary with distributional context about when issues were reported and which ecosystems dominate.
- Figure 1 shows the temporal and language distributions of the benchmark corpus.
- Figure 2 shows the mix of issue types and the distribution of patch sizes.
C.1. Per-language Performance with Confidence Intervals
The per-language plots report pass@1, standard error, 95% confidence intervals, and pass@3 for selected languages and pooled tasks. Their captions specify task counts but not comparative outcomes.
- Go: The Go panel reports pass@1, SEM, 95% CI, and pass@3 over 60 tasks.
- Across panels, the reported metrics are pass@1, SEM, 95% CI, and pass@3.
- Python: The Python panel reports pass@1, SEM, 95% CI, and pass@3 over 60 tasks.
- Scala: The Scala panel reports pass@1, SEM, 95% CI, and pass@3 over 60 tasks.
- JavaScript: The JavaScript panel reports pass@1, SEM, 95% CI, and pass@3 over 60 tasks.
- Rust: The Rust panel reports pass@1, SEM, 95% CI, and pass@3 over 60 tasks.
- All: The pooled All panel reports pass@1, SEM, 95% CI, and pass@3 over 300 tasks.