Source-linked AI summary
SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents
Ibragim Badertdinov, Alexander Golubev, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, Andrei Andriushchenko, Maria Trofimova, Daria Litvintseva, Boris Yangel
TL;DR
SWE agents need large-scale interactive training tasks and fresh, reliable evaluations because existing data is limited and static benchmarks face contamination and comparability problems. The paper introduces a fully automated pipeline that mines and verifies tasks from GitHub, producing SWE-rebench and continuously updated evaluations. The resulting resources provide over 21,000 interactive Python tasks, while benchmark comparisons indicate that some models’ performance may be inflated on older benchmarks by contamination.
Problem
Interactive, automatically verifiable SWE training data is scarce, while static benchmarks can become contaminated and inconsistent evaluation practices hinder reliable model comparisons.
Method
The paper presents a fully automated pipeline that continuously extracts, configures, executes, and assesses software engineering tasks from real-world GitHub repositories.
Results
The pipeline produces SWE-rebench, a public dataset of more than 21,000 interactive Python-based SWE tasks and a fresh benchmark for evaluating agents.
Takeaways & Limitations
SWE-rebench supplies scalable training data and continuously updated evaluations intended to support transparent assessment of agents in realistic SWE scenarios.
Takeaways & Limitations
Scalability reduces the ability to manually curate and verify every task, and LLM-based installation and quality assessment rely on heuristics and limited validation.
Abstract
from arXiv · showhide
LLM-based agents have shown promising capabilities in a growing range of software engineering (SWE) tasks. However, advancing this field faces two critical challenges. First, high-quality training data is scarce, especially data that reflects real-world SWE scenarios, where agents must interact with development environments, execute code and adapt behavior based on the outcomes of their actions. Existing datasets are either limited to one-shot code generation or comprise small, manually curated collections of interactive tasks, lacking both scale and diversity. Second, the lack of fresh interactive SWE tasks affects evaluation of rapidly improving models, as static benchmarks quickly become outdated due to contamination issues. To address these limitations, we introduce a novel, automated, and scalable pipeline to continuously extract real-world interactive SWE tasks from diverse GitHub repositories. Using this pipeline, we construct SWE-rebench, a public dataset comprising over 21,000 interactive Python-based SWE tasks, suitable for reinforcement learning of SWE agents at scale. Additionally, we use continuous supply of fresh tasks collected using SWE-rebench methodology to build a contamination-free benchmark for agentic software engineering. We compare results of various LLMs on this benchmark to results on SWE-bench Verified and show that performance of some language models might be inflated due to contamination issues.
1 Introduction
SWE-rebench addresses shortages of interactive training data and reliable evaluation by automating continuous task collection from real-world GitHub repositories. It provides a large public dataset and fresh, decontaminated benchmarking resources for software engineering agents.
- Interactive task data is needed for agents to learn real-world software engineering behaviors through actions, execution, feedback, and reinforcement learning.Existing programming-training approaches often emphasize open-source code or synthetic instruction data rather than executable, automatically verifiable tasks.
- Static benchmarks can become contaminated as improving models encounter test instances during pre- and post-training, while inconsistent scaffolds and reporting hinder comparisons.These issues can obscure models’ true capabilities and reduce evaluation reliability.
- The paper introduces a scalable, fully automated pipeline that continuously collects software engineering tasks from real-world GitHub repositories.The system eliminates manual intervention and is designed to expand task diversity and scale.
- SWE-rebench is a public dataset of more than 21,000 interactive Python-based SWE tasks designed for training and benchmarking agents, particularly with reinforcement learning.
- The project provides a continuously updated, decontaminated, standardized leaderboard for transparent comparisons across open- and closed-source models.
- By emphasizing scale and automation, SWE-rebench aims to support open-source research and improve the reliability and performance of software engineering agents.
2 An automated pipeline for collecting software engineering tasks
The pipeline converts large-scale GitHub issue–pull-request data into executable, annotated software engineering tasks through automated environment setup, verification, and quality assessment. Its design prioritizes scalable processing while retaining mechanisms for test-based solvability checks and task filtering.
- Pipeline overview: The four-stage pipeline produced 21,336 verifiable SWE tasks from 3,468 distinct GitHub repositories.The stages are preliminary collection, automated installation configuration, execution-based installation verification, and quality assessment.
- 2.1 Preliminary task collection: Selected pull requests must modify tests and non-test code, affect 1 to 15 files, and provide tests suitable for automatically evaluating solution patches.Each task separates the non-test solution patch from the test patch.
- 2.2 Automated installation instructions configuration: Repository versions are grouped by normalized major.minor tags, with the latest base commit selected to provide a shared dependency environment for tasks in each group.
- 2.2 Automated installation instructions configuration: An LLM generates up to three structured installation recipes, then refines them using installation or test-execution error logs when needed.The approach uses Qwen2.5-72B-Instruct and is preferred over an interactive installation agent because it is more computationally efficient at scale.
- 2.3–2.4 Verification and quality assessment: Execution-based verification installs each task in a container and runs pull-request tests, while automated labels assess issue clarity, task complexity, and test-patch correctness.For task complexity, the fine-tuned model achieved 81% accuracy and weighted F1 0.82, compared with 68% accuracy for Qwen-72B-Instruct; the labels are distributed as task metadata for filtering.
3 SWE-rebench benchmark
SWE-rebench addresses contamination, comparability, and run-variance concerns through a continuously updated benchmark with standardized evaluation. Results across temporal subsets and SWE-bench Verified reveal differences in model robustness, reliability, and possible contamination effects.
- Challenges: SWE-rebench targets contamination, scaffolding variability, inconsistent reporting, and high variance in agent performance.These issues can make benchmark results difficult to compare or potentially unrepresentative of actual capabilities.
- Evaluation principles: Evaluations use a fixed minimal ReAct-style scaffold, identical prompts, default model-recommended generation settings, and a standardized 128K-token context when supported.The framework is intended to provide a common environment for comparing models’ core abilities.
- Evaluation principles: SWE-rebench continuously supplies fresh tasks and tracks issue and pull-request creation dates against model release dates to flag potentially contaminated evaluations.Potential data leakage is explicitly marked on the leaderboard.
- Evaluation principles: Each model is run five times, with SEM and pass@5 reported to capture stochastic variation in agent behavior.This is intended to provide a more statistically grounded assessment than relying on a single run.
- Evaluation setup: The benchmark examines performance on January 2025 and March–April 2025 task subsets and compares March–April results with SWE-bench Verified.Table 1 compares the two SWE-rebench windows, while Table 2 compares SWE-bench Verified with the March–April slice.
- Results: GPT-4.1 was the only model whose performance noticeably declined on the March–April subset relative to January.This temporal comparison is used to examine changes across task windows.
- Results: LLaMa-4-Maverick combined relatively high pass@5 with a modest resolution rate, while Qwen2.5-Coder-32B-Instruct frequently hallucinated environment responses or entered formatting-error loops.These observations distinguish potential solution quality from consistency and instruction-following reliability.
- Results: DeepSeek models showed the strongest open-source performance across both SWE-rebench subsets and SWE-bench Verified, with similar SWE-rebench results but divergent Verified scores.The divergence on SWE-bench Verified may suggest contamination effects on the older benchmark.
4 Discussion and limitations
SWE-rebench addresses shortages of scalable, fresh interactive SWE data while acknowledging that automation limits manual quality control and introduces broader deployment risks.
- Discussion and limitations: Scalable automation supplies large, real-world tasks and continually refreshed benchmarks, but reduces manual curation of individual task quality and clarity.The trade-off follows from prioritizing scale over manual verification.
- Discussion and limitations: LLM-generated installation instructions and automated quality assessment rely on heuristics and cannot fully reproduce nuanced human judgment.The quality-assessment model was fine-tuned on SWE-bench Verified labels and may contain errors that reduce dataset quality.
- Discussion and limitations: Greater SWE-agent autonomy may increase risks including overreliance on AI-generated code and misuse of automated agents to introduce vulnerabilities.The paper frames openness, decontaminated evaluation, and rigorous benchmarking as mitigations.
- Discussion and limitations: Fully automated quality assessment can leave tasks imperfectly described or unsolvable from the issue alone, lowering absolute success rates versus manually curated benchmarks.This limitation concerns the quality and solvability of individual automatically collected tasks.
- Discussion and limitations: The initial dataset release covers only Python, limiting immediate applicability to other language ecosystems despite a language-agnostic underlying pipeline.Supporting additional languages requires language-specific components and is planned for future work.
5 Conclusion and future work
The paper concludes that continuous automated collection can provide fresh, decontaminated SWE-agent data and benchmarks, while future work expands coverage, filtering, language support, and evaluation.
- Conclusion: The pipeline continuously collects software engineering tasks from open-source repositories and underlies public datasets plus a continuously updated benchmark.The benchmark targets realistic SWE scenarios and robust, transparent agent evaluation.
- Future work: The authors plan to increase dataset volume by extending collection from GitHub issues to code changes represented by arbitrary pull requests.This broadens the collection target beyond issue-tied tasks.
- Future work: Future filtering improvements will refine the heuristics used to improve the overall quality of extracted tasks.
- Future work: The methodology is planned for application to JavaScript, Java, and C++ projects to expand SWE-rebench’s linguistic and technological diversity.
- Future work: The benchmark will be kept current through fresh-task evaluation of existing models, broader LLM coverage, and detailed performance analyses.
- Installation pipeline: The paper’s automated installation stage identifies relevant files, extracts structured JSON recipes, and revises failed recipes using error feedback.The recipes encode environment setup, dependency installation, and test execution instructions.
- Installation pipeline: The pipeline’s installation recipes are generated from repository files and can specify Python versions, dependencies, installation commands, test commands, and prerequisite system packages.
- Execution validation: During validation, flaky tests, non-interactive installation requirements, and unpinned or obsolete dependencies excluded otherwise valid tasks.These failure modes motivated repeated testing and dependency-version freezing.
C Evaluation of automated installation recipe generation
The evaluation compares automated and interactive installation-recipe generation on 18 manually configured SWE-bench tasks, using success rates summarized in Table 3.
- Validation setup: 18 SWE-bench task instances from distinct repositories formed the curated validation set with manually crafted installation instructions as a baseline.
- Comparison: The study compares agentless LLM recipe generation with varying candidate counts against an interactive installation agent.Some automated failures reflected generic log parsers’ difficulty with customized frameworks such as Django.
- Results: Table 3 reports installation-method success rates across the 18 SWE-bench tasks.
D Permissive licenses included in data collection
Data collection targeted repositories with permissive open-source licensing, using SPDX identifiers where available and manual review for unmatched or custom license text.
- License selection: Repositories were selected under permissive open-source licenses, identified primarily through SPDX identifiers.
- License selection: Repositories with unlisted or custom “Other” licenses underwent manual review to confirm permitted use cases.
E Example of a task instance with annotations
A SWE-rebench task instance combines a software issue, code and test patches, execution metadata, quality annotations, and installation configuration. The example illustrates how these fields support executable, verifiable task environments.
- The example identifies the repository, base commit, version, creation time, and issue description for a specific task.
- The solution patch records the source-code changes associated with resolving the reported issue.In the example, the patch updates argument handling in typedflow’s ConsumerNode implementation.
- The test patch records the corresponding test change used to check the task’s behavior.The example points to a test file and adds a test-related change for the issue.
- Additional annotations include LLM-generated quality scores, test outcomes, license information, and other task metadata.The example includes issue-text, test, and difficulty scores, FAIL_TO_PASS information, and an MIT License label.
- Installation metadata specifies Python and package requirements, installation commands, pre-install commands, and related environment settings.The shown configuration includes Python 3.8, requirements.txt, pip installation, pytest packages, and system dependencies.
- A task instance includes a problem statement, solution patch, test patch, metadata, and installation configuration.The example also contains fields describing repository, commit, license, test outcomes, and environment setup.
F Comparison of models for automatic task quality assessment
The paper evaluates automated task-quality assessment using a fine-tuned Qwen model and compares it with the vanilla model while examining file count as a difficulty heuristic. Direct quality labels provide a stronger assessment of difficulty than file count alone.
- The fine-tuned Qwen 2.5-72B-Instruct model predicts Test Patch Correctness, Task Complexity, and Issue Clarity from human annotations.The comparison uses the vanilla Qwen 2.5-72B-Instruct model as its baseline on a validation set.
- 28.6% ± 0.8% resolution was observed for one-file tasks, compared with 20.6% ± 0.4% for two-file tasks and 17.5% ± 2.2% for at least three files.These results are reported for DEEPSEEK-V3-0324 across January–July 2025 leaderboard tasks.
- Direct quality assessment captures difficulty better than file count alone because multi-file changes can be simple and single-file changes can be challenging.
G Refinements to the original SWE-bench methodology
SWE-rebench refines SWE-bench task construction and evaluation by emphasizing valid executable tasks, controlled benchmark filtering, and an interaction protocol that lets agents inspect, modify, and test repositories. These refinements target reliable reinforcement learning and standardized assessment.
- The dataset aims to support reinforcement learning by ensuring tasks remain unsolved because of agent limitations rather than faulty tests or specifications.
- Patch generation is refined to avoid unrelated changes introduced by intervening merges between a pull request’s base and merge commits.Such unrelated changes can invalidate tasks by adding tests for functionality external to the pull request.
- The benchmark subset filters tasks for clean pre-patch test execution, at most three modified files, patches of no more than 500 words, and problem statements of 16–1000 words.
- Agent runs begin with an issue description and return command output or errors after each command, including the current directory and open file.The environment provides terminal access and prohibits interactive session commands that require user input.
- The system prompt directs agents to inspect repositories, reproduce reported errors, edit source files, rerun checks, consider edge cases, and submit changes explicitly.Agents are instructed not to modify existing tests or add new tests to the repository.
- Figure 2 shows overlap in tasks solved across selected models, with some tasks uniquely solved by particular models and therefore indicating complementary strengths.
J.1 GPT-4.1 Trajectory Example
The trajectory example depicts an agent locating relevant code, correcting a variable-selection error, and validating the change with repository tests. It contrasts this successful interaction with a failure mode in which Qwen2.5-Coder-32B-Instruct hallucinates environment responses and loops on formatting and syntax errors.
- J.1 GPT-4.1 Trajectory Example: The trajectory follows repository search, code inspection, editing, testing, and feedback-driven correction.The example is presented as an illustration of how an agent attempts to resolve a SWE-rebench issue.
- J.1 GPT-4.1 Trajectory Example: The agent identifies that singularity handling should index y rather than x and replaces both assignments accordingly.The corrected lines adjust y values equal to 1.0 or -1.0 by 1e-12.
- J.1 GPT-4.1 Trajectory Example: All tests in the skrf/tests directory pass after the fix, apart from expected skips and xfails.
- J.2 Qwen2.5-32B Coder hallucination example: Qwen2.5-Coder-32B-Instruct repeatedly hallucinates environment responses and loops through formatting and syntax errors on SWE-rebench.The passage attributes this behavior to poor instruction following and limited ability to correct course from feedback.
- J.1 GPT-4.1 Trajectory Example: The example’s system interaction requires a single command block per response and exposes command results through the shell environment.
L Data collection funnel and potential enhancements
The data collection funnel processes raw data through multiple stages to curate SWE-rebench, while proposed enhancements target broader coverage and higher task yield. Key boundaries include Python-only scope, issue-linked pull requests, pre-existing tests, and low dependency-installation success.
- Data collection funnel: The pipeline processes raw data through several stages, with the funnel identifying acceptance rates, bottlenecks, and opportunities to improve task yield.Table 6 summarizes approximate input and output sizes at each key stage.
- Potential enhancements: The pipeline currently focuses exclusively on Python projects, limiting the pool of repositories and tasks available for collection.The authors propose extending support to JavaScript, Java, and C++.
- Potential enhancements: Requiring issue-linked pull requests narrows task sources, so general pull requests describing problems and solutions could broaden code-change coverage.The proposed extension would capture pull requests that are not linked to formal issues.
- Potential enhancements: Tasks without new or modified tests are currently excluded from some collection paths, although LLM-generated tests could expand coverage if validated.Generated tests would require validation before use.
- Potential enhancements: Approximately 14% dependency-installation success is a bottleneck that could be improved through better installation recipes or selective use of interactive agents.Suggested approaches include fine-tuning, enhanced prompting, more capable base models, and resource-intensive handling for complex repositories.
M SWE-rebench dataset statistics
SWE-rebench contains 21,336 task instances with concise descriptions, multi-file code changes, executable test dynamics, and automated quality assessments. These statistics characterize the dataset’s task structure and support filtering by clarity, complexity, and test correctness.
- Dataset scale and measures: 21,336 task instances are covered by dataset statistics measuring description length, code modification size, test-suite changes, and LLM-assessed quality.The dataset statistics are computed across all instances.
- Task descriptions: 142 words is the average issue-description length, while 75% of tasks have descriptions of 173 words or fewer.The statistics suggest that most task statements are concise.
- Code modifications: 3.46 files is the mean number of files edited, with a 75th percentile of 4 files; solutions add 97 lines on average.The passage reports the 75th percentile for added lines as 76, though the supplied text truncates its concluding interpretation.
- Test dynamics: 15 fail-to-pass tests and 86 pass-to-pass tests are the respective averages, yielding approximately 105 tests involved per task.The 75th percentile for fail-to-pass tests is 5, indicating many tasks have a focused set of critical tests.
- Automated quality assessment: 1.13 is the mean Difficulty Score and 1.04 is the mean Issue Score; the 75th percentile for both scores is 2.Lower difficulty scores generally indicate higher quality or lower difficulty, while the scores also characterize issue clarity and task complexity.