Source-linked AI summary
SWE-bench Goes Live!
Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, Elsie Nallipogu, Qingwei Lin, Yingnong Dang, Saravan Rajmohan, Dongmei Zhang
TL;DR
Existing SWE-bench-style benchmarks are static, vulnerable to contamination, and costly to curate. SWE-bench-Live addresses these limitations with a live benchmark and automated environment-building pipeline, while evaluations show lower performance than on static benchmarks and highlight the need for dynamic evaluation settings.
Problem
Existing SWE-bench derivatives are static, vulnerable to data contamination, and dependent on substantial manual effort for task and environment construction.
Method
SWE-bench-Live uses a live benchmark and REPOLAUNCH, an automated pipeline that streamlines issue mining, environment packaging, and reproducible execution.
Results
Leading agent frameworks show suboptimal performance on SWE-bench-Live, with the same agent-LLM pair performing worse there than on SWE-bench Verified.
Takeaways & Limitations
The findings suggest that models may overfit static benchmarks, underscoring the importance of dynamic and diverse evaluation settings.
Takeaways & Limitations
SWE-bench-Live primarily focuses on Python, limiting its language coverage.
Abstract
from arXiv · showhide
The issue-resolving task, where a model generates patches to fix real-world bugs, has emerged as a critical benchmark for evaluating the capabilities of large language models (LLMs). While SWE-bench and its variants have become standard in this domain, they suffer from key limitations: they have not been updated since their initial releases, cover a narrow set of repositories, and depend heavily on manual effort for instance construction and environment setup. These factors hinder scalability and introduce risks of overfitting and data contamination. In this work, we present SWE-bench-Live, a live-updatable benchmark designed to overcome these challenges. Our initial release consists of 1,319 tasks derived from real GitHub issues created since 2024, spanning 93 repositories. Each task is accompanied by a dedicated Docker image to ensure reproducible execution. Central to our benchmark is \method, an automated curation pipeline that streamlines the entire process from instance creation to environment setup, removing manual bottlenecks and enabling scalability and continuous updates. We evaluate a range of state-of-the-art agent frameworks and LLMs on SWE-bench-Live, revealing a substantial performance gap compared to static benchmarks like SWE-bench, even under controlled evaluation conditions. To better understand this discrepancy, we perform detailed analyses across repository origin, issue recency, and task difficulty. By providing a fresh, diverse, and executable benchmark grounded in live repository activity, SWE-bench-Live facilitates rigorous, contamination-resistant evaluation of LLMs and agents in dynamic, real-world software development settings.
1 Introduction
Existing issue-resolution benchmarks are static, narrow, and labor-intensive to construct, motivating SWE-bench-Live and its automated REPOLAUNCH pipeline. The benchmark targets fresh repository-level evaluation and reveals lower performance than static benchmarks.
- Benchmark limitations: SWE-bench and its variants remain standard for repository-level issue resolution but have not been updated since their initial releases.These tasks require models to understand complex codebases, interact with execution environments, and generate patches for real-world issues.
- Benchmark limitations: Static benchmark content risks data contamination, making apparent progress harder to distinguish from memorization.Models may have encountered public benchmark instances or solutions during training.
- Benchmark limitations: Limited repository coverage reduces diversity across codebases, domains, and programming practices, weakening evaluation generalizability.Existing benchmarks draw from a small set of repositories.
- Benchmark limitations: Manual instance construction requires issue-pair identification, test discovery, environment configuration, command composition, and workflow validation, creating scalability bottlenecks.Multi-SWE-bench reportedly required about one year and 68 expert annotators to create 1,632 instances.
- SWE-bench-Live: SWE-bench-Live provides a live, scalable benchmark for real-world issue resolution, with REPOLAUNCH automating issue mining, environment packaging, and validation.The initial release contains 1,319 tasks from GitHub issues created since 2024, spanning 93 repositories.
- Evaluation: Leading agent frameworks show suboptimal performance on SWE-bench-Live, and the strongest agent-model pairing performs significantly worse than on SWE-bench Verified.The evaluation covers OpenHands, SWE-Agent, and Agentless with GPT-4.1, GPT-4o, Claude 3.7 Sonnet, and DeepSeek V3.
2 Related Work
Prior coding benchmarks and agents increasingly address software engineering tasks, but their evaluations remain dominated by static datasets. This leaves repository-level systems exposed to overfitting and possible contamination concerns.
- Coding Benchmarks: Early benchmarks such as HumanEval and MBPP use single-file synthetic tasks, whereas SWE-bench introduced validated patch generation for real GitHub issues.Later variants extend the setting to multimodal interfaces and multiple programming languages.
- Coding Benchmarks: SWE-bench and its extensions are static, cover at most a few dozen repositories, and require labor-intensive environment construction.These characteristics limit freshness, repository diversity, and scalability.
- Coding Benchmarks: LiveCodeBench reduces contamination risk by streaming newly released algorithmic problems, but it does not address repository-level tasks requiring multi-file reasoning and execution.The comparison distinguishes algorithmic programming from complex software-engineering workflows.
- Coding Agents: Coding agents such as SWE-Agent, OpenHands, and Agentless are evaluated almost exclusively on static offline datasets.Reported improvements may partially reflect memorized solutions or configuration quirks rather than general capability gains.
3 SWE-bench-Live
SWE-bench-Live evaluates issue resolution by pairing real repository issues with validated execution environments, using an automated pipeline to construct a fresh, broad, continuously updateable benchmark.
- The issue-resolving task requires generating a patch for a repository issue that resolves the issue and passes test cases.
- SWE-bench-Live contains 1,319 instances from 93 open-source Python repositories, restricted to issues created between January 2024 and April 2025.
- Its three-stage construction pipeline extracts issue–PR pairs, automatically creates Docker-based environments with REPOLAUNCH, and validates instances through repeated test execution.
- The benchmark is designed for minimal manual maintenance, monthly updates, reproducible execution, and broader repository coverage than prior issue-resolution benchmarks.
- REPOLAUNCH recreates repository environments by identifying setup files, selecting a base image, and interactively installing dependencies and running tests through an agentic workflow.
- A time-restricted package mechanism prevents future dependency versions from causing incompatibilities during historical environment reconstruction.
4 Experiments
Experiments evaluate representative coding agents and models on SWE-bench-Live, comparing performance across benchmark origins, issue recency, and task difficulty. Results show substantially lower performance than on SWE-bench and sharp degradation as patch or repository complexity increases.
- 4.1 Setups: The evaluation uses OpenHands, SWE-Agent, and Agentless with controlled limits, reporting resolved, patch-apply, and localization success rates.The primary metric is Resolved Rate, supplemented by Patch Apply Rate and file-level Localization Success Rate.
- 4.2 Performance on SWE-bench-Live: The Lite and full evaluations report agent–model performance, with top combinations selected on Lite and then evaluated on the full benchmark.Table 3 covers the Lite subset, while Table 4 reports the top three combinations on the full version.
- 4.2 Performance on SWE-bench-Live: 43.20% resolved rate on SWE-bench Verified exceeded the 19.25% maximum on SWE-bench-Live under the authors’ matched rerun, despite identical issue-resolution settings.The authors report that leading agents often exceed 60% on SWE-bench Verified, but their matched OpenHands–Claude 3.7 Sonnet rerun reached 43.20%.
- 4.2 Performance on SWE-bench-Live: 22.96% resolved rate on instances from SWE-bench repositories exceeded 18.89% on instances from non-SWE-bench repositories.The comparison covers 216 instances from eight previously used repositories and 1,103 instances from repositories not previously used in SWE-bench.
- 4.3 Performance vs. Recency: Resolved rates remain relatively stable across issue-creation quarters from 2024Q1 to 2025Q1, with no clear correlation between recency and success.The analysis uses OpenHands with Claude 3.7 Sonnet on the full benchmark; a slight dip in 2024Q4 is followed by recovery in 2025Q1.
- 4.4 Performance vs. Difficulty: 48% of single-file patches changing fewer than five lines are solved, but success falls below 10% for patches touching at least three files or exceeding 100 changed lines.Patches touching seven or more files are never solved, indicating difficulty with coordinated multi-file edits and large intra-file changes.
- 4.4 Performance vs. Difficulty: Repositories under 100 files and 20,000 LOC often exceed 20% success, whereas projects over 500 files rarely exceed 5%.Repository size is informative but imperfect because some smaller projects remain difficult due to build systems or domain logic.
5 Conclusion
SWE-bench-Live is a scalable, continuously updating benchmark designed to address staleness, limited repository diversity, and manual curation in prior benchmarks. Its evaluations show greater difficulty than static datasets, especially for multi-file patches and large codebases.
- Conclusion: SWE-bench-Live provides a scalable, contamination-resistant, fully automated framework for fresh repository-level issue resolution.REPOLAUNCH builds reproducible Docker environments and validates issue–pull request pairs through test execution without manual intervention.
- Conclusion: Evaluations across multiple agent and model combinations show that SWE-bench-Live is significantly more challenging than static datasets.
- Conclusion: Low resolution rates on multi-file patches and large codebases expose limitations in current systems.
A Full Repositories List
The repository list organizes benchmark instances by repository type and records each repository’s license, instance count, file count, and lines of code.
- A Full Repositories List: The repository table reports repository type, name, license, number of instances, files, and lines of code.
- Cloud: Cloud repositories include aws-cloudformation/cfn-lint with 102 instances, 2,422 files, and 160.2k lines of code.
- Desktop: Desktop repositories include qtile/qtile with 6 instances, 405 files, and 81.6k lines of code.
B Task Formulation
Issue resolution requires a model to generate a patch for a reported issue using a snapshot of the target codebase, then tests determine whether the patch is correct.
- Task Formulation: Figure 6 depicts issue resolution as patch generation followed by correctness evaluation through test execution.
- Task Formulation: The issue-resolving task simulates a developer submitting a pull request to address an issue.
- Generating Patch: The model receives the issue description and codebase snapshot, then generates a .diff patch intended to fix the issue.
- Evaluating Patch: Patch correctness is assessed by applying the patch, executing the repository test suite, and parsing individual test outcomes.
C Performance vs Repository difficulty
Repository difficulty is examined by relating each repository’s resolved rate to its size, measured by Python-file count and lines of code.
- Performance vs Repository difficulty: Figure 7 plots each repository’s resolved rate against repository size.Python files are shown on the x-axis and lines of code on the y-axis.
- Performance vs Repository difficulty: The figure uses Python-file count as the x-axis and lines of code as the y-axis to compare repository size with resolution performance.
D Dataset Fields
Table 6 documents the fields included in SWE-bench-Live and explains how those fields are obtained during the curation process.
- Table 6 describes the fields included in the SWE-bench-Live dataset.
- The table explains how SWE-bench-Live dataset fields are obtained during curation.
- The field descriptions connect dataset contents with their curation process.
E Experimental Setup Details
The experimental setup section provides reproducibility details, including implementation settings, subset sampling, dataset fields, and computational resources.
- The paper presents additional experimental setup details to facilitate reproducibility.
- OpenHands uses up to 60 iterations per instance, while SWE-agent allows 100 LLM calls and Agentless uses one localization and one repair sample.
- The lite subset is sampled with random seed 42.
- Table 6 describes the required fields included in a typical issue-solving task instance.
- Experiments run on a 128-core Intel Xeon Gold 6338 CPU server with 2TB of RAM and parallel Docker containers.
F Limitations
The paper acknowledges experimental randomness from LLM calls and describes a benchmark setup centered on Docker-based test execution and automated environment preparation.
- LLM calls may introduce randomness, and the experiments are not repeated because of the extensive call requirements and limited budget.
- The setup reduces randomness by keeping the execution environment fixed and setting temperature and top_p to zero.
- The benchmark construction workflow uses Docker environments, repository files, setup commands, and test execution to verify project environments.
- Environment verification reports issues rather than modifying the environment, and tolerates some test failures when most tests pass.
- The workflow requires test commands to produce detailed pass/fail status for each test item for output parsing.
- Base image selection considers programming-language versions, system dependencies, and official images when possible.
2. Limitations
The paper reports limitations concerning reproducibility, scope, statistical significance, and experimental disclosure, while documenting open access and computational details.
- The paper identifies limitations and scope considerations as issues authors should discuss, including assumptions, robustness, and factors influencing performance.
- The work is presented as a dataset and experimental analysis rather than a theoretical paper, so theory assumptions and proofs are not applicable.
- The paper states that its code, data, and Docker environment are released to support reproduction of the experimental results.
- The paper provides experimental details in Section 4 and Appendix E, including compute-resource information.
- The paper acknowledges that LLM-driven experiments may lack statistical significance reporting because the budget limits repeated experiments.