Source-linked AI summary
Does SWE-Bench-Verified Test Agent Ability or Model Memory?
Thanosan Prathifkumar, Noble Saji Mathews, Meiyappan Nagappan
TL;DR
SWE-Bench-Verified’s GitHub-derived issues may overlap with model training data, making it important to distinguish task recall from issue-solving ability. The paper tests two Claude models on file localization with limited context across three benchmarks and finds substantially stronger performance on SWE-Bench-Verified, suggesting its tasks may have appeared in training data.
Problem
Because SWE-Bench-Verified draws on open-source GitHub repositories that may appear in LLM training corpora, its scores may reflect memorized examples rather than generalizable reasoning.
Source: §1 · p. 1Method
The authors test two Claude models on file localization using issue-only and issue-plus-file-structure inputs across SWE-Bench-Verified, BeetleBox, and SWE-rebench.
Source: §1 · p. 1, §3.2 · p. 2Results
With issue text alone, localization performance on SWE-Bench-Verified is almost 6 times that of BeetleBox and approximately 3 times better than SWE-rebench when evaluating all ground-truth files.
Source: §4 · p. 3Takeaways & Limitations
The authors conclude that SWE-Bench-Verified scores may not reflect true agent ability on real software issues and that continued reliance risks overstating progress.
Source: §5 · p. 3The authors frame the experiment as sufficient to test leakage into an earlier Claude model version, given Claude 4’s newer knowledge cutoff.
Source: §3.1 · p. 2
Abstract
from arXiv · showhide
SWE-Bench-Verified, a dataset comprising 500 issues, serves as a de facto benchmark for evaluating various large language models (LLMs) on their ability to resolve GitHub issues. But this benchmark may overlap with model training data. If that is true, scores may reflect training recall, not issue-solving skill. To study this, we test two Claude models that frequently appear in top-performing agents submitted to the benchmark. We ask them to find relevant files using only issue text, and then issue text plus file paths. We then run the same setup on BeetleBox and SWE-rebench. Despite both benchmarks involving popular open-source Python projects, models performed 3 times better on SWE-Bench-Verified. They were also 6 times better at finding edited files, without any additional context about the projects themselves. This gap suggests the models may have seen many SWE-Bench-Verified tasks during training. As a result, scores on this benchmark may not reflect an agent's ability to handle real software issues, yet it continues to be used in ways that can misrepresent progress and lead to choices that favour agents that use certain models over strong agent design. Our setup tests the localization step with minimal context to the extent that the task should be logically impossible to solve. Our results show the risk of relying on older popular benchmarks and support the shift toward newer datasets built with contamination in mind.
1 Introduction
SWE-Bench-Verified is a widely used benchmark for comparing LLM issue-fixing performance, but its GitHub-derived tasks may overlap with model training data. The paper examines possible leakage by comparing models’ bug-localization performance across SWE-Bench-Verified and other datasets with limited input.
1 Introduction
- SWE-Bench-Verified has become the de facto benchmark for comparing LLM-based systems on issue resolution.Its leaderboard ranks systems by the percentage of issues they resolve.Source: §1 · p. 1
- Because SWE-Bench-Verified uses open-source GitHub repositories, LLMs trained on internet-scale corpora may have seen its tasks during training.The authors warn that leaderboard scores could reflect memorized examples rather than generalizable reasoning.Source: §1 · p. 1
- The study probes possible benchmark leakage by testing Claude Sonnet on bug localization across SWE-Bench-Verified, BeetleBox, and SWE-rebench.Models receive issue text alone or issue text plus file paths, with insufficient input to rationally solve localization.Source: §1 · p. 1
2 Background and Related Work
Prior work reports evidence of SWE-Bench leakage and raises separate concerns about benchmark test quality and scope. This study differs by testing localization with minimal context to assess whether benchmark tasks themselves may have leaked.
2.1 Data Leakage in SWE-Bench
- Aleithan et al. [1] report that 94% of SWE-Bench instances predate common LLM training cutoffs and that resolution rates drop significantly on their post-cutoff SWE-Bench+ benchmark.They interpret the drop as evidence of data leakage in popular LLMs.Source: §2.1 · p. 1
- Zhou et al. [10] report duplicate-data leakage rates of 10.6% for SWE-Bench Verified and 8.7% for SWE-Bench in StarCoder.They link overlap to GitHub pretraining data, platform-specific sources, and shared repositories.Source: §2.1 · pp. 1–2
- Liang et al. [5] find issue-only localization accuracy of 60–76% on SWE-Bench-Verified but below 53% on external repositories, with up to a 47-point drop on other-language repositories.Their reported o3 accuracy on SWE-Bench without contextual information is 76%.Source: §2.1 · p. 2
- Unlike prior fix-generation comparisons using the issue and repository, this study tests localization with minimal context to assess possible leakage of benchmark tasks themselves.The authors distinguish task leakage from overlap attributable to repositories drawn from popular open-source projects.Source: §2.1 · p. 2
2.2 Other Faults in SWE-Bench
- Other critiques cite weak SWE-Bench tests and its Python-only scope, while this paper focuses specifically on possible data leakage rather than benchmark quality.Prior work reports a 4.5% performance drop when running all tests and says 28.6% of samples pass despite being obviously incorrect.Source: §2.2 · p. 2
3 Methodology
The study tests Claude Sonnet 3.5 and 3.7 on issue localization across SWE-Bench-Verified, BeetleBox, and SWE-rebench. It compares file predictions with ground truth under two limited-input settings and two measures of localization success.
3.1 Models
- The study selects Claude Sonnet 3.5 and 3.7, models used by several top-ranking SWE-Bench-Verified agents, to examine leakage into an earlier model version.The authors reason that Claude 4’s newer March 2025 knowledge cutoff makes the earlier models sufficient for this test.Source: §3.1 · p. 2
3.2 Data
- The dataset comprises all 500 SWE-Bench-Verified issues, 500 manually screened BeetleBox issues from five randomly selected Python repositories, and January and September 2025 SWE-rebench splits.The SWE-rebench splits contain 109 and 50 issues, respectively.Source: §3.2 · p. 2
3.3 Input Settings
- Models predict likely fix files using either the issue plus repository file paths or the issue alone, without file contents.Both settings provide insufficient information to rationally solve localization, let alone fix bugs.Source: §3.3 · p. 2
3.4 Evaluation
- Localization is evaluated by the percentage of issues for which model output includes all ground-truth files or at least one ground-truth file.Predicted file names are compared with benchmark ground truth.Source: §3.4 · p. 2
4 Results
With only issue text, Claude models localized all ground-truth files far more often on SWE-Bench-Verified than on BeetleBox or SWE-rebench. The advantage persisted with ticket and file-structure input, and was larger for complete file localization than for finding at least one file.
4 Results
- Approximately 4 times better than BeetleBox and 2 times better than SWE-rebench at predicting all ground-truth files with ticket and file structure.Both models showed this pattern.Source: §4 · p. 2, Table 1 · p. 3
- Almost 6 times better than BeetleBox and approximately 3 times better than SWE-rebench on all-file localization with ticket-only input.The issue-only gap was larger than under ticket-plus-file-structure input.Source: Table 3 · p. 3, §4 · p. 3
- SWE-Bench-Verified results were consistently higher despite all compared datasets involving popular Python projects, with a larger gap for all files than for at least one.The authors suggest chance, ticket mentions, or filename semantics may help explain some single-file predictions, but the all-files advantage remains difficult to explain.Source: §4 · p. 3
- The contrast with weak results on similar datasets suggests models may have seen SWE-Bench-Verified tasks before, reflecting memorization rather than problem-solving.The authors describe this as a possibility, not a demonstrated cause.Source: §4 · p. 3
5 Conclusion
The authors find that two Claude models localize issues much better on SWE-Bench-Verified than on BeetleBox and SWE-rebench, even with issue text alone. They argue this gap suggests training overlap and cautions that benchmark scores may not reflect real-world agent ability.
5 Conclusion
- Both Claude models localize issues very well on SWE-Bench-Verified using only issue text, while performance drops sharply on BeetleBox and SWE-rebench.The comparison uses other benchmarks involving popular open-source Python projects.Source: §5 · p. 3
- The performance gap suggests SWE-Bench-Verified tasks may be present in model training, rather than reflecting general skill transfer across similar projects.The authors note that stronger transfer would be expected if the benchmark measured real learning.Source: §5 · p. 3
- SWE-Bench-Verified scores may not reflect agent ability on real software issues, and continued reliance risks overstating progress and favouring systems benefiting from training overlap.Source: §5 · p. 3
- The authors support shifting toward refreshed, contamination-conscious benchmarks such as SWE-rebench and urge care when interpreting results from fixed, widely used datasets.Source: §5 · p. 3