PaperlayerView exact revision on arXiv

Source-linked AI summary

What's in a Benchmark? The Case of SWE-Bench in Automated Program Repair

Matias Martinez, Xavier Franch

arXiv:2602.04449v1cs.SE

TL;DR

SWE-Bench’s rapid growth motivates studying leaderboard dynamics and the types of proposals submitted. The paper systematically analyzes entries from Lite and Verified, finding that industry supplies most submissions and proprietary LLMs dominate among state-of-the-art entries.

  • Problem

    SWE-Bench’s rapid growth and impact motivate examining leaderboard dynamics and the types of proposals submitted.

    Source: §1 · p. 1

  • Method

    The study examines leaderboard entries and gathers metadata and external information about submitters, artifacts, LLMs, and solution openness.

    Source: §1 · p. 1

  • Results

    Most submissions come from industry, and most state-of-the-art submissions rely on proprietary LLMs, particularly Claude 4 Sonnet.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The analysis maps submitter origins, product characteristics, and LLM use to deepen understanding of the current SWE-Bench field.

    Source: §7 · p. 10

    The study covers only Lite and Verified, and the authors do not claim its findings apply to other benchmarks.

    Source: §5 · p. 10

Abstract

from arXiv · show

The rapid progress in Automated Program Repair (APR) has been fueled by advances in AI, particularly large language models (LLMs) and agent-based systems. SWE-Bench is a benchmark designed to evaluate repair systems using real issues mined from popular open-source Python repositories. Its public leaderboards-SWE-Bench Lite and Verified-have become central platforms for tracking progress and comparing solutions. In this paper, we present the first comprehensive study of these two leaderboards, examining who is submitting solutions, the products behind the submissions, the LLMs employed, and the openness of the approaches. We analyze 79 entries submitted to Lite leaderboard and 133 to Verified. Our results show that most entries on both leaderboards originate from industry, particularly small companies and large publicly traded companies. These submissions often achieve top results, although academic contributions-typically open source-also remain competitive. We also find a clear dominance of proprietary LLMs, especially Claude family, with state-of-the-art results on both leaderboards currently achieved by Claude 4 Sonnet. These findings offer insights into the SWE-Bench ecosystem that can guide greater transparency and diversity in future benchmark-driven research.

1 Introduction

SWE-Bench’s rapid growth motivates a systematic study of who submits solutions and how leaderboard submissions use products, LLMs, and openness. Across Lite and Verified, industry supplies most submissions, while most state-of-the-art entries rely on proprietary LLMs, especially Claude 4 Sonnet.

1 Introduction

  • SWE-Bench provides 2,294 issue-repair tasks from 12 Python repositories, a framework for computing repair metrics, and public leaderboards for publishing results.Unlike classical APR benchmarks described here, its instances do not include test cases or scripts that expose and validate fixes.Source: §1 · p. 1
  • The study examines submitter origins, associated software artifacts, LLM use, and solution openness by analyzing entries and gathering information from publications and grey literature.For each entry, the researchers collect leaderboard metadata and search sources such as blog posts and LinkedIn posts.Source: §1 · p. 1
  • Most submissions come from industry, and most state-of-the-art entries rely on proprietary LLMs, particularly Claude 4 Sonnet.The study analyzes 79 Lite entries and 133 Verified entries.Source: §1 · p. 2
  • The study’s findings are intended to inform understanding of SWE-Bench’s state of the art and practice and implications for researchers, practitioners, and leaderboard builders.The authors also discuss patch correctness and benchmark evolution.Source: §1 · p. 2

2 Methodology

The study defines SWE-Bench terms, collects and annotates entries from Lite and Verified, and codes submitters, products, availability, source-code openness, and LLM use. It excludes Full and Multimodal and supplements leaderboard metadata with external sources.

2.1 Terminology

  • SWE-Bench instances are issue-repair tasks, entries are leaderboard records, submitters are responsible individuals or teams, and approaches are evaluated configurations or methods.A leaderboard ranks submissions on a defined subset of instances.Source: §2.1 · p. 2

2.2 Data Collection

  • The study collects Lite and Verified entries, excluding Full because its solutions also appear in a considered leaderboard and Multimodal because the study restricts scope to language-based agents.Researchers inspect entry pages, follow external links, search the web, review repository metadata, and examine submitter LinkedIn pages.Source: §2.2 · p. 2

2.3 Data Annotation

  • The researchers use combined deductive and inductive content analysis, starting with predefined themes and refining codes as they process leaderboard content.Data availability sometimes constrained analyses; unclear country information led them to discard planned country-level analysis.Source: §2.3 · p. 2, §2.2 · p. 2

2.3.1 Submitter Category.

  • Submitters are coded by affiliation and organization type, including academia, industry size, academia-industry collaborations, open-source community, single developers, and unknown cases.Industry company-size categories use the Microsoft-LinkedIn classification system.Source: §2.3.1 · p. 3

2.3.2 Product and Accessibility.

  • Products are coded by purpose and delivery form, while availability distinguishes public products, request-based access, business agreements, non-commercial solutions, and unavailable products.Product availability concerns the product associated with an entry, not necessarily the code used in the benchmark experiment.Source: §2.3.2 · p. 3

2.3.3 Open-source solution.

  • Solution openness is classified by whether the solution’s source code is open or closed, independently of the underlying LLM’s openness.Source: §2.3.3 · p. 3

2.3.4 LLMs Employed in Solutions.

  • The study identifies all LLMs used by a solution, including combinations, drawing on submission names, metadata, and other sources when needed.It counts only the model corresponding to the leaderboard submission, not additional results reported in related articles.Source: §2.3.4 · p. 3

3 Results

Across SWE-Bench Lite and Verified, submissions and precision evolved alongside a shift toward industry participation. Results also show variation in product availability, solution openness, and LLM use.

3.1 RQ1: Evolution of submissions and precision

  • 76.8% maximum and 46.6% median % Resolved on Verified, versus 60.3% and 32% on Lite; recent submissions increasingly concentrate on Verified.Lite activity declined during 2025 while Verified submissions continued to grow steadily.Source: Figure 1 · p. 4, §3.1 · p. 4

3.2 RQ2: Types of Submitters

  • Industry entries grew from 32 (41%) in Lite to 74 (56%) in Verified, with small firms and large publicly traded companies contributing substantially.In Verified, companies of multiple sizes exceeded 73%, while academic and open-source approaches also exceeded 66%.Source: §3.2 · p. 5

3.3 RQ 3: Products and Availability Modes

  • Entries span public commercial products, non-commercial solutions, B2B tools, request-only products, and unavailable products; public and request-only solutions stand out in precision, particularly in Verified.Publicly Available Products have the highest reported median % Resolved across both leaderboards.Source: §3.3 · pp. 6–7

3.4 RQ 4: Open- and Closed- Source Solutions

  • Open-source entries account for 67% of Lite and 61.6% of Verified; closed-source entries generally have higher median scores, but open-source solutions remain competitive.When academia is involved, most resulting solutions are open source; industry–academia collaborations raise this share to at least 78%.Source: §3.4 · p. 7

4 Discussion and Implications

The discussion flags limits on interpreting SWE-Bench scores and proposes ways to improve validation, transparency, and comparability. It also notes cost and reporting factors that shape participation and claims of progress.

Patch Overfitting on SWE-Bench.

  • Test-passing patches can remain incorrect, so SWE-Bench resolution rates may overstate correctness and create unrealistic expectations.The authors call for post-submission correctness assessment and patch-validation tools in the evaluation pipeline.Source: Patch Overfitting on SWE-Bench · pp. 8–9
  • Researchers are encouraged to address overfitting and share usable open-source artifacts, while submitters are urged to provide metadata beyond minimum requirements.The authors also recommend balancing transparency against participation by setting lightweight but informative metadata requirements.Source: Patch Overfitting on SWE-Bench · pp. 8–9

Data Contamination.

  • Submitters are encouraged to document data-leakage mitigation, including cutoff dates, and researchers to consider benchmarks that control for training-cutoff overlap.Researchers are also encouraged to develop platforms designed to avoid contamination or update benchmarks with recent data.Source: Data Contamination · p. 9

Limited Real-World Representation.

Emerging Variants of SWE-Bench.

  • SWE-Bench does not track emerging subsets and derivatives; the authors encourage systematic registration and dialogue with leaderboard maintainers to improve their visibility and comparability.Source: Emerging Variants of SWE-Bench · p. 9

Benchmark versioning.

  • SWE-Bench lacks explicit versioning, and builders are encouraged to document revisions and relate results across versions to preserve traceability and comparability.Source: Benchmark versioning · p. 9

Saturation and the Cherry-Picking of Submissions.

  • Submissions reached up to 76.8% precision in September 2025, but selective reporting of the best model result may hide performance variability.All systems exceeding 70% precision relied on Claude 4 models, alone or combined; the authors say apparent saturation may also reflect selective reporting.Source: Saturation and the Cherry-Picking of… · pp. 9–10

Cost Barriers: Cloud-Based and Fine-Tuned LLMs.

  • Cloud-based LLM APIs and fine-tuning incur monetary or computational costs, which can constrain experimentation, particularly in academic settings.The authors encourage reporting open-source-model results and considering financial, computational, and energy costs in evaluation.Source: Cost Barriers: Cloud-Based and… · p. 10

5 Threats to Validity

The study's findings are bounded by its focus on two SWE-Bench leaderboards and by limitations in identifying systems and measuring submission characteristics.

External Validity

  • The findings cover two SWE-Bench leaderboards only; the authors do not claim they apply to other benchmarks.They note that other benchmarks may be equally representative of the broader issue-fixing landscape.Source: §5 · p. 10

Internal Validity

  • Missed artifacts may leave entries uncharacterized, and reported information may not accurately reflect the systems actually used.The authors searched beyond official leaderboard metadata, including Google, LinkedIn, Google Scholar, and arXiv.Source: §5 · p. 10

Construct Validity

  • The analysis omits LLM parameter counts and minor versions, focusing instead on model families and major releases.The authors excluded these finer-grained distinctions to simplify analysis and identify broader trends.Source: §5 · p. 10
  • Submission costs were excluded, although they matter for usability and practicality; falling token prices could make cross-submission comparisons misleading.The authors excluded cost for practical reasons.Source: §5 · p. 10

6 Related work

Prior SWE-Bench studies examine effectiveness, patch correctness, agent trajectories, and whether benchmark performance reflects genuine coding ability or memorization.

Effectiveness and patch correctness

  • Issue quality affects repair effectiveness, while prior patch analyses identified solution leakage and plausible patches that were incorrect.Meng et al. studied seven Lite systems; Aleithan et al. found complete solutions in 32.67% of issue descriptions and incorrect patches passing tests in 12.75%.Source: §6 · p. 10
  • On SWE-Bench Verified, an average of 7.8% of plausible patches were actually incorrect, exposing a flaw in patch validation.Wang et al. examined plausible patches from three systems.Source: §6 · p. 10

Agent trajectories

  • Successful agent trajectories balance exploration, explanation, and validation, whereas unsuccessful ones show repetitive, non-adaptive action cycles.Other trajectory studies analyzed SWE-Bench Verified submissions and complete trajectories and testing logs from eight agents.Source: §6 · p. 10

Benchmark generalization

  • Models including OpenAI o3-mini can repair most benchmark issues, but performance drops on issues outside SWE-Bench.Liang et al. investigate whether LLM-based SWE-Bench performance reflects coding capability or memorization.Source: §6 · p. 10

7 Conclusions

The paper studies submissions to the SWE-Bench Lite and Verified leaderboards, characterizing submitters, products, and LLM use. It finds that most submissions come from industry and that most high-performing submissions rely on proprietary LLMs, particularly Claude 4.

Study scope

  • The study analyzes 212 entries from 77 submitters across the SWE-Bench Lite and Verified leaderboards.Source: §7 · p. 10

Submitter profile

  • Most submissions came from industry, including large tech companies and small startups, alongside independent developers.Source: §7 · p. 10

LLM use

  • Most high-performing submissions relied on proprietary LLMs, particularly Claude 4.Source: §7 · p. 10
Loading 2602.04449v1…