PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-Bench+: Enhanced Coding Benchmark for LLMs

Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, Song Wang

arXiv:2410.06992v2cs.SE

TL;DR

The paper addresses the lack of systematic evaluation of SWE-bench’s quality by manually examining successful SWE-Agent+GPT-4 patches and introducing SWE-bench+. It finds answer leakage and weak tests among successful patches, and the correct resolution rate falls after suspicious fixes are filtered.

  • Problem

    A systematic evaluation of SWE-bench’s quality was missing despite its use to evaluate LLM coding capabilities.

    Source: §1 · p. 2

  • Method

    The authors manually compared SWE-Agent+GPT-4 generated patches with gold patches and created SWE-bench+ using post-cutoff issues without solutions in issue reports or comments.

    Source: §1 · pp. 2–3, §2 · p. 4

  • Results

    32.67% of resolved instances had answer leakage and 31.08% had weak-test issues; filtering suspicious fixes reduced the correct resolution rate from 12.47% to 3.97%.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    SWE-bench+ removes solution leakage and has the lowest proportion of weak-test issues among the SWE-bench variants, while weak tests remain a challenge.

    Source: §1 · p. 3, §7 · p. 11

    The authors call for further investigation into weak test cases and ways to improve test quality.

    Source: §7 · p. 11

Abstract

from arXiv · show

Large Language Models (LLMs) in Software Engineering (SE) can offer assistance for coding. To facilitate a rigorous evaluation of LLMs in practical coding contexts, Carlos et al. introduced the SWE-bench dataset, which comprises 2,294 real-world GitHub issues and their corresponding pull requests, collected from 12 widely used Python repositories. Several impressive LLM-based toolkits recently are developed and evaluated on this dataset. However, a systematic evaluation of the quality of SWE-bench remains missing. In this paper, we addressed this gap by presenting an empirical analysis of the SWE-bench dataset. We conducted a manual screening of instances where SWEAgent + GPT-4 successfully resolved issues by comparing the model-generated patches with the actual pull requests. SWE-Agent+GPT-4 was at the top of SWE-bench leaderboard during the time of our study. Our analysis reveals some critical issues with the SWE-bench dataset: 1) 32.67% of the successful patches involve cheating as the solutions were directly provided in the issue report or the comments. We refer to as solution leakage problem. 2) 31.08% of the passed patches are suspicious patches due to weak test cases, i.e., the tests were not adequate to verify the correctness of a patch. When we filtered out these problematic issues, the resolution rate of SWE-Agent+GPT-4 dropped from 12.47% to 3.97%. We also observed that the same data quality issues also exist in the two variants of SWE-bench, i.e., SWE-bench Lite and SWE-Bench Verified. In addition, over 94% of the issues were created before LLM's knowledge cutoff dates, posing potential data leakage issues.

1 Introduction

The paper asks whether LLMs truly resolve SWE-bench issues and examines both benchmark quality and generated patches. It identifies answer leakage and weak tests among successful patches, then introduces SWE-bench+ as a refined dataset.

1 Introduction

  • SWE-bench contains 2,294 GitHub issues, while Lite and Verified contain 300 bug-fixing issues and 500 verified issues, respectively.Each task asks an LLM to modify a codebase using issue information; the associated pull request includes developer changes and tests.Source: §1 · p. 2
  • The study compared SWE-Agent+GPT-4’s generated patches with gold patches and reviewed issue reports, tests, and discussions to identify fix patterns.SWE-Agent was at the top of the SWE-bench leaderboard during the study; other leading approaches cited were closed-source or lacked reproducibility verification.Source: §1 · p. 2
  • 32.67% of resolved instances had answers in issue reports or comments, while 31.08% passed despite incorrect, incomplete, or mismatched fixes.The authors attribute the latter pattern to tests too weak to catch such errors.Source: §1 · p. 2
  • 94% of SWE-bench instances and their pull requests predated the LLM training cutoffs, raising potential training-data leakage concerns.The authors note that issue reports may therefore have been exposed during training.Source: §1 · p. 2
  • Filtering suspicious fixes lowers SWE-Agent+GPT-4’s correct resolution rate from 12.47% to 3.97%.The suspicious fixes are associated with answer leakage and weak tests.Source: §1 · pp. 2–3
  • SWE-bench+ uses issues created after model training cutoffs and excludes issues with solutions in their descriptions or comments.The study reports no solution leakage and the lowest proportion of weak-test issues among the SWE-bench variants; SWE-Agent+GPT-4 resolved 0.55%.Source: §1 · p. 3

2 Robustness Analysis of SWE-Bench

A patch-level audit finds that passing tests can mask solution leakage and weak-test fixes; excluding suspicious patches lowers SWE-Agent+GPT-4’s Full resolution rate from 12.47% to 5.49%.

2 Robustness Analysis of SWE-Bench

  • The authors compared generated and gold patches for 251 resolved-and-tested instances, with three authors independently validating patches and resolving disagreements through discussion.They identified six patterns: four suspicious and two correct-fix patterns.Source: §2 · p. 4, §2.1 · p. 4

2.1.1 Patterns extracted from the suspicious fixes

  • 32.67% of successfully resolved issues had solutions directly stated in issue descriptions or comments supplied to the model as input.This was the most common suspicious pattern.Source: §2.1.1 · p. 4
  • 12.75% of passed instances had incorrect model patches that tests accepted, including a patch that reversed OrderedSet keys rather than the full dictionary.The example suggests the tests did not capture the requested behavior.Source: §2.1.1 · p. 5
  • Some passed patches changed files or functions unrelated to the issue, unlike the corresponding gold patches.The Matplotlib example changes cbook.py while the gold patch changes _axes.py.Source: §2.1.1 · p. 5
  • Incomplete patches could follow the right approach yet omit edge cases or critical logic, which weak tests failed to catch.The Django example omits error handling and event-loop checks present in the gold patch.Source: §2.1.1 · p. 6

2.1.2 Patterns extracted from the correct fixes

  • Two patterns counted as correct fixes: implementations that differed from the gold patch while resolving the issue, and more comprehensive fixes that handled additional cases.The authors characterize the latter as potentially safer and more robust.Source: §2.1.2 · pp. 6–7

2.2 Updated Resolution Rate of SWE-Agent + GPT-4 on SWE-Bench Full

  • 12.47% to 5.49%: SWE-Agent+GPT-4’s resolution rate on SWE-Bench Full after excluding suspicious fixes.The retained correct fixes were different but valid implementations (30.27%) or more comprehensive than the gold patch (5.98%).Source: §2.2 · p. 7

2.3 Updated Resolution Rate of SWE-Agent+GPT-4 on SWE-bench Lite and Verified

  • 48.14% of SWE-Bench Lite fixes and 55.36% of SWE-Bench Verified fixes were suspicious, reducing resolution rates from 18% to 9.33% and 22.4% to 10.0%, respectively.The model made no different-file or different-function changes in either variant, but both included incorrect and incomplete fixes.Source: §2.3 · p. 7

3 Building SWE-Bench+

SWE-Bench+ was built to evaluate issue resolution without clear solutions in issue reports and to reduce exposure to pre-cutoff data. Its 548 instances were gathered with SWE-Bench’s collection process and additional filtering.

3 Building SWE-Bench+

  • SWE-Bench+ targets real-world GitHub issues without clear solutions in reports and without potential risk of data leakage.The authors used SWE-Bench’s data-collection methodology to support consistent resolution-rate comparisons.Source: §3 · p. 8
  • Issues were collected from the same 12 SWE-Bench projects except Django, and were created after October 2023.The authors state that the models used were trained on data up to October 2023.Source: §3 · p. 8
  • Filtering for issues that resolve a problem, contribute tests, install successfully, and pass all tests, followed by manual leakage screening, yielded 548 instances.The issue and execution filters followed SWE-Bench’s process.Source: §3 · p. 8

4 Robustness of SWE-Bench+

The SWE-Bench+ evaluation manually checked resolved patches and found substantially lower validated resolution rates than leaderboard figures for SWE-Bench Full. Weak tests remained a major concern.

4 Robustness of SWE-Bench+

  • The study tested SWE-RAG with GPT-3.5 and GPT-4, SWE-Agent with GPT-4, and AutoCodeRover with GPT-4o, selecting models based on performance, context window, cost, and openness.Claude-based trials exceeded token limits before experiments could be completed across all 548 instances.Source: §4 · p. 8
  • On SWE-Bench+, validated resolution rates were 0.73% for SWE-RAG+GPT-4, 0.55% for SWE-RAG+GPT-3.5, 0.55% for SWE-Agent+GPT-4, and 3.83% for AutoCodeRover+GPT-4o.The reported SWE-Bench leaderboard rates were 1.31%, 0.17%, and 18.83% for the respective SWE-RAG+GPT-4, SWE-RAG+GPT-3.5, and AutoCodeRover+GPT-4o models.Source: Table 3 · p. 9, §4 · p. 9
  • Around 67.72% of resolved instances did not truly resolve the issue despite passing all tests, with incorrect bug localization a prominent pattern.Other patches were incomplete or incorrect; solution-leak-related issues were resolved, but weak tests persisted.Source: §4 · p. 9
  • The evaluation manually reviewed instances marked resolved, retained those passing all PASS_TO_PASS and FAIL_TO_PASS tests, then compared generated patches with gold patches.Patch validation followed resolution-rate analysis.Source: §4 · pp. 8–9

5 Effectiveness-aware Evaluation

The evaluation compares model costs per instance and per successfully resolved issue, alongside generation time. The reported figures show that resolution performance and cost-effectiveness do not always align.

5 Effectiveness-aware Evaluation

  • SWE-Agent+GPT-4 averaged around 4 minutes per instance, totaling approximately 37 hours; AutoCodeRover+GPT-4 averaged 4.5 minutes, totaling 41 hours.The authors identify time, cost, and resource allocation as trade-offs for real-world use.Source: §5 · p. 9
  • The two cost measures were average cost per instance across 548 tested instances and effectiveness-aware cost per successfully resolved issue.The detailed model costs are presented in Table 4.Source: §5 · p. 9
  • AutoCodeRover+GPT-4 achieved the highest resolution rate among all models at 3.83%, with an effectiveness-aware cost of $12.61 per issue fixed, versus $32.5 for SWE-Agent+GPT-4.SWE-Agent+GPT-4’s performance was comparable to RAG+GPT-4, whose effectiveness-aware cost was $10.0 per issue fixed.Source: §5 · p. 10
  • The authors recommend that future model evaluations consider implementation and operating costs alongside accuracy for large-scale and long-term use.Source: §5 · p. 10

6 Related Work

Related work spans LLM applications in software engineering and the development of reliable code-generation benchmarks. Prior research also highlights test adequacy and dataset quality as important evaluation concerns.

Related work on LLMs for Software Engineering

Related work on Benchmark Dataset Quality for Code Generation

  • Benchmark work has sought to improve code-generation evaluation through comprehensive tests and real-world issue datasets, including SWE-bench (Jimenez et al., 2024) and EvalPlus (Liu et al., 2024).Source: §6 · p. 10
  • EvalPlus augments datasets with LLM-generated test inputs and mutation-based strategies, and its findings point to insufficient testing and noisy benchmarks as quality concerns (Liu et al., 2024).Source: §6 · p. 10

7 Conclusion

SWE-Bench+ addresses solution leakage with post-cutoff issues, but weak test cases remain a challenge. The authors call for stronger tests and further study of other evaluation benchmarks.

7 Conclusion

  • SWE-Bench+ removes solution leakage by using issues created after LLM training cut-off dates, but weak test cases continue to pose challenges.The authors describe the refined dataset as a more reliable framework for assessing LLM capabilities.Source: §7 · p. 11
  • Further work should improve test robustness and investigate weak test cases to create more accurate and relevant test suites.Source: §7 · p. 11
  • The authors also propose studying causes of high failure rates and comparing results with other benchmarks, including Human-Eval.Source: §7 · p. 11
Loading 2410.06992v2…