Source-linked AI summary
Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study
You Wang, Michael Pradel, Zhongxin Liu
TL;DR
SWE-bench evaluations may accept plausible patches that do not match developer expectations, but the extent of this problem is unclear. This study examines plausible patches on SWE-bench Verified using PatchDiff and finds substantial behavioral divergence from ground-truth patches, motivating more reliable validation.
Problem
Because test suites are rarely exhaustive, it is unclear how often SWE-bench evaluations accept plausible but incorrect patches, and effective detection methods are lacking.
Method
The study analyzes patches from three state-of-the-art issue-solving tools on SWE-bench Verified, comparing them with oracle patches through PatchDiff-generated differentiating tests and manual inspection.
Results
29.6% of plausible patches were differentiated from their oracle patches by PatchDiff, indicating behavioral discrepancies in a substantial proportion of evaluated patches.
Takeaways & Limitations
The findings call for careful selection of developer tests and additional checking and filtering of patches during issue-solving evaluation.
Takeaways & Limitations
PatchDiff uses GPT-4o-mini, limiting access to potentially more advanced models, while manual analyses may introduce human bias despite consensus procedures.
Abstract
from arXiv · showhide
Automated issue solving aims to resolve real-world issues in software repositories. The most popular benchmarks for automated issue solving are SWE-bench and its human-filtered subset SWE-bench Verified. These benchmarks leverage testing to validate generated patches. However, because testing is rarely exhaustive, a patch may pass the tests but nevertheless fail to match the developers' expectations. Unfortunately, it is currently unclear to what extent evaluations performed with SWE-bench suffer from such plausible but incorrect patches. This paper presents an in-depth empirical study of the correctness of plausible patches generated by three state-of-the-art issue-solving tools evaluated on SWE-bench Verified. We extensively test and inspect generated patches, and compare them against human-written ground truth patches. The core of our methodology is a novel technique PatchDiff for differential patch testing, which automatically exposes behavioral discrepancies between two patches. Our findings reveal critical weaknesses in SWE-bench's patch validation mechanism, which causes 7.8% of all patches to count as correct while failing the developer-written test suite. Moreover, our novel automated technique reveals that even more (29.6%) plausible patches induce different behavior than the ground truth patches. These behavioral differences are often due to similar, but divergent implementations (46.8%) and due to generated patches that adapt more behavior than the ground truth patches (27.3%). Our manual inspection shows that 28.6% of behaviorally divergent patches are certainly incorrect. Combined, the different weaknesses lead to an inflation of reported resolution rates by 6.2 absolute percent points. Our findings are a call to arms for more robust and reliable evaluation of issue-solving tools. We envision our automated differential patch testing technique to be useful for this purpose.
1 Introduction
The paper examines whether SWE-bench evaluations reliably identify correct issue-solving patches. It studies this problem with PatchDiff and reports substantial behavioral divergence between plausible and oracle patches.
- Motivation: SWE-bench evaluates generated patches using issue-associated tests, but weak or incomplete suites can accept plausible patches that do not satisfy developers’ expectations.The study focuses on SWE-bench Verified, a human-filtered subset of 500 tasks.
- Study design: The study analyzes plausible patches from CodeStory, LearnByInteract, and OpenHands on SWE-bench Verified across four research questions.The questions address test coverage, behavioral discrepancies, difference patterns, and correctness.
- Method: PatchDiff generates tests that expose behavioral discrepancies between generated plausible patches and developer-written oracle patches.It uses call traces to identify target functions and provides focused evidence for manual assessment.
- Findings: 29.6% of plausible patches can be differentiated from their oracle patches by PatchDiff-generated tests.These patches are treated as suspicious because they may diverge from expected behavior.
- Findings: Behavioral discrepancies commonly involve similar but divergent implementations (46.8%) or plausible patches adapting more behavior than oracle patches (27.3%).The study manually analyzes a sample of 77 suspicious patches to characterize these patterns.
- Implications: The paper recommends stronger developer-test selection, filtering of plausible but incorrect patches, and attention to supplementary semantic changes in benchmark evaluation.It also provides a replication package containing PatchDiff and the study results.
2 Background and Motivating Example
SWE-bench and SWE-bench Verified validate patches by running test files modified in the issue’s pull request. The motivating example shows that this process can accept a patch that behaves differently from the oracle implementation.
- Background: SWE-bench is the most popular benchmark for automated issue solving and provides a test patch for each task.Tasks require generating patches from issue statements and buggy repository versions.
- Validation: SWE-bench and SWE-bench Verified regard a generated patch as correct when all test files modified in the test patch pass.These files may not cover every functionality affected by the generated patch.
- Motivating example: In the sympy-22714 example, the generated patch suppresses imaginary-input checking under evaluate(False), whereas the oracle patch explicitly tests whether the imaginary part is zero.The generated patch therefore allows invalid object creation to proceed.
- Motivating example: Weak tests can accept plausible but incorrect patches, causing performance overestimation and misleading comparisons between issue-solving tools.Manual identification is labor-intensive and must be repeated for new, similar patches.
3 Methodology
The methodology compares generated patches with oracle patches using PatchDiff, an LLM-assisted differential testing technique. It identifies relevant target functions through call traces, generates and repairs tests, and filters unqualified or flaky tests.
- Study setup: The study examines plausible patches from CodeStory, LearnByInteract, and OpenHands on the human-validated SWE-bench Verified subset.The selected tools open-source their implementations and/or results.
- Differential patch testing: PatchDiff takes the issue, buggy repository, test patch, oracle patch, and generated patch, then produces tests intended to reveal behavioral differences between the two patches.It skips generated patches that are syntactically identical to the oracle and uses call traces to provide contextual code.
- Target function identification: Target functions are non-test functions related to patch-modified code that are directly invoked by developer-written tests.PatchDiff identifies them by instrumenting patch-modified functions and collecting call traces in both patched repositories.
- Test generation: PatchDiff selects at most 10 target functions, prioritizing those with the smallest number of non-test functions in their call traces.The selection favors simpler relationships between modified functions and test-reachable targets.
- Test generation: For each target function, the LLM receives the function, both patches, contextual code, and shortest call traces to generate differentiating tests.The call traces show how developer tests invoke the target and how it reaches patch-affected functions.
- Test filtering: Generated tests may be repaired for two cycles when they fail under both patches, and tests are filtered unless they reliably distinguish the patches across repeated runs.A valid test passes under one patch for all 20 runs and fails under the other at least once.
4 Empirical Study
The study evaluates PatchDiff for identifying behavioral discrepancies between plausible and oracle patches, then analyzes why those discrepancies arise and how often suspicious patches are incorrect. It finds substantial divergence, including divergent implementations, supplementary semantic changes, and regression errors.
- PatchDiff: PatchDiff generates tests that expose behavioral discrepancies between plausible patches and developer-written oracle patches.It targets patch-modified functions and related functions invoked by developer-written tests.
- RQ2: Behavioral discrepancies: 29.6% of plausible patches are identified as suspicious, and filtering them reduces the three tools’ resolution rates by 17.3% on average.On average, 82.7% of suspicious patches cannot be identified by running all developer tests.
- PatchDiff: PatchDiff is reported as effective and cost-efficient for large-scale use, costing $0.105 per patch with two repair iterations.Reducing the repair bound to one iteration lowers the cost to $0.039 per patch while decreasing suspicious-patch detection by 3.6%.
- Comparison with existing generators: PatchDiff generates differentiating tests more successfully than Pynguin and CoverUp in the reported comparison.PatchDiff produces at least one test file for 117 patches and differentiating tests for 56, whereas none from Pynguin or CoverUp reveal discrepancies.
- RQ3: Patch-difference patterns: The comparison taxonomy attributes behavioral differences mainly to divergent implementations of aligned semantic changes (46.8%) and supplementary semantic changes (27.3%).Supplementary semantic changes are more common than absent semantic changes (27.3% versus 5.2%).
- RQ4: Correctness of suspicious patches: Manual validation identifies 28.6% of suspicious patches as incorrect, including patches that introduce regression errors or irrelevant behavioral changes.The estimated incorrect rate is 11.0% among plausible patches, inflating reported resolution rates by 6.4 points on average.
5 Discussion
The discussion recommends stronger patch validation, better-specified issue statements, and attention to supplementary semantic changes. It also identifies methodological limitations involving model choice, manual analysis, sampling, and reliance on suspicious patches.
- Carefully selecting developer tests for robust patch validation: Developer tests should be selected carefully: use all developer tests by default while excluding non-functional tests.Ignoring unmodified test files inflates performance, while some projects contain tests for code conventions rather than functionality.
- Awareness of plausible but incorrect patches: Patches that pass all developer tests can still be incorrect, so users and benchmark maintainers should check and filter plausible patches.The paper recommends this for more accurate evaluation.
- Paying more attention to patches introducing supplementary semantic changes: Supplementary semantic changes deserve scrutiny because they account for 36.4% of certainly incorrect patches.Additional changes may serve as an indicator of incorrectness.
- Handling under-specified issue statements: Under-specified issue statements can leave suspicious patches with uncertain correctness and may introduce unintended or undesired behaviors.The paper calls for tools that detect and collaboratively refine under-specified requirements.
- Building a new benchmark with well-specified statements: Benchmarks should use well-specified issue statements because vague requirements can misguide tools and constrain benchmark reliability.The paper identifies ambiguity as an ongoing issue despite human judgments of SWE-bench statement quality.
- PatchDiff as an evaluation aid: PatchDiff can support more accurate evaluation by generating differentiating tests before leaderboard submission and prompting manual examination of exposed discrepancies.The recommended workflow combines automated test generation with human assessment of whether differences indicate incorrectness.
- Threats to validity: The study is limited by its use of GPT-4o-mini, manual analysis, a 30% sample of suspicious patches, and restriction of later analyses to PatchDiff-identified suspicious patches.These choices may limit access to more advanced models and introduce human, sampling, or subset-selection bias.
6 Related Work
Related work spans weak test suites, automated test generation, and software engineering agents. PatchDiff differs by targeting behavioral discrepancies between patches in large, complex projects rather than general code coverage or function-level differential testing.
- Weak test suites: Weak test suites are a longstanding problem in test-based automated program repair, where plausible patches may be incorrect.Prior studies have manually assessed plausible patches and identified problematic functionality-deleting modifications.
- Automated test generation: Automated test generation includes search-based, randomization-based, constraint-based, and deep learning-based approaches.These approaches aim to reduce testing effort and help find bugs.
- Automated test generation: PatchDiff generates differentiating tests for large and complex projects, targeting patch-modified code and using patch-specific context to reveal meaningful behavioral differences.Existing LLM-based differential testing focuses on function-level programs, whereas PatchDiff is designed for differential patch testing.
- Software engineering agents: Software engineering agents automate tasks including fault localization, issue solving, program repair, repository setup, and issue-reproducing test generation.This study complements research examining such agents’ strengths and weaknesses.
7 Conclusion
The paper studies plausible patch correctness on SWE-bench, identifies validation weaknesses and behavioral discrepancies, and uses PatchDiff to expose differences between generated and ground-truth patches. It concludes that stronger validation and better-specified benchmarks are needed.
- Conclusion: The study finds that SWE-bench validation overlooks non-modified test files, causing significant performance overestimation.It also investigates behavioral discrepancies, their patch-difference patterns, and the correctness of discrepant patches.
- Conclusion: PatchDiff automatically exposes behavioral discrepancies between plausible and ground-truth patches through differential patch testing.The paper presents it as the core methodological technique for the empirical study.
- Conclusion: The results support carefully selecting developer tests, filtering incorrect plausible patches, and scrutinizing supplementary semantic changes during evaluation.The paper positions these practices as steps toward more accurate issue-solving benchmarks and tools.