Source-linked AI summary
The GitHub Recent Bugs Dataset for Evaluating LLM-based Debugging Applications
Jae Yong Lee, Sungmin Kang, Juyeon Yoon, Shin Yoo
TL;DR
Opaque LLM training data makes it difficult to know whether results on existing bug benchmarks reflect generalization or memorization. The authors assess Defects4J overlap in StarCoder and build GHRB, a post-cutoff real-world Java bug dataset; they report substantial Defects4J overlap and no GHRB overlap above their 90% criterion.
Problem
Because LLM training data are often undisclosed, researchers cannot readily assess whether existing bug benchmarks were included, making it unclear whether results reflect generalization or code memorization.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
GHRB is a supplementary benchmark for evaluating the potential generality of LLM-based software engineering tools.
Source: §7 · p. 3
Abstract
from arXiv · showhide
Large Language Models (LLMs) have demonstrated strong natural language processing and code synthesis capabilities, which has led to their rapid adoption in software engineering applications. However, details about LLM training data are often not made public, which has caused concern as to whether existing bug benchmarks are included. In lieu of the training data for the popular GPT models, we examine the training data of the open-source LLM StarCoder, and find it likely that data from the widely used Defects4J benchmark was included, raising the possibility of its inclusion in GPT training data as well. This makes it difficult to tell how well LLM-based results on Defects4J would generalize, as for any results it would be unclear whether a technique's performance is due to LLM generalization or memorization. To remedy this issue and facilitate continued research on LLM-based SE, we present the GitHub Recent Bugs (GHRB) dataset, which includes 76 real-world Java bugs that were gathered after the OpenAI data cut-off point.
1 Introduction
Open-source bug benchmarks support standardized evaluation, but their permissively licensed code may also enter LLM training data. The authors report Defects4J overlap in StarCoder and introduce GHRB, a post-cutoff real-world Java benchmark intended to reduce contamination concerns.
1 Introduction
- Defects4J bugs often appear in StarCoder training data, while popular OpenAI LLMs’ training data are not public.The authors say similar training data for OpenAI models is a reasonable assumption, not a confirmed fact.Source: §1 · p. 1
- The overlap raises the question of whether LLM results reflect generalization or code memorization; the authors know of no subsequent uncontaminated real-world bug dataset.The concern had been voiced in early literature [3].Source: §1 · p. 1
- The authors propose GHRB, 76 real-world Java bugs fixed after September 2021, and confirm these bugs were not used to train StarCoder.They present it as a benchmark for evaluating LLM applications without concern about data leakage.Source: §1 · p. 1
- GHRB is made publicly available to support evaluation of LLM-based bug techniques without data leakage concerns.The paper also reports a partial assessment of Defects4J leakage and an automated framework for gathering and filtering bug benchmarks.Source: §1 · p. 1
2 Motivation
The authors examine whether Defects4J data appears in StarCoder training, motivated by opaque LLM training data and the resulting uncertainty about benchmark validity. They find substantial overlap and argue for a standard real-world benchmark relatively free from contamination concerns.
2 Motivation
- The authors use Data Portraits to check whether Defects4J v1.0 bug-revealing tests and buggy methods occur in StarCoder training data.They assess 345 bugs with buggy methods and conservatively treat at least 90% overlap as likely inclusion.Source: §2 · p. 2
- Opaque OpenAI training details leave researchers unable to assess directly whether benchmark data was used, prompting an initial assessment with an open-source LLM.The approach assumes LLM developers are likely to gather similar training data.Source: §2 · p. 2
- 35% of tests and 39% of buggy methods in Defects4J v1.0 were included in StarCoder training data under the 90% criterion; 59% of bugs were likely compromised.Some matched subsequences break at the fix location, suggesting StarCoder was trained with fixed method versions; recent bugs in new Defects4J v2.0 projects also met the criterion.Source: §2 · p. 2
- The authors argue LLM-based SE research would benefit from a standard real-world benchmark relatively free from contamination concerns.Prior dataset efforts are described as ad hoc and uncoordinated.Source: §2 · p. 2
3 Data Collection
GHRB collection targets recent, source-code bugs that are reproducible and isolated. The authors automatically identify candidate pull requests, then filter and inspect them and verify that retained tests distinguish buggy from fixed versions.
3 Data Collection
- Each collected bug must be in project source code, have a test that fails before and passes after the fix, and isolate the bug from unrelated changes.Fixes to build scripts, configurations, documentation, or tests are excluded.Source: §3 · p. 2
- Collection and verification scripts are included in the artifact to facilitate GHRB expansion.Source: §3 · p. 2
3.1 Identifying Potential Bugs
- Candidate pull requests were gathered from repositories combining Kang et al.’s initial GHRB set [14] and 100 highly starred Java repositories, using post-September 2021 PRs linked to bug reports and adding or modifying tests.PRs without Java changes or English bug reports were filtered, followed by manual consistency inspection.Source: §3.1 · p. 2
3.2 Reproducing Bugs
- Tests passing or failing on both versions, unrelated to the bug, or flaky were removed through automated filtering and manual review.The final bugs are considered reproducible to the authors’ best knowledge.Source: §3.2 · p. 3
4 Database of Real Bugs
GHRB contains 76 bugs from 16 repositories, and the authors report no StarCoder overlap above their 90% criterion for its oldest bug-revealing tests. The dataset also records bug metadata, revealing tests, and patch information.
4 Database of Real Bugs
- GHRB contains 76 bugs from 16 repositories, and its oldest bug-revealing tests had no StarCoder overlap above the 90% criterion.The authors consider GHRB safe for evaluating StarCoder applications and, because its commits postdate September 2021, OpenAI LLM applications.Source: §4 · p. 3
- Each bug record includes pull-request metadata, one or more tests that fail on the buggy version and pass on the fixed version, and the patch diff.Test paths and root causes are also provided.Source: §4 · p. 3
5 Interface of GHRB
GHRB provides interfaces that abstract version-control access, build-environment setup, and test execution for its bug targets.
5 Interface of GHRB
- Users can access buggy and fixed repository versions through integer bug IDs and “b” or “f” flags, without knowing each project’s version-control system.Bug IDs are ordered chronologically by creation date.Source: §5 · p. 3
- GHRB automatically identifies build tools, project-specific compilation scripts, and required tool versions so users need not configure each target’s build environment manually.For some repositories, integrated wrapper scripts specify tool versions.Source: §5 · p. 3
- After checkout, GHRB uses a configuration file to find failing-test information and run tests without project-specific test commands.The configuration file is created when a user checks out a repository version.Source: §5 · p. 3
6 Related Work
GHRB builds on prior real-world bug benchmarks and expands an earlier post-cutoff Java bug dataset, while contrasting with Defects4J’s data-leakage risk.
6 Related Work
- GHRB expands Kang et al.’s dataset from 31 bugs across 6 repositories to 76 bugs across 16 repositories and adds a command-line interface.The earlier dataset included real-world Java bugs with reproducing tests added after the OpenAI training-data cutoff [14].Source: §6 · p. 3
- Unlike Defects4J, GHRB contains only data created after OpenAI LLMs’ cutoff, addressing the stated risk of data contamination in LLM evaluations.The paper reports data-leakage risk for state-of-the-art LLM evaluations using Defects4J.Source: §6 · p. 3
- The authors hope to extend the dataset to languages beyond Java in future work.Source: §6 · p. 3
7 Conclusion
The conclusion presents GHRB as a real-world Java benchmark intended to mitigate data leakage in LLM-based software-engineering evaluations. The public-release table lists projects and their available bug counts.
7 Conclusion
- GHRB is a real-world Java bug dataset designed to mitigate data-leakage concerns in evaluations of LLM-based software-engineering techniques.Source: §7 · p. 3
- The authors position GHRB as a supplementary benchmark for established bug datasets, supporting more convenient evaluation of LLM tools’ potential generality.Source: §7 · p. 3
- Table 2 lists projects and the number of real bugs in the public release as of 6 September 2023.Source: Table 2 · p. 4