Source-linked AI summary
How Often Do Single-Statement Bugs Occur? The ManySStuBs4J Dataset
Rafael-Michael Karampatsis, Charles Sutton
TL;DR
Program repair lacks real-world evidence for estimating how often simple bugs occur and how much recall template-based techniques achieve. The paper constructs the ManySStuBs4J dataset from single-statement Java bug fixes and annotates them with 16 templates. About one-third of the fixes match a template, while template-fitting bugs occur about once per 1,600–2,500 lines of code; the dataset is intended to support repair and empirical software-engineering research.
Problem
Real-world data was lacking for estimating the recall of repair templates for simple bugs and how often their associated bugs occur in code.
Method
The paper mines 153,652 single-statement bug-fix changes from 1,000 popular open-source Java projects and annotates them against 16 bug templates inspired by program-repair techniques.
Results
About 33% of mined single-statement bug fixes match at least one SStuB template, with template-fitting bugs occurring about once per 1,600–2,500 lines of code.
Takeaways & Limitations
ManySStuBs4J can support future program-repair research and empirical software-engineering studies, including evaluation of repair-template recall.
Takeaways & Limitations
The dataset is restricted to Java, and mined bugs are not annotated by severity.
Abstract
from arXiv · showhide
Program repair is an important but difficult software engineering problem. One way to achieve acceptable performance is to focus on classes of simple bugs, such as bugs with single statement fixes, or that match a small set of bug templates. However, it is very difficult to estimate the recall of repair techniques for simple bugs, as there are no datasets about how often the associated bugs occur in code. To fill this gap, we provide a dataset of 153,652 single statement bug-fix changes mined from 1,000 popular open-source Java projects, annotated by whether they match any of a set of 16 bug templates, inspired by state-of-the-art program repair techniques. In an initial analysis, we find that about 33% of the simple bug fixes match the templates, indicating that a remarkable number of single-statement bugs can be repaired with a relatively small set of templates. Further, we find that template fitting bugs appear with a frequency of about one bug per 1,600-2,500 lines of code (as measured by the size of the project's latest version). We hope that the dataset will prove a resource for both future work in program repair and studies in empirical software engineering.
1 INTRODUCTION
The paper addresses the difficulty of estimating repair-template recall by introducing ManySStuBs4J, a large annotated dataset of single-statement Java bug fixes. About one-third of mined fixes match one of 16 templates, and template-fitting bugs occur roughly once per 1,600–2,500 lines of code.
- Program repair requires analyzing failures, locating faults, synthesizing fixes, and validating that repairs introduce no new faults.
- Existing simple-bug and template-based repair evaluations lack enough real-world data to estimate template recall.Prior evaluations used relatively few projects or synthetic data, while another one-line-bug dataset did not classify bugs into templates.
- 153,652 single-statement bug-fix changes from 1,000 popular open-source Java projects were annotated against 16 repair-inspired bug templates.A smaller variant contains 25,539 changes from 100 popular open-source Java Maven projects.
- 33.04% of the smaller dataset and 33.47% of the larger dataset match at least one SStuB template.These correspond to 10,231 and 63,923 SStuB instances, respectively.
- Template-fitting SStuBs occur at about 1 per 1,600 lines of code in the smaller dataset and 1 per 2,500 lines in the larger dataset.The frequency uses the size of each project’s latest version.
2 METHODOLOGY
The dataset is built by mining popular Maven-based Java projects for bug-fixing commits, filtering for compilable single-statement changes, extracting AST differences, and matching them against 16 SStuB patterns.
- Project selection: The authors selected popular open-source Java Maven projects so dependencies could be downloaded automatically and projects could be built for repair-tool evaluation.The small dataset used the 100 most popular projects up to 1/4/2017.
- Change filtering: The filtering retains small, compilable fixes involving single statements and excludes commits that add or delete Java files.Modified statements are counted from diffs, including statements inside blocks and conditions in if and while statements.
- AST extraction: For each repaired line, the tool constructs pre-repair and line-repaired AST pairs and locates their first differing node through simultaneous depth-first traversal.The AST pair differs only at nodes associated with the modified line.
- Quality control: The methodology includes safeguards against non-fixing or behavior-preserving changes, including false-positive commit-message matches and tangled changes.The authors specifically examine refactorings that can produce small changes while not fixing bugs.
- Pattern annotation: The resulting changes are automatically checked against 16 SStuB patterns expressed as mutation operations on the original AST; matching instances are saved in the SStuBs dataset.The patterns were chosen because they appear often and have been used in pattern-based repair and mutation tools.
3 MANYSSTUBS4J DATASET STATISTICS
The dataset contains tens of thousands of mined single-statement bug instances across its versions, while the authors caution that bug severity was not annotated and some instances come from test code.
- Dataset scale: 10,231 and 63,923 single-statement bug instances were mined from 12,598 and 86,771 single-statement bug-fix commits, respectively, across the dataset versions.On average, the mining produced almost 2 single-statement bugs and 0.75 SStuBs per valid commit.
- Data format: The dataset is stored in JSON files, with detailed information available through the project’s GitHub repository.Each SStuB instance is annotated with its satisfied pattern and project name, among other information.
- Scope and caveats: Bug severity was not annotated, and some mined bugs occur in test code rather than production code.The authors note that test-code bugs can mask important bugs and complicate debugging despite not reaching the final product.
4 RESEARCH QUESTIONS
Although the paper focuses on the dataset, it performs a simple analysis to assess whether SStuB patterns are useful targets for program repair techniques.
- Research questions: The authors formulate two research questions to investigate whether SStuB patterns are useful targets for program repair techniques.The analysis supports the design decision to focus the new dataset on SStuBs.
RQ1. Are SStuBs common in open-source code?
SStuBs occur frequently in the mined datasets, with density estimates varying by denominator and dataset size. Using final-snapshot LOC, the larger dataset reports about one SStuB per 1,600 LOC, while modification-based densities are much lower.
- The study estimates two densities: SStuBs per final-snapshot LOC and SStuBs per LOC added and deleted.The authors note that final-snapshot LOC may not be the most informative denominator for bug density.
RQ2. Can SStuBs be spotted by existing tools such as static analyzers?
SpotBugs detects only a small fraction of SStuBs while producing an enormous warning volume, highlighting the need for tools specifically designed to detect them.
- 12% of SStuBs were located by SpotBugs, which reported more than 200 million possible bugs when all warnings were enabled.The authors state that actual recall is even lower and cite a study finding three static detectors located only 4.5% of bugs.
- SpotBugs’ low detection rate would require developers to inspect hundreds of thousands of warnings to find a single SStuB.This motivates tools specifically built to detect SStuBs.
5 RELATED WORK
Prior real-world bug datasets vary substantially in size and scope. ManySStuBs instead emphasizes estimating the frequency of template-matching single-statement bugs in Java projects.
- Defects4J contains 395 Java bugs, ManyBugs contains 185 C bugs, and Bugs.jar contains 1,158 Java bugs.These datasets contain fixes that may modify multiple source-code lines.
- Combined CodRep and Bugs2Fix datasets contain 40,289 one-line bugs, whereas ManySStuBs provides 25,539 and 153,652 single-statement bugs.The datasets are similar in broad scale, but ManySStuBs focuses on template frequency.
- ManySStuBs focuses on estimating the frequency of SStuB templates motivated by recent program-repair tools.Its dataset is restricted to single-statement bugs in Java projects.
6 LIMITATIONS - THREATS TO VALIDITY
The dataset has methodological and scope limitations: its mining process can produce unrelated aligned statements, and its coverage is limited to Java and small bugs.
- The SZZ-like methodology may align an unrelated deleted statement with an added statement.Refactoring detection is incomplete, and the heuristic for identifying bug-fixing commits can introduce false positives, although the authors say the false-positive rate is low.
- The dataset is not suitable for evaluating repair systems on larger bugs.It is restricted to Java, though the authors describe how the approach could be replicated for other languages.
7 CONCLUSIONS
The paper introduces a large-scale Java dataset of real-world single-statement bugs categorized by 16 repair-inspired templates. Its observed frequency and coverage make it a potential resource for evaluating repair recall and studying simple bugs.
- The dataset contains real-world Java SStuBs categorized into 16 templates inspired by state-of-the-art program repair methods.
- SStuBs occur at about one per 1,600 lines of code in the studied projects.
- The dataset could help estimate actual repair recall and support empirical studies of how simple bugs are introduced.
- Coverage information from tested Maven projects could be used to estimate how often tests cover SStuBs.