Source-linked AI summary
Bears: An Extensible Java Bug Benchmark for Automatic Program Repair Studies
Fernanda Madeiral, Simon Urli, Marcelo Maia, Martin Monperrus
TL;DR
Bug benchmarks support reproducible evaluation and comparison of automatic program repair, but collecting real, reproducible bugs across diverse systems is difficult. Bears uses Continuous Integration build pairs and a reproduction pipeline to construct an extensible Java benchmark, yielding 251 reproducible bugs from 72 projects while remaining limited to Maven projects and a uniform reproduction environment.
Problem
Bug benchmarks are needed to support reproducibility, generality, comparison, and progress measurement, yet collecting real, reproducible bugs from diverse systems is challenging.
Method
Bears scans Travis CI build pairs, reproduces candidate bugs by testing failing and passing program versions, and stores successful pairs in dedicated GitHub branches.
Results
251 reproducible bugs from 72 projects compose the first version of the BEARS-BENCHMARK.
Takeaways & Limitations
Using Continuous Integration enables a publicly available benchmark spanning diverse projects and supports community extension through repository branches and pull requests.
Takeaways & Limitations
The BEARS-COLLECTOR covers only Maven projects, and its uniform reproduction environment differs from project-specific Travis CI environments, potentially causing failures or false positives.
Abstract
from arXiv · showhide
Benchmarks of bugs are essential to empirically evaluate automatic program repair tools. In this paper, we present Bears, a project for collecting and storing bugs into an extensible bug benchmark for automatic repair studies in Java. The collection of bugs relies on commit building state from Continuous Integration (CI) to find potential pairs of buggy and patched program versions from open-source projects hosted on GitHub. Each pair of program versions passes through a pipeline where an attempt of reproducing a bug and its patch is performed. The core step of the reproduction pipeline is the execution of the test suite of the program on both program versions. If a test failure is found in the buggy program version candidate and no test failure is found in its patched program version candidate, a bug and its patch were successfully reproduced. The uniqueness of Bears is the usage of CI (builds) to identify buggy and patched program version candidates, which has been widely adopted in the last years in open-source projects. This approach allows us to collect bugs from a diversity of projects beyond mature projects that use bug tracking systems. Moreover, Bears was designed to be publicly available and to be easily extensible by the research community through automatic creation of branches with bugs in a given GitHub repository, which can be used for pull requests in the Bears repository. We present in this paper the approach employed by Bears, and we deliver the version 1.0 of Bears, which contains 251 reproducible bugs collected from 72 projects that use the Travis CI and Maven build environment.
I. INTRODUCTION
BEARS addresses the need for reproducible, diverse, and extensible Java bug benchmarks by collecting bugs from Continuous Integration build states. It delivers a publicly extensible collector and benchmark containing 251 reproducible bugs from 72 projects.
- Systematic bug benchmarks support reproducibility, generality, direct comparisons, and progress measurement in automatic program repair.
- Existing Java benchmarks rely on bug-fixing commits and cover only 13 mature projects in total, limiting project diversity.
- BEARS uses Continuous Integration build states to identify candidate buggy and patched versions rather than relying on traditional bug trackers.
- Successfully reproduced bugs are stored as publicly available GitHub branches, allowing extensions through pull requests to the benchmark repository.
- BEARS-COLLECTOR publicly implements the collection approach for Java, Travis CI, and Maven projects.
- 251 bugs from 72 projects comprise BEARS-BENCHMARK, which the authors describe as the largest reproducible benchmark by project diversity.
II. BEARS DESIGN DECISIONS
BEARS identifies candidate bugs by mining adjacent Travis CI builds and interpreting their statuses and file changes. It covers both failing-to-passing repairs and passing-to-passing builds whose new tests expose an earlier defect.
- BEARS uses Continuous Integration build statuses to locate test-failing program versions and their developer-written patches.
- A Travis build marked errored is excluded because failure occurred before tests, while a passed build suggests a compilable and testable version.
- Candidate pairs consist of immediately subsequent builds, with selection based on build statuses and files changed between their commits.
- Case #1: failing-passing builds with no test changes: In case #1, a failing build followed by a passing build without test changes indicates a candidate bug fixed in source code.
- Case #2: passing-passing builds with test changes: In case #2, both builds pass, but changed tests expose a failure when run against the earlier source version.
C. Inclusion Criteria for Projects
BEARS restricts source projects and benchmark entries to keep collection feasible and reproducible. Its repository design organizes reproduced bugs into standardized branches that can be contributed through pull requests.
- Inclusion Criteria for Projects: Eligible projects must be public GitHub repositories using Travis CI and Maven.
- Inclusion Criteria for Bugs: Each benchmark bug must be reproducible with at least one bug-triggering test case.
- Inclusion Criteria for Bugs: Each benchmark bug must have been fixed by a human-written patch.
- Bug Repository Design: The branch-based repository design supports organization, public availability, and future additions through pull requests.
- Bug Repository Design: The collector stores each successfully reproduced bug in a dedicated, standardized GitHub branch containing canonical commits and a bears.json metadata file.
III. THE BEARS-COLLECTOR PROCESS
The BEARS-COLLECTOR scans Travis CI builds, selects candidate pairs, reproduces their failures and fixes, and stores successful reproductions in GitHub branches. Scanning can operate over a fixed time window or continuously in real time.
- The process scans Travis CI build pairs and routes each selected pair through reproduction toward a failing test followed by a passing test suite.
- Build scanning supports either a fixed project-and-time-window search or repeated real-time monitoring of completed Travis builds.
- The scanner selects passing builds as patched candidates, retrieves their previous builds, and applies status and file-difference rules.
- The selection algorithm accepts failing-to-passing pairs when source code changes without test-file changes, or passing-to-passing pairs when source and test files change.
- Reproduction initializes a pipeline that checks out commits and test files, runs the program tests, and records whether the pair is successfully reproduced.
- Successful reproduction creates and pushes a dedicated branch containing the canonical commit sequence.
B. Phase II–Reproduction
The reproduction phase builds each candidate pair, runs its tests, and checks for a failure in the buggy version followed by a passing suite in the patched version. A successful reproduction therefore establishes both the bug and its patch.
- It then checks out the patched candidate, builds it, and runs the test suite to seek a passing result.
- A bug and patch are reproduced only when the buggy candidate fails tests and the patched candidate has no test failure.
- The pipeline constructs and executes an ordered sequence of steps for each candidate build pair.Its two phases are pipeline construction and pipeline execution.
- The process first checks out the buggy candidate, builds it, runs tests, and verifies that at least one test fails.A failure in the buggy version allows reproduction to continue.
C. Phase III–Validation
The validation phase stores successfully reproduced bugs as branches and subjects proposed additions to automated and manual checks before merging them into the benchmark. The collection process used three scanning rounds, including real-time Travis CI scanning.
- Phase III–Validation: Each successfully reproduced bug is submitted as a branch for inclusion in the BEARS-BENCHMARK repository through a pull request.The special base branch is “pr-add-bug”.
- Phase III–Validation: Automated validation triggers Travis CI checks, including verifying test failures on the buggy program version.
- Phase III–Validation: Manual validation examines whether the proposed branch contains a genuine bug using source differences and recorded test failures.
- Phase III–Validation: Branches that pass both validations are merged, while others are closed.
- Scanning: The benchmark was collected through three execution rounds using time-windowed project lists and real-time Travis CI scanning.The third round scanned builds in real time over approximately two months in 2018.
- Scanning: 168,772 builds yielded 12,355 build pairs: 741 failing-passing pairs and 11,614 passing-passing pairs.
B. Reproducing 12,355 Build Pairs
The reproduction pipeline processed 12,355 build pairs, producing 856 successful reproductions and identifying checkout and build failures as the most frequent failure statuses. Other failures included missing test failures, unreproduced patches, timeouts, and infrastructure problems.
- 856 of 12,355 reproduction attempts succeeded, representing 7% of all attempts.The remaining 93% were failed reproduction attempts.
- 40.8% of attempts failed during checkout because required commits or files were missing from repository history.Commits may have been deleted directly or become unavailable after branch deletion.
- 33.6% of attempts failed while building or compiling projects, often because dependencies were missing, source and test files were mixed, or Maven timed out.The pipeline aborted when Maven produced no output for 10 minutes.
- Testing failures affected 0.9% of attempts, directory analysis affected 1.8%, and nine attempts failed for other infrastructure or tooling reasons.
- 8.2% of attempts found no test failure in the buggy candidate, so reproduction was aborted.This commonly occurred for passing-passing pairs and could also reflect flaky tests.
- 7.5% of attempts failed to reproduce a fully passing patched candidate test suite.Possible explanations included flaky tests and differences between local and Travis environments, such as Java versions.
C. Validating 856 Branches
The validation pipeline reduced 856 generated branches to 251 benchmark entries through manual and automatic checks. Manual review exposed ignored tests, hidden fixes, refactorings, feature additions, duplicate patches, unrelated commits, and mixed changes.
- Validation outcome: 251 of 856 generated branches passed both manual and automatic validation.Manual validation accepted 295 branches, of which 251 passed automatic validation.
- Manual-validation findings: The validation process identified branches where bug-triggering tests were ignored in the first passing build.This caused apparently passing candidates to require manual inspection.
- Manual-validation findings: Some passing-passing pairs represented refactoring or cleaning rather than bug fixes.In one example, source code was removed and tests were adapted to pass on the cleaned version.
- Manual-validation findings: Some passing-passing pairs represented feature additions or enhancements rather than bug fixes.Changed tests could fail on the earlier source version because they exercised newly added functionality.
- Manual-validation findings: Duplicate patches arose when the same changes were applied to different branches and generated separate benchmark branches.The described FasterXML/jackson-databind example merged equivalent changes from branch 2.7 into branch 2.8.
- Manual-validation findings: Other invalid branches combined unrelated commits or mixed genuine bug fixes with refactoring and formatting changes.The benchmark discarded some mixed-change branches to make patch studies easier, without claiming all retained patches are isolated fixes.
> Invalid branches during the automatic validation
Automatic validation invalidated branches mainly when the patched version failed to build or pass its tests, while the resulting benchmark covers diverse projects and contains 251 reproducible bugs.
- Automatic validation: Automatic validation mainly rejected branches because the patched version failed during building or test execution.Such failures suggest issues including flaky tests.
- Benchmark outcome: 251 reproducible bugs constitute BEARS-BENCHMARK version 1.0.Nineteen bugs came from failing-passing builds and 232 from passing-passing builds with test changes.
- Project diversity: Table III presents project metadata including type, domain, age, contributors, commits, LOC, and average test counts.The excerpt includes five projects with more bugs and five randomly selected projects.
- Project diversity: The benchmark spans projects differing in domain, age, and size.Reported projects cover at least ten domains, range from less than one year to eight years old, span 759 LOC to 205 KLOC, and contain 16 to 8K tests.
B. Constituent Exception Types
BEARS records the exception and patch characteristics of reproducible Java bugs, supporting analysis of which failures expose bugs and how extensive developer patches are. The collection process is based on CI and experienced substantial reproduction challenges.
- Exception types: BEARS distinguishes JUnit test failures from test errors when characterizing reproduced bugs.Failures arise from unmet assertions, whereas errors arise from uncaught exceptions during execution.
- Exception types: AssertionError occurred in 108 bugs, ComparisonFailure in 31, and NullPointerException in 26.AssertionError and ComparisonFailure were the two most frequent exceptions reported.
- Patch characteristics: BEARS stores buggy and patched versions so their diff represents the developer-created patch.Patch size counts added, deleted, and modified lines, while spreading counts modified files.
- Patch characteristics: Patches ranged from 1 to 312 lines, with 50% involving at most 8 lines.These statistics summarize the collected patch sizes.
- Patch characteristics: 75% of patches changed at most one file, and no patch changed more than 10 files.Patch spreading was measured by the number of modified files.
- Process context: The study reports CI-based bug collection as novel and notes that 33.6% of reproduction attempts failed during building.The authors present these reproduction issues as insights for future bug-collection research.
A. Challenges
BEARS faced challenges in automating bug collection, validating candidates, and broadening benchmark coverage beyond mature projects and fixed datasets.
- Implementation challenges: Integrating automated scanning, reproduction, storage, and metadata generation made BEARS-COLLECTOR development challenging.The collector had to scan builds, perform reproduction attempts, and store successful cases in a standardized public repository.
- Implementation challenges: Multi-module and non-standard projects required identifying source and test files before combining versions for reproduction.This challenge arose when passing builds included test changes and source code from one commit had to be tested with another commit’s tests.
- Scope limitations: The collector supports only Maven projects, although its underlying approach is independent of the build tool and could be extended to Gradle.
- Scope limitations: Using a uniform reproduction environment instead of Travis CI’s project-specific environment may cause failed reproductions and false positive bugs.
- Validation challenges: Manual validation is time-consuming and remains a bottleneck for scaling benchmark construction.Validation may require examining source-code differences, although issue references can simplify some cases.
VIII. CONCLUSION
The paper presents BEARS-COLLECTOR and BEARS-BENCHMARK as an automated, extensible project for collecting reproducible bugs from Travis CI builds. Its first benchmark contains 251 bugs from 72 projects, while future work targets collector heuristics and benchmark analysis.
- VIII. CONCLUSION: BEARS-COLLECTOR reproduces buggy and patched program versions identified from Travis CI builds, scanning selected projects and time windows or projects in real time.
- VIII. CONCLUSION: The project was designed to maximize automation and project diversity while allowing the research community to contribute new benchmark bugs.
- VIII. CONCLUSION: The work provides publicly available artifacts, including the benchmark, collector, bug browser, and Section IV data.
- VIII. CONCLUSION: Future work includes heuristics to reduce manual validation effort and studies of bug and patch characteristics such as repair patterns.