GitBug-Java: A Reproducible Benchmark of Recent Java Bugs

• Defects4J mostly contains bugs 10–15 years old, motivating benchmarks aligned with current software stacks and development practices.Source: §1 · p. 1
• Reproducibility means compiling and running each bug years later; prior benchmarks reached 26.6%–96.9%, with none fully reproducible.Source: §1 · p. 1
• GitBug-Java uses GitBug-Actions and developers’ GitHub Actions workflows, selects 2023 commits, and packages the required files and dependencies.Source: §1 · p. 1
• 199 bugs from 55 repositories, all from 2023, are packaged with offline reproduction environments.Source: §1 · p. 1
• The benchmark provides a construction methodology and repository intended to facilitate APR and fault-localization research and address LLM benchmark leakage.The repository also includes documentation and a visualization companion website.Source: §1 · p. 1

2 BUILDING GITBUG-JAVA

• A bug-fix pairs subsequent versions where a test fails before and passes after; GitBug-Actions searches projects using GitHub Actions.The collection targets Java projects using Maven or Gradle and runs with 32 parallel workers.Source: §2 · p. 2

2.1 Finding Locally Executable Repositories

• 1,363 of 66,042 queried repositories were locally executable after filtering for Java, at least 20 stars, under 200 MB, active status, and test-running workflows.Source: §2.1 · p. 2

2.2 Selecting Locally Executable Bug-Fixes

• From January through October 2023, 51,276 commits yielded 508 candidate fixes; excluding 160 non-source-only changes left 348 candidates.Candidates follow two patterns: passing commits with source and test changes, or a failing previous commit followed by a passing source-only change.Source: §2.2 · p. 2

2.3 Exporting Reproduction Environments

• Saving Docker state after test execution produced offline environments for 341 of 348 candidates; seven failed due to unavailable dependencies or workflow timeouts.Each environment averaged 412.3 MiB.Source: §2.3 · p. 2

2.4 Checking Flakiness and Offline Reproduction

• Five offline runs retain fixes with identical test results; 100 fail reproduction or contain flaky tests, and 13 more encounter Docker failures, leaving 228.Source: §2.4 · p. 3

2.5 Curating Bug-Fixes

• 199 bug-fixes remain after manual curation, which removes refactors, feature-only changes, irrelevant cases, and one non-English source.Source: §2.5 · p. 3

3.1 Repositories

• The 55 repositories span diverse, active projects; across them, median commits per project are 395 (52 from 2023), and median tests per project are 221.The mean number of tests per project exceeds 511.Source: Table 1 · p. 3, §3.1 · p. 3

3.2 Bug-Fixes

• The median patch changes one file, two hunks, and 9 lines; the ranges extend to 17 files, 29 hunks, and 586 lines.Lines changed are the sum of added and removed lines.Source: Table 2 · p. 3, §3.2 · p. 4
• The median bug-fix has 497 tests and the median buggy version has 1 failing test; assertion errors account for over 50% of failures.The reported means are 635.37 tests and 2.38 failing tests per buggy version.Source: Table 3 · p. 4, §3.2 · p. 4
• From January to October 2023, the median number of bug-fixes per month was 18, with a minimum of 14 and a maximum of 32.The authors say this distribution shows bug-fixes were found across the collection period.Source: §3.2 · p. 4

3.3 Reproduction Environments

• The reproduction environments support offline execution and are published on Zenodo; the GitHub repository provides scripts, metadata, a command-line interface, and documentation.The environments contain files needed to rebuild a Docker image for each bug's execution.Source: §3.3 · p. 4

4 RELATED WORK

• Other-language benchmarks include FixJS and BugsJS for JavaScript and Defexts for Kotlin; this related-work discussion focuses on Java benchmarks.Source: §4 · p. 4
• Coding-contest benchmarks such as QuixBugs, Code4Bench, RunBugRun [15], EvalGPTFix [23], and DebugBench [20] use problems unlike those developers face in complex software systems.Source: §4 · p. 4
• HumanEval-Java uses artificially injected bugs, while Minecraft lacks test suites for each bug-fix instance, limiting execution-based studies.Source: §4 · p. 4
• Bugs.jar and Bears face execution and reproducibility challenges; Durieux and Abreu report that 96.4% of BugSwarm is unsuitable for APR and FL [4].Source: §4 · p. 4
• Defects4J [8] remains reproducible and adequate for APR and FL, but mostly contains old bugs and was last updated in 2020.The passage also notes evidence that many Defects4J examples appear in LLM training datasets.Source: §4 · p. 4
• Compared with GHRB [9], GitBug-Java is larger and more diverse, with 199 bugs from 55 repositories versus 76 bugs from 16 repositories, and ships files for offline reproduction.Source: §4 · p. 4

5 CONCLUSION

• 199 bug-fixes from 55 relevant open-source repositories were collected from 2023 to keep the benchmark aligned with current development practices.The paper states that this freshness mitigates the risk of data and benchmark leakage in LLM-based research.Source: §5 · p. 4
• Offline reproduction environments support very long-term reproducibility even if bug-fix dependencies become unavailable on the Internet.The included bug-fixes were also manually curated, and the benchmark is publicly available.Source: §5 · p. 4