Empirical Review of Java Program Repair Tools: A Large-Scale Experiment on 2,141 Bugs and 23,551 Repair Attempts

1 INTRODUCTION

• Because prior Java repair evaluations widely use Defects4J and benchmark representativeness is unknown, single-benchmark results may be hard to generalize.The paper notes that no benchmark is perfect.Source: §1 · p. 1
• The experiment uses five benchmarks and 2,141 bugs to compare each tool’s repairability across benchmarks, rather than comparing tools with one another.The authors use a framework to automate and simplify executions across benchmarks.Source: §1 · pp. 1–2
• The tools performed better on Defects4J than on the other benchmarks, and incorrect fault localization and multiple fault locations significantly affected patch generation.The study also identifies six main reasons for non-patch generation.Source: §1 · p. 2
• The study combines a large-scale experiment, an extensible repair-execution framework, a cross-benchmark repairability analysis, and an analysis of non-patch-generation cases.The experiment comprises 23,551 repair attempts.Source: §1 · p. 2

2 STATE OF AFFAIRS ON TEST-SUITE-BASED AUTOMATIC REPAIR TOOLS FOR JAVA

• The review gathers tools from a living review and collects papers that introduce repair approaches or empirically evaluate existing tools.The authors identified 24 Java test-suite-based repair tools in the living review.Source: §2 · p. 2
• Table 1 associates tools with evaluation benchmarks and reports input bugs, test-suite-adequate patches, and correct patches when those figures are available.It groups tools into generate-and-validate, semantics-based, and metaprogramming-based categories.Source: §2 · pp. 2–3, Table 1 · p. 2
• 38 evaluations cover 24 repair tools; 22 tools were evaluated on some Defects4J bugs, while nine were evaluated on QuixBugs.Bugs.jar and IntroClassJava appear in exceptional cases, and some benchmarks were rarely or never used.Source: §2 · p. 3
• Some tools were first evaluated on datasets not introduced in dedicated research papers, and PAR was later reimplemented as xPAR and evaluated on Defects4J.The initially used datasets included PARDataset, ConditionDataset, and NPEDataset.Source: §2 · p. 3

3 STUDY DESIGN

• Extensive use of Defects4J in prior evaluations motivates the study’s focus on performance across other bug benchmarks.Source: §3 · p. 3
• The study design includes research questions and the systematic selection of repair tools and bug benchmarks.Source: §3 · p. 3
• The design also covers data collection and analysis.Source: §3 · p. 3

3.1 Research Questions

• RQ1 asks to what extent test-suite-based repair tools generate patches for bugs across diverse benchmarks.Source: §3.1 · p. 3
• RQ2 asks whether tools repair bugs from benchmarks beyond Defects4J to a similar extent as bugs from Defects4J.Source: §3.1 · p. 3
• RQ3 asks what causes repair attempts to produce no patch, an unsuccessful case that prior evaluations have not investigated, to identify practical limitations.Source: §3.1 · p. 3

3.2 Subject Repair Tools

• 11 tools met the study’s criteria: public availability, runnability, support for multiple benchmarks, and use of only source code and tests.Twelve tools initially met the criteria, but ssFix was excluded because of execution issues; the final set covered all three approach categories.Source: §3.2 · p. 3
• GenProg implementations reuse existing code at the statement level, while Kali tools remove statements, alter conditions, or insert returns.GenProg’s statement operations include insertion, removal, and replacement.Source: §3.2 · pp. 3–4
• jMutRepair mutates relational, logical, or unary operators in suspicious if-condition statements.Source: §3.2 · p. 4
• Nopol uses runtime angelic values and an SMT formula to synthesize expressions, while DynaMoth builds expressions from runtime variables and method calls.Source: §3.2 · p. 4
• NPEFix activates different repair strategies across executions of a failing test, then uses successful strategies and their context to create a patch.Source: §3.2 · p. 4
• ARJA combines search-space representation, multi-objective optimization, and scope matching; Cardumen substitutes context-compatible templates, and RSRepair-A applies random search.RSRepair found patches faster than GenProg in 23 of 24 cases in its reported comparison.Source: §3.2 · p. 4

3.3 Subject Benchmarks of Bugs

• The five benchmarks met criteria requiring Java bugs, peer-reviewed publication, and at least one failing test case per bug.Source: §3.3 · p. 4
• Defects4J contains 395 bugs from six projects, averaging 129,592 lines of Java code per project; Bugs.jar contains 1,158 bugs from eight Apache projects, averaging 212,889 lines.Source: §3.3 · p. 4
• Bears contains 251 bugs from 72 GitHub projects, averaging 62,597 lines of code, and has the largest project diversity among prior bug benchmarks.Source: §3.3 · p. 4
• IntroClassJava contains 297 bugs from six student projects, transpiled from C, with projects averaging 230 lines of code.Source: §3.3 · p. 5
• QuixBugs contains 40 single-line bugs across 40 algorithm programs averaging 190 lines each, and provides Java and Python versions.Source: §3.3 · p. 5

3.4 Building a Repair Execution Framework

• RepairThemAll separates repair-tool plug-ins, benchmark plug-ins, and a runner that executes a selected tool on a specific bug.The framework abstracts both tools and benchmarks so additional ones can be added or removed.Source: §3.4 · p. 5
• The tool plug-in maps eight common execution parameters to tool-specific names and formats, while allowing extra tool parameters for tuning.Source: §3.4 · p. 5
• Benchmark plug-ins provide bug checkout, compilation, and tool-input information; only Defects4J fully supplied the required operations, so the other four needed additions.Source: §3.4 · p. 5
• The runner produces a log and normalized JSON containing generated-patch locations and textual differences; the framework supports 11 tools and five benchmarks and is extensible.Executions can also be configured with settings such as timeouts and patch limits.Source: §3.4 · p. 5

3.5 Data Collection and Analysis

• RepairThemAll executions supplied patches for analysis, while the study’s analysis stage investigated possible causes of attempts that generated no patch.Source: §3.5 · p. 5

3.5.1 Repair tools’ setup.

• Each repair attempt had a two-hour time budget, and tools were configured to stop after generating one patch when supported; the study used one predefined random seed.Five tools lacked the stop-after-one-patch option and ran until their own budget was consumed or timeout occurred.Source: §3.5.1 · p. 5
• 11 tools ran on 2,141 bugs from 130 projects, totaling 23,551 repair attempts and 314 days and 12.5 hours of combined execution.The experiment used computing resources from Grid’5000.Source: §3.5.2 · p. 6

3.5.2 Large scale execution.

• The analysis examined sampled attempt logs because manually reviewing all 23,551 was impractical and repeated behaviors made a predefined sample inefficient.The researchers manually identified major causes because automatic detection was unavailable; they observed shared issues, such as fault-localization problems across tools on a project’s bugs.Source: §3.5.3 · p. 6

4.1 [RQ1] Repairability of the 11 Repair Tools

• Overlap was greater among tools sharing a patch-generation framework; ARJA overlapped 45% with GenProg-A, 56% with Kali-A, and 55% with RSRepair-A.Tools in the Astor framework also overlapped substantially with Arja-framework tools that share similar repair approaches.Source: §4.1 · p. 6, Table 5 · p. 7
• The tools patched 15–213 of 2,141 bugs, and 10/11 repaired at least one unique bug; NPEFix patched 15, consistent with its null-pointer-only scope.The tools’ overlapping repairability was affected by similarities in their approaches and implementation frameworks.Source: §4.1 · p. 6, Figure 2 · p. 6

4.2 [RQ2] Benchmark Overfitting

• The authors propose technical problems, Defects4J bug-fix isolation, and differing bug-type distributions as possible explanations; further studies are needed to identify which contribute.They consider tool technical problems more plausible, based on their observation that implementation affects repairability.Source: §4.2 · p. 7
• 47.34% of Defects4J bugs were patched (187 unique patched bugs); QuixBugs and IntroClassJava had 30% and 20.87% patched, and all tools showed significantly higher repairability on Defects4J.Chi-square tests found the number of patched bugs by each tool depended on Defects4J.Source: §4.2 · p. 7, Table 6 · p. 8

4.3 [RQ3] Causes of Non-patch Generation

• 56.49% of repair attempts ended in error or timeout, with Bugs.jar the main contributor; NPEFix had the highest error rate, while jGenProg and Cardumen were more timeout-prone.The authors identify Bugs.jar project size and complexity as a limitation for current automatic patch-generation tools.Source: §4.3 · p. 8, Table 7 · p. 8, Table 8 · p. 8
• Six causes of non-patch generation were identified: limited repair scope, incorrect fault localization, multiple fault locations, insufficient time, incorrect configuration, and other technical issues.Examples include missing repair ingredients, misleading failing tests, and command lines too long for the operating system to execute.Source: §4.3 · pp. 8–9

Diversity of program repair benchmarks

• All 11 tools perform significantly better on Defects4J than on other benchmarks, threatening generalization from Defects4J-only evaluations.The authors recommend evaluating future tools on diverse benchmarks to mitigate this external-validity threat.Source: §5 · p. 9

Impact of the repair tools’ engineering on the repairability

• With the same input, Kali-A fixes 118 bugs while jKali fixes 52, despite sharing an approach, showing implementation differences affect observed repairability.The authors caution that comparisons measure tool effectiveness, not necessarily the repair approaches themselves.Source: §5 · p. 9

Challenges of creating RepairThemAll

• RepairThemAll required screening tools for availability, manual source analysis to understand diverse or undocumented tools, and integrating them into a common framework.The authors attribute these challenges mainly to the lack of well-organized open-science repositories.Source: §5 · p. 9

The observed repairability compared to the previous evaluations

• Prior Defects4J evaluations are difficult to compare because they did not include all benchmark bugs; only Cardumen fixed fewer bugs in this study.Differences in setup, including random seeds, and possible bugs in the Cardumen version may explain the result; no major drawbacks in RepairThemAll’s configurations were observed.Source: §5 · p. 9

Threats to validity

• A bug in RepairThemAll could affect reported results, although the framework and raw data are publicly available for validity checks.This is an implementation threat, not a reported finding that the results are affected.Source: §5 · p. 9
• The study evaluates test-suite-adequate patches, which pass the test suite but are not guaranteed to fix bugs; patch correctness is out of scope.The authors state that their analysis of tool behavior and potential flaws does not require patch-correctness knowledge.Source: §5 · p. 10
• The study includes 11 of 24 identified Java repair tools, leaving its intended full picture incomplete and limiting external validity.Many omitted tools could not be run; PAR is given as an example because it is unavailable.Source: §5 · p. 10

• C-language studies examined plausible versus correct patches, test-suite overfitting, and factors affecting overfitting in semantics-based repair.The studies described report few correct patches and overfitting even with high-quality, high-coverage tests.Source: §6 · p. 10
• Java studies examined correct patches on Defects4J, incorrect patches on QuixBugs, and whether repaired bugs are hard or important.Martinez et al. reported 9/47 bugs with test-suite-adequate patches that were also correct.Source: §6 · p. 10
• Unlike earlier studies focused on patch overfitting or correlations, this study investigates benchmark overfitting and non-patch generation at a larger scale.The comparison is with prior empirical studies on repair tools.Source: §6 · p. 10

• The experiment covers 11 tools, 2,141 bugs, and 23,551 repair attempts across five benchmarks containing bugs from 130 projects.The benchmarks collect bugs using different strategies.Source: §7 · p. 10
• The tools repaired bugs from benchmarks outside their original evaluations, but results suggest all tools overfit Defects4J; the study also identifies six causes of non-patch generation.The authors say these causes can help future repair-tool development.Source: §7 · p. 10
• Further work could test hypotheses for better Defects4J performance, run additional tools, adapt Defects4J-hardcoded tools, and examine benchmark bug-type distributions.One proposed hypothesis is that Defects4J buggy versions were changed through bug-fix isolation; confirming it would be a full contribution.Source: §7 · p. 10