Source-linked AI summary
Automatic Repair of Real Bugs in Java: A Large-Scale Experiment on the Defects4J Dataset
Matias Martinez, Thomas Durieux, Romain Sommerard, Jifeng Xuan, Martin Monperrus
TL;DR
Automatic repair needs stronger empirical evidence on whether test-suite based methods can repair real Java bugs correctly, not merely satisfy tests. The paper evaluates three repair systems on Defects4J and finds test-suite adequate patches for 47 of 224 bugs, while identifying overfitting and under-specified bugs as central challenges.
Problem
Few empirical evaluations had tested the practical ability of current repair techniques to repair real bugs, and test-suite adequacy does not establish correctness beyond the tests.
Method
The paper runs jGenProg, jKali, and Nopol on Defects4J bugs and manually analyzes generated patches for correctness and limitations.
Results
47 out of 224 bugs received test-suite adequate patches, while most manually analyzed patches were incorrect because they overfit test data.
Takeaways & Limitations
The results identify avoiding test overfitting and reasoning about expected behavior beyond the test suite as major challenges for automatic repair.
Takeaways & Limitations
The study analyzes only the first patch found per repair attempt, although multiple patches may exist and a correct patch may remain unanalyzed.
Abstract
from arXiv · showhide
Defects4J is a large, peer-reviewed, structured dataset of real-world Java bugs. Each bug in Defects4J comes with a test suite and at least one failing test case that triggers the bug. In this paper, we report on an experiment to explore the effectiveness of automatic test-suite based repair on Defects4J. The result of our experiment shows that the considered state-of-the-art repair methods can generate patches for 47 out of 224 bugs. However, those patches are only test-suite adequate, which means that they pass the test suite and may potentially be incorrect beyond the test-suite satisfaction correctness criterion. We have manually analyzed 84 different patches to assess their real correctness. In total, 9 real Java bugs can be correctly repaired with test-suite based repair. This analysis shows that test-suite based repair suffers from under-specified bugs, for which trivial or incorrect patches still pass the test suite. With respect to practical applicability, it takes on average 14.8 minutes to find a patch. The experiment was done on a scientific grid, totaling 17.6 days of computation time. All the repair systems and experimental results are publicly available on Github in order to facilitate future research on automatic repair.
1 Introduction
The paper evaluates test-suite based repair on real Java bugs to assess its practical effectiveness, correctness beyond tests, under-specification, and execution time. Across 224 Defects4J bugs, the systems generated test-suite adequate patches for 47 bugs, while manual analysis exposed substantial overfitting.
- Motivation: Few empirical evaluations had assessed whether current automatic repair techniques could practically repair real bugs.The study targets Java bugs in large-scale applications rather than toy or seeded programs.
- Dataset and systems: Defects4J provides 357 peer-reviewed, publicly available, structured real-world Java bugs with passing and failing tests.The experiment used four projects and selected publicly available systems supporting modern Java and large applications.
- Practicality: The study ran for over 17.6 days of computational time, while patch search for an individual bug took a matter of minutes.The authors present this execution profile as evidence relevant to practical applicability.
- Findings: 47 out of 224 bugs received test-suite adequate patches from jGenProg, jKali, or Nopol, yielding 84 different patches.Twelve of the 47 bugs were handled by all three approaches.
- Findings: Manual analysis found 11 of 84 patches correct, 61 incorrect, and 12 requiring domain expertise.Incorrect patches tended to overfit test cases, indicating limitations in test suites or repair techniques.
- Open science: The authors make the repair systems, code, and experimental data publicly available to support reproducible future research.The materials are hosted on Github.
2 Background
The paper frames test-suite based repair as patch synthesis guided by program tests and fault localization, and uses Defects4J as a realistic, large-scale Java benchmark. The approach evaluates whether generated patches make the complete test suite pass, without establishing correctness beyond that suite.
- Automatic repair: Automatic repair can operate offline by generating source patches or online by modifying runtime state to overcome failures.The paper focuses on offline, test-suite based repair.
- Test-suite based repair: Test-suite based repair takes a buggy program and its test suite as input and outputs a patch that makes the whole suite pass, if any.Fault localization ranks suspicious statements before patch generation modifies them.
- Defects4J: Defects4J contains 357 real-world bugs from five widely used open-source Java projects in a unified structure for programs, tests, and patches.The benchmark supports reproducible and controlled software-testing studies.
- Defects4J: The experiment excludes Closure Compiler because its script-based tests cannot run on the study platform as standard JUnit tests.The remaining considered projects are Commons Lang, JFreeChart, Commons Math, and Joda-Time.
- Benchmark rationale: Defects4J offers realism, scale, and novelty by containing real bugs in large software and enabling a new repair evaluation.These properties distinguish it from seeded bugs and student programs.
3 Experimental Protocol
The experimental protocol assesses automatic repair across test-suite adequacy, patch correctness, under-specified bugs, and performance. It is designed for systematic evaluation of repair approaches on real-world Defects4J bugs.
- Protocol scope: The protocol evaluates four dimensions: test-suite adequacy, patch correctness, under-specified bugs, and performance.The study first defines research questions, then describes experiment design and implementation details.
- Dataset scope: The protocol uses Defects4J project and bug data while excluding Closure Compiler because its tests use non-conventional scripts rather than standard JUnit cases.This platform constraint leaves Closure Compiler for future work.
3.1 Research Questions
The research questions examine whether repair systems can patch real Defects4J bugs, whether patches are correct beyond test-suite passing, which bugs are under-specified, and how long repair takes. The protocol treats test-suite adequacy as necessary but not sufficient for correctness.
- RQ1: Test-suite adequacy: The study counts bugs for which each repair system synthesizes a patch that makes the entire test suite pass.This is the basic evaluation criterion for test-suite based repair.
- RQ2: Patch correctness: The study manually examines generated patches to determine whether they are semantically correct beyond passing the test suite.A test-suite-passing patch may differ syntactically from the developer patch while remaining correct, or may be incorrect.
- RQ3: Under-specified bugs: An under-specified bug has expected behavior encoded by tests with low coverage and weak assertions.The study identifies such bugs using jKali results and manual analysis of generated patches.
- RQ4: Execution time: The study evaluates each repair approach's execution time to assess whether automatic repair is usable in practice.Manual repair is described as time-consuming, motivating this performance question.
3.2 Experiment Design
The experiment evaluates three Java repair systems on Defects4J, combining large-scale execution with manual patch assessment. It also documents execution constraints, analysis procedures, and limits on reproducibility and randomized search.
- Repair systems: Three publicly available Java repair systems—jGenProg, jKali, and Nopol—were selected using language, version-support, and availability criteria.jGenProg implements GenProg, jKali implements Kali’s code-removal strategy, and Nopol targets conditional bugs through SMT-based synthesis.
- Repair systems: jGenProg searches by randomly deleting, adding, and replacing AST nodes, with fault localization steering modification points and inserted code drawn from the same program.This design follows the redundancy hypothesis, while the paper notes a threat that the Java implementation may not fully reflect the original algorithm.
- Repair systems: jKali removes or skips code to expose weak test suites and under-specified bugs, whereas Nopol modifies conditions or adds guards for conditional faults.Nopol synthesizes conditions using input-output-based code synthesis with SMT solvers.
- Open science: The authors’ ownership of all three systems supports comparable system knowledge but limits the current practice of open and reproducible Java-repair research.The paper contrasts this situation with the broader goal of publicly sharing tools and results.
- Large-scale execution: The study assesses the three systems on 224 bugs using Grid’5000, with standardized hardware intended to avoid bias in time-cost measurement.The execution required 17.6 days of computation, and each repair attempt had a three-hour timeout.
- Large-scale execution: For all but five bugs, randomized jGenProg was run once, producing a conservative underestimation of effectiveness; each attempt stopped after the first patch.The authors identify this as an important threat to validity, while noting that found patches remain test-suite adequate.
- Patch analysis: Generated patches were manually assessed for correctness, readability, and validation difficulty through analyst review followed by live cross-validation with another co-author.Correctness meant being identical or semantically equivalent to the developer patch, based on the analyst’s understanding; assessment could take minutes to hours.
4 Empirical Results
The three repair systems generated test-suite adequate patches for 47 of 224 Defects4J bugs, but manual analysis found substantial overfitting and under-specification. Patch generation was practically feasible, while correctness beyond test-suite satisfaction remained limited.
- Test-suite Adequate Repair: 35 bugs were patched by Nopol, compared with 27 by jGenProg and 22 by jKali.Nopol also handled 18 bugs that neither jGenProg nor jKali handled.
- Research Contribution: The study provides an initial Defects4J benchmark result and releases implementations, experimental code, and results publicly for future comparisons.The authors describe these as the first automatic-repair results on Defects4J.
- Test-suite Adequate Repair: 47 of 224 bugs received test-suite adequate patches from at least one repair system.Together, jGenProg, jKali, and Nopol generated 84 patches for these bugs.
- Patch Correctness: Incorrect patches commonly removed or bypassed faulty behavior, exploited under-specified functionality, or overfit test data.For example, a patch may specialize behavior to the 2 × 2 matrices used by a failing test.
- Patch Correctness: Only 11 of 84 analyzed patches were semantically correct beyond passing the provided test suite.The assessment classified 61 patches as incorrect and 12 as unknown because domain expertise was unavailable.
- Under-specified Bugs: 21 bugs were identified as under-specified because their test suites did not accurately specify expected behavior and allowed trivial code removal.Repairing these bugs requires reasoning about functionality beyond what is encoded in the test suite.
- Performance: Test-adequate patches were generally found within minutes, while the full experiment consumed 17.6 days of computation.Both the median and average repair execution times were about one hour on server-like machines, and larger timeouts did not improve effectiveness in this experiment.
5 Case Studies
Three case studies show that automatic repair can generate correct patches for real Defects4J bugs, including developer-equivalent patches, but weak tests also permit clearly incorrect patches.
- Case Studies: Automatic repair generated correct patches for real bugs, including patches equivalent to developers’ manually written fixes.The case studies cover jGenProg, jKali, and Nopol, and report developer-equivalent patches for M70 and L55.
- M70: jGenProg repaired M70 by adding the missing parameter to an overloaded method call, exactly matching the manual patch.jGenProg replaced the call with another call selected from the same class; jKali and Nopol could not repair this bug.
- M8: jKali’s M8 patch passed all tests by removing the assignment, returning an empty array instead of fixing the array-type incompatibility.The single test assertion checked only array size, not returned contents, making the incorrect patch test-suite adequate; jGenProg found a semantically equivalent removal.
- M8: M8 exemplifies an under-specified bug because a trivial functionality-removing patch passes tests that omit assertions on the returned array’s contents.The developer fix changes the array declaration and instantiation from T[] to Object[].
- L55: Nopol correctly repaired L55 with a precondition equivalent to the developer’s condition, while jGenProg and jKali could not repair it.The two conditions are equivalent because running timers use stop time −1, whereas suspended timers store a millisecond stop time.
- Summary: Together, the cases demonstrate that automatic repair handles real bugs, but weak tests can yield patches that are test-suite adequate yet clearly incorrect.The examples contrast correct repairs with patches that merely satisfy the available test cases.
6 Discussion
The discussion identifies threats to validity involving benchmark scope, reimplementation fidelity, manual patch assessment, randomness, and the choice to inspect only the first generated patch.
- External Validity: The study covers four subject programs from a benchmark not designed to represent fault classes, limiting external validity.The authors call for experiments on additional benchmarks.
- Benchmark Setup: The unchanged Defects4J tests avoid using information from after bug reports and reduce potential experimental bias.The design measures repair effectiveness against the benchmark as it existed when bugs were reported.
- Implementations: jGenProg and jKali are Java reimplementations, so their results may differ from those of the original GenProg and Kali systems.The implementations were intended to be faithful and were released publicly for peer review.
- Assessment: Manual correctness assessment threatens internal validity because one author performed the initial analysis and no independent inter-annotator agreement study was conducted.A second author reviewed and validated the results, but the methodological setup remained a pilot study.
- Multiple Patches: Stopping after the first patch means a correct patch elsewhere among multiple generated patches may remain unanalyzed.The authors identify patch ordering as a needed research direction because manually analyzing every patch would require months.
6.2 Impact of Flaky Tests on Repair
A flaky test can distort automatic repair in either direction: falsely validating an incorrect patch or falsely rejecting an effective patch.
- Flaky Tests: A flaky failing test may make a repair system accept a patch as correct when it is not actually correct.The system can mistake nondeterministic test behavior for successful repair.
- Flaky Tests: A flaky passing test may make the system reject a patch as regressive, underestimating repair effectiveness.The issue arises when nondeterministic failures appear after a patch is applied.
6.3 Reflections on GenProg
The GenProg results suggest that its core idea remains useful, but effective repair should combine complementary techniques because no technique is universally appropriate for an unknown bug cause.
- Prior Evaluations: 55/105 GenProg bugs were test-suite adequately repaired in one evaluation, whereas another judged only 2/105 correctly repaired.The difference was attributed to experimental threats and under-specified bugs.
- jGenProg: jGenProg correctly repaired 5/224 Defects4J bugs and uniquely repaired 4, including M70.The authors caution that benchmark inclusion criteria may explain differences in repair rates and that neither benchmark reflects all bug kinds and difficulties.
- Implications: Correctly and uniquely repaired bugs indicate that GenProg’s core intuition is valid and should contribute to integrated repair tools.The proposed direction is to combine multiple repair techniques.
- Technique Selection: Because a new bug’s root cause may involve an incorrect condition, a missing statement, or something else, choosing one repair technique in advance is impractical.The authors suggest running available techniques sequentially or in parallel, while exploring failure-based root-cause identification.
7 Related Work
Prior work established diverse repair techniques and benchmarks, but this study positions Defects4J as a larger, more structured evaluation setting for real Java bugs.
- Real-world datasets: Defects4J is presented as a well-organized benchmark of real bugs and test suites from five widely-used Java projects.The paper describes it as the first evaluation of automatic repair techniques via Defects4J.
- Repair techniques: GenProg applies genetic programming to ASTs, while PAR uses human-derived patch patterns to generate readable patches.These methods represent evolutionary and pattern-based repair strategies.
- Repair techniques: Mutation-based and constraint-based approaches explore alternative repair spaces, including statement removal, symbolic synthesis, and condition repair.Kali examines simple removals, while SemFix and Nopol synthesize repairs for assignments, conditions, and preconditions.
- Repair settings: Other repair settings target contracts, atomicity violations, runtime errors, or domain-specific faults rather than general test-suite-based repair.These approaches broaden automatic repair beyond the setup evaluated in this paper.
- Empirical foundations: Empirical studies have examined patch maintainability, human debugging assistance, repair-model mining, redundancy, and patch characteristics.This research also includes analyses of machine-generated patches and thousands of manually written patches.
- Evaluation scope: Earlier comparative evaluations used seeded and smaller real-bug benchmarks, whereas this experiment evaluates larger and more complex Java applications.The cited comparison included 119 seeded bugs and 34 real bugs from Siemens and SIR benchmarks.
8 Conclusion
The evaluation found that three repair systems generated patches for 47 of 224 real Java bugs, but manual analysis showed that most patches overfit the test data. The results identify avoiding overfitting, reasoning beyond tests, and strengthening test suites as central challenges.
- Conclusion: 47 of 224 real Java bugs received patches from the three evaluated automatic repair systems.Because Defects4J contains real bugs from large-scale Java software, the result provides evidence about practical applicability.
- Conclusion: Most generated patches were incorrect beyond test-suite satisfaction because they overfit the test data.The paper reports this overfitting in Java software with state-of-the-art testing infrastructure and in Nopol as well as GenProg-like systems.
- Future challenges: The field must produce repair systems that avoid overfitting and reason about expected behavior beyond what tests directly encode.The paper identifies these as two grand challenges.
- Future challenges: Automatically enriching test suites with stronger assertions could prevent synthesis of incorrect patches.The paper identifies test-suite research as another need with direct impact on repair.