Source-linked AI summary
Practical Program Repair via Bytecode Mutation
Ali Ghanbari, Lingming Zhang
TL;DR
Automated program repair seeks to reduce the substantial effort of debugging, but simple mutation-based repair has not been thoroughly evaluated on real-world bugs. The paper implements PraPR, a JVM bytecode-mutation repair tool, and studies it on Defects4J. PraPR generates genuine patches for 43 bugs with augmented mutators and operates an order of magnitude faster than state-of-the-art techniques.
Problem
Mutation-based automated program repair has not been thoroughly studied on real-world bugs, despite APR's goal of fixing bugs with minimal human intervention.
Method
PraPR performs mutation-like patch generation at the JVM bytecode level and validates generated candidates using test-based generate-and-validate repair.
Results
PraPR generates genuine patches for 43 Defects4J bugs and is an order of magnitude faster than state-of-the-art APR techniques.
Takeaways & Limitations
Bytecode mutation provides a practical, simple baseline that can complement state-of-the-art APR in effectiveness, efficiency, and applicability.
Takeaways & Limitations
The experimental findings may not generalize beyond the subject systems used, and test-based validation remains vulnerable to APR overfitting.
Abstract
from arXiv · showhide
Software debugging is tedious, time-consuming, and even error-prone by itself. So, various automated debugging techniques have been proposed in the literature to facilitate the debugging process. Automated Program Repair (APR) is one of the most recent advances in automated debugging, and can directly produce patches for buggy programs with minimal human intervention. Although various advanced APR techniques (including those that are either search-based or semantic-based) have been proposed, the simplistic mutation-based APR technique, which simply uses pre-defined mutation operators (e.g., changing a>=b into a>b) to mutate programs for finding patches, has not yet been thoroughly studied. In this paper, we implement the first practical bytecode-level APR technique, PraPR, and present the first extensive study on fixing real-world bugs (e.g., Defects4J bugs) using bytecode mutation. The experimental results show that surprisingly even PraPR with only the basic traditional mutators can produce genuine patches for 18 bugs. Furthermore, with our augmented mutators, PraPR is able to produce genuine patches for 43 bugs, significantly outperforming state-of-the-art APR. It is also an order of magnitude faster, indicating a promising future for bytecode-mutation-based APR.
1 INTRODUCTION
PraPR studies whether simple bytecode mutation can provide a practical alternative to more complex automated program repair techniques. On Defects4J, it combines broad applicability and fast validation with substantial repair results.
- Motivation: Automated Program Repair aims to fix software bugs with minimal human intervention, addressing the high cost of manual debugging.APR can synthesize genuine patches or suggest patches that help developers repair bugs faster.
- Background: Existing APR techniques include dynamic-state healing and generate-and-validate approaches that use tests or formal specifications to validate candidate patches.Generate-and-validate methods are especially popular because testing is prevalent, whereas rigorous formal specifications are uncommon.
- Approach: PraPR is the first practical Java bytecode-level APR technique, generating patches that can be validated directly without compilation.Bytecode manipulation also avoids unexpected source changes, can target code without source information, and applies across JVM-based languages.
- Results: 43 Defects4J bugs received genuine patches with augmented mutators, while single-threaded PraPR was 26.1X faster per patch than CapGen and 15.7X faster than JAID.Using only basic traditional mutators, PraPR produced genuine patches for 18 bugs and plausible patches for 113 bugs.
- Study scope and implications: The study evaluates PraPR across all 395 Defects4J bugs and argues that mutation-based repair can complement state-of-the-art APR in effectiveness, efficiency, and applicability.The paper presents these findings as guidelines for advancing bytecode-mutation-based APR.
2 RELATED WORK
This section introduces mutation testing and generate-and-validate APR, then positions mutation-based repair as feasible but insufficiently studied on real-world bugs. It also outlines PraPR’s bytecode-level approach and test-based validation process.
- 2.1 Mutation Testing: Mutation testing evaluates test-suite quality by injecting artificial bugs through small, single-pointed program transformations called mutators.Applying a mutator produces a syntactically valid, typeable mutant intended to mimic a real bug.
- 2.2 Generate-and-Validate Program Repair: Generate-and-validate APR localizes suspicious code, generates candidate patches, and uses tests as oracles to identify plausible repairs.A plausible patch passes all original tests, whereas a genuine patch is also semantically equivalent to the programmer-written fix.
- 2.2 Generate-and-Validate Program Repair: APR overfitting means that passing the available test suite does not guarantee a patch generalizes to other potential tests.Consequently, plausible patches are not necessarily genuine patches.
- Related Mutation-Based APR: Existing mutation-based APR studies mainly evaluated small programs with artificial bugs or applied only limited mutation operators.This motivates a broader study on real-world benchmark bugs.
- PraPR: PraPR’s validation pipeline generates mutants from a buggy program and filters candidates using failing and passing tests before returning a plausible patch set.Validation continues or aborts when tests expose an invalid candidate.
- PraPR: PraPR performs bytecode-level mutation for Java, allowing generated patches to be validated without compilation and supporting larger search spaces on commodity hardware.Its bytecode-level design also supports JVM languages including Kotlin, Groovy, and Scala.
3 PRAPR
PraPR applies simple, single-statement mutation rules directly to JVM bytecode, generating and validating candidate patches without recompilation. Its mutators include traditional PIT operators and augmented transformations designed for broader repair coverage.
- Overall approach: PraPR applies mutation testing to a buggy program and its tests, producing plausible patches that pass the full test suite for developer inspection.Candidates are ranked by suspiciousness, tested first against failing tests, and then against the remaining tests only if they pass.
- Patch presentation: Bytecode-level patch reports provide debugging information, and PraPR can decompile mutated bytecode to show patches with source-code line numbers.These reports are intended to give developers enough information to confirm, reject, and apply patches to source code.
- Implementation: PraPR operates on JVM bytecode, using ASM, Java Agent, and PIT to support on-the-fly mutation, execution, coverage collection, and test-execution optimizations.The implementation supports the full set of JVM instructions and data types while targeting Maven-based Java and Kotlin projects.
- Presentation language: ClassicJava is used as a minimal presentation language for describing PraPR’s mutators and their transformations.The paper presents the language syntax and refers to prior work for its type rules and informal construct descriptions.
- Mutator design: Candidate patches are generated by mutating one expression instance while leaving all other program portions unchanged.The mutator rules use premises and type information to determine valid transformations.
- Mutator design: The mutator catalog augments PIT with broader arithmetic, conditional, field-access, and method-invocation transformations.Examples include arithmetic-operation deletion, additional conditional replacements, field-name alternatives, and method-name alternatives.
4 EXPERIMENTAL SETUP
The study evaluates PraPR’s effectiveness, efficiency, and comparison with state-of-the-art repair techniques on real-world bugs from the Defects4J benchmark. The setup uses six GitHub programs with reproducible bugs, their tests and code-size characteristics, and PraPR’s JVM-bytecode implementation.
- Research questions: The experiment addresses PraPR’s effectiveness, efficiency, and performance relative to state-of-the-art automated program repair techniques.These goals are stated as research questions RQ1, RQ2, and RQ3.
- Subject systems: Defects4J contains six real-world GitHub programs with known, reproducible bugs and varied programming idioms.The benchmark is widely used in peer-reviewed automated program repair research.
- Subject systems: The subject-system table reports the number of JUnit test methods and lines of code for each program’s HEAD buggy version.These values describe the test and code-size characteristics used in the benchmark setup.
- Implementation: PraPR supports Maven-based Java and Kotlin projects with JUnit or TestNG test suites and uses ASM, Java Agent, Ochiai fault localization, and PIT.The tool can be run with a single Maven command and was evaluated using both single-threaded and four-threaded execution.
5 RESULT ANALYSIS
PraPR fixes real Defects4J bugs using simple bytecode mutation, with augmented mutators substantially expanding repair coverage and bytecode-level validation enabling high efficiency. Its genuine patches often rank highly, while broad exploration also yields more non-genuine plausible patches than competing techniques.
- PraPR Effectiveness: 113 plausible and 18 genuine patches were produced using only the original PIT mutators on Defects4J.These results were comparable to CapGen’s 22 genuine fixes and exceeded prior source-level mutation results.
- PraPR Effectiveness: 148 plausible and 43 genuine patches were produced using all simple PraPR mutators, the largest reported Defects4J repair count at the time.The authors attribute this coverage partly to bytecode-level generation and validation, which allow rapid exploration of many potential patches.
- PraPR Effectiveness: PraPR generated genuine patches semantically equivalent to developer patches, including fixes produced by CONDITIONAL and RETURN VALUE mutators.One CONDITIONAL patch forces a subclass method to invoke its superclass implementation, semantically removing the overriding method.
- PraPR Efficiency: PraPR validated 29,850 Closure patches in under one hour with a single thread, while four threads provided a 2.1X speedup for Closure.The tool’s bytecode-level operation avoids recompilation and disk-based class or test loading for generated patches.
- PraPR Efficiency: Genuine patches ranked 2076.4th among 9696.5 validated patches on average with all mutators, placing them in the top 21.4% of the search space.The authors report that Ochiai spectrum-based fault localization provided effective ranking despite the absence of mining or learning information.
- Comparison with the State-of-Art: PraPR fixed 43 bugs across all plausible patches, 41 within the top 10, and 23 at the top 1, while also fixing bugs no other evaluated tool fixed.It fixed 16 bugs not fixed by either CapGen or JAID, and 13 not fixed by any technique in the broader comparison.
6 CONCLUSION
PraPR is a practical bytecode-level mutation approach for automatic program repair. On Defects4J, it generated genuine patches for 43 bugs, outperformed state-of-the-art techniques, and was an order of magnitude faster.
- PraPR performs mutation-like patch generation at the JVM bytecode level.
- 43 Defects4J bugs received genuine patches when PraPR used all its mutators.
- PraPR significantly outperformed state-of-the-art APR techniques on the Defects4J benchmark.
- PraPR was an order of magnitude faster than state-of-the-art APR techniques.
- Because PraPR requires neither learning nor mining, it can serve as a baseline for future repair techniques.