PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast and Precise On-the-fly Patch Validation for All

Lingchao Chen, Lingming Zhang

arXiv:2007.11449v1cs.SE

TL;DR

Test-based G&V APR incurs the costs of generating many patches and repeatedly running tests, while PraPR’s bytecode-level approach limits flexibility. UniAPR generalizes on-the-fly validation to source- and bytecode-level APR and resets JVM state between patches. The authors report over an order of magnitude speedup without imprecise validation and describe hybrid APR as a way to fix more bugs under the same time limit.

  • Problem

    Test-based G&V APR requires patch generation and repeated test execution, while PraPR’s bytecode-level approach limits flexibility.

    Source: §1 · p. 1

  • Method

    UniAPR validates source- and bytecode-level patches on the same JVM and resets JVM state after each patch execution.

    Source: §1 · p. 2, §2.2 · p. 3, §3.3.3 · p. 7

  • Results

    UniAPR speeds state-of-the-art APR by over an order of magnitude without incurring imprecision in patch validation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    UniAPR directly enables hybrid source- and bytecode-level APR to fix substantially more bugs than state-of-the-art techniques under the same time limit.

    Source: §1 · p. 2

    With vanilla on-the-fly validation, a patch that writes global state can affect later patches that read it, making their validation results imprecise.

    Source: §3.3.1 · p. 5

Abstract

from arXiv · show

Generate-and-validate (G&V) automated program repair (APR) techniques have been extensively studied during the past decade. Meanwhile, such techniques can be extremely time-consuming due to manipulation of the program code to fabricate a large number of patches and also repeated executions of tests on patches to identify potential fixes. PraPR, a recent G&V APR technique, reduces these costs by modifying program code directly at the level of compiled bytecode, and further performing on-the-fly patching by allowing multiple patches to be tested within the same JVM session. However, PraPR is limited due to its pattern-based, bytecode-level nature and it is basically unsound/imprecise as it assumes that patch executions do not change global JVM state and affect later patch executions on the same JVM session. Inspired by the PraPR work, we propose a unified patch validation framework, named UniAPR, which aims to speed up the patch validation for both bytecode and source-code APR via on-the-fly patching; furthermore, UniAPR addresses the imprecise patch validation issue by resetting the JVM global state via runtime bytecode transformation. We have implemented UniAPR as a fully automated Maven Plugin. We have also performed the first study of on-the-fly patch validation for state-of-the-art source-code-level APR. Our experiments show the first empirical evidence that vanilla on-the-fly patch validation can be imprecise/unsound; in contrast, our UniAPR framework can speed up state-of-the-art APR by over an order of magnitude without incurring any imprecision in patch validation, enabling all existing APR techniques to explore a larger search space to fix more bugs in the near future. Furthermore, UniAPR directly enables hybrid source and bytecode APR to fix substantially more bugs than all state-of-the-art APR techniques (under the same time limit) in the near future.

1 INTRODUCTION

Test-based generate-and-validate repair is costly, and PraPR’s bytecode-level, shared-JVM approach has flexibility and precision limitations. UniAPR generalizes on-the-fly validation across source and bytecode APR while resetting JVM state; experiments report faster validation without imprecision.

1 INTRODUCTION

  • Test-based G&V APR repeatedly generates patches and runs developer tests, making patch generation and validation major costs for scalability.PraPR reduces these costs by modifying compiled bytecode and reusing JVM sessions across patches.Source: §1 · pp. 1–2
  • PraPR’s bytecode-level nature limits flexibility, while shared JVM state can let one patch’s execution make later validation unreliable.For example, a patch may modify static fields read by a later patch.Source: §1 · p. 2
  • UniAPR reuses one JVM for bytecode- and source-level APR and resets JVM state after each patch execution to address imprecision.It uses HotSwap and Java Agent technology for on-the-fly patching and runtime bytecode transformation for JVM reset.Source: §1 · p. 2
  • UniAPR speeds state-of-the-art source-code APR by over an order of magnitude without imprecise patch validation.The study covers CapGen, SimFix, and ACS.Source: §1 · p. 2
  • UniAPR is a Maven plugin that accepts APR tools as patch-generation add-ons, and the paper reports that it can support hybrid APR.The authors envision hybrid source and bytecode APR fixing more bugs than state-of-the-art techniques under the same time limit.Source: §1 · p. 2

2 BACKGROUND

The background introduces APR and the rationale for PraPR’s on-the-fly validation, then explains Java Agent and HotSwap mechanisms used to reload patched classes within a shared JVM.

2.1 Automatic Program Repair

  • G&V APR generates and validates a pool of patches; PraPR [14] speeds this process through bytecode-level patching and JVM reuse, and this paper generalizes on-the-fly validation beyond bytecode APR.Plausible patches pass checks, but developers must still identify which are correct; PraPR also avoids repeated class loading and JVM warm-up.Source: §2.1 · p. 3

2.2 Java Agent and HotSwap

  • A Java Agent can modify loaded bytecode, and HotSwap lets the agent replace patched classes so tests validate patches within the same JVM.For each later patch, the agent first restores previously patched classes to their original versions.Source: §2.2 · p. 3

3 APPROACH

UniAPR validates source- and bytecode-level patches in a shared JVM, reloading patched classes rather than restarting for every patch. It resets JVM state between executions to address imprecision caused by one patch affecting another.

3.1 Overview

  • UniAPR accepts source-level patches compiled incrementally and bytecode patches directly, then reloads changed classes, runs tests, restores original bytecode, and resets JVM state.The same process is repeated for each patch without restarting the JVM after every execution.Source: §3.1 · p. 4

3.2 Fast Patch Validation via On-the-fly Patching

  • The vanilla validation algorithm tests remaining patches in one JVM, records each result, restores patched bytecode after normal completion, and restarts the JVM after an error.It stops running tests for a patch once a test falsifies it.Source: §3.2 · pp. 4–5

3.3.1 Limitations for vanilla on-the-fly patch validation

  • A patch that writes to a global pollution site can change later patches’ results, making vanilla on-the-fly validation imprecise.For example, P3 changes static field C.f, which P4 later reads; its assertion may fail even if P4 is correct.Source: §3.3.1 · p. 5

3.3.2 Technical challenges

  • Resetting pollution sites is challenging because final static fields cannot be reset directly, dependent fields require their original ordering, and only the JVM can invoke class initializers.Reinitializing classes with pollution sites also requires preserving the original order of dependent class initializers; the authors simulate JVM behavior at the application level rather than customize the JVM.Source: §3.3.2 · pp. 5–6

3.3.3 JVM reset via bytecode transformation

  • UniAPR combines static pollution analysis, runtime bytecode transformation, ordered class reinitialization, and dynamic state reset to prepare a clean JVM for each patch.It identifies mutable static-field pollution sites, reinitializes classes in their original order, resets JDK pollution sites through available APIs, and provides a public interface for project-specific reset content.Source: §3.3.3 · pp. 6–7

4 EXPERIMENTAL SETUP

The evaluation tests whether vanilla on-the-fly validation and JVM-reset UniAPR affect patch-validation cost and precision, using Defects4J and three representative source-level APR tools.

4.1 Research Questions

  • The research questions examine vanilla on-the-fly validation and JVM-reset validation for their cost reduction and precision.Source: §4.1 · p. 7

4.2 Benchmarks

  • Defects4J V1.0.0 supplies real-world bugs, and its statistics describe project bug counts, test counts, and lines of code.The benchmark is described as widely studied for program repair and software debugging.Source: §4.2 · p. 7

4.3 Studied Repair Tools

  • ACS, SimFix, and CapGen represent heuristic-, constraint-, and template-based Java APR tools applicable to Defects4J.The study evaluates bugs reported as fixed by each tool's original paper.Source: §4.3 · p. 7

4.4 Implementation

  • UniAPR is a Maven plugin that loads patches and project inputs from Maven configuration and validates patches in a forked JVM.Its implementation uses over 10K lines of Java code; validation runs originally failing tests before originally passing tests.Source: §4.4 · pp. 7–8

4.5 Experimental Setup

  • The study measures patch-validation speedup and whether UniAPR preserves the repair results of traditional validation.Experiments cover bugs reported as fixed by the original tools.Source: §4.5 · p. 8

5 RESULT ANALYSIS

Vanilla UniAPR substantially speeds up patch validation for the studied source-level APR tools, while the size of the speedup varies across bugs and tools.

5.1.1 Effectiveness.

  • For CapGen, the comparison uses a UniAPR variant that restarts a JVM for each patch because CapGen excludes some expensive tests on certain bugs.The study describes this adjustment as necessary for a fair comparison.Source: §5.1.1 · p. 8
  • 11.9X (CapGen on Math-80), 28.8X (SimFix on Closure-62), and 14.3X (ACS on Math-82) are reported patch-validation speedups.The cited examples compare traditional validation time with vanilla UniAPR while producing the same validation results.Source: §5.1.1 · p. 8
  • Speedups vary across bugs and tools with patch counts, bytecode files loaded, and test execution times among the influencing factors.The study reports a slight slowdown for ACS on one bug.Source: §5.1.1 · p. 8

(c) ACS

For ACS, vanilla UniAPR’s speedup tends to increase with patch count, while vanilla on-the-fly validation produces inconsistent results in 5.41% of studied cases. JVM-reset adds less than 10% overhead for studied tools and restores the same repair results as traditional validation, with limits on side-effect isolation and HotSwap patch support.

(c) ACS

  • For ACS, UniAPR’s speedup tends to be significantly larger for bugs with more patches, indicating scalability with larger patch pools.The reported correlation is significant at the 0.05 level.Source: (c) ACS · p. 9

5.1.2 Precision.

  • Vanilla UniAPR produced imprecise results in 5.41% of studied cases, with three of four inconsistent cases occurring on CapGen.CapGen averaged over 1,400 patches per studied bug, while SimFix averaged around 150.Source: §5.1.2 · p. 9

5.2.1 Effectiveness.

  • JVM-reset added less than 10% overhead across studied tools compared with vanilla UniAPR.For ACS, the reported average overhead was 8.33%; the authors attribute low overhead to reinitializing only classes with pollution sites and to implementation optimizations.Source: §5.2.1 · p. 10

5.2.2 Precision.

  • UniAPR with JVM-reset produced the same repair results as traditional validation and fixed all four bugs that vanilla UniAPR failed to fix.The examples include failures caused by earlier patch executions changing static-field or object state.Source: §5.2.2 · pp. 10–11

5.3 Discussion

  • UniAPR’s reset cannot fully isolate operating-system or network side effects, while HotSwap excludes class-layout edits and patches inside inner, anonymous, or lambda classes.The authors report no evaluated subject system required manual configuration for those external side effects.Source: §5.3 · p. 11

6 RELATED WORK

Related work surveys generate-and-validate APR families and cost-reduction efforts in APR and mutation testing. UniAPR extends on-the-fly validation beyond PraPR’s bytecode-level scope while addressing JVM-state pollution.

6.1 Automated Program Repair

  • Prior APR work spans monitoring-and-healing and generate-and-validate methods, including heuristic, constraint-based, and template-based approaches.Examples include GenProg, SimFix, Nopol, ACS, PAR, and CapGen.Source: §6.1 · pp. 11–12
  • JAID and SketchFix bundle patches but still validate each in a separate JVM, whereas PraPR uses bytecode mutation and HotSwap but is limited in scope and vulnerable to JVM pollution.UniAPR is positioned to retain on-the-fly validation while supporting source-level APR and resetting JVM state.Source: §6.1 · p. 12
  • UniAPR generalizes PraPR-style on-the-fly validation to source- and bytecode-level APR, resets JVM state between patches, and can integrate existing APR tools as patch-generation add-ons.This provides a framework for combining techniques while retaining their distinct patch-generation strengths.Source: §6.1 · p. 12

6.2 Cost Reduction for Mutation Testing

  • Mutation testing shares APR’s repeated-test cost; prior optimizations reuse execution state, while PIT uses one process but supports simple bytecode mutations and cannot handle JVM-state pollution.State-reuse approaches can be imprecise or struggle with external resource access.Source: §6.2 · p. 12

7 CONCLUSION

UniAPR extends on-the-fly patch validation to representative source-level APR tools and introduces JVM-state resetting for precision. Experiments report over an order-of-magnitude speedup without imprecise validation.

7 CONCLUSION

  • Over an order of magnitude faster, CapGen, SimFix, and ACS show no imprecision with UniAPR’s on-the-fly validation.The framework generalizes on-the-fly validation to state-of-the-art source-level APR and resets JVM state between patches.Source: §7 · p. 12
  • UniAPR generalizes on-the-fly validation from bytecode APR to existing state-of-the-art APR systems, including source-code techniques.The conclusion reports the first empirical evidence that on-the-fly validation can be imprecise or unsound and introduces JVM-state resetting for precision.Source: §7 · p. 12
Loading 2007.11449v1…