Source-linked AI summary
Less Training, More Repairing Please: Revisiting Automated Program Repair via Zero-shot Learning
Chunqiu Steven Xia, Lingming Zhang
TL;DR
Learning-based APR depends on costly, potentially noisy bug-fix datasets and may be limited to patterns represented in those datasets. AlphaRepair instead uses a pre-trained code model for zero-shot cloze-style repair, achieving up to 3.3X more fixes than the best baseline on Defects4J 2.0.
Problem
Learning-based APR requires buggy-fixed training pairs that are difficult to construct, can contain noise, and may limit repair edits to patterns present in the training data.
Method
AlphaRepair directly uses CodeBERT's masked-language-model objective to recover masked buggy code in a zero-shot cloze-style repair setting without additional fine-tuning.
Results
3.3X more bugs fixed than the best baseline in Defects4J 2.0, while AlphaRepair also achieved similar results on Python QuixBugs compared with Java.
Takeaways & Limitations
AlphaRepair opens a multilingual learning-based APR direction that does not require fine-tuning on repair datasets.
Takeaways & Limitations
The evaluation results may depend on benchmark choice, timeout settings, machine configuration, and reuse of prior baseline results.
Abstract
from arXiv · showhide
Due to the promising future of Automated Program Repair (APR), researchers have proposed various APR techniques, including heuristic-based, template-based, and constraint-based techniques. Among such classic APR techniques, template-based techniques have been widely recognized as state of the art. However, such template-based techniques require predefined templates to perform repair, and their effectiveness is thus limited. To this end, researchers leveraged the recent advances in Deep Learning to further improve APR. Such learning-based techniques view APR as a Neural Machine Translation problem, using the buggy/fixed code snippets as the source/target languages for translation. In this way, such techniques heavily rely on large numbers of high-quality bug-fixing commits, which can be extremely costly and challenging to construct. Furthermore, the edit variety of these learning-based techniques are limited to the available bug-fixes within their training datasets. Therefore, in this paper, we aim to revisit the learning-based APR problem, and propose AlphaRepair, to leverage zero-shot learning directly using large pre-trained code models for APR. Our main insight is instead of modeling what a repair edit should look like, we can directly predict what the correct code is based on the context information. We have implemented AlphaRepair as a practical multilingual APR tool based on the recent CodeBERT model. Our results on the widely used Defects4J benchmark show that AlphaRepair can substantially outperform state-of-the-art APR tools. We also studied the impact of different design choices and show that AlphaRepair performs even better on a newer version of Defects4J (2.0) with 3.3X more fixes than best performing baseline, indicating that AlphaRepair can potentially avoid the dataset-overfitting issue of existing learning-based techniques.
1 INTRODUCTION
Automated program repair reduces manual debugging effort, but established template- and learning-based approaches are constrained by predefined patterns, noisy and limited bug-fix data, and context representation. AlphaRepair addresses these issues through zero-shot cloze-style repair with large pre-trained code models, achieving strong Java and Python benchmark results.
- Developers spend an estimated 35 to 50% of their time debugging, motivating Automated Program Repair to generate patches automatically.
- Template-based APR is widely recognized as state of the art, but predefined fix patterns limit the number and expressiveness of candidate patches.
- Learning-based APR formulates repair as Neural Machine Translation, transforming buggy code into fixed code with encoder-decoder models trained on buggy-patch pairs.
- Historical bug-fix datasets can contain irrelevant commits and unrelated changes, adding noise to the training data.
- Limited historical fixes can restrict learning-based APR edit variety to patterns present in the training data.
- AlphaRepair directly queries pre-trained code models in a zero-shot setting, using cloze-style repair without additional fine-tuning and supporting multilingual extension through new code corpora.
- AlphaRepair outperforms existing APR techniques on Java and Python versions of Defects4J and QuixBugs, demonstrating multilingual capability.
2 BACKGROUND
Learning-based APR uses deep-learning translation models and large pre-trained code models, but still faces data, generalization, and context-representation limitations. This paper instead applies a pre-trained model directly for zero-shot cloze-style repair.
- Learning-based APR commonly uses encoder-decoder deep-learning models to translate buggy code into patched code.
- Training pairs of buggy and fixed code are difficult to obtain because commit-mining heuristics can include unrelated changes and introduce dataset noise.
- Learning-based APR may not generalize to fix patterns absent from its training dataset.
- Current approaches may miss important syntactic and semantic information when encoding context around buggy code.
- AlphaRepair uses a large pre-trained code model directly for cloze-style APR without training or fine-tuning on buggy-fixed code datasets.
- CodeBERT's masked-language-model objective supports recovering masked buggy tokens, while its bidirectional architecture uses context before and after the mask.
3 APPROACH
AlphaRepair reframes program repair as a zero-shot cloze task: it masks buggy code and uses CodeBERT to predict replacements from surrounding context. It generates candidates through multiple mask strategies, then re-ranks them using CodeBERT’s sequence likelihood.
- AlphaRepair treats repair as predicting correct code in context rather than translating buggy code into fixed code.
- CodeBERT’s masked-language-model objective is reused for zero-shot repair by masking all tokens in the buggy snippet and recovering replacement tokens.This avoids additional retraining or fine-tuning on bug-fixing datasets.
- 3.1 Input Processing: AlphaRepair combines context before and after the buggy line with the buggy line represented as a natural-language comment.The context is expanded around the buggy line up to CodeBERT’s 512-token input limit.
- 3.2 Mask Generation: It generates candidate patches with complete, partial, and template mask lines, including replacements of entire lines or insertions before and after buggy locations.Partial strategies retain prefixes or suffixes of the buggy line while masking the remainder.
- 3.3 Patch Generation: Grouped mask tokens are filled iteratively with beam search, retaining the highest-scoring token sequences as generation proceeds.At each step, CodeBERT supplies likely replacements, and the beam width controls how many sequences are retained.
- 3.4 Patch Re-Ranking: Generated patches are re-ranked using a length-normalized joint score computed from CodeBERT’s conditional token probabilities.The temporary generation score is only a proxy, so AlphaRepair recomputes likelihoods after complete patches are generated.
4 EXPERIMENTAL DESIGN
The evaluation compares AlphaRepair with state-of-the-art APR tools across localization settings, configurations, projects, and programming languages. It uses established bug benchmarks and distinguishes plausible from manually assessed correct patches.
- Research Questions: The study asks how AlphaRepair compares with state-of-the-art tools, which configurations affect performance, and whether it generalizes across projects and languages.
- Comparisons: The evaluation compares AlphaRepair with traditional and learning-based APR tools under perfect and non-perfect fault localization.Perfect localization supplies the exact fix location, while non-perfect localization uses suspicious locations from fault localization.
- Configurations: The study uses ablation experiments to measure the contribution of AlphaRepair’s individual design components.
- Datasets: Evaluation covers Defects4J 1.2, 82 single-line bugs from new bugs in Defects4J 2.0, and QuixBugs in Java and Python.Defects4J 1.2 contains 391 bugs across six Java projects; QuixBugs contains 40 small classic algorithms with single-line bugs.
- Baselines: The baselines include six recent learning-based APR tools and traditional APR tools, compared under matching fault-localization settings.Learning-based baselines include Recoder, DeepDebug, CURE, CoCoNuT, DLFix, and SequenceR.
- Metrics: Performance is measured using plausible patches that pass the full test suite and correct patches judged syntactically or semantically equivalent to developer patches.Correctness is determined by manually inspecting each plausible patch for semantic equivalence.
5 RESULT ANALYSIS
AlphaRepair outperforms prior APR tools across fault-localization settings, newer Defects4J versions, and multilingual repair benchmarks. Its results also show unique fixes, useful design components, improved patch ranking, and examples of repairs beyond common templates or training edits.
- 5.1.1 Perfect Fault Localization.: 74 bugs were correctly fixed under perfect fault localization, outperforming all previous traditional and learning-based baselines.
- 5.1.1 Perfect Fault Localization.: AlphaRepair fixed 14 unique bugs among learning-based tools and 8 unique bugs against the three best baselines plus all other APR tools.These results indicate that AlphaRepair can complement other techniques to increase correct patches.
- 5.1.1 Perfect Fault Localization.: Unique examples include inserting a length check on a method-call result and adding a missing switch case, repairs difficult for template- and training-based tools.AlphaRepair uses surrounding context and large open-source pre-training to identify these less conventional edits.
- 5.1.1 Perfect Fault Localization.: 50 correct patches and 7 unique bug fixes were produced without perfect fault localization, outperforming previous state-of-the-art tools.This result was achieved while generating patches for multiple suspicious lines with a reduced beam width.
- 5.2 RQ2: Ablation Study: Template mask lines added 21 new fixes in the ablation study, while complete masking alone produced 20 correct patches.Complete masking gives CodeBERT broad edit freedom but causes an exponentially growing search space as mask tokens increase.
- 5.2 RQ2: Ablation Study: Re-ranking improved the average correct-patch position from 612th to 418th, a 31.7% reduction, with 61 of 74 correct patches ranked higher.Re-ranking calculates likelihood without mask tokens and prioritizes patches likely to be validated when only a subset can be tested.
- 5.3 Additional Evaluations: On Defects4J 2.0, AlphaRepair achieved 36 correct patches, 3.3X more than the top baseline, while achieving 28 Java and 27 Python fixes on QuixBugs.The Defects4J 2.0 result is reported on a harder dataset, and AlphaRepair was directly usable for both QuixBugs languages.
6 THREATS TO VALIDITY
The evaluation faces internal threats from manual patch-correctness judgments, possible CodeBERT training overlap, and experimental setup differences, plus an external threat from benchmark choice.
- Manual assessment of plausible-patch correctness introduces an internal validity threat, mitigated through reviewer discussions and releasing correct patches and reproduction materials.The authors examined all plausible patches and discussed their correctness before releasing the confirmed patches and reproduction artifacts.
- 16.6% of Defects4J 1.2 bugs and 11.0% of Defects4J 2.0 bugs overlap with CodeBERT’s original training data.Among AlphaRepair’s correctly fixed bugs, 13.5% and 13.9% had corresponding developer patches in that training data; perturbation tests still yielded correct fixes for all 15 overlapping bugs.
- Experimental settings can change repair results because timeout thresholds, machine speed, and baseline execution procedures affect the number of fixes observed.The study used a five-hour per-bug timeout and prior reported baseline results; fully eliminating this threat would require rerunning all baselines under identical conditions.
- Claims about AlphaRepair’s performance may not generalize beyond the selected benchmarks.The authors assess this boundary using Defects4J 2.0 and Python and Java versions of QuixBugs.
7 CONCLUSION
The paper introduces AlphaRepair, a zero-shot cloze-style APR technique using large pre-trained code models directly, and reports state-of-the-art results across Java benchmarks with multilingual evidence.
- AlphaRepair directly uses a large pre-trained code model for zero-shot program repair without fine-tuning on repair datasets.Its inputs exploit CodeBERT’s pre-training objective to generate fix lines from surrounding context.
- 3.3X more bugs fixed than the best baseline is the highest improvement reported for AlphaRepair on Defects4J 2.0.The evaluation covers Defects4J and QuixBugs, with AlphaRepair achieving new state-of-the-art results on the reported Java benchmarks.
- Similar results on Python and Java versions of QuixBugs demonstrate AlphaRepair’s multilingual ability across the evaluated languages.