Source-linked AI summary
Practical Program Repair in the Era of Large Pre-trained Language Models
Chunqiu Steven Xia, Yuxiang Wei, Lingming Zhang
TL;DR
Existing automated program repair techniques are constrained by limited patch variety and reliance on bug-fixing datasets, while prior direct PLM-based studies lacked comprehensive evaluation. This study evaluates nine generative and infilling PLMs across three repair settings, five datasets, and three languages, finding that direct PLM application substantially outperforms existing APR techniques and supports additional patch-ranking and correctness-checking uses.
Problem
Existing APR techniques have limited patch variety because they rely on fix templates or bug-fixing datasets, hindering repairs for complicated bugs.
Method
The study evaluates nine generative and infilling PLMs, ranging from 125M to 20B parameters, across three repair settings, five datasets, and three programming languages.
Results
Directly applying state-of-the-art PLMs substantially outperforms existing APR techniques across the evaluated datasets; Codex fixes 32 more bugs than the existing best APR technique on Defects4J 1.2.
Takeaways & Limitations
Larger models tend to deliver stronger APR results, suffix code improves fix generation and compilation rate, and PLMs can rank or check patch correctness.
Takeaways & Limitations
Determining semantically correct patches involved manual validation of plausible patches, creating an internal threat to validity.
Abstract
from arXiv · showhide
Automated Program Repair (APR) aims to help developers automatically patch software bugs. However, current state-of-the-art traditional and learning-based APR techniques face the problem of limited patch variety, failing to fix complicated bugs. This is mainly due to the reliance on bug-fixing datasets to craft fix templates or directly predict potential patches. Large Pre-Trained Language Models (PLMs), trained using billions of text/code tokens, can potentially help avoid this issue. Very recently, researchers have directly leveraged PLMs for APR without relying on any bug-fixing datasets. Meanwhile, such existing work either failed to include state-of-the-art PLMs or was not evaluated on realistic datasets. In this work, we perform the first extensive study on directly applying PLMs for APR. We select 9 recent state-of-the-art PLMs, including both generative and infilling models, ranging from 125M to 20B in size. We designed 3 different repair settings to evaluate the different ways we can use PLMs to generate patches. We apply the PLMs under these repair settings on 5 datasets across 3 different languages and compare different PLMs in the number of bugs fixed, generation speed and compilation rate. Our study demonstrates that directly applying state-of-the-art PLMs can already substantially outperform all existing APR techniques on all our datasets. Among the studied PLMs, the scaling effect exists for APR where larger models tend to achieve better performance. Also, we show for the first time that suffix code after the buggy line (adopted in infilling-style APR) is important in not only generating more fixes but more patches with higher compilation rate. Besides patch generation, the PLMs consider correct patches to be more natural than other ones, and can even be leveraged for effective patch ranking or patch correctness checking.
I. INTRODUCTION
APR reduces developers’ manual bug-fixing effort, but template- and training-data-dependent methods have limited repair variety. This study evaluates state-of-the-art PLMs directly on realistic, multilingual repair tasks and finds strong performance and practical guidance.
- Motivation: Template-based APR is constrained to bug types covered by human-designed fix templates, while learning-based APR depends on supervised buggy–fixed code pairs.These dependencies limit the diversity of bugs and patches that the methods can address.
- Motivation: Recent direct PLM-based APR studies used smaller models or limited evaluations, including a small dataset of 40 bugs and simple programming tasks.Prior work focused mainly on AlphaRepair or Codex rather than a broad set of current PLMs and realistic projects.
- Findings: PLMs directly applied to APR substantially outperform existing APR tools on the studied datasets, with Codex fixing 32 more bugs than the previous best technique on Defects4J 1.2.The result is reported for the widely studied Defects4J 1.2 dataset and other datasets.
- Findings: Larger PLMs tend to deliver stronger APR results, while suffix code after the buggy line supports more fixes and higher compilation rates.The study also reports that correct patches receive higher naturalness and that PLMs can support patch ranking or correctness checking.
II. BACKGROUND AND RELATED WORK
PLMs are pretrained on large text and code corpora and can support generative or infilling repair. Earlier direct-PLM APR studies were narrow, motivating evaluation on realistic datasets and diverse repair settings.
- PLMs: PLMs use Transformer architectures and unsupervised pretraining on large text or code corpora before downstream use.They include encoder-only, decoder-only, and encoder-decoder architectures.
- PLMs: Encoder-only models predict masked tokens from bidirectional context, while decoder-only models generate tokens autoregressively.These architectural differences determine whether models naturally support infilling, generation, or both.
- APR: APR generates candidate patches from buggy code and validates them against a test suite; plausible patches pass all tests, whereas correct patches also fix the underlying bug.This distinction separates test-suite satisfaction from semantic correctness.
- APR: Template-based APR performs strongly but only addresses bug types represented by handcrafted templates, motivating learning-based approaches for more expressive patches.Learning-based APR commonly models repair as translating buggy code into fixed code.
- Related work: Prior direct PLM APR studies used small synthetic datasets and few models, with limited attention to infilling-style repair and multiple programming languages.The paper positions its broader evaluation as a response to these restrictions.
- Patch ranking: Patch-ranking experiments use model entropy to prioritize patches that are more likely to be correct.The study computes entropies with recent PLMs on real-world systems.
III. APPROACH
The approach compares diverse PLMs through three practical APR generation settings and examines how model architecture and scale affect repair. The evaluation includes models from 125M to 20B parameters and supports both generative and infilling workflows.
- Models: The selected models are chosen from popular code-trained PLMs across organizations and model types.The selection aims to provide a diverse evaluation set.
- Models: The chosen PLMs range from 125M to 20B parameters, with model type indicating support for infilling or generative APR.The overview also records each model’s pretraining dataset.
- Models: GPT-Neo models are evaluated at 125M, 1.3B, and 2.7B parameters, while GPT-J and GPT-NeoX provide 6.7B and 20B models.These models are open-source implementations of the GPT-3 transformer architecture.
- Models: Codex is a 12B-parameter GPT-3-based model designed for code generation and finetuned on 159GB of code files.It represents a large generative PLM included in the evaluation.
2) Infilling Models: •
The study evaluates generative and infilling PLMs across three APR settings, including complete function generation, correct code infilling, and single line generation.
- 2) Infilling Models: •: CodeT5 is a 220M-parameter code model trained with span prediction across eight programming languages.Its training combines CodeSearchNet with a C/C# dataset from BigQuery.
- 2) Infilling Models: •: INCODER is a code-infilling model evaluated in both 1.3B and 6.7B parameter versions.It uses a causal masking objective and is trained on GitHub/GitLab code and StackOverflow data.
- 2) Infilling Models: •: The study compares generative and infilling PLMs across three APR settings.The settings are complete function generation, correct code infilling, and single line generation.
- 2) Infilling Models: •: Complete function generation uses a buggy function as input and prompts the PLM to generate the entire patched function.This setting does not assume the bug location or bug type is known.
2) Correct code infilling:
Correct code infilling removes the buggy hunk and uses both surrounding code contexts to generate replacement code. Patch generation samples multiple outputs and ranks them using entropy-based naturalness.
- 2) Correct code infilling:: Correct code infilling provides the prefix and suffix surrounding the removed buggy hunk.Infilling models condition on both left and right code, unlike prefix-only generative models.
- 2) Correct code infilling:: The model generates a replacement chunk between an infilling token, after which the output is combined with the preserved prefix and suffix.The infilling token marks the location where new code should be generated.
- 2) Correct code infilling:: Single line generation supplies the bug location and restricts the repair to one changed line.Infilling models use prefix and suffix, while generative models can use only the prefix and stop after one line.
- 2) Correct code infilling:: Nucleus sampling and temperature generate multiple patches per bug, with lower temperatures producing more similar samples.A temperature of 0 deterministically selects the most likely token at each step.
- 2) Correct code infilling:: Entropy measures patch naturalness as the negative log probability of generated tokens.The study records entropy for each patch and evaluates its use for ranking and correctness checking.
IV. EXPERIMENTAL SETUP
The experiments examine PLM performance across APR tasks, baselines, scaling, and entropy-based patch assessment, using a reproducible generation pipeline and benchmark tables.
- IV. EXPERIMENTAL SETUP: RQ1 evaluates PLM types across repair datasets, programming languages, APR tasks, model sizes, computation time, and compilation rates.The question targets a holistic evaluation of APR ability and scaling behavior.
- IV. EXPERIMENTAL SETUP: RQ2 compares directly applied PLMs with state-of-the-art APR tools and examines bugs uniquely fixed by PLMs.The comparison uses existing APR baselines as reference points.
- IV. EXPERIMENTAL SETUP: RQ3 tests whether PLM entropy can assess patch naturalness, rank patches, and check patch correctness.The evaluation compares patched and buggy functions under the models’ built-in naturalness metric.
- IV. EXPERIMENTAL SETUP: RQ4 explores increasing samples and combining PLMs with repair templates as two performance-improvement directions.These experiments extend direct PLM-based repair.
- IV. EXPERIMENTAL SETUP: The implementation uses PyTorch and Hugging Face models, with Codex accessed through OpenAI’s API.The generation pipeline directly reuses model weights and supports Codex suffix-based infilling.
- IV. EXPERIMENTAL SETUP: The default generation setting uses nucleus sampling with top p = 0.95, temperature = 0.8, and 200 samples per bug.Experiments run on a 32-Core workstation with 256 GB RAM and an NVIDIA RTX A6000 GPU.
C. Subject Systems
The evaluation covers five APR benchmarks across three programming languages, filtered for repairs within single functions, and compares PLMs with 20 APR tools.
- C. Subject Systems: The evaluation uses five APR benchmarks spanning three programming languages and focuses on bugs fixed within a single function.The filtering also records single-hunk and single-line subsets.
- C. Subject Systems: Defects4J 1.2 contains 391 bugs from six Java projects, while Defects4J 2.0 contains 438 new bugs from nine additional projects.Both versions provide buggy and patched source-project pairs.
- C. Subject Systems: QuixBugs contains 40 classic programming problems in both Python and Java, with multiple test inputs and expected outputs.The Java version is translated from the original Python benchmark.
- C. Subject Systems: Only 91 of ManyBugs’ 185 C bugs are used because the remaining bugs were not reproducible with successful builds and passing reference patches.The dataset spans nine open-source projects and includes developer-written tests.
- C. Subject Systems: The comparison includes eight learning-based and twelve traditional APR tools, for 20 baselines in total.Baseline comparisons on several datasets use perfect fault localization, while ManyBugs is used only for RQ1.
E. Evaluation Metrics
The study measures repair quality using plausible and correct patches, then compares PLM repair settings on effectiveness, speed, and compilation-related errors. Larger models and infilling-oriented settings generally produce stronger repair outcomes.
- Evaluation Metrics: Plausible patches pass all test cases, while correct patches are syntactically or semantically equivalent to reference patches.Correctness is manually assessed for each plausible patch.
- Repair effectiveness: Model size consistently correlates with stronger repair effectiveness across the evaluated generative models and repair datasets.The study reports more correct and plausible patches as model size increases, although Codex outperforms GPT-NeoX despite having fewer parameters.
- Repair effectiveness: Correct code infilling and single line generation produce higher correct-fix ratios than complete function generation.These settings also yield higher ratios of correct to plausible patches.
- Repair effectiveness: Infilling models outperform generative counterparts in single line generation because they condition on both preceding and succeeding code.Generative models cannot use suffix snippets, whereas infilling models can.
2) Speed:
PLM-based repair achieves broad effectiveness across Defects4J 1.2, including fixes beyond existing tools, while exposing trade-offs between model size, generation speed, and repair coverage. Combining models increases the number of correct and unique fixes, including examples requiring non-template reasoning.
- Speed: Larger models improve repair effectiveness but substantially reduce patch-generation speed, creating a trade-off between repair quality and time cost.GPT-NeoX is 71x slower than GPT-Neo 125M for complete function generation.
- Compilation rate: Overall syntactic and semantic error rates decline with model size, although generative single line generation produces many syntactic errors.The error analysis covers all three repair settings on Defects4J 1.2.
- Defects4J 1.2 results: Combining all PLMs generates 36 unique bug fixes that no other tools have fixed on single-function Defects4J 1.2 bugs.The Venn diagram compares PLM fixes with the three best-performing baselines and an aggregated Others category.
- Unique fixes: PLMs can generate unique fixes for reference-equality handling and subtle numerical precision errors that do not fit common APR templates.The Math-69 example replaces an unstable p-value calculation with a more stable alternative using statistical knowledge.
2) Additional results:
Additional evaluations show that PLMs remain competitive with state-of-the-art APR tools across multiple datasets and can be applied directly across Java, Python, and C. The study also examines patch naturalness using entropy.
- Additional results: All nine studied PLMs outperform TBar on Defects4J 2.0 and are competitive with Recoder.Many models achieve similar or better performance than carefully designed APR tools on additional datasets.
- Additional results: PLMs support direct multilingual repair across Defects4J 2.0, QuixBugs-Java, and QuixBugs-Python without language-specific additional finetuning.The paper contrasts this with baselines designed for or requiring finetuning on a single language.
- Patch naturalness: Correct and plausible patches have lower average entropy than non-plausible patches across the studied models.Entropy is used as a measure of how natural a generated patch is.
C. RQ3: Patch Ranking and Correctness Checking Analysis
The study uses patch entropy to assess naturalness and rank candidate patches for validation. Lower-entropy ranking improves bug-fixing outcomes under tighter validation budgets, with sum entropy slightly outperforming mean entropy.
- Patch correctness checking: Correct and plausible patches have lower average entropy than non-plausible patches.The comparison is made across models and repair scenarios containing bugs for which a correct patch can be generated.
- Patch ranking: Entropy-based ranking fixes more bugs than random validation as the number of validated patches increases.The ranking analysis uses five PLMs with the highest numbers of correct patches on Defects4J 1.2.
- Patch ranking: Sum entropy performs slightly better than mean entropy for prioritizing patches.The authors hypothesize that sum entropy accounts for the entire sequence and favors shorter generated sequences.
D. RQ4: Improvements on Direct PLM APR
The study finds that PLM-based APR can be substantially improved by increasing sampling and combining PLMs with repair templates, while also examining data leakage and generalization limits.
- Performance improvements: More samples substantially improve the number of correctly fixed bugs across all three evaluated datasets.The improvement approaches the best baseline result.
- Performance improvements: Simple repair templates combined with PLMs achieve the highest number of correctly fixed bugs on all datasets.On Defects4J 1.2, this combination fixes 78 bugs, including 15 unique fixes absent from prior baseline tools.
- Data leakage: The evaluated PLM fixes are often different from developer patches, reducing concern that the results arise solely from memorization.Across 354 bug fixes on Defects4J 1.2, 234 fixes (66%) contain a patch different from the developer patch; excluding single-line bugs, this rises to 77% (196/255).
- Data leakage: Among 109 bugs fixable by combining all correct PLM patches, 93 bugs (85%) have at least one correct patch differing from the developer patch.Removing fixes identical to developer patches still leaves 31 bugs that prior tools cannot fix.
- Data leakage: For four models with accessible training data, only 15% (20/128) of fixes matching developer patches were found in the original training data.This supports the conclusion that most correct fixes from these PLMs are not simply memorized.
- Scope and validity: The evaluation covers five repair datasets and three programming languages, but the findings may not generalize to other datasets or languages.Further reducing data leakage would require retraining the PLMs, which could be extremely costly.