PaperlayerView exact revision on arXiv

Source-linked AI summary

Impact of Code Language Models on Automated Program Repair

Nan Jiang, Kevin Liu, Thibaud Lutellier, Lin Tan

arXiv:2302.05020v3cs.SE

TL;DR

APR research lacks comprehensive evidence on whether general-purpose CLMs can fix realistic bugs and how they compare with specialized DL-based repair tools. This paper evaluates ten CLMs across four benchmarks, introduces an unseen benchmark, and fine-tunes the models with APR data. CLMs show competitive performance, while fine-tuning substantially improves fixing, alongside concerns about buggy-line over-reliance and evaluation leakage.

  • Problem

    Comprehensive, in-depth comparisons of CLMs with specialized DL-based APR techniques on realistic bugs are limited, while abstract benchmarks and BLEU scores do not establish patch correctness.

  • Method

    The paper evaluates ten CLMs on four APR benchmarks using developer-written tests, compares prompts with and without buggy lines, and fine-tunes the models with APR training data.

  • Results

    Across four benchmarks, the best untuned CLM fixes 72% more bugs than state-of-the-art DL-based APR techniques, while fine-tuning improves CLMs by 31%–1,267% and yields 46%–164% more fixes.

  • Takeaways & Limitations

    CLM-based APR and APR-specific fine-tuning designs are promising directions, but evaluations should address data leakage and buggy-line over-reliance.

  • Takeaways & Limitations

    Codex is excluded from comparative results because it is a black box that cannot be fine-tuned, and its changing training data makes fair evaluation challenging.

Abstract

from arXiv · show

Automated program repair (APR) aims to help developers improve software reliability by generating patches for buggy programs. Although many code language models (CLM) are developed and effective in many software tasks such as code completion, there has been little comprehensive, in-depth work to evaluate CLMs' fixing capabilities and to fine-tune CLMs for the APR task. Firstly, this work is the first to evaluate ten CLMs on four APR benchmarks, which shows that surprisingly, the best CLM, as is, fixes 72% more bugs than the state-of-the-art deep-learning (DL)-based APR techniques. Secondly, one of the four APR benchmarks was created by us in this paper to avoid data leaking for a fair evaluation. Thirdly, it is the first work to fine-tune CLMs with APR training data, which shows that fine-tuning brings 31%-1,267% improvement to CLMs and enables them to fix 46%-164% more bugs than existing DL-based APR techniques. Fourthly, this work studies the impact of buggy lines, showing that CLMs, as is, cannot make good use of the buggy lines to fix bugs, yet fine-tuned CLMs could potentially over-rely on buggy lines. Lastly, this work analyzes the size, time, and memory efficiency of different CLMs. This work shows promising directions for the APR domain, such as fine-tuning CLMs with APR-specific designs, and also raises awareness of fair and comprehensive evaluations of CLMs and calls for more transparent reporting of open-source repositories used in the pre-training data to address the data leaking problem.

I. INTRODUCTION

APR seeks to improve software reliability by automatically generating patches, but existing DL-based tools often fix too few bugs and require costly patch generation and validation. This paper evaluates whether CLMs can improve APR fairly, using realistic benchmarks, test-based validation, and APR-specific fine-tuning.

  • APR automatically generates patches to repair software defects and improve software reliability.
  • 93% of developers review at most ten patches, while 63% expect APR tools to respond within one hour.Existing tools may generate hundreds to thousands of candidate patches and take hours to validate them.
  • CLMs are trained on large unlabeled code corpora for general tasks, whereas DL-based APR models use APR-specific designs and labeled buggy-fixed code pairs.
  • Prior CLM evaluations largely use abstracted CodeXGLUE code and BLEU scores rather than realistic projects, full context, test cases, and patch-correctness validation.Concrete variable and function names are necessary for bug fixing, and high BLEU scores can correspond to incorrect patches.
  • Training-data leakage may affect evaluations on existing APR benchmarks, so the study manually creates HumanEval-Java, unseen by the evaluated CLMs during training.
  • The study fine-tunes ten CLMs with APR training data and examines how fine-tuning-data size affects their fixing capabilities.

D. Contributions

The paper evaluates CLMs against DL-based APR, introduces a leakage-resistant benchmark, fine-tunes CLMs for APR, and studies buggy-line dependence and efficiency. It reports competitive unfine-tuned performance, substantial fine-tuning gains, and model-specific data and resource trade-offs.

  • Evaluation scope: The study evaluates ten CLMs and four DL-based APR techniques on four benchmarks, including the new HumanEval-Java benchmark.The benchmarks are Defects4J v1.2, Defects4J v2.0, QuixBugs, and HumanEval-Java.
  • Unfine-tuned CLMs: 72% more bugs are fixed by the best unfine-tuned CLM than by state-of-the-art DL-based APR techniques.This finding establishes competitive fixing capability without APR-specific fine-tuning.
  • Buggy-line impact: Buggy-line prompts reduce unfine-tuned CLMs’ fixing performance, indicating that these models do not make good use of explicitly supplied buggy lines.The buggy lines remain useful for guiding repair in general, but CLMs fix fewer bugs when they are explicitly given.
  • Fine-tuning: Fine-tuning improves all ten CLMs by 31%–1,267%, while fine-tuned CLMs fix 46%–164% more bugs than DL-based APR techniques.The fine-tuning experiment uses APR training data and directly compares the resulting models with existing DL-based techniques.
  • Fine-tuning: Fine-tuning can cause CLMs to over-rely on buggy lines, making them fail on some bugs that unfine-tuned models can fix.This identifies a trade-off between adapting models to APR data and preserving useful behavior from pre-training.
  • Fine-tuning data size: CodeT5 and CodeGen perform best after 10,000 APR instances, whereas InCoder performs best after 50,000; additional data can reduce performance.More data reduces CodeT5 and CodeGen performance by 8%–19% and InCoder performance by 9%.
  • Efficiency: CodeT5 and InCoder have the best size efficiency, while CodeT5, PLBART, and InCoder offer better time and memory efficiency than CodeGen.These findings identify model families suited to scaling or constrained-resource settings.
  • Implications: The paper adds implications for future APR research alongside its benchmark, evaluation, fine-tuning, and efficiency studies.The contribution list explicitly includes future-work implications.

II. CODE LANGUAGE MODELS

Code language models use encoder-only, decoder-only, or encoder-decoder Transformer architectures, with the architecture determining their generation capabilities. They are pre-trained through tasks such as token prediction, span recovery, identifier prediction, tagging, and dual generation.

  • Architectures: Most CLMs use Transformer architectures and belong to encoder-only, decoder-only, or encoder-decoder groups.The groups differ in whether they encode inputs, autoregressively generate sequences, or combine both capabilities.
  • Architectures: Encoder-only models support representation tasks, decoder-only models support open-ended code generation, and encoder-decoder models support both encoding and generation.CodeBERT and GraphCodeBERT are encoder-only; CodeGen, InCoder, and Codex are decoder-only; PLBART and CodeT5 are encoder-decoder.
  • Pre-training tasks: Next token prediction trains a model to iteratively generate complete programs from incomplete code.The task predicts the following token, such as x after `return` in a simple addition function.
  • Pre-training tasks: Masked span prediction trains models to reconstruct code snippets hidden behind placeholders.The example masks parts of a Java function and requires predicting the missing tokens.
  • Pre-training tasks: Masked identifier prediction, deleted span prediction, and identifier tagging target names, missing code, and identifier labels respectively.These tasks provide supervision for recovering identifiers, completing deleted code, and distinguishing identifiers from other tokens.
  • Pre-training tasks: Bimodal dual generation trains models to transform code among natural language and programming-language representations.Examples include natural language to Java, Java to natural language, and Java to Python.

III. EXPERIMENTAL DESIGN

The experimental design compares ten publicly accessible CLMs with DL-based APR techniques on Java single-hunk bugs across three established benchmarks and HumanEval-Java. It also controls prompting, addresses benchmark leakage, and studies model and fine-tuning configurations.

  • Experimental overview: The study applies ten CLMs and three state-of-the-art DL-based APR techniques to four benchmarks, then compares their generated patches.The design covers unfine-tuned fixing, fine-tuning with different data sizes, and efficiency comparisons.
  • Experimental scope: The evaluation focuses on Java single-hunk bugs so CLMs and DL-based APR techniques repair the same bug type.Single-hunk bugs contain continuous buggy lines, matching the target of the strongest DL-based APR techniques.
  • Existing benchmarks: Defects4J v1.2 contains 393 bugs, Defects4J v2.0 contains 444 additional bugs, and QuixBugs contains 40 algorithmic bugs.The established benchmarks include Java projects such as Google Closure compiler and Apache commons-math, plus algorithms such as quick sort and merge sort.
  • Benchmark leakage: Existing benchmarks may overlap CLM pre-training data because Defects4J repositories appear in CodeSearchNet and BigQuery.This creates a data-leakage concern when CLMs are evaluated on those benchmarks.
  • HumanEval-Java: HumanEval-Java converts HumanEval’s Python programs and tests into Java and injects bugs to create 164 single-hunk Java bugs.The benchmark is designed to reduce the risk that CLMs saw the evaluation data during pre-training.
  • Model selection: The selected CLMs are publicly accessible, trained on sufficiently large code corpora, and usable for APR without architectural modification.Encoder-only models and inaccessible models such as Codex are excluded.
  • Studied models: The study includes PLBART-base and PLBART-large, with 140M and 400M parameters, respectively.Both models use the same pre-training data and tasks.
  • Studied models: The study includes CodeT5-small, CodeT5-base, and CodeT5-large, with 60M, 220M, and 770M parameters.CodeT5 uses masked span prediction, identifier tagging, masked identifier prediction, and bimodal dual generation.

D. Applying Code Language Models

The study applies pretrained CLMs without fine-tuning and uses two prompts to compare fixing with and without explicit buggy-line information. Prompt formats and validation procedures vary by model design, while buggy lines are supplied as comments when included.

  • D. Applying Code Language Models: Pretrained CLMs are applied without fine-tuning to evaluate fixing capabilities learned from pretraining tasks.The models are configured using their papers and documentation.
  • D. Applying Code Language Models: Two prompts provide either the known bug location without buggy lines or the buggy lines as comments.The second prompt gives CLMs the same buggy-line information required by DL-based APR techniques.
  • D. Applying Code Language Models: PLBART masks buggy lines and generates the whole patched function, then validates the output with developer-written tests.The mask placeholder is <mask>.
  • D. Applying Code Language Models: CodeT5 masks buggy lines and generates patched lines, which replace the original lines before test-based validation.CodeT5 uses <extra_id_0> and need not generate the whole function.
  • D. Applying Code Language Models: CodeGen completes the function from its prefix, whereas InCoder generates the patched line and following code using post-buggy-line context.CodeGen lacks the code after the buggy lines by design, while InCoder uses it through masked infilling.
  • D. Applying Code Language Models: When buggy lines are included, they are supplied as comments before the buggy-line location.This prompt convention is applied across the CLM models.

E. Fine-tuning Code Language Models

The paper fine-tunes ten CLMs for APR using buggy–fixed code pairs and a common prompt and output format. It also standardizes the fine-tuning setup across models for fair comparison.

  • E. Fine-tuning Code Language Models: Ten CLMs are fine-tuned for APR by prompting them with buggy code to generate fixed code.All models use the same prompt and directly output patched lines, matching DL-based APR outputs.
  • E. Fine-tuning Code Language Models: The fine-tuning dataset contains 143,666 single-hunk fixes from commits in open-sourced GitHub Java projects.The data are split into 129,300 training instances and 14,366 validation instances.
  • E. Fine-tuning Code Language Models: All models use batch size one, Adam with learning rate 1e−5, one training epoch, and a fixed random seed.The common configuration supports consistent comparisons, while batch size one reflects hardware constraints.

F. Baseline DL-based APR Techniques

The comparison uses four open-sourced DL-based APR techniques and evaluates patches using top-ten generation, tests, and manual correctness checks. Without fine-tuning, several CLM families outperform the APR baselines overall, though performance varies by benchmark and model.

  • F. Baseline DL-based APR Techniques: CURE, RewardRepair, Recoder, and KNOD are selected as open-sourced DL-based APR baselines with APR-specific designs.Their designs include code-aware search, execution-aware loss, AST edits, and graph/tree decoding.
  • F. Baseline DL-based APR Techniques: Each tool generates ten patches per bug; tests identify plausible patches, which are manually checked for correctness.A correct patch must be identical or semantically equivalent to the developer-written patch.
  • F. Baseline DL-based APR Techniques: The evaluation reports correct patches within the top ten across four benchmarks, including the new HumanEval-Java benchmark.The top-ten scope reflects reported developer willingness to review up to ten patches.
  • F. Baseline DL-based APR Techniques: CodeGen produces the most compilable patches, averaging 73% compilation with a 97% median, while DL-based APR techniques average 44%–62%.PLBART and InCoder also produce more compilable patches than the DL-based techniques.
  • F. Baseline DL-based APR Techniques: CLMs and DL-based APR tools excel on different benchmarks: DL-based tools perform better on Defects4J v1.2, whereas several CLMs perform better on QuixBugs and HumanEval-Java.The examples attribute CLM success on GCD to learned semantic knowledge and APR-tool failures to reliance on buggy lines.
  • F. Baseline DL-based APR Techniques: Overall, PLBART, CodeGen, and InCoder are competitive without fine-tuning, while CodeT5 generates poor patches before APR-specific fine-tuning.The finding combines fixing and compilation outcomes across the benchmarks.

B. Impact of Buggy Lines

Providing buggy lines does not uniformly help CLMs: they fix fewer bugs overall when those lines are included, although buggy-line context enables some otherwise missed fixes. The figures compare compilation-rate distributions and benchmark-specific examples.

  • B. Impact of Buggy Lines: Figure 5 shows distributions of compilation rates for ten patches generated per bug across all four benchmarks.The figure is used to compare compilation behavior across models and benchmarks.
  • B. Impact of Buggy Lines: Figure 6 presents examples where CLMs and DL-based APR tools have different fixing outcomes.The examples span bugs on which either CLMs or APR tools succeed.
  • B. Impact of Buggy Lines: 6%–78% fewer bugs are fixed by CLMs when buggy lines are provided than when they are omitted.For example, PLBART-base fixes 36 bugs with buggy lines versus 72 without them.
  • B. Impact of Buggy Lines: Without the buggy line, CLMs correctly infer the HashSet-based repair for NEXT SMALLEST, but PLBART produces an uncompilable patch when the line is supplied.The supplied line leads to a patch in which numbers is undeclared.
  • B. Impact of Buggy Lines: Buggy-line context enables CLMs to fix Chart-8, whose correct patch they miss without that context.With the line provided, all CLMs generate the correct patch this(time, zone, Local.getDefault());.
  • B. Impact of Buggy Lines: Overall, buggy lines help CLMs on some bugs but yield fewer compilable patches and fewer fixes overall.The authors conclude that CLMs do not make good use of buggy-line information.

V. RQ2: IMPACT OF FINE-TUNING

Fine-tuning substantially improves all ten CLMs’ bug-fixing capabilities, enabling them to outperform existing DL-based APR techniques. However, its effect depends on the model and can increase reliance on buggy lines.

  • 31%–1,267% improvement: fine-tuning increases the fixing capabilities of all ten CLMs.Fine-tuned CLMs consistently outperform DL-based APR techniques across four benchmarks.
  • 100 (164%) more bugs: InCoder-6B fixes this many more bugs than the best DL-based APR technique.
  • 889%–1,267% improvement: CodeT5 gains the most from fine-tuning, while PLBART gains the least at 31%.The paper attributes CodeT5’s gains to programming-language knowledge learned during pre-training.
  • Fine-tuning helps CLMs use buggy-line information, but can also make them over-rely on those lines and miss the intended functionality.This produces a shortcoming shared with existing DL-based APR techniques.

C. Impact of Fine-tuning Data Size

Fine-tuning data size affects CLM fixing capability non-monotonically: several models peak with moderate data and decline with too much, while PLBART follows a different pattern. Model size and resource efficiency also vary substantially across architectures.

  • Impact of Fine-tuning Data Size: 59 correct fixes: CodeT5-large reaches its best HumanEval-Java performance after fine-tuning with 10,000 instances.
  • Impact of Fine-tuning Data Size: 76 correct fixes: InCoder-6B reaches its best performance after fine-tuning with 50,000 instances, despite initially declining with 100 instances.
  • Impact of Fine-tuning Data Size: 10,000–50,000 APR instances: CodeT5, CodeGen, and InCoder reach their best fixing capabilities, but excessive data can reduce performance.
  • Efficiency: Larger models consistently fix more bugs, with CodeT5 and InCoder showing the best size efficiency.
  • Efficiency: 0.70–0.89 seconds: PLBART has the best time efficiency, whereas CodeGen requires 3.64–13.88 seconds per correct patch.
  • Efficiency: 19.84–24.81GB: CodeGen-6B and InCoder-6B require substantially more GPU memory than the other CLMs.The other models fit on standard 8GB or 12GB cards, and CLMs outperform DL-based APR techniques at matched resources.

VII. IMPLICATIONS AND FUTURE WORK

The authors identify CLM-specific opportunities and evaluation requirements for APR. They recommend APR-aware fine-tuning, attention to buggy-line over-reliance, stronger benchmarks, transparent pre-training data reporting, and comprehensive efficiency measurement.

  • Improving fine-tuning: APR-specific designs such as syntax, structural, and test-execution information could further improve fine-tuned CLMs.
  • Addressing over-reliance on buggy lines: Over-reliance on buggy lines makes bugs requiring larger modifications especially challenging for fine-tuned CLMs.The authors suggest balancing fine-tuning data or developing models for bugs requiring large modifications.
  • Improving benchmarks: HumanEval-Java avoids known CLM pre-training exposure but mostly contains small programs, leaving larger unseen buggy programs needed.
  • Avoiding benchmark leaking in pre-training: Benchmark leakage remains a threat because CLM pre-training may include bugs or fixes from existing APR benchmarks.The paper calls for clearer reporting and documentation of repositories used in pre-training.
  • Evaluating size, time, and memory efficiency: Size, time, and memory efficiency should be reported alongside fixing capability for more comprehensive CLM evaluations.
  • Fair and comprehensive evaluation: Codex cannot be fine-tuned and is particularly susceptible to data leakage because its models continually update with new data and potential user input.
  • Fair and comprehensive evaluation: Manual patch-correctness checking may be subjective, while BLEU and CodeBLEU can misrepresent comparative APR performance.RewardRepair scores higher on CodeBLEU than fine-tuned CodeT5 on HumanEval-Java but fixes 19 fewer bugs.

IX. RELATED WORK

Prior work studies CLMs mainly through abstracted-code refinement and similarity metrics, while broader APR research uses real-world benchmarks and diverse repair techniques. This paper addresses those gaps by evaluating CLMs and validating patches through tests.

  • Code refinement studies fine-tune models on CodeXGLUE’s abstracted buggy functions to generate patched functions.
  • BLEU-based reporting does not establish how many bugs are correctly fixed because it measures similarity without executing test cases.
  • Template-, heuristic-, constraint-, and DL-based APR techniques have been evaluated on Defects4J and QuixBugs, but none is fully built on large pre-trained CLMs.
  • This paper evaluates ten CLMs with and without fine-tuning on four APR benchmarks, including HumanEval-Java, and reports competitive fixing capabilities.
Loading 2302.05020v3…