PaperlayerView exact revision on arXiv

Source-linked AI summary

Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program Repair

Yuxiang Wei, Chunqiu Steven Xia, Lingming Zhang

arXiv:2309.00608v3cs.SEcs.LGcs.PL

TL;DR

LLM-based APR can generate invalid patches because it lacks program-semantic guidance during token generation. Repilot combines an LLM with a Completion Engine that prunes infeasible tokens and proactively completes valid continuations. On Defects4J subsets, it achieves state-of-the-art bug-fixing results and improves patch validity and correctness with limited overhead.

  • Problem

    LLM-based APR treats programs as token sequences without structural or semantic understanding, which can produce invalid patches.

  • Method

    Repilot combines autoregressive LLM generation with a Completion Engine that prunes infeasible tokens and proactively completes valid continuations.

  • Results

    Repilot achieves state-of-the-art results on Defects4J 1.2 and 2.0 while generating more valid and compilable patches with minimal overhead compared with direct LLM-based APR.

  • Takeaways & Limitations

    The approach provides a general framework for combining LLMs with Completion Engines for APR and other code-generation tasks.

  • Takeaways & Limitations

    The evaluation covers only two LLMs and one programming language, and the modified Completion Engine requires manual inspection to guarantee soundness.

Abstract

from arXiv · show

During Automated Program Repair (APR), it can be challenging to synthesize correct patches for real-world systems in general-purpose programming languages. Recent Large Language Models (LLMs) have been shown to be helpful "copilots" in assisting developers with various coding tasks, and have also been directly applied for patch synthesis. However, most LLMs treat programs as sequences of tokens, meaning that they are ignorant of the underlying semantics constraints of the target programming language. This results in plenty of statically invalid generated patches, impeding the practicality of the technique. Therefore, we propose Repilot, a general code generation framework to further copilot the AI "copilots" (i.e., LLMs) by synthesizing more valid patches during the repair process. Our key insight is that many LLMs produce outputs autoregressively (i.e., token by token), resembling human writing programs, which can be significantly boosted and guided through a Completion Engine. Repilot synergistically synthesizes a candidate patch through the interaction between an LLM and a Completion Engine, which 1) prunes away infeasible tokens suggested by the LLM and 2) proactively completes the token based on the suggestions provided by the Completion Engine. Our evaluation on a subset of the widely-used Defects4j 1.2 and 2.0 datasets shows that Repilot outperforms state-of-the-art techniques by fixing 27% and 47% more bugs, respectively. Moreover, Repilot produces more valid and correct patches than the base LLM with the same budget. While we focus on leveraging Repilot for APR in this work, the overall approach is also generalizable to other code generation tasks.

1 INTRODUCTION

LLM-based APR can achieve strong bug-fixing performance but may generate semantically infeasible tokens because it treats programs as token sequences. Repilot guides autoregressive generation with a Completion Engine that prunes invalid tokens and proactively completes valid continuations.

  • Motivation: LLM-based APR tools generate patches without structural or semantic understanding, allowing invalid tokens and statically uncompilable patches.The paper highlights infeasible field accesses and difficulty generating rare identifiers as representative failure modes.
  • Approach: Repilot fuses an LLM with a Completion Engine to synthesize more valid patches during autoregressive generation.The Completion Engine provides semantic feedback on partial programs while the LLM supplies token probabilities.
  • Approach: Repilot prunes infeasible LLM-suggested tokens and directly completes valid continuations such as asEndTag without querying the LLM.These operations reduce invalid generation and can reduce the search space explored by the LLM.
  • Implementation: Repilot instantiates the framework with CodeT5-large and InCoder-6.7B and a Java Completion Engine based on Eclipse JDT Language Server.The two LLMs differ in architecture and size, supporting evaluation across distinct model configurations.
  • Evaluation: 66 Defects4J 1.2 single-hunk bugs and 50 Defects4J 2.0 single-line bugs were fixed, with 30 more combined fixes than the previous best baseline.The evaluation also reports improved patch validity and correctness with limited overhead.

2 BACKGROUND AND RELATED WORK

APR has progressed from handcrafted templates and learned translation models to LLM-based generation, but current LLM approaches still lack semantic guidance. Repilot addresses this gap by using a semantics-based Completion Engine to constrain the LLM search space during generation.

  • Code Completion: Code completion systems interactively suggest program constructs, including identifiers and library APIs, after the programmer’s caret position.The paper uses semantics-based Completion Engines rather than purely language-model-based completion.
  • Automated Program Repair: Traditional APR tools use handcrafted templates that target preset bug patterns but cannot generalize to new bug types.Template-based methods have achieved strong results among classic APR techniques.
  • Automated Program Repair: NMT-based APR translates buggy code into corrected code, but its training datasets may contain relatively few bug-fix types.This limits the diversity of fixes represented during learning.
  • Automated Program Repair: LLM-based APR achieves state-of-the-art bug-fixing results but mainly queries models as black boxes through beam search or sampling.Without generation guidance, LLMs may still produce invalid patches for the current code context.
  • Repilot: Repilot uses a semantics-based Completion Engine to guide and prune the LLM search space without domain-specific training.Its incremental analysis provides constraints during generation and can be combined with recent LLM-based APR techniques.

3 PRELIMINARIES

The preliminaries formalize static feasibility, Completion Engines, and LLMs as abstractions that support Repilot’s general framework. A strict Completion Engine provides semantically valid continuations or reports uncertainty, while an LLM assigns next-token probabilities.

  • Programming Languages: Repilot models programming languages with static specifications that determine whether complete and partial programs are valid or feasibly continuable.A partial program is feasible at a caret when some continuation can produce a statically valid program.
  • Completion Engines: A Completion Engine suggests continuations for a partially written program at a caret position.Its completion function may return unknown when the engine cannot determine suggestions from the code context.
  • Completion Engines: Strict Completion Engines guarantee that continuations outside their returned completion set yield infeasible programs.They should return unknown whenever they are unsure; an engine that always returns unknown is therefore trivially strict.
  • Large Language Models: The LLM abstraction covers encoder-decoder models and subsumes decoder-only models while also describing encoder-only generation from encoder outputs.The decoder maps generated token sequences to probabilities over the model vocabulary.

4 APPROACH

Repilot treats single-hunk repair as cloze-style generation and couples LLM token probabilities with Completion Engine feedback. It prunes infeasible tokens, actively completes valid prefixes, and uses memorization to reduce repeated engine queries while preserving soundness.

  • Cloze-Style Program Repair: Repilot represents single-hunk repair as a cloze task by replacing the buggy hunk with a masked span and generating its replacement.The framework assumes perfect fault localization and can be extended to multi-hunk bugs by infilling multiple replacement hunks.
  • Main Repair Loop: The main repair loop repeatedly feeds the current generation to the LLM, then uses GuidedPrune to select a feasible next token or terminate at the end token.The loop constructs a patch by replacing the buggy range with the generated hunk and advances the caret after each generation step.
  • Completion-Guided Search Space Pruning: GuidedPrune samples tokens from the LLM probability map, appends each candidate to the program, and prunes it when the Completion Engine returns no possible continuation.Pruned-token probabilities are set to zero before the search continues.
  • Optimization: Memorization stores rejected and accepted token information to reduce repeated Completion Engine analyses and speed patch synthesis.The memorization techniques do not change GuidedPrune’s behavior because they preserve its semantics while reducing engine-query frequency.
  • Active Completion: Active completion appends the Completion Engine’s shared completion prefix, enabling Repilot to advance without querying the LLM for every token.If the engine returns unknown, Repilot adds no completion tokens; otherwise it uses the available completion result to guide generation.
  • Soundness: Repilot’s pruning and active-completion procedures are sound: feasible programs are not missed, and tokens pruned by GuidedPrune produce infeasible programs.The overall algorithm therefore prunes the language model’s search space correctly, although soundness does not guarantee a valid patch every time.

5 EXPERIMENTAL SETUP

The evaluation studies Repilot through four research questions covering bug-fixing performance, compilation rate, component contributions, and generalizability across bug sets and models. Experiments use Defects4J, compare 19 APR baselines under perfect fault localization, and report correct-fix counts for two benchmark variants.

  • Research questions: Repilot is evaluated through questions on bug fixing, compilation rate, component contributions, and generalization.The study explicitly defines RQ1–RQ4 around these four aspects.
  • Datasets: The evaluation uses Defects4J 1.2 with 391 bugs and Defects4J 2.0 with 438 new bugs from additional Java projects.Defects4J provides buggy and patched source versions with developer test suites for validation.
  • Compared techniques: Repilot is compared with 19 traditional, NMT-based, and LLM-based APR baselines under perfect fault localization.The comparison includes AlphaRepair, six NMT-based tools, and twelve traditional tools.
  • Metrics: The reported correct-fix evaluation covers Defects4J 1.2 single-hunk bugs and Defects4J 2.0 single-line bugs.Table 1 is organized around correct fixes for these two benchmark settings.
  • Metrics: Plausible patches pass all tests, correct patches are semantically equivalent to developer patches, and compilation rate is the percentage of compilable generated patches.Correctness is determined by manually examining plausible patches.

6 RESULT ANALYSIS

Repilot fixes unique bugs that existing APR tools miss and uses semantic completion to construct difficult identifiers and valid continuations. Its results include unique fixes on both Defects4J versions and comparisons of correct-fix and compilation outcomes.

  • RQ1: Comparison with Existing Tools: Repilot fixes 7 Defects4J 1.2 bugs that no other evaluated baseline fixes.The result is shown in the comparison of unique bugs fixed by Repilot and other tools.
  • RQ1: Comparison with Existing Tools: Figure 6 presents correct-fix overlap among APR tools on Defects4J 1.2 using Venn diagrams.The figure compares correct fixes across Repilot, top-performing baselines, and other tools.
  • RQ1: Comparison with Existing Tools: Repilot uniquely fixes Closure-133 by actively completing rare identifiers such as unreadToken and NO_UNREAD_TOKEN.The Completion Engine supplies semantically valid continuations after the LLM generates identifier prefixes.
  • RQ1: Comparison with Existing Tools: On a Defects4J 2.0 unique bug, Repilot uses inferred type information to prune invalid String members and generate toLowerCase().The Completion Engine identifies the object type as String and removes incompatible continuations.

1 This is the top 200 rate for RewardRepair as it does not include top 1000

Ablation and generalization experiments show that pruning, memorization, and active completion improve patch validity and correctness while controlling overhead. These gains persist across Defects4J versions and model architectures, including CodeT5 and InCoder.

  • RQ3: Ablation Study: 63.4% compilation rate, 5.21% plausible percentage, 63 plausible fixes, and 42 correct fixes are achieved by full Repilot on Defects4J 1.2.Adding active completion improves over the pruned variant and yields the strongest reported ablation outcomes.
  • RQ3: Ablation Study: Pruning raises compilation rate from 43.2% to 60.7% and correct fixes from 37 to 41 compared with the base LLM.The comparison is between Repilot∅ and the pruning variant.
  • RQ3: Ablation Study: Memorization reduces pruning overhead from over 25% to around 10%, while active completion further reduces overhead to 7%.These reductions come from avoiding repeated Completion Engine queries and directly completing identifiers.
  • RQ4: Generalizability: On Defects4J 2.0, CodeT5 Repilot yields 18.1 percentage points more compilable patches, 3.0 points more plausible patches, 6 more plausible fixes, and 4 more correct fixes.The comparison is against the CodeT5 baseline Repilot∅ with a 7.4% overhead.
  • RQ4: Generalizability: With InCoder, Repilot produces more compilable and plausible patches and more plausible and correct fixes on both Defects4J versions.It adds 6 correct fixes on Defects4J 1.2 and 1 on Defects4J 2.0, with negligible overhead for the larger model.
  • RQ4: Generalizability: Overall, Repilot generalizes across Defects4J 1.2 and 2.0 single-hunk bugs and across CodeT5 and the larger InCoder model.The reported results support generalization across both bug sets and model scales.

7 LIMITATIONS

Repilot’s evaluation has important scope and interpretation limits. Its evidence covers a narrow language-and-model setting, and higher compilation rates do not necessarily yield proportionally more correct fixes.

  • Repilot’s effectiveness depends on a Completion Engine that provides useful guidance while remaining strict.
  • Balancing usefulness and strictness is generally more difficult in dynamically typed languages such as Python than in statically typed Java.
  • The evaluation covers only two LLMs, CodeT5 and InCoder, and one programming language, Java.
  • A higher compilation rate does not necessarily translate into a proportional increase in plausible and correct fixes.
  • The evaluation lacks strong empirical evidence that LLMs struggle to generate rare tokens or that Repilot solves this problem.
  • The evaluation limits Repilot’s demonstrated application to patch synthesis despite the claimed applicability to other code-generation tasks.

8 THREATS TO VALIDITY

The study identifies threats involving manual correctness assessment, possible training-data overlap, completion-engine soundness, reproducibility, and generalization beyond the evaluation dataset.

  • Correctness judgments require manually examining each plausible patch for semantic equivalence to the developer patch.
  • CodeT5’s GitHub training data may overlap with Defects4J, while InCoder’s undisclosed training data prevents explicit assessment of the same issue.
  • The modified Completion Engine requires manual inspection to guarantee soundness, creating a risk of false positives during pruning.
  • Reporting prior bug-fix results without reproducing them and running each experiment once may reduce result reliability and introduce statistical bias.
  • Repilot’s performance may not generalize to datasets beyond those used in the evaluation.

9 CONCLUSION

Repilot combines LLM-based patch synthesis with on-the-fly Completion Engine guidance to reduce invalid-token generation and search space. On Defects4J 1.2 and 2.0, it achieves state-of-the-art results while producing more valid and compilable patches with minimal overhead.

  • Repilot combines LLMs with on-the-fly Completion Engine guidance during autoregressive token generation.
  • The Completion Engine prunes invalid tokens and proactively completes partial programs, reducing the LLM’s search space.
  • Repilot achieves state-of-the-art results on subsets of Defects4J 1.2 and 2.0.
  • Repilot generates more valid and compilable patches than prior tools with minimal overhead compared with directly using LLMs for APR.
Loading 2309.00608v3…