PaperlayerView exact revision on arXiv

Source-linked AI summary

When Large Language Models Confront Repository-Level Automatic Program Repair: How Well They Done?

Yuxiao Chen, Jingzheng Wu, Xiang Ling, Changjiang Li, Zhiqing Rui, Tianyue Luo, Yanjun Wu

arXiv:2403.00448v1cs.SE

TL;DR

LLM repair evaluations have focused on the function or file containing a bug, leaving performance on repository-level repair underexplored. The paper introduces RepoBugs and RLCE, a repository-context extraction method; across three mainstream LLMs, RLCE improves repair performance by up to 160% over the preliminary method.

  • Problem

    LLM APR evaluations focus on the function or file containing a bug, while performance on repository-level repair remains underexplored.

    Source: §1 · pp. 1–2

  • Method

    The authors introduce RepoBugs, a benchmark of 124 repository-level bugs, and RLCE, which parses repository structure, filters fragments, and adds auxiliary repair context.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    RLCE significantly improves repository-level repair for three mainstream LLMs, with a maximum improvement of 160% over the preliminary method.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The RepoBugs evaluation and analysis of context sources, error types, and prompt length provide insights for future repository-level repair research.

    Source: §8 · p. 9

    RepoBugs primarily focuses on interface-inconsistency errors, so its coverage of repository-level bug types is limited.

    Source: §6 · p. 8

Abstract

from arXiv · show

In recent years, large language models (LLMs) have demonstrated substantial potential in addressing automatic program repair (APR) tasks. However, the current evaluation of these models for APR tasks focuses solely on the limited context of the single function or file where the bug is located, overlooking the valuable information in the repository-level context. This paper investigates the performance of popular LLMs in handling repository-level repair tasks. We introduce RepoBugs, a new benchmark comprising 124 typical repository-level bugs from open-source repositories. Preliminary experiments using GPT3.5 based on the function where the error is located, reveal that the repair rate on RepoBugs is only 22.58%, significantly diverging from the performance of GPT3.5 on function-level bugs in related studies. This underscores the importance of providing repository-level context when addressing bugs at this level. However, the repository-level context offered by the preliminary method often proves redundant and imprecise and easily exceeds the prompt length limit of LLMs. To solve the problem, we propose a simple and universal repository-level context extraction method (RLCE) designed to provide more precise context for repository-level code repair tasks. Evaluations of three mainstream LLMs show that RLCE significantly enhances the ability to repair repository-level bugs. The improvement reaches a maximum of 160% compared to the preliminary method. Additionally, we conduct a comprehensive analysis of the effectiveness and limitations of RLCE, along with the capacity of LLMs to address repository-level bugs, offering valuable insights for future research.

1 INTRODUCTION

The paper addresses the gap in evaluating LLM repair of repository-level bugs and introduces RepoBugs and RLCE to study the task. RLCE supplies more precise repository context and improves repair performance over the preliminary method.

1 INTRODUCTION

  • RepoBugs provides 124 typical repository-level bugs from open-source GitHub repositories for evaluating popular LLMs.The paper identifies the lack of a suitable repository-level benchmark as a central challenge.Source: §1 · p. 2
  • 22.58% was GPT3.5’s repair rate on RepoBugs when given only the function containing the error as context.The paper contrasts this with GPT3.5’s function-level results in related studies and notes that repository context produced the correct repair in its example.Source: §1 · p. 2
  • RLCE improves repository-level repair across three mainstream LLMs, with a maximum improvement of 160% over the preliminary method.The contribution summary reports over 100% improvement in repair rates for all experimental models.Source: §1 · p. 2
  • RLCE parses repository structure, filters code fragments, and adds auxiliary information to provide more precise repair context.The authors argue that repository context can be redundant, imprecise, and too large for prompt limits, while similarity-based slices may not match what programmers consult when fixing bugs.Source: §1 · p. 2

2 BENCHMARK CONSTRUCTION

RepoBugs is a manually constructed benchmark of repository-level interface-inconsistency bugs, assembled from selected open-source Python repositories. Its cases are generated through expert-designed disruptions between interacting functions.

2 BENCHMARK CONSTRUCTION

  • The benchmark requirements specify repository context, repairs that need that context, and repositories created after the LLM training-data collection date.These criteria target repository-level repair and reduce the risk of evaluating on repositories already seen during training.Source: §2 · p. 3
  • RepoBugs contains 124 repository-level interface-inconsistency bugs, a type the authors identify as the most prevalent repository-level issue.The authors describe the benchmark’s limited error-type selection as representative.Source: §2 · p. 3

2.1 Dataset Collection

  • The authors selected 11 GitHub repositories created after October 1, 2021, with at least 2,000 stars, no more than 1 MB, and at least four Python files.The collection was restricted to Python for conceptual validation and to reduce potential training-data leakage.Source: §2.1 · p. 3

2.2 Dataset Generation

  • Experts manually introduced six interface-inconsistency disruption types between main and context functions, keeping each change to one line and avoiding syntax errors.The types cover inconsistencies in parameter and return counts, order, or requirements.Source: §2.2 · p. 3

3 PROPOSED FRAMEWORK

RLCE retrieves repository context from the bug location, then processes the retrieved code and combines it with prompt templates. Its retriever models repository structure and selects code related to the error; the composer adds semantic information and slices caller code to limit redundancy.

3.1 Overall Framework

  • RLCE retrieves repository code fragments related to the repair task, then processes them with prompt-strategy templates to form the final prompt context.The framework describes repair as generating fixed code from the repository context and the function containing the error.Source: §3.1 · p. 3, §3.3 · p. 5

3.2 Context Retriever

  • The context retriever parses repository structure into a tree and uses error localization to select relevant code segments.Its tree represents directories, files, classes, functions, and global variables, with cross-file references marked on file nodes.Source: §3.2 · pp. 3–4
  • The retriever identifies Error-Invoking Functions (EIF) at the error location and searches four sources: their definitions and callers, the Error Function, and its callers.A source may return null when absent; the retrieved collection contains segments from these four sources.Source: §3.2 · p. 4

3.3 Prompt Composer

  • The prompt composer adds signatures and summaries to EIF definitions, slices EIF and Error Function callers around invocations, and includes the Error Function directly.EIF caller slices preserve five lines of context on each side of the invocation; the design principles can extend beyond the Python programs primarily studied.Source: §3.3 · p. 5

4 EVALUATION SETUP

The evaluation uses GPT-3.5, PaLM2, and GPT-4 with zero-shot, one-shot, and chain-of-thought prompting. It compares RLCE with function-only and slice-similarity context baselines, and experts manually assess response quality and repair outcomes.

4.1 Models

  • The experiments select GPT-3.5 Turbo, PaLM2 text bison-001, and GPT-4 gpt-4-0613, accessing the models through APIs because they are not open-source.GPT-3.5 Turbo supports a 4K-token sequence; the selected PaLM2 and GPT-4 models support up to 8K input tokens.Source: §4.1 · p. 5

4.2 Prompt Generation

  • The prompting strategies are zero-shot with Simple or Detail instructions, one-shot with a repair example, and zero-shot CoT decomposing repair into three logical steps.CoT prompts request an error explanation, a repair strategy, and the resulting fixed code.Source: §4.2 · p. 5

4.3 Compared Repair Baselines

  • The baselines are a preliminary method supplying only the erroneous function and a slice-similarity method selecting five repository segments most similar to the error segment.The similarity baseline uses a 10-unit slicing window, sparse word-bag vectors, and Jaccard similarity.Source: §4.3 · pp. 5–6

4.4 Evaluation Metrics

  • Two Python-programming experts manually score relevance, format, repair correctness, and—specifically for CoT—explanation correctness.They first align on criteria, evaluate independently, and resolve discrepancies through discussion and reassessment.Source: §4.4 · p. 6

5 RESULTS AND ANALYSIS

RLCE improves repair rates over the preliminary and slice-similarity baselines, with GPT3.5 reaching the largest reported improvement. The analyses also examine prompt strategy, context sources, error types, and prompt length, finding that useful context is not simply the longest context.

5.1 RLCE Outperforms the Baselines

  • Across experiments with three models, slice-similarity repair rates never exceed RLCE’s; on GPT4, improvement over preliminary is 100% for RLCE and 20% for slice-similarity.Source: §5.1 · p. 6

5.2 Impact of Prompt Strategies on RLCE

  • One-shot prompting improves GPT3.5’s repair rate by over 37% compared with the Simple zero-shot strategy, while GPT4 is described as optimal when accuracy and response stability are considered together.GPT3.5 shows the largest prompting-strategy fluctuation among the models.Source: §5.2 · p. 6

5.3 Correct Explanation is Important for CoT

  • CoT repair success is associated with correct explanations; for GPT4, every successfully repaired case also had an accurate explanation.The authors hypothesize that incorrect interpretations can interfere with repair and suggest more capable models or more accurate explanations may improve repair rates.Source: §5.3 · pp. 6–7, Figure 3 · p. 7

5.4 Validity of Context Sources

  • Complete context achieves the highest repair accuracy in the ablation experiments, while removing EIF definitions significantly reduces repair rates and extra semantic information does not noticeably improve success.GPT3.5's repair rate drops by more than half when EIF definitions are absent.Source: §5.4 · pp. 7–8, Table 3 · p. 7

5.5 Error Types Affect Repair Effectiveness

  • GPT-3.5 and GPT-4 perform worse on ORV and CRV errors, which require semantic understanding of interface parameter meanings or order and function behavior.The paper contrasts these errors with parameter-count mismatches that may be identifiable through syntax checks.Source: §5.5 · p. 8

5.6 Long Prompt vs. Short Prompt

  • Repair accuracy declines as prompt length increases for both GPT3.5 and GPT4, leading the authors to favor precise context with a higher density of useful information.The authors note that longer prompts may add information while also distracting the model and increasing repair difficulty.Source: §5.6 · p. 8

6 THREATS TO VALIDITY

The validity of the findings is bounded by the selected models and experimental settings, and by RepoBugs’ dataset scope. The authors note that results may vary with model characteristics, repository scale, and error type.

6 THREATS TO VALIDITY

  • Model differences in parameter count, training data, and fine-tuning may affect repair rates; the experiments use GPT-3.5, GPT-4, and PaLM2 at temperature 0.The authors also note that higher temperatures can make outputs unstable and potentially influence results.Source: §6 · p. 8
  • Results may be affected if dataset repositories predate the training-data cutoff, since models may already have learned them.Source: §6 · p. 8
  • The experiments focus primarily on interface inconsistency, so generalization to datasets covering other repository-level error types may vary.Larger and more complex repositories may also increase context-retrieval cost and difficulty, affecting reliability.Source: §6 · p. 8

7 RELATED WORK

Related work spans traditional search- and template-based repair, deep-learning approaches, and recent LLM evaluations. The supplied figures describe analyses of repair accuracy by error category and prompt length.

7 RELATED WORK

  • Traditional APR includes genetic-programming repair with test-case validation, template-based repair patterns, and searches for repair components in code repositories.Source: §7 · p. 8
  • Deep-learning repair methods learn from repair data, with examples using sequence-to-sequence models, copying mechanisms, or context-aware targeted search.Source: §7 · p. 8
  • Figure 4 organizes GPT-3.5 and GPT-4 repair accuracy by six error categories, while Figure 5 relates prompt length to accuracy across six strategies.Figure 4 uses colored bars for methods and gives category sample counts; Figure 5 uses bars for average prompt tokens and a line for repair accuracy.Source: Figure 4 · p. 9, Figure 5 · p. 9
  • Prior LLM work compared ten code language models with four deep-learning APR techniques across four benchmarks and reported competitive repair abilities.Source: §7 · p. 9

8 CONCLUSION

The paper evaluates LLMs on repository-level repair, introduces RepoBugs and RLCE, and reports improved performance with RLCE. It also analyzes context sources, error types, and prompt length.

8 CONCLUSION

  • RLCE significantly enhances LLM performance on repository-level repair tasks in RepoBugs.Source: §8 · p. 9
  • The paper introduces RepoBugs, a benchmark, and RLCE, a context-extraction method using repository-structure parsing and relevant-context retrieval.Source: §8 · p. 9
  • The analysis examines experimental results across context sources, error types, and prompt length.Source: §8 · p. 9

A.1 Prompt design

The prompt design combines task instructions, an optional one-shot example, RLCE-retrieved context, and the error function. Figure 7 presents instructions for each prompt strategy.

A.1 Prompt design

  • Figure 6 depicts a prompt composed, from top to bottom, of task instructions, a one-shot example, RLCE-retrieved context, and the error function.Source: §A.1 · p. 10, Figure 6 · p. 10
  • The one-shot prompt strategy includes an example between the task instruction and retrieved context.Source: §A.1 · p. 10
  • Figure 7 presents the designed instructions for each prompt strategy.Source: §A.1 · p. 10

A.2 Case Study

The case study illustrates repository-level repair through multiple valid fixes and an example in which GPT-4 repairs a parameter-order bug using RLCE-provided context. The accompanying figures cover prompt strategies, repair-method diversification, and Java repair.

A.2 Case Study

  • Both the manual fix and GPT-3.5's repair correct a missing returned parameter, despite using different receiving-variable names.The repository's original fix used “_”, while both repairs used “mask_C”.Source: §A.2.1 · p. 10

A.2.1 Diverse repair methods.

  • GPT-4 successfully repairs a bug caused by incorrect function-call parameter order using repository-level context provided by RLCE.The example is presented as an application of RLCE to Java program repair.Source: §A.2.2 · p. 10

A.2.2 RLCE for other program languages.

Loading 2403.00448v1…