PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical Knowledge Injection for Improving LLM-based Program Repair

Ramtin Ehsani, Esteban Parra, Sonia Haiduc, Preetha Chatterjee

arXiv:2506.24015v3cs.SE

TL;DR

LLM-based program repair can remain difficult when a fix depends on repository or project context beyond the buggy code. This paper incrementally injects bug, repository, and project knowledge, reaching a 79% fix rate with Llama 3.3 while finding that project-level gains are bug-type-specific.

  • Problem

    LLM-generated repairs may lack project constraints, dependencies, or intent, and prior context often centers on the buggy function and its surroundings,.

    Source: §I · p. 1, §II · p. 2

  • Method

    The framework incrementally adds bug-related, repository-level, and project-level context to prompts for unresolved bugs.

    Source: §I · p. 1, §VII · p. 11

  • Results

    79% (250/314) was Llama 3.3’s final fix rate, 23% above prior work; repository context helped all bug types, while project context further helped a subset.

    Source: §I · p. 2, §VII · p. 11

  • Takeaways & Limitations

    The results support progressively injecting context and indicate that different bug types benefit from different levels of information.

    Source: §VII · p. 11

    The findings may not generalize to all LLMs, and underrepresented bug categories may not yield broadly applicable insights.

    Source: §V · p. 10

Abstract

from arXiv · show

Prompting LLMs with bug-related context (e.g., error messages, stack traces) improves automated program repair, but many bugs still remain unresolved. In real-world projects, developers often rely on broader repository and project-level context beyond the local code to resolve such bugs. In this paper, we investigate how automatically extracting and providing such knowledge can improve LLM-based program repair. We propose a layered knowledge injection framework that incrementally augments LLMs with structured context. It starts with the Bug Knowledge Layer, which includes information such as the buggy function and failing tests; expands to the Repository Knowledge Layer, which adds structural dependencies, related files, and commit history; and finally injects the Project Knowledge Layer, which incorporates relevant details from documentation and previously fixed bugs. We evaluate this framework on a dataset of 314 bugs from BugsInPy using two LLMs (Llama 3.3 and GPT-4o-mini), and analyze fix rates across six bug types. By progressively injecting knowledge across layers, our approach achieves a fix rate of 79% (250/314) using Llama 3.3, a significant improvement of 23% over previous work. All bug types show improvement with the addition of repository-level context, while only a subset benefit further from project-level knowledge, highlighting that different bug types require different levels of contextual information for effective repair. We also analyze the remaining unresolved bugs and find that more complex and structurally isolated bugs, such as Program Anomaly and GUI bugs, remain difficult even after injecting all available information. Our results show that layered context injection improves program repair and suggest the need for interactive and adaptive APR systems.

I. INTRODUCTION

The paper investigates whether repository- and project-level context can improve LLM-based repair beyond local bug information. Its layered approach raises Llama 3.3’s fix rate to 79%, while the gains from project-level context vary by bug type.

I. INTRODUCTION

  • LLMs can struggle when bug fixes require information beyond the buggy function, and generated patches may miss project constraints, dependencies, or intent,.Prior findings report performance drops of more than 50% in such cases [6].Source: §I · p. 1
  • The study evaluates three context layers on 314 BugsInPy bugs, using Llama 3.3 and GPT-4o-mini and categorizing bugs into six types.The layers are Bug, Repository, and Project knowledge.Source: §I · pp. 1–2
  • 79% overall (250/314) was Llama 3.3’s final fix rate, a 23% improvement over previous work.The final rate follows repository- and project-level knowledge injection.Source: §I · p. 2
  • Repository-level context improved every bug type, while project-level context further improved only Program Anomaly, GUI, and Network bugs.The remaining unresolved bugs often involve user-facing behavior, implicit runtime conditions, or higher complexity.Source: §I · p. 2
  • The authors analyze unresolved bugs by type and complexity and provide bug-type annotations, prompt designs, and an evaluation pipeline for reproducibility.The analysis is intended to inform limitations and future work on context-aware APR.Source: §I · p. 2

II. MOTIVATING EXAMPLE

The motivating example argues that local bug facts can be insufficient when a fix depends on broader repository or project context. A Luigi bug illustrates how information from a related class changes the generated patch.

II. MOTIVATING EXAMPLE

  • Earlier context approaches added bug-related facts but achieved a maximum 56% fix rate on BugsInPy, leaving 137 of 314 bugs unresolved.The paper asks what broader context might help with the remaining bugs.Source: §II · p. 2
  • The authors hypothesize that repository structure, version history, and documentation contain context developers use but earlier approaches often overlook –.Prior context was often limited to the buggy function and its immediate surroundings.Source: §II · p. 2
  • In the Luigi example, Llama 3.3 fails with bug-related facts alone but produces the correct fix after receiving dependent class information from a related file.The fix disables the record task history attribute rather than modifying a hallucinated variable.Source: §II · pp. 2–3
  • Because excessive or unfiltered context may be detrimental, the paper motivates structured selection and prioritization of relevant information.The motivating example points to context spanning related files, dependencies, documentation, and past issue threads.Source: §II · p. 3

III. METHODOLOGY

The method progressively adds context for bugs that remain unresolved, moving from local bug facts to repository and project knowledge. The evaluation uses a manually annotated BugsInPy dataset spanning multiple projects and bug categories.

III. METHODOLOGY

  • The pipeline first supplies local bug information, then adds repository context for unresolved bugs, and finally adds project knowledge for bugs still unresolved.The layers include buggy functions and error messages; related files, dependencies, and commit history; then documentation and previously resolved bugs.Source: §III · p. 3
  • Progressive injection is designed to conserve tokens and computation on simpler bugs while scaling context only when necessary.The layered setup also supports analysis of which contextual signals benefit different bug types.Source: §III · p. 3

A. Dataset

  • The dataset contains 314 BugsInPy instances whose fixes change a single function; bug localization uses Function-granular Perfect Fault Localization.The model must still identify faulty lines within the localized function.Source: A. Dataset · p. 3
  • Two annotators reached Cohen’s Kappa 0.8 after refining the taxonomy, then independently labeled the remaining 214 data points.Their first round yielded 0.45; the second round used an additional 50 data points.Source: A. Dataset · p. 4
  • The taxonomy has nine categories, with six represented in the dataset; the sample spans 16 Python projects.Program Anomaly is the most frequent represented type, while Permission/Deprecation is the least; Database-Related, Security, and Test Code-Related bugs are absent.Source: A. Dataset · p. 4, Table I · p. 4

B. Prompting Strategy

The approach combines stepwise prompting with immediate bug information, then adds repository context for bugs still unresolved.

B. Prompting Strategy

  • Chain-of-Thought prompting guides the model to process context step by step, with code-only output enforced through triple-backtick formatting.The authors report consistent outputs across 200 runs on 10 bug instances per model.Source: B. Prompting Strategy · p. 4

RQ1: To what extent can LLMs resolve different types of bugs using only immediate bug knowledge?

  • The Bug Knowledge Layer supplies the buggy function, failing tests, error information, runtime and angelic variables, and a GitHub description as the baseline context.These information types are adopted from Parasaram et al. to evaluate subsequent layers.Source: RQ1: To what extent can LLMs resolve… · p. 4
  • The buggy function is extracted from the developer-provided fix commit, while failing tests, error messages, and stack traces provide evidence about the bug.Error outputs are captured by running failing tests after reverting each project to its pre-fix state.Source: RQ1: To what extent can LLMs resolve… · p. 4
  • Runtime variables record failing-execution values and types, while angelic variables give values that would make a failing test pass.Both are extracted from correct program versions, with at most three input-output cases per bug.Source: RQ1: To what extent can LLMs resolve… · pp. 4–5
  • GitHub issue or pull-request descriptions supply textual bug information, but 24 of 314 instances had neither linked source.The authors manually checked linked reports to ensure they described the bug without revealing its fix.Source: RQ1: To what extent can LLMs resolve… · p. 5
  • For bugs unresolved after the Bug Knowledge Layer, the Repository Knowledge Layer adds the buggy file, related files, and pre-fix repository history.The latest commit to the buggy function before the fix is used to suggest potential error sources.Source: RQ1: To what extent can LLMs resolve… · p. 5
  • Co-occurring files are identified from files frequently committed alongside the buggy file and serve as a heuristic for locating likely structural dependencies.They are not inserted directly into prompts.Source: RQ1: To what extent can LLMs resolve… · p. 5
  • Structural dependency extraction retrieves declarations invoked by the buggy function and relevant callers found within co-occurring files.Python AST and Jedi resolve referenced declarations; restricting caller searches to co-occurring files limits noise from widespread usages.Source: RQ1: To what extent can LLMs resolve… · p. 5

RQ3: How does injecting project-level knowledge further enhance LLM-based repair of different bug types?

For bugs unresolved after repository context, the Project Knowledge Layer injects retrieved documentation and knowledge from earlier resolved issues.

RQ3: How does injecting project-level knowledge further enhance LLM-based repair of different bug types?

  • The Project Knowledge Layer adds project documentation and developer knowledge from past resolved issues for bugs still unresolved after repository context.Its stated goal is to give the model a holistic view of project constraints and past repair strategies.Source: RQ3: How does injecting project-level… · p. 5
  • Documentation retrieval uses the buggy function as a query to select relevant chunks and extract intended behavior, usage, constraints, edge cases, and API guidance.The pipeline chunks documentation, embeds it with Sentence Transformers, indexes it in FAISS, and uses a QA Retrieval Agent.Source: RQ3: How does injecting project-level… · p. 5
  • Issue-history retrieval ranks earlier patched issues using an equal-weight average of textual and code similarity, then extracts relevant behaviors, fixes, and developer insights.Retrieved issues must predate the target bug's fix date.Source: RQ3: How does injecting project-level… · p. 5

RQ4: Which bug types remain challenging for LLMs, even after injecting all layers of contextual knowledge?

The section examines unresolved bugs, context availability, and code complexity, and specifies the baselines and evaluation measures used in the study.

RQ4: Which bug types remain challenging for LLMs, even after injecting all layers of contextual knowledge?

  • The unresolved-bug analysis checks whether repository and project information was available and usable, then compares complexity metrics for resolved and unresolved bugs.The comparison uses Mann–Whitney U tests and Cohen’s d because the data are not normally distributed.Source: RQ4: Which bug types remain… · pp. 5–6

D. Baselines

  • The baselines are Parasaram et al. and a variant that injects all three knowledge layers simultaneously for every bug.The latter tests progressive hierarchical injection against providing all context at once.Source: D. Baselines · p. 6

E. Evaluation Metrics

  • Pass@k estimates the chance that at least one of k queries succeeds, while #fixed counts bugs with a test-passing patch that is manually verified as correct.The study uses n = 10 and reports Pass@1, Pass@3, and Pass@5.Source: E. Evaluation Metrics · p. 6

IV. RESULTS AND DISCUSSION

The section frames RQ1 as asking how well LLMs resolve different types of bugs.

IV. RESULTS AND DISCUSSION

  • RQ1 asks to what extent LLMs can resolve different types of bugs.Source: §IV · p. 6

bugs using only immediate bug knowledge?

Repository-level context raises overall repair rates beyond bug-local prompting, and benefits both models across bug types. The section also shows how dependencies, related files, and commit history can resolve missing code-level context.

hance LLM-based repair of different bug types?

bugs using only immediate bug knowledge?

  • 78 bugs failed under both models with immediate bug context, marking a shared set of difficult cases for deeper context.Llama fixed 207 of 314 bugs (65%) with bug knowledge alone, compared with the previously reported 56%.Source: bugs using only immediate bug knowledge? · pp. 6–7

hance LLM-based repair of different bug types?

  • Repository context improved GPT-4o-mini and Llama 3.3 overall fix rates to 70% and 74%, respectively.The improvement counts varied by bug type, with Llama adding no Permission/Deprecation fixes.Source: hance LLM-based repair of different… · p. 7
  • Related files, structural dependencies, and commit history supplied missing clues in analyzed fixes, including valid attribute usage and class behavior.Examples include a Pandas attribute spelling fix and a Luigi scheduler-call fix.Source: hance LLM-based repair of different… · pp. 7–8
  • Both models improved across bug types with repository context, although Llama fixed no additional Permission/Deprecation bugs in this layer.The dataset contains only two Permission/Deprecation bugs.Source: hance LLM-based repair of different… · p. 7, Table III · p. 8

C. RQ3: How does injecting project-level knowledge further

Project-level knowledge adds documentation and past-issue context after repository context has been tried. It raises Llama 3.3's overall fix rate to 79%, but gains vary by bug type and depend on project materials.

C. RQ3: How does injecting project-level knowledge further

  • 79% (Llama 3.3) and 73% (GPT-4o-mini) overall fix rates followed project-level context, with 5% and 3% gains over the repository layer.The layer added 15 fixes for Llama and 7 for GPT-4o-mini; Pass@3 and Pass@5 improved by 2–3%.Source: enhance LLM-based repair of different… · p. 8
  • Project-level knowledge added fixes for Program Anomaly, Network, GUI, and Configuration bugs, but neither model gained fixes for Performance or Permission/Deprecation bugs.Llama added 10 Program Anomaly, 1 Network, 3 GUI, and 1 Configuration fix; GPT-4o-mini added 3, 1, and 3 for the first three types.Source: enhance LLM-based repair of different… · p. 8
  • Documentation and past issues provided project-specific API, intent, and usage constraints, with benefits dependent on how well projects maintain those materials.A documentation query supplied integer-format requirements relevant to a youtube-dl regex bug.Source: enhance LLM-based repair of different… · p. 8
  • Past issue threads offered relevant precedents for a Scrapy robots.txt KeyError and a configuration bug involving preservation of command arguments.The reported examples associate the context with correctly applied error handling and a fix direction.Source: enhance LLM-based repair of different… · pp. 8–9
  • Injecting all three knowledge layers simultaneously performed worse than hierarchical injection; GPT-4o-mini fixed 14% fewer bugs than with bug-level context alone.Llama's simultaneous-context fix rate was 65%, matching its bug-level result and trailing its hierarchical configurations.Source: enhance LLM-based repair of different… · p. 9

D. RQ4: Which bug types remain challenging for LLMs, even

After all knowledge layers, unresolved bugs clustered around shared hard cases, missing injected context, and higher complexity. Program Anomaly, Network, and GUI bugs remained especially challenging, with feedback-driven repair suggested for some cases.

after injecting all layers of contextual knowledge?

  • 85 GPT-4o-mini bugs and 64 Llama bugs remained unresolved; 99 were unique across models, with 50 overlapping.Program Anomaly accounted for 58 GPT-4o-mini and 45 Llama unresolved bugs.Source: after injecting all layers of… · p. 9
  • 39 of 99 unresolved bugs had all five added context pieces, while Co-occurring Files and Structural Dependencies were most often absent.Among the 50 bugs unresolved by both models, only 20 had all five pieces of information.Source: Table IV · p. 9, after injecting all layers of… · p. 9
  • Program Anomaly, Network, and GUI bugs remained most challenging, often requiring deeper data-flow or user-facing reasoning and interaction or feedback beyond static prompting.Over 66% of unresolved bugs lacked one or more repository- or project-level context pieces.Source: after injecting all layers of… · pp. 9–10
  • Unresolved bugs also had higher Cyclomatic Complexity and Lines of Code, consistent with greater difficulty reasoning about branching, nested control structures, and dense logic.The discussion identifies deeper semantic understanding and multi-step reasoning as limitations of current LLMs.Source: after injecting all layers of… · p. 10
  • Unresolved bugs scored higher across complexity metrics; Lines of Code differed significantly (p-value=0.02), while Cyclomatic Complexity and Halstead Difficulty were borderline (p-values=0.05 and 0.06).The Maintainability Index was higher for fixed bugs, but its difference was not statistically significant.Source: after injecting all layers of… · p. 9, Table V · p. 10

V. THREATS TO VALIDITY

The study identifies threats from manual bug-type annotation, undisclosed proprietary-model training data, and limits to generalization across models and bug categories.

V. THREATS TO VALIDITY

  • Manual bug-type annotations may threaten construct validity, although an iterative coding process and Cohen’s Kappa of 0.8 indicated strong annotator agreement.Annotators had at least three years of programming and qualitative research experience.Source: §V · p. 10
  • Undisclosed proprietary-model training data prevent verifying whether specific bugs or repository content appeared in pretraining, and the injected knowledge layers are not exhaustive.The study focuses on relative layer effectiveness and evaluates two architecturally distinct models to reduce leakage concerns.Source: §V · p. 10
  • Findings may not generalize to all LLMs, and imbalanced bug types limit broad conclusions for underrepresented categories such as Permission/Deprecation.The study evaluated Llama 3.3 and GPT-4o-mini and observed consistent performance trends across layers.Source: §V · p. 10

VI. RELATED WORK

Prior LLM-based repair work explored local-context prompting, richer bug signals, and agentic workflows, while this paper examines layered context across bug types. Its conclusion reports gains from incremental injection and identifies unresolved complex or isolated bugs as a continuing challenge.

VI. RELATED WORK

  • Earlier systems used local bug context, while later work incorporated fault localization, interactive failing-test prompts, identifiers,, stack traces,, bug reports, and blame history [10].MANIPLE selects bug-related facts for prompts and reported gains over naive prompting, especially with limited context windows.Source: §VI · pp. 10–11
  • Agentic and iterative APR systems search repositories, query repeatedly, and refine or validate patches; OpenHands [70] and AutoCodeRover reported gains on SWE-Bench.Self-iterative prompting yielded up to 78% improvement in plausible patch generation [71], while other work incorporated repository-level knowledge graphs [72].Source: §VI · p. 11
  • The paper frames prior work as limited by immediate bug context, uniform treatment of bugs, or opaque agentic methods, and presents transparent layered injection as its alternative.The framework incrementally adds bug-, repository-, and project-level context for analysis across bug types.Source: §VI · p. 11

VII. CONCLUSION AND FUTURE WORK

  • The framework iteratively injects bug-, repository-, and project-level context into prompts to support more context-aware repair across bug types.The conclusion reports consistent gains in #fixed and Pass@k across layers for Llama 3.3 and GPT-4o-mini.Source: §VII · p. 11
  • Llama achieved a 79% fix rate, a 23% improvement over prior work; all bug types improved with repository context, but only some benefited further from project context.Injecting all context at once underperformed the layered strategy for every bug type.Source: §VII · p. 11
  • Remaining failures were more complex, structurally isolated, or dependent on user interactions and edge-case reasoning, especially for Program Anomaly, Network, and GUI bugs.The authors propose future integration with agentic workflows and feedback-driven repair, as well as project-specific fine-tuning.Source: §VII · p. 11
Loading 2506.24015v3…