Source-linked AI summary
RepairLLaMA: Efficient Representations and Fine-Tuned Adapters for Program Repair
André Silva, Sen Fang, Martin Monperrus
TL;DR
Prior fine-tuning for program repair often used naive code representations, while full-parameter tuning can be problematic for limited repair data. RepairLLaMA combines repair-specific representations with a parameter-efficient adapter and reports results across three Java repair benchmarks that outperform strong baselines.
Problem
Prior program-repair fine-tuning used naive code representations and full-parameter tuning, overlooking fault localization and posing overfitting concerns when training data are limited.
Source: §I · p. 1Method
RepairLLaMA combines repair-specific code representations, including fault-localization signals, with LoRA-trained plug-and-play repair adapters for an underlying LLM.
Source: §I · p. 1, A. Overview · p. 2Results
RepairLLaMA correctly fixes 144 Defects4J bugs, 109 HumanEval-Java bugs, and 20 GitBug-Java bugs, outperforming strong baselines including GPT-3.5 and GPT-4.
Source: §I · p. 2, §VII · p. 13Takeaways & Limitations
The paper reports that parameter-efficient fine-tuning outperforms full fine-tuning of CodeLlama-7B across three benchmarks, while DeepSeek-Coder-6.7B shows similar improvements regardless of fine-tuning method.
Source: B. Results of RQ2 (Parameter-Efficient… · p. 10Three Defects4J samples with a patch function also found in Megadiff were excluded from evaluation; HumanEval-Java and GitBug-Java were constructed after Megadiff.
Source: D. Threats to Validity · p. 12
Abstract
from arXiv · showhide
Automated Program Repair (APR) has evolved significantly with the advent of Large Language Models (LLMs). Fine-tuning LLMs for program repair is a recent avenue of research, with many dimensions which have not been explored. Existing work mostly fine-tune LLMs with naive code representations and does not scale to frontier models. To address this problem, we propose RepairLLaMA, a novel program repair approach that 1) identifies optimal code representations for APR with fine-tuned models, and 2) pioneers state-of-the-art parameter-efficient fine-tuning technique (PEFT) for program repair. This results in RepairLLaMA producing a highly effective `program repair adapter' for fixing bugs with AI. Our experiments demonstrate the validity of both concepts. First, fine-tuning adapters with program repair specific code representations enables the model to use meaningful repair signals and produce better patches. Second, parameter-efficient fine-tuning helps fine-tuning to converge and clearly contributes to the effectiveness of RepairLLaMA in fixing bugs outside the fine-tuning data distribution. Overall, RepairLLaMA correctly fixes 144 Defects4J v2, 109 HumanEval-Java, and 20 GitBug-Java bugs, outperforming all baselines.
I. INTRODUCTION
RepairLLaMA addresses limits in fine-tuning for program repair by combining repair-specific code representations with a parameter-efficient repair adapter. Its experiments report strong results across three benchmarks and show the approach’s core design choices are effective.
I. INTRODUCTION
- The introduction presents fine-tuning as a way to learn repair patterns and task-specific representations, while noting its computational and training-data demands.Prompt- and agent-based approaches are described as relying on ad hoc workflows and being bounded by context-window size.Source: §I · p. 1
- Prior fine-tuning work used naive representations and full-parameter updates, overlooking fault localization and risking overfitting when repair training data are limited.The paper frames its goal as devising efficient fine-tuning techniques for program repair, focused on code representations and adapters.Source: §I · p. 1
- RepairLLaMA combines repair-specific representations with LoRA to train a smaller repair adapter intended to help prevent overfitting.The representations incorporate fault-localization signals and support multi-location bugs.Source: §I · pp. 1–2
- 144 Defects4J v2, 109 HumanEval-Java, and 20 GitBug-Java bugs were correctly fixed, outperforming strong baselines including GPT-3.5 and GPT-4.The paper also reports state-of-the-art fine-tuning performance across all three benchmarks.Source: §I · pp. 1–2
A. Overview
- Its three-stage pipeline selects the base model, representations, and dataset; trains an approximately 4M-parameter adapter; then applies it to real-world bugs.The adapter is described as a plug-and-play extension that transforms a tailored buggy-code input into a patch output.Source: A. Overview · p. 2
B. Target Bugs
The paper targets functional Java bugs and treats bug scope, base-model selection, and code representation as important fine-tuning design choices.
B. Target Bugs
- The target is functional Java bugs with at least one failing test, fixed within a single function, including bugs requiring changes at multiple locations.The approach makes no assumption about the length of the method under repair.Source: B. Target Bugs · p. 2
C. Choice of the Initial LLM
- The initial LLM criteria are public availability, code-focused pretraining, an infilling objective, and support from an off-the-shelf parameter-efficient fine-tuning library.The paper motivates infilling as enabling code synthesis from context both before and after the repair location.Source: C. Choice of the Initial LLM · p. 2
D. Choice of Code Representations
- Code representation is treated as a key design dimension because it can affect model effectiveness and is tailored here to program repair’s input and output needs.The section introduces and motivates task-specific input and output representations.Source: D. Choice of Code Representations · p. 2
1) Representation of Fault Localization:
RepairLLaMA represents fault localization as a suspicious region rather than a single line, then explores input and output formats tailored to program repair.
1) Representation of Fault Localization:
- Fault localization as a span can cover multi-location repairs, unlike a single suspicious line; exhaustively listing suspicious regions within a method has worst-case O(n^2) cost.The span runs from the beginning to the end of the suspicious region.Source: 1) Representation of Fault Localization · p. 3
2) Input Representation Space:
- Four input variants range from pretraining-familiar code-only IR1 to fault-localized comments IR2 and infilling IR3/IR4, with IR3 masking buggy code that may help generate a fix.IR1 lacks fault localization; IR3 shortens inputs and uses an infilling scheme, while IR4 retains buggy code in comments alongside the infilling token.Source: 2) Input Representation Space · p. 3
3) Output Representation Space:
- Output choices include full-function OR1 and shorter fixed-chunk OR2, which requires an input representation that includes fault localization.OR1 may produce a much longer sequence than the actual changes; OR2 outputs only the fixed chunk.Source: 3) Output Representation Space · p. 4
- OR3 and OR4 output contextual diffs; OR4 uses less context than OR3 but remains longer than OR2 and requires the model to locate where the patch applies.OR3 uses three lines of context, while OR4 uses one line.Source: 3) Output Representation Space · p. 4
4) Input/Output Representation Pairs:
The paper pairs input and output representations around fault localization, then selects fine-tuning data and uses LoRA to train a compact repair adapter. These choices aim to retain repair-specific signals while reducing training cost.
4) Input/Output Representation Pairs:
- The first three pairs omit fault localization and share IR1 input, while varying output between the full fixed function and two diff formats.Their key differences are output length and format.Source: 4) Input/Output Representation Pairs · p. 5
- IR2xOR2, IR3xOR2, and IR4xOR2 include fault-localization information, paired with OR2’s replacement-chunk output to minimize output length.IR3xOR2 aligns most closely with pretraining because it uses infilling; IR2 uses delimiter comments, while IR4 keeps buggy code as comments.Source: 4) Input/Output Representation Pairs · p. 5
E. Choice of Fine-Tuning Dataset
- Dataset curation weighs task relevance, similarity to target bugs, size versus quality, diversity, and legal and ethical considerations.The passage notes that a smaller high-quality dataset may be more beneficial than a larger low-quality one.Source: E. Choice of Fine-Tuning Dataset · p. 5
F. Program Repair Adapters for LLMs
- LoRA updates the query and value projection matrices in each self-attention layer, with inference weights formed as W = W0 + BA.W0 is the frozen pretrained weight; B and C are lower-rank matrices whose product supplies the update.Source: F. Program Repair Adapters for LLMs · p. 5
- LoRA freezes the pretrained weights and adds trainable low-rank matrices; at rank 8, the adapter trains 0.39% of the parameters required by full fine-tuning.The trained adapter is substantially smaller than the original model and can be trained on a single GPU.Source: F. Program Repair Adapters for LLMs · p. 5
G. Inference Time
The implementation evaluates three Java benchmarks, with the evaluation restricted to single-function bugs, and uses CodeLLaMA-7B with curated Megadiff training data. At inference, a fault-localized representation and the repair adapter generate candidate patches for plausibility and correctness checks.
G. Inference Time
- Inference feeds the buggy program through fault localization and an APR-specific representation, then uses the initial model and LoRA adapter to generate candidate patches for checking.The checking uses off-the-shelf plausibility and correctness techniques.Source: G. Inference Time · p. 5
A. Research Questions
- The study asks which code representation works best, how parameter-efficient compares with full fine-tuning, and how RepairLLaMA compares with ChatGPT-based repair.Inference uses beam search with a beam size of 10, producing 10 candidate patches per bug.Source: A. Research Questions · p. 5, B. Implementation · p. 6
B. Implementation
- The fine-tuning data comes from Megadiff function pairs, filtered for single-function changes, duplicates, and samples under 1024 tokens, yielding 30,000–50,000 pairs.The selected initial model is CodeLLaMA-7B.Source: B. Implementation · p. 6
- The three Java benchmarks are Defects4J, HumanEval-Java, and GitBug-Java, and evaluation is scoped to single-function bugs.GitBug-Java’s bugs postdate the training-data cutoffs of all models used in the experiments.Source: B. Implementation · p. 6
C. Patch Assessment
Patch assessment uses four metrics, including structural matching and expert-reviewed semantic equivalence. The RQ1 methodology compares six code-representation pairs using generated patches and pairwise significance tests.
C. Patch Assessment
- Plausible, exact-match, AST-match, and semantic-match patches are the four assessment metrics, with higher scores indicating better performance.Plausible patches pass all test cases; exact-match patches reproduce the developer patch textually, while semantic matches are manually assessed by an expert.Source: C. Patch Assessment · p. 6
- AST matching compares abstract syntax trees of plausible and reference patches, making assessment insensitive to formatting and more scalable than manual correctness checks.The process uses AST differencing to identify discrepancies.Source: C. Patch Assessment · p. 6
D. Methodology for RQ1
- RQ1 evaluates six realistic representation pairs by generating 10 patches per bug and uses McNemar tests on semantic-match outcomes for pairwise comparisons.The null hypothesis is that the two models’ marginal repair outcomes are indistinguishable.Source: D. Methodology for RQ1 · p. 6
E. Methodology for RQ2
RQ1 finds that IR4xOR2 delivers the strongest reported repair results, while representation signals and benchmark distributions shape performance. The supplied RQ2 and RQ3 passages describe comparison designs rather than their outcomes.
E. Methodology for RQ2
- RQ2 compares parameter-efficient and full-parameter fine-tuning on CodeLLaMA-7B and deepseek-coder-6.7b-base, alongside Incoder-6B and RAP-Gen benchmarks.The motivation frames parameter-efficient fine-tuning as a computational-cost/performance trade-off and notes the risk of overfitting when full fine-tuning uses small datasets.Source: E. Methodology for RQ2 · p. 7
F. Methodology for RQ3
- RQ3 zero-shot prompts GPT-3.5-Turbo and GPT-4 for 10 patches per bug, providing the same code context and fault-localization information as RepairLLaMA.The prompt asks for the fixed code chunk replacing <FILL_ME>.Source: F. Methodology for RQ3 · p. 7
A. Results of RQ1 (Code Representations for Fine-Tuning)
- Representation pairs without fault localization perform considerably worse, while the authors identify fault-localization signals as crucial for repair.The authors also state that fine-tuning is needed to build knowledge of these signals because pretraining objectives lack them.Source: A. Results of RQ1 (Code… · pp. 7–8
- 144 Defects4J v2, 109 HumanEval-Java, and 20 GitBug-Java bugs were correctly repaired by IR4xOR2, the reported state-of-the-art model.The table covers repair results by representation across test benchmarks and its caption reports significant improvement over baseline.Source: Table II · p. 8, A. Results of RQ1 (Code… · p. 9
- HumanEval-Java is an exception: fine-tuned IR3xOR2 performs worse than its baseline, which the authors hypothesize relates to its different bug and fix distribution.HumanEval-Java’s manually engineered tasks differ from the Java-project diffs in Megadiff; the authors say IR4xOR2 alleviates forgetting to some degree.Source: A. Results of RQ1 (Code… · p. 8
- IR4xOR2 repairs 195 plausible Defects4J bugs (+64), 124 exact-match bugs (+72), 125 AST-match bugs (+55), and 144 semantic-match bugs (+61) over baseline.The authors attribute this performance to buggy code in the input, which they say provides information for understanding the problem and generating a patch.Source: A. Results of RQ1 (Code… · p. 8
- IR4xOR2 significantly differs from the other models in statistical tests, except IR2xOR2, for which the test does not reject the null hypothesis.The reported significant comparisons have p ≤ 1.27e-04; the IR4xOR2–IR2xOR2 comparison has p ≈ 0.545.Source: A. Results of RQ1 (Code… · p. 8
- RepairLLaMA correctly repairs 51 multi-location bugs, including examples requiring simultaneous edits at distant locations.Math-86 has two error sections more than 20 lines apart, and the HumanEval-Java example also demonstrates multi-location editing.Source: A. Results of RQ1 (Code… · pp. 8–9, Figure 5 · p. 9, Figure 6 · p. 9
B. Results of RQ2 (Parameter-Efficient Fine-Tuning vs. Full Fine-Tuning)
RepairLLaMA’s parameter-efficient fine-tuning outperforms full fine-tuning for CodeLlama across three benchmarks, while results for DeepSeek vary by benchmark. Comparisons with prior repair systems also report gains under realistic fault-localization conditions.
B. Results of RQ2 (Parameter-Efficient Fine-Tuning vs. Full Fine-Tuning)
- LoRA optimizes an approximately 4M-parameter adapter rather than the approximately 7B-parameter LLM, reducing the trainable parameter count by approximately 1600x.The paper states this can reduce GPU-memory needs and potentially reduce overfitting.Source: B. Results of RQ2 (Parameter-Efficient… · p. 9
- RepairLLaMA plausibly repairs 49 (195 vs. 146) more Defects4J bugs and repairs 41 (125 vs. 84) more under AST Match than full fine-tuning.The semantic-match comparison is 144 versus 98 correctly repaired bugs.Source: B. Results of RQ2 (Parameter-Efficient… · p. 9
- 46 (144 vs. 98) more Defects4J bugs are correctly repaired by RepairLLaMA under semantic match than by full fine-tuning, while it also wins on HumanEval-Java and GitBug-Java.The table covers repair effectiveness on Defects4J v2, HumanEval-Java, and GitBug-Java.Source: B. Results of RQ2 (Parameter-Efficient… · p. 9, Table III · p. 10
- With DeepSeek-Coder-6.7B, both fine-tuning methods improve over the base model, but full fine-tuning leads on Defects4J and GitBug-Java while LoRA leads on HumanEval-Java.The best fine-tuned model repairs 34 more Defects4J, 24 more HumanEval-Java, and 6 more GitBug-Java bugs than the base model.Source: B. Results of RQ2 (Parameter-Efficient… · p. 10
- Against Jiang et al., RepairLLaMA repairs 25 more Defects4J and 39 more HumanEval-Java bugs; against RAP-Gen, it repairs 19 more Defects4J v2 bugs with 10x fewer patches.The Jiang et al. comparison uses the same Defects4J dataset and assumes perfect bug localization; RAP-Gen generates 100 patches per bug and uses perfect line-level localization.Source: B. Results of RQ2 (Parameter-Efficient… · p. 10
- RepairLLaMA fixes two distant buggy locations simultaneously in Math-86 without requiring perfect fault localization information.The example contrasts with Jiang et al.’s single-chunk limitation and RAG-Gen’s requirement for perfect fault localization.Source: B. Results of RQ2 (Parameter-Efficient… · p. 10
C. Results of RQ3 (RepairLLaMA vs. ChatGPT-based APR)
RepairLLaMA outperforms zero-shot ChatGPT-based repair on Defects4J and GitBug-Java, including against GPT-4, though GPT-4 leads on one HumanEval-Java metric. The authors conclude that specialized representations and parameter-efficient training together support the repair adapter’s performance.
C. Results of RQ3 (RepairLLaMA vs. ChatGPT-based APR)
- RepairLLaMA is more effective on Defects4J and GitBug-Java than ChatGPT-based approaches, including GPT-4, while GPT-4 repairs more HumanEval-Java bugs under semantic match.RepairLLaMA repairs more HumanEval-Java bugs under Exact Match and AST Match.Source: C. Results of RQ3 (RepairLLaMA vs.… · p. 11
- GPT-3.5 and GPT-4 plausibly fix 64% and 39% fewer Defects4J v2 bugs, respectively, than RepairLLaMA.RepairLLaMA correctly fixes twice as many bugs as GPT-4 under Exact, AST, and Semantic Match.Source: C. Results of RQ3 (RepairLLaMA vs.… · p. 11
- RepairLLaMA also plausibly and correctly fixes more GitBug-Java bugs than GPT-3.5 and GPT-4; those bugs all postdate the three models’ training-data cutoff.The passage reports a statistically significant difference between RepairLLaMA and GPT-3.5/GPT-4 (p ≤4.10e −08).Source: C. Results of RQ3 (RepairLLaMA vs.… · p. 11
V. DISCUSSION
The discussion emphasizes fewer candidate patches, a relaxed fault-localization assumption, and trade-offs between fine-tuning and prompt engineering. It presents these choices as relevant to practical repair and the resources needed to use LLMs for the task.
A. Sampling Candidate Patches
- RepairLLaMA generates only 10 candidate patches per bug, and the authors report that it prioritizes the best patches in its top-10 list while reducing end-to-end plausibility-checking resources.Evaluating candidate plausibility requires running test cases, which the authors describe as expensive.Source: A. Sampling Candidate Patches · p. 11
B. Fault Localization with Code Regions
- The code-region strategy assumes localization can identify a region containing all buggy lines, relaxing the perfect-localization assumption and supporting multi-location repairs.The authors say this can remain effective with imperfect localization if it narrows the search to a region containing all buggy lines.Source: B. Fault Localization with Code Regions · p. 11
C. Fine-Tuning vs. Prompt Engineering
- Prompt engineering avoids additional training but is ad hoc and context-window limited; fine-tuning learns repair-specific tokens, structures, and signals, with LoRA helping mitigate its data and computation costs.The paper describes fine-tuning as learning the relationship between buggy and fixed code by changing model weights.Source: C. Fine-Tuning vs. Prompt Engineering · pp. 11–12
D. Threats to Validity
The paper addresses leakage and generalizability threats while distinguishing RepairLLaMA from fine-tuning, prompting, and agent-based repair approaches. Its comparisons emphasize realistic fault localization and repair without iterative prompting or external tools.
D. Threats to Validity
- Three benchmarks are used to address possible pre-training leakage, while Defects4J evaluation excludes three samples with functions also found in Megadiff.The authors report no identical Megadiff and Defects4J samples; HumanEval-Java and GitBug-Java postdate Megadiff and are non-overlapping.Source: D. Threats to Validity · p. 12
- The evaluation is limited to one programming language and one LLM, although the authors argue their novelties are language agnostic and the approach is model agnostic.The approach relies on infilling; models not trained for that objective can also be fine-tuned with the proposed representations.Source: D. Threats to Validity · p. 12
A. Large Language Models for Program Repair
- Earlier fine-tuning work includes studies of code representations, but Huang et al. report results under an assumption of perfect multi-line fault localization that RepairLLaMA does not make.RepairLLaMA also distinguishes its representations by incorporating fault-localization signals under realistic assumptions.Source: A. Large Language Models for Program… · p. 12
- Unlike prompting and agent approaches, RepairLLaMA does not use prompt engineering or iterative repair; it stores repair knowledge in fine-tuned model weights.The authors describe it as zero-shot and without external tools or feedback.Source: A. Large Language Models for Program… · pp. 12–13
B. Code Representations for Program Repair
RepairLLaMA combines repair-specific representations with parameter-efficient fine-tuning, targeting a broader range of bugs than some prior representation work and avoiding perfect fault localization assumptions. The paper reports strong results across three Java benchmarks and positions LoRA as a new direction for repair.
B. Code Representations for Program Repair
- RepairLLaMA representations align with pretraining data and objectives, cover multi-location bugs, and avoid assuming perfect fault localization for those bugs.This distinguishes the approach from prior research in code representation.Source: B. Code Representations for Program… · p. 13
C. Parameter-Efficient Fine-Tuning in SE
- Prior software-engineering studies report competitive or better PEFT performance than full-parameter fine-tuning, while RepairLLaMA is the first to employ and evaluate LoRA for program repair.The authors call for further PEFT research in program repair and related tasks such as overfitting detection.Source: C. Parameter-Efficient Fine-Tuning in SE · p. 13
VII. CONCLUSION
- RepairLLaMA correctly fixes 144 Defects4J, 109 HumanEval-Java, and 20 GitBug-Java bugs, outperforming strong baselines including GPT-3.5 and GPT-4.The conclusion presents these results as validating the approach’s core design decisions.Source: §VII · p. 13