PaperlayerView exact revision on arXiv

Source-linked AI summary

MORepair: Teaching LLMs to Repair Code via Multi-Objective Fine-tuning

Boyang Yang, Haoye Tian, Jiadong Ren, Hongyu Zhang, Jacques Klein, Tegawendé F. Bissyandé, Claire Le Goues, Shunfu Jin

arXiv:2404.12636v4cs.SE

TL;DR

Program-repair fine-tuning often emphasizes code patterns rather than the reasoning behind changes, while high-performing approaches can require large datasets. MORepair adds natural-language repair guidance as a second training objective alongside repaired-code generation. It improves performance across function-level and repository-level benchmarks and outperforms the compared fine-tuning methods.

  • Problem

    Existing program-repair fine-tuning can learn repair patterns without explicitly learning the reasoning behind code changes, and high-performing approaches often use large datasets.

    Source: §1 · p. 2

  • Method

    MORepair fine-tunes models to generate repaired code and corresponding natural-language guidance explaining the repair logic.

    Source: §3 · p. 6

  • Results

    MORepair outperforms StdFT and baseline models, with up to 18.8% and 11.4% improvements over baseline in Top-10 on EvalRepair-C++ and EvalRepair-Java, respectively, and +20.3% Top-10 on D4J-Repair and +56.0% Top-1 on SWE-Repair.

    Source: §8 · pp. 33–34

  • Takeaways & Limitations

    Comparisons with brief LLM-generated explanations indicate that structural repair logic in training data, rather than explanation length, is key to improving repair performance.

    Source: §8 · p. 34

    Limited benchmark test cases may permit patch overfitting, so the authors add EvalPlus tests to diversify the benchmarks.

    Source: §6.2 · p. 32

Abstract

from arXiv · show

Within the realm of software engineering, specialized tasks on code, such as program repair, present unique challenges, necessitating fine-tuning Large language models~(LLMs) to unlock state-of-the-art performance. Fine-tuning approaches proposed in the literature for LLMs on program repair tasks generally overlook the need to reason about the logic behind code changes, beyond syntactic patterns in the data. High-performing fine-tuning experiments also usually come at very high computational costs. With MORepair, we propose a novel perspective on the learning focus of LLM fine-tuning for program repair: we not only adapt the LLM parameters to the syntactic nuances of the task of code transformation (objective 1), but we also specifically fine-tune the LLM with respect to the logical reason behind the code change in the training data (objective 2). Such a multi-objective fine-tuning will instruct LLMs to generate high-quality patches. We apply MORepair to fine-tune four open-source LLMs with different sizes and architectures. Experimental results on function-level and repository-level repair benchmarks show that the implemented fine-tuning effectively boosts LLM repair performance by 11.4% to 56.0%. We further show that our fine-tuning strategy yields superior performance compared to the state-of-the-art approaches, including standard fine-tuning, Fine-tune-CoT, and RepairLLaMA.

1 Introduction

Program-repair fine-tuning can capture repair patterns without explicitly learning why a change is needed, and high-performing approaches often rely on large, costly datasets. MORepair adds natural-language repair guidance as a second fine-tuning objective alongside code transformation, and the paper reports improvements across repair benchmarks and comparisons with prior fine-tuning approaches.

1 Introduction

  • Standard fine-tuning can teach repair patterns without explicitly teaching the logical reasoning behind a repair, while state-of-the-art performance often requires large datasets and substantial resources.Prior examples cited include 16k samples for GPT-2 and about 30–50k code pairs for RepairLLaMA.Source: §1 · p. 2
  • MORepair jointly trains models to generate repaired code and learn natural-language explanations of the logic behind each repair.The added conversational-guidance objective complements the classical code-transformation objective.Source: §1 · p. 2
  • MORepair improves repair performance across four open-source LLMs and is reported to outperform Fine-tune-CoT and RepairLLaMA.The introduction reports CodeLlama-13B-instruct gains of 18.8% on EvalRepair-C++ and 11.4% on EvalRepair-Java.Source: §1 · pp. 2–3
  • The paper introduces EvalRepair-C++ and EvalRepair-Java, plus repository-level D4J-Repair and SWE-Repair benchmarks.The function-level benchmarks contain 164 and 163 patch pairs, respectively; the repository subsets contain 371 Java bugs and 204 Python bugs.Source: §1 · p. 3
  • Ablations indicate that jointly optimizing patch correctness and explanations performs better than optimizing either objective alone.The paper also reports that the structural repair logic in LLM-generated guidance matters more than explanation length.Source: §1 · p. 3

2 Motivating Example

The motivating example uses an arithmetic-expression repair to illustrate how guidance can explain the logic behind a patch. The authors contrast concise human explanations with longer, structured LLM-generated guidance and hypothesize that its detail and range of fix strategies may provide a richer learning signal.

2 Motivating Example

  • The example patch addresses bugs in evaluating arithmetic expressions with mixed operations and parentheses.The example comes from a student solution to an Expression Evaluation problem in TutorLLMCode.Source: §2 · pp. 3–4
  • Human-written guidance identifies the buggy location and describes the missing code checks, abstracting the repair logic from the program's syntax.The authors posit that this information may help fine-tuning generalize to a broader set of programs.Source: §2 · p. 4
  • LLM-generated guidance averages 337 words versus 36 for human-written explanations and outlines multiple potential fixes.The example guidance discusses operator precedence, parentheses, parameter order, and error handling.Source: §2 · p. 4
  • The authors hypothesize that the LLM explanations' structure, length, and multiple fix strategies may provide a richer fine-tuning signal than human-written explanations.They suggest this could potentially improve generalization to a wider range of bugs.Source: §2 · p. 4

3 Approach

MORepair prepares repair data and guidance, trains LLMs on code repair and repair explanations, then uses the adapted models to generate candidate patches. Its approach also analyzes TutorLLMCode’s repair complexity and combines two training objectives.

3 Approach

  • MORepair combines preparation, multi-objective fine-tuning, and repair inference to train open-source LLMs to learn repair logic and generate patches.At inference, candidate patches are validated against benchmark test cases.Source: §3 · pp. 5–6

3.1 Training Preparation

  • TutorLLMCode contains 1,535 student-submitted buggy–repaired C++ code pairs across 45 tasks; GPT-4-generated rationales average 337 words and are manually checked on a random 10% sample.The dataset’s human-written rationales average 36 words and mainly identify fix locations; GPT-4 rationales are generated from code pairs without using those explanations.Source: §3.1 · p. 6

3.2 Dataset Analysis

  • 78.8% of TutorLLMCode repairs modify multiple hunks, while 61.4% of buggy codes contain one function and 64.6% of repairs modify one function.The corresponding figures show distributions of function counts per instance, modified hunks, and modified functions.Source: Figure 5 · p. 7, §3.2 · p. 7, Figure 4 · p. 7

3.3 Multi-objective Fine-tuning

  • MORepair trains on repaired code alone and on repaired code accompanied by explanatory guidance, combining their losses with equal weight (α = 0.5).The guidance explains the nature of bugs and their repair logic.Source: §3.3 · pp. 8–9

3.4 Repair Inference

  • At inference, a quantized LLM with QLoRA adapters generates candidate repairs; the system extracts the first code segment delimited by triple backticks from its response.Generation uses sampling with temperature and top_p set to 1.0.Source: §3.4 · pp. 9–10

4 Experimental Setup

The setup evaluates four code-focused LLMs on paired C++ and Java repair benchmarks, using Top-k repair metrics and shared fine-tuning controls. It also examines the effects of test augmentation and four research questions spanning model variation, guidance, prior methods, and repository-level bugs.

4.1 Model Selection

  • The study selects CodeLlama-13B, CodeLlama-7B, StarChat-alpha, and Mistral-7B to cover code-model sizes from 7B to 16B and varied architectures.The models were selected based on popularity and architectural diversity.Source: §4.1 · p. 11, Table 1 · p. 11

4.2 Evaluation Benchmark

  • HumanEval-X problems were adapted into C++ and Java benchmarks with equivalent injected bugs; the final single- and multi-hunk bug distributions are almost consistent.The benchmarks add original and translated EvalPlus tests, correcting original solutions that failed the expanded tests.Source: §4.2 · pp. 11–12, Table 2 · p. 12

4.3 Evaluation Metrics

  • Top-1, Top-5, and Top-10 count a repair when any of the top k candidate patches passes all benchmark tests.The selected cutoffs reflect findings that developers often stop after five attempts and are unlikely to consider more than ten patches.Source: §4.3 · p. 12

4.4 Fine-tuning Configurations

  • MORepair optimizes repaired-code generation and repaired-code generation with natural-language rationales, while StdFT and Fine-tune-CoT each optimize one objective.All three fine-tuning approaches use LoRA and NEFTune; MORepair weights its objectives equally (α=0.5), with the two losses exhibiting comparable magnitudes during training.Source: §4.4 · p. 13, Figure 7 · p. 13

4.5 Research Questions

  • RQ-1 tests multi-objective against standard fine-tuning on CodeLlama-13B, while RQ-2 examines size and architecture across four LLMs.Both questions use EvalRepair-C++ and EvalRepair-Java to assess repair performance and cross-language generalization.Source: §4.5 · pp. 13–14
  • RQ-3 compares LLM-generated with human guidance and evaluates MORepair against Fine-tune-CoT and RepairLLaMA.The comparisons span various LLMs.Source: §4.5 · p. 14
  • RQ-4 evaluates repository-level repair on subsets of Defects4J and SWE-Bench, comparing MORepair with standard fine-tuning.The question concerns effectiveness on real-world bugs and scalability to complex repair scenarios.Source: §4.5 · p. 14

5 Experiments & Results

The experiments evaluate MORepair across model types, guidance sources, and function- and repository-level repair benchmarks, including C++, Java, and Python. They examine repair performance alongside the role of natural-language guidance and rationales, including their potential to support error diagnosis and explain patch behavior.

5.1 Effectiveness of Multi-objective Fine-tuning for Program Repair

  • On EvalRepair-C++, MORepair gains 20.7 percentage points in Top-5 and 11.0 percentage points in Top-10 over baseline, and 12.2 and 5.5 points over StdFT, respectively.The corresponding relative increases are 50.6% and 18.8% over baseline, and 24.7% and 8.6% over StdFT.Source: §5.1 · pp. 14–15, Table 4 · p. 15
  • Case studies show MORepair correctly explains and repairs derivative logic and adapts a minimal Java loop fix, but its first all_prefixes patch remained off-by-one despite useful guidance.For all_prefixes, the incorrect Top-1 patch was fixed among the Top-10 results.Source: §5.1 · pp. 16, 17, 20
  • MORepair’s rationale can aid debugging when its Top-1 patch is wrong, and every such correct-guidance case had a correct fix in Top-10.Among 164 attempts, 45 explanations were correct (27.4%); 36 of those had correct patches (80.0%) and 9 did not (20.0%). Of 117 incorrect fixes, 9 (7.7%) had correct guidance.Source: §5.1 · pp. 17, 20, Table 5 · p. 18
  • For structural errors, MORepair outperforms StdFT by 72.8% and achieves the highest minimum accuracy across bug types, indicating stronger performance on complex logical bugs.The benchmark groups bugs into condition, expression, and structural errors.Source: §5.1 · pp. 17–18, Table 6 · p. 18
  • On EvalRepair-Java, MORepair reaches 77.9% Top-10, 11.4% above baseline and 1.6% above StdFT, supporting cross-language transfer from C++ training.MORepair also outperforms StdFT on Top-1, Top-5, and Top-10 in this cross-language evaluation.Source: Table 7 · p. 18, §5.1 · pp. 18, 20

5.2 Impact of Size or Type for Fine-tuning LLMs on Code Repair Performance

  • Across four LLMs on EvalRepair-C++, MORepair improved all repair metrics, with maximum Top-10 gains of 18.8% over baseline and 12.6% over StdFT, while StdFT sometimes fell below baseline.The tested models included CodeLlama-7B, StarChat-alpha (16B), Mistral-7B, and CodeLlama-13B.Source: Table 8 · p. 21, §5.2 · p. 21
  • MORepair improves Top-10 across varied model sizes and architectures, with reported gains of 11.0 percentage points over baseline and 7.4 over StdFT on EvalRepair-C++, and 8.0 and 9.9 points, respectively, on EvalRepair-Java.On EvalRepair-Java, the reported model-specific Top-10 gains range from 8.9% to 11.9% over baseline and 1.6% to 17.8% over StdFT; StdFT’s performance is inconsistent across languages and architectures.Source: §5.2 · pp. 21–22

5.3 Evaluating the Impact of Guidance Sources and Comparing MORepair against State-of-the-Art Fine-tuning Methods

  • LLM-generated guidance yields 3.0–7.9 percentage-point Top-10 improvements over human guidance on EvalRepair-C++ and 1.2–5.5 points on EvalRepair-Java.Full explanations outperform brief LLM guidance, while brief guidance still beats human-written guidance in most cases; the paper attributes the advantage to structured repair reasoning.Source: Table 10 · p. 23, Table 11 · p. 23, §5.3 · p. 24
  • MORepair surpasses Fine-tune-CoT and RepairLLaMA across Top-1, Top-5, and Top-10 on EvalRepair-C++ and EvalRepair-Java.It does so even though RepairLLaMA receives manually annotated fault-location information; RepairLLaMA fails to achieve a Top-10 enhancement on either benchmark.Source: Table 12 · p. 25, Table 13 · p. 25, §5.3 · pp. 25–26
  • Fine-tune-CoT has mixed results: it improves EvalRepair-C++ Top-10 by 9.8 percentage points for CodeLlama-13B but gains only 1.3 points on EvalRepair-Java and fails to beat StdFT for other models.For three EvalRepair-C++ bug types, the reported Top-1 figures of 22.0% versus 46.3% illustrate the difference between Fine-tune-CoT and MORepair; the paper links joint optimization to stronger generalization on structural errors.Source: §5.3 · p. 26, Table 14 · p. 26

5.4 Effectiveness of MORepair on Repository-Level Benchmarks

  • On D4J-Repair, MORepair achieves 41.5% Top-10, surpassing the baseline by 20.3% and StdFT by 14.0%.The evaluation uses a filtered subset of 371 Defects4J bugs and CodeLlama-13B.Source: Table 15 · p. 27, §5.4 · p. 27
  • In the Closure-124 case study, MORepair correctly fixes deep property access by traversing to the base identifier, while baseline and StdFT fail.MORepair’s explanation identifies the need to traverse the chain; the baseline patch introduces bugs, and StdFT’s fix is convoluted and does not address the issue effectively.Source: §5.4 · pp. 27–29
  • MORepair achieved 19.1% Top-1 accuracy on SWE-Repair with greedy decoding, versus 12.3% for the baseline and 16.7% for StdFT.With temperature sampling, MORepair achieved 14.7% Top-1, 28.4% Top-5, and 30.9% Top-10 accuracy.Source: Table 16 · p. 30, §5.4 · p. 30
  • On pydata__xarray_7391, MORepair’s explained patch preserved dataset attributes, while the baseline removed alignment logic and StdFT used a potentially error-causing None fallback.MORepair identified attribute loss during binary operations and made a minimal fix; the baseline overlooked that issue, and StdFT offered no explanation.Source: §5.4 · p. 31
  • MORepair improved over the baseline by 20.3% Top-10 on D4J-Repair and 56.0% Top-1 on SWE-Repair, and over StdFT by 14.0% and 10.8% Top-10, respectively.The benchmarks cover Java and Python, and the findings report that MORepair generates high-quality repair explanations on repository-level tasks.Source: §5.4 · p. 32

6 Threats to Validity

The authors identify threats involving base-model selection, benchmark test coverage, and randomness in LLM outputs. They describe measures intended to address each threat.

6.1 Threats to Internal Validity

  • Using four LLMs with varied sizes and architectures is intended to reduce the risk that findings depend on the choice of base model.The models include CodeLlama-13B-instruct, CodeLlama-7B-instruct, StarChat-alpha, and Mistral-Instruct-7B.Source: §6.1 · p. 32

6.2 Threats to External Validity

  • Adding EvalPlus test cases to EvalRepair-C++ and EvalRepair-Java is intended to reduce patch overfitting from insufficient benchmark tests.The authors say this improves the realism of performance assessment and the external validity of their conclusions.Source: §6.2 · p. 32

6.3 Threats to Construct Validity

  • Because LLM output randomness could undermine conclusions, the authors generate outputs ten times and calculate Top-1, Top-5, and Top-10 metrics.They intend multiple rounds to make findings more stable and representative.Source: §6.3 · p. 32

7 Related Work

Related work covers prompting and fine-tuning for LLM-based repair, including parameter-efficient adaptation. MORepair differs by combining multi-objective learning with LLM-generated guidance during fine-tuning.

Prompting

  • Prior prompting methods guide repair with buggy code and instructions, examples, or iterative dialogue, while prompting remains distinct from fine-tuning [8] [10] [26] [40] [59] [3] [49] [58] [59].The passage describes zero-shot and few-shot prompting and conversational prompting as variants.Source: §7 · pp. 32–33

Fine-tuning

  • Fine-tuning has improved repair performance, with studies reporting gains over baselines and learning-based or deep-learning APR approaches [22] [23] [25] [36].Examples include TFix, GPT-2 fine-tuning, and studies of LLMs across APR benchmarks [22].Source: §7 · p. 33

Parameter-efficient fine-tuning

  • LoRA and QLoRA reduce fine-tuning resource demands, while RepairLLaMA combines LoRA with specialized code representations for repair [17] [7] [48].The passage also notes NEFTune's use of embedding-level noise to enhance generalization [20].Source: §7 · p. 33

MORepair

  • Unlike prior single-objective fine-tuning, MORepair uses multi-objective learning and LLM-generated guidance, and reports superior repair performance to standard fine-tuning, RepairLLaMA, and Fine-tune-CoT [22].The passage characterizes this as the first use of this combination for LLM program repair.Source: §7 · p. 33

8 Conclusion

MORepair uses multi-objective fine-tuning to teach open-source LLMs repair logic alongside code generation. It reports gains on function-level and repository-level benchmarks and outperforms several fine-tuning alternatives.

8 Conclusion

  • Across EvalRepair-C++ and EvalRepair-Java, MORepair improves over baseline by up to 18.8% and 11.4% in Top-10, respectively.The paper reports these findings across four LLMs with different architectures and sizes.Source: §8 · p. 33
  • MORepair achieves +20.3% Top-10 on D4J-Repair and +56.0% Top-1 on SWE-Repair.D4J-Repair is a Defects4J subset for Java, and SWE-Repair is a SWE-Bench subset for Python.Source: §8 · pp. 33–34
  • MORepair surpasses Fine-tune-CoT and RepairLLaMA across four LLMs in both benchmarks, while ablations favor multi-objective learning and LLM-generated guidance over human-generated guidance.Comparisons with brief LLM-generated explanations indicate that structural repair logic, rather than explanation length, is key to improving performance.Source: §8 · p. 34

Related papers

Loading 2404.12636v4…