Exploring Parameter-Efficient Fine-Tuning of Large Language Model on Automated Program Repair

1 INTRODUCTION

• Limited research had investigated PEFT for APR, and execution-based evaluation matters because code similarity does not ensure correct execution.APR patches must be merged into the original code and execute correctly; test cases are one execution-based evaluation method.Source: §1 · p. 2
• APR-Instruction enriches existing APR data with descriptive information and fixes up to 20.1% more bugs than Code Alpaca or OSS-Instruction.The dataset consists of 30,000 instructions.Source: §1 · p. 2
• 58% more bugs are fixed by the best presented model than by state-of-the-art APR techniques.The study also reports PEFT methods fix 7.2%-31.3% more bugs than no fine-tuning on CodeLlama-7B.Source: §1 · p. 2
• The study fine-tunes four LLMs with LoRA, prefix-tuning, p-tuning v2, and (IA)3, then evaluates on three benchmarks.The benchmarks are Defects4J v2.0, QuixBugs, and HumanEval-Java.Source: §1 · p. 2
• Among the evaluated models and methods, DeepSeek-Coder-Base-6.7B with (IA)3 has the highest fixing capability.The authors attribute (IA)3's superior performance to improving LLM creativity more effectively through fine-tuning than the other three PEFT methods.Source: §1 · p. 2
• PEFT methods reduce peak GPU memory usage by around 45.6%-50.0% compared with FMFT.The contribution summary also reports CodeLlama-7B with PEFT fixes 7.2%-31.3% more bugs than no fine-tuning.Source: §1 · p. 2
• With (IA)3, reducing APR-Instruction by 50% yields only 3.67% fewer fixed bugs, while blindly increasing LoRA rank does not necessarily improve fixing.A LoRA rank of 16 already provides relatively strong fixing capability.Source: §1 · p. 2

2.1 Overview

• The workflow constructs APR-Instruction, fine-tunes four LLMs with four PEFT methods, then generates and test-validates patches on benchmarks.Bug code and fix code seed the instruction data; each benchmark bug receives 10 candidate patches.Source: §2.1 · pp. 2–3, Figure 1 · p. 3

2.2 Studied Large Language Models

• The study selects CodeLlama-7B, CodeLlama-13B, Llama-2-7B, and DeepSeek-Coder-Base-6.7B as publicly accessible decoder-only LLMs of roughly 7B parameters or more.The models differ in pretraining: CodeLlama is code-focused, while DeepSeek-Coder uses next-token prediction and fill-in-the-middle objectives.Source: §2.2 · p. 3

2.3.2 Data Construction.

• APR-Instruction expands a 143,666-instance APR dataset because its bug/fix pairs lack the varied instruction types needed for instruction-tuning.Prompt-engineered descriptions yielded 30,000 instructions from buggy-code inputs capped at 200 tokens.Source: §2.3.2 · p. 3

2.3.3 Impact of APR-Instruction.

• APR-Instruction produced the highest fixing capability across the compared instruction datasets, including 28 additional bugs for CodeLlama-7B with LoRA versus OSS-Instruction.It also fixed 15 more bugs than Code Alpaca in that comparison; DeepSeek-Coder comparisons tested the base-model and PEFT-method influences.Source: §2.3.3 · p. 4, Table 1 · p. 4

2.3.4 Evaluation of overlap between APR-Instruction and bench-

• The authors report no overlap between APR-Instruction and HumanEval-Java, QuixBugs, or Defects4J, citing dataset collection dates and exclusions.APR-Instruction adds descriptions to the existing dataset without adding code.Source: §2.3.4 · p. 4

2.3.5 Evaluation of APR-Instruction validity.

• The authors manually inspected nearly 100 generated samples but could not individually verify all 30,000 instructions.They report that any inaccurate data has a manageable impact on results.Source: §2.3.5 · p. 4

Method selection

• The experiments select LoRA, p-tuning, prefix-tuning, and (IA)3 from three PEFT categories available for causal language models.The methods are selected from the Hugging Face PEFT library for use with Transformers.Source: §2.4 · p. 4

LoRA and prompt-based tuning

• LoRA updates attention query and key weights with trainable low-rank matrices, while p-tuning and prefix-tuning add learnable prompts at different model locations.LoRA adds BA to the initial weight matrix; p-tuning adds parameters to input embeddings, and prefix-tuning adds them at all layers.Source: §2.4 · p. 4

(IA)3

• (IA)3 adds three learnable vectors that rescale attention keys and values and adjust feed-forward activations.The key and value vectors rescale their respective activations before softmax.Source: §2.4 · pp. 4–5

Hyperparameter settings

• Trainable parameter counts are kept within the same order of magnitude for LoRA, p-tuning, and prefix-tuning, but not exactly matched across methods.(IA)3 parameter counts depend on base-model structure, and LoRA rank choices constrain precise matching.Source: §2.4 · p. 5, Table 2 · p. 5

2.5 Benchmarks and Evaluation Metrics of APR

• Evaluation covers 217 Defects4J bugs, 40 QuixBugs bugs, and 163 HumanEval-Java bugs, all Java single-hunk tasks with labeled bug locations and test suites.HumanEval-Java is noted as reducing data-leakage risks between pretraining data and benchmarks.Source: §2.5.1 · p. 5

2.5.2 Patch generation and validation.

• For each benchmark problem, models generate 10 patches that are individually validated against its test suite; the reported metric is pass@10.A problem is solved if any of the k generated samples passes the unit tests.Source: §2.5.2 · pp. 5–6

FMFT and resource measurements

• For FMFT and no-instruction-following inference, buggy benchmark code is converted to CodeLlama’s fill-in-the-middle format to generate patches.The study records peak GPU memory and trainable-parameter counts during supervised fine-tuning.Source: §2.6 · p. 6, Figure 3 · p. 6

Experimental environment

• Experiments run on a server with an 80G A800 GPU, and the authors release SFT and inference code.The server also has an Intel Xeon Platinum 8358 CPU and 1TB RAM.Source: §2.6 · p. 6

RQ1

• RQ1 asks how PEFT affects LLM fixing capability and efficiency compared with no fine-tuning and state-of-the-art APR techniques.Source: §3 · p. 6

RQ2

• RQ2 asks how different PEFT methods affect fixing capability across different base LLMs.Source: §3 · p. 6

RQ3

• RQ3 asks how PEFT hyperparameters and instruction-dataset size affect APR fixing capability.Source: §3 · p. 6

3.1.1 Experiments Setup.

• The resource comparison used five epochs, a 1000-token maximum sequence length, and a 1e-4 learning rate, with generated patches validated by test cases.Source: §3.1.1 · p. 6

3.1.2 Analysis of Fixing Capabilities.

• 111 bugs were fixed by DeepSeek-Coder-Base-6.7B with (IA)^3, 41 more than INCODER-6B and a 58% improvement over the prior state of the art.Source: Table 3 · p. 6, §3.1.2 · p. 7
• HumanEval-Java is the only benchmark compared with prior APR results because Defects4J and QuixBugs carry data-leakage risks.The comparison reuses prior-work results; RewardRepair fixed 22 of 163 instances among the selected DL-based methods.Source: §3.1.2 · p. 6
• PEFT improved CodeLlama-7B over no fine-tuning by 7.2%-31.3% more fixed bugs; FMFT underperformed both, which the authors attribute to overfitting on limited data.Source: Table 4 · p. 7, §3.1.3 · p. 7, §3.1.2 · p. 7

3.1.3 Analysis of Computing Resource Consumption.

• 45.6%-50.0% less peak GPU memory was used by PEFT than FMFT; on CodeLlama-7B, FMFT used 126.56GB, versus 68.91GB for (IA)^3 and 63.30GB for LoRA.PEFT also used around 97-1,000 times fewer trainable parameters than FMFT.Source: §3.1.3 · p. 7
• (IA)^3 had the fewest trainable parameters but used the most memory; versus LoRA, it used approximately 8.86% more memory while fixing 9.55% more bugs.Source: Figure 4 · p. 7

3.2.3 CodeLlama-7B vs CodeLlama-13B vs DeepSeek-Coder-Base-6.7B.

• 243 bugs were fixed by DeepSeek-Coder-Base-6.7B with (IA)^3, the highest total; the model outperformed CodeLlama-7B with (IA)^3 by 25 bugs despite its smaller scale.Source: Table 5 · p. 8, §3.2.3 · p. 8

3.2.1 CodeLlama-7B vs CodeLlama-13B.

• At five epochs, CodeLlama-13B p-tuning and prefix-tuning could underperform their CodeLlama-7B counterparts; at ten epochs, their total fixes increased by 33 and 100, respectively.LoRA fixing capability decreased when CodeLlama-13B training increased to eight epochs; the authors note limited computing resources constrained further experiments.Source: §3.2.1 · pp. 7–8

3.2.2 CodeLlama-7B vs Llama-2-7B.

• Llama-2-7B had the lowest fixing capability despite matching CodeLlama-7B in parameter size; the authors point to a mismatch between its text pretraining and code instruction-tuning data.Source: §3.2.2 · p. 8

3.2.4 Impact of PEFT Methods.

• On Defects4J, (IA)^3 ranked lowest at pass@1 for CodeLlama-7B, but surpassed the other methods from pass@5 through pass@10; the same pattern appeared on two other models.The authors hypothesize that its more diverse outputs help explore candidate fixes, while noting this conclusion rests on experimental observations rather than theoretical proof.Source: §3.2.4 · p. 9, Figure 5 · p. 9
• Peak memory stayed below 70G for 7B models and reached about 120G for the 13B model; LoRA used the least memory and (IA)^3 tended to use the most.Among 7B-scale models, the lowest-to-highest PEFT memory difference was about 4GB.Source: §3.2.4 · pp. 9–10, Figure 7 · p. 10
• (IA)^3 had the highest fixing capability across the tested LLMs, followed by LoRA, p-tuning, and prefix-tuning.Source: §3.2.4 · pp. 8–10

3.3.1 Impact of Hyperparameters Setting.

• Because (IA)^3 has no adjustable hyperparameters, the hyperparameter study focuses on LoRA rank using CodeLlama-7B, despite LoRA's lower fixing capability than (IA)^3.Source: §3.3.1 · p. 10
• Rank 16 produced the highest fixing capability among the tested LoRA settings, fixing 13 more bugs than rank 32.The other tested ranks were 8, 32, and 64; LoRA alpha and dropout remained 16 and 0.05.Source: §3.3.1 · p. 10, Table 7 · p. 10
• LoRA fixing capability first increased and then decreased as rank rose, so increasing trainable parameters alone did not guarantee better results.Source: §3.3.1 · p. 10

3.3.2 Impact of Instruction Dataset Size.

• The dataset-size experiment fine-tuned CodeLlama-7B with (IA)^3 on 5K, 10K, 15K, 20K, 25K, and 30K APR-Instruction examples.Source: §3.3.2 · p. 10, Table 8 · p. 10
• 30K instructions yielded the most fixed bugs, but reducing the dataset to 25K degraded fixing capability by only 1.83%, and reducing it to 5K by only 4.59%.The 25K and 5K datasets were 16.7% and 83.3% smaller than the 30K dataset, respectively.Source: §3.3.2 · p. 10
• For (IA)^3, fixing capability did not increase substantially with a large expansion of instruction data; the authors suggest starting small and adding data until gains become insignificant.Source: §3.3.2 · p. 10

• The study focuses on Java APR and does not extend its tasks to multiple programming languages, limiting applicability to broader software engineering challenges.Source: §4 · p. 10
• The study concentrates on single-hunk APR, leaving PEFT's fixing capability on multi-hunk tasks unexplored.Source: §4 · p. 11
• The p-tuning and prefix-tuning settings are not guaranteed optimal, and resource limits make extensive searches for better configurations difficult.Source: §4 · p. 11
• Benchmark patches that pass test cases are plausible rather than necessarily correct because test coverage may not include every line; PEFT quantization methods such as QLoRA were not explored.Source: §4 · p. 11

5.1 Parameter-Efficient Fine-Tuning for Software Engineering

• Earlier SE studies applied PEFT across tasks including code completion, docstring generation, and code clone detection, but primarily used similarity metrics that miss actual code executability.One study evaluated five PEFT methods on eight PTMs and four SE tasks [73]; another examined PEFT for code generation under limited resources [62].Source: §5.1 · p. 11
• PEFT work also covered code search and summarization, CodePrompt for code generation, and selective layer freezing, with evaluation still relying on similarity-based metrics.Source: §5.1 · p. 11

5.2 LLM-based Automated Program Repair

• Pre-2022 APR studies fine-tuned models no larger than 7B, and FMFT of code language models improved fixing on Defects4J, QuixBugs, and Humaneval-Java.The studies cited include [48] and.Source: §5.2 · p. 11
• Earlier APR evaluations used models up to 6B, while newer code-focused LLMs such as StarCoder [29], CodeLlama [50], WizardCoder, and DeepSeek-coder [14] emerged afterward.Source: §5.2 · p. 11

• 58% more bugs than SOTA APR techniques were fixed by the best model presented in this work.Source: §6 · p. 11
• The study evaluates four LLMs and four PEFT methods across three benchmarks, and reports that (IA)^3 outperformed the other methods in fixing capability.An instance illustrates its creativity improvement; the study also investigates PEFT hyperparameters and reducing training-data size while preserving fixing capability.Source: §6 · p. 11
• The authors report that datasets, code, and weights will be available and suggest applying the workflow to other software engineering tasks.Source: §6 · p. 11