PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Models

Martin Weyssow, Xin Zhou, Kisub Kim, David Lo, Houari Sahraoui

arXiv:2308.10462v3cs.SEcs.CLcs.LG

TL;DR

Full fine-tuning is computationally costly, and prior code-intelligence PEFT research largely focused on smaller models, leaving its application to LLMs under resource constraints less explored. This study compares PEFT for code generation with full fine-tuning, ICL, and RAG; it reports that PEFT outperforms full fine-tuning for SLMs and ICL/RAG for LLMs.

  • Problem

    Full fine-tuning of LLMs is computationally costly, and prior code-intelligence PEFT studies focused on SLMs, leaving its applicability to LLMs underexplored.

    Source: §1 · p. 2

  • Method

    The study evaluates six PEFT techniques for code generation and compares them with SLMs, ICL, and RAG across three datasets under limited computational resources.

    Source: §1 · pp. 3–4

  • Results

    Across the evaluated settings, PEFT outperformed full fine-tuning for SLMs and ICL/RAG for LLMs.

    Source: §9 · p. 22

  • Takeaways & Limitations

    The study finds PEFT practical under limited-resource conditions, potentially reducing reliance on large, expensive computational infrastructures.

    Source: §9 · p. 22

    Resource constraints limited larger model variants to QLoRA; other PEFT techniques, ICL, and RAG were not tested on larger variants.

    Source: External validity · p. 20

Abstract

from arXiv · show

Large language models (LLMs) demonstrate impressive capabilities to generate accurate code snippets given natural language intents in a zero-shot manner, i.e., without the need for specific fine-tuning. While prior studies have highlighted the advantages of fine-tuning LLMs, this process incurs high computational costs, making it impractical in resource-scarce environments, particularly for models with billions of parameters. To address these challenges, previous research explored in-context learning (ICL) and retrieval-augmented generation (RAG) as strategies to guide the LLM generative process with task-specific prompt examples. However, ICL and RAG introduce inconveniences, such as the need for designing contextually relevant prompts and the absence of learning task-specific parameters, thereby limiting downstream task performance. In this context, we foresee parameter-efficient fine-tuning (PEFT) as a promising approach to efficiently specialize LLMs to task-specific data while maintaining reasonable resource consumption. In this paper, we deliver a comprehensive study of PEFT techniques for LLMs in the context of automated code generation. Our comprehensive investigation of PEFT techniques for LLMs reveals their superiority and potential over ICL and RAG across a diverse set of LLMs and three representative Python code generation datasets: Conala, CodeAlpacaPy, and APPS. Furthermore, our study highlights the potential for tuning larger LLMs and significant reductions in memory usage by combining PEFT with quantization. Therefore, this study opens opportunities for broader applications of PEFT in software engineering scenarios. Our code is available at https://github.com/martin-wey/peft-llm-code/.

1 INTRODUCTION

Full fine-tuning of billion-parameter LLMs is costly, while prior PEFT work largely focused on SLMs; this study evaluates PEFT for code generation and compares it with ICL and RAG. It reports LoRA gains over ICL and RAG on Conala and CodeAlpacaPy, alongside quantized-tuning memory savings.

1 INTRODUCTION

  • Full fine-tuning incurs substantial computational costs, while prior code-intelligence PEFT studies focused on SLMs below 0.25B parameters and overlooked LLMs of at least 1B.Source: §1 · p. 2
  • The study compares PEFT with SLMs, ICL, and RAG under limited computational resources, across Conala, CodeAlpacaPy, and APPS.Source: §1 · p. 3
  • LoRA, IA3, and Prompt tuning outperform fully fine-tuned SLMs, and LoRA achieves the highest effectiveness among the evaluated PEFT techniques.Source: §1 · p. 4
  • Among parameter-efficient fine-tuning techniques, QLoRA considerably reduces memory usage, achieving up to a 2-fold decrease compared to LoRA.Source: §1 · p. 4
  • LoRA significantly enhances all models versus ICL and RAG on Conala and CodeAlpacaPy; LoRA and QLoRA improve CodeLlama-7B-Instruct on APPS.Source: §1 · p. 4
  • The study evaluates six PEFT techniques for Python code generation across four model families, including eight large and three small variants.Source: §1 · pp. 3–4

2 BACKGROUND

ICL and RAG guide code generation by adding examples to prompts at inference, while PEFT specializes models by learning task-specific parameters. Each approach trades off computational cost, prompt design, and the relevance or size of the supplied context.

2.1 In-Context Learning (ICL) and Retrieval-Augmented Generation (RAG)

  • ICL adds context-specific information to prompts at inference without gradient-based training, but requires representative prompt design.The paper identifies contextual coherence and avoiding costly gradient-based training as advantages.Source: §2.1 · p. 4
  • RAG retrieves examples similar to a test query and injects them into the input prompt, unlike ICL's random example selection.The retrieval query can use information from the test example, such as its coding problem.Source: §2.1 · p. 4
  • Both ICL and RAG can exceed prompt capacity and depend on example quality; irrelevant retrievals can degrade performance.RAG specifically depends on retrieving examples genuinely similar or useful for the test query.Source: §2.1 · p. 5

2.2 Parameter-Efficient Fine-Tuning (PEFT)

  • PEFT updates a subset of parameters, including through LoRA's injected low-rank matrices, prompt or prefix tokens, and QLoRA's quantization.LoRA freezes model weights; QLoRA combines LoRA with reduced-precision model data to use less GPU memory.Source: §2.2 · p. 5

3 APPLYING LLMS WITH LIMITED RESOURCES

The section frames fine-tuning as a resource-access challenge and presents PEFT as a way to balance computational cost and effectiveness. It describes LoRA and QLoRA as reducing memory demands in a single-GPU setting.

3 APPLYING LLMS WITH LIMITED RESOURCES

  • Full fine-tuning may be impractical on a single 24GB consumer GPU because of its substantial memory demands.The section links this constraint to increasing model size and the number of trainable parameters.Source: §3 · p. 5
  • ICL and RAG avoid parameter updates but rely on representative examples and prompt design, and may be less effective on highly contextual software-engineering tasks.Table 1 frames the tuning-technique comparison as a computation-effectiveness trade-off.Source: §3 · p. 5, Table 1 · p. 6
  • Figure 1 compares peak GPU memory consumption for full fine-tuning, LoRA, and QLoRA.The caption identifies the techniques compared but supplies no values.Source: Figure 1 · p. 6
  • PEFT methods such as LoRA and QLoRA limit updated parameters, balancing computational cost and effectiveness while reducing memory consumption.The paper reports that LoRA can tune CodeLlama-7B below 19GB of GPU memory, while quantized QLoRA can tune CodeLlama-34B within 24GB VRAM.Source: §3 · p. 6

4 METHODOLOGY

The study evaluates code-generation approaches under a single-24GB-GPU constraint, using four research questions spanning ICL, PEFT, RAG, and APPS functional correctness. Its setup uses Python datasets with training examples and instruction-based prompts.

4 METHODOLOGY

  • Experiments constrain both fine-tuning and inference to a single 24GB GPU to assess PEFT's feasibility and desirability against prior approaches and smaller models.Source: §4 · p. 6

4.1 Research Questions

  • The four research questions examine ICL baselines, PEFT versus ICL, LoRA versus ICL and RAG, and LoRA or QLoRA on APPS.For APPS, the study assesses functional correctness using average test cases passed and Pass@k across three problem difficulty levels.Source: §4.1 · p. 7

4.2 Datasets and Task

  • The study selects Conala, CodeAlpaca, and APPS because they provide training examples for fine-tuning, unlike HumanEval's lack of a training corpus.Figure 2 presents token-length distributions for Conala, CodeAlpacaPy, and APPS.Source: §4.2 · pp. 7–8, Figure 2 · p. 8

Conala dataset.

  • The curated Conala data pairs natural-language intents with manually annotated code and separates test intents from training intents.The splits contain 2,135 training, 201 validation, and 543 test samples; solutions are mostly short one-liners.Source: Conala dataset · p. 8

APPS dataset.

  • APPS contains 10,000 Python code-generation problems across introductory, interview, and competition difficulty levels, with an average of 21.2 test cases per problem.The study uses 4,500 training, 500 validation, and 750 test samples, balanced at 250 test samples per difficulty level.Source: APPS dataset · pp. 8–9

Task design.

  • The instruction template separates natural-language intent from the code response, and training loss ignores instruction tokens while predicting code autoregressively.At inference, models generate up to 10 code candidates.Source: Task design · pp. 9–10

4.3 ICL and RAG

  • ICL uses randomly sampled training examples, while RAG retrieves instruction-similar examples; both append examples to the input prompt.The setup uses up to 16 ICL examples for Conala and 8 for CodeAlpacaPy; retrieval limits are 16 and 4, respectively.Source: §4.3 · p. 10

4.4 Small and Large Language Models

  • The model selection spans 11 open-source models sized to fit one 24GB GPU, including CodeGen-350M-mono and two CodeT5+ SLMs.Models are categorized as SLMs below 1B parameters and LLMs at or above 1B.Source: §4.4 · p. 10, – SLMs · p. 10

– SLMs.

The study describes its evaluated model families, metrics, and implementation settings for code-generation experiments. It also identifies the scope of its ICL effectiveness figure and provides its code repository.

– SLMs.

  • CodeGen2 models combine bidirectional-encoder and unidirectional-decoder learning schemes, while the study also uses three CodeLlama-7B variants.CodeLlama-13B-Python and CodeLlama-34B-Python are used with QLoRA in RQ4.Source: – SLMs · p. 11

4.5 Metrics

  • Conala and CodeAlpacaPy use Exact Match and CodeBLEU, whereas APPS uses average test cases passed and Pass@k.Pass@k measures the percentage of problems with at least one fully passing solution among the top k generated candidates.Source: §4.5 · p. 11

4.6 Implementation Details

  • All experiments use one RTX A5000 24GB GPU and compare full fine-tuning, ICL, LoRA, IA3, prompt tuning, prefix tuning, and QLoRA.LoRA and IA3 apply low-rank decomposition to attention layers; prompt and prefix tuning prepend 20 trainable virtual tokens.Source: §4.6 · p. 11
  • Figure 3 charts model effectiveness across counts of random ICL examples on Conala and CodeAlpacaPy.The paper attributes using fewer examples for CodeGen2 to its substantially higher GPU memory use.Source: Figure 3 · p. 12
  • The authors make their implementation code publicly available.Source: §4.6 · p. 12

5 EXPERIMENTAL RESULTS

The experiments compare code-generation effectiveness across zero-shot and ICL baselines, PEFT methods, and model sizes. Under the same 24GB GPU limit, LoRA-tuned LLMs outperform the best small model, while CodeLlama models lead among the evaluated LLMs.

5.1 RQ1: Baseline Effectiveness of Models Using Zero-Shot and ICL

  • ICL effectiveness rises with more examples but begins declining beyond eight on Conala; most models peak at three or four examples on CodeAlpacaPy.CodeAlpacaPy's smaller optimal example count reflects the setup's out-of-memory limit at 16 examples.Source: §5.1 · p. 12
  • CodeLlama-7B achieves EM@10 scores of 29.83 (7.73) and 11.62 (7.01) on Conala and CodeAlpacaPy with ICL (zero-shot), respectively.ICL improves effectiveness over zero-shot across the evaluated models.Source: §5.1 · p. 13

5.2 RQ2: Effectiveness of Models using PEFT Techniques

  • Under the same 24GB GPU limit, the best LoRA-tuned LLM surpasses the best small model by 39.8%, 41.7%, and 47.1% (72.3%, 48.8%, and 9.1%) in EM@1, EM@10, and CodeBLEU on Conala (CodeAlpacaPy).CodeLlama-7B-Python with LoRA is the best LLM on average, and CodeGen-350M-mono with LoRA is the best small model on average.Source: SLMs vs. LLMs · p. 13

SLMs vs. LLMs.

  • Among SLMs, CodeGen-350M-mono leads all metrics on both datasets; on Conala its EM@10 is 18.42 with full fine-tuning and 25.60 with LoRA, which surpasses full fine-tuning across nearly all configurations.CodeLlama models are the best-performing LLMs in the study, while similarly sized CodeGen2-7B lags behind all CodeLlama-7B variants.Source: SLMs vs. LLMs · p. 13, LLMs · p. 13

LLMs.

LoRA is the strongest PEFT technique in the reported comparisons, while QLoRA combines quantization with LoRA to reduce memory use and support larger models. LoRA also outperforms ICL on EM@10 across the evaluated datasets.

LLMs.

  • LoRA is the most effective studied PEFT technique; IA3 often scores lower, prompt tuning remains viable, and prefix tuning fails to adapt larger models effectively.These comparisons cover the study's evaluated models and both datasets.Source: Best PEFT technique · p. 13

Best PEFT technique.

  • Conala has higher EM scores than CodeAlpacaPy, while its CodeBLEU scores are comparatively lower because small code examples may not support dataflow graph computations.The paper attributes the EM difference to task complexity differences between the datasets.Source: Best PEFT technique · p. 13
  • CodeLlama-34B-Python with QLoRA-4bit reaches Conala EM@10 40.70, a 12.2% increase over CodeLlama-7B-Python with LoRA at 36.28.The reported peak GPU memory consumption is below 24GB for each tuning configuration; for CodeLlama-7B-Python, QLoRA-4bit halves peak memory while improving Conala EM@10.Source: Effect of quantization with QLoRA · p. 15, Figure 5 · p. 16

Effect of quantization with QLoRA.

  • Quantization with LoRA sharply reduces GPU usage while maintaining effectiveness on both datasets and enabling fine-tuning of models up to 34B parameters.The reported best-performing PEFT LLM surpasses the best small model by 39.8–72.3% in EM@k.Source: Effect of quantization with QLoRA · p. 15

5.3 RQ3: Comparative Analysis of LoRA, ICL, and RAG

  • All models fine-tuned with LoRA achieve higher EM@10 than ICL on both datasets; CodeLlama-7B-Python improves Conala EM@10 by 23.1% (36.28 versus 29.47).The figure compares LoRA and ICL effectiveness using EM@10 and CodeBLEU.Source: LoRA vs. ICL · p. 16, Figure 6 · p. 16

LoRA vs. ICL.

LoRA outperforms ICL and RAG on the evaluated datasets, with especially consistent gains on the more challenging CodeAlpacaPy. On APPS, LoRA and QLoRA improve introductory and interview-level results, but gains are less substantial on harder tasks.

LoRA vs. ICL.

  • On CodeAlpacaPy, CodeLlama-7B gains 2.36 CodeBLEU with LoRA, while LoRA's CodeBLEU advantage on Conala is less pronounced than ICL's.EM@10 gains are greater on CodeAlpacaPy; EM@10 requires exact matches, whereas CodeBLEU also rewards close solutions.Source: LoRA vs. ICL · p. 17

RAG vs. ICL vs. LoRA.

  • On Conala, CodeLlama-7B reaches EM@10 29.83 with ICL and 35.17 with RAG, below LoRA's 39.31.Across both datasets, RAG gains thin as examples increase and saturate around 8–16 examples on Conala and 3–4 on CodeAlpacaPy; on CodeAlpacaPy, RAG underperforms randomly selected ICL examples.Source: RAG vs. ICL vs. LoRA · p. 17

5.4 RQ4: Exploration of LoRA and QLoRA for Code Generation on APPs

  • On APPS, QLoRA-4bit increases average passed test cases by 52% over the base model, while LoRA and QLoRA-4bit improve introductory-level Pass@5 by +3.60%.Improvements are less substantial on interview- and competition-level tasks.Source: §5.4 · p. 18, Table 4 · p. 18

6 DISCUSSION

The discussion presents PEFT as a practical way to tune code LLMs with less dependence on expensive infrastructure. It also identifies scope limits and open questions around inference efficiency, continual learning, and broader task coverage.

6 DISCUSSION

  • The authors describe PEFT as practical for code LLM fine-tuning and as reducing practitioners' dependence on large, expensive infrastructure.They identify efficient techniques for low-cost inference and other software-engineering tasks as areas for further investigation.Source: §6 · p. 18, Efficient techniques for LLMs of code · p. 19

Efficient techniques for LLMs of code.

  • The study does not evaluate combining PEFT with ICL or RAG; the authors contend that longer inference prompts could make added computational demands outweigh potential benefits.The authors also note that PEFT requires more fine-tuning time than ICL and RAG, but adds no inference-time cost.Source: Efficient techniques for LLMs of code · p. 19
  • Whether PEFT can adapt code LLMs in continual-learning settings without compromising retention of previously learned knowledge remains unresolved.Source: Efficient techniques for LLMs of code · p. 19

Effectiveness of QLoRA.

  • Across the study datasets, QLoRA-4bit is competitive or comparable to other PEFT methods and outperforms LoRA and QLoRA-8bit on Conala and APPS.The authors hypothesize that 4-bit precision may regularize fine-tuning and mitigate overfitting, while noting that broader applicability needs further exploration.Source: Effectiveness of QLoRA · p. 19

New findings for PEFT in software engineering.

  • All studied PEFT techniques significantly outperform full fine-tuning for CodeGen-350M-mono and CodeT5+-770M on Conala and CodeAlpacaPy; full fine-tuning of LLMs was not evaluated because of resource constraints.The authors contrast this result with prior large-scale NLP evidence favoring full fine-tuning.Source: New findings for PEFT in software… · p. 19

SE tasks and multi-tasking.

  • The study deliberately limits its task and dataset coverage; evaluation across varied tasks and datasets is identified as future work.The discussion points to composing task-trained LoRA modules as a possible direction for broader software-engineering tasks.Source: SE tasks and multi-tasking · pp. 19–20

7 THREATS TO VALIDITY

The study’s validity is bounded by its model and dataset choices, its hyperparameter settings, and its evaluation metrics. Its evidence covers Python code generation and does not include unit-test-based metrics for Conala or CodeAlpacaPy.

7 THREATS TO VALIDITY

  • The study evaluates a diverse model set but excludes larger variants except with QLoRA because resource constraints limit the other methods.The authors identify model selection as a threat to validity.Source: External validity · p. 20

External validity.

  • The dataset scope excludes HumanEval and MBPP because they lack training examples, while APPS adds execution-based code generation.Conala was selected for its high-quality examples, and CodeAlpacaPy adds lengthier examples.Source: External validity · p. 20

Internal validity.

  • The PEFT hyperparameters were adopted from prior work, but a LoRA sensitivity analysis could further solidify its advantage over ICL and RAG.The authors identify hyperparameter choices as the main threat to internal validity and suggest testing rank r and scaling factor α across more software-engineering tasks.Source: Internal validity · p. 20

Construct validity.

  • Conala and CodeAlpacaPy lack unit tests, so the study uses EM@k rather than Pass@k for those datasets; APPS uses Pass@k with up to five candidates.The authors identify metric choice as the main threat to construct validity.Source: Construct validity · p. 20

8 RELATED WORK

Prior code-generation research includes deep-learning and LLM approaches, while efficient adaptation encompasses both parameter-efficient fine-tuning and parameter-free prompting. This study distinguishes itself by examining PEFT techniques for LLMs rather than focusing on smaller models.

8 RELATED WORK

  • Full fine-tuning’s computational cost is described as impractical for most researchers and practitioners seeking optimal LLM performance.The related-work discussion frames efficient, cost-effective fine-tuning as a way to mitigate adoption burdens.Source: Automated Code Generation · p. 21

Automated Code Generation.

  • Efficient adaptation means lowering fine-tuning computation costs or using parameter-free approaches such as prompting and ICL.Prior ICL work shows prompt-example selection, quantity, and order influence downstream performance, and this study uses random few-shot examples rather than exploring ICL fully.Source: Efficient Adaptation of Models of Code · p. 21

Efficient Adaptation of Models of Code.

  • Earlier code-intelligence PEFT work examined prompt tuning, prefix tuning, and adapters, including applications to generation and other code tasks.The cited work includes adapter-based knowledge transfer and code-specific prefix tuning for generation tasks.Source: Efficient Adaptation of Models of Code · p. 21
  • The study focuses on LLMs and includes LoRA, IA3, and QLoRA, techniques prior code-intelligence work had not considered for efficiently tuning LLMs of code.The authors did not include adapters and describe their selected techniques as a sufficiently thorough analysis.Source: Efficient Adaptation of Models of Code · p. 22

9 CONCLUSION AND FUTURE WORK

The study reports that PEFT is effective for fine-tuning LLMs for code generation, outperforming full fine-tuning for SLMs and ICL and RAG for LLMs. It also identifies limited-resource use and broader software-engineering tasks as important directions.

9 CONCLUSION AND FUTURE WORK

  • PEFT techniques outperform full fine-tuning for SLMs and ICL and RAG for LLMs in the study’s code-generation comparison.The paper presents PEFT as practical under limited-resource conditions and proposes future work on code review, comment generation, multitasking, and continual learning.Source: §9 · p. 22

Related papers

Loading 2308.10462v3…