A Systematic Literature Review of Parameter-Efficient Fine-Tuning for Large Code Models

1 Introduction

• Large Code Models can contain hundreds of millions to billions of parameters, making training, fine-tuning, and deployment resource-intensive and raising accessibility, scalability, and sustainability concerns.The paper notes energy consumption and carbon footprint concerns and the high cost of fully fine-tuning a model for each task.Source: §1 · pp. 1–2
• PEFT adapts models by modifying a small parameter subset or adding lightweight components, reducing computational and GPU-memory demands while sometimes matching or exceeding full fine-tuning.Examples include LoRA, adapter tuning, and prompt tuning.Source: §1 · p. 2
• The review targets limited understanding of PEFT’s practical effectiveness and limitations in software engineering, including which strategies and architectures suit different task types and what trade-offs they involve.Its research questions also compare PEFT performance and training efficiency with valid full fine-tuning baselines.Source: §1 · p. 2
• The review synthesizes PEFT applications into a taxonomy of generative and non-generative software engineering tasks, alongside empirical insights intended to guide research and practical adoption.It presents the taxonomy as a framework for aligning PEFT strategies with task types and informing future applications.Source: §1 · pp. 2–3

2.1 Large Code Models

• LCMs span encoder-only, encoder-decoder, and decoder-only architectures and serve diverse SE tasks, including code generation, summarization, and clone detection.The section also notes applications to low-resource languages and emerging areas such as autonomous agents.Source: §2.1 · pp. 3–4

2.2 Fine-Tuning Code Models

• Full fine-tuning updates all pretrained parameters but can be impractical at billion-parameter scales; PEFT updates a small subset to reduce memory and compute costs.The review describes PEFT as maintaining or sometimes improving performance across several SE tasks.Source: §2.2 · p. 4

2.3 Parameter-Efficient Fine-Tuning (PEFT)

• PEFT methods are grouped by mechanism into additive, selective, reparameterized, and hybrid approaches.These categories distinguish introducing components, selecting existing parameters, low-rank transformations, and combining techniques.Source: §2.3 · p. 4, §2.3.1 · p. 4, §2.3.2 · p. 6

2.3.1 Additive PEFT:

• Additive methods freeze core model parameters and add trainable components, notably adapter layers or soft prompts.Pass-Tuning adds a GNN prefix that captures AST structure; other examples include activation scaling and prompt-based methods.Source: §2.3.1 · pp. 4–5

2.3.2 Selective PEFT:

• Selective PEFT uses a binary mask to update only chosen existing parameters, with examples including bias updates, weight masking, and layer freezing.Telly-K freezes the bottom K layers and trains the remaining top layers.Source: §2.3.2 · p. 6

2.3.3 Reparameterized PEFT:

• Reparameterized methods use low-rank updates, with LoRA applying trainable matrices in parallel and allowing learned weights to be merged after fine-tuning.Extensions include AdaLoRA’s dynamic rank adjustment, FF-LoRA updates to feed-forward layers, and QLoRA’s quantization approach.Source: §2.3.3 · pp. 6–7

2.3.4 Hybrid/Others Technique:

• Hybrid methods combine techniques such as adapters, prefix tuning, and LoRA; cited examples include MAM Adapter, Compacter, and AutoPEFT.The section situates this review against broader PEFT research and notes that systematic coverage focused on SE remains largely unexplored.Source: §2.3.4 · p. 7

3 Methodology: Study Selection and Analysis

• The review follows the established systematic literature review guidelines proposed by Kitchenham et al..Source: §3 · p. 7
• Three research questions shape the investigation and focus its analysis on primary studies of PEFT techniques in software engineering.Source: §3 · p. 7

• RQ1: Which software engineering tasks have been explored using PEFT methods?

• RQ1 examines PEFT applications across SE tasks and compares generative tasks such as code summarization with non-generative tasks such as clone detection.The review aims to identify usage patterns, task-specific adaptations, and gaps for further exploration.Source: • RQ1: Which software engineering… · p. 7

• RQ2: Which deep learning models and architectures are commonly optimized with PEFT methods?

• The review’s RQ2 examines which base-model architectures are optimized with PEFT and how methods are instantiated across model types.Extracted attributes included architecture family, model scale, PEFT method, configuration parameters, and trainable-parameter proportion.Source: • RQ2: Which deep learning models and… · pp. 7–8, §3.3.2 · p. 11

3.2 Study Selection and Filtering Process

• The review searched four databases for PEFT studies in software engineering and used screening, manual review, and backward snowballing to select papers.The search covered 2019–2025 and returned 1,146 records initially.Source: §3.1 · pp. 8–9, • RQ2: Which deep learning models and… · p. 8

3.1 Identifying and Selecting Primary Studies

• The search used four databases and 23 software-engineering task names; Google Scholar queries were split into seven to meet its 256-character limit.Search terms covered PEFT methods and tasks across generative and non-generative software engineering.Source: §3.1 · pp. 9–10

3.2.1 Exclusion Criteria and Alignment Analysis

• After screening 1,146 records, 26 met the criteria; manual review proceeded with 25 after one paper was unavailable, and four disputed cases were included by consensus.Two independent reviewers assessed abstracts, methods, and contributions.Source: §3.2.1 · p. 11

3.2.2 Snowballing and Manual Addition of Studies

• Backward snowballing identified two additional arXiv studies, bringing the final corpus to 28 papers.The authors included these non-peer-reviewed studies because they appeared in references of peer-reviewed papers and were cited in related work.Source: §3.2.2 · p. 11

3.3 Data Extraction and Synthesis

• The review extracted study attributes aligned with its research questions and synthesized PEFT patterns across task types, deep-learning models, and architectures.For performance and efficiency, it compared each study’s best PEFT method with full fine-tuning when a one-to-one evaluation was available.Source: • RQ2: Which deep learning models and… · p. 8, §3.3 · p. 11, §3.3.2 · p. 12

3.3.1 Synthesis Approach

• Generative tasks produce or transform code artifacts or related text, whereas non-generative tasks analyze existing code through activities such as classification, retrieval, or detection.The distinction provides the basis for mapping PEFT methods to task types.Source: §3.3.1 · p. 11

3.3.2 Data Extraction

• 28 studies cover 19 software-engineering tasks: 12 generative and seven non-generative, organized in a taxonomy with corresponding study lists.Table 1 reports task category, task description, and the studies identifying each task.Source: §3.3.2 · p. 12, Table 1 · p. 12

4.1 RQ1: Which software engineering tasks have been explored using PEFT methods?

• PEFT was applied to 19 software engineering tasks, including 12 generative tasks explored through NL2Code (12 studies), Code2NL (16), and Code2Code (21).Non-generative tasks were grouped by output paradigm—classification, ranking, or regression—rather than by input-output modality.Source: §4.1 · pp. 13–14
• Code Summarization was the most studied task at ∼46.4% (13 studies), while Code Generation accounted for ∼35.7% (10 studies); ∼60.7% of studies addressed multiple tasks.Non-generative applications concentrated on classification (16 studies; ∼84% of non-generative applications), with ranking in 3 studies and no regression tasks identified. Future work is encouraged in underexplored areas such as testing, documentation, and code quality, and in cross-task adaptation.Source: §4.1 · pp. 14, 15, 17

4.2 RQ2: Which deep learning models and architectures are commonly optimized with PEFT methods?

• Encoder-decoder architectures were most prevalent, while encoder-only models were preferred for analysis-oriented tasks and decoder-only models appeared exclusively in generative tasks.The review describes growing use of larger, specialized models, including CodeLlama variants, DeepSeek-Coder-33, and LLaMA-2-70B.Source: §4.2.1 · pp. 17–18

4.2.1 Base Models and Architectures in PEFT Software Engineering Optimization.

• CodeT5 was the most frequently adapted base model, appearing 19 times across 8 studies, while CodeLlama variants were used across tasks including generation, summarization, translation, and repair.CodeLlama-7B appeared in 7 software engineering instances across 5 studies; CodeLlama-13B in 6 instances across 3 studies; and CodeLlama-34B in 5 studies.Source: §4.2.1 · p. 18

4.2.2 PEFT Methods Across Software Engineering Tasks and Model Architectures

• Base LoRA appeared in 29 software engineering tasks and the Adapter family in 23; Prompt Tuning, Prefix Tuning, and P-Tuning appeared in 14, 13, and 7 studies, respectively.For Code Generation, LoRA appeared in 7 of 10 studies; for Code Summarization, 13 studies explored a broad range of techniques, including adapters, prompt tuning, LoRA variants, and other approaches.Source: §4.2.2 · pp. 18–19
• PEFT strategies varied across code-change tasks: three studies used LoRA for Automated Program Repair, while Code Refinement research spanned prompting, adapters, Pass-Tuning, BitFit, and LoRA.Pass-Tuning tunes less than 1% of parameters; Code Translation appeared in 8 studies, and Unit Test Generation in 2 studies using QLoRA or Prefix Tuning.Source: §4.2.2 · pp. 18–19
• LoRA appears in 36 SE task instances, or 40 including its FF-LoRA–Adapter hybrid; generative tasks commonly use ranks 8 or 16, while non-generative FF-LoRA often uses ranks 256 or 512.Non-generative configurations also often use ranks 1, 4, or 8, keeping trainable parameters below 1% of the base model; the review suggests higher FF-LoRA ranks compensate for adaptation limited to Feed-Forward layers.Source: §4.2.2 · pp. 21–22
• Prompt Tuning appears in 14 studies, with task-specific hard, soft, or combined prompts; Wang et al. used hard and soft prompts for Code Summarization and Code Translation, while Peng et al. explored AST-edit-derived hard prompts for code repair [109].Defect Detection appears in five studies, three using Prompt Tuning [145]; Code Search used language-specific hard templates for retrieval, while Defect Detection used symbolic labels and cloze-style verbalizers.Source: §4.2.2 · pp. 20, 22, 23, 24
• Prefix Tuning appears in 13 studies and varies by task: configurations include 100-token prefixes with a 256-dimensional projection for program repair [67] and 16-token prefixes with a 400-dimensional projection in a MAM setup.Other reviewed configurations vary prefix lengths by task or freeze transformer layers while training only the final linear layer [163].Source: §4.2.2 · p. 24
• About 72% of 34 generative-task entries report performance gains and about 54% efficiency gains, with Code Summarization, Code Generation, and Code Refinement most studied.These comparisons are against full fine-tuning; Table 6 summarizes the best-performing PEFT variant identified for each study and task.Source: Table 6 · p. 26, §4.2.2 · pp. 26–27
• Among 16 non-generative task entries, about 62% report performance gains and 50% efficiency gains, while Code Clone Detection results are mixed and Defect Detection improves consistently.The review reports three improved or neutral and three degraded Code Clone Detection entries; Defect Detection gains are especially associated with Prompt Tuning or LoRA.Source: Table 6 · p. 26, §4.2.2 · p. 27
• LoRA-based methods, especially Base LoRA and QLoRA, generally improve generative-task performance, but PEFT can lose ground on private, domain-shifted code and Adapter-based methods often decline on Code Translation.Base LoRA improves performance across four generative tasks, with efficiency gains in APR and one Code Generation study; QLoRA reports performance and efficiency gains in Code Generation, Code Completion, and Code Summarization. Wang et al. report competitive public-data results but degradation on private codebases.Source: §4.2.2 · p. 27
• Generative results also favor Prompt Tuning in Code Summarization and Translation, while Pass-Tuning, (IA)3, and Telly-K show task-specific gains and Pass-Tuning and MAM have reported declines.Telly-K reports performance and efficiency gains in Code Generation, Summarization, and Completion; MAM declines in Summarization and Translation, potentially because modular adaptation struggles with cross-domain transformations.Source: §4.2.2 · p. 27
• Across non-generative tasks, LoRA improves Defect Detection and Code Review but declines on Code Clone Detection; Prompt Tuning gains in Defect Detection and Code Search, while hybrid methods improve Code Clone Detection and Code Search.Prompt-based methods have not been applied to Code Clone Detection or Code Review in the reviewed studies; Adapter results are mixed, and L-Adapter improves both performance and efficiency on Cloze Test.Source: §4.2.2 · pp. 20, 27, 28
• ∼72.7% of generative tasks reported performance improvements versus ∼62.5% of non-generative tasks, while efficiency gains accompanied improvements in over half of generative tasks and ∼25% of non-generative tasks.The review reports dual efficiency and performance gains in over half of generative tasks, compared with around ∼25% of non-generative tasks.Source: §4.2.2 · p. 28
• LoRA variants are favored for Code Generation, adapters are the most-used approach in non-generative tasks, and Base LoRA succeeds against full fine-tuning baselines in ∼60% of cases.The review also finds consistent effectiveness for LoRA-based and adapter-based methods across generative and non-generative tasks.Source: §4.2.2 · p. 28
• AdaLoRA, Prefix Tuning, P-Tuning, and BitFit lack empirical evaluation, motivating standardized cross-task benchmarks and domain-specific guidance for selecting and adapting PEFT methods.The review recommends preserving code structure and formatting and evaluating syntactic correctness and compilability alongside traditional performance measures.Source: §4.2.2 · p. 29

Methodological Strategy Implications

• Organizations are advised to prioritize code generation and summarization, where PEFT applications are described as the most mature and validated use cases.The review recommends decoder-only models for code generation and encoder-decoder architectures for mixed analytical and generative needs.Source: §5 · p. 29

Resource Optimization Implications

• The review recommends staged PEFT adoption, beginning with generative applications before expanding to non-generative tasks, to prioritize computational return.It also argues PEFT should not be viewed only as a resource-saving compromise, since findings suggest it can enhance performance beyond traditional fine-tuning.Source: §5 · p. 29

• PEFT methods, including LoRA, Prompt Tuning, and Prefix Tuning, often match or exceed full fine-tuning performance while requiring significantly fewer computational resources.The review says these findings support establishing strong performance baselines without computationally expensive full-model fine-tuning.Source: §5 · p. 29

Step 1: Identify Underexplored and Emerging SE Tasks

• Expand PEFT research into underrepresented tasks such as testing, documentation, code quality assessment, API recommendation, refactoring, and requirements analysis, and investigate cross-task adaptation.For emerging tasks, the review recommends defining scope, challenges, and dataset preparation to support reproducibility.Source: §6 · p. 30

Step 2: Align Base Models with Task Requirements

• Align model selection with task requirements, while examining larger models and hybrid or task-agnostic architectures that could support multiple software engineering tasks.The review notes that larger models including StarCoder-15B and DeepSeek-Coder-33B are gaining adoption but remain underutilized.Source: §6 · p. 30

Step 3: Expand and Diversify PEFT Techniques

• Broaden evaluations of PEFT methods, including under-evaluated variants, hybrids, and quantization-aware approaches such as QLoRA.The review identifies LoRA as widely adopted, adapters as prominent in non-generative settings, and calls for more study of methods such as AdaLoRA, BitFit, Prefix Tuning, and P-Tuning.Source: §6 · p. 30

Step 4: Evaluate Adaptability, Efficiency, and Generalizability

• Evaluate adaptability, efficiency, and generalizability beyond accuracy, including cross-task and cross-language performance, memory use, low-resource settings, and hardware suitability.The review says few studies provide comprehensive ablation or scalability analyses.Source: §6 · pp. 30–31

Step 5: Promote Reproducibility and Benchmarking Platforms

• Standardize datasets and benchmarking, share implementation details, and provide scripts, datasets, checkpoints, hardware specifications, and evaluation protocols.The review identifies inconsistent dataset usage and reporting, and missing configuration detail in adapter-based studies, as barriers to comparability.Source: §6 · p. 31

Step 6: Address Domain-Specific Knowledge Acquisition Limitations

• Address PEFT’s performance degradation on private codebases and specialized domains when required domain knowledge is absent from pre-training data.The review recommends distinguishing public from private dataset evaluations and assessing knowledge acquisition and domain characteristics.Source: §6 · p. 31

• The six-step roadmap aims to address current limitations and support more robust, efficient, and generalizable fine-tuning solutions for software engineering.It combines broader task coverage, model-task alignment, PEFT diversification, expanded evaluation, reproducibility, and domain-specific adaptation.Source: §6 · pp. 30–31

7 Threats to Validity

• Construct validity depends on correctly identifying and categorizing SE tasks, model architectures, and PEFT methods.Multiple authors independently reviewed task and method selection to mitigate misclassification risk.Source: §7 · p. 31
• Undocumented practical PEFT uses may lack the transparency and empirical evaluation needed for systematic analysis.Configuration toggles may not explain how a method was integrated, assessed, or compared with baselines.Source: §7 · pp. 31–32
• Potential classification inconsistencies remain because some primary studies provide limited methodological detail, particularly for adapter-based or hybrid configurations.At least two authors reviewed each included study, and classification results were cross-checked and refined.Source: §7 · p. 32
• Search constraints may have missed relevant work, especially studies using varied terminology or published in non-ranked venues.The review supplemented broad searches with snowballing and manual inclusion, but acknowledges possible selection bias and under-representation.Source: §7 · p. 32

9 Conclusions

• The review covers 28 primary studies spanning 19 diverse SE tasks grouped into generative and non-generative categories.The study selection followed Kitchenham’s methodology.Source: §9 · p. 32
• PEFT consistently achieves performance on par with full fine-tuning across generative and non-generative tasks while dramatically reducing learned parameters.Some best-performing PEFT approaches marginally outperform full fine-tuning baselines on certain metrics.Source: §9 · p. 32
• The review identifies gaps in evaluation rigor and benchmarking consistency and proposes future research directions.Source: §9 · p. 32
• The authors characterize PEFT as an optimization paradigm likely to play a central role in future software-engineering-focused LCMs.They associate this prospective role with greener, faster, and smarter fine-tuning.Source: §9 · p. 32