PaperlayerView exact revision on arXiv

Source-linked AI summary

Fine-Tuning and Prompt Engineering for Large Language Models-based Code Review Automation

Chanathip Pornprasit, Chakkrit Tantithamthavorn

arXiv:2402.00905v4cs.SE

TL;DR

The effectiveness of prompting strategies for LLM-based code review automation, particularly few-shot learning, remains uncertain. The paper evaluates fine-tuning and prompting for GPT-3.5 and Magicoder, finding that fine-tuned GPT-3.5 performs best and few-shot learning without a persona is the best prompting strategy without fine-tuning.

  • Problem

    The effectiveness of few-shot learning for LLM-based code review automation is unknown, making it difficult for practitioners to choose among prompting strategies.

    Source: §2.3 · p. 4

  • Method

    The study evaluates fine-tuning and zero-shot, few-shot, and persona prompting for GPT-3.5 and Magicoder, comparing them with existing code review automation approaches.

    Source: §1 · p. 2, §2.2 · p. 3

  • Results

    Fine-tuned GPT-3.5 performs best, with its zero-shot version achieving 73.17% -74.23% higher EM than Guo et al.’s approach [14].

    Source: §1 · p. 2, §7 · p. 13

  • Takeaways & Limitations

    The authors recommend fine-tuning when possible and few-shot learning without a persona when data is insufficient for fine-tuning.

    Source: §1 · p. 2

    Results may vary with model randomness, and the study did not explore all fine-tuning hyperparameter combinations because repeated runs and broad searches are expensive.

    Source: §6.2 · p. 12

Abstract

from arXiv · show

Context: The rapid evolution of Large Language Models (LLMs) has sparked significant interest in leveraging their capabilities for automating code review processes. Prior studies often focus on developing LLMs for code review automation, yet require expensive resources, which is infeasible for organizations with limited budgets and resources. Thus, fine-tuning and prompt engineering are the two common approaches to leveraging LLMs for code review automation. Objective: We aim to investigate the performance of LLMs-based code review automation based on two contexts, i.e., when LLMs are leveraged by fine-tuning and prompting. Fine-tuning involves training the model on a specific code review dataset, while prompting involves providing explicit instructions to guide the model's generation process without requiring a specific code review dataset. Method: We leverage model fine-tuning and inference techniques (i.e., zero-shot learning, few-shot learning and persona) on LLMs-based code review automation. In total, we investigate 12 variations of two LLMs-based code review automation (i.e., GPT- 3.5 and Magicoder), and compare them with the Guo et al.'s approach and three existing code review automation approaches. Results: The fine-tuning of GPT 3.5 with zero-shot learning helps GPT-3.5 to achieve 73.17% -74.23% higher EM than the Guo et al.'s approach. In addition, when GPT-3.5 is not fine-tuned, GPT-3.5 with few-shot learning achieves 46.38% - 659.09% higher EM than GPT-3.5 with zero-shot learning. Conclusions: Based on our results, we recommend that (1) LLMs for code review automation should be fine-tuned to achieve the highest performance; and (2) when data is not sufficient for model fine-tuning (e.g., a cold-start problem), few-shot learning without a persona should be used for LLMs for code review automation.

1. Introduction

The introduction motivates evaluating fine-tuning and prompting because prior GPT-3.5 code-review work covered zero-shot use but not fine-tuning or few-shot strategies. It reports gains for fine-tuned GPT-3.5 and identifies few-shot learning without a persona as the best prompting strategy when GPT-3.5 is not fine-tuned.

1. Introduction

  • Prior GPT-3.5 work by Guo et al. [14] used zero-shot learning, leaving fine-tuning and few-shot prompting unevaluated and their relative effectiveness unclear.The introduction notes that training CodeReviewer required two DGX-2 servers with 32 NVIDIA V100 GPUs, resources infeasible for organizations with limited budgets.Source: §1 · pp. 1–2
  • The study evaluates GPT-3.5 and Magicoder with fine-tuning and inference techniques, comparing them with existing approaches across three code-review datasets using EM and CodeBLEU.The techniques include zero-shot learning, few-shot learning, and persona; the paper identifies this investigation as its first contribution.Source: §1 · p. 2
  • 73.17%–74.23% higher EM than Guo et al.’s approach [14] is achieved by fine-tuned GPT-3.5 with zero-shot learning.The authors infer that GPT-3.5 should be fine-tuned to achieve the highest performance.Source: §1 · p. 2
  • 63.91%–1,100% higher Exact Match is achieved by fine-tuned GPT-3.5 with few-shot learning than by its non-fine-tuned variants.The authors say fine-tuned GPT-3.5 can generate more correct revised code than GPT-3.5 without fine-tuning.Source: §1 · p. 2
  • 46.38%–659.09% higher Exact Match is achieved by GPT-3.5 with few-shot than zero-shot learning, while adding a persona yields 1.02%–54.17% lower Exact Match.The reported best prompting strategy for GPT-3.5 without fine-tuning is few-shot learning without a persona.Source: §1 · p. 2
  • The authors recommend fine-tuning for highest performance and few-shot learning without a persona when data is insufficient for fine-tuning.They also identify practitioner recommendations as a contribution of the study.Source: §1 · p. 2

2. Related Work and Research Questions

The section reviews code-review automation, LLM approaches, and the resource constraints that motivate fine-tuning and prompting. It frames open questions about fine-tuning and few-shot learning for GPT-3.5.

2.1. Code Review Automation

  • Code review automation is framed as sequence generation, but NMT approaches trained on small datasets remain imperfect.Code review itself is time-consuming and expensive, and developers can face delays in receiving feedback.Source: §2.1 · p. 3

2.2. LLMs-based Code Review Automation Approaches

  • Prior studies found LLM-based code-review automation often outperforms NMT-based approaches; Li et al. [5] reported an 11.76% gain over a transformer-based NMT model, while limited-resource organizations may be unable to develop LLMs.Source: §2.2 · p. 3
  • Fine-tuning adapts a pre-trained model to a code-review dataset, while prompting guides generation with instructions without requiring a specific dataset.Prior work on LLaMa-Reviewer [9] found that fine-tuning can improve performance; zero-shot, few-shot, and persona are instruction-based strategies considered suitable for software-engineering tasks.Source: §2.2 · p. 3

2.3. GPT-3.5 for Code Review Automation

  • Guo et al. [14] evaluated only zero-shot GPT-3.5, leaving fine-tuning and few-shot learning unevaluated and making it difficult to identify the most effective approach.The section asks which approach best leverages LLMs, whether fine-tuning benefits GPT-3.5, and which prompting strategy is most effective.Source: §2.3 · p. 4

3. Experimental Design

The experimental design compares fine-tuning and prompting for GPT-3.5 and Magicoder across three code-review datasets. It specifies dataset selection, training and prompting procedures, evaluation measures, and model settings.

3.1. Overview

  • The study tests GPT-3.5 and Magicoder across six fine-tuning and inference settings on three datasets, for 36 experiments in total.Fine-tuned models use zero-shot prompting with or without a persona; non-fine-tuned models use zero-shot or few-shot prompting, each with or without a persona.Source: §3.1 · p. 4, Figure 2 · p. 5

3.2. The Studied Datasets

  • The three selected datasets avoid the constraint that revised code cannot contain new code tokens, which the authors say does not reflect real code review practice.The datasets are CodeReviewerdata, Tufanodata, and D-ACTdata.Source: §3.1 · p. 4, §3.2 · p. 5, Table 3 · p. 6

3.3. Model Fine-Tuning

  • The authors randomly select 6% of training examples for fine-tuning, choosing that proportion after trials showed GPT-3.5 outperformed existing approaches.GPT-3.5 is fine-tuned through the OpenAI API, while Magicoder uses DoRA.Source: §3.3 · p. 5

3.4. Inference via Prompting

  • Zero-shot prompts contain an instruction and review input; few-shot prompts add three BM25-selected demonstrations, and persona prompts ask GPT-3.5 to act as a software developer.Each demonstration contains submitted code, a reviewer comment, and revised code; the prompt templates are shown in Figure 3.Source: §3.4 · p. 6, Figure 3 · p. 6

3.5. The Evaluation Measures

  • The evaluation uses Exact Match for identical revised code and CodeBLEU for similarity that includes weighted n-gram, syntactic, and semantic information.The authors use CodeBLEU rather than BLEU because BLEU ignores syntactic and semantic correctness.Source: §3.5 · pp. 6–7

3.6. The Hyper-Parameter Settings

  • GPT-3.5 generation uses temperature 0.0, top p 1.0, and maximum length 512; Magicoder uses the same generation settings.Magicoder fine-tuning uses DoRA attention dimension 16, alpha 8, and dropout 0.1; GPT-3.5 fine-tuning uses OpenAI API-provided hyperparameters.Source: §3.6 · p. 7

4. Result

The results compare fine-tuned and prompted GPT-3.5 and Magicoder for code review automation. Fine-tuning performs best against Guo et al.’s approach, while few-shot prompting without a persona is the strongest non-fine-tuned GPT-3.5 strategy.

RQ1

  • Fine-tuned GPT-3.5 achieved 73.17% -74.23% higher EM, and fine-tuned Magicoder achieved 26.00% - 28.53% higher EM, than Guo et al.’s approach [14].Both comparisons use Guo et al.’s approach [14] as the baseline.Source: §4 · p. 7

RQ2

  • Fine-tuned GPT-3.5 with zero-shot learning achieved 63.91% - 1,100% higher EM and 5.91% - 63.9% higher CodeBLEU than non-fine-tuned GPT-3.5.Fine-tuning adapts the model by learning the relationship between submitted code and reviewer comments and the revised code from training examples.Source: §4 · p. 7

RQ3

  • Few-shot GPT-3.5 achieved 46.38% - 659.09% higher EM than zero-shot GPT-3.5 without fine-tuning.Demonstration examples in the prompt guide revised-code generation.Source: §4 · p. 8
  • Adding a persona yielded 1.02% - 54.17% lower EM than omitting one in GPT-3.5 prompts.The examples describe persona prompts producing additional incorrect code changes.Source: §4 · pp. 8–9
  • Few-shot GPT-3.5 without a persona correctly suggested a new if condition, while the persona variant added an incorrect if statement and else block.The paper identifies few-shot learning without a persona as the best non-fine-tuned prompting strategy.Source: §4 · p. 9

5. Discussion

The discussion interprets the findings for practitioners, examines correctly generated code changes, and reports additional experiments on training-set size, prompt design, and usage costs.

5.1. Implications of Our Findings

  • GPT-3.5 fine-tuned on approximately 20k examples outperformed the studied approaches, and the authors recommend fine-tuning or, when data is insufficient, few-shot prompting without a persona.The recommendation follows the reported fine-tuning and prompting comparisons.Source: §5.1 · p. 9

5.2. The Characteristics of the Revised Code that are Correctly Generated by GPT-3.5

  • Across three datasets, fine-tuned GPT-3.5 led on Refactoring and Other, while few-shot GPT-3.5 led across categories on Tufano without comments.The figure compares Exact Match by code-change type across the studied GPT-3.5 variants.Source: Figure 6 · p. 10, §5.2 · p. 10
  • The qualitative sample was randomly selected at a 95% confidence level and 5% confidence interval, then classified as bug fixes, refactoring, or other.The categories distinguish fixing past bugs, changing code structure without behavior changes, and changes fitting neither category.Source: §5.2 · p. 10

5.3. The Impact of the Size of Training Dataset on Fine-Tuned GPT-3.5

  • Fine-tuning with 20% of a training set yielded 2.29% - 12.07% higher EM and 0.54% - 2.55% higher CodeBLEU than with 6%.Compared with 10% of a training set, 20% yielded 2.38% - 25.75% higher EM and 0.01% - 1.25% higher CodeBLEU.Source: §5.3 · pp. 10–11, Table 6 · p. 11

5.4. The Impact of Prompt Design on GPT-3.5

  • Simple-instruction prompts achieved 16.44% - 45.45% higher EM than stepwise prompts and 37.12% - 880.00% higher EM than detailed prompts in zero-shot learning.In few-shot learning, simple prompts achieved 5.15% - 290.00% higher EM than stepwise prompts and 5.61% - 515.79% higher EM than detailed prompts.Source: §5.4 · pp. 11–12, Table 7 · p. 12

5.5. Cost and Benefits of Using GPT-3.5 for Code Review Automation

  • At 300,000 queries per year, estimated GPT-3.5 costs were $600 for zero-shot, $1,050 for few-shot, and $2,740 for fine-tuning and use.The authors recommend fine-tuning for the highest EM, with few-shot learning without a persona as an alternative.Source: §5.5 · p. 12

6. Threats to Validity

The section identifies construct-validity concerns in example selection and persona design, internal-validity concerns in model randomness and fine-tuning settings, and limits on generalizing findings beyond the studied projects and model version.

6.1. Threats to Construct Validity

  • Three BM25-selected examples were used for few-shot learning; different example counts or selection techniques may produce different results.The authors suggest exploring more examples and alternative selection techniques.Source: §6.1 · p. 12
  • The persona was limited to an expert software developer; other roles, such as senior engineer or front-end developer, were not explored.The authors suggest testing other persona and prompt designs.Source: §6.1 · p. 12

6.2. Threats to Internal Validity

  • Results may vary with GPT-3.5 and Magicoder randomness, and the study did not search all fine-tuning hyperparameter combinations.Repeated experimental runs and broad hyperparameter searches were considered expensive; finding optimal settings was not the study’s goal.Source: §6.2 · p. 12

6.3. Threats to External Validity

  • Experiments used a dataset from recent work [5] [6], so findings may not generalize to other software projects.The authors identify evaluation on other projects as future work.Source: §6.3 · p. 13
  • Future OpenAI updates to GPT-3.5 may cause reproduced results to differ from those reported.Source: §6.3 · p. 13

7. Conclusion

The study compares fine-tuning and prompting strategies for GPT-3.5 and Magicoder in code review automation. It reports that fine-tuned GPT-3.5 performs best, while few-shot learning without a persona is the preferred prompting strategy when GPT-3.5 is not fine-tuned.

7. Conclusion

  • Fine-tuned GPT-3.5 performs best, while few-shot learning without a persona is the best prompting strategy for GPT-3.5 without fine-tuning.The authors recommend fine-tuning for highest performance, or few-shot learning without a persona when data is insufficient for fine-tuning.Source: §7 · p. 13
Loading 2402.00905v4…