PaperlayerView exact revision on arXiv

Source-linked AI summary

LLaMA-Reviewer: Advancing Code Review Automation with Large Language Models through Parameter-Efficient Fine-Tuning

Junyi Lu, Lei Yu, Xiaojia Li, Li Yang, Chun Zuo

arXiv:2308.11148v2cs.SEcs.CLcs.LG

TL;DR

Code-review automation has relied on domain-specific pre-trained models that can require substantial resources, while LLMs’ potential for these tasks remains underexplored. LLaMA-Reviewer uses PEFT with LLaMA, and the authors report matching state-of-the-art code-review-focused models with a 6.7B-parameter base model and fewer than 1% trainable parameters.

  • Problem

    The potential of LLMs for automating code-review tasks remains largely unexplored despite the resource demands of domain-specific pre-training.

    Source: §I · p. 1, A. Research Questions · p. 5

  • Method

    LLaMA-Reviewer applies PEFT to LLaMA for code-review automation and evaluates two PEFT methods and fine-tuning components.

    Source: §I · p. 1

  • Results

    With the 6.7B-parameter LLaMA model and fewer than 1% trainable parameters, LLaMA-Reviewer matches state-of-the-art code-review-focused models.

    Source: §VIII · p. 11

  • Takeaways & Limitations

    The ablations suggest that pre-training-aligned input formats, initial instruction tuning, and appropriately ranked LoRA can benefit task performance.

    Source: §VIII · p. 11

Abstract

from arXiv · show

The automation of code review activities, a long-standing pursuit in software engineering, has been primarily addressed by numerous domain-specific pre-trained models. Despite their success, these models frequently demand extensive resources for pre-training from scratch. In contrast, Large Language Models (LLMs) provide an intriguing alternative, given their remarkable capabilities when supplemented with domain-specific knowledge. However, their potential for automating code review tasks remains largely unexplored. In response to this research gap, we present LLaMA-Reviewer, an innovative framework that leverages the capabilities of LLaMA, a popular LLM, in the realm of code review. Mindful of resource constraints, this framework employs parameter-efficient fine-tuning (PEFT) methods, delivering high performance while using less than 1% of trainable parameters. An extensive evaluation of LLaMA-Reviewer is conducted on two diverse, publicly available datasets. Notably, even with the smallest LLaMA base model consisting of 6.7B parameters and a limited number of tuning epochs, LLaMA-Reviewer equals the performance of existing code-review-focused models. The ablation experiments provide insights into the influence of various fine-tuning process components, including input representation, instruction tuning, and different PEFT methods. To foster continuous progress in this field, the code and all PEFT-weight plugins have been made open-source.

I. INTRODUCTION

Code review remains a substantial manual workload despite automation research, while the potential of large language models for these tasks is underexplored. LLaMA-Reviewer applies PEFT to LLaMA for code-review automation, building on a three-task review pipeline.

I. INTRODUCTION

  • Code review supports defect identification, quality improvement, and knowledge sharing, but still imposes substantial developer effort and has prompted automation research across several tasks.Prior work includes reviewer recommendation, code-quality evaluation, code refinement, and review-comment generation.Source: §I · p. 1
  • Domain-specific pre-trained models can require substantial resources, whereas large language models offer a promising code-review alternative without domain-specific pre-training.The paper notes that LLMs’ potential for code-review automation remains largely unexplored.Source: §I · p. 1
  • LLaMA-Reviewer applies PEFT to LLaMA for code-review automation, building on review-necessity prediction, comment generation, and code refinement.The contributions include an offline, privacy-conscious alternative to closed-source APIs, a unified-model-plus-PEFT paradigm, and evaluation of two PEFT methods and fine-tuning ablations.Source: §I · p. 1

A. Automation in Code Review

  • The review cycle covers pull requests through merge or rejection, with reviewers predicting review need and commenting, and committers refining code.The study maps these steps to review-necessity prediction, review-comment generation, and code refinement.Source: A. Automation in Code Review · p. 2

B. Large Language Models

  • LLMs remain underexplored in software-engineering tasks; the paper uses unified LLaMA because code review differs from code generation.The cited background distinguishes unified LLMs, trained predominantly on natural language with some code, from code LLMs trained primarily on code.Source: B. Large Language Models · p. 2

C. Parameter-Efficient Fine-Tuning

LLaMA-Reviewer uses instruction tuning followed by task-specific fine-tuning, with PEFT methods providing lightweight task-specific plugins. The section describes the tuning setup and zero-init attention prefix-tuning.

C. Parameter-Efficient Fine-Tuning

  • The study fine-tunes LLaMA using zero-init attention prefix-tuning [49] and LoRA tuning [48], reporting that these PEFT methods introduce no extra model latency and have demonstrated effectiveness in natural language tasks.Source: C. Parameter-Efficient Fine-Tuning · p. 2

A. Overview

  • LLaMA-Reviewer first instruction-tunes LLaMA on code-centric data, then fine-tunes it for each code-review subtask using prefix-tuning or LoRA.The task-specific plugins are independent of the base model weights and can be integrated at inference.Source: A. Overview · p. 2, C. Parameter-Efficient Fine-Tuning · p. 2
  • The framework is evaluated on the CRer and Tuf. datasets.These refer to the CodeReviewer dataset and the dataset from Tufano et al., respectively.Source: A. Overview · p. 2

B. Instruction Tuning on LLaMA

  • The instruction-tuning stage adapts LLaMA to code review using Code Alpaca data and an instruction, optional input, and output format.Combining Alpaca and Code Alpaca data did not improve performance in the reported ablation experiments.Source: B. Instruction Tuning on LLaMA · p. 3

C. Zero-init Attention Prefix-tuning

  • Zero-init attention prefix-tuning freezes the base model and trains added prompt tokens in the topmost transformer layers.A learnable gating factor, initialized to zero, regulates attention between inserted prompt tokens and original tokens.Source: C. Zero-init Attention Prefix-tuning · p. 3

D. Low-Rank Adaptation

LoRA approximates weight adjustments with trainable low-rank matrices while keeping the pretrained model weights fixed.

  • LoRA keeps the original model weights fixed and uses trainable low-rank matrices to approximate weight adjustments.The approach relies on the premise that adaptation has low intrinsic rank.Source: D. Low-Rank Adaptation · p. 4
  • The LoRA update uses Wdown ∈ R^(d×r) and Wup ∈ R^(r×k), with r ≪ min(d, k).W0 remains unchanged during fine-tuning, while Wdown and Wup are trainable.Source: D. Low-Rank Adaptation · p. 4
  • For input x and original output h, the adjusted output is computed by adding the low-rank update to the original model output.The displayed expression is ¯h = W0x + WdownWupx.Source: D. Low-Rank Adaptation · p. 4

E. Code Review Automation Tasks

LLaMA-Reviewer targets three sequential code-review tasks, from deciding whether review is needed to generating comments and refining code.

E. Code Review Automation Tasks

  • The framework covers review necessity prediction, review comment generation, and code refinement, corresponding to sequential stages of code review.Inputs and outputs are represented as code snippets, natural-language comments, and binary review-needed labels.Source: E. Code Review Automation Tasks · p. 4
  • Table I summarizes the tasks, their input and output formats, and associated datasets.The dataset abbreviations are Crer. for CRer and Tuf. for the Tufano dataset.Source: E. Code Review Automation Tasks · p. 4

1) Review Necessity Prediction:

  • Review necessity prediction checks whether a diff hunk requires review comments; diff hunks can be long enough to challenge input handling.The task is attributed to Li et al., who proposed the diff-hunk framing.Source: 1) Review Necessity Prediction · p. 4

2) Code Review Comment Generation:

The section describes comment generation at line and method levels and frames experiments across code-review tasks, input representations, instruction tuning, and PEFT choices.

2) Code Review Comment Generation:

  • Comment generation is evaluated from line-level code on CRer and method-level context on Tufano.The section also introduces code refinement as minor adjustments or reordering of existing code.Source: 2) Code Review Comment Generation · p. 4

3) Code Refinement:

  • Code-refinement inputs follow the Tufano and CRer datasets, while language-type information is omitted for a fair comparison with baselines.The section notes that language-type information is available in only one task of one published dataset and says its implications are examined later.Source: 3) Code Refinement · p. 4

A. Research Questions

  • The experimental design is organized around research questions comparing LLaMA with state-of-the-art approaches across review necessity, comment generation, and code refinement.The study uses the smallest LLaMA variant to assess these tasks.Source: §IV · p. 4, A. Research Questions · p. 5
  • Input-representation questions examine code formatting and whether programming-language labels are included and where they appear in prompts.The formatting comparison uses raw code versus modified formatting, including removal of consecutive spaces.Source: A. Research Questions · p. 5
  • Instruction-tuning questions test its influence on prefix-tuning and LoRA, and compare mixed natural- and programming-language instructions with programming-language-only instructions.The section motivates these tests as a way to examine domain knowledge and subtask understanding.Source: A. Research Questions · p. 5
  • PEFT questions compare the two methods, examine how LoRA rank affects trainable parameters and performance, and compare parameter and storage efficiency with prior approaches.The section identifies balancing efficiency and effectiveness as a challenge.Source: A. Research Questions · p. 5

B. Datasets

The study uses CRer and Tufano datasets spanning code-review tasks, then applies task-specific evaluation metrics and a top-1 output criterion.

B. Datasets

  • CRer and Tufano were selected because they cover the full code-review process and draw from diverse repositories.The section contrasts this coverage with datasets limited to particular subtasks or a smaller repository pool.Source: B. Datasets · p. 5
  • Tufano is presented as a more recent and comprehensive version than datasets used in earlier studies,,.Source: B. Datasets · p. 5
  • CRer is a multi-language, diff-aware, line-grained GitHub corpus that preserves inline comments, docstrings, and consecutive spaces.It has separate subsets for review-necessity prediction, comment generation, and code refinement.Source: B. Datasets · p. 5
  • Tufano is a Java, function-grained dataset from GitHub and Gerrit; it removes comments and consecutive spaces and does not represent commit-to-base differences.For refinement, comments mark focus areas with ⟨START⟩ and ⟨END⟩; the study uses its comment-generation and refinement subsets.Source: B. Datasets · p. 6

C. Evaluation Criteria

  • Review-necessity prediction is treated as binary classification, with requiring review as the positive class, and is measured using precision, recall, and F1-score.Source: C. Evaluation Criteria · p. 6
  • Comment generation and code refinement use BLEU-4, while codeBLEU is excluded because it is incompatible with CRer; all tasks use the top-1 result.Top-1 aligns with the stated objective of focusing on the most relevant feedback.Source: C. Evaluation Criteria · p. 6

D. Baselines

The section specifies baseline selection and implementation, then reports review-necessity results in which LLaMA-Reviewer improves recall while retaining comparable F1.

D. Baselines

  • Baselines were selected for task and dataset needs, with some studies excluded because they target smaller datasets or require additional input information.CRer baseline results came from CodeReviewer, CommentFinder was reproduced on both datasets, and other results used supplied models.Source: D. Baselines · p. 6, E. Implementation Details · p. 6

E. Implementation Details

  • Experiments used NVIDIA A100-SXM4-80GB GPUs, a 2048-token limit, batch size 64, AdamW, and 5 training epochs for review necessity.Prefix-tuning and LoRA used different specified learning rates, weight decay, and configuration settings.Source: E. Implementation Details · p. 6

1) (RQ1.1) Review Necessity Prediction Performance:

  • The review-necessity experiment uses CRer because it is the only dataset with the required data; results are compared with those reported in CodeReviewer.Source: 1) (RQ1.1) Review Necessity Prediction… · p. 6
  • LLaMA-Reviewer achieves superior recall with a comparable F1 score, and at threshold 0.5 its precision is 88.61%, exceeding all baselines.The reported recall and F1 results follow threshold adjustment; prefix-tuning is omitted because its rigid training structure does not suit classification.Source: 1) (RQ1.1) Review Necessity Prediction… · p. 6

2) (RQ1.2) Review Comment Generation Performance:

LLaMA-Reviewer performs especially well on CRer comment generation, while its code-refinement performance remains competitive with specialized models and exceeds other baselines.

2) (RQ1.2) Review Comment Generation Performance:

3) (RQ1.3) Code Refinement Performance:

  • On both datasets, LLaMA-Reviewer competes closely with CodeReviewer or Tufano et al. [23] in code refinement and exceeds other baselines.The comparison uses the smallest LLaMA version and limited tuning epochs; more diverse valid refinements may score lower in BLEU against a singular reference.Source: 3) (RQ1.3) Code Refinement Performance · pp. 7–8, Table VI · p. 8

B. RQ2: The Influence of Input Representation

The input-representation experiments find that LLaMA-Reviewer performs better when code resembles its pre-training format. Language labels help when paired with instruction tuning, but not without it.

1) (RQ2.1) Consequences of Code Formatting:

  • Pre-training-like code representation is associated with superior relative performance on CRer than Tufano, whose code is more heavily processed.The authors suggest that pre-training-like representations better leverage the model’s understanding of code structures and semantics.Source: 1) (RQ2.1) Consequences of Code… · p. 8

2) (RQ2.2) Role of Language Label:

  • Language labels do not improve performance without instruction tuning, but improve it when instruction tuning is used (paired-bootstrap p-value 0.0032).The experiments used the CRer dataset, integrating labels into either the instruction or input while holding other settings constant.Source: 2) (RQ2.2) Role of Language Label · p. 8, Table VII · p. 8

B. RQ2: The Influence of Input Representation

C. RQ3: The Impact of Instruction Tuning

The instruction-tuning experiments find different effects across PEFT methods and instruction data. Instruction tuning can help LoRA, while prefix tuning does not benefit and Alpaca data may reduce performance.

1) (RQ3.1) Consequences for Zero-init Attention Prefix-tuning:

2) (RQ3.2) Consequences for Low-Rank Adaptation:

  • With LoRA, instruction tuning raises review-necessity precision from 81.56% to 83.99% on PL, but does not improve its F1-score.The precision increase is reported when PL is the sole tuning set.Source: 2) (RQ3.2) Consequences for Low-Rank… · p. 9
  • With LoRA, instruction tuning increases BLEU for review-comment generation, while code-refinement improvement is not substantial.The authors infer that instruction tuning improves the model’s ability to incorporate language labels in code refinement.Source: 2) (RQ3.2) Consequences for Low-Rank… · p. 9

3) (RQ3.3) Influence of Instruction Types:

  • Adding Alpaca data alongside Code Alpaca during LoRA instruction tuning diminishes performance across all tasks.The authors suggest that Alpaca’s diverse instructions, including rewrite and classify tasks, may overwhelm code-review tasks.Source: 3) (RQ3.3) Influence of Instruction… · p. 9
  • The additional instruction-data experiments contrasted PL+NL, using Alpaca and Code Alpaca, with PL, using only Code Alpaca.For LoRA, the reported PL+NL results were worse across tasks than PL.Source: 3) (RQ3.3) Influence of Instruction… · p. 9
  • Instruction tuning can potentially improve task performance or handling of additional natural-language information, but its effectiveness is minor amid instruction–downstream-dataset word-habit inconsistency.The instruction-type experiments used LoRA on the CRer dataset; supplementary instructions were investigated to identify suitable tuning data.Source: C. RQ3: The Impact of Instruction Tuning · p. 8, 3) (RQ3.3) Influence of Instruction… · p. 9

2) (RQ4.2) Impact of LoRA Rank r:

Increasing LoRA rank improves performance, although it adds trainable parameters and requires balancing performance against efficiency. The paper also describes LLaMA tuning and the code-review task pipeline motivating its application.

2) (RQ4.2) Impact of LoRA Rank r:

  • Increasing LoRA rank r from 8 to 16 improves performance, while adding trainable parameters makes balancing performance and efficiency important.A higher rank brings the model closer to full-parameter tuning.Source: 2) (RQ4.2) Impact of LoRA Rank r · p. 9

3) (RQ4.3) Efficiency of PEFT Methods:

  • PEFT methods use under 1% trainable parameters and reduce storage from 13GB to under 20MB; the authors judge LoRA more suitable for code-review automation.The base-model weights remain constant, and independent plug-in weights support multi-task processes.Source: 3) (RQ4.3) Efficiency of PEFT Methods · p. 9

A. Tuning on LLaMA

  • Prior LLaMA tuning work spans language adaptation, text quality, multimodal input, and code capabilities; this study evaluates LLaMA with state-of-the-art PEFT methods.Prior work also includes Alpaca and PEFT approaches such as Alpaca LoRA and zero-init attention prefix-tuning.Source: A. Tuning on LLaMA · pp. 9–10

B. Automation of Code Review Activities

  • The study focuses on Li et al.’s code-review pipeline of review-necessity prediction, comment generation, and code refinement, an area where unified LLMs had received limited attention.The authors motivate using LLaMA to assess unified LLM suitability for tasks involving both programming and natural language.Source: B. Automation of Code Review Activities · p. 10

VII. THREATS TO VALIDITY

The paper reports competitive code-review performance with a small LLaMA model and PEFT, while noting evaluation, training, and dataset boundaries. It proposes broader model and PEFT exploration as future work.

A. Construct Validity

  • Evaluation relies primarily on a BLEU-4 variant, which the authors say is not universally recognized as definitive for comments and refined snippets.ROUGE was omitted because several baselines lacked direct results or fine-tuned models; reproduced baseline results may vary with data formats and distributions.Source: A. Construct Validity · p. 10

B. Internal Validity

  • The PEFT assessment covers only two methods and lacks a full-parameter comparison; resource constraints also limited ablation settings.The authors estimate full-parameter tuning would require 8 A100-SXM4-80G over half a month.Source: B. Internal Validity · p. 10
  • Using the smallest LLaMA and a finite number of epochs may understate latent LLM capacity as model size and tuning duration increase.The authors identify these choices as an internal-validity threat.Source: B. Internal Validity · p. 10

C. External Validity

  • Findings may not generalize beyond two datasets derived exclusively from open-source projects, and retaining one comment per change may bias filtering.The authors note that the findings may not fully apply to industrial and other contexts.Source: C. External Validity · pp. 10–11

VIII. CONCLUSION AND FUTURE WORK

  • With 6.7B parameters and under 1% trainable parameters, LLaMA-Reviewer matches state-of-the-art code-review-focused models after limited tuning epochs.The conclusion also reports reduced storage requirements from plug-in models.Source: §VIII · p. 11
  • Future work will examine models of varied sizes and types, additional PEFT methods, and prompt, code, comment, and sequence-block token lengths.Source: §VIII · p. 11
Loading 2308.11148v2…