PaperlayerView exact revision on arXiv

Source-linked AI summary

Distilling Desired Comments for Enhanced Code Review with Large Language Models

Yongda Yu, Lei Zhang, Guoping Rong, Haifeng Shen, Jiahao Zhang, Haoxiang Yan, Guohao Shi, Dong Shao, Ruiqi Pan, Yuan Li, Qiushi Wang, Zhao Tian

arXiv:2412.20340v2cs.SEcs.AI

TL;DR

Code-review LLMs need to generate comments that identify issues and prompt fixes, but suitable DRC-rich training data is lacking and manual annotation is impractical. Desiview distills DRCs from a code-review dataset to train and align LLaMA models. It achieves 86.67% accuracy on CodeReviewer, and human evaluation finds its models outperform base LLMs in issue identification and description.

  • Problem

    Existing LLM-based code review can be ineffective at generating desired review comments, and manual annotation of DRC-rich datasets is impractical.

    Source: §I · pp. 1–2

  • Method

    Desiview identifies DRCs in a code-review dataset to build a distilled dataset used to fine-tune LLaMA 3 and 3.1 and KTO-align the resulting models.

    Source: §I · p. 2

  • Results

    86.67% accuracy on CodeReviewer, versus GPT-4o’s 76.50%; human evaluation found both models identify issues more accurately and describe them better than base LLMs.

    Source: §I · p. 2

  • Takeaways & Limitations

    The distilled dataset supports training DRC classifiers and fine-tuning or aligning LLMs for improved code-review tasks.

    Source: §VII · p. 11

    The study uses CodeReviewer alone for training and testing because it is, to the authors’ knowledge, the only public multi-programming-language dataset with original submissions, fixes, and review comments.

    Source: §VI · p. 10

Abstract

from arXiv · show

There has been a growing interest in using Large Language Models (LLMs) for code review thanks to their proven proficiency in code comprehension. The primary objective of most review scenarios is to generate desired review comments (DRCs) that explicitly identify issues to trigger code fixes. However, existing LLM-based solutions are not so effective in generating DRCs for various reasons such as hallucination. To enhance their code review ability, they need to be fine-tuned with a customized dataset that is ideally full of DRCs. Nevertheless, such a dataset is not yet available, while manual annotation of DRCs is too laborious to be practical. In this paper, we propose a dataset distillation method, Desiview, which can automatically construct a distilled dataset by identifying DRCs from a code review dataset. Experiments on the CodeReviewer dataset comprising more than 150K review entries show that Desiview achieves an impressive performance of 88.93%, 80.37%, 86.67%, and 84.44% in terms of Precision, Recall, Accuracy, and F1, respectively, surpassing state-of-the-art methods. To validate the effect of such a distilled dataset on enhancing LLMs' code review ability, we first fine-tune the latest LLaMA series (i.e., LLaMA 3 and LLaMA 3.1) to build model Desiview4FT. We then enhance the model training effect through KTO alignment by feeding those review comments identified as non-DRCs to the LLMs, resulting in model Desiview4FA. Verification results indicate that Desiview4FA slightly outperforms Desiview4FT, while both models have significantly improved against the base models in terms of generating DRCs. Human evaluation confirms that both models identify issues more accurately and tend to generate review comments that better describe the issues contained in the code than the base LLMs do.

I. INTRODUCTION

The introduction frames desired review comments (DRCs) as comments that identify code issues and motivate fixes, while noting that existing LLMs and review datasets may not reliably support this goal. It introduces Desiview to identify DRCs automatically and use them to build code-review models.

  • DRCs identify code issues, describe their nature, and/or lead to meaningful code repairs, but existing LLM solutions may not generate them effectively.Source: §I · pp. 1–2
  • Manual annotation is impractical for a dataset with more than 150,000 review entries, and its quality can vary.Source: §I · p. 2
  • Desiview automatically identifies DRCs to construct a distilled dataset, then fine-tunes LLaMA 3 and 3.1 and KTO-aligns the resulting models.Source: §I · p. 2
  • 86.67% accuracy on CodeReviewer, surpassing GPT-4o’s 76.50%, is reported for Desiview’s identification of DRCs.Source: §I · p. 2
  • Both Desiview4FT and Desiview4FA significantly improve DRC generation over their base models on CodeReviewer.Source: §I · p. 2
  • Human evaluation finds both models identify issues more accurately and describe code issues better than the base LLMs.Source: §I · p. 2

II. RELATED WORK

Related work covers automated code review and LLM use in software engineering, including prompting, fine-tuning, and alignment. It notes that existing code-review studies treat review comments as ground truth without checking whether they correspond to code fixes, and that code-review alignment data remains an open need.

A. Automated code review

  • Automated code review includes defect detection and review-comment recommendation or generation, while existing studies may treat comments as ground truth without checking whether they relate to fixes.Source: A. Automated code review · p. 3

B. Large language models for software engineering

III. METHODOLOGY

The methodology aims to improve LLM generation of DRCs by constructing a customized dataset with a high proportion of desired comments. Desiview comprises DRC identification followed by dataset preparation and preprocessing.

III. METHODOLOGY

  • The methodology constructs a customized, high-DRC dataset, fine-tunes LLaMA 3 and LLaMA 3.1 to create Desiview4FT, and then aligns those models to create Desiview4FA.Source: §III · p. 4

A. Desiview: Constructing a distilled dataset

1) Identification of DRCs:

Desiview identifies desired review comments (DRCs) through their relationship to code fixes, using perplexity differences to score comments. It applies this method to CodeReviewer and reports that fewer than half of comments are DRCs, with similar proportions in the training and test sets.

1) Identification of DRCs:

  • Perplexity measures how uniformly a model predicts answer tokens; higher perplexity indicates a lower probability of generating the given content.Source: 1) Identification of DRCs · p. 4
  • Desiview scores a review comment by comparing code-fix perplexity with and without the comment, treating positive scores as desired and nonpositive scores as undesired.Source: 1) Identification of DRCs · p. 4

2) Dataset preparation and pre-processing:

B. Desiview4FT: Fine-tuning Large Language Models

Desiview4FT fine-tunes LLaMA-3 and LLaMA-3.1 on the distilled dataset using LoRA. The setup uses the smallest 8B versions because of GPU resource limitations.

B. Desiview4FT: Fine-tuning Large Language Models

C. Desiview4FA: Aligning Large Language Models

Desiview4FA aligns the fine-tuned model to encourage desired review comments, using KTO because code review usually provides only one comment per code piece. Its objective handles desired and undesired comments without paired data.

C. Desiview4FA: Aligning Large Language Models

IV. EVALUATION

The evaluation examines Desiview’s ability to identify desired review comments and whether models trained or aligned with its distilled data generate better comments. It uses CodeReviewer and compares against existing identification and code-review methods.

IV. EVALUATION

  • The evaluation asks how accurately Desiview identifies DRCs and whether fine-tuning and alignment with its distilled data improve LLM code review.The RQ2 comparison is against LLaMA-Reviewer, which uses the original dataset.Source: §IV · p. 6, A. Experimental settings · p. 7

A. Experimental settings

  • CodeReviewer is the evaluation dataset because it is publicly available and includes code before and after review alongside review comments.The paper describes it as the only publicly available dataset meeting the study’s needs.Source: A. Experimental settings · p. 6
  • DRC-identification benchmarks include the 10-line rule, GPT-3.5, and GPT-4o, while LLaMA-Reviewer serves as the code-generation baseline.LLaMA-Reviewer uses the same dataset and LLaMA base model, facilitating comparison.Source: A. Experimental settings · pp. 6–7
  • Figure 6 presents the evaluation process.The paper evaluates the method and the models trained or aligned using the distilled dataset.Source: Figure 6 · p. 7

1) The experiment for RQ1:

For RQ1, the study compares Desiview with rule-based and LLM identification methods using a manually annotated test sample. For RQ2, it compares original-data and distilled-data LLaMA models through automated and human evaluation.

1) The experiment for RQ1:

  • The manually annotated RQ1 sample contains 600 entries drawn randomly from 150,406 CodeReviewer training entries.Two graduate students annotated entries; 300 were duplicated to check consistency, with a Chi-Squared test p-value of 0.965.Source: 1) The experiment for RQ1 · p. 7
  • The RQ1 comparison includes the 10-line rule, GPT-3.5-turbo, and GPT-4o prompted to identify DRCs.The rule was simplified to whether modifications were made subsequently; GPT prompts were selected after testing different phrasings.Source: 1) The experiment for RQ1 · p. 7, Figure 7 · p. 7

2) The experiment for RQ2:

  • RQ2 compares LLaMA-Reviewer with Desiview4FT and Desiview4FA, using the same fine-tuning settings as the baseline study.Desiview4FT uses distilled data, while Desiview4FA aligns on distilled DRCs and dropped non-DRCs.Source: 2) The experiment for RQ2 · p. 7
  • Automated evaluation compares generated comments with the 5,727-entry test set using BLEU-4 text similarity.The study treats the test-set comments as ground truth because the distilled dataset has a high proportion of comments that lead to fixes.Source: 2) The experiment for RQ2 · p. 7
  • Human evaluation samples 300 of the 5,727 DRC entries and assesses issue localization separately from issue description.Human Position checks issue location; Human Perfect requires matching issues and/or solutions, building on localization.Source: 2) The experiment for RQ2 · pp. 7–8

B. Results analysis

Desiview outperforms the compared identification methods on comprehensive metrics, while models trained with the smaller distilled dataset outperform original-data models in issue localization and description. Alignment further extends the advantage.

B. Results analysis

  • Desiview significantly outperforms the 10-line rule, GPT-3.5-turbo, and GPT-4o on accuracy and F1, while the rule reaches 100% recall but performs poorly on comprehensive metrics.Identifying DRCs requires understanding both code and comments, not only fix proximity, phrasing, or keywords.Source: Table III · p. 8, B. Results analysis · p. 8
  • The distilled dataset, less than half the original dataset’s size, significantly outperforms full-data fine-tuning in localizing and describing code issues.Alignment further extends this advantage, and the distilled dataset improves both LLaMA-3 and LLaMA-3.1.Source: Table IV · p. 8, B. Results analysis · p. 8
  • LLaMA-3.1 shows no clear advantage over LLaMA-3 but appears more sensitive to fine-tuning and alignment with the distilled dataset.The authors report that the distilled dataset improves both model versions.Source: B. Results analysis · p. 8
  • Examples show that distilled-data training improves issue identification, while alignment improves accuracy and reduces irrelevant information.Figure 9 presents examples from the three methods based on LLaMA-3.Source: B. Results analysis · p. 8

V. DISCUSSION

The discussion presents Desiview as a way to obtain high-quality review data for model training and explores reuse of the distilled data for review-comment quality prediction.

A. Distilled dataset for training code review models

  • Desiview uses the relationship between review comments and code fixes to assess comment desiredness and acquire large-scale review data automatically at low cost.The authors frame the method as addressing scarcity of high-quality code-review datasets.Source: A. Distilled dataset for training code… · p. 8, §V · p. 8
  • The desiredness criterion could be adapted to other review goals, including performance bottlenecks and security risks, though further exploration is needed.The authors also identify vulnerability detection as a possible software-engineering objective.Source: A. Distilled dataset for training code… · pp. 8–9

B. Distilled dataset for training review quality prediction models

  • The distilled data can train a binary classifier to predict whether a generated review comment will trigger a code fix, using a code diff and comment as input.The study uses RoBERTa and CodeBERT as base models.Source: B. Distilled dataset for training… · pp. 9–10
  • A comment-quality assessment mechanism could help developers decide whether to accept generated comments or filter low-quality comments from system outputs.The reported prediction results indicate potential for this use, but the passage gives no metric values.Source: B. Distilled dataset for training… · p. 10, Table VI · p. 10

VI. THREATS TO VALIDITY

The validity risks concern the scope of the DRC definition, residual dataset noise, evaluation and model choices, training resources, and reliance on CodeReviewer.

Definition of quality in terms of desiredness

  • The DRC-based quality definition focuses on comments that trigger fixes, leaving other meaningful review purposes, such as acknowledging a fix, outside its scope.The authors report that the resulting dataset still contained close to 65,000 DRCs, enough for LLaMA fine-tuning.Source: §VI · p. 10

Noise in the distilled dataset

  • A small number of non-DRCs may remain in the distilled CodeReviewer dataset, including its test set, and could bias RQ2, though the authors consider the risk controllable.They describe the residual amount as minimal and cite human-evaluation results as corroboration.Source: §VI · p. 10

Pre-trained models

  • Because pre-trained models vary in code-review performance, using different LLMs to assess DRCs could affect results despite the multi-model voting strategy.The voting used multiple models without preference and showed a positive effect.Source: §VI · p. 10

Model parameter size

  • GPU limits restricted computations to models under 16B parameters and 8B models for fine-tuning and alignment; larger models could potentially improve performance.The authors also identify parameter scale as one possible reason for the review-comment quality gap with GPT-4o.Source: §VI · p. 10

Model training methods

  • Resource constraints led the study to use low-parameter training, while full-parameter training may perform better at significantly higher cost.The authors cite prior research indicating that full-parameter training outperforms low-parameter training.Source: §VI · p. 10

Dataset

  • The study used only CodeReviewer, the only public multi-language dataset it knew of with submissions, fixes, and review comments, limiting generalization evidence.The authors encourage constructing other datasets to validate the work.Source: §VI · pp. 10–11

Errors from human evaluation

  • Manual-evaluation errors remain possible despite standardized methods and consistency checks; software-engineering expertise, evaluator overlap, and chi-square tests were used to mitigate this risk.The authors note that evaluators were selected for expertise in relating code to review comments.Source: §VI · p. 11

VII. CONCLUSIONS

The conclusion presents Desiview as a way to identify DRCs and construct distilled code-review datasets. Experiments and evaluations report stronger DRC identification and generation outcomes, while future work includes broader objectives and stronger models.

VII. CONCLUSIONS

  • Desiview surpassed other methods, including GPT-4o, in accuracy for identifying DRCs.The conclusion reports this comparison without specifying a value.Source: §VII · p. 11
  • Automated and human evaluations found that LLMs trained with the distilled dataset outperformed those trained with the original dataset on code-review tasks generating DRCs.Source: §VII · p. 11
  • The distilled dataset contains a high proportion of DRCs and supports both predicting whether a new comment is a DRC and fine-tuning or aligning LLMs for DRC generation.Source: §VII · p. 11
  • Future work includes distilling datasets for different code-review objectives and exploring stronger base models and newer fine-tuning and alignment techniques.Source: §VII · p. 11

Related papers

Loading 2412.20340v2…