PaperlayerView exact revision on arXiv

Source-linked AI summary

Automating Code Review Activities by Large-Scale Pre-training

Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Majumder, Jared Green, Alexey Svyatkovskiy, Shengyu Fu, Neel Sundaresan

arXiv:2203.09095v2cs.SEcs.AI

TL;DR

Code review is important but requires developers to spend substantial time examining code changes from multiple perspectives. The paper addresses this by pre-training CodeReviewer on multilingual code-review data with four review-specific tasks, and reports that it outperforms prior state-of-the-art models across three code-review tasks. The authors also report benefits from the proposed tasks and multilingual dataset for understanding code changes and reviews.

  • Problem

    Code review requires substantial effort to assess code changes across logic, functionality, complexity, style, and documentation, motivating automation of review activities.

  • Method

    The paper pre-trains an encoder-decoder CodeReviewer model on a large GitHub dataset of code diffs and review comments using four code-review-specific pre-training tasks.

  • Results

    CodeReviewer outperforms state-of-the-art models pre-trained on source code across code change quality estimation, review comment generation, and code refinement.

  • Takeaways & Limitations

    The proposed pre-training tasks and multilingual dataset benefit the model's understanding of code changes and reviews.

  • Takeaways & Limitations

    The evaluation data comes only from open-source GitHub projects, and each code change is represented with a single review comment despite reviews often involving multiple reviewers.

Abstract

from arXiv · show

Code review is an essential part to software development lifecycle since it aims at guaranteeing the quality of codes. Modern code review activities necessitate developers viewing, understanding and even running the programs to assess logic, functionality, latency, style and other factors. It turns out that developers have to spend far too much time reviewing the code of their peers. Accordingly, it is in significant demand to automate the code review process. In this research, we focus on utilizing pre-training techniques for the tasks in the code review scenario. We collect a large-scale dataset of real-world code changes and code reviews from open-source projects in nine of the most popular programming languages. To better understand code diffs and reviews, we propose CodeReviewer, a pre-trained model that utilizes four pre-training tasks tailored specifically for the code review scenario. To evaluate our model, we focus on three key tasks related to code review activities, including code change quality estimation, review comment generation and code refinement. Furthermore, we establish a high-quality benchmark dataset based on our collected data for these three tasks and conduct comprehensive experiments on it. The experimental results demonstrate that our model outperforms the previous state-of-the-art pre-training approaches in all tasks. Further analysis show that our proposed pre-training tasks and the multilingual pre-training dataset benefit the model on the understanding of code changes and reviews.

1 INTRODUCTION

Code review is valuable but labor-intensive, motivating automation that understands code changes and review comments. The paper introduces CodeReviewer, review-specific pre-training tasks, and multilingual datasets, reporting improvements over prior models across three code-review tasks.

  • Motivation: Reviewing code across logic, functionality, complexity, style, and documentation requires substantial developer time, creating demand for automation.In Qt, 1,437 reviewers provided more than 1M reviews over four years.
  • Motivation: Existing source-code pre-trained models are poorly suited to code review because they are trained on raw source files rather than code changes and reviews.The paper therefore targets understanding code diffs and their associated review comments.
  • Approach: CodeReviewer is an encoder-decoder transformer pre-trained on code-diff hunks and review comments using four review-specific tasks.The tasks are diff tag prediction, denoising code diff, denoising review comment, and review comment generation.
  • Datasets: The authors construct a large-scale GitHub dataset and benchmark covering code changes and reviews in nine popular programming languages.They describe it as the largest multilingual code-review dataset with complete information on code changes and reviews.
  • Evaluation: The benchmark evaluates code change quality estimation, review comment generation, and code refinement, with CodeReviewer outperforming previous state-of-the-art models on all three tasks.Further analysis supports the effectiveness of the proposed pre-training tasks and multilingual dataset.

2 CODE REVIEW AUTOMATION TASKS

The paper frames code review automation around three separate tasks operating on diff hunks: estimating change quality, generating review comments, and refining code from comments.

  • 2 CODE REVIEW AUTOMATION TASKS: Code review automation targets three tasks: quality estimation, review comment generation, and code refinement.These tasks address both reviewers’ and contributors’ workloads.
  • 2.2 Code Change Quality Estimation: Quality estimation predicts whether a code change is high-quality and ready for acceptance.The estimates let reviewers prioritize questionable changes and contributors improve low-quality changes before review.
  • 2.3 Code Review Generation: Review generation maps a code diff and its context to a natural-language review comment focused on changed code.The model can generate candidate comments that reviewers may choose instead of writing manually.
  • 2.4 Code Refinement: Code refinement takes contributor code and a reviewer comment as input and generates revised code implementing the comment’s requirements.This task specifically uses review feedback as guidance for assisting contributors.
  • 2.5 Data Format: All three tasks use diff-hunk-level inputs because files or functions can contain multiple dispersed comments and revisions.For refinement, the input and output lines are extracted from the revision diff hunk.

3 CODE REVIEW DATASET

The dataset is built from publicly available GitHub pull-request data in popular, high-quality repositories across nine programming languages, then processed into pre-training and benchmark datasets.

  • 3 CODE REVIEW DATASET: GitHub pull requests provide publicly available code changes, review comments, authors, and timestamps for dataset construction.The collection uses the GitHub REST API to access repository and review information.
  • 3 CODE REVIEW DATASET: Projects span C, C++, C#, Go, Java, JavaScript, PHP, Python, and Ruby, with the top 10,000 projects selected per language.Projects are ranked by stars and repositories without explicit redistribution permission are removed.
  • 3.3 Dataset Construction: Commented changes support quality estimation, review generation, and code refinement datasets through task-specific filtering and processing.Quality estimation balances commented and uncommented changes, generation removes author comments and keeps the earliest comment per hunk, and refinement follows revisions.
  • 3.3 Dataset Construction: Project-level splitting separates repositories with more than 2,500 pull requests for pre-training and benchmark training from repositories used for validation and testing.This split is intended to prevent information leakage.

4 CODEREVIEWER

CodeReviewer is an encoder-decoder Transformer designed for code review, using code-review data and four specialized pre-training tasks. Its inputs and outputs vary across code understanding, comment generation, and code refinement tasks.

  • Model and pre-training: CodeReviewer is an encoder-decoder Transformer model designed with four pre-training tasks for automating code review activities.The tasks target understanding code diffs, denoising code and comments, and generating review comments.
  • Task interfaces: For classification, the encoder’s [CLS] representation produces predictions, whereas generation tasks use the full encoder-decoder to produce target sequences.Code refinement additionally takes original source code and review comments as input.
  • Pre-training tasks: Diff Tag Prediction masks diff markers and trains the model to recover [ADD], [DEL], or [KEEP] tags, distinguishing unchanged from updated lines.This task is intended to improve understanding of the diff format.
  • Pre-training tasks: Denoising Code Diff masks 15% of code lines, while Denoising Review Comment masks spans with a 20% corruption rate for sequence reconstruction.Code-diff corruption occurs at line level to preserve diff integrity; review comments are generated by the decoder.
  • Pre-training tasks: Review Comment Generation uses code changes as input and generates associated natural-language comments to model the relationship between code changes and reviews.This task uses bimodal data consisting of programming-language code changes and natural-language review comments.

5 STUDY DESIGN

The study evaluates CodeReviewer on three code-review tasks and examines the roles of individual pre-training tasks and multilingual data. It compares the model with a Transformer trained from scratch and two pre-trained baselines using task-specific metrics.

  • Research questions: The evaluation covers code change quality estimation, review comment generation, and code refinement.These tasks respectively predict whether a change needs review, generate reviewer-like comments, and revise code using review feedback.
  • Research questions: Ablation experiments remove pre-training tasks one at a time to measure their influence on downstream performance.The full model is pre-trained on all four tasks before comparison with each reduced-task variant.
  • Research questions: Monolingual Java, C#, and Ruby datasets are compared with the nine-language dataset to test whether multilingual pre-training benefits single-language performance.Java represents a popular language, while Ruby represents a low-resource language in this analysis.
  • Baseline models: The baselines are a Transformer trained from scratch, T5 for code review, and CodeT5-base.The comparison is intended to assess the multilingual dataset and tailored pre-training tasks.
  • Evaluation metrics: Code change quality estimation uses accuracy, precision, recall, and F1, treating changes requiring comments or updates as the positive class.This is evaluated as a binary classification task.
  • Evaluation metrics: Review comment generation uses BLEU-4 and human ratings of information and relevance, while code refinement uses BLEU and exact match.Exact match is emphasized for refinement because similar code can still compile or execute incorrectly.

6 RESULTS ANALYSIS

CodeReviewer is evaluated across code change quality estimation, review comment generation, and code refinement, with additional analyses of pre-training tasks and multilingual data. It outperforms prior models across the evaluated tasks, while review generation remains difficult.

  • RQ1: Performance on Code Change Quality Estimation: 8.24% F1 and 7.07% accuracy improvements over T5 are reported for code change quality estimation.The model also improves over CodeT5 by over/about 7%, while a scratch-trained Transformer performs worse than the other models.
  • RQ2: Performance on Review Generation: Review generation achieves a higher BLEU score than baseline models, but BLEU remains below 10 because review comments are diverse and non-unique.Human evaluation is used because similar review intent can receive substantially different wording.
  • RQ3: Performance on Code Refinement: CodeReviewer generates repaired code exactly matching the ground truth in more than 30% of cases.This is reported as twice T5’s result and 25% higher than CodeT5 relatively; its BLEU score also exceeds T5 and CodeT5.
  • RQ4: Influence of Pre-training Tasks: Removing pre-training tasks reduces performance, with Diff Tag Prediction and Denoising Code Diff identified as especially important for quality estimation.The ablation study evaluates models without DTP, DCD, or the comment-related tasks.
  • RQ5: Influence of Multilingual Dataset: Multilingual pre-training improves average accuracy by 2.32% and F1 by 1.10% over monolingual models.The multilingual model consistently outperforms monolingual Java, C#, and Ruby models.

7 RELATED WORKS

Related work applies deep learning and pre-training to software engineering, but prior models largely overlook code-review-specific tasks and code changes as inputs. Research also spans reviewer recommendation, review analysis, comment generation, code refinement, and dataset construction.

  • Pre-trained Models: Pre-trained software-engineering models support downstream code tasks, including classification, understanding, and generation.Examples include CuBERT, CodeBERT, and GraphCodeBERT, which target different programming-language and task scopes.
  • Pre-trained Models for Code Review: Earlier pre-trained models generally do not model code review tasks, whose distinctive input is a code change rather than a raw source file.Tufano et al. use T5 for code-review automation but do not tailor pre-training objectives to code changes.
  • Automating Code Review Activities: Automated code-review research includes reviewer recommendation, comment-location prediction, review-comment recommendation, and code refinement.Prior systems use approaches such as CNN-LSTM, Transformers, multi-instance learning, retrieval, and LSTM-based recommendation.
  • Code Review Datasets: Existing code-review datasets vary in scale and completeness, while this work reports a multilingual dataset with complete information from over 1,000 repositories and more than 7.9M pull requests.The authors contrast their dataset with collections that provide comments or source-code metadata without complete code information.

8 THREATS TO VALIDITY

The paper identifies internal-validity concerns about limited hyperparameter exploration and external-validity concerns about its GitHub-based, single-comment dataset. It also questions BLEU's suitability for review generation and supplements it with human annotation.

  • Internal validity: Limited hyperparameter tuning leaves the separate effects of model architecture and hyperparameter settings unresolved.The authors performed only a small grid search over learning rate and batch size, while keeping other settings from CodeT5; more tuning may improve results.
  • External validity: The dataset covers open-source GitHub projects but not industrial projects, constraining external validity.The collection source limits the evidence base to open-source development settings.
  • External validity: Using one review comment per code change may bias the dataset because real changes can receive multiple perspectives.The paper notes that code review often involves multiple reviewers who may provide different comments.
  • Construct validity: BLEU may not adequately evaluate review-comment generation, so the authors add human annotation.They identify metric suitability as a construct-validity concern and use human annotation to improve evaluation.

9 CONCLUSION

The paper presents a large-scale, nine-language GitHub dataset and CodeReviewer, a transformer encoder-decoder with four code-review-specific pre-training tasks. CodeReviewer outperforms state-of-the-art source-code-pre-trained models on all three evaluated code-review tasks.

  • Conclusion: The study formulates three code-review tasks, builds GitHub pre-training and benchmark datasets, and covers nine popular programming languages.The paper describes this as the largest dataset in the code-review scenario.
  • Conclusion: CodeReviewer is a transformer-based encoder-decoder pre-trained on the collected dataset with four code-review-specific pre-training tasks.The tasks are designed for understanding code diffs and reviews.
  • Conclusion: CodeReviewer outperforms state-of-the-art models pre-trained on source code in all three evaluated tasks.The conclusion reports this cross-task outcome without specifying individual metric values.
Loading 2203.09095v2…