PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Automating Code Review Activities

Rosalia Tufano, Luca Pascarella, Michele Tufano, Denys Poshyvanyk, Gabriele Bavota

arXiv:2101.02518v4cs.SE

TL;DR

Code reviews improve code quality but require substantial developer time, motivating partial automation. The paper trains two transformer-based systems for contributor-side recommendations and reviewer-comment implementation, achieving up to 16% and 31% correct cases respectively, while noting data noise and analysis subjectivity as limitations.

  • Problem

    Code reviews provide quality benefits but add substantial developer time and context-switching costs because they are manual.

  • Method

    The paper mines and links reviewed methods with reviewer comments, then trains one-encoder and two-encoder transformer models to generate revised code.

  • Results

    At up to 10 generated candidates, the one-encoder model produced 15.76% perfect predictions and the two-encoder model produced 30.72%.

  • Takeaways & Limitations

    The results constitute a first step toward partially automating contributor-side recommendations and reviewer-side implementation of natural-language comments.

  • Takeaways & Limitations

    The training data still contained a small percentage of noisy comments caused by failures in automated filtering heuristics.

Abstract

from arXiv · show

Code reviews are popular in both industrial and open source projects. The benefits of code reviews are widely recognized and include better code quality and lower likelihood of introducing bugs. However, since code review is a manual activity it comes at the cost of spending developers' time on reviewing their teammates' code. Our goal is to make the first step towards partially automating the code review process, thus, possibly reducing the manual costs associated with it. We focus on both the contributor and the reviewer sides of the process, by training two different Deep Learning architectures. The first one learns code changes performed by developers during real code review activities, thus providing the contributor with a revised version of her code implementing code transformations usually recommended during code review before the code is even submitted for review. The second one automatically provides the reviewer commenting on a submitted code with the revised code implementing her comments expressed in natural language. The empirical evaluation of the two models shows that, on the contributor side, the trained model succeeds in replicating the code transformations applied during code reviews in up to 16% of cases. On the reviewer side, the model can correctly implement a comment provided in natural language in up to 31% of cases. While these results are encouraging, more research is needed to make these models usable by developers.

I. INTRODUCTION

Code reviews improve code quality and help detect defects, but they consume substantial developer time. This paper takes an initial step toward partial automation by training separate deep-learning models for contributor-side recommendations and reviewer-comment implementation.

  • Reviewed code has lower chances of being buggy and higher internal quality, motivating widespread adoption in industrial and open source projects.
  • Developers spend more than six hours per week reviewing code, while projects such as Microsoft Bing can undergo approximately 3,000 reviews per month.
  • Code review also imposes context-switching costs beyond the time allocated to reviewing.
  • The long-term goal is to partially automate review tasks while preserving developers’ roles and knowledge sharing.
  • Contributor scenario: The contributor-side transformer learns reviewer-recommended changes from 17,194 submitted-to-revised code pairs and can provide preliminary feedback before review submission.
  • Reviewer scenario: The reviewer-side model uses code and a natural-language comment to generate revised code implementing that recommendation, using two encoders and one decoder.
  • The two scenarios support automation before review submission or after a reviewer has provided a specific comment.
  • Evaluation: 3% to 16% of contributor-side cases and 12% to 31% of reviewer-side cases were correctly implemented, depending on the number of candidates generated.

II. USING TRANSFORMERS TO AUTOMATE CODE REVIEW

The approach mines linked code-review data from GitHub and Gerrit, constructs paired and triplet training examples, and trains transformer models to generate revised Java methods. It uses one encoder without comments and two encoders when reviewer comments are available.

  • A. Mining Code Review Data: The approach begins by mining code reviews from Java projects hosted on GitHub or using Gerrit.
  • A. Mining Code Review Data: Submitted code is parsed into methods, and reviewer comments are collected by linking them to specific source-code lines.
  • A. Mining Code Review Data: Methods before and after review are abstracted, and only methods receiving a single comment in a review round are retained to associate one revision with one recommendation.
  • B. Model Construction: The resulting Reviewed Commented Code Triplets dataset has the form ⟨ms, rnl⟩→mr and trains a two-encoder, one-decoder transformer.
  • B. Model Construction: Removing comments from the triplets produces code pairs ms →mr for training a one-encoder, one-decoder model that recommends revisions without reviewer input.
  • A. Mining Code Review Data: The mining process uses two dedicated tools because Gerrit and GitHub expose different APIs for retrieving review data.
  • A. Mining Code Review Data: The GitHub mining effort covered 2,566 Java repositories with at least 50 pull requests.

B. Data Preprocessing

Preprocessing extracts Java methods, links comments to the methods they reference, and abstracts code and comment identifiers into a consistent vocabulary. The resulting representations support triplet and pair datasets for model training.

  • Preprocessing builds triplet ⟨ms, rnl⟩→mr and pair ms →mr datasets from collected review data.
  • Java files are parsed with Lizard to extract methods, establishing method-level granularity for the deep-learning models.
  • Code abstraction limits vocabulary while retaining expressiveness, and samples with abstraction parsing errors are removed.
  • Identifiers and literals receive position-based abstraction IDs, while frequent idioms such as i, j, 0, and 1 remain unabstracted.
  • 2) Linking and Abstracting Reviewer Comments:: Reviewer comments are associated with the specific code lines they reference in both Gerrit and GitHub.
  • 2) Linking and Abstracting Reviewer Comments:: A comment is linked to a method when its referenced start and end lines fall within the method body, signature, or annotations; otherwise it is discarded.
  • 2) Linking and Abstracting Reviewer Comments:: Commented code components are mapped to the same abstraction identifiers as the code, while unmatched camel-case identifiers become CODE tokens.

3) Filtering Out Noisy Comments:

The authors filter reviewer comments that are unlikely to trigger code changes before constructing training data. Manual labels and machine-learning experiments assess how precisely relevant comments can be identified.

  • Some collected comments are linked to code lines but are unlikely to result in code changes, making them irrelevant for the study.
  • A manually labeled sample contained 1,875 comments drawn from 500 Gerrit reviews and 500 GitHub reviews.
  • 21% of the analyzed comments produced labeling conflicts, including both obvious acknowledgments and harder-to-classify statements.
  • The final labeled dataset contained 1,676 relevant and 199 irrelevant comments, and models used unigram, bigram, and trigram features.
  • Random Forest, J48, and Bayesian-network classifiers were evaluated with 10-fold cross-validation and SMOTE rebalancing.
  • 91.6% precision was achieved by the Random Forest classifier with SMOTE when classifying comments as relevant.
  • The reported baseline was 89% precision for a constant classifier that always labeled comments as relevant.

C. Automating Code Review

The study constructs filtered datasets pairing reviewed code changes with reviewer comments, producing separate inputs for models with and without comments. The filtering enforces learnable, attributable transformations and yields 17,194 triplets split into training, evaluation, and test sets.

  • Filtering: Author comments, code comments, unchanged code pairs, multi-comment cases, and overlong sequences are excluded during dataset preparation.Sequences longer than 100 abstracted tokens account for the removal of 148,539 triplets.
  • Assumptions: Generated revised code cannot introduce identifiers or literals absent from the original code because abstraction cannot recover their concrete values.This constraint limits the model’s usable output vocabulary while supporting generation of concrete source code.
  • Filtering: The dataset requires reviewer comments to be singletons so the triggering comment can be associated with the observed code transformation.This restriction supports attribution of a transformation to one reviewer recommendation.
  • Comment preprocessing: Reviewer comments are cleaned by removing stopwords, links, superfluous punctuation, and lowercasing non-code words while preserving code identifiers.The preprocessing converts natural-language comments into a normalized representation for model input.
  • Dataset construction: 17,194 remaining triplets form the Dt dataset for the 2-encoder model, while removing comments produces Dp for the 1-encoder model.Both datasets use 80% training, 10% evaluation, and 10% test splits.

2) Scenario 1: Recommending Changes (1-encoder):

The paper uses transformer encoder-decoder models to generate reviewer-like code changes, with one encoder for prediction before review and two encoders when a natural-language recommendation is available. Bayesian optimization selects configurations, and beam search generates multiple candidate revisions.

  • Scenario 1: Recommending Changes (1-encoder): The 1-encoder transformer maps submitted code to one or more suggested reviewer changes before a review comment is available.Its encoder reads the code sequence and its decoder generates revised-code suggestions.
  • Model configuration: Table I lists the models’ hyperparameters and their best configuration.
  • Scenario 2: Implementing Recommendations (2-encoder): The 2-encoder transformer takes submitted code and a natural-language recommendation as separate inputs, then generates revised-code suggestions implementing that recommendation.This scenario supports previewing the recommended change for reviewers and clarifying the recommendation for contributors.
  • Model configuration: Bayesian optimization with a Tree Parzen Estimator searches configurations over 10 hyperparameters for both models.The search is used to select the best configuration of each architecture.
  • Decoding: Beam search generates multiple hypotheses by extending sequences token by token according to conditional likelihood.Multiple outputs are appropriate because model generations are intended as suggestions.

III. STUDY DESIGN

The study evaluates neural machine translation as a partial approach to automating code-review activities from contributor and reviewer perspectives. It tests whether models can recommend reviewer-like changes or implement natural-language recommendations, using exact-match, BLEU-4, edit-distance, and qualitative analyses.

  • Research questions: RQ1 tests whether NMT can recommend code changes for submitted contributions as reviewers would, using the Dp dataset.The evaluation runs the tuned 1-encoder model on unseen test samples with beam-search inference.
  • Research questions: RQ2 tests whether NMT can implement a reviewer’s natural-language recommendation in submitted code, using the Dt dataset.This evaluates the 2-encoder reviewer-perspective model.
  • Evaluation measures: A prediction is perfect when generated code is identical to the code manually written after review, and results report perfect-prediction counts and percentages by beam size.The analysis checks whether at least one perfect prediction appears among the k generated solutions.
  • Evaluation measures: BLEU-4 evaluates generated predictions using 4-gram overlap, with 100% indicating identity between prediction and reference.
  • Evaluation measures: Token-level Levenshtein distance measures the minimum insertions, deletions, or substitutions needed to convert predictions into reference code.The study normalizes this distance by the token count of the longer predicted or reference sequence.
  • Qualitative analysis: Qualitative analyses classify perfect predictions by code-change type and examine whether non-perfect predictions could still benefit developers.For RQ2, the analysis focuses on 300 cases perfect for the 2-encoder but not the 1-encoder model.

IV. RESULTS DISCUSSION

The 2-encoder model substantially outperformed the 1-encoder model, especially when reviewer comments were provided, while both models learned diverse code transformations. Some non-perfect predictions were still meaningful, and the additional input enabled only a limited number of entirely new change types.

  • Quantitative comparison: With k = 10, the 1-encoder produced 271 perfect predictions (15.76%), versus 528 (30.72%) for the 2-encoder model.The 2-encoder model used the reviewer’s natural-language comment as additional input.
  • Quantitative comparison: At k = 1, the 2-encoder achieved 209 perfect predictions (12.16%), compared with 50 (2.91%) for the 1-encoder, a 4× improvement.The performance gap remained across beam sizes but became less pronounced as k increased.
  • Quantitative comparison: The 2-encoder generated code closer to the reference, with one of three predictions requiring approximately 13% token changes on average and a median of 9%.BLEU-4 and normalized Levenshtein distance confirmed the same performance trend.
  • Qualitative analysis: The 1-encoder learned varied transformations, mostly simple code changes, that sometimes addressed functional or non-functional quality issues difficult to spot.Examples included visibility and variable-type changes, readability improvements, and possible bug fixes involving conditions or method-call parameters.
  • Qualitative analysis: Among 300 2-encoder predictions missed by the 1-encoder, only 32 belonged to entirely new change categories, despite 528 versus 271 perfect predictions at k = 10.The additional reviewer comments substantially improved performance but enabled relatively few new types of learned changes.
  • Qualitative analysis: The comment-filtering heuristic achieved approximately 93% precision, with 22 of 300 inspected comments judged irrelevant to the performed code changes.These comments were false positives that should have been discarded.
  • Qualitative analysis: The 2-encoder implemented complex reviewer recommendations, including try-with-resources, while extract-method cases were limited to identifying statements rather than their destination.It also learned simple changes such as adding final to a variable modifier once the reviewer comment was supplied.
  • Qualitative analysis: Manual inspection found meaningful, semantically equivalent non-perfect predictions in 5% of sampled 1-encoder cases and 6% of sampled 2-encoder cases.The authors estimated approximately 5% additional performance beyond perfect-prediction results.

V. THREATS TO VALIDITY

The study identifies threats to construct, internal, and external validity. These include noisy comments, subjectivity in manual analyses, and limited generalizability beyond Java systems.

  • Construct validity: Noisy comments that were unlikely to trigger code changes remained in the training and testing data despite preprocessing heuristics.The authors describe this residual noise as a limitation of the proposed approach.
  • Construct validity: BLEU-4 and Levenshtein distance may be influenced by source-code syntactic sugar and the similarity between input and output code.The authors therefore also considered perfect predictions.
  • Internal validity: Manual classification and qualitative analysis could contain imprecisions despite independent classification by two authors and conflict resolution by a third.
  • External validity: The findings are limited in generalizability because the datasets cover Java systems only.The authors state that the approach can be adapted to other languages by replacing the abstraction component.

VI. RELATED WORK

Related work applies deep learning to automated code changes, including bug fixing, software migration, code completion, and test-oracle generation. This paper distinguishes itself by targeting code review automation from both contributor and reviewer perspectives with a new dataset and two-encoder architecture.

  • Deep Learning for Automating Code Changes: Deep learning has been applied to source-code changes such as bug fixing, software migration, code completion, and assert-statement generation.
  • Deep Learning for Automating Code Changes: Existing bug-fixing systems include DeepFix, SequenceR, and neural machine translation models that generate patches for defective code.
  • Deep Learning for Automating Code Changes: Tufano et al. learned generic Java method changes from pull requests using an encoder-decoder RNN model.
  • Code Review Automation: The paper claims to be the first attempt to automate code review activities from both contributor and reviewer perspectives.
  • Code Review Automation: Compared with Tufano et al., the study targets a different problem requiring a new dataset and a two-encoder architecture processing reviewer comments and submitted code.

B. Code Review

Code reviews are associated with improved code quality and a lower likelihood of bugs, and identifying defects is a principal reason for conducting them. However, reviews consume developer time and can require context switching, motivating automation that supports specific review steps.

  • Code Review: Reviewed code has a lower likelihood of introducing bugs and higher code quality than non-reviewed code.
  • Code Review: Studies identify defect detection as the main reason for performing code review.
  • Code Review: ClusterChanges decomposes submitted changesets into cohesive smaller changes, whereas this paper automates specific code review steps.

VII. CONCLUSIONS

The paper explores transformer-based automation of two code review activities: recommending contributor changes and generating code that implements reviewer comments. The models produce meaningful recommendations in up to 16% and 31% of cases, but substantial improvement is still needed for developer use.

  • Conclusions: The study automates contributor-side recommendations and reviewer-side code implementation of natural-language comments.
  • Conclusions: Two different transformer-based architectures are defined for these activities.
  • Conclusions: Up to 16% of contributor-side cases and 31% of reviewer-side cases receive meaningful generated recommendations.
  • Future work: The authors plan to explore different deep-learning architectures and increase training data to learn a larger variety of code changes.
  • Conclusions: The study releases its code and datasets.
Loading 2101.02518v4…