Source-linked AI summary
Using Pre-Trained Models to Boost Code Review Automation
Rosalia Tufano, Simone Masiero, Antonio Mastropaolo, Luca Pascarella, Denys Poshyvanyk, Gabriele Bavota
TL;DR
Code review is valuable but costly, and earlier automation models were evaluated mainly on simplified scenarios. This paper applies pre-trained T5 to raw code and a larger, more realistic dataset, reporting improved applicability and performance over the prior approach while adding comment generation.
Problem
Earlier code-review automation models addressed two tasks but were tested on relatively simple scenarios because abstraction and length restrictions simplified the problem.
Method
The paper uses a pre-trained T5 model with SentencePiece on raw source code and a substantially larger dataset, and evaluates an additional task that generates reviewer comments.
Results
T5 shows improved applicability and performance over the previous approach for code-review automation; for code-to-code at k=1, models achieve approximately 5% perfect predictions versus 2.91% previously.
Takeaways & Limitations
The work represents a step forward toward automating code-review tasks in more realistic and challenging scenarios.
Takeaways & Limitations
The findings are limited to Java, and direct comparison with the previous baseline uses only its original dataset.
Abstract
from arXiv · showhide
Code review is a practice widely adopted in open source and industrial projects. Given the non-negligible cost of such a process, researchers started investigating the possibility of automating specific code review tasks. We recently proposed Deep Learning (DL) models targeting the automation of two tasks: the first model takes as input a code submitted for review and implements in it changes likely to be recommended by a reviewer; the second takes as input the submitted code and a reviewer comment posted in natural language and automatically implements the change required by the reviewer. While the preliminary results we achieved are encouraging, both models had been tested in rather simple code review scenarios, substantially simplifying the targeted problem. This was also due to the choices we made when designing both the technique and the experiments. In this paper, we build on top of that work by demonstrating that a pre-trained Text-To-Text Transfer Transformer (T5) model can outperform previous DL models for automating code review tasks. Also, we conducted our experiments on a larger and more realistic (and challenging) dataset of code review activities.
1 INTRODUCTION
The paper targets the high cost of code review by automating contributor- and reviewer-facing tasks, while addressing limitations of earlier models through T5 and a larger, more realistic dataset. Its contributions include a third comment-generation task, comprehensive evaluation, and a reusable dataset and replication package.
- Motivation: Large projects conduct hundreds to thousands of code reviews monthly, requiring developers to spend many hours reviewing code.Examples include approximately 500 monthly reviews in Linux and 3,000 in Microsoft Bing.
- Prior automation tasks: Earlier models generated revised code either from submitted code alone or from submitted code paired with a reviewer’s natural-language change request.The first supports rapid contributor feedback; the second provides reviewers with concrete implementation examples.
- Prior results and limitations: Earlier experiments achieved 3%–16% success for reviewer-like changes and 12%–31% for implementing reviewer comments, depending on the number of predictions.These models were trained on approximately 17,000 code-review triplets.
- Prior results and limitations: Code abstraction excluded reviews introducing identifiers or literals absent from the submitted code, restricting evaluation to relatively simple transformations.The abstraction map built on submitted code could not map newly introduced elements back to raw source code.
- Prior results and limitations: Restricting submitted and revised code to at most 100 tokens further simplified the task, leaving the models representative of only a minority of review transformations.Both the abstraction and length restrictions reduced the complexity of the targeted problem.
- This paper’s approach and contributions: The paper trains pre-trained T5 on raw source code with SentencePiece, increases the length limit to 512 tokens, and expands the dataset from 17,000 to 168,000 instances.The new dataset includes challenging transformations introducing identifiers and literals, which account for 63% of its instances.
- This paper’s approach and contributions: The study adds automatic generation of natural-language reviewer comments requesting code changes, alongside a comprehensive evaluation, comparison with the previous technique, and replication package.These contributions extend automation beyond implementing requested changes.
2 T5 TO AUTOMATE CODE REVIEW
The study adapts pre-trained T5 for code-review automation by using raw source-code tokenization, longer inputs, and domain-relevant pre-training and fine-tuning datasets. It evaluates T5 on reviewer-oriented code and comment transformations using mined GitHub and Gerrit review data.
- 2.1 Text-to-Text Transfer Transformer (T5): T5 is a Transformer-based text-to-text model that is pre-trained to acquire general knowledge before supervised fine-tuning on downstream tasks.The adopted small version has approximately 60M parameters and uses six encoder and six decoder layers.
- 2.2.1 Pre-training Dataset: The pre-training dataset combines Java source code and technical English from Stack Overflow and CodeSearchNet.The resulting dataset contains 1,485,326 instances and uses a denoising objective that masks 15% of tokens.
- 2.2.2 Fine-tuning Datasets: The fine-tuning data consists of review triplets linking submitted methods, reviewer comments requesting changes, and revised methods implementing those comments.The mining process targeted accepted reviews and extracted 382,955 valid triplets from GitHub and Gerrit.
- 2.2.2 Fine-tuning Datasets: The review-mining pipeline excludes author comments, inline comments, and review rounds with multiple reviewer comments to isolate single-comment code changes.It also requires the revised method to differ from the submitted method.
- 2.3 Training and Hyperparameter Search: Among eight tuned models, ST-LR has the best overall performance and is selected for subsequent experiments.The models are assessed using perfect predictions, defined as outputs identical to the expected target string.
3 STUDY DESIGN
The evaluation studies T5 across three code-review generation tasks, examines pre-training and prediction confidence, and compares T5 with the earlier baseline. It uses perfect predictions together with BLEU or CodeBLEU, plus statistical comparisons for selected analyses.
- Research Questions: The study evaluates T5 through five research questions covering code-change recommendation, reviewer-comment implementation, comment generation, pre-training and confidence, and comparison with the baseline.The first three tasks include two previously studied tasks and one newly introduced natural-language recommendation task.
- RQ1: T5 receives a submitted Java method for automatic code-change recommendation and produces a revised method representing changes likely to be requested during review.This task is framed as an automated check available before submission.
- RQ2: T5 receives a submitted method and a natural-language reviewer comment, then generates a revised method addressing the requested changes.This task evaluates automated implementation of reviewer recommendations.
- RQ3: T5 also generates natural-language review comments from submitted Java methods, introducing a code-review task not covered by the previous work.The generated comment requests code changes as a reviewer would do.
- RQ4: The study compares pre-trained and non-pre-trained T5 models and tests whether prediction confidence can serve as a proxy for prediction quality.The confidence analysis considers whether developers can filter recommendations using a confidence threshold.
- Metrics: Performance is measured by perfect predictions for all tasks, BLEU for generated comments, and CodeBLEU for generated code.CodeBLEU incorporates n-gram, abstract-syntax-tree, and data-flow similarity, while perfect predictions require exact output matches.
- RQ5: The experiments compare T5 with the previous baseline on the two earlier code-generation tasks using perfect-prediction percentages and CodeBLEU.The comparison uses the earlier dataset after excluding 97 instances with non-English comments or invalid Unicode characters.
4 RESULTS DISCUSSION
T5 performance varies across code-review tasks, pre-training conditions, confidence levels, and beam sizes. Manual inspection shows that exact-match scores underestimate valuable predictions, while comparisons on the prior dataset favor T5 over the baseline.
- RQ1-RQ3: Performance of T5: T5’s perfect-prediction performance differs across tasks: code-to-code reaches about 5% at k=1, code&comment-to-code reaches 14.08%, and code-to-comment reaches 2.12% with pre-training.The code-to-code values are approximately 5% for both variants; the other figures refer to the pre-trained model at k=1.
- RQ1-RQ3: Performance of T5: Increasing beam size from 1 to 10 yields only marginal gains; for pre-trained code&comment-to-code, performance rises from 14.08% to 18.88%.The authors consider k=1 most relevant because presenting many entire-method recommendations may burden developers.
- Manual analysis: 62% of inspected code&comment-to-code non-perfect predictions were valid implementations, while 36 code-to-comment cases were semantically equivalent.For code-to-code, 89% of inspected non-perfect predictions were actually wrong and not aligned with the developer’s changes.
- Pre-training and confidence: Pre-training improves code&comment-to-code and code-to-comment performance but reduces code-to-code odds of a perfect prediction at k=1.The odds ratios are 1.85 and 1.59 in favor of pre-training for the two tasks involving natural language, versus 0.66 for code-to-code.
- Pre-training and confidence: The highest confidence bucket raises perfect predictions to 14.24% for code-to-code, 28.23% for code&comment-to-code, and 22.23% for code-to-comment.These values compare predictions in the 0.9-1.0 confidence bucket with overall rates of 4.48%, 14.08%, and 2.12%, respectively.
- Comparison with the baseline [46]: On the earlier, simpler test set, T5 significantly outperforms the baseline at k=1, with odds ratios ranging from 1.69 to 11.48.For code&comment-to-code, perfect predictions are 29.74% for pre-trained T5, 15.46% for non-pre-trained T5, and 9.48% for the baseline.
5 THREATS TO VALIDITY
The study identifies dataset quality, limited hyperparameter exploration, and Java-only evaluation as threats to validity. These constraints bound confidence in the findings and their generalizability.
- Construct validity: Dataset noise and incorrect comment–implementation links may remain despite cleaning, particularly in the larger dataset.Examples include non-English comments and links associating a comment with the wrong implementation change.
- Internal validity: Hyperparameter tuning varied only the learning rate, while other T5 settings were taken from the best architecture identified by Raffel et al.The authors acknowledge that additional tuning could improve performance.
- External validity: The main research questions were evaluated on a dataset one order of magnitude larger than before, but the findings remain limited to Java.The baseline comparison additionally used only the earlier dataset because the previous approach requires code abstraction.
6 RELATED WORK
Related work spans deep-learning software automation, empirical code-review research, and techniques for optimizing or partially automating review. The paper’s approach differs by generating code changes rather than only assessing or routing reviews.
- Research areas: The paper relates its contribution to deep-learning software automation and empirical code-review research while focusing its discussion on review optimization and partial automation.Systematic literature reviews are cited for the first two research areas.
- Optimizing/automating the code review process: Existing code-review platforms mostly provide basic functionality with little support for automating tasks.This finding was reported for platforms including Gerrit, Code Flow, and Phabricator and confirmed by later work.
- Optimizing/automating the code review process: Researchers have studied patch size, test-driven code review, defect prediction, and static analysis for improving review effectiveness.The cited studies examine reviewer behavior, review effectiveness, defect prediction, coding-standard violations, and common defects.
- Optimizing/automating the code review process: Search-based approaches have been proposed to simplify reviewer-assignment and code-review triaging tasks in open-source and industrial contexts.These techniques target review assignment rather than generating the reviewed code changes themselves.
- Optimizing/automating the code review process: Other machine-learning approaches assess whether a change should be accepted or whether code is well written, rather than implementing requested review changes.The paper characterizes these approaches as complementary to its code-change automation.
7 CONCLUSION AND FUTURE WORK
The paper replaces code abstraction with a pre-trained T5 model using SentencePiece and evaluates it on a larger, more realistic code-review dataset. T5 improves applicability and performance over the prior approach, but observed performance remains insufficient for practical deployment.
- Conclusion: T5 with SentencePiece works directly on raw source code, overcoming the previous approach’s exclusion of changes introducing new identifiers or literals.The model also supports longer code components than the earlier abstraction-based setup.
- Conclusion: Evaluation on a much larger and more realistic dataset shows improvements over the previous state-of-the-art approach in applicability and performance.The conclusion frames these gains as a step forward in code-review automation.
- Conclusion: The techniques remain far from deployable in practice because their actual performance is still limited.The authors call for further research in code-review automation.
- Future work: Future work targets prediction accuracy through combining code representations and using model confidence to filter for higher-quality recommendations.These are proposed directions rather than evaluated results in this paper.
- Reproducibility: The study’s code and data are publicly available.The paper points readers to its replication package.