Towards Automating Code Review Activities

• Reviewed code has lower chances of being buggy and higher internal quality, motivating widespread adoption in industrial and open source projects.Source: §I · p. 1
• Developers spend more than six hours per week reviewing code, while projects such as Microsoft Bing can undergo approximately 3,000 reviews per month.Source: §I · p. 1
• Code review also imposes context-switching costs beyond the time allocated to reviewing.Source: §I · p. 1
• The long-term goal is to partially automate review tasks while preserving developers’ roles and knowledge sharing.Source: §I · p. 1

Contributor scenario

• The contributor-side transformer learns reviewer-recommended changes from 17,194 submitted-to-revised code pairs and can provide preliminary feedback before review submission.Source: §I · p. 1

Reviewer scenario

• The reviewer-side model uses code and a natural-language comment to generate revised code implementing that recommendation, using two encoders and one decoder.Source: §I · p. 1

• The two scenarios support automation before review submission or after a reviewer has provided a specific comment.Source: §I · p. 2

Evaluation

• 3% to 16% of contributor-side cases and 12% to 31% of reviewer-side cases were correctly implemented, depending on the number of candidates generated.Source: §I · p. 2

A. Mining Code Review Data

• The approach begins by mining code reviews from Java projects hosted on GitHub or using Gerrit.Source: §II · p. 2
• Submitted code is parsed into methods, and reviewer comments are collected by linking them to specific source-code lines.Source: §II · p. 2
• Methods before and after review are abstracted, and only methods receiving a single comment in a review round are retained to associate one revision with one recommendation.Source: §II · p. 2

B. Model Construction

• The resulting Reviewed Commented Code Triplets dataset has the form ⟨ms, rnl⟩→mr and trains a two-encoder, one-decoder transformer.Source: §II · p. 2
• Removing comments from the triplets produces code pairs ms →mr for training a one-encoder, one-decoder model that recommends revisions without reviewer input.Source: §II · p. 2

A. Mining Code Review Data

• The mining process uses two dedicated tools because Gerrit and GitHub expose different APIs for retrieving review data.Source: A. Mining Code Review Data · p. 2
• The GitHub mining effort covered 2,566 Java repositories with at least 50 pull requests.Source: A. Mining Code Review Data · p. 2

• Preprocessing builds triplet ⟨ms, rnl⟩→mr and pair ms →mr datasets from collected review data.Source: B. Data Preprocessing · p. 3
• Java files are parsed with Lizard to extract methods, establishing method-level granularity for the deep-learning models.Source: B. Data Preprocessing · p. 3
• Code abstraction limits vocabulary while retaining expressiveness, and samples with abstraction parsing errors are removed.Source: B. Data Preprocessing · p. 3
• Identifiers and literals receive position-based abstraction IDs, while frequent idioms such as i, j, 0, and 1 remain unabstracted.Source: B. Data Preprocessing · p. 3

2) Linking and Abstracting Reviewer Comments:

• Reviewer comments are associated with the specific code lines they reference in both Gerrit and GitHub.Source: 2) Linking and Abstracting Reviewer… · p. 3
• A comment is linked to a method when its referenced start and end lines fall within the method body, signature, or annotations; otherwise it is discarded.Source: 2) Linking and Abstracting Reviewer… · p. 4
• Commented code components are mapped to the same abstraction identifiers as the code, while unmatched camel-case identifiers become CODE tokens.Source: 2) Linking and Abstracting Reviewer… · p. 4

• Some collected comments are linked to code lines but are unlikely to result in code changes, making them irrelevant for the study.Source: 3) Filtering Out Noisy Comments · p. 4
• A manually labeled sample contained 1,875 comments drawn from 500 Gerrit reviews and 500 GitHub reviews.Source: 3) Filtering Out Noisy Comments · p. 4
• 21% of the analyzed comments produced labeling conflicts, including both obvious acknowledgments and harder-to-classify statements.Source: 3) Filtering Out Noisy Comments · p. 4
• The final labeled dataset contained 1,676 relevant and 199 irrelevant comments, and models used unigram, bigram, and trigram features.Source: 3) Filtering Out Noisy Comments · p. 4
• Random Forest, J48, and Bayesian-network classifiers were evaluated with 10-fold cross-validation and SMOTE rebalancing.Source: 3) Filtering Out Noisy Comments · p. 4
• 91.6% precision was achieved by the Random Forest classifier with SMOTE when classifying comments as relevant.Source: 3) Filtering Out Noisy Comments · p. 4
• The reported baseline was 89% precision for a constant classifier that always labeled comments as relevant.Source: 3) Filtering Out Noisy Comments · p. 5

Filtering

• Author comments, code comments, unchanged code pairs, multi-comment cases, and overlong sequences are excluded during dataset preparation.Sequences longer than 100 abstracted tokens account for the removal of 148,539 triplets.Source: 1) Dataset Preparation · p. 5

Assumptions

• Generated revised code cannot introduce identifiers or literals absent from the original code because abstraction cannot recover their concrete values.This constraint limits the model’s usable output vocabulary while supporting generation of concrete source code.Source: 1) Dataset Preparation · p. 5

Filtering

• The dataset requires reviewer comments to be singletons so the triggering comment can be associated with the observed code transformation.This restriction supports attribution of a transformation to one reviewer recommendation.Source: 1) Dataset Preparation · p. 5

Comment preprocessing

• Reviewer comments are cleaned by removing stopwords, links, superfluous punctuation, and lowercasing non-code words while preserving code identifiers.The preprocessing converts natural-language comments into a normalized representation for model input.Source: 1) Dataset Preparation · p. 5

Dataset construction

• 17,194 remaining triplets form the Dt dataset for the 2-encoder model, while removing comments produces Dp for the 1-encoder model.Both datasets use 80% training, 10% evaluation, and 10% test splits.Source: 1) Dataset Preparation · p. 5

Scenario 1: Recommending Changes (1-encoder)

• The 1-encoder transformer maps submitted code to one or more suggested reviewer changes before a review comment is available.Its encoder reads the code sequence and its decoder generates revised-code suggestions.Source: 2) Scenario 1: Recommending Changes… · p. 5

Model configuration

• Table I lists the models’ hyperparameters and their best configuration.Source: Table I · p. 6

Scenario 2: Implementing Recommendations (2-encoder)

• The 2-encoder transformer takes submitted code and a natural-language recommendation as separate inputs, then generates revised-code suggestions implementing that recommendation.This scenario supports previewing the recommended change for reviewers and clarifying the recommendation for contributors.Source: 3) Scenario 2: Implementing Changes… · p. 6

Model configuration

• Bayesian optimization with a Tree Parzen Estimator searches configurations over 10 hyperparameters for both models.The search is used to select the best configuration of each architecture.Source: 4) Hyperparameter Search · p. 6

Decoding

• Beam search generates multiple hypotheses by extending sequences token by token according to conditional likelihood.Multiple outputs are appropriate because model generations are intended as suggestions.Source: 5) Generating Multiple Solutions via… · p. 6

Research questions

• RQ1 tests whether NMT can recommend code changes for submitted contributions as reviewers would, using the Dp dataset.The evaluation runs the tuned 1-encoder model on unseen test samples with beam-search inference.Source: §III · p. 6, A. Data Collection and Analysis · p. 6
• RQ2 tests whether NMT can implement a reviewer’s natural-language recommendation in submitted code, using the Dt dataset.This evaluates the 2-encoder reviewer-perspective model.Source: §III · p. 6

Evaluation measures

• A prediction is perfect when generated code is identical to the code manually written after review, and results report perfect-prediction counts and percentages by beam size.The analysis checks whether at least one perfect prediction appears among the k generated solutions.Source: A. Data Collection and Analysis · p. 6
• BLEU-4 evaluates generated predictions using 4-gram overlap, with 100% indicating identity between prediction and reference.Source: A. Data Collection and Analysis · p. 6
• Token-level Levenshtein distance measures the minimum insertions, deletions, or substitutions needed to convert predictions into reference code.The study normalizes this distance by the token count of the longer predicted or reference sequence.Source: A. Data Collection and Analysis · pp. 6–7

Qualitative analysis

• Qualitative analyses classify perfect predictions by code-change type and examine whether non-perfect predictions could still benefit developers.For RQ2, the analysis focuses on 300 cases perfect for the 2-encoder but not the 1-encoder model.Source: A. Data Collection and Analysis · p. 7

Quantitative comparison

• With k = 10, the 1-encoder produced 271 perfect predictions (15.76%), versus 528 (30.72%) for the 2-encoder model.The 2-encoder model used the reviewer’s natural-language comment as additional input.Source: §IV · p. 7
• At k = 1, the 2-encoder achieved 209 perfect predictions (12.16%), compared with 50 (2.91%) for the 1-encoder, a 4× improvement.The performance gap remained across beam sizes but became less pronounced as k increased.Source: §IV · p. 7
• The 2-encoder generated code closer to the reference, with one of three predictions requiring approximately 13% token changes on average and a median of 9%.BLEU-4 and normalized Levenshtein distance confirmed the same performance trend.Source: §IV · p. 7

Qualitative analysis

• The 1-encoder learned varied transformations, mostly simple code changes, that sometimes addressed functional or non-functional quality issues difficult to spot.Examples included visibility and variable-type changes, readability improvements, and possible bug fixes involving conditions or method-call parameters.Source: §IV · pp. 7–8
• Among 300 2-encoder predictions missed by the 1-encoder, only 32 belonged to entirely new change categories, despite 528 versus 271 perfect predictions at k = 10.The additional reviewer comments substantially improved performance but enabled relatively few new types of learned changes.Source: §IV · p. 8
• The comment-filtering heuristic achieved approximately 93% precision, with 22 of 300 inspected comments judged irrelevant to the performed code changes.These comments were false positives that should have been discarded.Source: §IV · p. 9
• The 2-encoder implemented complex reviewer recommendations, including try-with-resources, while extract-method cases were limited to identifying statements rather than their destination.It also learned simple changes such as adding final to a variable modifier once the reviewer comment was supplied.Source: §IV · p. 9
• Manual inspection found meaningful, semantically equivalent non-perfect predictions in 5% of sampled 1-encoder cases and 6% of sampled 2-encoder cases.The authors estimated approximately 5% additional performance beyond perfect-prediction results.Source: §IV · p. 9

Construct validity

• Noisy comments that were unlikely to trigger code changes remained in the training and testing data despite preprocessing heuristics.The authors describe this residual noise as a limitation of the proposed approach.Source: §V · p. 9
• BLEU-4 and Levenshtein distance may be influenced by source-code syntactic sugar and the similarity between input and output code.The authors therefore also considered perfect predictions.Source: §V · p. 9

Internal validity

• Manual classification and qualitative analysis could contain imprecisions despite independent classification by two authors and conflict resolution by a third.Source: §V · p. 9

External validity

• The findings are limited in generalizability because the datasets cover Java systems only.The authors state that the approach can be adapted to other languages by replacing the abstraction component.Source: §V · p. 9

Deep Learning for Automating Code Changes

• Deep learning has been applied to source-code changes such as bug fixing, software migration, code completion, and assert-statement generation.Source: §VI · p. 9, A. Deep Learning for Automating Code… · pp. 9–10
• Existing bug-fixing systems include DeepFix, SequenceR, and neural machine translation models that generate patches for defective code.Source: A. Deep Learning for Automating Code… · p. 10
• Tufano et al. learned generic Java method changes from pull requests using an encoder-decoder RNN model.Source: A. Deep Learning for Automating Code… · p. 10

Code Review Automation

• The paper claims to be the first attempt to automate code review activities from both contributor and reviewer perspectives.Source: A. Deep Learning for Automating Code… · p. 10
• Compared with Tufano et al., the study targets a different problem requiring a new dataset and a two-encoder architecture processing reviewer comments and submitted code.Source: A. Deep Learning for Automating Code… · p. 10

Code Review

• Reviewed code has a lower likelihood of introducing bugs and higher code quality than non-reviewed code.Source: B. Code Review · p. 10
• Studies identify defect detection as the main reason for performing code review.Source: B. Code Review · p. 10
• ClusterChanges decomposes submitted changesets into cohesive smaller changes, whereas this paper automates specific code review steps.Source: B. Code Review · p. 10

Conclusions

• The study automates contributor-side recommendations and reviewer-side code implementation of natural-language comments.Source: §VII · p. 10
• Two different transformer-based architectures are defined for these activities.Source: §VII · p. 11
• Up to 16% of contributor-side cases and 31% of reviewer-side cases receive meaningful generated recommendations.Source: §VII · p. 11

Future work

• The authors plan to explore different deep-learning architectures and increase training data to learn a larger variety of code changes.Source: §VII · p. 11

Conclusions

• The study releases its code and datasets.Source: §VIII · p. 11