PaperlayerView exact revision on arXiv

Source-linked AI summary

Predicting Usefulness of Code Review Comments using Textual Features and Developer Experience

Mohammad Masudur Rahman, Chanchal K. Roy, Raula G. Kula

arXiv:1807.04485v1cs.SE

TL;DR

Existing tools did not predict usefulness during review submission, despite a substantial share of comments being non-useful. The authors study comment text and reviewer experience, then develop RevHelper. They report 66% overall prediction accuracy and validation against baseline variants.

  • Problem

    No tools predicted comment usefulness during review submission, although prior research found that 34.50% of comments were non-useful.

    Source: §I · p. 1

  • Method

    The authors compare useful and non-useful comments using textual characteristics and reviewer experience, then build RevHelper to predict usefulness during submission.

    Source: §I · pp. 1–2

  • Results

    66% overall prediction accuracy was reported on 1,482 review comments, with comparison against three baseline variants in a case study and validation set.

    Source: §VII · p. 10

  • Takeaways & Limitations

    The authors report that automatic, instant usefulness prediction using comment text and developer experience demonstrates potential for code-review support.

    Source: §VII · p. 10

    Usefulness labels relied on manual application of the change-triggering heuristic, so human error could not be avoided despite training and cross-checks.

    Source: §V · p. 10

Abstract

from arXiv · show

Although peer code review is widely adopted in both commercial and open source development, existing studies suggest that such code reviews often contain a significant amount of non-useful review comments. Unfortunately, to date, no tools or techniques exist that can provide automatic support in improving those non-useful comments. In this paper, we first report a comparative study between useful and non-useful review comments where we contrast between them using their textual characteristics, and reviewers' experience. Then, based on the findings from the study, we develop RevHelper, a prediction model that can help the developers improve their code review comments through automatic prediction of their usefulness during review submission. Comparative study using 1,116 review comments suggested that useful comments share more vocabulary with the changed code, contain salient items like relevant code elements, and their reviewers are generally more experienced. Experiments using 1,482 review comments report that our model can predict comment usefulness with 66\% prediction accuracy which is promising. Comparison with three variants of a baseline model using a case study validates our empirical findings and demonstrates the potential of our model.

I. INTRODUCTION

The paper addresses the lack of submission-time support for identifying non-useful review comments by studying comment text and reviewer experience, then building RevHelper.

I. INTRODUCTION

  • Earlier models used constrained features unavailable during submission, and no existing tools predicted usefulness for new comments at submission time.The authors identify this as a gap in applying earlier qualitative findings to practical automated support.Source: §I · p. 1
  • RevHelper predicts comment usefulness during review submission using findings from a comparative study of textual characteristics and reviewer expertise.The study contrasts five textual characteristics and three reviewer-experience features.Source: §I · pp. 1–2
  • The authors describe their work as an extensive two-dimension, eight-feature comparison paired with automatic usefulness prediction, and report that the datasets are hosted for replication.The earlier studies they discuss primarily analyzed comments qualitatively.Source: §I · p. 2
  • 66% overall accuracy, 66% precision, and 66% recall were reported for RevHelper on 1,116 comments; a 366-comment case study compared it with three baseline variants.The authors say the case study validates their empirical findings and demonstrates the model's potential.Source: §I · p. 2
  • 44.47% of 1,116 comments were non-useful; useful comments shared more vocabulary with changed code, contained more relevant code elements, used fewer stop words, and had more experienced reviewers.Non-usefulness was defined as not triggering code changes.Source: §I · p. 2

II. COMPARATIVE STUDY BETWEEN USEFUL AND NON-USEFUL REVIEW COMMENTS

The comparative study investigates which textual features distinguish useful from non-useful comments and how developer experience relates to usefulness, using comments from four commercial systems.

II. COMPARATIVE STUDY BETWEEN USEFUL AND NON-USEFUL REVIEW COMMENTS

  • The authors examine 1,116 comments from four commercial ABC Company systems to investigate features associated with useful and non-useful comments.The study seeks to identify what non-useful comments lack and which features distinguish useful ones.Source: §II · pp. 2–3
  • Table I is presented as an overview of the variables and dimensions used in the study.Source: Table I · p. 3
  • The study asks whether useful and non-useful comments differ in textual properties and how developer experience relates to writing useful comments.These are the two research questions that organize the study.Source: §II · p. 3

A. Dataset for Study

The study compares useful and non-useful in-line review comments from four commercial systems, using a change-triggering heuristic for labels. It examines textual properties and reviewer experience through eight independent variables.

A. Dataset for Study

  • 1,116 in-line comments from four commercial systems remained after scaffolding pull requests and their comments were excluded.The researchers collected 300 recent comments per system before exclusions.Source: A. Dataset for Study · p. 3
  • Comments were manually labeled useful when they triggered code changes within 1–10 lines, and non-useful when changes were unrelated or absent.The annotation took about 20 person-hours; 55.53% were useful and 44.47% non-useful.Source: A. Dataset for Study · p. 3

B. Determining Independent Variables and Dimensions

  • The comparative study tests eight variables: five textual properties and three proxies for reviewer experience.Readability and reviewer experience motivate examining clarity and expertise alongside comment text.Source: B. Determining Independent Variables… · p. 3

C. Textual Features

Useful comments differ from non-useful ones in several textual properties, particularly their relation to changed code, while reading ease and question ratio do not significantly distinguish them.

C. Textual Features

  • Reading ease and question ratio did not significantly distinguish useful from non-useful comments, including across reading-ease quartiles.Removing code elements raised reading ease in both groups, but their difference remained nonsignificant; question-ratio p-value=0.08.Source: C. Textual Features · p. 4, Figure 5 · p. 4
  • Non-useful comments had higher stop-word ratios, including when programming keywords were counted; the difference was significant but small (Cohen’s D=0.14).For stop-word ratios, Q1 was not significant, while Q2–Q4 were significant.Source: C. Textual Features · p. 4, Figure 4 · p. 4
  • About 51% of comments contained source-code elements, but their ratios did not differ significantly by usefulness (paired t-test, p-value=0.27).Useful-comment rates with at least one code element were 45%–61%, compared with up to 36%–53% for non-useful comments.Source: C. Textual Features · p. 5, Table III · p. 5
  • Useful comments had significantly higher source-token ratios across all four quartiles; the overall difference had Cohen’s D=0.17.Quartile effect sizes ranged from 0.42 to 0.92.Source: C. Textual Features · p. 5, Figure 6 · p. 5
  • Useful comments had a median relevance of 0.24 to changed code, versus 0.17 for non-useful comments; the difference was significant (p-value=0.00, Cohen’s D=0.42).Quartile analyses also found significantly higher relevance for useful comments across all four quartiles.Source: C. Textual Features · p. 5, Figure 7 · p. 5
  • Useful comments differ significantly from non-useful comments in code-element ratio, stop-word ratio, and conceptual similarity, but not in reading ease or question ratio.These are the study’s summary findings for textual properties.Source: C. Textual Features · p. 6

D. Developer Experience

Prior file authorship and review activity can significantly influence comment usefulness, but their relationship with usefulness is not consistently linear. Reviewers’ external-library experience shows mixed evidence, with some quartile differences but no consistent overall separation.

D. Developer Experience

  • Reviewers’ file-level and whole-system commit distributions differed by usefulness, although file-change prevalence was 46.12% for useful comments and 49.00% for non-useful comments.The reported tests found significant differences in commit-count distributions for both file-level and whole-system authorship experience.Source: D. Developer Experience · p. 6
  • Useful comments were more often associated with prior reviewing of the target file and repeated reviewing, but quartile differences appeared only at lower-to-medium experience levels.Reviewing the same file occurred for 87.22% of useful comments versus 84.54% of non-useful comments; reviewers had reviewed more than once for 86.25% versus 79.09%, respectively.Source: D. Developer Experience · pp. 6–7
  • Total reviewed commits and pull requests were not significantly different between comment types, although both review-activity measures significantly influenced usefulness in Kruskal-Wallis tests.The respective between-type p-values were 0.63 for commits and 0.33 for pull requests; both influence tests reported p-value=0.00<0.05.Source: D. Developer Experience · p. 7
  • External-library experience showed mixed evidence: useful comments had higher minimum experience, Q3 differed significantly, and Q4 did not.The Q3 comparison reported p-value=0.03<0.05, while Q4 reported p-value=0.62>0.05; the overall difference was not statistically significant.Source: Figure 11 · p. 7, D. Developer Experience · p. 7
  • Prior file authorship and reviewing activity can significantly influence comment usefulness, but experience and usefulness do not always have a linear relationship.This summary supports earlier findings.Source: D. Developer Experience · p. 7

III. REVHELPER: AUTOMATIC PREDICTION MODEL FOR REVIEW COMMENT USEFULNESS

The paper motivates automated usefulness prediction with the prevalence of non-useful comments and findings that useful comments differ in textual properties and reviewer experience. It develops RevHelper to predict usefulness from text and experience features.

III. REVHELPER: AUTOMATIC PREDICTION MODEL FOR REVIEW COMMENT USEFULNESS

  • A prior study found that 34.5% of code review comments were non-useful, motivating automatic usefulness prediction before review submission.The cited study is.Source: §III · p. 7
  • The comparative study found that useful and non-useful comments differ significantly in textual properties and that reviewers of useful comments have more experience.Source: §III · p. 7
  • RevHelper applies machine learning to textual and developer-experience features to predict whether a review comment is useful or non-useful.Source: §III · p. 7

A. Feature Calculation

The model uses comment-text and developer-experience features to predict usefulness, with experience derived from system histories and usefulness labels assigned manually.

A. Feature Calculation

  • The feature set contains 15 independent variables from comment text and developer experience, with usefulness labeled as useful or non-useful.Some features yield multiple variables; the response variable is the manually annotated comment label.Source: A. Feature Calculation · p. 7
  • Textual features come from comments, while reviewer experience is calculated from version-control and code-review histories.Source: A. Feature Calculation · p. 7
  • Fewer than 4% of samples had missing feature values, which were discarded or replaced with the corresponding feature mean.Source: A. Feature Calculation · p. 8

B. Development of the Usefulness Prediction Model

The model-development section compares three classifiers with different learning strategies. Experiments use commercial review data, cross-validation, a separate case study, and three baseline models.

B. Development of the Usefulness Prediction Model

  • Random Forest, Logistic Regression, and Naive Bayes were selected to represent different learning strategies for potentially complex feature-usefulness relationships.Random Forest assumes non-linear relationships, Logistic Regression the opposite, and Naive Bayes predictor independence.Source: B. Development of the Usefulness… · p. 8
  • Naive Bayes and Logistic Regression use 10-fold cross-validation, while Random Forest averages 10 runs trained on 65% of randomly sampled data with 2,000 trees.Random Forest runs use different seed points.Source: B. Development of the Usefulness… · p. 8

IV. EXPERIMENT

  • The experiments use 1,482 ABC Company review comments, 10-fold cross-validation, a separate case study, and three baseline comment-classification models.The baselines were models used by Bosu et al..Source: §IV · p. 8
  • The experiment asks how well RevHelper predicts usefulness, whether text and reviewer experience act as useful proxies, and whether it outperforms baseline models.Source: §IV · p. 8

A. Experimental Dataset

The study evaluates RevHelper using manually annotated review comments and reports its predictive performance and the relative contribution of reviewer experience and comment text.

A. Experimental Dataset

  • The experimental dataset comprised 1,482 review comments extracted from four commercial subject systems.Source: A. Experimental Dataset · p. 8
  • Comments were annotated as useful when they triggered a code change within 1–10 lines in subsequent commits, following Bosu et al..Source: A. Experimental Dataset · p. 8

B. Evaluation of RevHelper

C. Comparison with Baseline Models using Case Study

The case study compares RevHelper with three Bosu et al. variants built from features available during review submission. RevHelper performed better, including on non-useful comments.

C. Comparison with Baseline Models using Case Study

V. THREATS TO VALIDITY

The authors identify manual-analysis error and the study’s single-company, Python-system scope as threats to validity, and describe steps taken to address or qualify them.

V. THREATS TO VALIDITY

  • Manual annotation could involve human error; participants received pilot-example training and supervision, with random cross-checks to reduce false positives.Source: §V · p. 10
  • MSR-Splat found no comments negative, which the authors considered unlikely; Stanford Sentiment Analyzer produced unchanged overall validation findings.Source: §V · p. 10
  • The study covers one company and Python-based systems, so its findings’ generalizability beyond that setting remains uncertain.Source: §V · p. 10

VI. RELATED WORK

The related-work discussion distinguishes prior analyses of review practices and comment usefulness from automated support for improving comments. The conclusion presents RevHelper’s prediction results and identifies personalized recommendation systems as future work.

VI. RELATED WORK

  • Earlier work studied review practices and comment usefulness, but the authors describe a lack of tools applying such insights to automated support for code review.Source: §VI · p. 10

VII. CONCLUSION AND FUTURE WORK

  • The paper concludes that RevHelper predicts comment usefulness instantly, using comment text and developer experience; evaluation on 1,482 comments reported 66% accuracy.Source: §VII · p. 10
  • The authors identify personalized recommendation systems as a future extension of their code-review support.Source: §VII · p. 10
Loading 1807.04485v1…