PaperlayerView exact revision on arXiv

Source-linked AI summary

Does AI Code Review Lead to Code Changes? A Case Study of GitHub Actions

Kexin Sun, Hongyu Kuang, Sebastian Baltes, Xin Zhou, He Zhang, Xiaoxing Ma, Guoping Rong, Dong Shao, Christoph Treude

arXiv:2508.18771v2cs.SE

TL;DR

The real-world adoption and development impact of AI-based code review actions remain insufficiently understood. This study examines these actions using an LLM-assisted assessment of addressed comments and finds that human comments are more often addressed, while AI comment outcomes vary by tool.

  • Problem

    The adoption of AI-based review actions and whether their comments lead to code changes remain empirically underexplored.

    Source: §I · p. 1

  • Method

    The authors use an LLM-assisted framework to assess comment validity and addressing, then analyze addressed comments across AI-based review actions.

    Source: §I · p. 2, lead to code changes compared to human… · p. 5

  • Results

    60% of valid human review comments led to code changes, compared with 0.9%–19.2% of valid AI-generated comments depending on the tool.

    Source: §V · p. 11

  • Takeaways & Limitations

    AI review actions have potential to influence development if designed well, and concise, specific comments are more likely to be addressed.

    Source: §V · p. 11, §VIII · p. 15

    Addressing was inferred from file-level code changes, which may miss edge cases; uncertain cases made up 1.3% of the addressing dataset and were excluded from impact-factor analysis.

    Source: §VI · p. 13

Abstract

from arXiv · show

AI-based code review tools automatically review and comment on pull requests to improve code quality. Despite their growing presence, little is known about their actual impact. We present a large-scale empirical study of 16 popular AI-based code review actions for GitHub workflows, analyzing more than 22,000 review comments in 178 repositories. We investigate (1) how these tools are adopted and configured, (2) whether their comments lead to code changes, and (3) which factors influence their effectiveness. We develop a two-stage LLM-assisted framework to determine whether review comments are addressed, and use interpretable machine learning to identify influencing factors. Our findings show that, while adoption is growing, effectiveness varies widely. Comments that are concise, contain code snippets, and are manually triggered, particularly those from hunk-level review tools, are more likely to result in code changes. These results highlight the importance of careful tool design and suggest directions for improving AI-based code review systems.

I. INTRODUCTION

AI-based code review actions are increasingly integrated into GitHub workflows, but their real-world adoption and effects on development remain poorly understood. The study examines how developers use these tools, whether their comments lead to code changes, and which factors influence that outcome.

I. INTRODUCTION

  • Prior research has examined human code review, while the adoption and effects of GenAI-powered reviews remain largely unexplored.Open questions include which tools developers adopt, whether comments lead to code changes, and what comment characteristics influence action.Source: §I · p. 1
  • The study asks how LLM-based review actions are adopted on GitHub, whether their comments lead to code changes, and which factors affect that likelihood.The questions address tool configuration and use, comparison with human comments, and contextual and technical factors.Source: §I · p. 1
  • The authors analyze 16 popular GitHub code-review actions across 178 repositories, examining more than 22,000 review comments.They use a two-stage LLM-based framework to assess whether comments are addressed and interpretable machine learning to identify influencing factors.Source: §I · pp. 1–2
  • The study categorizes AI-based review actions by feedback granularity and behavior, including PR-, file-, and hunk-level review.The authors present this as the first systematic study of adoption and usage of these actions on GitHub.Source: §I · p. 2
  • The authors introduce an LLM-assisted framework to assess whether review comments are actionable and addressed, reporting high accuracy against human annotations.The framework is intended to support large-scale analysis of comment addressing.Source: §I · pp. 1–2
  • The study identifies factors associated with whether AI-generated comments lead to code changes and offers design implications for review tools.Its broader motivation is that AI-driven actions post suggestions on pull requests, while their impact on development has little empirical evidence.Source: §I · pp. 1–2

II. BACKGROUND

The background illustrates how a GitHub Action is configured and runs on pull-request changes. The study setup selects 16 actions and compares their feedback granularity and design choices.

II. BACKGROUND

  • A YAML workflow can invoke anc95/ChatGPT-CodeReview on pull requests and require repository-access and LLM-authentication credentials.Optional parameters can select the model, prompt, and comment language.Source: §II · p. 2, Figure 2 · p. 2
  • After a trigger, the example action compares the base and latest commits, reviews changes file by file, and posts inline comments on the pull request.It may skip files using ignore patterns and divide large change blocks according to token limits.Source: §II · p. 2

III. STUDY SETUP

  • The authors screened about 240 popular Marketplace actions, excluded four of 20 candidates, and selected 16 actions focused on comment generation.The selection process began January 7, 2025.Source: §III · p. 3
  • Review actions are grouped as PR-, file-, or hunk-level, reflecting how specifically comments are linked to changed code.Hunk-level feedback targets code lines; PR-level feedback can span multiple files.Source: §III · p. 3, Table I · p. 3
  • Actions also differ in compared commits, change segmentation, model, prompt context, and comment format.The study illustrates design choices across the three granularity categories; Table I compares selected examples.Source: §III · p. 3, Table I · p. 3
  • The setup presents categories of optional configuration parameters alongside descriptions and examples.Table III summarizes these categories.Source: Table III · p. 4

IV. ANALYSES & RESULTS

RQ1 finds that use was concentrated among four actions, while many mature repositories that declared an action showed no generated comments. Most repositories also customized at least one optional parameter.

A. RQ1: How are LLM-based code review actions adopted in GitHub repositories?

  • The adoption analysis measured action use across repositories, pull requests, and comments, and examined developers’ configuration settings.Optional parameters were grouped into nine categories, and configuration evolution was assessed using the latest and initial workflow configurations.Source: A. RQ1: How are LLM-based code review… · p. 4
  • The analysis identified repositories invoking the target actions, excluded repositories with ambiguous action mappings, and limited the sample to projects with at least 50 pull requests.Comments were retrieved through GitHub’s REST API after identifying relevant pull requests.Source: A. RQ1: How are LLM-based code review… · p. 4
  • 37.1% of mature repositories declared an action but showed no comments; four actions accounted for 91.1% of repositories, 95.2% of pull requests, and 98.9% of comments.The four included two file-level and two hunk-level reviewers, and activity was largely driven by anc95/ChatGPT-CodeReview.Source: Table IV · p. 4, A. RQ1: How are LLM-based code review… · pp. 4–5
  • 82.6% of the 178 mature repositories customized at least one optional parameter.The most frequently configured options were prompt context augmentation (103/147, 70.1%), LLM selection (95, 64.6%), and input settings (70, 47.6%).Source: A. RQ1: How are LLM-based code review… · p. 5
  • Repositories changed their configuration files an average of four times after setup, while 68 repositories (38.2%) retained their original configurations.Among post-setup changes, 50 projects finalized modifications within one week and 40 continued adjustments beyond one month.Source: A. RQ1: How are LLM-based code review… · p. 5

B. RQ2: To what extent do AI-generated review comments

The study uses a two-stage LLM-assisted analysis, validated against human annotations, to assess whether AI review comments lead to code changes. Hunk-level comments were addressed more often than file-level comments, but AI comments lagged behind human reviews.

Approach

  • The two-stage analysis first assessed comment validity, then whether identified issues or suggestions were addressed, applying the best-performing setup to the full dataset.The authors created a comment dataset, manually annotated a subset, evaluated LLM performance, and then used the selected setup for large-scale analysis.Source: lead to code changes compared to human… · pp. 5, 7, Figure 3 · p. 5

Phase I

  • Filtering retained first-in-thread English comments from merged pull requests, leaving 2,908, 399, 713, and 169 comments for Actions ID-1 to ID-4, respectively.The analysis excluded PR-level actions and used reviewed changes and subsequent changes in the reviewed file as context.Source: lead to code changes compared to human… · pp. 5–6

Phase II

  • Human annotators distinguished comments without suggestions, overly general comments, and valid actionable comments before judging whether valid suggestions were unaddressed, partially addressed, or fully addressed.The scheme also included uncertain cases; it was designed for comments containing multiple suggestions and for suggestions whose resolution could not be determined.Source: lead to code changes compared to human… · p. 6
  • Agreement across the six-class annotation scheme was substantial, with Cohen’s κ of 0.674 for file-level actions, 0.734 for hunk-level actions, and 0.764 for human reviews.A third author resolved disagreements between the two annotators.Source: lead to code changes compared to human… · p. 6

Phase III

Results

  • 60% of valid human comments were addressed, compared with 0.9%–19.2% for valid AI comments; hunk-level actions reached 6.5%–19.2% versus 0.9%–4.2% for file-level actions.The automated labels covered comments across the full dataset, including many comments with no subsequent file changes.Source: Table IX · p. 7, lead to code changes compared to human… · p. 8

comments lead to code changes?

An interpretable analysis associates comment addressing with feedback source and comment characteristics. Hunk-level, manually triggered, concise, code-rich comments and comments directed at less experienced contributors were associated with higher addressing rates.

Approach

  • The analysis modeled 3,879 valid comments with definitive addressing labels using Random Forest and SHAP, after reducing 45 candidate features to 36.The full-dataset model achieved 88.6% overall accuracy and Macro-F1=0.846 under stratified five-fold cross-validation.Source: comments lead to code changes? · pp. 8–9

Results

  • Source Features and Comment Features ranked first and second in total importance, associating both feedback source and comment content with whether comments were addressed.The analysis used a Random Forest model and SHAP to interpret feature influence and directionality.Source: comments lead to code changes? · pp. 8–9

Source Features

  • Hunk-level review actions generally outperformed file-level counterparts, while manually triggered comments had consistently higher addressing rates in the two actions with manual triggering; automatic triggering correlated negatively with addressing (ρ = −0.96).Source: comments lead to code changes? · p. 10

Comment Features

  • Concise comments and comments with higher code-to-text ratios were more likely to be addressed, with addressing rates increasing when code made up more than half the comment.Multiline code suggestions were also positively associated with addressing.Source: comments lead to code changes? · p. 10, Table XV · p. 10

Modification and Repository Features

  • Comments targeting newcomers with Author Prior Commits≤124 had a 16% addressing rate, compared with 3.3% for the most experienced contributors.The analysis also associated comments with larger and more active repositories with higher addressing rates.Source: Table XVI · p. 10, comments lead to code changes? · p. 10

AI-Specific Modeling

  • In the AI-only analysis, the main feature influence directions remained consistent, while error-handling and documentation topics were positively associated with addressing.Author Prior Commits ranked highly with a negative correlation in the AI-only model.Source: Table XII · p. 9, comments lead to code changes? · p. 10

V. DISCUSSION

The discussion finds that AI-generated reviews are addressed less often than human reviews, while suggesting that well-designed tools can still influence development.

Discussion

  • 60% of valid human review comments led to code changes, compared with 0.9%–19.2% of valid AI-generated comments, depending on the tool.The authors conclude that AI review still has potential to influence development if tools are designed well.Source: §V · p. 11
  • The contrast between human and AI addressing rates shows that AI review comments are not uniformly ineffective across tools.The AI range varies by tool, from 0.9% to 19.2%.Source: §V · p. 11
  • The authors frame effective tool design as a condition for AI-generated review to influence development.Source: §V · p. 11

A. Design Matters

The discussion connects addressing rates to tool and comment design, including review granularity, triggering, concrete suggestions, and prompt quality. It also advocates more selective review generation and iterative dialogue.

Design Matters

  • Among the four deeply analyzed actions, coderabbitai/ai-pr-reviewer had 19.2% of valid comments addressed, while mattzcarey/code-review-gpt had 0.9%.The authors suggest that tool design choices directly affect developers’ responses.Source: A. Design Matters · p. 11
  • Hunk-level feedback, manual triggering, concise comments, and code-rich suggestions were associated with higher addressing rates.For anc95/ChatGPT-CodeReview, manually triggered comments had a 12.8% addressing rate versus 6.8% for automatically triggered comments; rates rose when code exceeded half of comment content.Source: A. Design Matters · p. 11
  • Comments aimed at contributors with little prior commit history were more likely to be addressed, while most sampled PR-level comments were ignored.Only one of 30 sampled PR-level comments was partially addressed, and it offered a specific performance-improvement suggestion.Source: A. Design Matters · p. 11
  • In the illustrative case, interactive dialogue let the tool revise a strict equality check to a fuzzy comparison after the developer raised rounding-error concerns.The developer initially rejected the suggestion; the revised proposal became an actionable improvement.Source: Figure 4 · p. 11, A. Design Matters · pp. 11–12
  • Detailed prompts that discourage generalities and suppress unnecessary praise contrast with short generic prompts that can yield vague comments.The authors describe coderabbitai/ai-pr-reviewer’s prompt as over 400 words and note its logic for suppressing “LGTM!” comments unless no issues are found.Source: A. Design Matters · p. 11
  • The authors advocate a multi-stage review process that decides whether a change needs review, whether to comment, and whether to offer a code suggestion.They contrast this with a one-in-one-out approach, which they say produces low-precision output and reviewer fatigue.Source: A. Design Matters · p. 12

B. Beyond Code Changes: Impact on Closed PRs and Feedback on Unaddressed Comments

The authors focused their main analysis on merged PRs and ran two supplementary analyses to examine possible impact beyond merged PRs and direct code changes.

B. Beyond Code Changes: Impact on Closed PRs and Feedback on Unaddressed Comments

1) Analysis of AI Comment Impact in Closed PRs:

AI comments in closed PRs were addressed less often than in merged PRs, and the authors suggest that valid feedback may not prevent abandonment when it misses the root cause.

1) Analysis of AI Comment Impact in Closed PRs:

2) Analysis of Feedback on Unaddressed AI Comments:

Replies to unaddressed AI comments ranged from defending current code to reflection and willingness to act later; the authors recommend design choices intended to make feedback more actionable and useful.

2) Analysis of Feedback on Unaddressed AI Comments:

C. Recommended Features for Effective AI Code Review

D. Implications for AI Code Review Adopters

The authors advise adopters to tune tools, treat AI review as a complement to human review, and monitor effectiveness over time.

D. Implications for AI Code Review Adopters

  • 82.6% of analyzed repositories customized their action configurations, and 22.5% continued adjusting them beyond one month.The authors recommend tool selection, customization, and ongoing optimization rather than relying on defaults.Source: D. Implications for AI Code Review… · pp. 12–13
  • 19.2% was the addressing rate for the best-performing AI tool, compared with 60% for human reviewers.The authors recommend framing AI code review as a complement to human review, not a replacement.Source: D. Implications for AI Code Review… · p. 13
  • The authors recommend positioning AI reviewers as defenders, given their strengths in defensive programming and developer responsiveness to error-handling comments.They suggest leaving higher-level architecture, development context, and business logic to human reviewers.Source: D. Implications for AI Code Review… · p. 13
  • The authors recommend monitoring comment validity and addressing rates over time as a feedback loop for tuning the AI reviewer.Source: D. Implications for AI Code Review… · p. 13

E. Implications for GitHub

The paper recommends GitHub improve action traceability and support code-change detection, while noting limits on attribution, generalizability, and causal interpretation. It situates its contribution in work that has focused more on comment quality than downstream impact.

E. Implications for GitHub

  • GitHub should make action behavior attributable to individual actions and provide Marketplace usage lists to help users and researchers assess popularity.The shared github-action[bot] account complicates attribution; a usage list could indicate successful runs.Source: E. Implications for GitHub · p. 13
  • GitHub could support incremental review by tracing the last-reviewed commit and providing file-content comparison that ignores rebase noise.Most tools re-review the full base-to-head diff because reviewed commits are difficult to trace.Source: E. Implications for GitHub · p. 13

VI. THREATS TO VALIDITY

  • The study identifies AI comments through a shared bot account, so unrelated actions may be misclassified; file-level change labels may also miss edge cases.Uncertain cases were 1.3% of the addressing dataset and were excluded from impact-factor analysis.Source: §VI · p. 13
  • Findings are bounded by the study's scope, including English-language comments, repositories with at least 50 pull requests, and file- or hunk-level actions.The analysis also focuses on small- to medium-sized projects and uses a binary addressed/not-addressed outcome that does not capture the degree or impact of code changes.Source: §VI · p. 14

VII. RELATED WORK

  • The study addresses a gap in prior work, which has mostly assessed generated-comment quality rather than whether comments are used in workflows or lead to code changes.It analyzes whether comments are addressed and examines factors associated with their effectiveness.Source: §VII · p. 14

VIII. CONCLUSION AND FUTURE WORK

The study finds adoption concentrated on four popular actions and that vague or context-poor comments are often not addressed, while concise, specific comments are more likely to be addressed. It recommends treating AI reviewers as collaborators and exploring richer context, better prompts, and multi-stage strategies.

VIII. CONCLUSION AND FUTURE WORK

  • Adoption is highly concentrated on four popular actions, and many AI-generated comments are not addressed, especially when vague or lacking context.Concise, specific comments are more likely to be addressed, particularly by less experienced contributors.Source: §VIII · p. 15
  • Even without immediate code changes, AI comments can prompt developer reflection and inform future improvements.The paper recommends treating AI code reviewers as collaborators with human reviewers.Source: §VIII · p. 15
  • Future work should explore richer context, better prompts, and multi-stage strategies to produce more specific and actionable feedback.The paper also calls for clarifying AI capabilities to identify effective forms of human-AI collaboration.Source: §VIII · p. 15
Loading 2508.18771v2…