PaperlayerView exact revision on arXiv

Source-linked AI summary

From Human-Centric to Agentic Code Review: The Impact of Different Generations of Generative AI Technology on Review Quality

Suzhen Zhong, Shayan Noei, Bram Adams, Ying Zou

arXiv:2607.13196v1cs.SE

TL;DR

As AI reviewers enter code review, evidence remains limited on how transitions from human-centric to LLM-assisted and agentic review affect efficiency and quality. Analyzing 1.02 million pull requests across 207 projects, the paper finds that some agent-involved patterns speed decisions but do not improve review quality.

  • Problem

    Evidence remains limited on how code review practices and quality evolve as projects transition from human-centric to LLM-assisted and agentic review.

  • Method

    The study analyzes 1.02 million pull requests from 207 GitHub projects across review eras, adoption practices, and reviewer interaction patterns modeled with Markov chains.

  • Results

    Agent-initiated and multi-agent reviews reach decisions faster under Gradual AI Adoption and Rapid AI Agent Adoption, but exhibit review smells more often than human-only reviews.

  • Takeaways & Limitations

    Developers should adopt AI reviewers selectively, considering adoption history, reviewer participation and timing, and pull request characteristics.

  • Takeaways & Limitations

    The agent era is chronological and may include LLM reviewers, so it does not guarantee that every AI reviewer is an agent.

Abstract

from arXiv · show

Code review helps maintain software quality before code integration, but it also imposes a substantial workload on human reviewers. As generative artificial intelligence becomes part of software development, code review is shifting from a primarily human review process toward AI-supported review processes in which large language model (LLM) reviewers and AI agent reviewers participate alongside human reviewers. However, we still lack empirical evidence on how this transition affects review efficiency and review quality. In this paper, we study 1.02 million reviewed pull requests from 207 GitHub projects that transition across three code review eras: human-centric review, LLM-assisted review, and agentic code review. We identify three AI reviewer adoption practices: Gradual AI Adoption, Rapid LLM Adoption, and Rapid AI Agent Adoption. We further model pull request review discussions as reviewer interaction sequences to characterize how human, LLM, and AI agent reviewers collaborate during the review process. Our results show that agent-involved collaboration patterns, especially reviews initiated by AI agents or involving multiple AI agents, are associated with faster review decisions under Gradual AI Adoption and Rapid AI Agent Adoption. However, these efficiency gains do not translate into better review quality. We also find that review activity and pull request type remain important across eras, while human-AI collaboration patterns become the strongest explanatory factor for review efficiency once LLM and AI agent reviewers participate. These findings provide empirical guidance for designing AI-supported code review processes that improve efficiency without weakening review quality.

I. INTRODUCTION

The paper examines how code review changes as projects move from human-centric to LLM-assisted and agentic review, using 1.02 million pull requests from 207 open-source projects. It identifies three AI reviewer adoption practices and finds that agent-era efficiency gains do not consistently imply better review quality.

  • Study scope: 1.02 million pull requests from 207 open-source projects form a longitudinal study spanning human-centric, LLM-assisted, and agentic review eras.The authors also release a replication package containing the longitudinal dataset.
  • AI adoption practices: Three AI reviewer adoption practices are identified: Gradual AI Adoption, Rapid LLM Adoption, and Rapid AI Agent Adoption.The study examines how these practices relate to review quality.
  • Review efficiency: Gradual AI Adoption and Rapid AI Agent Adoption show significantly more efficient review in the agent era.This result addresses how adoption practices relate to review efficiency and quality.
  • Human-AI collaboration: No human-AI collaboration pattern consistently outperforms human-only review in both efficiency and quality.The study analyzes interactions among human, LLM, and AI agent reviewers within individual pull requests.
  • Explanatory factors: Human-AI collaboration patterns become a strong explanatory factor once LLM and AI agent reviewers join, alongside traditional review factors.Traditional factors include pull request characteristics and review activity.

II. CASE STUDY SETUP · A. Project Selection

The study selects active GitHub projects with continuous review activity across pre-LLM, LLM, and agent eras, narrowing 2,490 candidates to 207 projects. It then examines AI adoption practices, pull request types, review quality, and human-AI collaboration patterns.

  • II. CASE STUDY SETUP: The case study analyzes data collection and analysis approaches for research questions presented in Sections III-A–III-C.
  • II. CASE STUDY SETUP: 2,490 candidate GitHub projects with continuous review activity were identified, and 207 were retained with sufficient reviewed pull requests across the pre-LLM, LLM, and agent eras.
  • II. CASE STUDY SETUP: The study clusters each project’s generative AI adoption series into common AI reviewer adoption practices.
  • II. CASE STUDY SETUP: Within each adoption practice, the study examines pull request types increasingly involving AI reviewers and changes in review quality across the three eras.
  • II. CASE STUDY SETUP: The study identifies common human-AI collaboration patterns and compares them by review efficiency and quality.
  • A. Project Selection: Projects were selected for continuous review activity before and after the emergence of generative AI reviewers.
  • A. Project Selection: Eligible projects had at least 100 stars, indicating sufficient community adoption and active maintenance.
  • A. Project Selection: Projects had to be created before May 2022 and maintain at least one reviewed pull request per month from May 2022 to February 2026.These requirements provided at least six months of review history before ChatGPT’s public release and ensured continuous activity throughout the observation window.

B. Data Collection and Preprocessing · C. Pull Request Types

The study collects reviewed pull requests and conversations, labels reviewers by the tools behind their accounts, and defines project-specific review eras by AI reviewer participation. Pull requests are classified into code-change-purpose categories using GPT-4.1-mini, with strong agreement against manual labels.

  • B. Data Collection and Preprocessing: 2,490 candidate projects provide reviewed pull requests, review conversations, reviewer identities, timestamps, decisions, and reviewer comments through the GitHub REST API.Decisions are recorded as accepted when merged and rejected when closed without merge.
  • B. Data Collection and Preprocessing: Reviewer accounts are labeled by the reviewer type represented by their underlying tool, distinguishing human and bot accounts before manually classifying bot tools.The process uses the GitHub REST API and official tool documentation to distinguish rule-based bots, traditional machine-learning tools, and generative-AI reviewers.
  • B. Data Collection and Preprocessing: 24 sampled cases with only natural-language comments were labeled as involving AI agent reviewers after documentation and configuration checks found no evidence that agentic review was disabled.Official documentation identifying the reviewers as AI agents supported this classification.
  • B. Data Collection and Preprocessing: Each project’s pre-LLM, LLM, and agent eras are defined chronologically by the first reviewed pull request involving an LLM or AI agent reviewer.The pre-LLM era precedes generative-AI participation; the LLM era begins with the first LLM-involving review and ends before the first AI-agent-involving review.
  • C. Pull Request Types: Pull request type is a shared measure across all three research questions and is assigned at the pull-request level.The paper describes the classification procedure in this section because pull request type is used throughout the analyses.
  • C. Pull Request Types: 11 pull request types characterize code-change purpose, and the study applies Li et al.’s GPT-4.1-mini classification process to titles and descriptions.The taxonomy was originally proposed by Watanabe et al. and is presented in Table I.
  • C. Pull Request Types: 384 manually labeled pull requests produce a Cohen’s κ of 0.91 against GPT-4.1-mini classifications, indicating almost perfect agreement.The sample uses a 95% confidence level and 5% margin of error.

D. Review Quality Evaluation … E. Explanatory Factors for Review Efficiency and Quality

The paper evaluates review efficiency through change-normalized decision time and treats review smells as quality-risk indicators across human-only and human-AI reviews. It also models human-AI collaboration alongside pull request characteristics, review activity, and reviewer experience as explanatory factors.

  • D. Review Quality Evaluation: Review efficiency and review smells are the shared review-quality measures used across all three research questions during the transition toward agentic review.The paper examines whether movement from human-centric to agentic review is associated with changes in these measures.
  • 1) Review Efficiency:: Review efficiency measures the time from pull-request creation to final review decision, normalized by the code change size in thousands of lines of code.This measure follows prior work and accounts for the greater effort potentially required by larger changes.
  • 1) Review Efficiency:: Review Efficiency = Tdecision − Tcreation # Thousand lines of code.Tcreation denotes pull-request creation, while Tdecision denotes the final decision timestamp.
  • 2) Review Smells:: Review smells represent observable review-process anti-patterns that may reduce effectiveness and introduce quality risks.Examples include low participation and large changes receiving less useful feedback.
  • E. Explanatory Factors for Review Efficiency and Quality: The explanatory-factor analysis jointly models reviewer collaboration patterns with alternative factors affecting review efficiency and review smells.Table III identifies the independent variables used for this analysis and abbreviations later used in Table VI.
  • 2) Review Smells:: Review smells remain relevant indicators of process-quality risk in both human-only and human-AI reviews because AI participation does not remove missing context, repeated back-and-forth, or narrow feedback.The paper applies these indicators across both review settings.
  • E. Explanatory Factors for Review Efficiency and Quality: Traditional explanatory factors include pull request characteristics, review activity, and reviewer experience, following prior code-review research.Prior work associates review quality with these review-process factors.

1) Pull Request Characteristics: … A. RQ1: What are the common AI adoption practices and how

The paper examines pull request characteristics, review activity, participant experience, and reviewer adoption practices to understand how AI reviewer participation relates to review quality. AI adoption is measured through time-series proportions of pull requests reviewed by different reviewer types, aggregated monthly to reduce noise.

  • 1) Pull Request Characteristics:: Pull request characteristics describe a change’s purpose and initial scope before review, using pull request type, initial code churn, files changed, and commit count.These measures are summarized in Table III.
  • 2) Review activity:: Review activity captures revision and discussion during review through commit count, unique reviewer count, and reviewer discussion threads.Threads are opened on specific code changes.
  • 3) Participant experience:: Participant experience reflects the author’s and reviewers’ familiarity with the project before the current review.Reviewer type, including LLM reviewers, is also included as a factor that may explain review quality.
  • A. RQ1: What are the common AI adoption practices and how do they relate to review quality?: The RQ1 motivation is to identify common AI reviewer adoption practices and measure their effects on code review quality.The study addresses how AI reviewers are introduced and used over time in software projects.
  • A. RQ1: What are the common AI adoption practices and how do they relate to review quality?: AI adoption is represented by time series showing the proportion of pull requests reviewed by each reviewer type, including human and AI agent reviewers.The approach distinguishes reviewer types to characterize adoption across projects.
  • A. RQ1: What are the common AI adoption practices and how do they relate to review quality?: Monthly aggregation of reviewer-type proportions reduces noise caused by AI reviewer participation varying between individual pull requests.The aggregation uses the era-specific generative AI reviewer type.

N LLM

The study identifies three AI reviewer adoption practices, each describing a distinct transition from human-centric to LLM-assisted and agentic review. These practices differ in reviewer participation, pull request targeting, review efficiency, and review-smell outcomes.

  • Adoption practices: Three practices cover 46% Gradual AI Adoption, 22% Rapid LLM Adoption, and 32% Rapid AI Agent Adoption of the 207 studied projects.The three-cluster solution has a silhouette score of 0.40, indicating fair separation.
  • Adoption practices: Gradual AI Adoption averages 8% LLM participation in the LLM era and 36% AI-agent participation in the agent era.LLM reviewers focus on chore pull requests at 2.0 times the rate of other types and fix pull requests at 1.2 times; agents focus on feature pull requests at 1.4 times and refactoring pull requests at 1.3 times.
  • Adoption practices: Rapid LLM Adoption averages 91% LLM participation in the LLM era, followed by 93% AI-agent participation in the agent era.This practice comprises 45 projects, including 12 web applications.
  • Adoption practices: Rapid AI Agent Adoption averages 19% LLM participation in the LLM era, followed by 76% AI-agent participation in the agent era.The practice includes 12 Microsoft projects and 3 Google projects.
  • Review quality: Rapid LLM Adoption increases review-smell rates by 8.0 percentage points in the LLM era and 4.4 percentage points in the agent era versus the pre-LLM baseline.Review Buddies increase by 26% in the LLM era and 23.2% in the agent era, while Gradual AI Adoption and Rapid AI Agent Adoption significantly improve review efficiency in the agent era.

B. RQ2: What is the impact of human-AI collaboration pat-

Human-AI collaboration patterns are modeled from ordered reviewer-type sequences and compared across adoption practices and review eras. Agent-initiated and multi-agent reviews can improve efficiency in some settings, but AI-involved patterns generally exhibit more review smells than human-only review.

  • Collaboration-pattern analysis: Reviewer-type comment sequences are clustered into collaboration patterns using Markov chains and expectation-maximization, then compared across review eras.Each sequence preserves reviewer order and repeated participation before the pull request decision.
  • Evaluation: Efficiency is ranked within each adoption practice and era using the Scott-Knott ESD test, while review smells and pull request types are tested with Bonferroni-corrected chi-square tests.The analyses compare AI-involved collaboration patterns with human-only review.
  • Review efficiency: Under Rapid LLM Adoption, all LLM-involved review patterns are significantly slower than human-only review, whereas human-bot and multi-LLM patterns are as efficient as human-only review in Gradual AI Adoption and Rapid AI Agent Adoption.LLM-Assist reviews involve more human back-and-forth, with medians of four versus two human comments in human-only reviews.
  • Review efficiency: Under Gradual AI Adoption and Rapid AI Agent Adoption, agent-initiated and multi-agent reviews are significantly faster than human-only review.These patterns rank R1–R2 versus R2–R3 for human-only review; under Rapid LLM Adoption, no significant efficiency difference is observed.
  • Review quality: Review smell prevalence is 69%–76% for human-only reviews versus 78%–94% for AI-involved patterns across the LLM and agent eras.Review Buddies is the main driver, averaging 16% for human-only reviews, 60% for LLM-involved patterns, and 53% for agent-involved patterns.

traditional factors impacting review quality?

Traditional review factors remain important after AI reviewers are adopted, although their influence varies across review eras and outcomes. Review activity, author experience, and pull request characteristics are associated with review efficiency and review smells alongside human-AI collaboration patterns.

  • Approach: Logistic-regression models assess how collaboration patterns, pull request characteristics, and traditional review factors associate with review efficiency and review smells.Efficiency is classified using the median #days/KLOC; review-smell models focus on Review Buddies, Sleeping Review, and Large Changeset.
  • Review efficiency: Traditional review factors remain important after generative AI reviewers are adopted, while their associations vary across review eras and outcomes.In the agent era, AI agent collaboration becomes the dominant factor associated with review efficiency, whereas traditional factors remain relevant but less consistently.
  • Sleeping Review: More commits and inline discussion threads consistently increase the probability of Sleeping Review across review eras.Impact scores range from +27 to +70 for commits and from +26 to +51 for inline discussion threads.
  • Sleeping Review: Greater author experience and build or chore pull requests are associated with a lower probability of Sleeping Review.The reported impact scores range from −10 to −50.
  • Review smells: Agent-era collaboration patterns reduce Sleeping Review under Gradual AI Adoption and Rapid AI Agent Adoption but are associated with more Review Buddies and larger changesets.Among traditional factors, build pull requests show mixed associations with Large Changeset across practices and eras.

IV. IMPLICATIONS

The implications support selective, context-aware AI reviewer adoption that balances efficiency gains with traditional review practices and quality safeguards. They also call for research into developer decisions and context-sensitive benchmarks for evaluating human-AI review.

  • Practitioner implications: Gradual AI Adoption and Rapid AI Agent Adoption are associated with faster reviews, whereas Rapid LLM Adoption is associated with lower review quality.Practitioners may therefore select AI reviewers based on pull request context and observed efficiency and quality rather than applying uniform adoption.
  • Tool-builder implications: Human-AI collaboration patterns, pull request type, changeset size, discussion activity, and author or reviewer history can inform context-aware AI reviewer assignment.These signals can guide whether a pull request remains human-led, uses lightweight LLM summarization, or involves AI agent inspection.
  • Practitioner implications: AI collaboration does not uniformly improve review quality, while traditional factors such as author experience and review activity remain strongly associated with review efficiency and review smells.Some agent-era patterns coincide with fewer Sleeping Reviews but more Review Buddies and larger changesets, supporting integration with traditional review practices.
  • Research implications: Future research should investigate why developers assign AI reviewers, how they interpret AI feedback, and when they keep humans in the lead.Interviews or surveys could examine how these decisions shape review efficiency and quality across projects.
  • Research implications: Researchers and tool builders should develop benchmarks that incorporate pull request type, changeset size, discussion activity, author or reviewer history, and human-AI collaboration patterns.Context-sensitive benchmarks should cover routine maintenance, bug fixing, refactoring, and large changesets rather than relying only on generic issue-finding ability.

V. THREATS TO VALIDITY

The study’s validity is limited by imperfect era classification, non-causal regression analyses, and the possibility that observed associations may change as AI reviewer capabilities evolve. Its longitudinal dataset nevertheless provides a baseline across preLLM, LLM, and agent eras.

  • Construct validity: The agent era is defined chronologically after a project’s first AI-agent-reviewed pull request, so agent-era reviews may still include LLM reviewers.This classification does not guarantee that every AI reviewer in the agent era is an AI agent.
  • Construct validity: LLM-involved patterns in the agent era had less than 5% support and were excluded from the reported pairwise comparisons.Complete pattern results and frequencies are available in the replication package.
  • Internal validity: The RQ3 logistic regression models are explanatory rather than causal, so reported associations do not show that changing a factor directly changes review efficiency or review smells.The associations concern factors and review efficiency and quality within each practice and era.
  • External validity: AI capability evolution may shift the observed collaboration patterns and review-quality associations, although the dataset covers May 2022 through February 2026.The longitudinal analysis across preLLM, LLM, and agent eras provides a baseline even as tool-specific findings evolve.

VI. RELATED WORK

Prior work has examined code review practices and the technical and social factors influencing review efficiency and quality. This study extends that work by jointly analyzing human-AI collaboration with pull request characteristics, review activity, and participant experience.

  • Code review practices: Prior studies characterize code review practices, identify review smells, and link reviewer experience and changeset characteristics to review comment quality.They also examine review intervals, participation convergence, and modern review practices at scale.
  • Review efficiency and quality factors: Human-centric review research associates efficiency and quality with organizational structure, developer relationships, review participation, patch description length, reviewer experience, and changeset characteristics.These factors capture both technical and social influences on review outcomes.
  • Study contribution: This study models review outcomes with human-AI collaboration patterns, pull request characteristics, review activity, and participant experience to assess whether differences are associated with AI participation or alternative factors.The analysis considers both AI reviewer participation and established explanatory factors.

VII. CONCLUSION

Across 1.02 million pull requests from 207 GitHub projects, AI reviewer adoption affects review efficiency and review-smell prevalence differently across adoption practices. The findings support selective AI reviewer adoption and motivate future work on improving efficiency while preserving human-established quality standards.

  • Key Findings: Gradual AI Adoption and Rapid AI Agent Adoption are associated with faster reviews but no improvement in review smells.Rapid LLM Adoption is associated with higher review-smell prevalence and no efficiency gain.
  • Implications: Developers should adopt AI reviewers selectively based on adoption history, reviewer identity and timing, and pull request characteristics.These factors are identified as relevant considerations for AI-supported review processes.
  • Future Work: Future work should investigate how agentic review systems can improve review efficiency while preserving human code-review quality standards.The proposed direction addresses the gap between efficiency gains and review quality preservation.
Loading 2607.13196v1…