PaperlayerView exact revision on arXiv

Source-linked AI summary

Human-AI Synergy in Agentic Code Review

Suzhen Zhong, Shayan Noei, Ying Zou, Bram Adams

arXiv:2603.15911v1cs.SE

TL;DR

The paper addresses limited empirical evidence about how AI-agent and human code reviewers differ and how they collaborate. It analyzes 278,790 review conversations from 300 open-source GitHub projects, comparing feedback, interaction patterns, suggestion adoption, and code-quality effects. The results show that AI agents provide narrower feedback and have lower suggestion adoption, while human oversight remains important for contextual feedback and suggestion quality.

  • Problem

    Prior work lacks empirical evidence comparing AI-agent and human feedback, their collaboration patterns, and the code-quality effects of their adopted suggestions.

  • Method

    The study analyzes 278,790 code review conversations from 300 open-source GitHub projects, comparing reviewer feedback, interaction sequences, suggestion adoption, and code-quality changes.

  • Results

    Human reviewers provide broader feedback, exchange 11.8% more rounds when reviewing AI-generated code, and achieve a 56.5% suggestion-adoption rate versus 16.6% for AI agents.

  • Takeaways & Limitations

    AI agents can support scalable defect screening, but human oversight remains critical for contextual feedback and suggestion quality.

  • Takeaways & Limitations

    The findings may not generalize to proprietary enterprise systems, smaller niche repositories, traditional machine-learning tools, or future agent capabilities.

Abstract

from arXiv · show

Code review is a critical software engineering practice where developers review code changes before integration to ensure code quality, detect defects, and improve maintainability. In recent years, AI agents that can understand code context, plan review actions, and interact with development environments have been increasingly integrated into the code review process. However, there is limited empirical evidence to compare the effectiveness of AI agents and human reviewers in collaborative workflows. To address this gap, we conduct a large-scale empirical analysis of 278,790 code review conversations across 300 open-source GitHub projects. In our study, we aim to compare the feedback differences provided by human reviewers and AI agents. We investigate human-AI collaboration patterns in review conversations to understand how interaction shapes review outcomes. Moreover, we analyze the adoption of code suggestions provided by human reviewers and AI agents into the codebase and how adopted suggestions change code quality. We find that human reviewers provide additional feedback than AI agents, including understanding, testing, and knowledge transfer. Human reviewers exchange 11.8% more rounds when reviewing AI-generated code than human-written code. Moreover, code suggestions made by AI agents are adopted into the codebase at a significantly lower rate than suggestions proposed by human reviewers. Over half of unadopted suggestions from AI agents are either incorrect or addressed through alternative fixes by developers. When adopted, suggestions provided by AI agents produce significantly larger increases in code complexity and code size than suggestions provided by human reviewers. Our findings suggest that while AI agents can scale defect screening, human oversight remains critical for ensuring suggestion quality and providing contextual feedback that AI agents lack.

I. INTRODUCTION

This study examines how AI agents and human reviewers differ in code-review feedback, collaboration patterns, and the adoption and code-quality effects of their suggestions. Using a large dataset of open-source review conversations, it finds distinct feedback roles and highlights continued human importance in review workflows.

  • Feedback differences: AI agents provide more verbose but narrower feedback, focusing on Code Improvement and Defect Detection, whereas human reviewers also provide Understanding and Knowledge Transfer.AI-agent reviews average 29.6 tokens per line of code versus 4.1 for human reviews.
  • Collaboration patterns: Human reviewers exchange 11.8% more rounds when reviewing AI-generated code than human-written code.Additionally, 85–87% of AI-agent-initiated reviews end after the first comment without follow-up discussions.
  • Collaboration patterns: Conversations ending at AI-agent responses have higher rejection rates than conversations ending at human responses.The reported rejection-rate ranges are 7.1%-25.8% for AI-agent-ending conversations and 0.9%-7.8% for human-ending conversations.
  • Suggestion outcomes: Human reviewers’ suggestions are adopted more often than AI agents’ suggestions, and adopted AI-agent suggestions produce larger increases in code complexity and code size.AI agents generated 88,011 suggestions versus 25,673 from human reviewers, while adoption rates were 16.6% versus 56.5%, respectively.
  • Study scope: 278,790 inline code review conversations from 300 mature open-source GitHub projects form the study’s dataset.The dataset covers reviews involving human reviewers and AI agents across multiple review configurations.
  • Implications: The study characterizes feedback and suggestion outcomes to help practitioners assign review tasks and identify where human review remains essential.Its contributions also examine how human-AI interactions integrate into collaborative code-review workflows.

II. EXPERIMENT SETUP

The study mines inline review conversations from systematically selected GitHub projects, classifies human and AI involvement, and compares review feedback and interaction outcomes across four categories.

  • Data collection and analysis: The experimental pipeline selects projects, mines closed-pull-request conversations, classifies review categories, compares feedback and interaction patterns, and assesses code-modification impact.The approach combines data collection, identity-based categorization, review analysis, interaction analysis, and code-quality assessment.
  • Project selection: 300 projects and 54,330 closed pull requests satisfy filters for sustained activity, AI-agent review volume, and project adoption.Projects require at least 100 stars, monthly closed pull requests from 2022 to November 2025, and at least 100 pull requests reviewed by AI agents.
  • Agent identification: 16 AI-based review bots are identified manually, and reviews count as agentic only from each tool’s announced agent-capability date.Agentic capability means reasoning about code context, planning review actions, and interacting with the development environment.
  • Review categorization: The dataset distinguishes human reviews of human-written code, human reviews of agent-generated code, AI reviews of human-written code, and AI reviews of agent-generated code.The PR author determines code origin, while the first commenter determines review initiator.
  • Review categorization: 55.7% of conversations involve AI-agent reviews, while human reviews account for 44.3%; HRA and ARA represent 2.3% and 0.3%, respectively.AI-agent reviews are dominated by reviews of human-written code, whereas human reviews are dominated by the traditional human-written-code category.

C. Labeling Conversations

The study labels first comments using a nine-category code-review taxonomy and an LLM-based annotation process to compare the aspects emphasized by human reviewers and AI agents.

  • Feedback taxonomy: The Bacchelli–Bird taxonomy defines nine feedback categories, including Code Improvement, Defect Detection, Knowledge Transfer, Testing, and Understanding.The categories capture actual code-review outcomes and provide the basis for comparing human and AI feedback.
  • Automated labeling: GPT-4.1-mini classifies the first comment of each conversation because it reflects the reviewer’s independent assessment before discussion begins.The approach is used because manually labeling 278,790 conversations is infeasible.

D. Interaction Pattern Extraction

Interaction sequences track whether human or agent-generated code is reviewed by humans or AI agents and connect each conversation to pull-request acceptance or rejection.

  • Sequence construction: Each sequence begins with human-written or agent-generated code, records human or AI comment authors, and ends with the pull request’s accepted or rejected status.The sequence represents the ordered flow of participants and the associated review outcome.
  • Outcome assignment: Because GitHub lacks outcomes for individual inline conversations, all conversations within a pull request inherit its merge status.This can bias conversation-level analysis when pull requests contain different numbers of inline conversations, so the study checks for that risk.

E. Code Metric Assessment

The study evaluates code suggestions by measuring software metrics before and after application, using a broad static-analysis metric set beyond complexity alone.

  • Assessment design: Code suggestions are assessed by comparing quantitative software-property metrics before and after each suggestion is applied.The measured properties include cohesion, coupling, and complexity.
  • Assessment design: SciTools Understand measures 111 code metrics to capture multiple dimensions of code quality beyond complexity.The broader assessment reflects the view that code quality includes cohesion and coupling as well as complexity.

review comments by AI agents and human reviewers?

The study compares AI-agent and human review comments by feedback content, comment-to-code density, and discussion effort. AI agents focus narrowly on code improvement and defect detection, whereas human reviewers provide more diverse, context-oriented feedback and trigger richer discussion.

  • Feedback content: AI agent comments focus on Code Improvement and Defect Detection, while human reviewers provide Understanding, Testing, Knowledge Transfer, and Social feedback.AI-agent comments contain over 95% Code Improvement and Defect Detection feedback; human reviewers provide broader feedback categories.
  • Contextual feedback: Human reviewers provide Understanding feedback in 31% of comments on human-written code and 17% on agent-generated code, while Knowledge Transfer accounts for 4–6%.These categories reflect clarification of design intent, implementation rationale, and project-specific knowledge sharing.
  • Comment-to-code density: 29.6 tokens per line of code versus 4.1 makes AI-agent comments significantly denser than human-review comments.ARH has the highest density at 29.6 tokens per line, while ARA has 11.3; human reviewers form the lowest-density cluster.
  • Comment content: AI agents and human reviewers identify the same core problems and propose the same fixes, but AI comments add severity labels, summary titles, tool evidence, and downstream-file lists.These additional features occur in manually inspected Code Improvement and Defect Detection comments.
  • Discussion effort: Understanding feedback ranks first in human-initiated reviews, with 69% of conversations extending beyond one exchange and 2.1–2.3 average discussion rounds.AI-agent reviews average only 1.2–1.3 discussion rounds and skip clarification about unclear design intent.

B. RQ2: How do interaction patterns differ between human

The study models human-AI review conversations as finite state machines and finds that AI involvement changes review effort, follow-up, and pull-request outcomes. Human reviews of AI-generated code involve more back-and-forth, whereas conversations ending with AI responses show higher rejection rates.

  • Modeling interaction sequences: Finite state machines represent reviewer states, comment transitions, and terminal pull-request acceptance or rejection.The model includes human and AI comment states, self-loops, cross-transitions, and terminal transitions.
  • Comparing interaction patterns: Review effort is measured as average comments per conversation against the human-reviewing-human-written-code baseline.The analysis also measures consecutive comments within contributor states and transition probabilities toward acceptance or rejection.
  • Review effort: 11.8% more exchange rounds occur when humans review AI-generated code than when they review human-written code.In human reviews of human-written code, 99.8% of conversations end after the initial round, averaging 1.76 exchange rounds; AI-generated code prompts more human-AI back-and-forth.
  • AI-initiated reviews: 85.2–86.7% of AI-initiated reviews terminate at the agent’s first review state without follow-up discussion.These conversations end in 30.6% fewer exchange rounds than the human-reviewing-human-written-code baseline.
  • Transition outcomes: 15.6% of conversations ending with an AI agent’s response are rejected, versus 7.8% when a human contributor responds last.Across categories, AI-terminal conversations have rejection rates of 7.1%–25.8%, compared with 0.9%–7.8% for human-terminal conversations.

C. RQ3: What is the impact of code suggestions from human

The study measures suggestion adoption, reasons for AI non-adoption, and changes in source-code metrics after adopted suggestions. Human suggestions are adopted more often, while adopted AI suggestions produce larger increases in complexity and code size.

  • Approach: The analysis extracts explicit code suggestions, tests whether subsequent commits adopt them, and compares code metrics before and after adoption.Adoption requires similarity to the suggested code, no later reversion, and a successfully merged pull request.
  • Measuring suggestion adoption: Adoption rate is defined separately for human reviewers and AI agents as adopted suggestions divided by total suggestions.Rates are also calculated across feedback types.
  • Adoption by feedback type: Human defect-detection suggestions achieve a 53.7% adoption rate, compared with 16.7% for AI suggestions.The study links the largest adoption gap to project-specific knowledge, such as renamed classes and repository conventions.
  • Unadopted suggestions: 28.7% of unadopted AI suggestions are incorrect, while 24.0% propose fixes that differ from developers’ chosen alternatives.Incorrect suggestions may break the build or contradict project behavior.
  • Code quality impact: Adopted AI suggestions significantly increase code complexity and code size more than adopted human suggestions.AI suggestions add more branching logic, with average complexity increases ranging from 0.085 to 0.106.
  • Suggestion adoption: 56.5% of human suggestions versus 16.6% of AI suggestions are adopted across feedback types.Human reviewers provide 25,673 suggestions and AI agents provide 88,011 suggestions in the analyzed conversations.

IV. IMPLICATION

The implications emphasize that AI agents and human reviewers contribute different strengths to code review. The authors recommend stronger project context, targeted feedback, and improved multi-turn responsiveness.

  • Bridging the socio-technical gap: Over 95% of AI-agent comments concern Code Improvement or Defect Detection, while human feedback more often addresses understanding and developer intent.The authors frame this difference as a socio-technical gap in contextual review feedback.
  • Project-specific context: Project-specific context is recommended because over half of unadopted AI suggestions are incorrect or replaced by developer-chosen alternatives.The implication connects low adoption with missing project-specific context in AI-generated suggestions.
  • Reducing verbosity: Targeted, actionable feedback is recommended to reduce the cognitive load associated with AI agents’ greater verbosity.The authors suggest refining prompts so agents produce less exhaustive explanations.
  • Multi-turn responsiveness: Stronger context tracking and self-verification are recommended because AI-terminal conversations have higher rejection rates after multi-turn review.These mechanisms are intended to help agents address human reviewer concerns before presenting revised patches.

V. THREATS TO VALIDITY

The study’s validity is constrained by possible classification errors, ambiguity about human-authored comments, incomplete metric coverage, and limited generalizability beyond selected public GitHub projects.

  • Threats to Construct Validity: AI-agent identification may miss early or unrecognized interactions, while LLM feedback labels may vary with prompts or ambiguous comments.Manual validation of 383 comments produced Cohen’s kappa of 0.85.
  • Threats to Construct Validity: Human-account comments may have been drafted with external LLMs, complicating the separation between human and AI-authored review text.The authors report that human-terminal conversations still have lower rejection rates than AI-terminal conversations.
  • Threats to Internal Validity: Code-quality analysis excludes unsupported languages and measures only file-level effects, omitting project-wide architecture and security attributes.Consequently, comparable tools or broader quality measures could yield different results.
  • Threats to External Validity: Findings may not generalize to proprietary enterprises, smaller repositories, traditional machine-learning tools, or future agent capabilities.The dataset covers public GitHub projects with more than 100 stars and sustained development histories.

VI. RELATED WORK

Prior research examines human review practices and AI-generated artifacts, but this study focuses on multi-turn human-agent review conversations and their downstream technical outcomes.

  • Human code review and suggestions: Human-review studies link reviewer experience and prior codebase engagement to more useful feedback and examine how suggestions affect pull-request outcomes.
  • Study positioning: This study explicitly compares human reviewers and AI agents within the same open-source projects using aligned feedback categories.
  • Study positioning: Unlike single-comment analyses, the study models review conversations as multi-turn interaction sequences and examines four reviewer pairings.
  • Study positioning: The study connects interaction patterns to downstream code changes and quality outcomes across 278,790 conversations.
  • AI-based code review: Recent AI-review research studies generated artifacts in pull-request workflows, including adoption, review time, merge likelihood, and developer interventions.

VII. CONCLUSION

Across 278,790 conversations in 300 open-source GitHub projects, AI agents and human reviewers show different feedback roles and suggestion-quality outcomes. Human reviewers provide broader contextual feedback, while human involvement remains important because agent suggestions face higher rejection and complexity costs.

  • AI agents provide verbose, narrowly focused feedback on code improvement and defect detection, whereas human reviewers contribute understanding and knowledge transfer.
  • Human involvement remains critical because conversations ending at AI agent states show consistently higher rejection rates.
  • Adopted agent suggestions introduce more code complexity than suggestions from human reviewers.
  • The authors propose improving agent context awareness, enabling project-knowledge sharing across agents, and reducing incorrect suggestions.
Loading 2603.15911v1…