PaperlayerView exact revision on arXiv

Source-linked AI summary

AI-to-AI Code Reviews of GitHub Pull Requests

Niruthiha Selvanayagam, Taher A. Ghaleb

arXiv:2608.21311v1cs.SE

TL;DR

The paper addresses limited evidence about AI reviewers evaluating AI-authored GitHub pull requests and whether behavior varies by author–reviewer setup. It constructs a large-scale dataset by linking attributed pull requests with attributed review events, finding that closed-loop review is growing rapidly but remains a minority of identified agent activity and varies across configurations.

  • Problem

    Prior work largely studied AI coding tools and review systems in isolation, leaving product-resolved AI author–AI reviewer pairings in public GitHub repositories insufficiently characterized.

  • Method

    The paper applies signature-based attribution to CodAGE GitHub events and aggregates AI-authored pull requests with AI-reviewer events into cross-product and same-product datasets.

  • Results

    248,641 AI-attributed pull requests received at least one AI review; cross-product review increased by over two orders of magnitude from 2025-Q1 to 2025-Q3, while reviewer output varied across configurations.

  • Takeaways & Limitations

    Closed-loop AI review is already substantial enough to require explicit accounting for AI-generated artifacts in empirical software-engineering studies, despite remaining a minority of agent activity.

  • Takeaways & Limitations

    Signature-based attribution undercounts agents with missing or uncatalogued signatures, and observational associations plus unequal timestamp availability constrain comparisons.

Abstract

from arXiv · show

AI coding agents are increasingly integrated into software development workflows, operating on both sides of the pull-request (PR) process: AI authoring agents create or modify PRs, while AI reviewers evaluate them. This creates a closed loop in which one AI coding agent reviews a contribution attributed to another. We construct a large-scale dataset of AI-to-AI code review by linking AI-attributed PRs with AI-attributed review events from CodAGE, a public dataset of coding-agent-generated GitHub events. Our dataset contains 248,641 unique AI-attributed PRs that received at least one AI-attributed review. Of these, 45,269 received cross-product review and 208,145 received same-product review; 4,773 PRs received both. Cross-product AI-to-AI review occurred in approximately 1.6% of identified agent-authored PRs but was substantial in absolute terms, and its volume increased by more than two orders of magnitude from 2025-Q1 to 2025-Q3. Reviewer output varied across author-reviewer configurations. CodeRabbit labeled 35.0% of its comments on Claude Code-authored PRs as refactor comments, compared with 10.5% on Copilot-authored PRs, although this difference may reflect characteristics of the PRs rather than the reviewer. For three of four dual-role reviewers, mean comments per PR were 58-65% higher in the same-product group, although effect sizes were small or negligible and the difference was concentrated in the upper tail. Among pairs with complete, nonnegative timestamps, the observed median latency was 1.2 minutes for cross-product pairs and 4.7 minutes for same-product pairs; differential timestamp availability and reviewer composition limit this comparison. Overall, closed-loop AI-to-AI review is increasing but remains a minority of identified agent activity, with review output varying across authoring-agent groups and product configurations.

1 Introduction

The paper frames closed-loop AI review as AI occupying both sides of GitHub pull requests and studies its prevalence and reviewer behavior across author–reviewer configurations.

  • Closed-loop AI review means an AI agent contributes to a pull request and one or more AI agents review it.The observable definition does not imply that human review was absent.
  • The paper asks how prevalent closed-loop review is and how reviewer behavior varies with the author–reviewer setup.These questions address growth over time and whether authoring agents influence reviewer output.
  • Prior work largely studied coding tools, reviewer bots, or attribution separately rather than product-resolved AI author–AI reviewer pairings.This paper addresses that gap with a large-scale empirical study of AI-to-AI code reviews.
  • 248,641 pull requests received at least one AI review, including 45,269 cross-product and 208,145 same-product reviews.The dataset is the paper’s large-scale empirical foundation.
  • The study tracks closed-loop review patterns over time and analyzes comment categories, comment volume, and time to first review across author–reviewer setups.These analyses operationalize the paper’s prevalence and reviewer-behavior questions.

2 Background and Related Work

Earlier research examined AI coding tools, reviewer bots, and automated attribution largely in isolation. This paper focuses on deployed, product-resolved pairings between AI authors and AI reviewers in public GitHub repositories.

  • Prior studies emphasized code-suggestion productivity, usability, pair programming, and autonomous agent-authored pull requests.This literature characterizes AI assistance and agent-authored contributions but not their combined review loop.
  • Reviewer-bot research studied bot adoption, developer interaction, industrial LLM review, and multi-agent review pipelines.These strands examine reviewer systems or workflows without resolving the full author–reviewer product pairing at scale.
  • Existing attribution methods identify automated contributions or infer originating agents but generally do not identify the corresponding AI product or vendor.The paper addresses this gap with signature-based product attribution.
  • The paper studies how often AI reviewers comment on AI-authored pull requests, which agents participate, and how behavior varies by authoring agent and configuration.It positions deployed GitHub interactions as a less-studied software-engineering setting for multi-agent evaluation.

3 Data and Methodology

The study builds AI-to-AI review datasets from attributed GitHub events using signature-based identification, cleaning, aggregation, and deterministic comment classification. Its methodology explicitly bounds interpretation through attribution, coverage, and dataset-construction caveats.

  • Data source: CodAGE events from GHArchive provide the source for identifying AI-authored pull requests and AI-attributed reviews.The study uses CodAGE-PRs, CodAGE-Reviews, and CodAGE-ReviewComments from a snapshot covering 2024-01-01 to 2026-04-15.
  • AI attribution: A two-tier signature framework combines body signatures with vendor-controlled login signatures, while branch-name prefixes are never sufficient alone.S1 agents require body-level evidence; S2 agents accept body or login evidence because logins and quoted text can mislead attribution.
  • Cleaning and coverage: 38.0% of 4,563,819 candidate author PRs were quarantined, with 96.9% quarantined as branch_only, whereas quarantine affected 0.01% of review events and comments.The asymmetry means the analyzed agent-authored population contains only candidates meeting body or login signature criteria.
  • Cleaning and coverage: Relabeling uses signature attribution when it disagrees with provisional labels, but this comparison is an internal consistency check rather than external validation.The largest relabel consolidates OpenAI_Codex_Cloud and the CLI Codex trailer into OpenAI_Codex.
  • Dataset construction: The analysis aggregates events by PR and reviewer into cross-product, same-product, and no-detected-review datasets, while allowing overlap between the two reviewed datasets.The PR key is (repo_name, pr_number), and absence of an attributed reviewer signature does not prove that no AI reviewed the PR.
  • Comment classification: CodeRabbit categories are extracted deterministically from explicit comment headers using ordered regular expressions, without treating categories as an ordered severity scale.The classifier emits substantive and structural labels; unmatched comments remain unlabeled, and the first matching rule wins.

4 RQ1: Prevalence and Composition of Closed-Loop AI-to-AI Code Reviews

Closed-loop AI review was substantial but remained a minority of identified agent-authored PR activity, with cross-product and same-product review growing sharply and differing by authoring agent.

  • Volume and growth: 248,641 agent-authored PRs received at least one AI review, including 45,269 cross-product and 208,145 same-product PRs.The counts are lower bounds, and the dataset aggregates review activity at the PR level.
  • Volume and growth: AI review activity increased by over two orders of magnitude from 2025-Q1 to 2025-Q3 for both cross-product and same-product groups.Figure 1 reports quarterly counts on a log-scaled y-axis.
  • Who reviews whom?: OpenAI Codex dominated cross-product authorship with 31,601 PRs, while Copilot was the dominant reviewer with 21,022 author–reviewer pairs.The modal cross-product pair was Codex-authored and Copilot-reviewed, with 18,114 pairs.
  • Same- vs. cross-product composition: Same- versus cross-product review shares were sharply non-uniform: Copilot, Amazon Q, and Devin were mostly reviewed internally, whereas Cursor, Google Jules, and Claude Code were reviewed almost entirely across products.OpenAI Codex was the only author with substantial populations in both datasets, with 54.1% same-product review.
  • Volume and growth: Cross-product review comprised 1.6% of identified agent-authored PRs and spanned 10,345 repositories.This percentage is conditioned on signature attribution.

5 RQ2: Reviewer Behavior Across Authors and Product Pairings

Reviewer behavior varies with both the authoring agent and the author–reviewer product pairing. CodeRabbit’s category mix differs across authoring agents, same-product review often has higher mean comment volume, and observed latency differences are constrained by composition and timestamp availability.

  • CodeRabbit comment categories by authoring agent: CodeRabbit’s comment-category distributions differed across authoring agents, with refactor shares ranging from approximately 10% to 35%.The analysis held CodeRabbit fixed while varying the authoring agent.
  • CodeRabbit comment categories by authoring agent: 35.0% of CodeRabbit comments on Claude Code-authored PRs were refactor comments, compared with 10.5% on Copilot-authored PRs.The difference was 24.5 percentage points, with 95% CI [23.1, 25.9].
  • Same- vs. cross-product review volume: Three of four reviewer bots produced 58–65% more comments per PR on same-product than cross-product PRs on average.The distributions overlapped heavily, and the gap was concentrated in the upper tail; OpenAI Codex was the exception.
  • Same- vs. cross-product review volume: Observed mean comment gaps were not population-wide shifts: Cliff’s δ was small-to-negligible across all four bots.The comparisons used per-PR comment counts for Copilot, OpenAI Codex, Devin, and Amazon Q.
  • Latency from PR open to first AI review: Among pairs with complete, nonnegative timestamps, median first-review latency was 1.2 minutes for cross-product pairs and 4.7 minutes for same-product pairs.The comparison is limited by differential timestamp availability and reviewer composition.

6 Discussion

Closed-loop AI review is frequent enough in absolute terms to affect software-engineering sampling assumptions, although AI-authored and AI-reviewed pull requests remain a minority of public GitHub activity. Reviewer output and same- versus cross-product patterns vary across authoring agents, reviewer composition, and product configurations, with causal interpretations unsupported.

  • Scope and implications: AI-authored and AI-reviewed pull requests remain a minority of public GitHub activity but are frequent enough to challenge purely human sampling assumptions.Studies may mix fundamentally different populations unless they account for AI-generated artifacts.
  • Reviewer-output variation: With CodeRabbit held constant, comment-category distributions differ across authoring-agent groups.The analysis cannot separate effects of PR characteristics, repository context, and reviewer behavior.
  • Product-pairing interpretation: Same- and cross-product gaps may reflect reviewer composition, product integration, or triggering rather than review strictness or reviewer preference.Product boundaries also need not correspond to model boundaries.

7 Threats to Validity

The study’s validity is constrained by attribution coverage, observational confounding, incomplete timestamps, and limited generalizability beyond attributable public GitHub activity. Accordingly, absolute counts are lower bounds, and latency differences are descriptive rather than causal.

  • Construct validity: Signature-based attribution can miss AI activity and misclassify copied or shared signatures, making absolute counts lower bounds.Under-attribution is greater for authors than reviewers because author attribution depends on trailers that agents may omit and squash merges may strip.
  • Construct validity: Observed rates per agent-authored PR describe the attributable population rather than all AI-authored PRs.The population excludes activity whose signatures are missing or removed.
  • Internal validity: The study reports observational associations, while pull-request size, language, repositories, and product integrations may confound the observed patterns.These factors may influence reviewer behavior and other comparisons.
  • Internal validity: 79.2% of cross-product and 31.9% of same-product PR–reviewer pairs were retained for latency analysis, so complete-case estimates may not represent either full population.The authors therefore interpret latency differences descriptively and do not attribute them to product pairing.
  • External validity: The dataset covers public CodAGE GitHub events from 2024-01-01 to 2026-04-15, excluding private repositories, GitHub Enterprise, and other version-control platforms.Results may not generalize to non-adopting projects.

8 Conclusion and Future Work

The paper concludes that AI review of AI-authored GitHub pull requests remains a minority of agent activity but grew rapidly and spans substantial cross-product interaction. It calls for validated content assessment, model-level experiments, security benchmarks, and richer provenance over time.

  • Conclusion: Closed-loop review remains a minority of agent activity but grew rapidly over 2025 and spans tens of thousands of cross-product interactions.This conclusion concerns identified AI-authored and AI-reviewed public GitHub pull requests.
  • Conclusion: Reviewer output differs across authoring-agent groups and author–reviewer configurations in comment-category mix and per-PR comment volume.The paper identifies these differences as empirical variation, not validated differences in review quality.
  • Future work: CodeRabbit’s categories describe its output but are not validated measures of correctness, severity, or usefulness.Future evaluation should use human annotation of correctness, localization, severity, and actionability with inter-rater agreement.
  • Future work: Product-level attribution cannot distinguish same- and different-model configurations because public records rarely reveal the underlying model.Controlled experiments with known models, fixed configurations, source blinding, and functionally equivalent pull requests are proposed.
  • Future work: AI authors and reviewers may share failure modes, but the dataset cannot assess correlated blind spots because model identity and ground-truth defects are unavailable.A controlled benchmark could inject known functional defects and security vulnerabilities and compare reviewer types.
  • Future work: Structured provenance should record products, model versions, review triggers, configurations, and human involvement for longitudinal evaluation.Repeated measurement could assess whether workflows improve software correctness, security, and maintainability.

Data Availability

The replication package and source CodAGE dataset are publicly available, enabling reproduction of the paper’s reported counts, tables, and figures.

  • Artifacts: The replication package includes the agent signature registry, data-processing scripts, classifier, and analysis code.It is available at the paper’s GitHub repository.
  • Artifacts: CodAGE is available on Hugging Face under CC BY 4.0, and the artifacts reproduce the reported counts, tables, and figures.The passage provides links to both the replication package and source dataset.
Loading 2608.21311v1…