Source-linked AI summary
On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub
Miku Watanabe, Hao Li, Yutaro Kashiwa, Brittany Reid, Hajimu Iida, Ahmed E. Hassan
TL;DR
Prior work had not examined how agentic coding affects software development processes and resulting artifacts. This study analyzes Claude Code-generated pull requests across open-source projects and finds they are widely adopted, while still benefiting from human oversight and refinement.
Problem
Prior studies focused mainly on LLM-based chatbots, leaving the practical implications of agentic coding for software processes and artifacts insufficiently investigated.
Method
The paper empirically analyzes 567 Claude Code-generated pull requests across 157 open-source projects and compares their purposes with human pull requests.
Results
83.8% of Agentic-PRs were accepted, compared with 91.0% of Human-PRs, while Agentic-PRs more often addressed refactoring, documentation, and testing.
Takeaways & Limitations
Agentic coding provides a strong starting point for real-world contributions, but human oversight helps ensure correctness, maintainability, and alignment with project conventions.
Takeaways & Limitations
Excluding open pull requests creates survival bias, so reported acceptance rates and merge times reflect only completed workflows and may underestimate ongoing contributions’ latency.
Abstract
from arXiv · showhide
Large language models (LLMs) are increasingly being integrated into software development processes. The ability to generate code and submit pull requests with minimal human intervention, through the use of autonomous AI agents, is poised to become a standard practice. However, little is known about the practical usefulness of these pull requests and the extent to which their contributions are accepted in real-world projects. In this paper, we empirically study 567 GitHub pull requests (PRs) generated using Claude Code, an agentic coding tool, across 157 diverse open-source projects. Our analysis reveals that developers tend to rely on agents for tasks such as refactoring, documentation, and testing. The results indicate that 83.8% of these agent-assisted PRs are eventually accepted and merged by project maintainers, with 54.9% of the merged PRs are integrated without further modification. The remaining 45.1% require additional changes benefit from human revisions, especially for bug fixes, documentation, and adherence to project-specific standards. These findings suggest that while agent-assisted PRs are largely acceptable, they still benefit from human oversight and refinement.
1 Introduction
Agentic coding lets autonomous AI agents plan, execute, test, and iterate on software tasks with minimal human intervention. This study examines how such Agentic-PRs differ from Human-PRs and how maintainers receive them.
- Agentic coding enables autonomous agents to plan, execute, test, and iterate on development tasks with minimal human intervention.
- The study addresses the previously unexamined impact of agentic coding on software processes and resulting artifacts in real-world projects.
- 83.8% of Agentic-PRs are accepted, compared with 91.0% of Human-PRs.
- Agentic-PRs emphasize tests, refactoring, and documentation, whereas Human-PRs handle project maintenance such as CI and chores.
- Agentic-PRs can address bugs and features while also introducing non-functional improvements and multiple purposes in one submission.
2 Motivating Example
Claude Code can modify code, run development tools, and generate pull requests with synthesized descriptions. The motivating example illustrates both the usefulness of these artifacts and their variable quality during review.
- Claude Code can edit files, repair bugs, search documentation, execute tests, and interact with version-control systems.
- A refactoring workflow has Claude analyze files, apply transformations such as extracting helper methods, and package the modifications into a pull request.
- Agentic-PR descriptions automatically summarize the change and identify its provenance as generated with Claude Code.
- Some Agentic-PRs are merged with little modification, while others receive substantial follow-up revisions or are closed without merging.
- The study therefore examines PR content, acceptance and rejection, revision extent, and the types of reviewer changes.
3 Data Collection
The researchers collected Claude Code pull requests from GitHub using explicit provenance markers and constructed a matched human comparison set. The resulting dataset supports comparisons within the same project and author context.
- Agentic-PRs were identified through the description marker “Generated with Claude Code” using the GitHub GraphQL API.
- The search covered PRs submitted between February 24, 2025 and April 30, 2025.
- Human-PRs were selected from similar periods, the same authors, and the same project repositories as Agentic-PRs.
- The study defines Agentic-PRs as PRs created with agentic coding assistance and Human-PRs as PRs created without such assistance.
4 Results
Agentic-PRs address similar core tasks as Human-PRs but are used more for testing, refactoring, documentation, and code refinement. They are usually accepted, although maintainers reject some for project evolution, complexity, procedural issues, and technical shortcomings.
- 31.0% of Agentic-PRs and 30.8% of Human-PRs address bug fixing, while feature development accounts for 26.8% and 27.6%, respectively.
- 24.9% of Agentic-PRs involve refactoring versus 14.9% of Human-PRs, often restructuring code without changing external behavior.
- 18.8% of Agentic-PRs improve testing versus 4.5% of Human-PRs, including targeted tests that raised one project’s coverage from 70% to 94%.
- Documentation updates occur in 22.1% of Agentic-PRs versus 14.0% of Human-PRs, while maintenance and configuration tasks occur at similar rates, 20.7% versus 21.2%.
- 39.9% of Agentic-PRs pursue multiple objectives versus 12.2% of Human-PRs, commonly combining features, refactoring, or bug fixes with testing.
- 83.8% of Agentic-PRs were merged versus 91.0% of Human-PRs; rejected Agentic-PRs commonly reflected alternative solutions, obsolescence, oversized changes, verification-only submissions, conflicts, or technical shortcomings.
5 Implications
The implications emphasize measuring hidden review costs, improving agent transparency and project alignment, and structuring agent-assisted work for easier review. They also call for workflow-level benchmarks and instrumentation rather than evaluating code generation in isolation.
- Implications for researchers: 52.4% of Agentic-PR revisions involved documentation drift, style adherence, or reviewer negotiation, exposing gaps in code-centric benchmarks.The paper recommends PR-centric benchmarks that reward alignment across code, documentation, and review resolution.
- Implications for researchers: 34.1% of revision commits were co-authored by Claude Code, supporting full-trace studies of human-AI development workflows.Review threads also referenced GitHub Copilot and other agents.
- Implications for developers: Nearly 40% of Agentic-PRs combined multiple tasks, while “too large” ranked among the top three rejection reasons.The paper recommends small, self-contained submissions or sequences of smaller follow-up PRs.
- Implications for developers: Style mismatches accounted for 23.4% of revisions and refactoring for 27.1%, motivating project-specific instructions for agents.The recommended guidance should include formatting rules, design principles, and architectural constraints.
- Implications for coding agent builders: Rejected PRs often lacked implementation rationale, leaving reviewers to infer design decisions and increasing review friction.The paper recommends uncertainty presentation and review scaffolding alongside generated code.
- Implications for coding agent builders: 1.1% of rejections involved conflict resolution, motivating automation for rebases, conflict resolution, and stale-response handling.These rejections often reflected timing misalignments as target branches evolved.
- Implications for coding agent builders: 23.4% style and 27.1% refactoring revisions indicate generic models do not fully capture project-specific conventions.The paper recommends additional tools to improve adherence to local standards.
6 Related Work
Related work has studied LLM assistance across software-engineering tasks, reviews, code changes, and multi-agent collaboration. However, the paper identifies a lack of empirical research on agentic coding tools and their resulting artifacts in real-world development.
- LLM-assisted software engineering: Prior studies examined LLM assistance for requirements documentation, code search, code generation, issue reports, and pull-request activities.Examples include test generation, PR-title generation, issue-report clarification, and review assistance.
- Human-AI collaboration: Human oversight remains necessary because AI-assisted development can affect result quality, knowledge sharing, and collective accountability.The cited literature describes persistent knowledge gaps in multidisciplinary teams and disruptions to accountability mechanisms in reviews.
- Multi-agent systems: Multi-agent frameworks assign specialized planner, coder, and tester roles to cooperating LLMs, while systematic reviews organize collaboration scenarios and prompting techniques.These approaches emulate Agile practices and aim to improve robustness.
- LLM-based chatbots: Earlier research found that 30.7% of AI-generated code-review suggestions were met with skepticism, despite ChatGPT reducing effort in review-intensive PRs.These studies focused on chatbots rather than autonomous agentic coding tools.
- Research gap: The paper distinguishes its contribution by investigating agentic coding tools and their effects on software-development processes and resulting artifacts in real-world contexts.Prior work primarily centered on LLM-based chatbots such as ChatGPT.
- AI-generated code: Research on generated code has addressed correctness, efficiency, security, error patterns, technical debt, iterative repair, and autonomous bug-fixing plans.Findings include weak correspondence between functional correctness and efficiency and the need for repair or domain-specific prompting.
7 Threats to Validity
The study’s validity is constrained by uncertain human-agent provenance, temporal differences between samples, completed-PR selection, and limited measurement of long-term code quality and generalizability.
- Internal validity: The Claude Code labeling scheme may overlook hybrid workflows in which developers modify generated code or commit changes themselves.The authors call for finer-grained provenance tracking to separate agent-generated code from human modifications.
- Internal validity: 42.0% of Human-PRs were created in the exact Agentic-PR timeframe, while the remaining baseline required an approximately 10-month backward sampling window.A sensitivity analysis found consistent task-type distributions in the time-aligned subset.
- Construct validity: The dataset excludes open PRs, creating survival bias because unresolved complex or controversial contributions are not captured.Acceptance rates and merge times therefore reflect completed workflows and may underestimate latency for undecided contributions.
- Construct validity: The study does not isolate agentic capabilities from Model Context Protocol integration, so their individual contributions to outcomes remain unresolved.The authors identify disentangling these impacts as future work.
- Construct validity: PR acceptance does not measure long-term code quality or maintainability because latent bugs, regressions, and later reverts may remain undetected.The study lacks longitudinal data on downstream stability.
- External validity: The sample contains 567 Claude Code PRs collected soon after launch, so observed behavior may reflect an enthusiastic experimentation phase.The authors call for larger datasets collected over longer periods.
- External validity: Because the analysis focuses exclusively on Claude Code, its findings may not generalize to other agentic coding tools.Replication across tools such as GitHub Copilot is identified as future research.
8 Conclusion
This first empirical study of Claude Code contributions analyzes 567 pull requests across 157 open-source projects. Agentic pull requests are widely adopted, but human oversight remains important for correctness, maintainability, and alignment with project conventions.
- 567 pull requests across 157 open-source projects were analyzed to examine how Claude Code contributions are received by developers.
- 83.8% of Agentic-PRs were accepted, compared with 91.0% of Human-PRs, yet Agentic-PRs were still widely adopted in real-world projects.
- When revisions were required, Agentic-PRs and Human-PRs did not differ significantly in the extent of modifications needed for integration.
- Agentic coding provides a strong starting point that benefits from human oversight for correctness, maintainability, and alignment with project conventions.
- Developers can reduce review friction by keeping pull requests small and encoding project-specific rules and architecture guidance for agents.