Source-linked AI summary
Where Do AI Coding Agents Fail? An Empirical Study of Failed Agentic Pull Requests in GitHub
Ramtin Ehsani, Sakshi Pathak, Shriya Rawal, Abdullah Al Mujahid, Mia Mohammad Imran, Preetha Chatterjee
TL;DR
As AI coding agents increasingly submit pull requests, evidence is limited on how their contributions behave and why many are not merged. This paper studies over 33k agent-authored PRs quantitatively and analyzes 600 rejected PRs qualitatively, finding that success varies by task and that failures combine technical, workflow, and socio-technical problems.
Problem
Evidence is limited on how agent-authored pull requests behave in real repositories and why many fail to be merged.
Method
The study analyzes over 33k pull requests from five coding agents across GitHub and qualitatively examines 600 rejected PRs to derive rejection patterns.
Results
Documentation, CI, and build tasks have the highest success, while performance and fix tasks have the lowest; not-merged PRs are larger, touch more files, and more often fail CI.
Takeaways & Limitations
Improving agentic workflows requires better task and duplicate detection, adherence to contribution norms, localized changes, and CI validation before opening PRs.
Abstract
from arXiv · showhide
AI coding agents are now submitting pull requests (PRs) to software projects, acting not just as assistants but as autonomous contributors. As these agentic contributions are rapidly increasing across real repositories, little is known about how they behave in practice and why many of them fail to be merged. In this paper, we conduct a large-scale study of 33k agent-authored PRs made by five coding agents across GitHub. (RQ1) We first quantitatively characterize merged and not-merged PRs along four broad dimensions: 1) merge outcomes across task types, 2) code changes, 3) CI build results, and 4) review dynamics. We observe that tasks related to documentation, CI, and build update achieve the highest merge success, whereas performance and bug-fix tasks perform the worst. Not-merged PRs tend to involve larger code changes, touch more files, and often do not pass the project's CI/CD pipeline validation. (RQ2) To further investigate why some agentic PRs are not merged, we qualitatively analyze 600 PRs to derive a hierarchical taxonomy of rejection patterns. This analysis complements the quantitative findings in RQ1 by uncovering rejection reasons not captured by quantitative metrics, including lack of meaningful reviewer engagement, duplicate PRs, unwanted feature implementations, and agent misalignment. Together, our findings highlight key socio-technical and human-AI collaboration factors that are critical to improving the success of future agentic workflows.
1 Introduction
This study examines how agent-authored pull requests are evaluated in real repositories and where they fail. It combines large-scale characterization with analysis of rejection patterns spanning technical, workflow, and reviewer factors.
- Agent-authored pull requests increasingly participate in software development beyond inline suggestions, including generating changes, responding to reviews, and contributing autonomously.
- Existing evidence links pull-request acceptance to correctness, scope, contributor reputation, tests, CI, and localized changes, but their relevance to agent-authored PRs remains unclear.
- The study analyzes over 33k pull requests from five coding agents across GitHub projects with more than 100 stars.
- Documentation, CI, and build-update contributions have higher merge rates, whereas performance and bug-fix contributions have the lowest acceptance.
- Not-merged pull requests tend to involve larger changes, more modified files, more reviewer revisions, and frequent CI-check failures.
- Agentic failures reflect misalignment with repository workflows, developer expectations, and project coordination, including CI failures, unwanted features, and reviewer abandonment.
2 Methodology
The methodology quantitatively compares merged and not-merged agentic pull requests across task, code, CI, and review dimensions, then qualitatively derives a taxonomy of rejection patterns. It uses effect sizes and iterative manual coding to characterize both measurable differences and human-centered failure modes.
- The quantitative analysis compares merged and not-merged pull requests across task types, code changes, CI outcomes, and review dynamics.
- Task labels cover 11 categories, while code-change magnitude is measured by added and removed lines and the number of modified files.
- Because the dataset exceeds 33k pull requests, the study uses Cliff’s delta rather than relying on statistical significance alone.
- Kernel density estimates complement effect sizes by visualizing distribution shape, spread, and shifts between merged and not-merged pull requests.
- Two authors independently open-coded an initial sample of 100 rejected pull requests and iteratively developed rejection patterns while assessing inter-rater reliability with Cohen’s kappa.
- The resulting hierarchical taxonomy spans reviewer-level abandonment, pull-request integration problems, code-level failures, and agentic failures.
3 Results
Across 33,596 agentic PRs, merge success varies by task type and agent, while not-merged PRs generally involve larger changes, more CI failures, and more review activity. Qualitative analysis of rejected PRs identifies reviewer abandonment, duplicate or unwanted submissions, CI/test failures, and agent misalignment as major rejection patterns.
- Merge outcomes: 71.48% of agentic PRs were merged, with Codex achieving the highest agent-level merge rate at 82.59% and Copilot the lowest at 43.04%.Codex contributed 18,004 merged PRs, while Copilot contributed 2,139.
- Code changes: Not-merged PRs introduced larger changes, with Cliff’s δ differences of 17% for total LOC changes and 10% for changed files.Their distributions were shifted toward larger changes on log10-scale density plots.
- CI and review dynamics: Not-merged PRs experienced more CI failures, with Cliff’s δ of 24% and each additional failed CI check associated with approximately 15% lower merge odds.Merged PRs clustered near zero failures, whereas not-merged PRs had a heavier tail of multiple failing checks.
- Merge outcomes: Documentation, CI, and build tasks had the highest merge success, whereas performance and fix tasks had the lowest.Codex’s task-specific rates exceeded 80% for documentation, CI, and build, while its lowest category was performance at 0.68.
- Rejection patterns: Among 562 categorized rejected PRs, reviewer abandonment was most frequent at 228 PRs (38%), followed by pull-request-level reasons at 188 (31%) and code-level reasons at 133 (22%).Duplicate PRs accounted for 142 cases (23%), while CI/test failures accounted for 99 (17%).
- Rejection patterns: The study’s rejection taxonomy also identifies unwanted features, incomplete or incorrect implementations, agent misalignment, and licensing issues.Agentic-level issues were least frequent, comprising 13 PRs (2%), with misalignment appearing in 9 PRs (1%).
4 Conclusion
Not-merged agentic PRs tend to involve broader, more invasive changes and higher CI/test failure rates, while rejections also reflect reviewer abandonment, duplication, and unwanted features. The findings motivate workflows that improve task selection, repository-context alignment, localized changes, and CI validation.
- Not-merged PRs tend to introduce larger, more invasive code changes, broader feature additions, and higher CI/test failure rates.
- Rejections arise from reviewer abandonment, duplicate PRs, unwanted features, and other failures of task selection, coordination, and repository alignment.
- Improving agentic workflows requires identifying existing work, following contribution norms, decomposing tasks into localized changes, and validating submissions against CI pipelines.
- The study provides empirical grounding for context-aware and collaboration-sensitive coding agents integrated into real-world software development workflows.