Source-linked AI summary
Not All Agents Are Equal: Code Quality and Post-Merge Maintenance Across Five Autonomous Coding Agents in the Wild
Obada Kraishan
TL;DR
Evidence about autonomous coding agents’ behavior in real projects after merge remains limited. This paper follows provenance-labeled agent and matched human PRs, measuring code patterns and post-merge outcomes. Its results show that vendor differences are substantial: Codex PRs were reverted at half the human rate, while Devin PRs were reverted at 1.3 times it.
Problem
Most evidence measures models in controlled tasks or generated snippets, leaving less field evidence about agent code in real projects and after merge.
Source: §1 · p. 1Method
The study combines AIDev provenance labels with GitHub enrichment and a matched human baseline to measure added-code quality and post-merge outcomes.
Source: §1 · p. 2, §3.1 · p. 2, §3.4 · p. 4Results
Vendor differences dominate: Codex PRs were reverted at half the human rate, while Devin PRs were reverted at 1.3 times it; pooled agents had fewer security smells than humans.
Source: §4.1 · p. 4, §5 · p. 7, §7 · p. 8Takeaways & Limitations
Treating agents as a single category averages away differences between vendors that matter to maintainers.
Source: §5 · p. 7Churn on the most-changed file is a proxy for whole-PR churn, and revert detection by commit message misses silent rewrites.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Autonomous coding agents now open pull requests in public repositories at a scale that was out of reach two years ago, yet little is known about what happens to that code after it lands. This paper studies 37,623 provenance-labeled pull requests (PRs) from five commercial agents (OpenAI Codex, Devin, GitHub Copilot, Cursor, and Claude Code) and a matched human baseline, drawn from 2,807 GitHub repositories between December 2024 and July 2025. We combine the AIDev dataset with 58,792 cached GitHub API responses to measure security smells in added code, structural maintainability, post-merge churn, revert rates, and human review behavior. Three results stand out. First, quality differences are vendor-specific rather than uniform: Codex-authored PRs were reverted about half as often as human PRs (6.1% vs. 11.5%, odds ratio 0.50), while Devin PRs were reverted more often (14.5%, odds ratio 1.31). Second, agent code pooled across vendors was less likely than human code to contain a security smell (odds ratio 0.63), driven by fewer hardcoded credentials and eval-style constructs. Third, review effort concentrates unevenly: Copilot PRs drew the most human reviews and change requests, and Claude Code PRs waited the longest for a first human review (median 12.6 hours). All pipeline code, statistical reports, and figures are released for replication.
1 Introduction
Research on AI-generated code has largely measured controlled tasks or isolated outputs, leaving limited evidence about autonomous agents’ code in real projects after merge. This study compares agent products with matched human PRs and makes vendor-specific differences visible through a reproducible measurement pipeline.
1 Introduction
- Existing benchmarks and lab studies say less about whether autonomous agents’ code fits real projects and survives after merge.Field evidence was thinner and mostly concerned Copilot in autocomplete form.Source: §1 · p. 1
- The study examines provenance-labeled agent and human PRs in the same repositories, following merged changes for 90 days.The comparison covers five dimensions, including code patterns, maintainability, post-merge churn, reverts, and human review.Source: §1 · pp. 1–2
- Vendor labels let the analysis distinguish pooled agent behavior from individual products, a distinction the authors say matters.The study is organized around five research questions on code patterns, maintainability, maintenance, vulnerability classes, and review behavior.Source: §1 · p. 2
- The paper contributes a reproducible pipeline, statistical evidence on agent-specific outcomes, and released materials for replication and extension.The artifact includes pipeline code, a 63-variable codebook, per-question reports, and figures.Source: §1 · p. 2
2 Related Work
Prior work evaluates code-generation models and agents, examines security in generated code, and mines AI-assisted development. This study extends that work toward repository outcomes after agent-authored code is accepted.
2.1 Evaluating code-generating models and agents
- Controlled benchmark and agent-framework studies score task success, while this study addresses how accepted agent code behaves in living projects afterward.Prior multi-agent failure analysis used annotated execution traces rather than repository outcomes.Source: §2.1 · p. 2
2.2 Security of AI-generated code
- Security studies have often examined suggested or freshly generated code, whereas this study analyzes full autonomous-agent PRs against human PRs in the same repositories.Prior work includes findings on vulnerable Copilot completions and partly conflicting user-study conclusions.Source: §2.2 · p. 2
2.3 Mining AI-assisted development at scale
- Repository-mining work measured Copilot productivity and churn, while AIDev made agent PRs observable at scale with per-vendor labels.This study adds follow-up maintenance signals, patch-level quality measures, and a matched human baseline to AIDev.Source: §2.3 · p. 2
3 Data and Method
The study assembles a matched agent–human PR corpus, enriches it with GitHub history, measures added-code patterns and follow-up activity, and applies non-parametric statistical tests. Figure 1 summarizes the pipeline from corpus construction through validation and analysis.
3 Data and Method
- Figure 1 outlines four stages: corpus construction, GitHub enrichment with a 90-day follow-up, patch and post-merge measurement, and validation and statistical analysis.Agent and language marks identify the five products and three analyzed languages.Source: §3 · p. 2, Figure 1 · p. 3
3.1 Corpus construction
- The corpus starts from AIDev’s agent PRs and a human sample matched within repositories and the agents’ activity window.The matched sample contains 4,027 human PRs alongside 9,750 agent PRs from the same 810 repositories; Table 1 summarizes corpus composition.Source: §3.1 · pp. 2–3, Table 1 · p. 3
3.2 GitHub enrichment
- GitHub enrichment retrieves changed files and post-merge commits touching each PR’s most-changed file during the 90-day window.Of 30,721 PRs submitted for enrichment, 29,730 (96.8%) succeeded; 991 (3.2%) referred to deleted or legally blocked repositories or PRs.Source: §3.2 · p. 3
3.3 Patch-level static analysis
- Patch analysis covers Python, JavaScript, and TypeScript diffs, using maintainability counts and regex-matched security smells mapped to eight CWE classes.The analysis includes 8,933 PRs and 1,348,822 added lines; pattern hits are interpreted as smells, not confirmed exploitability.Source: §3.3 · pp. 3–4
3.4 Review measures
- Review latency measures time to first human review, and review latency and depth use human reviewers only.AIDev’s review tables cover agent PRs, not the human sample, so RQ5 compares agents with one another and reports group coverage.Source: §3.4 · p. 4
3.5 Statistical design
- Because software metrics are heavy-tailed and zero-inflated, the analysis uses non-parametric tests and corrects related comparisons with Benjamini–Hochberg at a 0.05 false discovery rate.Group comparisons use Mann–Whitney U with Cliff’s δ; six-group omnibus comparisons use Kruskal–Wallis followed by Dunn’s pairwise tests, while binary outcomes use χ2 or Fisher’s exact tests with odds ratios and 95% confidence intervals.Source: §3.5 · p. 4
4 Results
Across the five research questions, outcomes vary by vendor rather than supporting a single judgment about agent code: pooled agents have fewer PRs with security smells, while maintenance and review patterns differ across products.
4.1 RQ1: Defect-prone patterns in added code
- Per-line security-smell density was similar across authorship, with all five agent–human effects negligible by conventional bands.Codex, Copilot, and Cursor were slightly below human levels, Devin was indistinguishable, and Claude Code was above them.Source: §4.1 · pp. 4–5
- 2.9% vs. 4.6% of PRs contained a security smell for pooled agents and humans, respectively (odds ratio 0.63, 95% CI [0.47, 0.85]).Cursor (1.6%) and Codex (2.5%) had the lowest rates; Claude Code was highest at 9.5%.Source: §4.1 · p. 4, Figure 2 · p. 5
4.2 RQ2: Maintainability across agents
- All four structural measures differed across groups (all p < .001), and no single agent dominated across them.Claude Code had the highest median nesting (4 levels vs. 3 elsewhere) and branch density (.063 per line); Codex, Devin, and human diffs had no median comment lines.Source: §4.2 · p. 5
4.3 RQ3: Post-merge maintenance overhead
- Claude Code had the lowest churn per changed line (δ = −.33 vs. humans, medium, p < .001; median ci = 0.5 vs. 5.3 for humans).Copilot (δ = −.16, small) and Cursor (δ = −.08) were also below humans; Codex and Devin were at the human level.Source: §4.3 · p. 5
- Codex PRs were reverted at about half the human rate, Devin PRs at about 1.3 times it, and the other three agents were statistically indistinguishable from humans.Across churn and reverts, none of the five agents imposed a measurably higher per-line maintenance burden than human contributors in the same repositories.Source: §4.3 · pp. 5–6, Figure 3 · p. 6
4.4 RQ4: Vulnerability classes
- Hardcoded credentials and eval/exec injection were under-represented in pooled agent code, with no CWE class over-represented after correction.Hardcoded credentials: 0.9% of agent PRs vs. 2.2% of human PRs (OR 0.39 [0.25, 0.62]); eval/exec injection: 0.2% vs. 0.8% (OR 0.25 [0.11, 0.56]).Source: Figure 4 · p. 6, §4.4 · p. 6
4.5 RQ5: Human review behavior
- Copilot PRs drew the most scrutiny, averaging 3.6 human reviews and 0.43 change requests; Claude Code waited longest for first human review (median 12.6 hours).Review records cover agent PRs only, from 5.4% of Codex PRs to 51.2% of Copilot PRs.Source: §4.5 · pp. 6–7, Figure 5 · p. 7
5 Discussion
Maintenance and quality outcomes vary by vendor, so pooled labels can obscure differences that matter to maintainers. Within the measured window, agent PRs do not show a general maintenance penalty, but size confounding and human review capacity remain important concerns.
5 Discussion
- On reverts, the best- and worst-performing vendors had odds ratios of 0.50 and 1.31 against the same baseline, a gap far wider than any pooled agent–human gap.Credential smells showed a fourfold spread across agents, and the same pattern held for review latency.Source: §5 · p. 7
- Within 90 days in repositories with more than 100 stars, agents had no higher measured per-line maintenance burden, and several outcomes were lower than for humans.The authors say conservative, template-like code and bounded, well-specified tasks may contribute, but their design cannot distinguish these explanations.Source: §5 · p. 7
- Claude Code’s higher smell prevalence, heavier structure, and slower first review co-occurred with PRs about eight times the median size of other groups.Per-line densities and size-stratified checks limit but do not remove this entanglement; per-agent task-mix data would be needed to address it.Source: §5 · p. 7
- Copilot PRs already consume several human reviews each, and bot reviews outnumber human reviews for some agents, making human attention a bottleneck as agent volume grows.The discussion calls for research on routing and prioritizing agent PRs alongside research on code generation.Source: §5 · p. 7
6 Threats to Validity
The study’s measures and comparisons have limits: smell detection identifies patterns rather than confirmed vulnerabilities, maintenance signals are proxies, and agent assignment is nonrandom. Its corpus also covers a bounded set of repositories, languages, and versions.
6 Threats to Validity
- Regex smell detection flags patterns rather than confirmed vulnerabilities, misses context-dependent flaws, and should be read as an upper bound on pattern presence—not exploitability.The catalog was validated on planted-defect patches, and a path-traversal detector was corrected after relative-import strings inflated that class roughly 150-fold.Source: §6 · p. 7
- Churn on the most-changed file is only a proxy for whole-PR churn, while commit-message revert detection misses silent rewrites.The proxy was chosen to keep API cost linear.Source: §6 · p. 8
- Because agents are not randomly assigned and repositories and users self-select agents and tasks, group differences mix code quality with task mix.A shared-repository, shared-window human baseline and size-bucket sensitivity analysis reduce but do not eliminate this confounding.Source: §6 · p. 8
- The corpus covers public repositories with more than 100 stars, three languages, and a December 2024–July 2025 window, limiting transfer to private codebases, other languages, or later agent versions.Codex contributes 58% of agent PRs, while Claude Code contributes 459 PRs.Source: §6 · p. 8
- Static-analysis coverage was 23.7% of PRs and review coverage ranged from 5–51% by agent group; analysis scripts regenerate results deterministically from a fixed seed.Source: §6 · p. 8
7 Conclusion
The study follows agent and matched human PRs through review, merge, and 90 days of maintenance. It finds fewer security smells in pooled agent code and no higher per-line maintenance burden, while the largest quality gaps are between vendors.
7 Conclusion
- Across 37,623 provenance-labeled PRs, pooled agents shipped fewer security smells than humans and none of the five imposed a higher per-line maintenance burden.Source: §7 · p. 8
- Codex PRs were reverted at half the human rate, while Devin PRs were reverted at 1.3 times the human rate.Source: §7 · p. 8
- The authors frame agent choice as a measurable engineering decision and release a pipeline for tracking outcomes as products evolve.Future work should add per-task context, extend follow-up beyond 90 days, and track whether review capacity keeps pace with agent volume.Source: §7 · p. 8