Source-linked AI summary
How AI Coding Agents Modify Code: A Large-Scale Study of GitHub Pull Requests
Daniel Ogenrwot, John Businge
TL;DR
The paper addresses limited empirical evidence about how AI-generated pull requests differ from human contributions in code modification and communication. It compares merged Agentic and Human pull requests using structural metrics and lexical and semantic description-to-diff measures, finding that Agentic PRs are structurally distinct yet generally coherent in describing their edits.
Problem
Empirical evidence is limited on how AI-generated pull requests differ from human-authored ones in code modifications and description-to-diff alignment.
Method
The study compares merged Agentic and Human pull requests using structural metrics and lexical and semantic similarity between pull-request descriptions and code diffs.
Results
Agentic PRs differ most clearly in commit structure and modified-file breadth, while both PR types show high semantic alignment and Agentic PRs exhibit slightly stronger consistency.
Takeaways & Limitations
AI-generated pull requests are structurally distinct yet generally coherent in how they describe their edits.
Takeaways & Limitations
The findings depend on complete AIDev data and reconstructed Human-PR commit data, while missing or truncated patches may bias distributions and similarity metrics.
Abstract
from arXiv · showhide
AI coding agents are increasingly acting as autonomous contributors by generating and submitting pull requests (PRs). However, we lack empirical evidence on how these agent-generated PRs differ from human contributions, particularly in how they modify code and describe their changes. Understanding these differences is essential for assessing their reliability and impact on development workflows. Using the MSR 2026 Mining Challenge version of the AIDev dataset, we analyze 24,014 merged Agentic PRs (440,295 commits) and 5,081 merged Human PRs (23,242 commits). We examine additions, deletions, commits, and files touched, and evaluate the consistency between PR descriptions and their diffs using lexical and semantic similarity. Agentic PRs differ substantially from Human PRs in commit count (Cliff's $δ= 0.5429$) and show moderate differences in files touched and deleted lines. They also exhibit slightly higher description-to-diff similarity across all measures. These findings provide a large-scale empirical characterization of how AI coding agents contribute to open source development.
1 Introduction
AI coding agents are becoming autonomous contributors, but empirical evidence remains limited on how their pull requests differ from human-authored contributions. This study compares their code modifications and description-to-diff alignment to assess reliability and communication in development workflows.
- AI coding agents can autonomously generate code, fix bugs, and submit pull requests.
- Empirical evidence remains scarce on how AI-generated pull requests modify source code and describe associated edits in real-world repositories.
- The study compares Agentic and Human pull requests using structural metrics including additions, deletions, files touched, and commit count.
- It also evaluates whether pull-request descriptions align with their underlying code edits through lexical and semantic similarity.
- These questions address how AI agents behave as contributors and communicate intent during software-engineering workflows.
2 Method
The study combines AIDev data with reconstructed Human-PR commit details, filters merged pull requests with valid patches, and computes structural and description-to-diff similarity metrics. Similarity analysis uses lexical baselines and embedding-based semantic measures, with score distributions guiding interpretation.
- The workflow comprises dataset collection, Human-PR commit-data extension, merged-PR filtering, and structural and similarity analysis.
- Human pull requests were enriched through GitHub API retrieval of commits, modified files, additions, deletions, and unified diffs.
- Files touched are counted as unique modified paths across commits, while GitHub-reported renames count as single-file modifications.
- Lexical alignment uses TF–IDF cosine similarity and Okapi BM25 after concatenating and normalizing pull-request titles and descriptions.
- Semantic alignment uses cosine similarity between CodeBERT and GraphCodeBERT embeddings of descriptions and cleaned diffs.
- TF–IDF and embedding cosine scores are bounded indicators, whereas Okapi BM25 is unbounded and should be interpreted as a relative lexical signal.
3 Results ?
Agentic and Human PRs differ most in how changes are structured across commits and files, while agent-generated descriptions show slightly stronger alignment with their diffs. Agentic edits are generally smaller and more localized, although variation differs substantially across tools.
- Structural changes: Human PRs exhibit larger and more variable additions and deletions, while Agentic PRs tend toward smaller, localized edits.Claude Code and OpenAI Codex show broader addition variability; Devin, Cursor, and Copilot produce more consistently small changes.
- Structural changes: Human PRs touch more files and often involve more commits, whereas Agentic tools range from compact single-focus contributions to broader edits.Claude Code and OpenAI Codex have wider distributions, while Devin, Cursor, and Copilot show narrower distributions.
- Structural changes: δ=0.54 marks the largest practical difference in commit count between Agentic and Human PRs.Medium effects also occur for files touched and deletions, whereas additions and total line changes show only small effects.
- Description-to-diff alignment: Lexical similarity remains near zero for both groups, while semantic similarity forms tight peaks between 0.9 and 1.0.The distributions therefore separate surface vocabulary overlap from similarity in the meaning captured by descriptions and patches.
- Description-to-diff alignment: Agentic PRs show a slight right shift and slightly higher central tendency across all four similarity metrics than Human PRs.Differences are modest, with the clearest differences appearing in semantic measures and Agentic scores showing tighter clustering.
4 Implications
Agentic PRs may warrant structural review cues beyond lines changed, while their descriptions generally align coherently with their edits. Commit count, files touched, and unusually low alignment could support triage, but links to concrete risks remain unresolved.
- Structural implications: Commit count and files touched may serve as early indicators of Agentic PR scope beyond LOC-based measures.Deletion-heavy or wide-scope PRs may warrant closer review, although concrete risks remain to be established.
- Structural implications: Agentic PR behavior varies across agent types, making agent identity and task context relevant to studying review effort, latency, or defect likelihood.
- Description–diff alignment: High description–diff consistency may support release-note generation, change-log construction, reviewer routing, and checks for unusually low alignment.
5 Related Work
Prior AI-assisted programming research largely examines snippet-level tools and developer interactions, while human-patch studies address structure and review-related outcomes. Large-scale PR-level analyses of autonomous AI contributions remain limited, and the AIDev dataset enables such analysis.
- AI-assisted programming: Research on Codex and GitHub Copilot primarily studies code completion, developer productivity, and interaction patterns rather than project-scale pull requests.
- Patch studies: Human-authored patch studies examine patch size, structure, review effort, maintainability, and defect risk, whereas empirical analyses of AI-generated patches remain limited.
- Dataset opportunity: The AIDev dataset enables large-scale, pull-request-level analysis of AI-generated contributions.
6 Threats to Validity
The findings are constrained by dataset completeness, reconstructed Human-PR histories, metric validity, and the open-source GitHub settings represented in AIDev. These limitations affect patch completeness, interpretation, and generalization.
- Internal validity: Missing or truncated patches and gaps in reconstructed Human-PR histories may bias distributions and similarity metrics.Potential causes include API rate limits, deleted repositories, rewritten histories, and truncation of very large diffs.
- Construct validity: Structural metrics approximate PR scope without capturing intent or correctness, while similarity metrics measure description–diff alignment rather than code quality or reviewer understanding.
- Construct validity: Okapi BM25 is unbounded and tokenization-sensitive, so it should be interpreted only as a relative lexical indicator.
- External validity: The findings generalize to open-source GitHub projects and agent versions represented in AIDev, but may differ in private or industrial settings or as agents evolve.
7 Conclusion
This study finds that Agentic PRs are structurally distinct from Human PRs while remaining generally coherent in how they describe their edits. It provides a large-scale empirical comparison and identifies future questions about review effort, defect risk, and maintainability.
- Conclusion: Agentic PRs differ most clearly in commit structure and the breadth of modified files, while differences in added or changed lines are relatively small.
- Conclusion: Both PR types show high semantic alignment between descriptions and code, with Agentic PRs exhibiting slightly stronger consistency.
- Conclusion: The study provides a large-scale empirical comparison of Agentic and Human PRs in the AIDev dataset.
- Conclusion: Future work should examine how these patterns relate to review effort, defect risk, and maintainability as agents and workflows evolve.