Source-linked AI summary
AIDev: Studying AI Coding Agents on GitHub
Hao Li, Haoxiang Zhang, Ahmed E. Hassan
TL;DR
Software-engineering research lacks comprehensive evidence on how coding agents are used in real-world projects. The paper introduces AIDev, a large-scale GitHub dataset of Agentic-PRs with metadata and enriched development artifacts, providing a foundation for studying adoption, quality, review, and risks.
Problem
Existing evidence on coding-agent use often comes from controlled studies, benchmarks, or small-scale deployments rather than real-world repositories.
Method
AIDev aggregates Agentic-PR data from GitHub and enriches a starred-repository subset with review comments, commit diffs, issues, timelines, and purpose annotations.
Results
932,791 Agentic-PRs from five agents cover 116,211 repositories and 72,189 developers, including 33,596 enriched PRs from 2,807 repositories.
Takeaways & Limitations
AIDev provides a foundation for studying coding-agent adoption, quality, review dynamics, and risks in real-world software engineering.
Takeaways & Limitations
Coding-agent use remains associated with security, latency, context-handling, and user-trust challenges, especially with proactive or repository-scale actions.
Abstract
from arXiv · showhide
AI coding agents are rapidly transforming software engineering by performing tasks such as feature development, debugging, and testing. Despite their growing impact, the research community lacks a comprehensive dataset capturing how these agents are used in real-world projects. To address this gap, we introduce AIDev, a large-scale dataset focused on agent-authored pull requests (Agentic-PRs) in real-world GitHub repositories. AIDev aggregates 932,791 Agentic-PRs produced by five agents: OpenAI Codex, Devin, GitHub Copilot, Cursor, and Claude Code. These PRs span 116,211 repositories and involve 72,189 developers. In addition, AIDev includes a curated subset of 33,596 Agentic-PRs from 2,807 repositories with over 100 stars, providing further information such as comments, reviews, commits, and related issues. This dataset offers a foundation for future research on AI adoption, developer productivity, and human-AI collaboration in the new era of software engineering. > AI Agent, Agentic AI, Coding Agent, Agentic Coding, Agentic Software Engineering, Agentic Engineering
1 High-Level Overview
AIDev frames Coding Agents as increasingly routine participants in collaborative software development and introduces a large-scale dataset of Agentic-PRs from real-world GitHub projects.
- Coding Agents increasingly act as AI Teammates in core development workflows, contributing thousands of pull requests daily.
- Figure 1’s workflow shows an agent generating a patch, submitting a PR, and addressing human review feedback in a follow-up commit.
- An Agentic-PR is a pull request authored by a Coding Agent.
- 932,791 Agentic-PRs from five agents span 116,211 repositories and involve 72,189 developers.The agents are OpenAI Codex, Devin, GitHub Copilot, Cursor, and Claude Code.
- 33,596 Agentic-PRs from 2,807 repositories with over 100 stars include comments, commit-level diffs, event timelines, and related issues.
2 Internal Structure
AIDev organizes Agentic-PR information across multiple relational levels and enriches a starred-repository subset with detailed development artifacts and automated annotations.
- AIDev provides pull-request, repository, and developer-level metadata.
- Repositories with more than 100 GitHub stars contribute inline review comments, commit-level diffs, linked issues, and full pull-request timelines.
- Automated annotations classify pull-request purpose using categories such as bug fix, feature, and documentation.The annotations follow Conventional Commits categories.
3 How to Access (Links)
AIDev is distributed through Hugging Face and Zenodo, with interactive exploration and example notebooks supporting reproducible dataset analysis.
- The AIDev dataset is available for download on Hugging Face and Zenodo.
- Hugging Face’s Data Studio supports in-browser SQL queries for interactive dataset exploration.
- Example Jupyter notebooks with Google Colab links demonstrate how to download, filter, and analyze the dataset.
4 Potential Research Questions
The proposed research questions use AIDev to examine adoption, code changes, testing, review, quality, security, and production outcomes of Agentic-PRs.
- Adoption and practices: Research questions examine who adopts Coding Agents and which repository, developer, and pull-request practices correlate with Agentic-PR quality.
- Code changes: Code-change questions compare Agentic-PRs with Human-PRs in changes, originality, reuse, convention conformance, review effort, and merge outcomes.
- Testing behavior: Testing questions ask how often agents contribute tests, which test types occur, and whether developers add testing through follow-up commits or related pull requests.
- Review dynamics: Review questions investigate which aspects receive attention, how often agents address comments, which comments are difficult, and how review gaps can guide human collaboration skills.
- Quality and security: Quality and risk questions target failure patterns, code-quality issues, early predictors of rejection or review effort, and security vulnerabilities in Agentic-PRs.
- Production outcomes: Outcome questions ask which Agentic-PRs reach production and persist, and how outcomes vary by language, repository maturity, and task type.
5 Related Work
Related work shows a transition toward increasingly autonomous, tool-using AI coding agents, while evidence about real-world adoption and productivity remains mixed. AIDev addresses the limited scale of existing evidence by capturing Agentic-PRs and associated artifacts from real GitHub repositories.
- AI Coding Agents: AI coding agents integrate environment awareness, long-horizon planning, and structured tool use beyond traditional code-completion models.Their interfaces expose file operations, build and test execution, code search, and version-control actions.
- Risks and Reliability: Security, usability, and trust remain concerns as agents receive write or commit permissions in real repositories.Reported challenges include vulnerabilities in AI-generated code, latency, context handling, and user trust, motivating verifiable analyses, richer testing, and auditable traces.
- Evidence Gaps: Prior evidence largely comes from controlled studies, benchmarks, or small deployments rather than large-scale observation of real repository workflows.This motivates collecting Agentic-PRs together with review artifacts, commit diffs, issue relations, and event timelines.
- Empirical Studies: Telemetry studies find that acceptance of suggestions predicts perceived productivity more strongly than coarse output metrics, with substantial variation across users and time.Other studies report increased searching, validation, and cognitive load when developers know code is AI-generated, alongside improved performance on some tasks.
- Empirical Studies: Direct productivity measurements are mixed but cautiously positive, with perceived productivity and starting-point quality improving more consistently than completion time or task success.Broader frameworks therefore evaluate outcomes across participants, tasks, and assistance modalities.