PaperlayerView exact revision on arXiv

Source-linked AI summary

The Rise of AI Teammates in Software Engineering (SE) 3.0: How Autonomous Coding Agents Are Reshaping Software Engineering

Hao Li, Haoxiang Zhang, Ahmed E. Hassan

arXiv:2507.15003v1cs.SEcs.AIcs.CEcs.LG

TL;DR

Real-world use of autonomous coding agents lacked systematic empirical evidence. The paper introduces AIDev, a large-scale dataset of agent-authored pull requests and associated development records. It shows that agents can close PRs much faster than humans but are less likely to be merged, especially for complex changes.

  • Problem

    Existing discussions of autonomous software engineering were largely conceptual, leaving empirical questions about agent operation, productivity, review, mergeability, and team adaptation unanswered.

    Source: §1 · p. 2

  • Method

    The paper constructs AIDev, a dataset of autonomous-agent pull requests and software-development activity across five agents, repositories, developers, reviews, and code changes.

    Source: §1 · p. 2, §6 · p. 20

  • Results

    Agents like OpenAI Codex can close PRs up to ten times faster than humans, yet Agentic-PRs are significantly less likely to be merged, particularly for complex changes.

    Source: §6 · p. 20

  • Takeaways & Limitations

    AIDev grounds SE 3.0 research in real-world agent activity and supports evaluation beyond static benchmarks.

    Source: §6 · p. 20

    Current benchmarks are static, often single-language, and omit important software-engineering context.

    Source: §5.2 · p. 19

Abstract

from arXiv · show

The future of software engineering--SE 3.0--is unfolding with the rise of AI teammates: autonomous, goal-driven systems collaborating with human developers. Among these, autonomous coding agents are especially transformative, now actively initiating, reviewing, and evolving code at scale. This paper introduces AIDev, the first large-scale dataset capturing how such agents operate in the wild. Spanning over 456,000 pull requests by five leading agents--OpenAI Codex, Devin, GitHub Copilot, Cursor, and Claude Code--across 61,000 repositories and 47,000 developers, AIDev provides an unprecedented empirical foundation for studying autonomous teammates in software development. Unlike prior work that has largely theorized the rise of AI-native software engineering, AIDev offers structured, open data to support research in benchmarking, agent readiness, optimization, collaboration modeling, and AI governance. The dataset includes rich metadata on PRs, authorship, review timelines, code changes, and integration outcomes--enabling exploration beyond synthetic benchmarks like SWE-bench. For instance, although agents often outperform humans in speed, their PRs are accepted less frequently, revealing a trust and utility gap. Furthermore, while agents accelerate code submission--one developer submitted as many PRs in three days as they had in three years--these are structurally simpler (via code complexity metrics). We envision AIDev as a living resource: extensible, analyzable, and ready for the SE and AI communities. Grounding SE 3.0 in real-world evidence, AIDev enables a new generation of research into AI-native workflows and supports building the next wave of symbiotic human-AI collaboration. The dataset is publicly available at https://github.com/SAILResearch/AI_Teammates_in_SE3. > AI Agent, Agentic AI, Coding Agent, Agentic Coding, Software Engineering Agent

1 Introduction

Software engineering is shifting toward autonomous AI teammates that initiate and contribute to development workflows, but their real-world operation and effects were previously underdocumented. AIDev provides large-scale empirical evidence and shows both rapid agent activity and important gaps in acceptance, review, and code complexity.

  • OpenAI Codex created over 400,000 pull requests in open-source repositories within less than two months of its May 2025 release.Source: §1 · p. 2
  • AIDev contains 456,535 Agentic-PRs from five autonomous coding agents across 47,303 developers and 61,453 repositories.Source: §1 · p. 2
  • AIDev is intended as a foundation for studying human-AI cooperation, productivity, review behavior, and governance in real-world software projects.Source: §1 · p. 3
  • Agents complete PRs rapidly, but their PRs are less likely to be accepted than human-authored PRs, indicating quality and trust gaps.GitHub Copilot can complete half of its PRs within 13 minutes, while Agentic-PRs diverge from traditional benchmark results.Source: §1 · p. 3
  • OpenAI Codex PRs are reviewed and merged faster than human-authored PRs, while the thoroughness of review remains a concern.Source: §1 · p. 3
  • Agents can greatly increase developer output, yet their PRs alter fewer structural code properties, including cyclomatic complexity.One developer produced 164 Agentic-PRs in three days versus 176 Human-PRs over three years.Source: §1 · p. 3

2 Background

The paper frames SE 3.0 as a progression from deterministic development through predictive and assistant-based tools toward autonomous, task-level AI teammates. It also situates these agents against current benchmarks and human-AI productivity research.

2.1 From SE 1.0 to SE 3.0

  • SE 1.0 used hand-coded deterministic workflows, SE 1.5 introduced token-level assistance, and SE 2.0 generated broader code while retaining human control.Source: §2.1 · p. 4
  • SE 3.0 shifts developers from manual coding toward orchestrating autonomous agents that plan, execute tools, modify code, test, and submit pull requests.Source: §2.1 · p. 4

2.2 Autonomous Coding Agent in Software Engineering

  • Autonomous coding agents combine persistent memory, tool-use planning, self-reflection, and negotiated handoffs to act as collaborators rather than simple assistants.Source: §2.2 · p. 5
  • Prior agent systems include RepairAgent for automated defect repair and BRT Agent for generating bug-reproduction tests that improve repair effectiveness.Source: §2.2 · p. 5

2.3 Benchmarking Autonomous Capabilities

  • Benchmarks such as SWE-bench evaluate repository-level capabilities, but static and research-oriented evaluations leave a gap from real-world software engineering.Source: §2.3 · p. 5

2.4 Human-AI Collaboration and Productivity

  • Human-AI productivity depends on balancing acceleration with validation, testing, trust calibration, and alignment with developers’ cognitive models and workflows.Source: §2.4 · p. 5

3 The AIDev Dataset

AIDev mines GitHub pull requests associated with five autonomous coding agents and organizes their activity into a broad dataset plus a popular-repository subset. Its schema links PRs to repositories, users, timelines, reviews, commits, diffs, issues, and raw patches.

3.1 Data Collection

  • AIDev collects GitHub pull requests attributed to OpenAI Codex, Devin, GitHub Copilot, Cursor, and Claude Code using REST APIs and targeted search queries.Source: §3.1 · p. 6
  • The collection uses documented release dates when available and a conservative January 1, 2025 start date for Cursor and GitHub Copilot.Data collection ends on June 22, 2025; most Cursor and Claude Code PRs occur after March 2025.Source: §3.1 · p. 6

3.2 Dataset Overview

  • The full dataset contains 456,535 pull requests from 47,303 developers across 61,453 repositories.Source: §3.2 · p. 6
  • OpenAI Codex dominates overall PR volume, exceeding Devin’s volume by approximately 17 times after its May 2025 release.Source: §3.2 · p. 6
  • AIDev-pop filters to repositories with at least 500 stars, yielding 7,122 PRs from 1,240 developers across 856 repositories while retaining all five agents.Source: §3.2 · p. 6

3.3 Dataset Schema

  • The schema supports lifecycle, collaboration, code-change, and issue-tracing analyses through PR timelines, reviews, comments, commits, diffs, issues, and repository metadata.Source: §3.3 · pp. 7–8

4 Applications of the AIDev dataset

This section applies the AIDev dataset to study autonomous coding agents’ productivity, task participation, contribution speed, and integration workflows. It also examines review and latency bottlenecks, failure modes, benchmarks, and questions of accountability and human–bot responsibility.

4.1 Productivity in Autonomous Coding Agents Era

  • Across evaluated autonomous coding agents, PR acceptance rates consistently lag human performance, particularly for feature development, bug fixing, and performance optimization; OpenAI Codex leads at 64%, followed by Devin at 49%.Source: §4.1 · p. 9
  • Documentation is an agent strength: OpenAI Codex reaches 88.6% acceptance and Claude Code 85.7%, exceeding the human baseline of 76.5%.Documentation’s natural-language emphasis and lower functional risk may align with language-model capabilities and increase reviewer confidence.Source: Figure 3 · p. 10, §4.1 · p. 10
  • GitHub Copilot completes 50% of PR jobs within 12.8 minutes and 75% within 18.5 minutes, although its 95th percentile exceeds one hour.The long tail may reflect cold starts, dependency resolution, or API throttling, potentially hindering real-time feedback and CI/CD responsiveness.Source: §4.1 · p. 10, Figure 4 · p. 11
  • AIDev supports integration-oriented benchmarks that evaluate real-world acceptance, maintainability, style, and project context rather than functional correctness alone.This shifts evaluation from whether agents generate code to whether their contributions improve everyday software development.Source: §4.1 · p. 11
  • AIDev also enables failure-mode analysis of rejected PRs by linking review comments, timeline events, and patch feedback to issues such as logic bugs.These preserved diagnostic artifacts can guide targeted improvements to autonomous coding agents.Source: §4.1 · p. 11
  • AIDev supports latency-aware orchestration by exposing GitHub Copilot’s heavy-tailed latency and telemetry for environment setup, MCP startup, and inference.The dataset motivates caching, speculative execution, incremental previews, warm starts, and prompt-level efficiency techniques to reduce median and tail latency.Source: §4.1 · p. 11

4.2 Agentic Code Review Dynamics

  • OpenAI Codex accepted PRs close in 0.3 hours versus 3.9 hours for Human-PRs, while rejected PRs close in 2.4 versus 27.6 hours.These differences are statistically significant, with medium and small effect sizes respectively, suggesting rapid decisions may reflect clear correctness or flaws rather than deep review.Source: §4.2 · pp. 12–13, Table 5 · p. 12, Figure 5 · p. 13
  • GitHub Copilot accepted PRs take 17.2 hours to review versus 3.9 hours for accepted Human-PRs, whereas rejected Copilot PRs take 4.6 versus 27.6 hours.This slow-to-accept, fast-to-reject pattern may reflect quality inconsistency or ambiguity in Copilot outputs.Source: §4.2 · p. 13
  • Rejected Agentic-PRs close significantly faster than Human-PRs for every agent except Devin, raising concerns that rapid triage may involve superficial rather than meaningful scrutiny.The analysis does not assess review-comment depth or formal change requests, so the extent of genuine review remains uncertain.Source: §4.2 · p. 13
  • Human reviewers remain dominant across Agentic-PRs, yet review shifts toward automated hybrid collaboration: bot reviewers appear in 20.1% of Agentic-PRs versus 10.0% of Human-PRs.Most PRs receive no explicit review—75.3% of Human-PRs and 58.2% of Agentic-PRs—while human-only review accounts for 14.7% and 21.8%, respectively.Source: §4.2 · p. 13

provider, forming closed PR-review loops that streamline workflows; but risk reinforcing provider-specific

This section examines how provider-aligned review bots and agentic coding shape integrated workflows, review burden, triage, throughput, complexity, authorship accountability, and programming-language preferences. It also outlines research directions and provides AIDev data and traces for empirical study of these issues, including long-term code quality, planning, and language effects.

provider, forming closed PR-review loops that streamline workflows; but risk reinforcing provider-specific

  • 91.0% of bot reviews on GitHub Copilot PRs came from copilot-pull-request-reviewer[bot], while cursor[bot] reviewed 13.5% of Cursor PRs, representing 53.8% of its bot reviews.These provider-aligned loops provide tightly integrated workflows, consistent UX, and faster iteration cycles.Source: provider, forming closed PR-review… · p. 14
  • Cross-provider bots such as coderabbitai[bot] review both human- and agent-authored code, potentially enabling more robust and diverse evaluation.Their broader exposure is contrasted with bots confined to a single provider ecosystem.Source: provider, forming closed PR-review… · p. 14
  • Traditional static analysis still outperforms current AI systems for vulnerability detection, leaving trusted AI-powered security review as an open frontier.Among the top 10 review bots, only github-advanced-security[bot] remains rule-based rather than AI-powered.Source: provider, forming closed PR-review… · p. 14
  • Agent-generated PRs are frequently reviewed by bots from the same provider, creating streamlined workflows but risking provider-specific blind spots.Figure 7 shows strong provider association; such bots may overlook stylistic issues or latent errors outside their tuned assumptions.Source: provider, forming closed PR-review… · p. 14, Figure 7 · p. 15
  • AIDev enables analysis of review effort, feedback, merger outcomes, and human-bot collaboration through PR timelines, review histories, and reviewer metadata.These data support future efforts to make autonomous contributions more interpretable, reviewable, and trustworthy.Source: provider, forming closed PR-review… · p. 15

4.3 Beyond Agentic/Vibe Coding for Software Engineering

  • AI-assisted coding prioritized throughput over complexity: one developer submitted 164 Codex-assisted PRs in five days versus 176 human-authored PRs over the preceding three and a half years.Only 9.1% of Agentic-PRs changed cyclomatic complexity, compared with 23.3% of Human-PRs, suggesting minimal or boilerplate-style updates.Source: §4.3 · p. 16, Figure 8 · p. 17
  • Authorship attribution remains inconsistent: Devin, GitHub Copilot, and Cursor identify authorship, whereas OpenAI Codex provides none and Claude Code uses an optional co-authorship message.The resulting traceability gap undermines transparency, auditability, and accountability, motivating standardized authorship labels.Source: §4.3 · pp. 16–17
  • Standardized ownership labels could support provenance tracking, human-AI collaboration, responsible-AI governance, and longitudinal studies of software quality and maintainability.Linking ownership-labelled code with issue reports and bug fixes would enable analysis of AI- versus human-authored code over time.Source: §4.3 · p. 17
  • Long-term agentic code quality remains unresolved because static complexity metrics do not capture defect density, review latency, test coverage, or bug-fix effort.Future evaluations should examine how AI-authored code performs over time rather than only its immediate simplicity.Source: §4.3 · p. 17
  • Quality evaluations should account for project type, scale, and team practices because infrastructure libraries, mobile apps, and backend systems may value simplicity and optimization differently.This is especially important for real-world and high-stakes deployments where quality cannot be compromised.Source: §4.3 · pp. 17–18
  • AIDev releases curated Agentic-PRs and developers’ contribution histories, including pre-agent periods, enabling benchmarking across time, authorship, and tool usage.The dataset supports empirical study of agentic code quality beyond anecdotal impressions.Source: §4.3 · p. 17
  • TypeScript is the most common language across agents, while Codex skews toward Python and GitHub Copilot favors C#, reflecting domain-specific integration and user-base affinities.These patterns motivate language-aware evaluation and agents tuned to diverse development environments.Source: §4.3 · p. 17, Table 6 · p. 18
  • Research should analyze task planning because translating human intent into executable steps shapes AI-generated-code success, yet many agents obscure this reasoning stage.Improved planning analysis could increase reliability, transparency, and software quality.Source: §4.3 · p. 18
  • GitHub Copilot’s PR descriptions preserve planning traces, including a bug-fix plan iteratively updated across five versions, enabling study of prompts, developer interaction, and agent capability.These traces can reveal which planning structures produce more effective outcomes and distinguish stronger agents from weaker ones.Source: §4.3 · p. 18
  • Programming-language features may influence agent effectiveness and reliability: stronger type systems such as TypeScript and Rust could provide compile-time feedback and structural constraints.The open question is how language-level affordances support, amplify, or constrain agentic behavior.Source: §4.3 · p. 18
  • Studying language–agent interplay could guide language selection and the design of language-aware AI teammates that better support autonomous capabilities.This direction connects programming-language design with AI-powered automation and agent reliability.Source: §4.3 · p. 18

5 The Road Ahead

The Road Ahead positions software repositories as real-world learning and evaluation environments for autonomous coding agents, while calling for dynamic benchmarks and SE 3.0 practices that preserve human oversight.

5 The Road Ahead

  • AIDev shows autonomous agents are embedded in everyday software development and motivates a roadmap for research and practice.Source: §5 · p. 18

5.1 Software Repositories as Training Environments for AI Agents

  • Software repositories can serve as interactive agent training environments with feedback from builds, merged pull requests, reviewer feedback, failed tests, and rejected pull requests.Source: §5.1 · p. 19

5.2 Dynamic Benchmarking and Living Leaderboards

  • Existing benchmarks are often static, single-language, weak on review dynamics, and vulnerable to contamination and pretraining leakage.Source: §5.2 · p. 19
  • Dynamic, domain-specific benchmarks should evaluate both task solutions and integration into real projects using outcomes such as acceptance, review turnaround, and code quality.Source: §5.2 · pp. 19–20
  • Reliable real-world benchmarks remain costly and computationally or manually intensive, requiring careful design tradeoffs.Source: §5.2 · p. 19

5.3 Towards SE 3.0 Methodologies

  • Always-active agents create review-load, triage, and conflict-resolution challenges while enabling a single developer to coordinate multiple autonomous contributors.Source: §5.3 · p. 20
  • SE 3.0 requires orchestration frameworks, adaptive review systems, and governance defining autonomous action, validation, and accountability.Source: §5.3 · p. 20

6 Conclusion

The paper introduces AIDev as the first systematic empirical dataset of widely used autonomous coding agents in modern codebases. Its analyses show faster contributions but lower acceptance and a gap between benchmark performance and real-world effectiveness.

  • AIDev is the first empirical and systematic evidence of how widely used autonomous coding agents contribute to modern codebases.Source: §6 · p. 20
  • Over 456K agent-authored pull requests span 61K GitHub repositories and 47K developers.Source: §6 · p. 20
  • Agents can close pull requests up to ten times faster than humans, but substantial disparities remain in acceptance rates and review dynamics.Source: §6 · p. 20
  • Agentic pull requests are significantly less likely to be merged, especially for complex feature-development or bug-fixing tasks.Source: §6 · p. 20
  • These findings expose a gap between benchmark performance and real-world effectiveness, questioning the ecological validity of static evaluations.Source: §6 · p. 20
  • AIDev is released as a living, extensible resource for research on productivity, authorship, code review, collaboration, governance, and AI-native tooling.Source: §6 · p. 20
Loading 2507.15003v1…