Source-linked AI summary
Investigating Autonomous Agent Contributions in the Wild: Activity Patterns and Code Change over Time
Razvan Mihai Popescu, David Gros, Andrei Botocan, Rahul Pandita, Prem Devanbu, Maliheh Izadi
TL;DR
The paper addresses limited real-world evidence about how autonomous coding agents contribute to software development and how their code evolves over time. It mines GitHub pull requests and related metadata across five agents, comparing their activity and longitudinal code changes with human contributions. Agent activity is increasing, but agent-generated code is less likely to be retained and more prone to churn.
Problem
Real-world assessment of autonomous coding agents remains limited, especially for collaborative activity and the longitudinal evolution of their code.
Method
The study analyzes approximately 110,000 GitHub pull requests and related metadata across five coding agents, using agent-specific signals to identify contributions.
Results
Agent-generated code is significantly less likely to be retained and more prone to churn than human-authored code, while agent activity is increasing in open-source projects.
Takeaways & Limitations
Public pull requests provide a lens for studying agentic software development, including activity patterns and the subsequent stability of contributed code.
Takeaways & Limitations
Agent systems changed rapidly during data collection, so the findings emphasize aggregated behavioral patterns rather than version-specific effects.
Abstract
from arXiv · showhide
The rise of large language models for code has reshaped software development. Autonomous coding agents, able to create branches, open pull requests, and perform code reviews, now actively contribute to real-world projects. Their growing role offers a unique and timely opportunity to investigate AI-driven contributions and their effects on code quality, team dynamics, and software maintainability. In this work, we construct a novel dataset of approximately $110,000$ open-source pull requests, including associated commits, comments, reviews, issues, and file changes, collectively representing millions of lines of source code. We compare five popular coding agents, including OpenAI Codex, Claude Code, GitHub Copilot, Google Jules, and Devin, examining how their usage differs in various development aspects such as merge frequency, edited file types, and developer interaction signals, including comments and reviews. Furthermore, we emphasize that code authoring and review are only a small part of the larger software engineering process, as the resulting code must also be maintained and updated over time. Hence, we offer several longitudinal estimates of survival and churn rates for agent-generated versus human-authored code. Ultimately, our findings indicate an increasing agent activity in open-source projects, although their contributions are associated with more churn over time compared to human-authored code.
1 Introduction
Autonomous coding agents have made real-world software contributions increasingly observable, motivating large-scale study of their activity and code evolution. This paper compares agent and human contributions across collaboration patterns and longitudinal maintenance signals.
- Motivation: Early coding results established that LLMs could generate complete code units, with Codex solving 28.8% of HumanEval problems while GPT-3 solved none.HumanEval evaluates relatively simple Python programs.
- Motivation: Assistive systems such as Copilot and ChatGPT spread LLM use, but their limited workflow integration made real-world usage and software quality difficult to assess.Prior research often emphasized acceptance rates, which did not capture representative software quality attributes.
- Motivation: Autonomous agents now extend beyond passive assistance by managing development workflows from requirements through implementation.Their defining features include autonomy, expanded scope, and engineering practicality.
- Motivation: Agentic systems account for 10% of public GitHub pull requests, increasing the need to examine their effects on code maintainability.The paper highlights maintenance as a traditionally resource-intensive software development phase.
- Research scope: The study investigates five coding agents and introduces a longitudinal focus on change signals associated with agent-authored code quality.The agents are OpenAI Codex, GitHub Copilot, Claude Code, Google Jules, and Devin.
- Main finding: Agent activity is growing, particularly in lower-starred repositories, and agent contributions are correlated with more extensive code changes over time than human-authored code.The finding concerns longitudinal code-change behavior rather than a demonstrated causal effect.
- Contributions: The paper curates over 110,000 pull requests and compares agent- and human-authored activity across merging, changes, collaboration, repository characteristics, survival, and churn.The dataset includes related metadata and supports research on agentic software development.
2 Related Work
Prior work has progressed from assistive tools and controlled evaluations toward autonomous agents and repository activity studies. However, existing evidence remains limited in scale, agent coverage, repository diversity, and longitudinal analysis.
- Assistants and agents: AI assistants support developers through predictive and assistive capabilities, including code generation and issue or vulnerability identification.They are commonly integrated into developer workflows, such as through IDE extensions.
- Assistants and agents: More proactive agents can plan tasks, act across tools, run self-evaluations, and coordinate over broader project contexts.This marks a transition from reactive coding assistance toward autonomous software-development activity.
- Assistants and agents: Multi-agent systems distribute programming, testing, documentation, design, evaluation, and optimization roles across cooperating agents.Examples include AgentCoder, MaintainCoder, and CodeSim.
- Assistants and agents: Feedback loops allow autonomous agents to adjust behavior based on observed effects while pursuing user-provided goals.The feedback may involve self-correction, human collaboration, or negotiation among agents.
- Evaluation: Controlled user studies and static benchmarks dominate evaluation, but they have limited external validity or known issues such as contamination, faulty ground truths, and saturation.These settings do not fully capture the complexity, pace, and nuances of real-world development.
- Real-world activity: Repository studies have examined documentation, bug fixing, code generation, and AI-generated review comments, showing growing but variable AI participation.The cited work covers pull requests, commits, issues, and code-review workflows.
- Research gap: Existing agent studies often use small samples, a single agent, or highly popular repositories, leaving large-scale multi-agent and longitudinal evidence needed.The paper addresses this gap by studying activity levels and ongoing evolution of agent-generated code.
3 Methodology
The methodology builds a large GitHub dataset of agent- and human-authored pull requests, identifies agent contributions using attribution signals, and evaluates activity and code-change characteristics with defined metrics.
- Dataset construction: Five autonomous coding agents—Devin, Claude Code, OpenAI Codex, GitHub Copilot, and Google Jules—are targeted to represent established and recent agentic coding paradigms.The study uses the GitHub GraphQL search API to collect pull requests associated with these agents.
- Attribution and sampling: Agent-generated pull requests are identified with agent-specific signals such as branch prefixes, authorship, PR-description watermarks, and other filtering rules.Manual sanity checks and duplicate removal are applied after collection, while human pull requests are gathered with the same scraping procedure for comparison.
- Attribution and sampling: The dataset uses representative sampling across agents, with shorter intervals for higher-activity agents and longer intervals for lower-activity agents, plus daily sampling limits.The sampling intervals were empirically tuned to balance coverage throughout the day.
- Dataset construction: The dataset contains 111,969 pull requests contributed by agents and humans, with associated activity information and nested repository, commit, review, comment, issue, and file data.The collection includes auxiliary metadata for up to the first 100 commits, comments, issues, reviews, and modified files per pull request when available.
- Metrics: Activity is characterized using merge rate, merge time, change size, comments, reviews, repository popularity, additions, and modified-file variety.Change size is based on additions plus deletions and reflects the magnitude of a pull request; merge time measures the duration from opening to merging.
- Metrics: Code changes from agent and human commits are assessed with longitudinal survival, churn, and deletion metrics, including whether added lines remain syntactically unchanged over time.The analysis applies these metrics to commits from both contributor groups; change-size values from the GitHub API reflect the PR when first opened and may evolve afterward.
4 Results and Discussion
Agent contributions differ from human pull requests in repository context, change patterns, merging, interaction, and longitudinal code stability. Agents often produce larger, addition-heavy changes, while their code survives less completely and exhibits higher churn over time.
- Pull Request and Repositories Characteristics: About 75% of Codex and Jules PRs came from 0-star repositories, while Devin’s nonzero-star median more closely matched humans.Agent PRs were less common in very popular repositories, though 100+ star repositories approached 15% of some agents’ PRs.
- Pull Request and Repositories Characteristics: Roughly half of Copilot PRs linked issues, compared with about 8% of human PRs, while Codex and Devin rarely had assigned issues.The paper attributes Copilot’s higher issue linkage partly to issue assignment being central to its user experience.
- Pull Request Changes: Claude PRs had a median change size of 376 lines versus approximately 60 lines for human PRs, whereas Codex was used for smaller changes.Across agents, changes were generally more addition-heavy than in the human sample, but effect sizes were moderated by high between-PR variance.
- Pull Request Changes: Agent PRs more often included Python and noncode files: Codex had 27% Python PRs, while Markdown appeared in close to half of Claude PRs.TS/JS appeared in over a third of agent PRs but approximately a quarter of human PRs, with Devin especially high.
- Pull Request Merges: Claude and Codex PRs merged at higher rates than human PRs, while Copilot and Devin merged at lower rates.The merge-rate pattern varied with repository star count by author; Codex’s rate was roughly similar across star counts.
- Pull Request Merges: The median human PR merged in 0.4 hours, versus 0.5 minutes for Codex, while merge times generally increased in higher-starred repositories.The p75 of Codex merge times remained under 1 hour even for higher-starred repositories; the paper speculates that simpler work or pre-submission review may contribute.
- Developer Interaction: Human PRs received comments on 21% of submissions, while agent PRs typically received fewer comments but more reviews than human PRs.Copilot PRs received marginally more reviews than human PRs.
- Code Change over Time: Across three intervals, the fraction of commits whose initial additions all survived was consistently higher for humans than for every agent.The differences were statistically significant, with small but consistently negative Cliff’s δ values of -0.05 to -0.14, indicating slightly longer human-code persistence.
5 Future Work
The dataset supports future studies of agent-generated code quality, stability, collaboration, and behavioral change over longer periods and across broader settings.
- The dataset can support benchmarking agent-generated code quality, stability, and collaboration patterns using real-world outcomes such as post-merge churn and acceptance rates.
- Future extensions could cover longer periods, additional agents, richer metadata, CI/CD results, review iterations, and contributor information.
- Long-term studies could examine technical-debt accumulation in relation to agent versions, project characteristics, and review practices.
6 Threats to Validity
The study’s validity is constrained by changing agent systems, incomplete authorship observability, uncertain human attribution, repository-popularity imbalance, and possible data-collection coverage bias.
- Rapidly changing agent systems make individual version-specific effects opaque, so the analysis emphasizes aggregated behavioral patterns across pull requests.
- Untraceable CLI contributions and disabled watermarks, especially Codex commits recorded under human accounts, limit granular authorship tracking.
- Because the full set of active GitHub coding agents is unknown, supposedly human-authored data may contain undetected agentic contributions.
- Most collected pull requests originate from low-star repositories, requiring activity results to be reported by repository-popularity level and code-change analysis to mitigate potential bias.
- For agents with many pull requests, GitHub API concentration in specific time frames may underrepresent the global distribution of agentic contributions.
7 Conclusion
AI coding agents are rapidly shaping software development, and this study uses public pull requests to examine their activity and code evolution. It finds agent-generated code is less likely to be retained and more prone to churn, while most agent activity targets zero-star repositories.
- Agent-generated code is significantly less likely to be retained and more prone to churn than human-authored code.
- Agent-generated pull requests increasingly target zero-star repositories, which currently represent the majority of agent activity despite differing from pull requests in more popular repositories.
- The broader adoption of agents in higher-starred repositories remains uncertain, and the reasons for current repository-targeting patterns require further investigation.