Source-linked AI summary
Security in the Age of AI Teammates: An Empirical Study of Agentic Pull Requests on GitHub
Mohammed Latif Siddiq, Xinye Zhao, Vinicius Carvalho Lopes, Beatrice Casey, Joanna C. S. Santos
TL;DR
Autonomous coding agents increasingly contribute security-relevant changes, but evidence about their behavior and oversight in realistic workflows remains limited. This study analyzes manually validated agentic pull requests using quantitative, qualitative, and predictive methods. Security-related work forms a meaningful minority of activity, receives heightened scrutiny, and has rejection associated more with complexity and verbosity than explicit security topics.
Problem
Existing research provides limited systematic evidence about how autonomous coding agents perform security work and how humans evaluate those contributions in realistic workflows.
Method
The study analyzes AIDev agent-authored pull requests through keyword filtering, manual validation, quantitative outcome and review analysis, qualitative coding, and predictive modeling.
Results
Security-related agentic pull requests constitute approximately 4% of activity, show lower merge rates and longer review latency, and face rejection associated more with complexity and verbosity than explicit security topics.
Takeaways & Limitations
Agents’ security contributions often involve supportive hardening, while human review remains especially important for security-related pull requests.
Takeaways & Limitations
The study’s findings are subject to threats to validity, and rejection prediction is constrained by contextual factors unavailable in pull-request metadata and text.
Abstract
from arXiv · showhide
Autonomous coding agents are increasingly deployed as AI teammates in modern software engineering, independently authoring pull requests (PRs) that modify production code at scale. This study aims to systematically characterize how autonomous coding agents contribute to software security in practice, how these security-related contributions are reviewed and accepted, and which observable signals are associated with PR rejection. We conduct a large-scale empirical analysis of agent-authored PRs using the AIDev dataset, comprising of over 33,000 curated PRs from popular GitHub repositories. Security-relevant PRs are identified using a keyword filtering strategy, followed by manual validation, resulting in 1,293 confirmed security-related agentic-PRs. We then analyze prevalence, acceptance outcomes, and review latency across autonomous agents, programming ecosystems, and types of code changes. Moreover, we apply qualitative open coding to identify recurring security-related actions and underlying intents, and examine review metadata to identify early signals associated with PR rejection. Security-related Agentic-PRs constitute a meaningful share of agent activity (approximately 4\%). Rather than focusing solely on narrow vulnerability fixes, agents most frequently perform supportive security hardening activities, including testing, documentation, configuration, and improved error handling. Compared to non-security PRs, security-related Agentic-PRs exhibit lower merge rates and longer review latency, reflecting heightened human scrutiny, with variation across agents and programming ecosystems. PR rejection is more strongly associated with PR complexity and verbosity than with explicit security topics.
1. Introduction
Autonomous coding agents increasingly modify production code, but prior research provides limited evidence about their security-relevant behavior in realistic workflows. This study addresses that gap through large-scale quantitative analysis and qualitative interpretation of agent-authored pull requests and human review.
- Autonomous coding agents now independently author pull requests that modify production code in real-world repositories.
- Prior studies emphasize productivity outcomes, leaving security implications comparatively underexplored.
- Security-relevant agent contributions require reasoning about threat models, privilege boundaries, cryptography, configuration, and compatibility.
- The study examines prevalence, work types, security intents, and human evaluation of agent-authored security changes.
- The methodology combines quantitative measurement with open coding of manually validated security-related pull requests.
2. Background & Related Work
Related work has studied agentic software engineering, productivity, collaboration, secure code generation, and pull-request review, but rarely security behavior within realistic agent-authored workflows. This paper integrates quantitative and qualitative analysis to examine security actions, intents, and review scrutiny.
- Prior agent studies commonly emphasize productivity, collaboration, acceptance, and developer efficiency rather than security-specific behavior.
- This study instead analyzes security-related work within real pull-request development workflows.
- Existing security benchmarks largely evaluate isolated tasks such as vulnerability detection or secure code generation.
- Security-related pull requests involve explicit concerns such as vulnerability mitigation, authentication, authorization, cryptography, configuration, dependencies, or compliance.
- Automated security-focused pull requests can still experience review delays and rejection, reflecting developer caution.
- The paper addresses the open problem of how security-related agent-authored pull requests are evaluated and governed.
3. Methodology
The study follows a structured, multi-stage empirical process that identifies security-relevant agentic pull requests, quantifies their prevalence and review outcomes, and analyzes security actions, intents, and rejection signals.
- Security-relevant agentic pull requests are identified through keyword filtering followed by manual validation.
- Quantitative analyses examine prevalence, outcomes, and human review patterns.
- Qualitative open coding and predictive modeling analyze agent security actions, design intents, and early rejection indicators.
3.1. Research Questions (RQs)
The research questions cover how often agents perform security-relevant work, how outcomes vary, what actions and intents they exhibit, how humans review them, and which early signals relate to rejection.
- RQ1: RQ1 measures how frequently autonomous coding agents contribute security-relevant changes to GitHub repositories.
- RQ2: RQ2 compares outcomes across agents, programming ecosystems, and code-change types.
- RQ3: RQ3 identifies recurring security actions and underlying design intents through open coding.
- RQ4: RQ4 examines how human review shapes outcomes using merge decisions and review latency as scrutiny proxies.
- RQ5: RQ5 investigates whether PR metadata and text provide early signals associated with rejection or perceived risk.
3.2. Dataset of Pull Requests Authored by Autonomous Coding Agents
The study uses AIDev’s curated dataset of 33,596 agent-authored pull requests from 2,807 popular GitHub repositories, covering five autonomous coding agents and enriched review artifacts.
- 33,596 agent-authored PRs form the curated AIDev subset used for analysis.The broader dataset contains 932,791 agent-authored PRs, but the curated subset provides the artifacts needed to study security outcomes and human review behavior.
- 2,807 GitHub repositories with at least 100 stars contribute PRs from five widely deployed agents.The agents are OpenAI Codex, Copilot, Devin, Cursor, and Claude Code.
- The dataset includes PR titles, descriptions, timestamps, merge outcomes, review comments, review decisions, commit metadata, and file-level diffs.
3.3. Answering RQ1: Prevalence of Security-Relevant Agentic-PRs
Security-relevant Agentic-PRs are identified through inclusive keyword filtering followed by expert manual validation, yielding 1,293 confirmed cases from 2,598 candidates.
- Six keyword categories cover general security, identity and access, cryptography, attack vectors, privacy and compliance, and defensive operations.
- Candidate PRs are flagged when a case-insensitive, word-boundary-constrained keyword appears in the title or body.
- Two experienced authors independently manually inspected candidates instead of relying on LLM-based labeling.
- κ=0.91 indicates almost perfect agreement between annotators on a sampled set of 335 candidate PRs.
- 1,293 security-relevant Agentic-PRs were confirmed from 2,598 keyword-filtered candidates.The resulting overall keyword-filter precision was 49.8%.
- Keyword precision varies substantially, reaching 100% for CVE, vulnerability, malicious, and encryption but only 13–37% for patch, compliance, and token.This variation motivates manual validation after the high-recall keyword pass.
3.4. Answering RQ2: Differences Across Agents, Ecosystems, and Code Change Types
The study compares security-PR outcomes across agents, programming ecosystems, and manually assigned coarse-grained change categories, while accounting for limitations in automated PR-type tags.
- Security-PR outcomes are compared across agents using counts, closed PRs, merge rates, and dominant programming languages.
- Repositories are mapped into Web, Systems, Data/ML, Enterprise, and Ops domains using their primary GitHub language.
- AIDev’s rule-based PR-type tags provide approximate intent characterization but may overlap or remain ambiguous.
- All 1,293 security-relevant PRs were manually assigned to one coarse-grained category using diffs, descriptions, discussions, and tags.
- The categories include dependency updates and vulnerability fixes, with vulnerability fixes covering concrete flaws regardless of CVE assignment.
- Category volumes and merge rates are computed across agents, ecosystems, and change types to avoid noisy fine-grained labels.
3.5. Answering RQ3: Security Actions and Intents in Agentic-PRs
The study uses open coding to distinguish what security-related Agentic-PRs do from why they do it, producing an inductively refined codebook applied across the validated dataset.
- 1,293 security PRs are analyzed through open coding of titles and descriptions to identify recurring actions and intents.
- Actions describe implementation mechanisms, whereas intents capture the underlying security motivation.The two dimensions are coded separately to avoid conflating how a change is implemented with why it is made.
- Codes are derived inductively rather than from predefined taxonomies such as CWE or OWASP.
- Figure 2 illustrates the process by mapping highlighted title and description elements to action and intent codes.
- Open redirect and header-injection examples separate concrete remediation steps from their stated security rationales.
- Constant comparison, analytic memos, code merging, and consensus resolution support iterative refinement and consistent application of the final codebook.
3.6. Answering RQ4: Reviewer Behavior and Scrutiny of Security Agentic-PRs
The study operationalizes reviewer scrutiny through review duration, merge outcomes, and interaction signals, while controlling for PR complexity and comparing agent-authored with human-authored PRs.
- Scrutiny measures: Review scrutiny is measured using review latency and merge rate, capturing process duration and acceptance outcomes.Latency is elapsed time from PR creation to closure; merge rate is the proportion of closed PRs that were merged.
- Control variables: PR complexity is controlled using code churn, files touched, and description length, with merge rates repeated across description-length tertiles.The tertiles are small (<36 words), medium (36–69 words), and large (>69 words).
- Analysis design: The latency analysis compares 1,130 closed security-related Agentic-PRs with 30,154 closed non-security Agentic-PRs after excluding invalid timestamps.A Mann–Whitney U test is used because review times are highly skewed.
- Analysis design: Agent-specific median latencies compare security and non-security PRs within each agent to account for variation in reviewer trust.The within-agent comparison assesses whether particular agents receive disproportionately different scrutiny for security-relevant changes.
- Human-authored baseline: The study extends the comparison to 4,306 human-authored PRs from the same 198 repositories, including 634 manually validated human security PRs.The same keyword filtering and manual analysis process is applied to human-authored PRs.
3.7. Answering RQ5: Early Predictors of Risky or Rejected PRs
The study tests whether signals available when a security-related Agentic-PR is created can predict rejection, using structured, text-based, and prompted LLM models.
- Prediction target: The prediction task uses 1,130 closed security-relevant Agentic-PRs, including 435 rejected and 695 merged PRs.A PR is rejected when it closes without being merged.
- Early signals: Early features combine PR text structure, security keywords and categories, and agent identity without relying on code-churn or file-level metadata.Security signals include title-keyword presence, body keyword counts, Vulnerability Fix, and Dependency Update indicators.
- Model families: Structured prediction compares interpretable logistic regression with Random Forests that can model nonlinear relationships and feature interactions.Both classifiers operate on early structured signals.
- Model families: Text prediction evaluates TF–IDF with linear SVM, fastText, and fine-tuned DistilBERT using concatenated titles and descriptions.These models test whether PR text alone provides predictive information.
- Model families: GPT-4o-mini is evaluated as a prompting-based baseline in zero-shot and few-shot settings using truncated PR descriptions.The few-shot setting prepends four training examples, while the zero-shot setting provides no task-specific examples.
4. Results
Security-related agentic work is a small but recurring and unevenly distributed part of agent activity, with substantial variation in acceptance, ecosystems, and implementation patterns.
- Prevalence and distribution: 1,293 security-related PRs represent 3.85% of 33,596 Agentic-PRs in the curated dataset.Security-relevant PRs were identified through filtering followed by manual validation.
- Prevalence and distribution: Security-PR prevalence varies sharply by agent, from 1.3% for OpenAI Codex to 14.6% for Claude Code.Copilot and Devin fall between these extremes at 10% and 7.6%, respectively.
- Prevalence and distribution: Security PRs are spread across 520 repositories more evenly than overall Agentic-PRs, although 61.7% of those repositories contribute only one security PR.Overall Agentic-PRs are highly concentrated: 22 repositories account for 50% of all PRs.
- Prevalence and distribution: Monthly security-PR rates remain broadly stable at 3.8–6.2% after excluding an 11.4% March 2025 spike driven by concentrated Devin activity.The result suggests no material change in security-PR propensity as adoption grew.
- Acceptance outcomes: OpenAI Codex has the highest security merge rate at 86.59%, while Copilot has the lowest at 49.60%.Cursor, Claude Code, and Devin achieve 76.47%, 58.62%, and 52.12%, respectively.
- Ecosystem variation: Security merge rates differ across languages, with Python at 68.30%, TypeScript at 56.51%, and Rust lowest at 51.16%.Small-sample languages such as Ruby and HTML exceed 80% but require cautious interpretation.
- Ecosystem variation: Across domains, Operations has the highest merge rate at 69.70% based on only 35 PRs, while Systems has the lowest at 56.15%.Data/ML, Web, and Enterprise record 63.71%, 62.11%, and 60.18%, respectively.
- Security change types: Security work is concentrated in Security Feature and Vulnerability Fix categories, but agents differ in their category preferences.Claude Code emphasizes Security Features at 68.7%, whereas Copilot emphasizes Vulnerability Fixes at 50.5%.
5. Discussion
Security-related agentic PRs represent a substantial and varied form of software work, but human reviewers scrutinize them more heavily than non-security PRs. Rejection signals are dominated by complexity and explanatory burden, while observed patterns differ across agents and change types.
- Security work is meaningful but secondary: 3.85% of agentic PRs were manually confirmed as security-relevant, representing over a thousand security-related changes in real-world repositories.This work is a minority of total agent activity but is already occurring at scale.
- Security work is meaningful but secondary: Agents frequently perform security hardening through refactoring, testing, documentation, error handling, configuration, authentication, input validation, and dependency management.These actions are often embedded in routine functionality, usability, maintainability, or compatibility improvements rather than isolated vulnerability remediation.
- Human review is heightened and differentiated: Security PRs merge less often than non-security PRs, at 61.5% versus 77.3%, and have more than tenfold higher median review latency, at 3.92 versus 0.11 hours.They also receive more review comments and formal review events and more often trigger Changes Requested actions.
- Human review is heightened and differentiated: Review scrutiny varies by agent and change type: security feature PRs take longest to review, while configuration and compliance PRs are reviewed faster.Devin and Copilot show pronounced delays and lower merge rates, whereas Codex has comparatively low observed latency in this dataset.
- Complexity and explanatory burden predict rejection: Rejection is most strongly associated with body length, structured sections, and title length, whereas explicit security keywords contribute only modestly.The reported associations are not causal, and trained models reach only moderate macro-F1 scores of 0.58–0.60.
- Implications: Review-aware agent design should treat PR scope, presentation, semantic content, and review context as evaluation concerns rather than relying on repository-agnostic assumptions.The study argues that autonomous agents must be technically capable, review-aware, and context-sensitive within human-centered workflows.
6. Conclusion
This study finds that security-related Agentic-PRs are a meaningful but minority activity, typically involving supportive hardening and receiving heightened human scrutiny. Rejection is more strongly associated with PR complexity and verbosity than with explicit security topics.
- Main findings: Security-related Agentic-PRs constitute a meaningful but minority share of agent activity and often involve supportive security hardening rather than narrowly scoped vulnerability fixes.The study analyzes security-related PRs authored by five autonomous coding agents in real-world GitHub repositories.
- Main findings: Security-related Agentic-PRs receive lower merge rates and substantially longer review latency than non-security PRs, with variation across agents, ecosystems, and code-change types.These outcomes indicate heightened human scrutiny of security-related agent work.
- Rejection signals: PR rejection is more strongly associated with complexity and verbosity than with explicit security topics.The finding suggests that observable characteristics of PR content matter more than security labeling alone for rejection prediction.
- Rejection signals: Trained text models outperform structured-feature and prompting-based LLM baselines for predicting PR rejection.This suggests reviewer judgments are more strongly reflected in PR text semantics than in lightweight metadata or zero-/few-shot prompting alone.
7. Declarations
The authors disclose their contributions, provide access to the replication package, and report no known competing financial interests or personal relationships.
- Author contributions: M. L. Siddiq led conceptualization, methodology design, data collection, analysis, writing, and editing.
- Author contributions: X. Zhao, V. C. Lopes, B. Casey, and J. C. S. Santos contributed to analysis, validation, supervision, administration, writing, or editing.
- Data and materials: The replication package is available at the cited reference.
- Competing interests: The authors declare no known competing financial interests or personal relationships that could have influenced the reported work.