PaperlayerView exact revision on arXiv

Source-linked AI summary

Do AI Coding Agents Write Infrastructure as Code Like Humans? An Empirical Study of GitHub Pull Requests

Amine Barrak, Emna Ksontini, Charles Wehbe

arXiv:2610.34613v1cs.SE

TL;DR

Evidence remains limited on how AI agents’ real-world IaC contributions differ from humans and bots, how maintainers evaluate them, and whether they introduce different sustainability concerns. This study analyzes IaC PRs by contributor provenance, change type, acceptance, and sustainability smells. Agent and human PRs show comparable smell rates, while acceptance patterns differ by agent disclosure.

  • Problem

    It remains unclear whether agents perform the same IaC changes as humans, whether acceptance differences persist across change types and repositories, and whether sustainability-smell patterns differ.

    Source: §I · p. 1

  • Method

    The study analyzes IaC PRs by contributor provenance, change semantics, review outcomes, and matched agent–human sustainability-smell comparisons.

    Source: D. Statistical Analyses · p. 5, §IX · p. 10

  • Results

    Agent and human PRs introduced sustainability smells at comparable rates (10.4% vs. 10.3%), while disclosed-agent PRs were merged less often within repositories than human PRs across nearly every change type.

    Source: §IX · p. 10

  • Takeaways & Limitations

    PR metadata alone misses silent-agent contributions and misclassifies bots, and aggregate merge rates can obscure repository differences.

    Source: §VII · p. 10, §IX · p. 10

    Agent use without PR or commit traces and bots using user accounts are missed, making silent-agent counts lower bounds; merge outcomes measure acceptance, not quality.

    Source: §VIII · p. 10

Abstract

from arXiv · show

Infrastructure-as-Code (IaC) configurations define cloud resources, networking, and access control, and AI coding agents increasingly write them. Little is known about how agent contributions differ from those of humans and automation bots, how maintainers evaluate them, and whether they introduce different sustainability concerns. We study 349,156 IaC pull requests (PRs) from 49,241 GitHub repositories, distinguishing humans, automation bots, silent agents, and disclosed agents. Bots account for 32.9% of ostensibly human PRs, and 12.3% of the remaining PRs contain agent traces, a share that grows over the study period. Agent PRs are about four times larger than human PRs and more often modify core infrastructure modules, whereas bots mainly maintain dependencies. Silent-agent PRs have higher overall merge rates than human PRs (87.4% vs. 78.9%) but are merged 4.0 percentage points less often within the same repositories. Disclosed agents create new infrastructure more often than humans (34.0% vs. 21.4%), and even for the same change type within the same repository, their PRs are 8.4 points less likely to be merged; silent-agent PRs show comparable acceptance for most change types. Finally, in 2,645 matched agent-human Terraform PR pairs, agent and human PRs introduce sustainability smells at similar rates (10.4% vs. 10.3%; paired odds ratio 1.02, 95% CI 0.83-1.25), most commonly missing remote state and monolithic configurations.

I. INTRODUCTION

IaC changes govern deployed resources and can open network paths, grant permissions, or provision costly resources. The study examines who contributes to IaC, what they change, how PRs are evaluated, and whether agent changes introduce sustainability smells.

Motivation

  • IaC specifies infrastructure as versioned configurations for repeatable provisioning, but applied changes can open network paths, grant permissions, or provision costly resources.Potential sustainability smells include oversized instances, fixed fleets without autoscaling, indefinite log retention, local Terraform state, and monolithic configurations.Source: §I · p. 1
  • Existing research examined agent PR outcomes, generated-IaC correctness and security, and IaC smells separately, leaving real-world agent changes and their acceptance and sustainability patterns unclear.The study connects contributor provenance, change semantics, review outcomes, and sustainability in real-world IaC repositories.Source: §I · pp. 1–2

Authorship and participation

  • Silent-agent PRs were merged 4.0 percentage points less often than human PRs within repositories, despite higher overall acceptance.Contributor populations include humans, automation bots, silent agents, and disclosed agents.Source: §I · p. 2

Infrastructure changes and acceptance

  • Disclosed agents created new infrastructure more often than humans (34.0% vs. 21.4%), and their within-repository acceptance deficit persisted after adjusting for change type (8.4 percentage points).The study uses an IaC-specific taxonomy and human-audited annotations.Source: §I · p. 2

Sustainability implications

  • Across matched AWS Terraform PR pairs, sustainability smells appeared at similar rates for agents and humans (10.4% vs. 10.3%; paired odds ratio 1.02, 95% CI 0.83–1.25).The study treats a detected smell as an indicator, not a measurement of energy, cost, or waste.Source: §I · pp. 1–2

II. RELATED WORK

Prior work examines AI coding agents and automation in software repositories, alongside IaC quality and generated configurations. This study combines provenance signals to distinguish contributor populations and examine their IaC changes and acceptance.

A. AI Coding Agents and Automation in Software Repositories

  • Prior agent research examined correctness and security, then shifted toward autonomous agents and their PR acceptance, rejection, size, and CI failures.AIDev primarily identifies disclosed contributions, while commit-level traces can reveal undisclosed agent involvement [9].Source: A. AI Coding Agents and Automation in… · p. 2
  • This study combines agent-signature detection, commit-level provenance analysis, and bot identification to distinguish four contributor populations in IaC repositories.It investigates the infrastructure changes they produce and how acceptance varies across repositories and change types.Source: A. AI Coding Agents and Automation in… · p. 2

B. IaC Quality and AI-Generated Configurations

Prior IaC quality research largely examines existing configurations or benchmark-generated code. The study analyzes real PRs, classifies contributor provenance, and screens the corpus for its change and sustainability analyses.

B. IaC Quality and AI-Generated Configurations

  • Earlier work studied IaC smells, security, defects, and generated configurations, whereas this study examines proposed changes in real repositories and matched agent–human PRs.Its sustainability comparison uses the taxonomy of Kosbar and Hamdaqa.Source: B. IaC Quality and AI-Generated… · p. 2

A. Data Collection and Authorship

  • The study collects IaC PRs opened from January 2025 through May 2026, identifying tool-related changes by file path, extension, or content.The collection contains 349,975 PRs from 49,241 repositories across Terraform, Ansible, CloudFormation, Puppet, and Chef.Source: A. Data Collection and Authorship · p. 2

III. STUDY DESIGN

  • Figure 1 outlines the shared study design, with held-out validation covering 240 PRs and an audit sampling 240 annotations per task.Smells are detected in all matched Terraform pairs, while the main analysis uses AWS pairs.Source: §III · p. 2, Figure 1 · p. 3

A. Data Collection and Authorship

  • PR authorship is classified as disclosed agent, silent agent, automation bot, or human using account and metadata signatures, commit traces, and GitHub account type.The final labeled set contains 349,156 PRs after excluding 819 PRs whose commits were no longer reachable.Source: A. Data Collection and Authorship · p. 3
  • For RQ2 and RQ3, the study excludes forks, insecure-by-design and demo repositories, then caps each repository-population group at 50 PRs.This screening yields 237,585 PRs; the top 1% most active repositories otherwise hold 19–36% of a population’s PRs.Source: A. Data Collection and Authorship · p. 3

B. Annotation of Changes and Rejection Reasons

The annotation approach defines independent action and target dimensions for IaC changes, then uses model labels checked against human coding, validation, and an audited correction procedure.

B. Annotation of Changes and Rejection Reasons

  • The change codebook assigns each codable PR an action and target, with security, destructive, and broadened-permission flags cutting across those dimensions.Twelve ordered rules resolve ties; rejection reasons have seven categories.Source: B. Annotation of Changes and Rejection… · p. 3, Table I · p. 4
  • Two models independently label each PR, and a human audit corrects reported label shares for errors found in sampled labels.The models agree on 82% of actions, 74% of targets, and 93% of rejection reasons.Source: B. Annotation of Changes and Rejection… · p. 4
  • On 240 held-out PRs, the primary annotator reached 85% accuracy for actions, 76% for targets, and 92% for rejection reasons.Conclusions from a class require precision and recall of at least 0.70 with at least ten instances; fixes, tests, host targets, and destructive flags are tentative, while security is reported only through error-adjusted estimates.Source: B. Annotation of Changes and Rejection… · p. 4
  • The 480-PR audit estimates weighted accuracy at 88% for actions, 81% for targets, and 88% for rejection reasons, then uses stratified difference estimates to correct label shares.Because interval coverage was low for rare classes, population differences count as established only when conservative and design-based intervals both exclude zero; rare classes such as removals and formatting are not interpreted at population level.Source: B. Annotation of Changes and Rejection… · p. 4

C. Detection of Sustainability Smells

The study operationalizes sustainability smells as static heuristics on parsed Terraform configurations, then compares introduced and removed smells in matched agent–human PRs. The main analysis is limited to matched pairs changing AWS configurations, and the introduced-finding detector was manually validated.

C. Detection of Sustainability Smells

  • Six sustainability smells adapted from Kosbar and Hamdaqa are detected in parsed Terraform configurations using thresholds derived from their examples.The detector was tested on 42 cases.Source: C. Detection of Sustainability Smells · p. 4, Table II · p. 4
  • 2,645 matched pairs that both changed AWS configurations form the main sustainability-smell analysis.The study initially formed 6,688 pairs; 321 could not be analyzed, and Azure or Google Cloud pairs were reserved for sensitivity analysis.Source: C. Detection of Sustainability Smells · pp. 4–5
  • A smell is counted as introduced when it appears between the merge-base configuration and the PR head; unreadable files are excluded from both snapshots.Terragrunt state management and example, test, demo, LocalStack, and state-bootstrap configurations are excluded under specified conditions.Source: C. Detection of Sustainability Smells · p. 5
  • Two authors independently checked up to 30 random introduced findings per smell, with retained smells reaching precision of 0.86 to 1.00.Reviewers were blind to population and recorded repository or code justifications for confirmed smells.Source: C. Detection of Sustainability Smells · p. 5

D. Statistical Analyses

The analyses compare contribution and acceptance across populations, classify change types and rejection reasons, and assess paired sustainability-smell outcomes. Acceptance models progressively adjust for PR and repository differences; the delegation criteria are explicitly exploratory and unvalidated.

D. Statistical Analyses

  • Merge outcomes are modeled against human PRs, first adjusting for PR size, IaC tool, and month, then adding repository fixed effects.Standard errors are clustered by repository; change-type models estimate population differences for the same type.Source: D. Statistical Analyses · p. 5, Table III · p. 5
  • Change-type shares and rejection reasons are compared using error-aware estimates, while raw-label Cramér’s V measures association with contributor population.The table reports error-aware shares of labeled PRs per population; bold marks differences from humans that exclude zero, and † denotes tentative classes.Source: D. Statistical Analyses · p. 5
  • Delegation groups use within-repository acceptance intervals and risk indicators for broadened permissions, possible state destruction, or introduced sustainability smells.The thresholds are exploratory, selected after examining results, and not validated; the groups do not apply to disclosed agents.Source: D. Statistical Analyses · p. 5
  • For lower-acceptance actions, two authors review 20 randomly selected refused agent PRs per action to assess intent, closure reasons, and action-label accuracy.The sample is split evenly between silent and disclosed agents, with at most one PR per repository.Source: D. Statistical Analyses · p. 5
  • Paired agent–human PRs are tested for differences in whether and how many smells they introduce, with paired odds ratios and repository-clustered bootstrap intervals.The analysis also examines removed smells, smells per touched configuration, agent modes, merged-pair subsets, and sensitivity analyses.Source: D. Statistical Analyses · p. 5

IV. RQ1 — IAC CONTRIBUTORS AND MERGE OUTCOMES

Across the labeled IaC PRs, bots and agents are substantial contributor populations, and agent volume grows rapidly. Silent agents lead overall merge rates but lag humans within repositories; agents submit larger PRs and more often change Terraform module skeletons, while bots concentrate on dependencies.

IV. RQ1 — IAC CONTRIBUTORS AND MERGE OUTCOMES

  • 8,057 agent PRs were opened in May 2026, up from fewer than 100 per month before May 2025, while silent-agent PRs had outnumbered disclosed-agent PRs since November 2025.Monthly IaC PR volume rose from 12.7k in January 2025 to 35.4k in May 2026.Source: §IV · p. 6, Figure 2 · p. 6
  • Of 349,156 labeled PRs, 57.1% were human-authored, 31.8% bot-authored, 8.0% silent-agent, and 3.1% disclosed-agent PRs.Silent-agent PRs outnumbered disclosed-agent PRs 2.6 to 1.Source: §IV · pp. 5–6
  • Silent-agent PRs lead overall acceptance at 87.4%, but within repositories they are merged 4.0 points less often than human PRs; disclosed agents lag by 10.4 points.Adjusting for size, IaC tool, and month leaves a 6.2-point silent-agent advantage before repository fixed effects reverse the association.Source: §IV · p. 6
  • Median silent-agent and disclosed-agent PRs changed 302 lines in 8 files and 247 lines in 7 files, respectively, versus 71 lines in 4 files for humans and 26 lines in one file for bots.Agents deliver more code per PR rather than more commits; non-bot populations typically use two commits per PR.Source: §IV · p. 6
  • Silent and disclosed agents changed main.tf in 43% and 48% of PRs, respectively, versus 37% for humans; bots most often changed Terraform dependency files.Bots changed the Terraform lock file in 31% of PRs and provider or backend configurations in 30%.Source: §IV · p. 6, Figure 4 · p. 6

V. RQ2 — CHANGES AND ACCEPTANCE

Within the screened corpus, silent-agent PRs have lower merge rates than human PRs within repositories, and disclosed-agent gaps persist after adjusting for change type. Disclosed agents create new infrastructure more often, while rejection records and exploratory task evidence point to different review needs by action.

Change types

  • 34.0% of disclosed-agent PRs create new infrastructure, versus 21.4% of human PRs; no robust change-type differences are established between silent agents and humans.Disclosed agents extend existing infrastructure less often than humans (7.5% vs. 20.2%); bots mainly make upgrades (88.2%) and act on IaC plumbing (78.5%).Source: §V · p. 7

Acceptance by action

  • Silent-agent PRs are merged 2.1 points less often than human PRs within repositories, and disclosed-agent PRs 8.4 points less often; adjusting for action leaves both gaps unchanged.Disclosed-agent deficits range from 6.1 to 12.2 points across actions except tests, where the estimate is inconclusive; silent-agent deficits concentrate in upgrades, tests, and documentation. These estimates use the screened, capped corpus.Source: §V · p. 7, Figure 5 · p. 7

Refusal reasons

  • Explicit maintainer rejection accounts for 1.4% of refused bot PRs to 6.4% of refused disclosed-agent PRs, but only 19 audited cases make comparisons unreliable.Most refused agent PRs have no stated closure reason (54.0% silent; 75.4% disclosed); many refused agent upgrades instead reflect cleanup or replacement, while documentation and test findings rely on unreliable labels.Source: §V · pp. 7–8, Table VI · p. 8

Delegation evidence

  • Modifications show comparable acceptance and few risk indicators, while creations, extensions, migrations, and removals warrant targeted checks for smells, permissions, and destruction.Creations introduce sustainability smells in 39.4% of agent and 39.8% of human PRs; 24.1% of agent extensions broaden permissions, and roughly a quarter of agent migrations and removals may destroy stateful resources. Acceptance evidence is inconclusive for removals; upgrades merit further evidence because agents are merged less often.Source: §V · p. 8, Table V · p. 8

VI. RQ3 — AGENT VS. HUMAN SUSTAINABILITY SMELLS

In matched agent-human PR pairs from the same repository and period, agents introduce sustainability smells at similar rates. Smells cluster around infrastructure creation and state configuration, with no detectable difference by authorship for the same action.

Overall comparison

  • 10.4% of agent PRs and 10.3% of their human counterparts introduce at least one smell (paired OR 1.02, 95% CI 0.83–1.25).The main analysis covers 2,645 matched AWS pairs; results also hold for silent agents and merged pairs, while the disclosed-agent estimate is less precise.Source: §VI · p. 9, Table VII · p. 9

Smell types and validation

  • Missing remote state is the most common smell, appearing in 7.7% of agent and 7.6% of human PRs; monolithic configurations follow at 2.1% and 2.3%.Two authors confirmed 114 of 116 sampled detected smells (precision 0.98, 95% CI 0.94–1.00); 28 confirmed smells had a stated code or repository justification.Source: §VI · p. 9

Introduction and removal

  • Agent PRs introduced 400 smells and removed 166 (net increase 234), while human PRs introduced 416 and removed 204 (net increase 212); both introduced more than they removed.Removing smells was rare: agent PRs removed at least one in 3.7% of pairs and human counterparts in 4.5%, with no statistically detectable difference (paired OR 0.81, 95% CI 0.60–1.08; p = 0.18).Source: §VI · p. 9

Smells by action

  • 39.4% of agent creations and 39.8% of human creations introduce a smell, and no detectable difference appears between agent and human PRs of the same action (−1.0 points, 95% CI −2.5 to 0.6).Creations account for 62% of agent and 53% of human smell-introducing PRs; modifications, fixes, and upgrades rarely introduce smells (at most 2.7%).Source: §VI · p. 9, Figure 6 · p. 9

Sensitivity analyses

  • No statistically detectable difference appears across sensitivity analyses, including alternative detection rules and 1,707 Azure and Google Cloud pairs.Three of 42 sensitivity comparisons reached p < 0.05 before correction, all for Azure and Google Cloud; none remained significant after correcting for multiple tests.Source: §VI · p. 10

VII. DISCUSSION

Contributor provenance, maintainer acceptance, and sustainability smells capture distinct aspects of agent participation in IaC. The discussion cautions against judging agents by PR metadata or merge rates alone and recommends targeted checks for certain changes.

Interpreting participation

  • PR authorship alone misses silent-agent contributions and misclassifies automation bots, while aggregate merge rates obscure repository differences.Within repositories, disclosed-agent PRs are merged less often for nearly every change type, whereas silent-agent PRs are not; the acceptance gap is not explained by change type.Source: §VII · p. 10

Practical implications

  • Creations, extensions, migrations, and removals warrant targeted checks of resource provisioning, permissions, and destructive changes.Sustainability smells occur at comparable rates in matched agent and human PRs, mostly when infrastructure is created; some smells have stated justifications rather than indicating waste.Source: §VII · p. 10
  • A repository-level CI check for a remote backend would catch most smells, regardless of authorship.Tool vendors can improve transparency through machine-readable agent provenance and clearer explanations of proposed changes.Source: §VII · p. 10

VIII. THREATS TO VALIDITY

The study identifies construct, internal, and external validity constraints. Its findings depend on trace-based agent identification, observational comparisons, and a public GitHub sample dominated by Terraform.

Construct validity

  • Agent use that leaves no trace in PRs or commits is missed, so silent-agent counts are lower bounds; merge outcomes measure acceptance, not quality.Change types and rejection reasons use LLM labels corrected with a stratified human audit, but rare classes remain limited by the small audit.Source: §VIII · p. 10

Internal validity

  • The comparisons are observational, and undetected agent use among human PRs would bias comparisons toward no difference.Diffs were unavailable for 2.7% of PRs, more often for agent PRs, which could bias RQ2 and RQ3 if these differ systematically.Source: §VIII · p. 10

External validity

  • The sample covers public GitHub repositories from January 2025 to May 2026, is dominated by Terraform, and focuses on AWS for RQ3.Private deployments, other tools, and later agent versions may differ; Azure and Google Cloud pairs point in the same direction.Source: §VIII · p. 10

IX. CONCLUSION

The study finds that PR metadata can obscure agent contributions, acceptance varies with how agent work is submitted, and agent and human PRs introduce sustainability smells at comparable rates. It recommends accounting for undisclosed agents in empirical studies and applying IaC checks independently of authorship.

IX. CONCLUSION

  • Agent and human PRs introduce sustainability smells at comparable rates (10.4% vs. 10.3%), with no statistically detectable difference.Smells arise predominantly when new infrastructure is created and reflect project conventions rather than authorship.Source: §IX · p. 10
  • Bots open about a third of PRs attributed to humans, and a substantial share of agent contributions is undisclosed.PR metadata alone does not reliably identify contributor type.Source: §IX · p. 10
  • Within repositories, silent-agent PRs are accepted at rates comparable to human PRs for most change types, while disclosed-agent PRs are merged 6 to 12 points less often.Change type does not explain the disclosed-agent acceptance gap.Source: §IX · p. 10
  • The findings support accounting for undisclosed agent contributions in empirical studies and applying project-wide IaC checks independently of authorship.The study identifies relating IaC changes to deployed resources and operational cost as future work.Source: §IX · p. 10

Related papers

Loading 2610.34613v1…