PaperlayerView exact revision on arXiv

Source-linked AI summary

Co-Modifying Application Code and Deployment Infrastructure: A Causal Study of AI Agent Pull Requests

Mohammad Wathiq Soualhi, Basem Suleiman

arXiv:2610.12849v1cs.SEcs.AI

TL;DR

Benchmarks commonly test agents in isolated sandboxes, while empirical studies lack causal evidence about cross-domain integration overhead and oversight. The authors analyze 26,999 PRs using a deterministic taxonomy and exact matching framework. Dual-scope PRs show a bimodal integration pattern, with reviewed changes delayed and many others merging silently.

  • Problem

    Existing agent benchmarks isolate code generation from collaborative deployment, and repository studies lack causal evidence about integration overhead and maintainer oversight.

    Source: §1 · pp. 1–2

  • Method

    The authors evaluate 26,999 PRs across two datasets using a deterministic four-tier file taxonomy and exact 4D matching that controls for application volume.

    Source: §2.4 · p. 6, §7 · p. 18

  • Results

    Dual-scope PRs show bimodal integration outcomes: explicitly reviewed changes incur a +13.13-hour median delay, while 43.1% to 56.2% merge silently.

    Source: §1 · p. 3, §7 · p. 18

  • Takeaways & Limitations

    The study provides a reusable framework for evaluating cross-domain contributions under conditional exchangeability.

    Source: §7 · p. 18

    Conditioning on successfully merged PRs leaves abandoned PRs unobserved, so latency estimates are a conservative lower bound on integration delay.

    Source: §5 · p. 15

Abstract

from arXiv · show

Current software engineering agents are evaluated predominantly inside isolated sandboxes against synchronous local tests. However, deploying features in collaborative repositories requires modifying continuous integration (CI/CD) pipelines, containers, and provisioning manifests alongside application logic. Because agents lack deployment awareness, attempting cross-domain modifications introduces unverified dependencies into asynchronous integration pipelines. To isolate the causal integration overhead of dual-scope pull requests (PRs)-modifying both application code and deployment infrastructure-we evaluated 26,999 PRs across two large datasets (AIDev v5 and MOSAIC-3M) encompassing six agent architectures and an aligned human baseline. Using a deterministic four-tier file taxonomy and an exact 4D strata-matching framework, we isolated integration metrics under conditional exchangeability while controlling for application volume. We find agents operate primarily as application coders or chore workers, with maintenance tasks capturing 17.70% to 25.41% of agent PRs. Dual-scope modifications represent only 4.48% to 4.55% of agentic PRs, yet trigger bimodal integration outcomes. In the first mode, explicitly reviewed contributions experience protracted delays, adding a median penalty of +13.13 hours and multiplying time-to-merge by 3.24x, driven mechanically by CI/CD and provisioning manifests. In the second mode, maintainer gatekeeping exhibits a structural bypass: even for external contributors lacking repository write access, agent identity reduces the odds of human review by 63.7% compared to human peers. Consequently, 43.1% to 56.2% of dual-scope PRs bypass inspection to merge silently in a median of under 31 minutes. These uninspected modifications compromise build verification, demonstrating the need for deployment-aware benchmarks and strict path-based access controls.

1 Introduction

Existing agent benchmarks isolate code generation from collaborative deployment, and empirical studies have lacked causal evidence about cross-domain integration overhead and maintainer oversight. This study measures dual-scope pull requests and finds both substantial delays under review and frequent silent merges.

1 Introduction

  • Existing benchmarks test agents in isolated sandboxes, while prior repository studies lack conditional exchangeability to separate integration penalties from structural confounders.The paper frames deployment infrastructure as the operational context needed to deploy functional application code.Source: §1 · pp. 1–2
  • Dual-scope modifications add +13.13 hours to median merge latency and multiply time-to-merge by 3.24× in AIDev v5.The paper attributes the delay primarily to CI manifests (3.8×) and provisioning (2.7×).Source: §1 · p. 2
  • Dual-scope infrastructure modifications make up 4.48% to 4.55% of agent-authored PRs, versus 17.70% to 25.41% for non-functional repository maintenance chores, yet introduce statistically significant integration friction.Source: §1 · p. 2
  • 43.1% to 56.2% of agent-authored dual-scope PRs merge silently in under 31 minutes, while explicitly reviewed contributions take 15.59 to 49.80 hours.Even for external contributors with zero repository write access, agent identity reduces the odds of human review by 63.7%.Source: §1 · p. 3
  • The authors contribute a deterministic four-tier file taxonomy and exact 4D matching framework to measure dual-scope PRs while controlling for application volume.Their stated contributions also include decomposing latency bottlenecks and quantifying review deficits.Source: §1 · pp. 2–3

2 Empirical Study Design

The study builds a temporally bounded, cross-dataset cohort and a deterministic file-domain classification, then matches dual-scope PRs to application-only controls within exact strata. It defines the cohort, classification, matching, and outcome measures to support comparisons of integration latency and human oversight.

2 Empirical Study Design

  • The study evaluates 26,999 PRs from AIDev v5 and MOSAIC-3M using a multidimensional observational design to estimate infrastructure co-modification overhead.Table 1 summarizes dataset provenance, experimental controls, and the cohort-filtering funnel.Source: §2 · p. 3, §2.1 · p. 3, Table 1 · p. 4

2.1 Target Datasets and Repository Quality Filtering

  • Both datasets are restricted to repositories with over 100 stars; MOSAIC-3M includes five agent architectures and a 2,574-PR human baseline, while AIDev v5 uniquely includes Cursor.The datasets share five agent architectures for cross-dataset validation.Source: §2.1 · p. 3

2.2 Temporal Cohort Controls and Data Attrition Audit

  • With a 31-day observation buffer mitigating right-censoring, escalating API-payload attrition (65.64% by March 2026; 117,292 omitted PRs) led authors to replace full 15-month evaluation with June-to-August cross-validation.In the aligned target window, 18.66% of merged AIDev v5 PRs lacked usable file paths; 99.78% of permanently dropped records belonged to Claude Code.Source: §2.2 · p. 4

2.3 Deterministic File Domain Taxonomy

  • A deterministic four-tier classifier prioritizes explicit infrastructure manifests and test assets before contextual infrastructure, application files, and auxiliary maintenance.It replaces NLP task labels with file-level classification, achieving 100% classification coverage; test assets are intercepted before contextual infrastructure classification.Source: §2.3 · p. 5
  • A dual-scope PR must modify at least one valid application file and one valid infrastructure file, with at least two modified files overall.Pure documentation or lockfile changes are auxiliary maintenance and are excluded from causal matching.Source: §2.3 · pp. 5–6

2.4 Causal Identification and Exact Strata Matching

  • Greedy 1:1 matching pairs dual-scope PRs with application-only controls within exact strata for agent, archetype, test maturity, and language, using application-file volume and a 0.28-standard-deviation caliper.Matching is without replacement; the caliper pruned 29.82% of AIDev v5 and 33.88% of MOSAIC-3M agent treatment units.Source: §2.4 · pp. 6–7
  • The study estimates latency and review outcomes using paired statistical tests and regression, and applies the same matching protocol to the MOSAIC-3M human baseline.Silent Integration means a PR merges with zero external human comments and zero external review iterations; human-to-human matching retained 79 balanced pairs.Source: §2.4 · pp. 7–8

3 Empirical Results

The empirical results show that dual-scope PRs are uncommon, but their integration outcomes diverge: AIDev v5 PRs take longer to merge, while review of agent-authored changes varies across datasets. Infrastructure changes concentrate in CI/CD and provisioning, and a substantial share of agent PRs are repository-maintenance chores.

3.1 RQ1: Empirical Scale and the “Chore Agent” Phenomenon

  • Infrastructure changes are active authoring: 97.45% of AIDev v5 and 99.01% of MOSAIC-3M dual-scope PRs add or modify infrastructure files.CI/CD pipelines account for 77.12% of AIDev v5 and 71.79% of MOSAIC-3M infrastructure co-modifications; categories are non-mutually exclusive.Source: §3.1 · p. 8, Figure 1 · p. 9
  • The apparent 17.87% AIDev v5 dual-scope prevalence for Claude Code is a survivorship-bias artifact; the authors use MOSAIC-3M for its baseline observations.The passage attributes the artifact to upstream omission of 5,004 Claude Code PRs, or 94.50% data loss.Source: §3.1 · p. 8
  • Dual-scope PRs account for 4.48% of AIDev v5 and 4.55% of MOSAIC-3M agentic PRs; pure maintenance chores account for 25.41% and 17.70%, respectively.The dual-scope changes are often targeted: 47.0% to 63.9% modify one infrastructure file, overwhelmingly a CI/CD manifest.Source: §3.1 · p. 9

3.2 RQ2: Integration Latency and Contextual Bottlenecks

  • In AIDev v5, dual-scope PRs add +13.13 hours median merge latency and multiply time-to-merge by TR 3.237 (p<0.001); MOSAIC-3M shows no significant penalty (TR = 0.794, p = 0.544).Figure 2 compares the matched control and treatment latency curves; the MOSAIC-3M estimate reflects 80 matched pairs.Source: Figure 2 · p. 10, §3.2 · p. 10
  • CI/CD manifests and system provisioning multiply merge time by TR 3.827 (p<0.001) and TR 2.727 (p=0.0070), respectively; automation scripts and containerization show no significant latency penalty.Procedural automation scripts nevertheless draw the most review discourse: 3.11 external review iterations in AIDev v5 and 3.42 in MOSAIC-3M.Source: §3.2 · p. 11, Table 3 · p. 11, §3.3 · p. 12

3.3 RQ3: Review Friction and the Intervention Bifurcation

  • In AIDev v5, dual-scope PRs lower silent integration from 47.2% to 43.1%, while in MOSAIC-3M external review falls from 52.5% to 36.2%.Human review rates remain statistically flat when developers co-modify infrastructure, shifting from 72.2% to 73.4%.Source: §3.3 · p. 12
  • Among external contributors with zero repository write access, agent identity reduces the odds of human review by 63.7% compared with human contributors (OR = 0.363, p<0.001).The regression controls for application volume, line churn, test inclusion, and repository popularity.Source: §3.3 · p. 12
  • Silent dual-scope PRs merge in median times of 0.43 hours in AIDev and 0.51 hours in MOSAIC, whereas explicitly reviewed ones take 49.80 and 15.59 hours, respectively.The split links review bypass with very rapid integration and explicit maintainer intervention with longer latency.Source: §3.3 · p. 13

4 Discussion and Implications

The discussion argues that sandbox-focused evaluation misses the integration and governance costs of deployment-aware agent contributions. Dual-scope PRs can be delayed when reviewed or merge rapidly without inspection, motivating changes to evaluation, agent tooling, and repository controls.

4.1 The DevOps Integration Gap: Operational Context vs. Sandbox Benchmarks

  • Sandbox benchmarks omit asynchronous integration, where dual-scope PRs add a +13.13-hour median penalty and multiply time-to-merge by 3.24× in AIDev v5.Agents lack mechanisms to ingest and resolve remote build failures, leaving maintainers to translate pipeline failures into source-code changes.Source: §4.1 · pp. 13–14
  • In AIDev v5, strict dual-scope modifications delayed integration to a median of 19.64 hours, versus 1.13 hours for weak dual-scope modifications.CI/CD manifests drive temporal stalling, while procedural automation scripts trigger the most external review iterations.Source: §4.1 · p. 13
  • 17.70% to 25.41% of agentic PRs are pure repository-maintenance chores, a pattern the discussion links to agents’ limited deployment awareness.README.md files alone comprise 25.0% to 44.7% of this maintenance slice.Source: §4.1 · p. 14

4.2 The Intervention Bifurcation and Supply Chain Vulnerabilities

  • 43.1% to 56.2% of agentic dual-scope PRs merged silently in under 31 minutes, while others stalled for 15.59 to 49.80 hours.The review split varies across datasets: explicit review rates rose from 47.5% to 53.1% in AIDev v5 but fell from 52.5% to 36.2% in MOSAIC-3M.Source: §4.2 · p. 14
  • For external contributors without write access, agent identity reduces the odds of human review by 63.7%, and unreviewed CI changes can compromise build verification.A PR that alters its own validation workflow can receive green checks without independent assurance of build integrity.Source: §4.2 · p. 14

4.3 Actionable Implications for Tooling, Governance, and Evaluation

  • The authors call for deployment-aware benchmarks, local pipeline verification, and path-based controls that require human sign-off on sensitive infrastructure changes.They also recommend applying the matching framework to unmerged PRs and studying why maintainers bypass review.Source: §4.3 · pp. 14–15

5 Threats to Validity

The study’s estimates are bounded by measurement and selection choices, residual confounding, and the populations represented in the matched cohorts. The authors therefore limit their generalization to typical PRs in mature collaborative repositories.

Construct Validity

  • Because the analysis conditions on successfully merged PRs, abandoned PRs are unobserved and reported latency is a conservative lower bound.Governance stratification is also limited to MOSAIC-3M because AIDev v5 lacks author-permission metadata.Source: §5 · p. 15

Internal Validity

  • Unobserved developer intent and offline communication remain potential confounders, while human baseline PRs may include uninstrumented AI assistance.The authors characterize possible AI contamination as making observed review deficits a conservative lower bound.Source: §5 · p. 16

External Validity

  • Strict caliper matching excludes outlier file volumes, so estimated effects apply to typical PRs rather than extreme repository-wide refactorings.The >100-star filter also limits generalization to mature collaborative open-source repositories, not necessarily hobby projects or proprietary enterprise monorepos.Source: §5 · p. 16
  • The June–July 2025 cohort may not reflect future agents that autonomously ingest asynchronous CI failures, and the taxonomy requires path tuning for non-standard monorepos.Evaluation-tool coverage is asymmetric across datasets, and large cohorts are needed to withstand the matching framework’s dimensionality requirements.Source: §5 · p. 16

6 Related Work

The related work contrasts isolated agent benchmarks and unadjusted repository mining with this study’s live-repository, matched evaluation. It also connects review bypass to circular CI verification and supply-chain risk.

6.1 From Execution Sandboxes to Socio-Technical Deployment Context

  • Sandbox benchmarks separate code generation from asynchronous integration, although real software engineering requires application logic and deployment infrastructure to co-evolve.This separation can leave a blind spot similar to patches that pass local tests but fail beyond the immediate test oracle.Source: §6.1 · p. 17
  • In live repositories, dual-scope contributions multiply time-to-merge by 3.24×, while CI manifests carry a 3.8× penalty that local sandboxes miss.The section argues that benchmarks should account for real-world deployment dependencies, not just local pass rates.Source: §6.1 · p. 17

6.2 Empirical Mining of Automated Code: Rule-Based Bots to Causal Agent Analytics

  • 17.70% to 25.41% of merged agentic PRs are pure non-functional maintenance, paralleling earlier automated bots more than autonomous feature developers.Prior repository studies also risk conflating infrastructure complexity with PR size when analyzing raw outcomes.Source: §6.2 · p. 17
  • The study uses a deterministic four-tier taxonomy and exact four-dimensional matching to isolate integration costs while controlling for application file volume.Matching strata are Agent, Architectural Archetype, Test Verification, and Language Ecosystem, with an application-file-volume caliper.Source: §6.2 · p. 17

6.3 Modern Code Review, Automated Reliance, and Circular CI/CD Verification

  • Agentic dual-scope PRs exhibit a review bifurcation: explicitly reviewed contributions stall for 15.59 to 49.80 hours, while 43.1% to 56.2% merge without inspection.Even among external contributors without repository write access, agent identity reduces the odds of human review by 63.7%.Source: §6.3 · p. 18
  • When an unreviewed PR changes its own CI validation logic, passing checks provide zero independent assurance of build integrity.The section relates this circular verification paradox to Poisoned Pipeline Execution, in which workflow manipulation can compromise build runners.Source: §6.3 · p. 18

7 Conclusion

The conclusion presents a matched study of agent contributions and their integration outcomes across two datasets. It finds a split between delayed reviewed changes and rapidly merged, uninspected changes, and calls for deployment-aware evaluation and repository safeguards.

7 Conclusion

  • The study evaluates 26,999 PRs across AIDev v5 and MOSAIC-3M using a deterministic four-tier taxonomy and exact 4D strata matching.The design controls for application volume under conditional exchangeability.Source: §7 · p. 18
  • When agents co-modify infrastructure, reviewed PRs add a median +13.13-hour delay and 3.24× time-to-merge in AIDev v5, while 43.1% to 56.2% merge silently in under 31 minutes.Agent identity reduces human-review odds by 63.7% for external contributors, and the delays are driven by CI manifests and system provisioning.Source: §7 · p. 18
  • The authors call for local container dry-runs, path-based CODEOWNERS rules, deployment-aware benchmarks, and study of unmerged PRs and review bypass.They also identify qualitative field studies as a way to examine why maintainers systematically bypass review for automated dual-scope contributions.Source: §7 · p. 18
Loading 2610.12849v1…