PaperlayerView exact revision on arXiv

Source-linked AI summary

Merged, Not Measured: An Empirical Study of Performance Issues Fixed by Coding Agents

Zhenyu Qi, Haotang Li, Jinfu Chen, Huashan Chen, Yutong Zhao, Derui Zhu, Tomas Cerny, Bo Liu, Sen He

arXiv:2609.37985v1cs.SE

TL;DR

Studies of human performance fixes and agent-PR acceptance have not established how maintainers respond to agent performance fixes or whether their speed-up claims hold when run. The authors code and re-execute fixes from AIDev v4, finding that acceptance tracks repository and agent history more than coded fix content, and that merging does not guarantee a measured gain.

  • Problem

    Prior studies had not hand-confirmed agent performance fixes, separated fix content from the receiving repository, or re-executed the fixes’ claims.

    Source: §1 · p. 2, §10 · p. 45

  • Method

    The authors filtered AIDev v4 PRs, coded 1,262 performance issues and their tests, and re-executed rejected and merged fixes.

    Source: §1 · p. 3

  • Results

    Acceptance tracked agent and repository history rather than coded fix content, while 9 of 30 merged fixes showed no significant gain or regressed.

    Source: §10 · p. 45

  • Takeaways & Limitations

    Maintainers who re-run claimed speed-ups would put evidence about the fix into the decision, and acceptance-based quality studies need a within-repository control.

    Source: §10 · p. 45

    The merged-fix re-execution sample favored small, self-contained fixes, so its 9-of-30 non-delivery rate may overstate the population’s.

    Source: §9 · p. 45

Abstract

from arXiv · show

Coding agents open pull requests (PRs) that claim to speed up software, but studies of human performance fixes say little about how maintainers respond to such a fix or whether its claim holds. From the 71,677 agent PRs of AIDev v4, a text filter and codebook coding by language models and by the authors select 1,262 performance issues fixed by six agents in 582 repositories. We code each issue and its tests and re-execute 23 rejected and 30 merged fixes. (1) 57% of closed fixes are merged, 61% of rejections give no stated reason, and only 6 of the 23 re-executed rejected claims held under our three-run pilot on mostly agent-built workloads. (2) Acceptance rises with the agent's track record in the repository (31-37% to 70%) and with the repository's pre-opening merge rate on its other agent PRs (33% to 84%). Merged fixes delete a larger share of the lines they change (0.26 versus 0.15), a difference that holds within agent and within repository, with no such difference detected in the coded content, description, tests or measurements. (3) Repeated computation and redundant data processing cause 44% of the issues, and 46% of fixes are architectural-level. (4) Agents change tests in 37% of fixes and 11% carry a performance test or benchmark; of the 30 merged fixes, 18 met our delivery criterion, 3 fell short of the claim, 9 showed no significant gain or regressed, and 14 change behavior on untested inputs. The outcome tracks the repository's history with the agent rather than the coded content of the fix, and a merge does not show that the fix delivers what it claims.

1 Introduction

The study examines how maintainers decide on agents’ performance fixes and whether the claimed improvements hold when run. It finds that acceptance tracks repository and agent history more than coded fix properties, while re-execution shows that merging does not ensure a delivered gain.

1 Introduction

  • Prior research studied human performance fixes and agent-PR acceptance separately, leaving unclear how maintainers respond to agents’ performance fixes and whether their claims hold when run.The authors identify hand-confirmation of each fix, separation of fix content from repository context, and re-execution of claims as gaps in prior studies.Source: §1 · pp. 2–3
  • A text filter followed by codebook coding by language models and the authors selected 1,262 performance issues from 71,677 agent PRs for study.The issues came from six agents in 582 repositories; the study coded issues, fixes, scope, and tests and re-executed rejected and merged fixes.Source: §1 · p. 3, RQ4: What evidence do agents provide… · p. 4
  • Only 6 of 23 re-executed rejected claims held in a three-run pilot, while 57% of closed fixes were merged and 61% of rejections gave no stated reason.The pilot used mostly agent-built workloads.Source: §1 · p. 3
  • Acceptance was associated with the agent’s repository track record and the repository’s prior merge rate on other agent PRs; only deleted-line share persisted within agent and repository.The study detected no acceptance association with coded fix properties, descriptions, tests, or measurements.Source: §1 · p. 3
  • Agents’ fixes were architectural-level in 46% of cases, compared with 27% of human fixes coded by Zhao et al. (2023).Agents addressed similar inefficiencies as humans, most often through caching.Source: §1 · p. 3

RQ4: What evidence do agents provide to support their performance fixes?

  • Agents changed tests in 37% of fixes, but only 11% included a performance test or benchmark, and acceptance was no higher with any kind of test.Of 30 re-executed merged fixes, 18 met the delivery criterion, 3 improved less than claimed, and 9 showed no significant gain or regressed; 14 changed behavior on untested inputs.Source: RQ4: What evidence do agents provide… · p. 3

2 Background and Related Work

The background connects performance measurement, PR acceptance, optimization benchmarks, and studies of human performance fixes. These strands provide the measurement protocol, comparison context, and coding vocabulary for examining agents’ fixes in repositories.

2 Background and Related Work

  • The study brings together four strands of prior work: performance measurement, PR acceptance, optimizer benchmarks, and studies of human performance fixes.The strands inform its re-execution protocol, separation of acceptance from fix content, comparison of merge outcomes with measured speed-ups, and coding scheme.Source: §2 · p. 4

2.1 Performance Analysis

  • Repeated runs and control matter because benchmark comparisons can reverse with run selection, process environment, or workload instability.Open-source projects often use limited performance-testing rigor, and generated benchmarks can suffer from insufficient workloads and instability.Source: §2.1 · pp. 4–5

2.2 LLMs and Agents for Software Engineering Tasks

  • Research on GitHub PRs links acceptance to author standing, while agent-PR studies often pool task types and report limited human review.Prior work reports differing acceptance patterns for bots and humans and has not separated author-standing effects from the content of a particular fix.Source: §2.2 · p. 6

2.3 LLMs and Agents for Performance Optimizations

  • Repository-level optimization benchmarks find agents behind experts, while prior AIDev studies selected performance PRs using task labels or a language-model classifier.This study confirms performance issues by coding, examines repository context alongside fix content, and re-executes rejected claims in the submitting repository.Source: §2.3 · pp. 6–7

2.4 Human Fixes of Performance Issues

  • Studies of human fixes supplied root-cause and scope codes, finding 27% architectural-level fixes and 15% with test changes in Zhao et al. (2023).The human and agent corpora come from different populations, so their shares are not comparisons of the same population under different authors.Source: §2.4 · pp. 7–8

3 Study Design

The study starts with AIDev v4 agent PRs, confirms performance fixes through coding, and applies four research-question analyses to the resulting dataset.

3 Study Design

  • The study starts from AIDev v4 agent PRs, filters for text describing a performance fix, and confirms candidates through codebook coding by language models and then the authors.The authors also code each issue, fix, and its tests.Source: §3 · p. 8
  • The study uses the resulting dataset to answer four research questions, with Sections 3.2–3.5 describing each question’s design and metrics.Source: §3 · p. 8
  • Table 1 presents the progression from AIDev v4 to the study’s analysis sets.Source: Table 1 · p. 9

3.1 Dataset Construction

The dataset construction screens AIDev v4 PRs in stages, then applies language-model and author codebook coding to identify the consensus performance-issue set. The final set contains 1,262 fixes from 582 repositories.

3.1 Dataset Construction

  • The three construction stages reduced 71,677 agent PRs to 7,660 text-filter candidates, 1,613 model-positive candidates, and 1,262 author-confirmed fixes.The consensus set forms the basis of the main tables.Source: §3.1 · pp. 9–11
  • AIDev v4 covers six agents’ PRs in repositories with more than 100 stars, with outcomes recorded through October 24, 2025.Closed PRs without a merge are called rejected; the study distinguishes inactivity-driven closures from person-made closures.Source: §3.1 · p. 9
  • The text screen applied Zhao et al.’s heuristic linguistic patterns plus fix-specific patterns, then combined scores from two logistic regressions to select candidates.The authors retained 7,660 PRs scoring at least 0.5; reported precision and recall for the original patterns apply to issue reports, not this screen.Source: §3.1 · pp. 9–10
  • Two language models independently coded candidates against a codebook, with a third model resolving disagreements; the authors then dual-coded every model-positive fix and resolved disagreements.The author coding confirmed 1,262 of 1,613 fixes, and the two authors agreed on the verdict with κ = 0.91.Source: §3.1 · p. 10
  • For the 351 model-positive fixes not confirmed by the authors, 270 were features or refactors that mentioned performance.The model-positive set’s acceptance rate was 58%, close to the consensus set’s 57%.Source: §3.1 · p. 11
  • The consensus set comprised 488 Copilot, 438 Codex, 147 Devin, 112 Cursor, 60 Claude Code, and 17 Jules fixes.It included 625 merged, 480 rejected, and 157 open fixes; per-file analyses covered 1,259 fixes because patches were missing for three.Source: §3.1 · p. 11

3.2 RQ1: Acceptance and Rejection of the Fixes

The analysis measures how closed fixes are decided and examines rejection reasons, then re-executes a sample of rejected performance claims. Acceptance excludes open fixes, and claim reproduction is judged against study-defined thresholds.

3.2.1 Analysis Design

  • The analysis breaks acceptance down by repository category and agent, examines decision timing and closure type, and codes stated rejection grounds.Repository categories are labeled from repository descriptions, topics, and homepages; two model passes agreed on 85% of primary labels (κ = 0.83).Source: §3.2.1 · pp. 11–12
  • The rejected-fix pilot sampled 30 measured claims, but seven could not run, leaving 23 fixes tested with three repetitions per tree.Workloads came from the PR, a test or benchmark added by the fix, or a small constructed workload; measured improvement was the median base-to-head cost ratio.Source: §3.2.1 · p. 12

3.2.2 Evaluation Metrics

  • Acceptance is the merged share of closed fixes; open fixes are excluded from both numerator and denominator.The analysis also splits outcomes by repository category, agent, and their combination.Source: §3.2.2 · p. 12
  • A claim is reproduced when measured improvement reaches half the claimed improvement, or 1.2 when the claim gives no number.A faster result below the applicable bound is partly reproduced; no improvement or a regression is not reproduced, using thresholds the study identifies as operational choices.Source: §3.2.2 · p. 13

3.3 RQ2: Factors Associated with Acceptance

The study compares merged and rejected fixes across coded fix properties and repository context, progressively controlling for agent and repository differences. It also models acceptance using measured feature blocks.

3.3.1 Analysis Design

  • The pooled comparison tests whether merged and rejected fixes differ in coded content, tests, measurements, diff properties, and review process.Within-agent comparisons address differences in agents’ description templates, while within-repository comparisons use repositories with both outcomes.Source: §3.3.1 · p. 13
  • A blind-pair review compares one merged and one rejected fix from the same repository and agent, with outcomes hidden from the judge.The judge assesses change style, added machinery, readability, description quality, and which fix they would merge or predict was merged.Source: §3.3.1 · p. 14

3.3.2 Evaluation Metrics

  • The study defines deleted-line share as the fraction of changed lines deleted and repository merge rate using agent PR decisions made before the fix opened.An agent’s track record counts that same agent’s PRs previously merged by the repository.Source: §3.3.2 · p. 14
  • Logistic regressions add agent and description, diff shape, added machinery, and repository history as successive feature blocks.Models use 1,102 closed fixes with file data, cluster standard errors by repository, and exclude 157 open fixes.Source: §3.3.2 · p. 15

3.4 RQ3: Content and Reach of the Fixes

The study codes each fix’s inefficiency, resolution, scope, and depth, relating these descriptions to acceptance. It distinguishes architectural-level changes from multi-file changes that include noise or unrelated work.

3.4.1 Analysis Design

  • Each fix is coded by root cause, resolution, scope, and depth, and root causes are crossed with resolutions to identify dominant pairings.Resolutions include mechanisms such as caching or memoization and batching; scope distinguishes functional-level fixes affecting at most one production file from multi-file fixes.Source: §3.4.1 · p. 15
  • Architectural-level fixes change files together to implement one performance mechanism, while functional-level-plus-noise and mixed-purpose changes are distinguished separately.Patterns include change propagation, optimization clones, parallel optimization, classic design patterns, and new shared abstractions.Source: §3.4.1 · p. 16

3.4.2 Evaluation Metrics

  • Architectural-level prevalence uses fixes with file data as its denominator, and extra files that are only noise do not make a fix architectural.Function counts use distinct patch hunk-header contexts and are treated as a lower bound; pattern comparisons use Kruskal–Wallis tests and outcome comparisons use Mann–Whitney U tests.Source: §3.4.2 · p. 16

3.5 RQ4: Tests in the Fixes

The study examines how often fixes change tests, what those tests exercise, and whether test changes relate to acceptance or predict measured improvement. It re-executes a purposive sample of merged fixes under a stricter delivery criterion.

3.5.1 Analysis Design

  • Test analysis records test-file changes, test-case changes, what the tests exercise, and whether they run in CI.Test kinds distinguish performance tests or benchmarks, functional tests, adaptations, and fixes with no test change.Source: §3.5.1 · p. 17, §3.5.2 · pp. 17–18
  • The merged-fix re-execution selects 30 test-changing fixes that can be built and driven by a deterministic in-process harness.Each fix is measured at three workload sizes with 12 interleaved forks per version; the sample excludes settings such as network, GPU, user-interface, or database-dependent changes.Source: §3.5.1 · p. 17

3.5.2 Evaluation Metrics

  • A fix delivers at the primary workload when median cost improves by at least 5%, with Mann–Whitney p < 0.05 and |δ| ≥ 0.147.For a numeric PR claim, the confidence interval must also reach the claimed value; the merged and rejected arms are not compared statistically because their sampling, repetitions, and verdict rules differ.Source: §3.5.2 · p. 18

4 RQ1: How often are agents’ performance fixes accepted or rejected?

Agents’ performance fixes were merged in 57% of closed cases, but rejection threads often gave no reason and re-executed rejected claims usually failed to meet their claims. Decisions were quick, and some re-executed gains came with behavior changes.

4.1 Acceptance Rate by Agent and Repository Category

  • 57% of 1,105 closed fixes merged; acceptance ranged from 30% for Devin to 73% for Codex, Claude Code, and Jules, and varied across project categories.Content, Library, and Server-client fixes were accepted in 67–69% of cases, while Build Management fixes were accepted in 40%.Source: §4.1 · pp. 18–19, Table 6 · p. 19

4.2 Time to Acceptance or Rejection

  • Merged fixes closed in a median 1.2 hours, compared with 24.3 hours for rejections; 37% of rejections were direct, silent closures within 24 hours.Another 26% of rejections were prolonged, taking at least seven days.Source: §4.2 · p. 21, Figure 3 · p. 21

4.3 Reasons for Rejected Performance Issue Fixes

  • 61% of rejection threads stated no reason, including 86% of direct rejections; only 3% asked for a measurement.A human engaged substantively in 34% of rejection threads, and the threads rarely showed whether anyone checked the claimed improvement.Source: §4.3 · p. 22
  • Only 6 of 23 re-executed rejected claims reproduced (26%); 4 partly reproduced and 13 did not, with a median measured speed-up of 1.02× on changed paths.The pilot used changed paths rather than production-scale workloads.Source: §4.3 · p. 22, Figure 4 · p. 23
  • Among the 13 claims that did not reproduce, six ran slower, five showed no measurable change, one crashed, and one lacked the claimed change.Examples included a 2–7× speed-up claim that ran slower at every tested input size and an optimization measured at 0.55×.Source: §4.3 · pp. 22–23
  • A reproduced speed-up did not ensure correct behavior: two of ten reproduced or partly reproduced fixes broke behavior on tested inputs.One fix sped up a bursty stream but stalled terminal animation on a steady stream; another mislabelled computed styles.Source: §4.3 · p. 23
  • Other reproduced fixes delivered concrete gains on their target paths, including 37× fewer Gantt re-renders and 8× fewer allocations per operation.The allocation count fell from 218 to 27 per operation.Source: §4.3 · p. 23

5 RQ2: What factors are associated with the acceptance or rejection of agents’ performance fixes?

Acceptance was associated most strongly with repositories’ prior treatment of agent PRs, while the deleted-line share was the fix property that remained associated within agents and repositories. Description style and coded fix content generally did not distinguish outcomes; these are observational associations, not measured causes.

5.1 Overall Comparison of Merged and Rejected Fixes

  • Root cause, performance category, test co-change, and attached measurements did not reliably separate outcomes once agent identity was considered.Merged and rejected fixes had test co-change rates of 35% and 37%, respectively; the attached-measurement difference was not significant after controlling for agent.Source: §5.1 · p. 24, §5 · p. 24
  • The pre-opening repository merge-rate block raised model pseudo R² from 0.145 to 0.232; deleted-line share was the only stable fix-content term, with OR 10.5 in M4.These are associations in observational data, not measured causes of a merge decision.Source: §5 · p. 24, §5.1 · p. 25, Table 8 · p. 25

5.2 Comparison Within Each Agent

  • Description verbosity differed in pooled comparisons but not within agents, because agent-specific templates aligned with different acceptance rates.Codex bodies were about 40 words, while Copilot and Devin bodies were 250–400 words; the pooled difference did not persist within categories and agents.Source: §5.2 · pp. 25–26, Figure 5 · p. 26

5.3 Whether the Patch Adds or Removes Code

  • Merged fixes had a deleted-line share of 0.26 versus 0.15 for rejected fixes, a difference also found within each of the three largest agents and in 50 of 74 repositories.Within-agent shares were 0.20 versus 0.10 for Copilot, 0.33 versus 0.21 for Codex, and 0.28 versus 0.20 for Devin.Source: §5.3 · pp. 26–27, Figure 6 · p. 27
  • In 33 of 40 blind same-agent, same-repository pairs, a judge selected the merged fix; its reasons included readability and replacing code rather than adding machinery.The coded machinery flags did not distinguish outcomes, and the study notes that the language-model judge and small pair sample limit this evidence.Source: Figure 7 · p. 27, §5.3 · p. 28

5.4 Repository History with Agent PRs

  • Acceptance rose with repository history: 33% in the lowest pre-opening merge-rate band versus 84% in the highest, and 70% after 20 or more prior merged PRs from the same agent.The pre-opening rate’s association was strongest in repositories that decided case by case; the deleted-line share did not distinguish outcomes in the low-rate band.Source: §5.4 · pp. 29–30

6 RQ3: How frequently do agents’ performance fixes involve architectural changes?

Agents often address recurring computation through caching, but many fixes are architectural, spanning files to introduce shared mechanisms. These fixes vary in reach and acceptance across repository categories, while deleted-line share connects the characterization to acceptance patterns.

6.1 Root Causes and How Agents Fix Them

  • Repeated computation and redundant data processing account for 44% of issues; caching or memoization is the most common resolution, at 20%.Repeated computation is resolved by caching in 67% of cases; synchronization by concurrency changes in 84%, and API inefficiency by API replacement in 63%.Source: §6.1 · pp. 31–32
  • Resolution is unrelated to acceptance overall (p = 0.11), except lazy evaluation, accepted 40% of the time versus 57% overall.Allocation reduction has the highest acceptance at 69%.Source: §6.1 · p. 32

6.2 Functional-level and Architectural-level Fixes

  • 46% of fixes are architectural-level; among multi-file fixes, 81% are architectural, while 11% are functional changes plus noise.Architectural-level fixes are accepted 53% of the time versus 60% for the rest (p = 0.044).Source: §6.2 · p. 32
  • Architectural-level acceptance varies by repository category: it is 37% versus 56% for functional fixes in lower-acceptance categories, but 67% versus 70% elsewhere.The lower-acceptance group comprises Language, Big-data and Build Management projects; the other group comprises Content, Library and Server-client projects.Source: §6.2 · p. 33, Figure 9 · p. 33
  • Among architectural-level fixes with GitHub patches, propagated changed interfaces and new shared abstractions each account for 31%, cloned optimizations 23%, and acceptance does not differ across patterns (47–59%, p = 0.48).Source: §6.2 · pp. 33–34

6.3 Reach and Acceptance of Architectural-level Fixes

  • Architectural fixes touch a median of 7 functions across 4 files; new shared abstractions change 258 production lines, while optimization clones change 10 lines per function.Shared abstractions have a 0.10 deleted-line share, compared with 0.40 for clones; within stratified comparisons, deleted-line share is the fix-level feature that separates outcomes.Source: §6.3 · pp. 34–35
  • A shared cache serving several call sites changed 188 lines across five production files and deleted 13; a five-site optimization clone deleted 45 of 101 lines and was merged.The shared-cache change was closed after the maintainer noted that the repository's query system already cached.Source: Figure 10 · p. 34, §6.3 · p. 35

7 RQ4: What evidence do agents provide to support their performance fixes?

The paper examines what tests agents add and whether those tests support performance claims. Few fixes include performance tests, and re-execution finds that many merged fixes do not deliver the claimed gain or change behavior on untested inputs.

7.1 Test Changes in Agent Fixes

  • Agents changed a test file in 37% of fixes, but only 11% included a performance test or benchmark.Among test-changing fixes, 56% added test cases and 25% modified existing ones.Source: §7.1 · pp. 36–37

7.2 The Impact of Tests on Acceptance

  • Only 11% of fixes included a performance test or benchmark, and test kind was not associated with acceptance.Acceptance was 51% with a performance test, 54% with functional tests only, 60% with adaptation only, and 58% with no test.Source: §7.2 · p. 37
  • A sized workload can make a performance test check a bound in CI, but the merged cal.com test's 2,000 ms bound would not catch loss of most of its reported gain.The PR reported a drop from about 6 seconds to 70 ms.Source: §7.2 · p. 37

7.3 Accepted Fixes That Do Not Deliver

  • 18 of 30 merged fixes met the delivery criterion, 3 fell short of the claimed gain, and 9 showed no significant gain or regressed; 14 changed behavior on untested inputs.The tests accompanying the merged fixes would not have distinguished delivered improvements from missing ones.Source: §7.3 · pp. 38, 40, Figure 13 · p. 39
  • The re-executed rejected and merged arms are not comparable: they used different sampling and measurement protocols, so their delivery counts cannot establish whether maintainers recognize effective fixes.The merged arm used 12 forks at three workload sizes; the rejected arm used three repetitions of workloads mostly constructed by the executing agent.Source: §7.3 · p. 39, Table 13 · p. 40
  • In 17 of 28 fixes, the PR's tests also passed on the base tree, and only 2 of 30 failed there for a performance reason.Four fixes' tests failed on the merged head itself; full diff coverage did not ensure delivery.Source: §7.3 · pp. 39–40

8 Discussion

Across the five repositories, agent PRs were less often approved than human PRs, but approved PRs merged at similar rates. The discussion connects acceptance to repository history and argues for measuring fixes rather than treating merges as evidence of delivered speed-ups.

8.1 Implications

  • Agent acceptance varies with repository history and patch shape; deleted-line share is the only fix-level signal that holds within both agent and repository.Smaller fixes that remove code are associated with higher acceptance, but whether changing a fix’s shape changes its outcome remains untested.Source: §8.1 · p. 41
  • A re-runnable measurement could put fix-specific evidence into merge decisions: 61% of rejections state no reason, and reviewers request a measurement in 3%.Of 30 re-executed merged fixes, 18 met the delivery criterion and 9 showed no significant gain or regressed; tests passed on the unfixed code in 17 of 28 cases.Source: §8.1 · p. 41
  • For agent evaluation, 19 of 23 rejected-fix claims were resolved by the pilot, while the merged arm found 8 of 30 fixes slower at one workload size and 14 with unmentioned functional changes.An evaluation using one workload and one number can miss regressions at the PR’s own scale or functional changes on untested inputs.Source: §8.1 · p. 41
  • Agents change tests in more than a third of fixes, but only one in nine includes a performance test; changed tests alone are weak evidence of verification.About a third of performance tests and benchmarks use toy workloads, and some added tests pass on the unfixed base.Source: §8.1 · p. 41

8.2 The Approval Funnel against Human PRs in the Same Repositories

  • 41% of agent PRs versus 80% of human PRs were merged, with the gap appearing at approval; after approval, 95% versus 96% merged.The paired performance-fix set showed the same funnel: 40% versus 79% merged, 24% versus 41% approved, and 95% versus 94% merged after approval.Source: Figure 14 · p. 42, Table 14 · p. 42, §8.2 · p. 43
  • Among unapproved PRs, 61% of human PRs merged anyway, usually by their authors, while only 12% of unapproved agent PRs merged.Counting only merges by someone other than the author, the rates were 39% for human PRs and 35% for agent PRs.Source: §8.2 · p. 43
  • In airbyte, the human-agent merge gap arose largely from authors merging unapproved human PRs; in crewAI, it arose at approval, including for otherwise similar deletions.A two-line deletion rejected under a Devin account later merged in a human-authored PR after approval.Source: §8.2 · p. 43
  • The approval-step pattern differs by account model: operator-held merge accounts are accepted like maintainers’ own work, while bot-account acceptance rises with repository track record.The authors note that what approvers weigh at this step remains a question for interview research.Source: §8.2 · p. 43

9 Threats to Validity

The authors identify threats involving coding and measurement, interpretation of associations, and generalization beyond the sampled fixes and repositories. Several findings rely on dependent robustness checks in one dataset rather than independent replications.

Construct validity

  • Coding and measurement may be biased by model and coder exposure to outcomes, same-vendor judging of Claude Code fixes, and a filter that misses some performance fixes.The text filter’s estimated miss rate is roughly a tenth of the population, so population shares are conditioned on that filter; function counts are a lower bound.Source: §9 · p. 44

Internal validity

  • Pre-opening merge rate reflects maintainers’ or operators’ habits rather than an independent cause, while raw track record mixes prior success with exposure and repository activity.For bot accounts, author association of NONE coincides with an empty track record, making the variables partly collinear.Source: §9 · p. 44

Conclusion validity

  • The central results recur across dependent robustness checks, but subgroup and category comparisons are exploratory and come from one dataset, not independent replications.Dropping the two most concentrated repositories moves pooled acceptance to 55%; isolated content differences are treated as suggestive because they would weaken under correction.Source: §9 · pp. 44–45

External validity

  • The re-execution arms are small and selected differently; the merged arm’s feasibility criteria favor small, self-contained fixes, so its non-delivery rate may overstate the population’s.For 16 of 23 rejected-fix pilot runs, the executing agent constructed a workload on the changed path rather than using one named in the PR.Source: §9 · p. 45

Reliability

  • A rerun of model labeling, pair judging, or execution may produce different outputs; the authors release artifacts to support reproducibility.Released materials include data, codebooks, coding sheets, model outputs, container results, and scripts.Source: §9 · p. 45

10 Conclusion and Future Work

Acceptance tracks repository history with the agent and patch shape, while re-executed merged fixes do not always deliver their claimed benefit. The authors call for controlled tests and broader, comparable re-execution to address the study’s limits.

10 Conclusion and Future Work

  • Acceptance varies with agent track record, repository merge history, and whether a fix removes or adds code, while coded content, descriptions, tests, and measurements show no detected association.When re-executed, rejected claims mostly failed, and 9 of 30 merged fixes showed no significant gain or regressed.Source: §10 · p. 45
  • Because acceptance is confounded with a repository’s prior treatment of an agent, studies using acceptance as a quality measure need a within-repository control.The authors also recommend that maintainers re-run claimed speed-ups before deciding, to put evidence about the fix itself into the decision.Source: §10 · p. 45
  • Whether changing a fix’s shape changes its outcome is untested; the authors propose submitting removing and adding forms of the same optimization to the same repositories.The findings are associations in one snapshot of AIDev v4.Source: §10 · p. 45
  • The re-execution arms are small, use different protocols, and the merged arm was feasibility-selected; the funnel comparison covers five repositories and one paired set.The authors propose a random merged-fix sample under the same three-size protocol and a broader comparison of repositories where humans and multiple agents submit fixes.Source: §10 · p. 46

Related papers

Loading 2609.37985v1…