PaperlayerView exact revision on arXiv

Source-linked AI summary

VeeReview: Measuring Build-Verify Pairs in Agent-Authored Pull Requests from a V-Model Perspective

Gabriel Dautovi, Jie JW Wu

arXiv:2610.29143v1cs.SEcs.CL

TL;DR

Benchmarks and pull-request outcomes do not directly assess whether verification evidence addresses a proposed change. VeeReview classifies build–verify pairs in pull-request artifacts and compares matched agent- and human-authored requests. Agent-authored requests received lower build-only rates, but the authors treat these as instrument-specific observations and find the rule-decided pair indistinguishable after standardization.

  • Problem

    Task-resolution benchmarks and pull-request outcome studies do not directly assess whether submitted verification evidence addresses the change, leaving this relationship unclear.

    Source: §1 · p. 2

  • Method

    VeeReview classifies four build–verify pairs using repository signals, model judgments for three pairs and a separate diff-based rule for Module Design ↔ Unit Test.

    Source: §1 · p. 2

  • Results

    Agent-authored pull requests had 3.3 to 13.3 percentage points lower build-only rates across all four pairs, while standardization reduced differences by 34% to 60% and the rule-decided pair’s interval included zero.

    Source: §9 · pp. 37–38

  • Takeaways & Limitations

    The rule-decided pair indicates that a language-model-judged process measure partly reflects pull-request presentation, motivating a mechanically decided component.

    Source: §9 · p. 38

    Only 62 of 958 repositories contain multiple agents, leaving agent identity strongly confounded with repository context.

    Source: §6.3 · p. 32

Abstract

from arXiv · show

Context: Reviewing a pull request requires verification evidence that addresses the proposed change. Coding agent benchmarks and studies of pull request outcomes do not directly assess this relationship. Objective: We test whether the V-Model's build and verify pairs can measure verification evidence in pull requests and support comparisons between agent-authored and human-authored changes. Method: VeeReview classifies four build and verify pairs using repository signals and a language-model judge. Three pairs use model judgements constrained by deterministic signals, and a rule over the diff and repository tree decides Module Design-Unit Test. We evaluated VeeReview against a 25-pull request reference set labelled by two raters and applied it to 1,454 matched pairs of agent and human pull requests from 741 repositories. Results: VeeReview assigned lower build-only rates to agent-authored pull requests on all four pairs, contrary to the preregistered hypothesis (mixed-effects odds ratios 0.52 to 0.86). Architecture-Integration was not significant. Standardization for linked issues, shipped tests and size reduced matched-rate differences by 34% to 60%, and the rule-decided pair's standardized interval included zero. The instrument did not meet its preregistered calibration criterion: agreement with adjudicated labels was 4% to 60% against a required 80%, and inter-rater agreement was weak (kappa = 0.154). Conclusions: The rule-decided pair shows that a process measure judged by a language model partly reflects how a pull request is presented rather than the verification work behind it. We report the cohort differences as observations made with this instrument, and we argue that such measures need a mechanically decided component and a validated human reference before they can support comparisons of verification practice.

1 Introduction

Pull-request outcomes and task-resolution benchmarks do not directly measure whether verification evidence addresses a proposed change. VeeReview applies V-Model build–verify pairs to compare agent- and human-authored pull requests, but its findings remain observations from an instrument that did not meet its calibration criterion.

1 Introduction

  • Existing task-resolution benchmarks and pull-request outcome studies do not directly assess whether submitted verification evidence addresses a change.The missing measure matters to reviewers deciding whether a proposed change has been adequately verified.Source: §1 · p. 2
  • VeeReview classifies four build–verify pairs using repository signals and model judgments, with a separate rule deciding Module Design ↔ Unit Test from changed files and repository-tree heuristics.The rule-decided pair permits comparison with model-judged states using a model-independent procedure.Source: §1 · pp. 2–3
  • The study matches agent-authored pull requests with human-authored requests in the same repositories on time and approximate size, while noting that matching does not make cohorts interchangeable.It analyzes 1,454 pairs from 741 repositories and standardizes for issue linking, shipped tests and size.Source: §1 · pp. 2–3
  • Agent-authored pull requests had 3.3 to 13.3 percentage points lower build-only rates across all four pairs, contrary to the preregistered hypothesis; three differences were supported by the mixed-effects analysis.Standardizing for linked issues, shipped tests and size reduced matched-rate gaps by 34% to 60%, and the standardized interval for the rule-decided pair included zero.Source: §1 · p. 3
  • Agreement with adjudicated labels ranged from 4% to 60%, below the required 80%, while inter-rater agreement was weak (κ = 0.154).The authors therefore report cohort findings as observations made with this instrument, not confirmed differences in verification practice.Source: §1 · pp. 3–4
  • The paper contributes an inspectable, reproducible procedure and reports its repeatability and human-rater agreement, while recognizing verdict instability across identical reruns.The released code and configuration support further validation; the calibration results qualify the cohort comparison.Source: §1 · pp. 3–4

2 Background

The background uses the V-Model’s same-level pairing principle to organize verification evidence visible in pull-request artifacts, without treating it as a required lifecycle. It also frames pull requests as the unit for comparing agent and human work, while recognizing that available descriptions and evidence can differ by workflow.

2.1 The V-Model and its pairing claim

  • The V-Model pairs development and verification activities at corresponding abstraction levels, defining four categories that organize evidence such as unit tests for modified logic.The paper uses this relationship to classify evidence rather than to measure adherence to a V-Model lifecycle.Source: §2.1 · pp. 4–5, Figure 1 · p. 4
  • Each applicable pair is classified separately as matched, build-only or verify-only; three model-judged pairs can also be unverified, while out-of-scope pairs are not applicable.The preregistered outcome is the build-only rate among applicable pairs, whose assessed subsets may differ between cohorts.Source: §2.1 · p. 5, Table 1 · p. 5
  • Four pairs are scored, and a deterministic diff-based rule decides Module Design ↔ Unit Test independently of pull-request wording.The rule uses repository-tree heuristics; an existing related test can produce a matched result even when it was not updated, and a filename match does not prove the test exercises changed logic.Source: §2.1 · pp. 5–6, Table 2 · p. 6, Figure 2 · p. 6
  • The claim concerns review-visible build–verify evidence, not lifecycle compliance; verification outside retrieved artifacts may go undetected.An existing test can satisfy the instrument’s rule even if it was not created or updated with the change.Source: §2.1 · p. 7

2.2 Agent-authored pull requests and the AIDev dataset

  • The study samples agent-authored requests from AIDev and constructs a human cohort because AIDev lacks a comparable human cohort.Sampling reflects uneven agent frequencies, repository clustering, matching eligibility and scoring costs; 2,194 of 3,760 sampled agent requests had no eligible human candidate.Source: §2.2 · p. 7
  • Pull requests are the unit of analysis, and their GitHub artifacts provide evidence for both build and verification without requiring agent instrumentation.Workflows may differ in descriptions, issue references and other review artifacts, and the study does not observe harness settings or author intent directly.Source: §2.2 · pp. 7–8

3 VeeReview and Its Build and Verify Classification Procedure

VeeReview classifies pull-request evidence through deterministic signals, model judgments, and a model-independent rule for one build–verify pair. Its outputs distinguish evidence states and retain model claims that cannot be substantiated as unverified.

3 VeeReview and Its Build and Verify Classification Procedure

  • VeeReview’s pipeline combines deterministic signals, model judgments, and a rule-decided pair; its run infrastructure is designed to keep verdict corpora comparable.The model cannot overturn facts established by deterministic rules.Source: Figure 3 · p. 8, §3 · p. 8

3.1 Overview and definitions

  • A pair is matched when both build and verification halves are present, build-only or verify-only when just one is present, and not applicable when neither applies.Model-judged claims without cited evidence become unverified; the Module Design–Unit Test pair is decided by a rule.Source: §3.1 · pp. 8–9

3.2 Inputs

  • VeeReview retrieves pull-request metadata, diffs, review submissions, head-commit CI checks, linked-issue information, and the repository tree for candidate existing tests.Diffs use a fixed 120,000-character budget, and the export records truncation.Source: §3.2 · p. 9

3.3 Deterministic signals

  • Deterministic signals classify changed paths and test levels, detect tests in the diff, retrieve CI results, and parse linked issues.A comment-only check prevents documentation-only edits in source files from counting as source changes.Source: §3.3 · p. 9

3.4 Model-judged pairs and the evidence check

  • For three pairs, the model interprets the diff and description, while an evidence check verifies that cited paths or artifacts occur in retrieved material.This check validates references, not whether a cited test covers changed behavior or acceptance evidence is adequate.Source: §3.4 · p. 10

3.5 The rule-decided pair

  • A deterministic rule assigns Module Design–Unit Test from changed production files, tests in the diff, and filename-based repository-tree heuristics.A potentially related test filename does not establish coverage of modified logic.Source: §3.5 · p. 10

3.6 Outputs

  • Each pair exports applicability, both half-states, a match state, a score, a color band, a finding, and supporting references.Inferential analyses use match states; numerical scores are summarized only at cohort level because individual scores are unstable.Source: §3.6 · p. 10

3.7 Design evolution

  • Design changes included splitting the original three pairs into four, constrained output, evidence checks, one rule-decided pair, and configuration-aware caching.The all-rule alternative missed its preset agreement threshold, while majority voting did not justify its inference cost; the authors also dropped repository-level analysis to keep focus on individual pull requests.Source: §3.7 · p. 11

4 Study Design

The study preregistered questions and analysis choices, sampled agent-authored pull requests, matched human-authored requests, and evaluated the instrument before comparing cohorts. Its design also documents deviations from the preregistration.

4 Study Design

  • The study preregistered its research questions, analysis choices, and falsification criteria; its pipeline samples, matches, and scores pull requests.Documented deviations and post-result analyses are distinguished from preregistered tests and robustness checks.Source: §4 · p. 11, Figure 4 · p. 12

4.1 Research questions and hypotheses

  • The preregistered questions address cohort differences in pair-level match states, where build-only rates occur, and how pull-request size relates to matching.The RQ1 hypothesis predicted higher agent build-only rates, especially for Module Design–Unit Test.Source: §4.1 · p. 12

4.2 Agent cohort

  • The agent cohort was sampled from a pinned AIDev revision, with proportional agent strata and minimum sizes for the two smallest strata.Each sampled request has a sampling weight, and the frozen sample reproduces from its seed.Source: §4.2 · p. 12

4.3 Matched human cohort

  • Human requests were selected from the same repositories within ±90 days, prioritizing fixed size-band matches; unmatched agent requests were dropped.The resulting 1,454 complete pairs cover 741 repositories, or 39% of the agent sample; matching did not equalize issue linking, test shipping, change purpose, or unobserved verification practices.Source: §4.3 · p. 13, Table 3 · p. 14

4.4 Instrument configuration and run

  • Both cohorts were scored with the same locally served judge model, prompt, character budget, temperature-zero setting, and one draw per pull request.One draw was used to limit cost, and the study separately measured verdict instability.Source: §4.4 · p. 14

4.5 Instrument evaluation protocol

  • Instrument evaluation tested rerun consistency and agreement with human labels; the preregistered calibration criterion required at least 80% agreement for every applicable pair.Two raters independently labelled 25 pull requests, and the first author joined them to resolve disagreements, which could influence adjudicated labels.Source: §4.5 · pp. 14–15

4.6 Analysis methods

  • The preregistered analysis used mixed-effects logistic regression with repository random intercepts, alongside methods addressing clustering and matched-pair dependence.The plan also used repository-bootstrap intervals and paired tests, and tested agent-by-pair profiles on a log-odds scale.Source: §4.6 · pp. 16–17

4.7 Deviations from the preregistration

  • The final analysis restored the preregistered mixed-effects model as primary and reported that Architecture–Integration was not significant after adjustment.The preregistered design and falsification rules covered three pairs although the instrument had four; its null-based rule could not distinguish predicted from reversed effects.Source: §4.7 · p. 17

5 Results

The results report VeeReview’s instrument-generated observations, including calibration and matched-pair findings, alongside patterns in matched classifications across pull-request size bands in the agent corpus. These findings are presented with substantial rerun variation and limits on interpretation.

5.1.1 Rerun consistency

  • Across 19 rerun pull requests, overall scores agreed across all three draws for 5 (26%), complete verdicts matched for 33%, and match state changed in 56 of 228 comparisons.The rule-decided pair changed state on 0% of reruns versus 26% to 39% for model-judged pairs; this shows repeatability, not validity, and the cause of model-judged variation was not isolated.Source: §5.1.1 · p. 18

5.1.2 Agreement with human raters

  • Across four pairings, VeeReview’s agreement with adjudicated labels was 4%, 32%, 60%, and 44%, all below the required 80%; inter-rater κ was 0.154, so calibration does not establish accuracy.Because raters often disagreed, disagreement with the instrument cannot be attributed solely to instrument error.Source: §5.1.2 · pp. 18–19

5.1.3 Interpretation scope

  • The remaining cohort results are reported as instrument-generated observations, not established differences; RQ1 covers a selected subset of 1,454 of 3,648 scored agent pull requests.The matched subset differed from unretained pull requests in issue links, test shipping, merge rate, repository contribution, and agent mix; the authors do not use calibration asymmetry to correct its estimates.Source: §5 · p. 18, §5.1.3 · p. 20, §5.2 · p. 25

5.2 RQ1 Prevalence

  • Agent pull requests had lower build-only rates on all four pairs; odds ratios ranged from 0.52 to 0.86, with three significant after correction, and matched-rate difference intervals excluded zero.The matched-rate differences were +16.3 to +18.0 percentage points for model-judged pairs and +6.1 points for the rule-decided pair. These are observations from the matched subset, not validated cohort differences.Source: §5.2 · pp. 20, 21, 25, 26, Table 5 · p. 21
  • Standardization reduced matched-rate differences by 34% to 60% for model-judged pairs, while the rule-decided pair’s interval included zero under both specifications.The rule-decided difference was −1.2 points [−3.9, +2.4] with the full stratum and +3.1 points [−0.4, +7.1] without test shipping.Source: §5.2 · pp. 22, 23, 26, Table 8 · p. 24

5.3 RQ2 Locus

  • Across the agent corpus, Requirement ↔Acceptance had the highest build-only rate at 71.6% of assessed rows, while Module Design ↔Unit Test was lowest at 37.2%; middle-pair ordering was less clear.The endpoints held their ranks in every repository-bootstrap resample. Agent profiles differed, but comparisons were largely between different repositories, so the authors report them descriptively rather than as rankings.Source: §5.3 · pp. 26, 28, Table 9 · p. 27

5.4 RQ3 Size

  • Not-applicable classifications fell from 45.8% in the smallest band to 18.5% at 1,000–5,000 lines, and size patterns were similar across lines changed, diff characters, and files touched.Applicability also varied with breadth: 52.9% of single-file pull requests were not applicable, versus 14.9% of pull requests touching 5–19 files.Source: §5.4 · pp. 28–29
  • 22.6% of rows in the 1,000-to-5,000-line band and 68.2% in the largest band were truncated, limiting interpretation of the largest-band results.Among untruncated largest-band rows (n = 136), the matched share was 30.1% versus 32.5% overall, and build-only was 45.6% versus 31.8%; this comparison cannot bound truncation’s effect because the rows also differ in size and potentially content.Source: §5.4 · pp. 28–29, Table 10 · p. 29
  • 13.3% of pair rows were matched below 50 lines, rising to 38.0% at 1,000–5,000 lines before declining to 32.5% above 5,000 lines.The largest-band decline is observed, but extensive truncation limits its interpretation.Source: §5.4 · pp. 28–29, Table 10 · p. 29
  • The repository-level permutation test did not detect a size-band-by-agent interaction (observed statistic 0.40, null 95th percentile 1.03, p = 0.91), but this is not evidence of equal agent relationships.A within-repository test could not be estimated because only one agent met the cell-size threshold across all five bands in the 62 multi-agent repositories.Source: §5.4 · p. 29
  • Weighting preserved the rise through the 1,000-to-5,000-line band and decline above it, with the largest changes in bands containing strata sampled with floors.Weighted versus unweighted matched shares were 13.4% versus 13.3% at XS, 27.4% versus 27.2% at S, 34.3% versus 34.3% at M, 34.6% versus 38.0% at L, and 27.5% versus 32.5% at XL.Source: §5.4 · p. 29

6 Discussion

The discussion considers verification work, judge-visible artifacts, and applicability as possible explanations for the cohort difference, but the study cannot distinguish among them. It cautions against treating the measure as validated evidence of verification practice and draws implications for review, agent evaluation, and the V-Model taxonomy.

6.1 Three Explanations for the RQ1 Difference

  • The standardized rule-decided gap includes zero, while model-judged gaps remain; calibration and applicability differences prevent identifying which explanation accounts for the cohort contrast.The rule detects test-file presence and filename correspondence, not actual coverage, and the comparison does not establish cohort equivalence.Source: §6.1 · pp. 30–31

6.2 Implications for reviewing agent-authored pull requests

  • Reviewers should check whether cited artifacts address changed behavior, because fluent descriptions and linked issues do not establish verification.Weak agreement on Requirement ↔Acceptance prevents using its build-only rates to prioritize review effort; the study did not measure defects, reviewer effort, or post-merge outcomes.Source: §6.2 · p. 32

6.3 Implications for agent and harness builders

  • The five agent cohorts differ in issue-linking and test-shipping rates, but repository context and unobserved harness configurations prevent attributing those patterns to harness design.Only 62 of 958 repositories contain multiple agents, so Figure 8 is descriptive rather than an agent ranking.Source: §6.3 · p. 32

6.4 Implications for evaluating agents

  • Task-resolution benchmarks and repository-based verification measures target different outcomes, but real artifacts alone do not validate a process measure.The deterministic pair helps identify results that depend on reading prose, yet its heuristics also need evaluation against an independent reference.Source: §6.4 · pp. 32–33

6.5 The V-Model as a measurement lens

  • The four V-Model pairs distinguish evidence at requirement, system, interface, and module levels, but poor calibration prevents concluding that their output ordering reflects real verification practice.The middle pairs exchange rank in about 21% of repository-bootstrap resamples; any alternative taxonomy needs a separately tested rubric and reference set.Source: §6.5 · p. 33

7 Threats to Validity

The validity threats concern whether VeeReview measures verification evidence, whether matching and adjustment support cohort comparisons, and how far the findings generalize. These constraints limit conclusions to instrument-generated states in the selected matched sample.

7.1 Construct Validity

  • Human-reference agreement ranges from 4% to 60% by pair, and weak inter-rater agreement means neither the reference standard nor the instrument is validated.Visible pull-request artifacts do not capture all verification, and test presence does not establish relevant coverage or successful execution.Source: §7.1 · p. 34

7.2 Internal Validity

  • Matching cannot identify an authorship effect, and rerun instability, differing applicability rates, and authorship misclassification leave cohort contrasts uncertain.The judge changed match state in 25% of rerun comparisons; AIDev-recorded authorship cannot capture editor-level assistance or substantial human editing.Source: §7.2 · pp. 34–35

7.3 External Validity

  • The 1,454 retained agent pull requests are a selected subset of 3,648, drawn from repositories with eligible human comparisons, so results do not generalize to agent-authored changes generally.The study covers five named agents using one judging model and configuration.Source: §7.3 · p. 35, §7.2 · p. 35

7.4 Conclusion Validity

  • Repository-bootstrap intervals address clustering, but Architecture ↔Integration is not significant for build-only after adjustment, and cells below n = 30 are not interpreted.The preregistered mixed-effects model uses Benjamini-Hochberg correction across four pairs.Source: §7.4 · p. 35

8 Related Work

Prior work studies agent pull-request outcomes, testing and review, task resolution, and language models as judges. This paper positions its contribution as classifying change-linked verification evidence, matching human pull requests, and evaluating the measure against human labels.

8.1 Empirical studies of agent-authored pull requests

  • Earlier agent pull-request studies examine rejection reasons, test contributions, security, or merge behavior; this study classifies pre-decision verification evidence and compares matched human pull requests.It does not analyze how build-and-verify states relate to review outcomes, a link the paper identifies as future connective work.Source: §8.1 · pp. 35–36

8.2 Testing practice and code review in pull-based development

  • Prior work shows tests, descriptions, and contributor history influence acceptance, motivating measures of test co-change and observable artifacts that a judge may respond to.GitHub mining also carries known selection risks (Kalliamvakou et al. 2014).Source: §8.2 · pp. 36–37

8.3 Evaluating coding agents

  • Coding-agent benchmarks test task resolution, while this study examines verification evidence in live-repository pull requests as a complementary evaluation target.The paper distinguishes task benchmarks from evidence connecting changes to verification at multiple levels.Source: §8.3 · p. 37

8.4 LLM-as-a-judge for software artifacts

  • The paper contributes a preregistered calibration criterion and a comparison of model-judged with rule-decided states, while arguing that judge-based measures need validity, agreement, and repeatability tests.The authors report that among software-artifact judging studies reviewed through early 2026, they found no process metric with the same kind of rule-decided comparison.Source: §8.4 · p. 37

9 Conclusion

VeeReview applies V-Model build–verify pairs to matched agent and human pull requests, finding lower agent build-only rates across all four pairs. Standardization reduced the gaps, while the rule-decided pair’s interval included zero; calibration limits what these observations establish.

9 Conclusion

  • Agent-authored pull requests had lower build-only rates on all four pairs by 3.3 to 13.3 percentage points, contrary to the preregistered hypothesis; standardization removed 34% to 60% of the difference.The rule-decided pair’s interval included zero under both specifications.Source: §9 · pp. 37–38
  • Agreement with human references was 4% to 60% against an 80% criterion, and weak rater agreement means cohort differences remain observations made with this instrument.The authors call for a trained rater protocol and adjudication independent of the instrument’s developers.Source: §9 · p. 38
  • Reviewers should check whether cited evidence addresses changed behavior at the relevant level; language-model measures require validation before supporting comparisons of verification practice.A fluent description and linked issue are not themselves evidence that a change was verified.Source: §9 · p. 38

Declarations

The authors report study funding, ethics procedures, contributions, and AI-tool use. The replication package and calibration materials are archived on Zenodo.

Declarations

  • The replication package includes frozen samples, instrument verdicts, analysis scripts, figure and table scripts, and calibration materials archived on Zenodo.The calibration materials include the 25-pull-request reference sample, both raters’ labels, adjudicated labels, and scoring scripts.Source: Declarations · p. 38
  • The first author received a US$2,000 undergraduate research scholarship, and the JWare Lab provided a workstation GPU.Source: Declarations · p. 39
  • Two volunteer raters labelled 25 publicly available pull requests, consented to analysis of their labels, and had no personal data collected.Source: Declarations · p. 39
  • The first author designed and built VeeReview, collected and analyzed data, and wrote the manuscript; the second author supervised, reviewed, and revised it.Source: Declarations · p. 39
  • VeeReview uses qwen3:32b as judge; Claude tools assisted with instrument and script development and manuscript drafting, revision, editing, and figures.The authors state they directed the work, reviewed AI-assisted output, and take responsibility for the paper’s content.Source: Declarations · p. 39

A The pairs and their rules

VeeReview scores four build–verification pairs separately, using constrained model judgements for three and a repository-signal rule for Module Design–Unit Test. Its rubric distinguishes matched, build-only, verify-only, and out-of-scope states, while file heuristics supply evidence inputs.

A The pairs and their rules

  • The rule-decided Module Design–Unit Test pair uses changed production logic, test-file presence, and a repository-tree heuristic to identify a potential existing test.A matching existing test path is marked not updated; no production logic change is not applicable.Source: A The pairs and their rules · p. 43
  • The four pairs connect requirements to acceptance tests, system design to system tests, architecture design to integration tests, and module design to unit tests.The rubric distinguishes system failures affecting end-to-end product behavior from architecture failures at component seams.Source: A The pairs and their rules · p. 43
  • Each applicable pair is classified separately as matched, build-only, verify-only, or not applicable; model-judged pairs can also be unverified when evidence cannot substantiate a half.Build-only marks a present build half with missing verification; out-of-scope pairs are marked not applicable rather than scored low.Source: A The pairs and their rules · p. 44
  • The rubric first classifies pull-request type to determine pair scope, then argues the missing verification half first and requires concrete artifact evidence for each non-not-applicable verdict.It also calls for pair-specific findings, explanations, improvements, and actionable suggestions grounded in files or identifiers.Source: A The pairs and their rules · p. 44
  • Three pairs receive model judgements constrained by deterministic repository facts; the named Module Design–Unit Test verdict, score, and color are computed mechanically.The model’s role is to judge whether tests cover changed logic and whether validation addresses requirements; it must treat supplied facts as ground truth.Source: A The pairs and their rules · p. 45

B File classification heuristics

  • File heuristics separate tests, documentation, configuration, and source changes; configuration is excluded from production logic, and comment-only edits do not count as source changes.The repository-tree heuristic identifies possible file correspondence rather than demonstrated test coverage, a construct threat that may make the rule too lenient for both cohorts.Source: A The pairs and their rules · p. 43, B File classification heuristics · p. 45

C Instrument configuration for the reported run

The reported run used a local qwen3:32b judge and prompt version v2. After failed attempts and exclusions, it retained 3,648 agent pull requests and 1,454 complete matched agent–human pairs.

C Instrument configuration for the reported run

  • The run used qwen3:32b locally with temperature 0, one draw per pull request, and rubric prompt version v2.Source: C Instrument configuration for the… · p. 45
  • 3,648 agent pull requests formed the final scored corpus after 112 requests lacked verdicts following retries and failed attempts.The sample began with 3,760 requests from 977 repositories; 31 of 180 failed attempts were recovered by retries.Source: C Instrument configuration for the… · p. 46
  • 1,454 complete agent–human pull-request pairs across 741 repositories were retained after scoring 1,459 of 1,464 matched human requests.Source: C Instrument configuration for the… · p. 46
  • The instrument’s components separate retrieval, deterministic signal extraction, model rubric, evidence validation, and the mechanical module-unit verdict.The components are implemented in github_client.py, signals.py, pairs_v2.md, evidence_check.py, and verify_rule.py, respectively.Source: C Instrument configuration for the… · p. 46

D Deviations from the preregistration

The paper documents preregistration departures, including a restored primary model, four-pair rubric changes, and added analyses. Its reproduction instructions warn that an export option is essential to prevent unintended records from affecting analyses.

D Deviations from the preregistration

  • The primary analysis now reports the preregistered mixed-effects logistic model, repository random intercepts, odds ratios, and Benjamini–Hochberg correction after the original analysis used paired tests and bootstrap risk differences.Source: D Deviations from the preregistration · p. 46
  • The instrument expanded from three preregistered pairs to four, and the paper reconstructs rubric v1 rather than rescoring with it.Both permissive and strict reconstructions had the same direction and significance, but exact reproduction of v1 would require rescoring.Source: D Deviations from the preregistration · p. 46
  • The preregistered interval-includes-null falsification rule could detect an effect versus no effect, but not distinguish a reversed effect from the hypothesized direction.Source: D Deviations from the preregistration · p. 46
  • The model-versus-rule contrast was exploratory, while four robustness analyses and an unplanned analysis of requests without issue links were added during review.Source: D Deviations from the preregistration · p. 46

E Reproduction

  • The --only-sampled export flag is essential because otherwise cached requests from earlier draws enter the export with null strata and default cohort labels, affecting analyses other than RQ1.The last four reproduction commands generate the figures and tables together with CSV files of plotted values.Source: E Reproduction · p. 47

F Calibration detail

The calibration detail shows that raters disagreed about the cohort difference, while the instrument reported little difference in the 25-request reference set. The human reference also has limitations, including merge-outcome information entering the labels.

F Calibration detail

  • Against adjudicated labels, the instrument withheld matches more often from human than agent requests (38% versus 23%), adding about 20 points to the apparent agent advantage.The asymmetry was unstable across raters: +11 points against one rater and −1 against the other.Source: F Calibration detail · p. 47
  • Raters favored opposite cohorts by 2 and 10 points, adjudicated labels favored humans by 19 points, and the instrument reported 15.4% versus 14.6% matched.The 25-request sample was too small to assess the larger cohort gap directly.Source: F Calibration detail · p. 47
  • Requirement ↔Acceptance was never classified as matched in the reference set; its 4% agreement reflects no matched classifications rather than graded disagreement.The instrument returned build-only 19 times, not applicable 5 times, and verify-only once.Source: F Calibration detail · p. 47
  • One human rater marked 20 of 21 merged requests matched but only 1 of 4 unmerged requests, and adjudicated labels followed that rater exactly.Although merge outcomes were prohibited, outcome information entered the process labels; this did not explain the overall calibration failure.Source: F Calibration detail · p. 47
  • Even excluding Requirement ↔Acceptance, no pair met the agreement threshold against any reference; agreement with the second rater ranged from 28% to 72%.Adjudicated labels followed the first rater exactly.Source: F Calibration detail · p. 47

Related papers

Loading 2610.29143v1…