PaperlayerView exact revision on arXiv

Source-linked AI summary

Plausible Is Not Enough: Trust and Evidence in AI-Authored Pull Request Review

Md Shamimur Rahman, Khairul Alam, Banani Roy, Chanchal K. Roy

arXiv:2610.21144v1cs.SE

TL;DR

Whether reviewers use the same approval criteria for AI-authored and human-authored pull requests remains unclear, despite its relevance to safe and scalable AI-assisted development. A survey comparing practitioners’ reported judgments found lower initial trust in AIPRs and frames their review as an evidence-ownership problem.

  • Problem

    It remains unclear how reviewers’ approval judgments differ between HPRs and AIPRs, including their trust, evidence, and uncertainty criteria.

    Source: §1 · p. 2

  • Method

    A survey of 239 practitioners compared reported judgments about human- and AI-authored pull requests.

    Source: §7 · p. 24

  • Results

    Respondents reported lower initial trust in AIPRs and greater emphasis on prompter expertise, change size, independent reasoning, ownership statements, and code provenance.

    Source: §7 · p. 24

  • Takeaways & Limitations

    The findings frame AIPR review as an evidence-ownership problem involving credible validation and clear human responsibility for generated changes.

    Source: §1 · p. 2

    Findings concern stated judgments rather than directly observed review behavior, and understanding and ownership debt is interpretive rather than directly measured.

    Source: §6.1 · p. 23

Abstract

from arXiv · show

Code review is where AI-authored pull requests (AIPRs) must earn reviewer trust before integration, yet it is unclear whether reviewers apply the same criteria used for human-authored pull requests (HPRs). We surveyed 239 practitioners from 31 countries using matched HPR and AIPR questions to compare trust, evidence requirements, and verification practices within respondents. Most respondents (64.9%) reported lower initial trust in AIPRs. They relied less on author track record and subsystem familiarity, emphasizing instead human prompter expertise and bounded change size. For approval, respondents more often considered their own reasoning through critical cases sufficient for AIPRs than HPRs (55.2% versus 28.5%), while relying less on test coverage. Overall, 61.1% reported higher evidence requirements for AIPRs, and the share who would block approval under residual uncertainty increased from 19.4% for HPRs to 43.9% for AIPRs. Respondents prioritized test inspection, project context, and local execution, and more often selected partial reimplementation to verify non-trivial logic. Unclear ownership was also more likely to pause AIPR review (29.3% versus 14.6%). Open-ended responses pointed to the difficulty of validating plausible code without reliable context and human accountability. These findings frame AIPR review as an evidence-ownership problem, not merely a correctness check. Reviewers need to connect generated changes to validated behavior and responsible maintainers. Review tools and agentic workflows should expose provenance, distinguish generated claims from human-validated evidence, and make uncertainty visible before approval.

1 Introduction

The paper asks whether reviewers use the same trust and approval criteria for AI-authored and human-authored pull requests. A within-respondent survey frames AIPR review as a question of whether evidence and human accountability justify merging plausible generated code.

1 Introduction

  • AIPRs make conventional trust cues less direct, leaving reviewers to judge whether plausible code and passing tests are backed by reliable rationale and accountability.Reviewers must decide when to rely on AI-produced artifacts, demand more evidence, or block approval under unresolved uncertainty.Source: §1 · p. 2
  • Prior studies have not explained whether perceived authorship changes reviewers’ trust, evidence thresholds, verification strategies, and tolerance for uncertainty across otherwise comparable review decisions.The gap matters for assessing AI-generated code without excessive trust in apparently correct outputs or unnecessary rejection of useful contributions.Source: §1 · p. 2
  • A mixed-method survey of 239 practitioners used matched HPR and AIPR questions to compare reported review judgments within the same respondents.The comparisons covered trust, evidence requirements, uncertainty handling, verification depth, and effort allocation.Source: §1 · p. 2
  • Respondents reported that AIPR review shifts emphasis toward connections between generated code, human verification, and long-term maintenance responsibility.They considered passing tests, plausible explanations, and familiar review practices alone insufficient, while retaining technical checks used for HPRs.Source: §1 · p. 2
  • The paper frames AIPR review as an evidence-ownership problem and proposes implications for independent validation, provenance, human ownership, and reviewer verification burden.It describes understanding and ownership debt as a possible long-term maintenance risk.Source: §1 · p. 2

2 Background and Related Work

Prior work describes code review as a judgment under incomplete information that draws on technical evidence, author context, and project history. Research on AI-assisted and agent-authored code motivates examining how reviewers establish trust, validation, and accountability in AIPRs.

2.1 Review Judgment and Authorship Cues

  • Reviewers combine evidence about rationale, dependencies, expected behavior, context, and tests with author experience, ownership, and interaction history when deciding whether to integrate a change.Missing or complex information can be associated with confusion, delays, and increased review effort.Source: §2.1 · p. 3

2.2 From AI Assistance to Agentic PRs

  • AI assistance spans human-controlled coding to substantial AI authorship, while exact line-level provenance is often unavailable and responsibility may be distributed across people and systems.Reviewers may infer authorship from descriptions, tests, commits, contributor behavior, and manual adaptation.Source: §2.2 · p. 3

2.3 Human Review of Agent-authored PRs

  • Repository studies describe review activity and outcomes for agent-authored PRs, but provide limited insight into individual reviewers’ trust, evidence requirements, and verification reasoning.This study compares practitioners’ reported criteria across HPR and AIPR contexts.Source: §2.3 · pp. 3–4

2.4 Trust Calibration and Human Oversight

  • Trust calibration frames appropriate reliance as matching trust to capability, task context, and available evidence, with uncertainty and system limitations made visible to reviewers.The paper uses this framing to compare trust, evidence reliance, and responses to uncertainty, but does not test a formal theoretical model.Source: §2.4 · p. 4

2.5 Trust and Provenance in the Evaluation of AI-Generated Code

  • Prior studies of AI-code use, provenance labels, and accountability examine different tasks and produce mixed evidence about provenance effects, leaving responsibility and credible validation central to AIPR review.The cited work includes studies of generated-code validation, authorship labels, reviewer accountability, and risk-aware review workflows.Source: §2.5 · pp. 4–5

2.6 Positioning of This Study

  • The study addresses limited evidence about HPR–AIPR approval differences with a paired questionnaire comparing reported trust cues, evidence requirements, uncertainty handling, and verification priorities.Its focus is whether reviewers consider available evidence sufficient to approve and merge generated code.Source: §2.6 · p. 5

3.1 Research Questions

The study asks how perceived authorship shapes trust, approval evidence, and verification work in pull-request review. Its questions compare reviewers’ judgments about HPRs and AIPRs across the decision process.

3.1 Research Questions

  • RQ1 asks whether reviewers begin with different trust assumptions for HPRs and AIPRs and rely on different cues when forming trust.The comparison includes author- or AI-centered cues and shared cues such as change size, tests, and description clarity.Source: §3.1 · p. 6
  • RQ2 asks whether AI authorship changes what reviewers consider sufficient evidence and how they respond to unresolved uncertainty before approval.It compares approval criteria, uncertainty tolerance, and reported evidence thresholds.Source: §3.1 · p. 6
  • RQ3 asks whether AI authorship shifts verification strategies and review effort toward defensive validation and accountability.It covers non-trivial logic, time allocation, stop-review conditions, and up-front artifacts.Source: §3.1 · p. 6

3.2 Survey Design and Data Analysis Plan

The study uses a cross-sectional mixed-method survey with matched HPR and AIPR questions to compare practitioners’ reported review judgments. Quantitative comparisons are paired within respondents, while open-ended responses contextualize those patterns.

3.2 Survey Design and Data Analysis Plan

  • A cross-sectional mixed-method questionnaire uses matched HPR and AIPR items to compare reported judgments within respondents, rather than estimate causal effects on actual review behavior.Structured quantitative items are combined with open-ended questions about respondents’ reasoning.Source: §3.2 · p. 6

3.2.1 Target Population and Sampling Objective

  • The target population is practitioners with code-review experience in settings where AI-authored PRs are relevant, recruited from complementary sources because no comprehensive sampling frame exists.The sample aims for varied experience and contexts, not demographic representativeness; conclusions are restricted to contexts supported by participants.Source: §3.2.1 · p. 6

3.2.2 Instrument Construction and Validation

  • Questionnaire response options drew on prior research, and pre-deployment reviews plus an 11-practitioner pilot were used to refine wording and coverage.The instrument uses concrete single-item judgments rather than an established psychometric scale; response options are not an exhaustive taxonomy.Source: §3.2.2 · pp. 6–7

3.2.3 Questionnaire Structure and Measures

  • Questionnaire blocks follow the review decision from initial trust through verification and approval to pausing review and requesting information.Matched items use identical or parallel options where appropriate, while open-text responses capture reasoning beyond predefined alternatives.Source: §3.2.3 · p. 7

3.2.4 Recruitment and Sampling Frame

  • Recruitment combined reviewers identified through agent-authored PR repositories, developers from active and mature projects, and professional-network dissemination.The repository-based frame broadened beyond projects with known AIPR activity to include active Apache and GitHub projects.Source: §3.2.4 · pp. 7–8

3.2.5 Response Screening and Final Sample

  • After screening 261 response records, the analysis retained 239 participants and excluded one response without consent and 21 substantially incomplete responses.Optional and open-ended items use their valid-response counts when denominators differ from the full sample.Source: §3.2.5 · p. 8

3.2.6 Quantitative Analysis

  • Paired judgments were analyzed with tests matched to item type, with effect sizes and response distributions considered alongside significance.Exploratory subgroup analyses were treated as contextual, and Holm correction controlled family-wise error within related comparison sets.Source: §3.2.6 · p. 9

3.2.7 Qualitative Analysis

  • Open-ended responses were analyzed using codebook-based thematic coding with independent coders, agreement measurement, and adjudication of unresolved disagreements.Qualitative results contextualize or qualify quantitative patterns rather than estimate population prevalence.Source: §3.2.7 · pp. 9–10

4.1 Overview of Participant Demographics

The survey analyzed 239 practitioners from 31 countries, including many experienced developers and reviewers with substantial exposure to AIPRs.

4.1 Overview of Participant Demographics

  • 239 valid responses came from 31 countries, with the top five countries accounting for 64.0% of responses.The top 10 countries accounted for 83.7%.Source: §4.1 · p. 10
  • 58.2% reported more than seven years of software development experience, while 48.5% reported more than seven years of code review experience.The sample also included early-career perspectives.Source: §4.1 · p. 10
  • 78.2% had reviewed at least 10 AIPRs, including 49.4% who had reviewed more than 40.The reported exposure grounds the sample in practical AIPR review experience while including respondents with more limited exposure.Source: §4.1 · p. 10
  • Figure 1 summarizes AIPR-review exposure, software development and code review experience, and the eight most represented countries.Percentages use the 239 valid responses.Source: Figure 1 · p. 10

4.2 RQ1: Authorship and Trust Calibration

Respondents reported lower initial trust in AIPRs than HPRs and shifted toward a narrower set of trust cues. For AIPRs, those cues emphasized human oversight and bounded changes.

4.2.1 Initial trust

  • 64.9% rated AIPRs lower than HPRs, with a median one-point penalty and a large paired effect (rrb = −0.77).Figure 2 presents the paired trust distribution.Source: Figure 2 · p. 11, §4.2.1 · p. 11
  • The median trust penalty was one point in every background group; AIPR exposure had only a weak association with the penalty (ρ = −0.13, p = .05).Differences across exposure groups were not significant (p = .32), and experience was not a significant predictor.Source: §4.2.1 · p. 11

4.2.2 Trust cues

  • HPR trust cues centered on author track record (77.9%), while the leading AIPR-specific cue was human prompter expertise (57.5%); change size rose from 45.0% to 57.7%.AI tool reputation was selected by 32.3%.Source: §4.2.2 · p. 11
  • Subsystem familiarity fell from 72.5% for HPRs to 44.1% for AIPRs (−28.4 pp), though differing item wording may partly explain the shift.The HPR item referred to the author's familiarity and the AIPR item to the human prompter's familiarity.Source: §4.2.2 · p. 11
  • Respondents selected fewer trust cues for AIPRs than HPRs (3.76 versus 4.33); AIPR free-text responses more often named process and ownership cues.These included codebase adaptation, human prompter understanding, and the extent of AI generation.Source: §4.2.2 · p. 12
  • AIPR trust was framed as requiring reviewable, supervised contributions tied to accountable human judgment, rather than reliance on the AI tool alone.The paper interprets this as re-establishing accountability around generated changes.Source: §4.2.2 · p. 12

4.3 RQ2: Evidence and Uncertainty

Under perceived AI authorship, respondents shifted toward reviewer-owned assurance and a higher evidentiary bar, and were less willing to accept unresolved uncertainty. Their concerns centered on whether generated code and its supporting evidence could be trusted in project context and tied to a responsible human.

4.3.1 Sucient evidence for approval

  • 55.2% of respondents considered reasoning through critical cases sufficient for AIPRs, versus 28.5% for HPRs; reliance on test coverage fell from 41.0% to 23.8%.Overall criteria differed significantly (χ2 = 48.71, df = 4, p < .001); 49.8% changed their criterion, with 18.4% shifting from tests to personal reasoning.Source: §4.3.1 · pp. 12–13, Figure 4 · p. 13

4.3.2 Residual uncertainty before approval

  • 43.9% would block an AIPR until uncertainty was resolved, versus 19.4% for an HPR; approval with documented uncertainty fell from 19.8% to 5.5%.Respondents more often escalated to blocking under AIPR authorship, while open-ended responses cited safeguards such as splitting or redoing a PR and seeking human explanation.Source: §4.3.2 · pp. 13–14, Figure 5 · p. 14

4.3.3 Stated evidence requirements and rationales

  • 61.1% reported higher evidence requirements for AIPRs, attributing the added burden to plausible code, project-context gaps, and supporting evidence that itself needs verification.Among 225 respondents, 158 (70.2%) described AIPR approval as harder; the reported bar favored project-grounded, independently verifiable support tied to a responsible human.Source: §4.3.3 · p. 14

4.4 RQ3: Verication and Eort

Respondents described AIPR review as a shift in role rather than uniformly broader effort: they emphasized inspectable evidence, defensive validation, and clear human responsibility. Across verification, time allocation, stopping conditions, and up-front artifacts, AI authorship changed what reviewers prioritized and when they paused.

4.4.1 Verication of non-trivial logic

  • Verification breadth was similar (3.03 strategies for HPRs vs. 2.90 for AIPRs), but AIPRs shifted toward concrete validation, stricter scrutiny, and partial reimplementation.Running or inspecting tests rose from 52.1% to 65.0%, and re-implementing or partially rewriting logic from 11.1% to 20.5%; open responses also cited risk containment and human accountability.Source: §4.4.1 · pp. 15–16, Figure 6 · p. 16

4.4.2 Review time allocation

  • Reviewers moved intent comprehension and feedback later, while local validation, test checking, context inspection, and coordination moved earlier for AIPRs.Local execution shifted from mean rank 5.98 to 4.91; the defensive-validation index moved from 4.22 to 3.72, while comprehension moved from 4.12 to 5.04.Source: Table 4 · p. 16, Figure 7 · p. 17, §4.4.2 · p. 17

4.4.3 Stop-review conditions

  • Ownership uncertainty was a stop-review trigger for 29.3% of AIPR reviews versus 14.6% of HPR reviews, while common technical triggers remained similar.Open responses added AIPR-specific gates around provenance, accountability, and stronger evidence before reviewers invested further effort.Source: §4.4.3 · pp. 17–18, Figure 8 · p. 18

4.4.4 Up-front review-eciency artifacts

  • AIPR reviewers more often prioritized AI-generation disclosure and explicit ownership, while clear intent and constraints remained the most-selected artifact but declined from 70.3% to 47.7%.A summary of AI-generated versus manually edited material rose from 11.7% to 35.6%, and explicit ownership statements from 16.3% to 33.9%.Source: §4.4.4 · pp. 18–19, Figure 9 · p. 19

5 Discussion

The discussion frames AIPR review as an evidence-ownership problem: reviewers need to connect generated changes to validated behavior and a responsible person. It argues for calibrated, inspectable evidence rather than blanket distrust, while noting that several costs and risks remain unmeasured.

5.1 Evidence Ownership in AIPR Review

  • AIPR review shifts trust toward human prompter responsibility, while tests remain useful checks but are less often accepted as sufficient approval evidence.Provenance alone does not establish that behavior was checked against project requirements or that a person can answer for the supporting evidence.Source: §5.1 · pp. 19–20

5.2 The Hidden Cost of Validation

  • Review work is redistributed toward reconstructing assurance—checking assumptions, risky-path coverage, and representative conditions—rather than simply adding strategies; elapsed review time was not measured.The authors suggest assessing how much reviewer reconstruction agent outputs require alongside generation success.Source: §5.2 · p. 20

5.3 Understanding and Ownership Debt

  • Respondents valued accepted responsibility, not merely a named maintainer; “understanding and ownership debt” describes an unmeasured downstream risk when no human can explain decisions after merge.The survey did not track long-term maintenance outcomes, so it cannot establish that AIPRs actually accumulate more debt.Source: §5.3 · pp. 20–21

5.4 Limits of the Authorship Penalty

  • Stricter AIPR judgments were concentrated in particular dimensions, while trust, evidence criteria, verification breadth, and technical stop reasons also showed stable or mixed patterns.The authors caution that blanket extra checks, regardless of supplied evidence, could waste effort or disadvantage adequately validated work.Source: §5.4 · p. 21

5.5 Implications for Review Workows

  • Review workflows should connect consequential changes to claims, project context, validation, and human stewardship, distinguishing generated tests from tests whose expectations were independently checked.Routine changes may need concise provenance and test summaries, while higher-risk changes may warrant more explicit assumptions, measurements, or domain-owner review.Source: §5.5 · p. 21
  • Future studies should test which project-grounded evidence improves review decisions without unnecessary work, measuring accuracy, confidence, review time, and reliance on generated tests.The authors also call for separating agent-produced evidence from evidence independently validated by a responsible human or another sufficiently independent source.Source: §5.5 · pp. 21–22

6 Threats to Validity

The authors qualify their findings through limits in measurement, study design, sampling, and survey execution. The results characterize practitioners’ stated judgments in this sample, not causal effects or direct observations of review behavior.

6.1 Construct Validity

  • Study-specific questionnaire items indicate stated judgments rather than validated latent measures, and AIPR refers to respondent-perceived primarily AI-generated authorship rather than verified line-level provenance.Mixed-authorship workflows make a strict boundary between AI-assisted and AI-authored code difficult to observe during review.Source: §6.1 · p. 22
  • Predefined alternatives and question wording may frame answers; selection frequencies reflect endorsement of presented options, not their exhaustiveness or prevalence of themes.Open-ended responses surfaced concerns also represented by structured alternatives, supporting their relevance but not proving completeness.Source: §6.1 · p. 22

6.2 Internal Validity

  • The observational comparison cannot isolate an effect caused solely by AI authorship because the HPRs and AIPRs respondents considered may differ in risk, quality, familiarity, or policy.The study was designed to characterize judgments, not establish a causal effect of authorship on review behavior.Source: §6.2 · p. 23
  • HPR items appeared first without randomized order, and explicit authorship framing may create order, contrast, salience, or demand effects that the observed response patterns cannot eliminate.The authors interpret results as within-respondent contrasts under explicit authorship comparison, not causal estimates in natural review settings.Source: §6.2 · p. 23

6.3 External Validity

  • Purposive, non-probabilistic recruitment may underrepresent private, smaller, regulated, or non-GitHub settings, supporting analytical generalization only to comparable review contexts.Self-selection may also distinguish participants from nonparticipants in AI exposure, workload, policy, or strength of opinion.Source: §6.3 · p. 23
  • Findings are temporally situated as coding agents, provenance mechanisms, policies, and reviewer familiarity evolve, so replication is needed to assess longer-term transferability.The authors call for replication across industrial settings, repository ecosystems, organizational policies, and later generations of agents.Source: §6.3 · p. 23

6.4 Conclusion Validity

  • Analyses used scale-appropriate paired tests, effect sizes, and Holm correction; main shifts remained significant under Bonferroni correction across 51 comparisons, while nonsignificant results remain inconclusive.The largest adjusted p-value under the all-comparison Bonferroni correction was .039.Source: §6.4 · p. 24
  • Undetected inattentive or machine-assisted responses cannot be ruled out because the anonymous survey lacked dedicated attention checks or automated-response detection items.Most respondents answered the open-ended question substantively, and uninterpretable responses were not assigned themes.Source: §6.4 · p. 24

7 Conclusion

The survey frames AI-authored PR review as an evidence-ownership problem, with respondents emphasizing trust, validation, and human accountability. The authors call for studies of these judgments in actual review settings.

7 Conclusion

  • Respondents reported lower initial trust in AIPRs, greater emphasis on prompter expertise and change size, and more reliance on independent reasoning and clear ownership.They were more likely to withhold approval under residual uncertainty and valued code provenance; these findings reflect reported judgments, not observed review behavior.Source: §7 · p. 24
  • Differences were selective: technical reasons for stopping review and the range of verification activities remained largely similar across AIPRs and HPRs.Source: §7 · p. 24
  • Future studies could test these reported judgments in actual reviews and examine how provenance summaries, ownership statements, and human-validated tests influence approval.Repository and longitudinal studies could also examine relationships with review practices and post-merge outcomes.Source: §7 · p. 24

Declarations

The declarations report funding, ethics approval, participant consent, data availability, and no conflict of interest.

Funding

  • The research received support from NSERC Discovery Grants, the CFI-JELF, and the NSERC CREATE program in Software Analytics Research.Source: Funding · p. 24

Ethical approval

  • The University of Saskatchewan Behavioural Research Ethics Board approved the study before data collection.Approval No. BEH#6412.Source: Ethical approval · p. 24

Informed consent

Data Availability Statement

Conict of Interest

Related papers

Loading 2610.21144v1…