PaperlayerView exact revision on arXiv

Source-linked AI summary

Who Pays the Review Cost? Triage, Fairness, and Accountability in AI-authored Pull Requests

Md Shamimur Rahman, Khairul Alam, Banani Roy, Chanchal K. Roy

arXiv:2610.11179v1cs.SE

TL;DR

Reviewers face a governance problem as AI-generated patches enter workflows where production can be cheap but review attention remains scarce. This mixed-method survey of 239 practitioners finds that AI authorship alone does not determine review effort; respondents condition it on visible human grounding, validation, scope, responsiveness, and ownership.

  • Problem

    As AI-generated patches become easier to produce, less is known about how reviewers govern AIPR entry, credibility, and fairness while review attention remains scarce.

    Source: §1 · p. 2

  • Method

    The authors surveyed 239 practitioners with code-review experience using quantitative questions and open-ended prompts about AIPR review and governance.

    Source: §1 · p. 2

  • Results

    Respondents did not reject AIPRs categorically; they described review effort as conditional on clear intent, bounded scope, validation, responsive human engagement, and identifiable ownership.

    Source: §1 · p. 2

  • Takeaways & Limitations

    AIPR governance centers on making human judgment and responsibility visible before generated contributions consume reviewer attention.

    Source: §7 · p. 23

    Question wording made low-effort submissions and stricter criteria salient, so findings reflect judgments under the stated scenarios rather than estimates of how often those situations occur.

    Source: §6.1 · p. 21

Abstract

from arXiv · show

AI coding agents are moving from local code assistance into pull-based workflows, where generated contributions must be reviewed, explained, and maintained within existing project norms. Although recent work has begun to characterize AI-authored pull requests (AIPRs), less is known about how reviewers govern their entry into review, how AI authorship reshapes credibility and fairness, and what intake mechanisms protect review sustainability. We report a mixed-method questionnaire survey of 239 practitioners from 31 countries with code-review experience and varying exposure to AIPRs. In the scenarios and self-reports elicited by the survey, AI authorship was not a categorical rejection signal. Instead, respondents described review effort as conditional on whether an AIPR arrived as an accountable contribution, with bounded scope, project-grounded rationale, validation beyond Continuous Integration (CI), contributor responsiveness, and identifiable post-merge ownership. This conditional logic extended to newcomer AIPRs, where respondents emphasized visible participation in the current review process over profile-level reputation alone. Qualitative responses further described shifts in mentoring, scrutiny, deferral, and routing when human stewardship was difficult to observe. We conceptualize missing rationale, validation, and ownership as reviewability debt, the work reviewers must absorb when generated code lacks sufficient human grounding. These findings reframe AIPR governance around making human judgment observable before generated contributions consume scarce reviewer attention.

1 Introduction

AI-generated patches can lower production costs without reducing the scarce reviewer effort needed to assess context, risk, and ownership. This study frames AIPR intake as a question of reviewability and governance, finding that reviewers condition effort on accountable preparation rather than AI authorship alone.

1 Introduction

  • AI tools can make patch production cheaper without necessarily reducing review costs, leaving reviewers uncertain about context, edge cases, maintainability, and ownership.This creates a tension between cheaper code production and scarce review attention.Source: §1 · p. 2
  • Reviewability concerns whether a PR supplies enough context, evidence, scope control, and accountable engagement for productive review; it is distinct from correctness.A correct change can remain hard to review when rationale, test evidence, decomposition, or an accountable maintainer is missing.Source: §1 · p. 2
  • Reviewers condition AIPR review on intent, bounded scope, validation, responsive human engagement, and ownership—not AI authorship alone.When these signals are missing, explanation, validation, and ownership costs shift to maintainers, who may defer, route, audit, or reject the contribution.Source: §1 · p. 2
  • The study contributes a reviewability-centered account of AIPR triage, an empirical account of fairness and effort asymmetry, and workflow safeguards for accountable review.Proposed mechanisms include AI-use boundaries, risk-specific evidence, queue controls, owner routing, and generated-diff containment.Source: §1 · p. 2

2 Background and Related Work

Prior research establishes that pull-request review depends on technical evidence and contributor context, while offering less direct evidence about reviewers’ stated criteria for triaging AIPRs. This study positions its contribution at that reviewer-level intersection of reviewability, fairness, and governance.

2.1 Signals and Decision-making in PR Review

  • PR review combines technical evidence—such as tests, rationale, and change quality—with social and historical signals, including contributor history, familiarity, and responsiveness.These signals shape decisions about review effort and ownership, while AI authorship complicates how reviewers interpret them.Source: §2.1 · pp. 2–3

2.2 Newcomers, Casual Contributors, and Fairness in Review

  • Newcomer outcomes can reflect project-fit judgments and interaction dynamics, while AI use adds uncertainty about whether contributors understand, validate, and own their changes.This motivates examining observable participation and preparation rather than treating perceived AI authorship as sufficient evidence of commitment or reviewability.Source: §2.2 · p. 3

2.3 AI-assisted Development and Agent-authored PRs

  • Agentic tools can create and submit PRs, but reviewers infer production and accountability from visible artifacts because they rarely know how each line was produced.Generated code still requires inspection, adaptation, debugging, and contextualization before safe use.Source: §2.3 · pp. 3–4

2.4 Human Review of Agent-authored PRs

  • Repository studies describe uneven human review of AIPRs, recurring concerns beyond correctness, and more review rounds for AI-generated code.They provide less access to reviewers’ unstated pre-review criteria and perceptions of responsibility, reciprocity, and fairness.Source: §2.4 · p. 4

2.5 Human Oversight and Accountability in AI-mediated Work

  • Human-oversight research emphasizes calibrated reliance and visible uncertainty, but offers limited empirical evidence about how reviewers convert these concerns into AIPR decisions and team rules.Relevant considerations include understanding, validation, accountability, and organizational risk.Source: §2.5 · p. 4

2.6 Review Burden and Governance of AI-assisted Contributions

  • Prior work documents review friction, community-level burden, and project AI policies that commonly permit some AI use with conditions on human involvement or accountability.It leaves open how individual reviewers operationalize conditions for investing effort in everyday decisions.Source: §2.6 · pp. 4–5

2.7 Interpretive Theoretical Lenses

  • The study retrospectively uses signaling, social exchange, and organizational justice theories to interpret credibility, reciprocity, and fairness patterns.These lenses were not used to construct the questionnaire, code responses, or formulate confirmatory hypotheses.Source: §2.7 · p. 5

2.8 Positioning of This Study

  • The study complements repository evidence by eliciting reviewers’ stated reasoning about when AIPRs merit effort and which signals make them reviewable.Table 1 distinguishes prior-work coverage as central, partial, or limited across principal research streams.Source: §2.8 · p. 6, Table 1 · p. 6

3 Methodology

The study organizes its inquiry around how AIPRs reshape review work, reviewer triage, and governance expectations. Three research questions examine these issues, including fairness and accountability in sustainable review.

3.1 Research Questions

  • RQ2 asks which conditions make an AIPR reviewable enough to receive effort, including entry criteria, deferral, CI-passing cases, suspected drive-by submissions, and newcomer credibility.The question covers triage before detailed review.Source: §3.1 · p. 6
  • RQ1 examines how AIPRs change review work, including communication expectations, volume pressure, low-effort submissions, and review practices.Source: §3.1 · p. 6
  • RQ3 examines how reviewers expect workflows to preserve fairness and accountability, including effort asymmetry and whether they mentor, give feedback, scrutinize, or reject contributions.Source: §3.1 · p. 6

3.2 Survey Design and Data Analysis Plan

The study uses a cross-sectional mixed-method questionnaire to compare reviewer judgments and examine their stated reasoning. Its design combines purposive recruitment, validated survey items, structured comparisons, and codebook-based analysis, with conclusions bounded by the sample and instrument.

3.2 Survey Design and Data Analysis Plan

  • A cross-sectional mixed-method questionnaire paired quantitative comparisons of reviewer judgments with open-ended questions about respondents’ reasoning.The study followed empirical survey guidance and ACM SIGSOFT standards.Source: §3.2 · p. 6

3.2.1 Target Population and Sampling Objective

  • The target was a heterogeneous sample of experienced reviewers relevant to AIPRs, not a demographic replica of the software-development population.The authors restrict conclusions to the contexts supported by recruited participants.Source: §3.2.1 · pp. 6–7

3.2.2 Instrument Construction and Validation

  • The literature-informed questionnaire was iteratively reviewed and piloted, but lacked a previously established psychometric scale and its predefined alternatives were not exhaustive.Option-level results therefore represent endorsement of presented alternatives; open responses cover additional considerations.Source: §3.2.2 · p. 7

3.2.3 Questionnaire Structure and Measures

  • Questionnaire blocks combined structured and open-ended formats; paired HPR/AIPR items held change type and passing CI constant to compare willingness without additional explanation.AIPR-specific blocks covered entry criteria, deferral, newcomer signals, and open-ended views on fairness and effort asymmetry.Source: §3.2.3 · pp. 7–8, Table 2 · p. 8

3.2.4 Recruitment and Sampling Frame

  • Purposive recruitment combined reviewers from agent-active repositories, mature open-source projects, and professional networks.The strategy aimed to broaden participation beyond a single sampling frame.Source: §3.2.4 · p. 8

3.2.5 Response Screening and Final Sample

  • Of 261 response records, 239 remained after excluding one non-consenting and 21 substantially incomplete responses.The combined email and LinkedIn recruitment made the total number of people exposed to the survey unobservable, so no invitation-level response rate was reported.Source: §3.2.5 · p. 9

3.2.6 Quantitative Analysis

  • Quantitative analyses matched tests to item scales and dependencies, using paired non-parametric comparisons, effect sizes, response distributions, and Holm correction for related tests.Exploratory subgroup analyses were treated as contextual rather than confirmatory.Source: §3.2.6 · p. 9

3.2.7 Qualitative Analysis

  • Open-ended responses were analyzed with a shared codebook, independent coding, Cohen’s κ, and third-evaluator adjudication when discussion did not resolve disagreements.Multi-label themes contextualize and qualify quantitative patterns rather than estimate prevalence or claim theoretical saturation.Source: §3.2.7 · pp. 9–10

4.1 Overview of Participant Demographics

The survey analyzed 239 valid responses from practitioners with varied development, review, and AIPR experience, drawn from 31 countries.

4.1 Overview of Participant Demographics

  • 239 valid responses were analyzed after 22 exclusions during a two-month survey.The figure summarizes participants’ AIPR exposure, software development and code review experience, and the eight most represented countries.Source: §4.1 · p. 10, Figure 1 · p. 10
  • Participants came from 31 countries, with the top five countries accounting for 64.0% of responses.The United States was the largest group at 19.7% (47), followed by Canada at 15.5% (37).Source: §4.1 · p. 10
  • 58.2% reported more than seven years of software development experience, while 48.5% reported more than seven years of code review experience.AIPR exposure was also substantial: 78.2% had reviewed at least 10 AIPRs and 49.4% had reviewed more than 40.Source: §4.1 · p. 10

4.2 RQ1- Review Work Under AI Authorship

RQ1 examines how AI authorship changes review work and what makes an AIPR worth reviewer effort. Respondents described cost-aware intake rather than categorical rejection, with sustained review depending on accountable contributions.

4.2.1 Communication and workload conditions

  • 90.8% reported changed communication expectations, 77.8% increased review PR volume, and 60.7% saw low-effort newcomer AIPRs as a problem.Moderate or significant changes were reported by 74.1% for communication and 57.7% for PR volume; 25.9% called low-effort newcomer AIPRs a major problem.Source: §4.2.1 · p. 11, Figure 2 · p. 11

4.2.2 Communication shifts toward evidence and accountability

  • Communication shifted toward evidence-gated accountability, with requests for concrete validation, reconstructed intent and context, human ownership, and clearer reviewability conditions.Respondents reported minimal difference when a trusted human clearly owned the AI output; the evidence-gated orientation did not significantly differ by reviewers’ code-review experience.Source: §4.2.2 · pp. 11–12, Table 3 · p. 12

4.2.3 Reviewers adopt concrete safeguards for AIPR work

  • Reviewers described safeguards including active validation, intent, provenance, and ownership gates, plus reviewability or bandwidth controls.These adaptations often co-occurred as a compound safeguard: establish technical evidence, confirm human ownership, then decide whether deeper review is warranted.Source: §4.2.3 · pp. 12–13, Table 4 · p. 13
  • Reviewers did not treat AI authorship as unacceptable; productive review depended on a responsible human scoping, explaining, validating, and owning the change.Respondents resisted broad, weakly explained, or unvalidated changes that shifted preparation work to maintainers.Source: §4.2.3 · p. 13

4.3 RQ2- Reviewability Triage for AIPRs

Reviewers were less willing to accept CI-passing AIPRs without further explanation, and applied entry and deferral gates based on evidence, scope, ownership, and responsiveness. Newcomers could earn deeper review through visible participation in the current PR rather than profile reputation alone.

4.3.1 Willingness to proceed across change types

  • 52.3% to 19.7%: willingness to accept behavior-preserving refactors without explanation fell from HPRs to AIPRs, the largest drop across eight change types.The AIPR explanation penalty was not reliably moderated by code-review experience.Source: §4.3.1 · pp. 13–15, Figure 3 · p. 14

4.3.2 Entry gates and deferral triggers

  • 66.5% required passing CI or tests before AIPR review, while 56.9% required a clear description; vague descriptions were the most common deferral trigger at 64.9%.Large undecomposed PRs and missing tests despite behavior changes each triggered deferral for 51.0%; reviewers also cited absent context, unresolved CI failures, and nonresponsive contributors.Source: §4.3.2 · p. 15, Figure 4 · p. 15

4.3.3 Conditional credibility for newcomer AIPRs

  • 82.4% rated professional responses to feedback highly or extremely influential for newcomer credibility, compared with 23.4% for an active GitHub profile.Reviewers also valued testing evidence (81.2%) and appropriate scope (77.8%); the most common special handling asked newcomers to demonstrate understanding (55.6%).Source: §4.3.3 · pp. 16–17

4.4 RQ3- Fairness and Governance in AIPR Workflows

Respondents wanted AIPR rules embedded in project workflows, while judging stricter handling fair when it addressed effort asymmetry rather than AI authorship alone. Visible human engagement supported constructive reviewer investment; apparent offloading shifted responses toward audit, triage, or rejection.

4.4.1 Governance rules for AIPR intake

  • 171 cleaned responses were thematically coded into six rule families, reflecting requests for concrete workflow mechanisms beyond individual reviewer judgment.The item used multi-label thematic coding because responses could propose more than one rule.Source: §4.4.1 · p. 17
  • 36.8% of 171 responses proposed AI-use admissibility policies, and 33.3% called for evidence bundles matched to the change type.Other proposed mechanisms included queue controls, risk-based owner routing, generated-diff containment, and agent-workflow hygiene.Source: §4.4.1 · pp. 17–18

4.4.2 Fairness under effort asymmetry

  • Reviewers viewed stricter AIPR handling as fair when it addressed effort asymmetry, extending mentoring when human engagement was visible and shifting to audit, triage, or rejection when work was offloaded.Respondents described repairing the asymmetry through explanations of intent, validation, responsiveness, and post-merge responsibility; fairness triggers transferred to AIPRs when human engagement was observable.Source: §4.4.2 · pp. 18–19

5 Discussion

The discussion frames AIPR review as a conditional allocation of reviewer effort: generated code requires visible human accountability because artifact quality alone may not show understanding. It develops this account through reviewability debt, stewardship-focused intake, and directions for further study.

5.1 A Conditional Review Model for AIPRs

  • AIPR review requires calibrating reviewer effort alongside trust in generated contributions, because a polished diff or passing CI may not reveal who checked assumptions or will maintain the change.The discussion connects this parallel calibration to human-AI trust research.Source: §5.1 · p. 19

5.2 Cheap Signals, Costly Signals, and Reciprocity

  • Current-review behavior outweighed profile cues: professional feedback responses and testing evidence rated highly or extremely influential by 82.4% and 81.2%, versus 23.4% for an active profile.The authors interpret these findings through signaling and reciprocity, but note that the interpretations are post hoc and need direct tests using designs such as vignette experiments.Source: §5.2 · p. 19

5.3 Reviewability Debt and Recoverable Fairness

  • Reviewability debt is reviewer-side work caused when an apparently complete PR leaves intent, scope, validation, or ownership unresolved.Its immediate costs include defensive checking, repeated clarification, queue delay, or reduced mentoring; CI cannot establish behavior preservation or accountable human understanding.Source: §5.3 · p. 19

5.4 Designing for Stewardship at PR Intake

  • The proposed stewardship trace makes visible what was generated, revised, validated, left uncertain, and who will maintain the change.Suggested intake artifacts include provenance summaries, risk-specific evidence, validation logs, and ownership declarations; policies should scale evidence and ownership requirements to change risk.Source: §5.4 · pp. 19–20

5.5 From Project Policies to Reviewer Judgments

  • Compared with policy studies focused on AI use and disclosure [68] [69], respondents also wanted change-matched evidence, risky-change routing, and focused generated diffs.Unlike commonly reported aggressive closure policies [69], respondents most often favored asking suspected newcomer drive-by submitters to demonstrate understanding (55.6%) or discuss the issue first (38.1%).Source: §5.5 · p. 20

5.6 Research Agenda

  • Future research should track AIPRs after merge, study reviewer cognition in controlled or field settings, and test which evidence improves review.The proposed outcomes include defects and maintenance effort; cognition topics include automation bias, fatigue, calibration, and skill development.Source: §5.6 · p. 20

6 Threats to Validity

The study’s conclusions are bounded by study-specific measures and scenario framing, self-reported judgments, purposive recruitment, and exploratory analyses. These constraints limit measurement precision and generalization without negating the convergent patterns reported.

6.1 Construct Validity

  • AIPR authorship was perceived rather than verified line by line, and exposure was self-reported in ordinal bands, limiting precise provenance and exposure measurement.Source: §6.1 · p. 21
  • Predefined alternatives and salient scenarios may shape responses; selection frequencies indicate endorsement, not exhaustiveness or spontaneous prevalence.Findings concern judgments under stated scenarios, not reconstructed behavior; reviewability debt is interpretive, not a measured quantity of time, defects, or maintenance cost.Source: §6.1 · p. 21

6.2 Internal Validity

  • The cross-sectional observational design characterizes reviewer judgments, not causal effects of AI authorship; paired comparisons also face fixed-order and framing concerns.Source: §6.2 · p. 21
  • Fixed questionnaire order may have influenced later responses, and explicit human–AI comparison may create demand characteristics that the observed countervailing responses cannot eliminate.Source: §6.2 · pp. 21–22

6.3 External Validity

  • Purposive, non-probabilistic recruitment, self-selection, and nonresponse constrain population estimates; results support analytical generalization to comparable review settings.Private, smaller, regulated, and non-GitHub settings may be less represented; 78.2% reported reviewing at least 10 AIPRs and 49.4% more than 40.Source: §6.3 · p. 22
  • Findings are temporally situated as agents and review practices evolve, so replication across settings and later agent generations is needed to assess stability.Source: §6.3 · p. 22

6.4 Conclusion Validity

  • Analyses use methods suited to ordinal paired responses, but exploratory subgroup analyses and sparse categories reduce precision; non-significance is not evidence of no association.Multiple-selection frequencies are not a complete ranking of concerns.Source: §6.4 · p. 22
  • Anonymous survey responses lacked dedicated attention checks, so inattentive, repeated, or machine-assisted responses cannot be ruled out; qualitative coding is interpretive and open-ended responses cannot establish saturation or prevalence.Two coders used a shared codebook, resolved disagreements through discussion, and used a third evaluator for unresolved cases; Cohen’s κ indicated strong agreement.Source: §6.4 · p. 23

7 Conclusion

The conclusion frames AIPR governance around observable human stewardship rather than AI authorship alone. It presents reviewability debt as reviewer work left by unresolved grounding and calls for workflows that make responsibility visible.

7 Conclusion

  • The central governance challenge is whether generated contributions show enough understanding, validation, scope control, responsiveness, and responsibility for reviewers to invest attention.Source: §7 · p. 23
  • Reviewability debt names work left to reviewers when rationale, validation, context, or ownership remain unresolved.Source: §7 · p. 23
  • Tools and workflows should support evidence of readiness for sustained review while preserving a path for responsible AI-assisted contributors.Source: §7 · p. 23
  • Future repository and longitudinal studies can test whether reviewability signals predict latency, rework, maintenance effort, or post-merge outcomes.Controlled and field studies can evaluate which forms of provenance, validation evidence, rationale, and ownership reduce reviewer burden.Source: §7 · p. 23

Declarations

The declarations report funding, ethics approval, consent, data availability, and conflicts of interest.

Funding

  • The research received support from NSERC Discovery Grants, the CFI-JELF, and the NSERC CREATE Software Analytics Research program.Source: Funding · p. 23

Ethical approval

  • The University of Saskatchewan Behavioural Research Ethics Board approved the study before data collection (BEH#6412).Source: Ethical approval · p. 23

Informed consent

Data Availability Statement

Conflict of Interest

Loading 2610.11179v1…