PaperlayerView exact revision on arXiv

Source-linked AI summary

AI Writes Faster Than Humans Can Review: A Longitudinal Study of an Enterprise 2x Mandate

Hao He, Shyam Agarwal, Yegor Denisov-Blanch, Pavel Azaletskiy, Sanmi Koyejo, Bogdan Vasilescu

arXiv:2607.01904v1cs.SE

TL;DR

This study examines how a documented enterprise AI mandate affects throughput and reorganizes work. It analyzes internal company data with a staggered difference-in-differences design centered on developers’ adoption timing. Throughput roughly doubled, with evidence implicating adoption and accumulated use while review shifted toward automation.

  • Problem

    Whether mandate-driven throughput gains vary across developers, codebase age, and model generations matters because velocity metrics miss displaced effort and resulting debt.

    Source: §III · p. 3, B. Implications for Research · p. 10

  • Method

    A longitudinal case study uses internal company data and staggered difference-in-differences, comparing developers’ output around their individual AI adoption.

    Source: A. Case Company and the “2×” Mandate · p. 3, C. Analysis Approach · p. 4

  • Results

    Roughly doubled, per-capita PR throughput was linked within developers to AI adoption and growing accumulated use; automated review overtook human review while merge and revert rates stayed about flat.

    Source: §I · p. 1, §VI · p. 10

  • Takeaways & Limitations

    The authors conclude that an enterprise AI mandate is a process-redesign problem: gains grow with use and relocate work downstream rather than removing it.

    Source: §I · p. 2

    Adoption timing and intensity were not randomized, and cumulative use is endogenous to workload, motivation, task mix, and shipped code, limiting causal interpretation.

    Source: C. Threats to Validity · p. 10

Abstract

from arXiv · show

Enterprises increasingly mandate AI coding tools and report large productivity gains, yet longitudinal evidence on how such a mandate unfolds is scarce. In this paper, we present a quantitative case study of a documented enterprise "2x" mandate at a mid-sized, AI-forward company that has been committed to doubling merged pull requests per engineer since mid-2025. In a panel of 802 developers and 196,212 pull requests (January 2024-April 2026), per-capita throughput eventually doubled, reaching 2.09x the pre-mandate baseline in April 2026, among the largest gains reported from a field deployment of AI coding tools to our knowledge. A staggered difference-in-differences design links the within-developer share of this gain to AI adoption and to a further gain that grows with accumulated use, with the mandate acting as a catalyst rather than a direct driver. Because adoption and usage intensity were not randomly assigned, we read this evidence as strongly implicating an adoption-and-use channel rather than as exact causal attribution. The gain is broadly shared across seniority yet concentrated in newer code and not separable across model generations. Adoption also restructured code review around automation: per-reviewer load roughly doubled and automated review overtook human review, while merge and revert rates held steady.

I. INTRODUCTION

The paper examines whether a documented enterprise AI mandate can deliver the large gains proponents promise and how adoption unfolds. It situates a longitudinal case study against mixed, mostly point-in-time evidence on AI productivity and review.

A. Empirical Evidence on AI Coding Tools

  • Prior credible field estimates range from a 19% slowdown to roughly 40% more PRs in heavy-use weeks, leaving enterprise-scale gains uncertain.Earlier studies mostly measure point-in-time effects and do not follow how a rollout unfolds within an organization.Source: §I · p. 1

I. INTRODUCTION

  • The study follows a mid-sized B2B software company’s June 2025 2× mandate using internal AI-use and PR records in a developer-month panel.The panel covers 802 developers and 196,212 PRs from January 2024 to April 2026, analyzed with staggered difference-in-differences.Source: §I · p. 1
  • Per-capita PR throughput roughly doubled; within-developer evidence links much of the gain to AI adoption and accumulated use, while nonrandom adoption bounds exact causal attribution.The authors describe the gain as among the largest within-developer gains reported from a field deployment, reached gradually under unusually favorable conditions.Source: §I · pp. 1–2
  • The paper frames enterprise AI mandates as process redesign: gains grow with use and shift work downstream rather than simply removing it.Its longitudinal case examines how far a near-best-case rollout moved productivity, for whom, and how it restructured code review around automation.Source: §I · p. 2

A. Empirical Evidence on AI Coding Tools

  • Existing research reports mixed productivity and quality findings, while a close within-engineer study found roughly 40% more PRs in heaviest-usage weeks but left experience-versus-model effects unresolved.Related work also describes review-capacity pressures and unsettled quality outcomes as AI-generated work increases.Source: A. Empirical Evidence on AI Coding Tools · p. 2

B. Complementary Investments and the Productivity Paradox

The paper connects complementary investment and productivity-paradox research to a study of a documented 2× mandate at a favorable enterprise site. It assembles internal PR and AI-use data and uses staggered adoption timing to study throughput and organizational change.

B. Complementary Investments and the Productivity Paradox

  • Prior complementary-investment research links technology returns to reorganized workflows and skills, while whether developer-side accumulating returns can be separated from model-side gains remains open.The cited literature includes a productivity-pressure paradox in which mandates without time to build skills may undermine their intended gains.Source: B. Complementary Investments and the… · p. 2

III. METHODS

  • The study asks whether throughput gains exceed prior field estimates, vary across seniority, codebase age, and model generations, and how review and quality outcomes respond.These questions address unresolved developer-side accumulating returns and whether organizations absorb added production through backlog, reduced review, or another response.Source: §III · pp. 2–3

A. Case Company and the “2×” Mandate

  • The company’s young, AI-forward, permissive setting makes its rollout a near-ideal case whose effects are better read as an upper envelope than an industry average.In June 2025, its CTO set a twelve-month goal to double productivity using merged PRs per engineer per month as the progress metric.Source: A. Case Company and the “2×” Mandate · p. 3
  • The firm offered uncapped commercial coding tools, and three frontier-model generations arrived during the study; release dates mark intended exposure because internal uptake lagged and overlapped rollout.The tools included Cursor and Claude Code, alongside other tools and custom agents.Source: A. Case Company and the “2×” Mandate · p. 3

B. Data Collection and Processing

  • PR history supplies throughput, review, revert, and timing measures, while corporate tool logs provide monthly AI-written and suggested lines, token spend, and acceptance rates.The timing measures separate coding, pickup, and review phases; the first human review defines the pickup/review boundary because automated review events are common.Source: B. Data Collection and Processing · pp. 3–4
  • The company’s created-by-AI label identifies PRs it attributes to AI, but its assignment mechanism is uncontrolled, so per-PR AI-versus-non-AI comparisons are correlational.The label also captures AI use from tools without usage logs.Source: B. Data Collection and Processing · p. 3
  • The resulting panel includes 802 developers and the PR-level dataset 196,212 non-bot PRs; panel estimation uses 564 developers observed for at least three active months.Developer identities are linked to tool logs, and role data cover about 91% of current employees for seniority analysis.Source: B. Data Collection and Processing · p. 4

C. Analysis Approach

The analysis separates AI adoption from the shared mandate date by comparing developers with their own earlier output and aligning them by adoption timing. It estimates both the initial adoption change and whether gains grow with accumulated use, while treating the latter cautiously because usage intensity is not randomly assigned.

Identification

  • The staggered DiD aligns developers by first AI use, comparing their post-adoption output with their own earlier output and never-adopters’ shared trend.The mandate is a shared date, so the design estimates returns to adoption rather than a separately identified direct mandate effect.Source: C. Analysis Approach · pp. 4–5

Developer-panel model

  • The developer-panel model uses fixed effects and pooled Cursor/Claude Code adoption, preferring calendar-month fixed effects to absorb organization-wide shocks.The pooled adoption measure reflects that Cursor use was largely pre-mandate and Claude Code use largely post-mandate, making tool effects difficult to separate from calendar time.Source: C. Analysis Approach · pp. 4–5

Measurement checks

  • Adoption is measured from tool-usage logs rather than the created-by-AI PR label, which may miss or misattribute AI assistance.The study also re-estimates the adoption coefficient using merged PRs, the mandate’s own metric; it changes only marginally, from +0.354 to +0.347.Source: C. Analysis Approach · p. 5

Downstream analyses

  • The analysis treats review-process changes as part of adoption’s total effect, then tests heterogeneity by seniority and repository cohort and models PR-level review, cycle-time, merge, and revert outcomes.PR-level models control for change size; organization-wide review-supply trends are reported descriptively.Source: C. Analysis Approach · pp. 5–6

Robustness checks

  • Robustness checks trace pre-adoption trends, use estimators comparing adopters only with not-yet-adopters, and test a fake release plus seasonal controls.The fake August 2025 release tests whether the release indicators capture generic calendar drift rather than model-release changes.Source: C. Analysis Approach · pp. 5–6

Accumulated use

  • A lagged cumulative-AI-lines term estimates whether gains grow with use, while frontier-model release indicators separate release-associated changes from accumulated-use gains.The authors interpret the cumulative-use coefficient as an association, not a clean causal slope, because accumulated use can co-move with workload, task mix, motivation, and output.Source: C. Analysis Approach · p. 5

A. RQ1: A Near-Doubling That Grows with Accumulated Use

Per-capita throughput reached 2.09× the pre-mandate baseline, while within-developer estimates link gains to both AI adoption and accumulated use. The usage-related gain grows over time, although its dose–response pattern is descriptive rather than causal.

A near-doubling

  • 44.3 mean PRs per active developer in April 2026 versus 21.2 in the pre-mandate baseline, a 2.09× increase.The reported count is authored PRs per active developer, not the mandate’s merged PRs per engineer.Source: A. RQ1: A Near-Doubling That Grows… · p. 6
  • Among 246 developers active throughout the study, mean throughput rose 2.3×, exceeding the company-wide average.This comparison isolates developers observed across the full study period.Source: A. RQ1: A Near-Doubling That Grows… · p. 6
  • At April 2026 usage, the within-developer gain is +0.38 log points (1.46×) with calendar-month fixed effects, versus +0.54 (1.72×) in the baseline specification.The decomposition attributes positive contributions to both adoption and accumulating use; accumulated use supplies the larger share, while the mandate leaves a small residual.Source: A. RQ1: A Near-Doubling That Grows… · pp. 6–7, Table II · p. 7

Accumulated use

  • The traced adoption effect rises from 1.51× at three months to 1.55× at six and 1.99× at nine; heavier use also tracks larger gains.The tiered dose–response is descriptive because post-adoption usage may itself reflect productivity; the month-by-month path does not condition on later usage but retains nonrandom adoption timing. Claude Code token spend is a lower bound on the intensity ramp because Cursor usage is not dollar-denominated.Source: A. RQ1: A Near-Doubling That Grows… · pp. 6–7, Figure 5 · p. 7

A near-doubling

  • The pooled adoption estimate is +0.354 (approximately +42%), smaller than the decomposition total and nine-month effect because it averages short and long horizons.It remains +0.201 with calendar-month fixed effects.Source: A. RQ1: A Near-Doubling That Grows… · p. 7

B. RQ2: The Gain Is Broadly Shared but Uneven

Estimated adoption gains appear broadly similar across engineering seniority, with management an exception, but gains differ by code age. Productivity rises with accumulated use, while the contribution of particular model generations cannot be separated credibly.

Across the seniority ladder

  • Adoption gains are statistically indistinguishable across the IC–principal ladder, while management has the largest proportional jump at +0.62 (approximately +86%).The principal estimate is imprecise because it covers 16 developers.Source: B. RQ2: The Gain Is Broadly Shared but… · p. 7

Across the codebase

  • Adoption raises throughput by +0.36 (+44%) in post-2022 repositories but only +0.12 (+12%, not significant) in pre-2022 repositories; the difference is +0.25 (p < 0.01).The gain is concentrated in newer code.Source: B. RQ2: The Gain Is Broadly Shared but… · pp. 7–8

Review patterns

  • Human review falls to 68% of PRs while automated review reaches 84%, overtaking human review shortly after the mandate.Review activity rises after adoption, with approve-only review increasing faster than substantive review.Source: Figure 6 · p. 8, Figure 7 · p. 8

Across model generations

  • The cumulative-use gain is positive and significant, but productivity gains cannot be pinned to particular frontier-model releases.Release dates coincide with the organization-wide usage ramp, and an August 2025 placebo without a release moves as much as the real events; the cumulative-use slope remains an association rather than a clean causal estimate.Source: B. RQ2: The Gain Is Broadly Shared but… · p. 8

C. RQ3: Adoption Reshaped the Review Process

As AI-authored PR volume rose, review demand outpaced human capacity and the organization shifted much of the work to automated review. Human review became less substantive, while quality proxies stayed stable and AI-authored PRs took longer to merge.

C. RQ3: Adoption Reshaped the Review Process

  • 2.0× per-reviewer load followed PR volume growth of 3.1×, while the reviewer pool grew only 1.5×.The capacity gap was associated with rising end-to-end cycle time and a substantive-review queue that never regained its earlier speed.Source: C. RQ3: Adoption Reshaped the Review… · pp. 8–9
  • Automated review coverage rose from ∼19% to ∼84%, overtaking human review as the share receiving human review fell from 89% to 68%.Substantive human review fell from ∼39% to ∼21% of PRs while silent approvals stayed roughly flat at ∼50%.Source: C. RQ3: Adoption Reshaped the Review… · p. 8
  • Merge rates stayed essentially flat and revert rates declined slightly as review shifted to automation, providing evidence against an acute quality collapse, not a clean bill of health.Merge and revert rates are coarse, short-horizon proxies; the AI-authored label is not randomized and may select safer changes.Source: C. RQ3: Adoption Reshaped the Review… · p. 8, Figure 8 · p. 8
  • AI-authored PRs took about 20% longer from first human review to merge and 22% longer in total cycle time.The longer interval involved more reviewer engagement without more revision rounds, rather than additional fixing.Source: C. RQ3: Adoption Reshaped the Review… · pp. 8–9
  • Late-window AI-driven review and auto-approval collapsed human-review latency for a growing share of PRs, biasing the measured premium toward zero.The per-PR estimates are correlational; the created-by-AI label may reflect unmeasured complexity.Source: C. RQ3: Adoption Reshaped the Review… · p. 9

V. DISCUSSION

The near-doubling emerged gradually in a favorable, AI-forward setting and was measured on a single activity metric. The discussion emphasizes monitoring downstream work, supporting accumulated use, and anticipating changes in engineering roles.

V. DISCUSSION

  • The within-developer gain was roughly 1.5× at observed usage and rose to 2× by nine months on tool, but the near-best-case setting and activity metric mean it is not evidence that gains are typical, immediate, or free.The firm was young and AI-forward, with uncapped tool budgets and a concerted adoption push.Source: §V · p. 9

A. Implications for Practice

  • Merged-PR counts should be paired with review wait time, queue depth, and revert or incident rates because authoring speedups can shift the binding constraint to stages paced by human judgment.The company reactively routed review to automation as PR volume rose and per-reviewer load doubled.Source: A. Implications for Practice · p. 9
  • Throughput grew with accumulated use, so the discussion recommends targeting use intensity and judging the mandate over a long horizon rather than focusing on adoption alone.The authors caution that accumulated use is an association: heavier use may partly reflect, rather than cause, higher output.Source: A. Implications for Practice · p. 9
  • The management tier had the largest proportional gain (+0.62, ≈+86%), which the authors interpret as a re-entry effect and a reason to plan for blurred coding and management roles.The proportional figure is amplified by managers’ low starting level of coding.Source: A. Implications for Practice · p. 9

B. Implications for Research

The paper calls for broader outcome measures, replication across organizations, and longer observation of AI rollouts. It also bounds causal interpretation because adoption and usage were not randomized, while concluding that throughput doubled during a near-best-case rollout.

B. Implications for Research

  • The panel stops at merged PRs and cannot measure downstream costs or separate model capability from repeated use because releases followed the same firm-wide usage ramp.The unmeasured outcomes include technical debt, diluted ownership and understanding, and what automated review misses or developers stop learning.Source: B. Implications for Research · p. 10
  • Estimates from one young, AI-forward firm with an official merged-PR target may not transfer to firms with different resources or older codebases, and the design cannot fully separate acceleration from metric inflation.The authors propose comparisons with similar firms that adopted AI without making throughput an official target.Source: B. Implications for Research · p. 10
  • Because gains accrue with use, short study windows can catch only the adoption spike and understate or misdate the mature effect; multi-year panels are needed to follow adoption cohorts and model generations.The authors cite the Callaway–Sant’Anna estimate as an instance of a short window misdating the effect.Source: B. Implications for Research · p. 10

C. Threats to Validity

  • The rollout was not randomized: adoption timing and use intensity were developer choices, so the accumulated-use slope is an association rather than a clean causal return.The adoption jump depends on a parallel-trends assumption, while the heavy/average/light dose-response conditions on post-adoption usage.Source: C. Threats to Validity · p. 10

VI. CONCLUSION

  • Per-developer throughput doubled during a near-best-case rollout, but review load rose just as fast and the organization leaned on automation as human review thinned toward bare approval.The authors emphasize that throughput alone hides this shift in how the resulting work gets reviewed.Source: §VI · p. 10

APPENDIX Supplementary Materials

The supplementary material documents dataset summaries, automated-account flagging, and additional review analyses. It reports that the mean reviewer series is outlier-driven, while the qualitative human-review composition finding is consistent across analyses.

APPENDIX Supplementary Materials

  • Table V summarizes both datasets used throughout the paper for January 2024–April 2026.The supplied caption identifies the table's scope but does not provide dataset values.Source: Table V · p. 13, APPENDIX Supplementary Materials · p. 13
  • Automated accounts were flagged using username and name patterns, email markers, and behavioral signals, with borderline cases adjudicated manually.Reviewer-side signals included templated event bodies, low body diversity, and many empty-bodied events.Source: APPENDIX Supplementary Materials · p. 13

A. Mean Per-Reviewer Decomposition

  • Mean commented-review counts show recurring March–April spikes driven by a few high-volume reviewers, motivating the use of medians; the qualitative pattern remains rising silent approvals and comparatively flat commented reviews.Figure 9 presents means, while Figure 7 presents monthly medians.Source: A. Mean Per-Reviewer Decomposition · p. 13, Figure 9 · p. 13

B. Human-Review Composition

  • Substantive human review fell from ∼39% to ∼21% of PRs while silent approvals stayed roughly flat at ∼50%, leaving remaining human review increasingly bare approval.Figure 10 decomposes the human-review share into substantive written feedback and silent approval-only reviews.Source: B. Human-Review Composition · p. 13, Figure 10 · p. 13

C. Per-PR Cycle-Time Decomposition

AI-related PR latency is concentrated in human review rather than coding or pickup. As volume outpaced review capacity, the organization kept aggregate cycle time from rising further by routing fewer PRs through substantive human review, while Claude Code spend climbed sharply.

C. Per-PR Cycle-Time Decomposition

  • The post-mandate AI latency premium concentrates between first human review and merge; coding lead is statistically indistinguishable and pickup delay is comparatively small.Pickup and review lead are measured to the first human review or comment, excluding automated reviews.Source: C. Per-PR Cycle-Time Decomposition · p. 13, Figure 11 · p. 14

D. Organization-Level Latency

  • Substantively human-reviewed PRs’ 90th-percentile cycle time peaked near 114 hours and remained around 73 hours in 2026, while review coverage fell from ∼48% to ∼21%.Aggregate cycle time peaked near 66 hours and then receded because more PRs bypassed the still-slow substantive-review queue, not because that queue decongested; the timing is observational and the monthly tail is noisy.Source: D. Organization-Level Latency · pp. 13–14, Figure 12 · p. 14

E. Usage-Intensity Ramp

  • Claude Code spend per developer-month rose roughly 20×, from $42 to $856, between July 2025 and April 2026.This is a Claude-only lower bound on the intensity ramp because Cursor usage is not dollar-denominated; spend steepened through the model-release window rather than at any single cutoff.Source: E. Usage-Intensity Ramp · p. 14, Figure 13 · p. 14

F. Adoption Signal Robustness

An independent PR label changes adoption classifications and strengthens the estimated adoption coefficient, while correcting for inactive months barely changes it. Output is flat before first tool use and rises afterward, supporting the parallel-trends assumption.

F. Adoption Signal Robustness

  • Adding created-by-AI PRs to telemetry reclassified 36 developers as adopters and raised the adoption coefficient from +0.354 to +0.410 (p < 0.001).The revised definition moved 87 adoption dates earlier; the residual mandate term fell to near zero as adopters starting around the mandate were attributed to adoption.Source: F. Adoption Signal Robustness · p. 14
  • Redating adoption to account for tool use in inactive months changed the coefficient by less than 0.01.Recovered inactive months carry no PR outcome and drop out of the log-count regression.Source: F. Adoption Signal Robustness · pp. 14–15

G. Adoption Event Study (Parallel-Trends Check)

  • Output is flat before developers’ first AI-tool use and rises afterward, supporting the parallel-trends assumption.The event study controls for the company-wide mandate and uses τ = −1 as the reference month.Source: G. Adoption Event Study… · p. 15, Figure 14 · p. 15

H. Throughput Across Staggered-Adoption Estimators

The adoption estimate remains positive under calendar-month fixed effects, while some alternative estimators attenuate it further. Denominator checks preserve or increase throughput gains, but release-ladder estimates cannot separate model capability from calendar drift and accumulated use.

H. Throughput Across Staggered-Adoption Estimators

  • The adoption coefficient on log(Monthly PRs) is +0.201 (p < 0.001) with full calendar-month fixed effects, versus +0.354 under the baseline design.Estimates are +0.369 when never-treated developers are dropped, +0.132 under Borusyak imputation, and a non-significant +0.162 under Callaway–Sant’Anna; the further drop occurs under estimators that forbid already-adopted developers from serving as controls.Source: H. Throughput Across… · p. 15, Table VI · p. 15
  • Counting zero-output months within engineers’ tenure yields a 2.42× increase for authored and 2.31× for merged pull requests.The full-calendar-month-FE Poisson estimate remains +0.24 (p < 0.001) with 11.8% zero-output months, so conditioning on active developers is, if anything, conservative.Source: H. Throughput Across… · p. 15
  • Release indicators switch on in the first full month after launch, but shared uptake means they can absorb a generic upward calendar trend instead of capability gains.The placebo replaces the release indicators with an indicator for August 2025, a month with no model launch; a comparable placebo coefficient signals calendar drift rather than capability.Source: H. Throughput Across… · p. 15
  • The placebo release moved as much as genuine releases, and the incremental ladder coefficients flipped sign, so the design captures calendar drift rather than identified capability gains.The August 2025 placebo had no model launch; its coefficient was −0.236 (p < 0.001), while the release coefficients were Sonnet 4.5 −0.123, Opus 4.5 −0.208, and Opus 4.6 +0.613. The sign instability persisted with month-of-year fixed effects.Source: H. Throughput Across… · p. 16, Figure 15 · p. 16, Table VII · p. 16
Loading 2607.01904v1…