PaperlayerView exact revision on arXiv

Source-linked AI summary

Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

Joel Becker, Nate Rush, Elizabeth Barnes, David Rein

arXiv:2507.09089v2cs.AIcs.HCcs.SE

TL;DR

Real-world evidence on how AI affects software developer productivity remains limited because synthetic evaluations and indirect field metrics may not capture actual productivity. This randomized trial measures completion time for experienced open-source developers and finds that allowing early-2025 AI tools increased completion time by 19%.

  • Problem

    Real-world AI effects on developer productivity remain understudied because synthetic tasks and indirect field metrics may not reflect actual productivity.

    Source: §1 · p. 1

  • Method

    A randomized controlled trial measured task completion times for 16 experienced developers completing 246 real tasks in repositories they regularly contribute to, with AI use randomly allowed or disallowed.

    Source: §1 · p. 2, §2.1 · p. 4

  • Results

    Allowing early-2025 AI tools increased completion time by 19%, despite developers and experts forecasting substantial speedups.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Field experiments with robust outcome measures are important because expert forecasts and developer surveys can substantially overestimate AI’s productivity benefits.

    Source: §4 · pp. 10, 12

    The observed slowdown may not generalize beyond highly familiar, large, mature repositories, and the factor analysis provides suggestive rather than statistically powered evidence.

    Source: §3.3 · p. 10, §4.1 · p. 12

Abstract

from arXiv · show

Despite widespread adoption, the impact of AI tools on software development in the wild remains understudied. We conduct a randomized controlled trial (RCT) to understand how AI tools at the February-June 2025 frontier affect the productivity of experienced open-source developers. 16 developers with moderate AI experience complete 246 tasks in mature projects on which they have an average of 5 years of prior experience. Each task is randomly assigned to allow or disallow usage of early 2025 AI tools. When AI tools are allowed, developers primarily use Cursor Pro, a popular code editor, and Claude 3.5/3.7 Sonnet. Before starting tasks, developers forecast that allowing AI will reduce completion time by 24%. After completing the study, developers estimate that allowing AI reduced completion time by 20%. Surprisingly, we find that allowing AI actually increases completion time by 19%--AI tooling slowed developers down. This slowdown also contradicts predictions from experts in economics (39% shorter) and ML (38% shorter). To understand this result, we collect and evaluate evidence for 20 properties of our setting that a priori could contribute to the observed slowdown effect--for example, the size and quality standards of projects, or prior developer experience with AI tooling. Although the influence of experimental artifacts cannot be entirely ruled out, the robustness of the slowdown effect across our analyses suggests it is unlikely to primarily be a function of our experimental design.

1 Introduction

This study uses a randomized controlled trial on experienced open-source developers’ real repository tasks to measure AI’s actual productivity impact. Contrary to forecasts and prior synthetic-task findings, AI increased completion time by 19%, although the effect is setting-specific and broadly robust across design analyses.

1.1 Background

  • The study addresses realism gaps in prior evaluations by measuring real-world open-source work rather than self-contained synthetic tasks with limited context and algorithmic outcomes.Prior synthetic-task studies reported 56% and 21% coding speedups and a 65% increase in requirements-satisfied rate, but their tasks may not represent ordinary software development.Source: §1 · p. 1, §1.1 · p. 3
  • Prior field experiments found 14–51% gains in output metrics, but lines of code, commits, and pull requests were not fixed in advance and may not measure true productivity.AI could increase these outcomes through more verbose equivalent code or by splitting work into smaller pull requests.Source: §1 · p. 1, §1.1 · p. 3

1 Introduction

  • The randomized trial covered 246 tasks completed by 16 developers on familiar open-source repositories, comparing randomly assigned AI-allowed and AI-disallowed conditions.Developers primarily used state-of-the-art February–June 2025 tools, including Cursor Pro and Claude 3.5/3.7 Sonnet; screen recordings supported compliance and analysis.Source: §1 · p. 2
  • 19% slower completion: allowing AI increased experienced developers’ task completion time, despite 24% pre-task and 20% post-study forecasts of faster completion.The result contrasted with forecasts from both developers and economics and machine-learning experts, who substantially overestimated AI’s speedup.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • The slowdown remained broadly robust across experimental design decisions, though the study could not completely rule out experimental artifacts.Researchers evaluated 21 hypothesized properties, finding evidence for five contributors, mixed or unclear evidence for ten, and evidence against six.Source: §1 · pp. 2–3
  • The findings do not imply that current AI systems are generally unhelpful or that future models and improved prompting, scaffolding, or domain-specific fine-tuning cannot produce speedups.Many slowdown factors are specific to this study’s setting, and rapid AI capability progress leaves positive future effects possible.Source: §1 · p. 3

1.1 Background

  • The study focuses on highly skilled open-source developers because existing evidence suggests less experienced workers benefit more from generative AI, leaving this population relatively understudied.Participants had years of expertise in the target repositories, directly addressing heterogeneous effects by experience.Source: §1.1 · pp. 3–4
  • Its design combines frontier-era models, unfiltered live repository tasks, fixed pre-randomization outcomes, experienced engineers, and rich behavioral and qualitative data.The study also collected time-usage data, AI suggestions, screen recordings, interviews, and surveys to analyze mechanisms behind the observed effect.Source: §1 · p. 2, §1.1 · p. 4

2 Methodology

The study uses experienced contributors completing real repository issues under randomized AI-allowed or AI-disallowed conditions. It combines self-reported completion times with forecasts, analytics, interviews, qualitative evidence, and screen-recording labels to estimate AI’s effect.

2.1 Developers and Repositories

  • Experienced contributors worked on large, mature repositories where they averaged 5 years of repository experience and 1,500 commits.Repositories averaged 23,000 stars, 1,100,000 lines of code, 4,900 forks, 20,000 commits, and 710 committers.Source: §2.1 · p. 4

2.2 Experimental Design

  • Developers supplied real issues, forecast completion times with and without AI, and were assigned conditions by simulated fair-coin randomization.AI-allowed developers could use any tools or none, while AI-disallowed developers could not use generative AI; issues were completed through normal PR review and merging.Source: §2.2 · pp. 5–6

2.2.1 AI Tools and Training

  • Cursor Pro subscriptions and live basic training were provided, but developers could use other AI tools or no AI; 93% had prior ChatGPT-like experience and 44% had Cursor experience.Cursor offered chat and agent features for searching, editing, running commands, and iterating on programs.Source: §2.2.1 · p. 6

2.2.2 Data Collection

  • Developers largely followed their usual workflows while recording screens and reporting time spent before and after pull-request review.Screen recordings provided data on AI usage and working patterns, despite some developers using Cursor instead of their normal environments.Source: §2.2.2 · p. 6

2.3 Effect Estimation

  • The treatment effect is the percentage change in total implementation time, estimated with log-linear regression using forecasted issue difficulty to reduce noise.The measure includes pre- and post-review time; missing post-review times are imputed by treatment-condition means, with robustness checks for the imputation method.Source: §2.3 · pp. 6–7

2.4 Supplementary Data Collection

  • Supplementary data compare developer and expert forecasts with outcomes and examine AI use, perceptions, and qualitative experiences through Cursor analytics, exit interviews and surveys, and inductively coded notes.Source: §2.4 · p. 7

3 Results

Allowing AI increased implementation time by 19%, despite developers and experts forecasting substantial speedups. Activity labels suggest that AI shifted time from coding and information search toward prompting, reviewing outputs, waiting, and idling, while factor analysis found suggestive but inconclusive evidence for mechanisms behind the slowdown.

3 Results

  • 19%: AI-allowed issues took longer on average than AI-disallowed issues, with nearly all observed implementation-time quantiles also showing a slowdown.The raw percentage difference was 34%, but the regression-estimated slowdown was 19% because AI-allowed issues were slightly more difficult after randomization.Source: §3 · p. 8, §3.2 · p. 9

3.1 Forecasts

  • 24% and 20%: developers forecast and later estimated speedups from AI, while economics and machine-learning experts predicted 39% and 38% reductions in implementation time.Developer forecasts correlated with actual completion times at 0.64 for AI-allowed issues and 0.59 for AI-disallowed issues, indicating reasonable time calibration but reversed expectations about AI’s usefulness.Source: §3.1 · pp. 8–9

3.2 Activity Labels

  • AI allowance shifted developers’ time away from active coding and information search toward prompting, reviewing AI outputs, waiting for generations, and idling.This activity-label analysis covered 74 issues with valid screen recordings.Source: §3.2 · p. 9

3.3 Factor Analysis

  • 5 of 21 investigated factors had evidence contributing to slowdown, 10 had mixed or unclear evidence, and 6 had evidence against contribution.The analysis grouped potential factors into direct productivity loss, experimental artifacts, raised developer performance, and limited AI performance; the authors characterize the evidence as speculative and suggestive because subset analyses were not powered for multiple comparisons.Source: §3.3 · p. 10

4 Discussion … C.2.1 Experimentally driven overuse of AI (Experimental artifact)

The discussion concludes that AI slowed experienced developers in familiar, large open-source repositories, while emphasizing that this result is setting-specific and that experts and developers overestimated AI’s productivity benefits. It attributes the slowdown to developer over-optimism, repository familiarity and complexity, limited reliability and tacit context, while noting possible experimental overuse and future AI improvements.

4 Discussion

  • AI slowed experienced developers on real issues in large, familiar repositories, providing evidence that real-world AI capabilities may lag benchmark impressions.The authors stress that field experiments with robust outcome measures are important because expert forecasts and developer surveys substantially overstated AI’s usefulness.Source: §4 · pp. 10, 12

B Extended Discussion

  • The discussion distinguishes its evidence from prior studies and potential misreadings while acknowledging possible experimental overuse; Joel Becker and Nate Rush led the project, Beth Barnes provided guidance, and David Rein contributed to writing and framing.Prior work generally did not study frontier-beyond-GPT-4 systems in the wild with experienced developers and fixed productivity outcomes; the unblinded experiment may also have encouraged some participants to overuse AI.Source: B Extended Discussion · p. 17, Table 2 · p. 17, Table 3 · p. 18

4.1 Key Caveats

  • The study identifies 5 setting-specific factors contributing to slowdown and cautions against generalizing the result to all software development settings.Small greenfield projects or unfamiliar codebases may instead see substantial speedup; higher AI reliability, lower latency, better elicitation, or repository fine-tuning could also reverse the effect, and autonomous agents may soon improve further.Source: §4.1 · p. 12, §C.1 · p. 18, §C.2 · p. 20

C.1.1 Over-optimism about AI usefulness (Direct productivity loss)

  • 24%: developers forecast that AI would reduce completion time, but after completing issues they estimated a 20% reduction, encouraging potentially unproductive overuse.Developers were instructed to use AI as much or as little as they found helpful, yet this overoptimism may have led them to continue using tools that slowed them down.Source: §C.1.1 · p. 18

C.1.2 High developer familiarity with repositories (Raises developer performance)

  • High repository and task familiarity contributed to greater slowdown, whereas AI was described as more helpful on unfamiliar issues and less helpful when developers were experts.The study reports moderate evidence for greater slowdown with high prior task exposure and lower external resource needs, consistent with developers’ average 5 years and 1,500 commits of repository experience.Source: §C.1.2 · pp. 18–19

C.1.3 Large and complex repositories (Limits AI performance)

  • Repository complexity limited AI performance: participants reported erroneous edits, and the repositories averaged about 10 years old and >1,100,000 lines of code.The authors hypothesize that these characteristics explain weaker assistance than in more greenfield randomized trials, consistently with literature on environment complexity [3] [39].Source: §C.1.3 · p. 19

C.1.4 Low AI reliability (Limits AI performance)

  • <44%: developers accepted fewer than 44% of Cursor generations, while reviewing, testing, editing, and cleaning accepted outputs consumed substantial time.75% read every generated line, 56% often made major changes, 100% modified generated code, and review and cleaning occupied approximately 9% of AI-allowed time on 44 issues.Source: §C.1.4 · p. 20

C.1.5 Implicit repository context (Limits AI performance, Raises developer performance)

  • AI lacked developers’ tacit repository knowledge, such as hidden data assumptions and backwards-compatibility requirements, making assistance less useful in mature codebases.The authors hypothesize that repository size and maturity increase the tacit knowledge experienced developers use while limiting AI access to it.Source: §C.1.5 · p. 20

C.2.2 Unrepresentative task distribution (Experimental artifact)

The study’s issues resemble normal repository work, but their deliberate restriction to small, well-scoped tasks omits larger and non-issue contributions, making the selection’s effect on AI’s relative usefulness unclear.

Task representativeness

  • Issues were judged representative of normal repository work, but the study sampled from the small end by asking developers to split larger issues into ≤2-hour tasks.Developers described the issues as “completely standard,” while also noting they were smaller than their usual work.Source: §C.2.2 · p. 21
  • The task distribution excluded contributions such as pull-request review and design discussions, so it did not capture all developer work on these repositories.Source: §C.2.2 · p. 21

Experimental artifact

  • Selection may have produced better-scoped, clearer issues; this could help AI, but it could also make AI less relatively useful to expert developers, leaving the net effect unclear.Existing literature suggests AI performs better on better-scoped issues, whereas experts may also perform better on such tasks.Source: §C.2.2 · p. 21

C.2.3 AI increasing issue scope (Experimental artifact) · C.2.4 Bias from issue completion order (Experimental artifact)

The study examines whether AI-related scope expansion or issue-completion order could artifactually explain the observed slowdown. Evidence provides little support for scope expansion as the primary cause, while ordering effects cannot be fully excluded.

C.2.3 AI increasing issue scope (Experimental artifact)

  • Pre-randomizing issues avoids confounding, but flexible issue definitions leave open whether AI causes developers to expand implementation scope.If so, longer AI-allowed completion times might reflect more work rather than lower productivity.Source: §C.2.3 · p. 21
  • Developers reporting scope creep were slowed down less than those reporting no scope creep, arguing against scope expansion as the cause of slowdown.This evidence relies on subjective self-reports, so it may be unreliable.Source: §C.2.3 · p. 21, Figure 9 · p. 22
  • Some experienced developers reported that AI did not change their implementations or issue scope.They attributed this to knowing how changes should be made and finding AI more suggestible than themselves.Source: §C.2.3 · p. 21
  • Other developers reported that AI encouraged them to undertake more extensive or tedious code changes.These accounts indicate that AI can sometimes increase issue scope, even though the broader evidence does not establish it as the main slowdown mechanism.Source: §C.2.3 · pp. 21–22
  • 47% more code per hour was observed on AI-allowed issues relative to AI-disallowed forecasts, but the difference was not statistically significant (p = 0.16).Lines of code are noisy because verbosity, generated files, and unavailable human-only line counts may not reflect productivity or scope.Source: §C.2.3 · p. 22

C.2.4 Bias from issue completion order (Experimental artifact)

  • Completion-order bias could make AI-allowed issues appear slower if developers preferentially complete AI-disallowed issues when they have more energy.Although developers did not qualitatively report this behavior, incomplete workflow data means the effect cannot be fully ruled out.Source: §C.2.4 · p. 22

C.2.5 Sampling bias in developer recruitment (Experimental artifact) · C.2.6 Trading speed for ease (Direct productivity loss) · C.2.7 Low quality initial pull requests (Direct productivity loss)

The study considers recruitment bias and two direct productivity-loss explanations for AI-related slowdowns. Evidence suggests continued Cursor use reflects meaningful value, while lower-quality initial pull requests do not explain the observed delay.

C.2.5 Sampling bias in developer recruitment (Experimental artifact)

  • Recruitment may systematically exclude developers who rely heavily on AI because they would dislike losing AI access on 50% of tasks.One developer later said they were unlikely to participate again for this reason.Source: §C.2.5 · p. 22
  • Prior Cursor, Copilot, or web-LLM experience did not visibly change the slowdown pattern, with prior-Cursor users slowed similarly to users without prior Cursor experience.The figure reports no difference between developers with and without Copilot or web-LLM experience.Source: Figure 10 · p. 23

C.2.6 Trading speed for ease (Direct productivity loss)

  • Developers may trade speed for ease because AI assistance can make work feel less effortful or reduce deliberation, although qualitative reports were mixed.Some developers described AI-assisted issue completion as requiring less effort or thought, while another reported no difference in effort or approach.Source: §C.2.6 · p. 23
  • 69% of developers continued using Cursor after the study, suggesting substantial value beyond productivity gains and making pure productivity miscalibration unlikely.Developers gave mixed qualitative reports about whether AI-assisted work felt easier or required less thought.Source: §C.2.6 · p. 23

C.2.7 Low quality initial pull requests (Direct productivity loss)

  • Lower-quality AI-generated code could require extra post-review fixes, but the observed qualitative PR quality and statistically insignificant timing difference provide little support for this explanation.This explanation concerns insufficient review or correction of AI-generated code before submission.Source: §C.2.7 · p. 23
  • Post-review implementation time averaged 9 for AI-disallowed issues versus 15 for AI-allowed issues, a statistically insignificant difference.Thus, the data do not establish that lower-quality AI-allowed PRs caused extra post-review work.Source: §C.2.7 · p. 23
  • 100% of developers reported submitting only high-quality pull requests, and AI-allowed PRs did not qualitatively appear lower quality.Experienced contributors anticipated their repositories’ demanding review standards.Source: §C.2.7 · p. 23

C.2.8 Below-average use of AI tools (Limits AI performance) … C.3 Factors unlikely to contribute to slowdown

The paper finds suggestive evidence that six examined factors do not contribute substantially to the observed slowdown. In particular, limited AI experience, generation latency, and ordinary elicitation do not fully account for the result, though learning effects and alternative elicitation strategies remain uncertain.

C.2.8 Below-average use of AI tools (Limits AI performance)

  • Prior Cursor experience shows no meaningful difference in AI-related completion-time changes across developers.Only 44% of developers had prior Cursor experience, but experience level did not predict the slowdown.Source: §C.2.8 · p. 23
  • Excluding up to the first eight AI-allowed issues leaves similar slowdown percentages, arguing against short-term skill acquisition as the main explanation.Developers also appeared broadly comparable to the authors in Cursor use, although not necessarily optimal.Source: Figure 11 · p. 24, §C.2.8 · pp. 24–25
  • Beyond 50 hours of Cursor experience, one developer showed positive speedup, but the analysis is underpowered and difficult to interpret.A later correction for another developer with more than 100 prior hours would move the >50-hour estimate from roughly 25% speedup to roughly 0%; AI use might also weaken unaided skills.Source: §C.2.8 · pp. 24–25

C.2.9 AI generation latency (Limits AI performance)

  • 4% of developers’ time on 44 valid screen-recorded issues was spent waiting for AI-generated outputs when AI was allowed.Waiting was substantial for a minority of developers, but not for everyone.Source: §C.2.9 · p. 25

C.2.10 Suboptimal elicitation (Limits AI performance)

  • Limited token spending could contribute to weak performance, but it cannot explain differences from prior programming studies, and higher-spend strategies were not evaluated.The study’s developers typically used Cursor and web LLMs rather than parallel sampling with an LLM judge or self-consistency.Source: §C.2.10 · p. 25

C.3 Factors unlikely to contribute to slowdown

  • Six examined factors are suggested not to contribute to the observed slowdown, as summarized in Table 1.Source: §C.3 · p. 25

C.3.1 Unfamiliar development environment (Experimental artifact)

The slowdown persists when developers use comparable IDEs across conditions, and it does not vary meaningfully with the first 50 hours of Cursor use, suggesting unfamiliarity with Cursor is not the main cause.

Unfamiliar development environment (Experimental artifact)

  • A 24% slowdown remains among developers using comparable IDEs between treatment conditions, indicating that unfamiliar development environments do not explain the effect.This restriction addresses the possibility that differences in keyboard shortcuts or personalized tooling caused the slowdown.Source: §C.3.1 · p. 26
  • Cursor usage shows no meaningful change in slowdown across the first 50 hours, while developers rarely report usability bottlenecks.Together, these findings suggest that learning Cursor and personalizing its features does not substantially contribute to the slowdown.Source: §C.3.1 · p. 26

C.3.2 Cheating or under-use of AI (Experimental artifact)

Screen-recording evidence indicates that the observed slowdown was not primarily caused by developers under-using AI when allowed or cheating when AI was disallowed. AI was used in most allowed tasks, while observed cheating was uncommon.

C.3.2 Cheating or under-use of AI (Experimental artifact)

  • Screen recordings provide evidence that slowdown was not driven by AI under-use in allowed tasks or cheating in disallowed tasks.Source: §C.3.2 · p. 26
  • 16.4% of 44 manually labeled valid recordings with AI allowed showed no AI use, indicating that assistance was typically used when permitted.Source: §C.3.2 · p. 26
  • ~6% of 54 labeled disallowed recordings contained observed AI use, with at least one cheating instance appearing accidental.Source: §C.3.2 · p. 26

C.3.3 Issue dropout (Experimental artifact)

Issue dropout could bias the comparison if developers selectively abandoned tasks by AI condition, but multiple robustness checks and manual review indicate dropout is unlikely to explain the observed slowdown.

Potential selection bias

  • Selective issue completion could bias results if developers chose easier AI-disallowed issues after assignment.Under that mechanism, AI-allowed issues would be harder on average and could produce an apparent slowdown.Source: §C.3.3 · p. 26

Uncompleted assigned issues

  • Developers completed all assigned issues by commitment, but 7 AI-disallowed and 6 AI-allowed issues remained incomplete.Because developers were not full-time repository workers, some failed to meet their completion commitments.Source: §C.3.3 · p. 26

Robustness to dropout

  • Slowdowns remain 21–42% when excluding developers with uncompleted issues in either condition or with any dropout.The slowdown is 42% among developers with no uncompleted AI-disallowed issues, 21% with no uncompleted AI-allowed issues, and 21% with no dropout on any issue.Source: §C.3.3 · p. 26

Intentional issue dropout

  • Manual review found no evidence that 24 intentionally dropped issues were abandoned because of AI condition.The leading reasons were excessive implementation difficulty (11 issues), loss of relevance (5), and implementation by another developer (3).Source: §C.3.3 · p. 26

C.3.4 Non-robust outcome measure (Experimental artifact) … D.2 Confidence intervals

Robustness checks indicate that the observed AI slowdown is not driven by the outcome measure, estimator choice, or use of older models. The empirical strategy uses randomized treatment assignment with forecast-adjusted regression and multiple confidence-interval procedures.

C.3.4 Non-robust outcome measure (Experimental artifact)

  • Extreme imputation biases still produce 14% and 23% slowdowns when AI is allowed, leaving speedup estimates substantially unchanged.The extreme scenarios assign one condition one hour and the other zero minutes for unreviewed post-review times.Source: §C.3.4 · p. 27
  • 25% slowdown from screen-recording duration closely matches the 24% slowdown from self-reported implementation times.This comparison supports the robustness of the slowdown measurement to self-reported time bias.Source: §C.3.4 · p. 27

C.3.5 Non-robust estimator (Experimental artifact)

  • Alternative estimators—including a simple ratio and three regression specifications—report similar results, indicating slowdown is robust to estimator choice.The ratio compares mean time with AI allowed against mean time with AI disallowed, while regression variants change covariate specifications.Source: §C.3.5 · p. 27

C.3.6 Non-frontier model usage (Limits AI performance)

  • Developers used Claude 3.7 Sonnet thinking mode, Claude 3.7 Sonnet, and Claude 3.5 Sonnet in 25%, 34%, and 23% of labeled AI-allowed issues.Other models were GPT-4o (11%), Gemini 2.5 Pro (3%), and o1 (2%); developers prompted no AI model 16.4% of the time, excluding widespread autocomplete use.Source: §C.3.6 · p. 28

D.1 Regression

  • The regression models each issue’s realized completion time using a binary AI-treatment indicator and the developer’s ex-ante forecast.Ordinary least squares is used for the log-linear specification.Source: §D.1 · p. 28

D Empirical Strategy

  • Random assignment identifies the treatment effect, while pre-treatment completion forecasts proxy for issue difficulty and improve statistical power without introducing bias.The regression estimates the difference in expected log completion times between AI-allowed and AI-disallowed issues.Source: §D.1 · p. 28

D.2 Confidence intervals

  • 95% HC3 confidence intervals are the default; developer-clustered and bias-corrected cluster-robust errors are similar, while hierarchical bootstrap intervals are wider.The hierarchical bootstrap resamples developers and then issues within developers.Source: §D.2 · p. 28

D.3 Converting to speedup … E.1 Balance and proportions checks

The analysis converts log-scale treatment effects into speedup measures, estimates subgroup-specific effects, and checks robustness with ratio-based estimators. Randomization checks show balanced forecasted difficulty and no statistically significant deviation from intended treatment proportions.

D.3 Converting to speedup

  • The regression specification expresses conditional expected log time as α + β + δ log bT NoAI with AI and α + δ log bT NoAI without AI.These conditional expectations motivate interpreting β as the conditional log-scale treatment effect.Source: §D.3 · p. 28
  • β equals the average treatment effect on log implementation time, conditional on forecast, and speedup is obtained as S = exp(β) − 1.Confidence intervals for S are constructed by transforming the endpoints of the confidence interval for β.Source: §D.3 · pp. 28, 30

D.4 Heterogeneous treatment effects

  • For binary subgroup characteristic X, treatment effects are β1 for X = 0 and β1 + β3 for X = 1, transformed into SX=0 = exp(β1)−1 and SX=1 = exp(β1+β3)−1.Interaction terms estimate differential treatment effects across subgroups, with confidence intervals based on linear hypotheses and the same transformation.Source: §D.4 · p. 30

D.5 Ratio estimator

  • The ratio-of-means estimator is Ŝratio = ¯TAI-allowed / ¯TAI-disallowed − 1, with 95% confidence intervals obtained from pooled-bootstrap quantiles.The estimator also converts aggregate percentage changes implied by developer forecasts, distinct from averaging task-level percentage changes.Source: §D.5 · pp. 30–31

D.6 Participant post-study estimator

  • Post-study speedup estimates are weighted arithmetic means of developers’ reported time ratios, weighted proportionally to each developer’s completed issue count.The weighted mean’s 95% confidence interval uses a weighted-variance standard error and normal approximation.Source: §D.6 · p. 31

E.1 Balance and proportions checks

  • Randomization produced balanced forecasted issue difficulty, and treatment-assignment proportions did not differ significantly from the intended 50/50 allocation.Table 4 reports the balance comparison, while Table 5 reports the allocation-proportion test.Source: §E.1 · p. 31, Table 4 · p. 31, Table 5 · p. 31

E.2 Per-developer speedup and forecast calibration … G.2 Developer Instructions and Survey Data

This section examines per-developer speed effects and forecast calibration, alongside the study’s open-source setting, tooling, recruitment, incentives, and task procedures. It describes how tasks were randomized and how implementation details, time, effort, recordings, and resulting pull requests were documented.

E.2 Per-developer speedup and forecast calibration

  • 75% of developers experienced slowdown with AI, while AI-allowed issues took longer on average despite nearly identical median no-AI time forecasts.These per-developer estimates use the study’s standard methodology for heterogeneous effects.Source: §E.2 · p. 32, Figure 17 · p. 32
  • Developers remained relatively calibrated: predicted speedups of at least 33% corresponded to no slowdown, with slowdown decreasing monotonically as forecasted speedup increased.Forecasted-speedup bins are approximately similarly sized, though duplicate forecasts make the tertiles imbalanced.Source: §E.2 · p. 32, Figure 18 · p. 33

E.3 Randomization

  • Excluding the 25 early block-randomized issues left the estimated slowdown at 20%, so the study retained those issues to increase statistical power.The researchers abandoned block randomization because developer issue lists were too small and switched to a simulated fair coin flip.Source: §E.3 · p. 33

E.4 Fine-Grained Screen Recording Labels

  • The study analyzed time spent on activities using coarse and 27-category fine-grained screen-recording labels, including comparisons between AI-allowed and AI-disallowed issues.Figures 19–21 provide activity breakdowns, with additional labeling detail referenced in Sections 2.4 and G.8.Source: §E.4 · p. 33, Figure 19 · p. 33, Figure 20 · p. 34

G Recruitment and Onboarding

  • Participants were recruited from active contributors to open-source repositories, filtered for substantial recent contribution activity and prior repository experience, then trained to use Cursor Pro.The recruitment process began with 51 interest-survey respondents and narrowed to about 20 participants before attrition.Source: G Recruitment and Onboarding · p. 37

G.1 Incentivization Scheme

  • Developers were paid $150 per hour rather than per completed issue, a choice intended to avoid incentives to split tasks or reduce quality standards.The researchers expected hourly payment to have minimal behavioral effects beyond encouraging participation.Source: §G.1 · p. 37

G.2.1 Developer Instructions

  • Issues were randomized into an AI bucket permitting AI assistance and a no-AI bucket prohibiting AI tooling.Participants could work in any issue order and use any non-AI tools, with Cursor Pro access and basic training provided when needed.Source: §G.2.1 · p. 40
  • Participants recorded models and usage locations, implementation notes, pull-request links, screen recordings, and separate active times before and after review.Total completion time was defined as initial implementation time plus post-review time until the change was mergeable.Source: §G.2.1 · pp. 40–41
  • Initial implementation time covered understanding, coding, testing or checking work, and preparing the pull request for review.Only active participant time counted, excluding waiting periods such as time awaiting review.Source: §G.2.1 · p. 41
  • Post-review implementation time covered active work on requested changes and merge conflicts, and was zero when no substantive review work was required.Waiting for review did not count toward this category.Source: §G.2.1 · p. 41
  • Participants rated each issue’s perceived effort from 1, minimal effort, to 5, maximum effort, using examples ranging from simple text changes to complex production debugging.Intermediate ratings represented below-average, average, and above-average effort.Source: §G.2.1 · pp. 41–42

G.3 Onboarding call and Cursor Training … Background

This section describes the study’s onboarding, training, task assignment, compliance, and measurement procedures, along with screen-recording labeling and expert-forecaster briefing. It also documents participant experiences and potential sources of bias related to exclusions and recording analysis.

G.3 Onboarding call and Cursor Training

  • Developers completed a 30-minute onboarding call after setting up Loom, installing Cursor for their codebase, and reviewing required study data.Training covered creating and managing Cursor agent-mode instances, adding files to context, prompting changes, accepting suggestions, and reverting them; developers also received a data-collection overview and could ask questions.Source: §G.3 · p. 42

G.4 Mid-experiment check-in calls

  • Most developers had 1–4 periodic 15-minute check-in calls to monitor progress, answer questions, and discuss their experiences using AI during the experiment.The calls were also intended to help ensure timely issue completion.Source: §G.4 · p. 42

G.5 Exit Interview

  • All developers participated in a 30-minute–1-hour unstructured exit interview designed to elicit qualitative experiences after study completion.Interviews confirmed prior AI and Cursor use, reviewed tracked issue data, and asked about task selection, effort, code cleaning, scope creep, future AI use, skill development, waiting or distraction, effectiveness, and study experience.Source: §G.5 · pp. 42–43, Exit interview · p. 43

Exit interview:

  • The exit interview covered developers’ comparisons with normal repository work, effort and focus, AI-output quality and cleaning, issue scope, future AI use, AI skill, time effects, effective tools, and study experience.Questions also addressed whether AI sped developers up or slowed them down, preferred interfaces and models, and whether the study changed their beliefs or usage.Source: Exit interview · pp. 43–44

G.5.1 Exit Survey

  • The exit survey measured prior LLM and Cursor experience, end-of-study Cursor skill, repository quality practices, and perceived AI-related changes in issue-completion time.For time effects, respondents entered 2 when AI made issue completion twice as fast and .5 when it made completion twice as slow.Source: §G.5.1 · pp. 44–45

G.6 Participant Dropout

  • Three developers’ work was excluded from all results: two because their repositories paused development indefinitely and one because of widespread cheating in the first issues submitted.All three developers were compensated fully.Source: §G.6 · p. 45

G.8 Screen Recordings

  • Screen-recording labeling may bias time-allocation estimates because labeling began early, prioritized shorter recordings, and developers often omitted recordings for small review-related changes.The labeling protocol categorized general development activities and AI interactions, including prompts, generation waiting, reviewing, accepting or rejecting suggestions, and AI code cleaning, alongside model and interface labels.Source: §G.8 · pp. 45, 46, 48, 49
  • Screen-recording labels used roughly accurate timestamps, distinguished writing from sustained reading, and assigned unclear Cursor interactions to “unknown.”Writing was labeled when it occupied more than half of a time chunk, while reading required roughly 30 seconds of reading activity.Source: §G.8 · p. 50

G.9 Instructions Given to Expert Forecasters

  • Expert forecasters were told the study randomized experienced open-source developers’ tasks to AI access and measured the ratio of completion times with versus without AI.At the time of these instructions, the study was still underway and its final result was not yet known.Source: §G.9 · p. 50

Background

  • The study recruited active contributors to repositories with more than 500 stars and trained 16 retained developers on Cursor agent mode during individual 30-minute calls.Developers were considered trained after independently prompting, accepting, and reverting file changes; 94% had previously used web-based LLMs.Source: Background · pp. 50–51
  • Issues were randomized to AI-allowed or AI-disallowed conditions, with participants recording screens and writing post-issue implementation notes to support compliance checks.AI-allowed developers could use any AI tools, including none; AI-disallowed developers could use no generative AI tooling.Source: Background · p. 51
  • Speedup was defined as S = mean(completion time with no AI) / mean(completion time with AI allowed), where S = 1 denotes equal completion times.The briefing explained that S = 2 represents a 100% speedup and S = 0.5 represents a -50% speedup.Source: Background · p. 51

Related papers

Loading 2507.09089v2…