PaperlayerView exact revision on arXiv

Source-linked AI summary

Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

Joel Becker, Nate Rush, Elizabeth Barnes, David Rein

arXiv:2507.09089v2cs.AIcs.HCcs.SE

TL;DR

AI’s real-world effect on software-development productivity remains uncertain because synthetic benchmarks and existing field measures may not capture actual productivity. This randomized controlled trial tests frontier early-2025 tools on experienced open-source developers’ real repository tasks and finds a 19% slowdown despite developers’ and experts’ expectations of substantial speedup. The paper examines potential contributors and cautions that the result is setting-specific.

  • Problem

    Synthetic software evaluations and existing field experiments provide limited evidence about AI’s actual productivity impact on realistic development tasks.

  • Method

    A randomized controlled trial assigns 246 tasks completed by 16 experienced open-source developers to allow or disallow early-2025 AI tools.

  • Results

    19% more time is required when AI tools are allowed, contrary to developers’ and experts’ predicted speedups.

  • Takeaways & Limitations

    Field experiments with robust outcome measures are important because both experts and developers can drastically overestimate AI’s productivity usefulness.

  • Takeaways & Limitations

    The slowdown is setting-specific and does not imply that current AI tools generally fail to improve productivity or that future models will not speed developers in this setting.

Abstract

from arXiv · show

Despite widespread adoption, the impact of AI tools on software development in the wild remains understudied. We conduct a randomized controlled trial (RCT) to understand how AI tools at the February-June 2025 frontier affect the productivity of experienced open-source developers. 16 developers with moderate AI experience complete 246 tasks in mature projects on which they have an average of 5 years of prior experience. Each task is randomly assigned to allow or disallow usage of early 2025 AI tools. When AI tools are allowed, developers primarily use Cursor Pro, a popular code editor, and Claude 3.5/3.7 Sonnet. Before starting tasks, developers forecast that allowing AI will reduce completion time by 24%. After completing the study, developers estimate that allowing AI reduced completion time by 20%. Surprisingly, we find that allowing AI actually increases completion time by 19%--AI tooling slowed developers down. This slowdown also contradicts predictions from experts in economics (39% shorter) and ML (38% shorter). To understand this result, we collect and evaluate evidence for 20 properties of our setting that a priori could contribute to the observed slowdown effect--for example, the size and quality standards of projects, or prior developer experience with AI tooling. Although the influence of experimental artifacts cannot be entirely ruled out, the robustness of the slowdown effect across our analyses suggests it is unlikely to primarily be a function of our experimental design.

1 Introduction

Existing software benchmarks and synthetic-task experiments provide limited evidence about AI’s impact on real-world development. This study addresses that gap with randomized evaluation of experienced open-source developers and finds that AI slowed them despite widespread expectations of speedup.

  • Motivation: Synthetic software tasks often sacrifice realism, limiting inferences about AI’s practical productivity impact.They are typically self-contained, require little prior context, and use algorithmic metrics that omit important capabilities.
  • Study design: The study randomly assigns 246 tasks completed by 16 experienced developers to allow or disallow February–June 2025 AI tools.Tasks come from well-known repositories contributors regularly use; developers primarily use Cursor Pro with Claude 3.5/3.7 Sonnet.
  • Main result: 19% more time is required when AI tools are allowed, reversing developers’ expected productivity gain.Developers forecast a 24% reduction before randomization and estimated a 20% reduction after participation.
  • Mechanisms and robustness: The analysis evaluates 21 hypothesized contributors to slowdown, finding evidence for 5, mixed or unclear evidence for 10, and evidence against 6.Screen recordings, source-code statistics, interviews, surveys, and subset analyses support the factor analysis, though experimental artifacts cannot be completely ruled out.
  • Caveats: The findings should not be generalized to all software settings or future AI systems.Repository familiarity and project size and maturity contribute to the observed slowdown, while small greenfield projects or unfamiliar codebases may see speedup.
  • Expectation gap: Experts and developers substantially overestimate AI’s usefulness for productivity in this setting.The observed slowdown contrasts with forecasts from both study participants and external economics and machine-learning experts.

2 Methodology

The study measures AI’s effect on real software-development work through randomized assignments of experienced contributors’ repository issues to AI-allowed or AI-disallowed conditions. It uses developer forecasts, self-reported implementation times, repository workflows, and supplementary observations to estimate completion-time differences and examine design constraints.

  • 2.1 Developers and Repositories: Experienced developers contribute real issues to large, mature repositories where they have substantial prior experience.Repositories average 23,000 stars, 1,100,000 lines of code, 4,900 forks, 20,000 commits, and 710 committers.
  • 2.2 Experimental Design: Each issue is forecast under both AI conditions, then randomly assigned by a simulated fair coin flip to allow or disallow generative AI.Forecasts proxy issue difficulty and measure anticipated per-issue speedup; AI-allowed developers may use any tools or none, while AI-disallowed developers cannot use generative AI.
  • 2.2 Experimental Design: Developers complete assigned issues in flexible workflows, submit pull requests for review, make requested changes, and report total implementation time before and after review.The study includes real repository work submitted through standard source-code management systems, with high review standards and rare mistakes in merged pull requests.
  • 2.2.1 AI Tools and Training: The study provides Cursor Pro training but permits developers to choose any AI tools or no AI assistance.Cursor offers chat and agent features for searching and editing files, running commands, and iterating on programs; developers may retain their preferred workflows otherwise.
  • 2.3 Effect Estimation: The estimated effect S is the percentage change in expected completion time with AI allowed relative to AI disallowed.S = E[completion time with AI allowed] / E[completion time with AI disallowed] − 1; negative values are called speedup and positive values slowdown.
  • 2.3 Effect Estimation: The analysis uses total self-reported implementation time, imputes missing post-review time by treatment-condition means, and estimates S with difficulty-adjusted log-linear regressions.The study also adapts the regression for subset analyses and reports robustness to the specific imputation method.
  • 2.4 Supplementary Data Collection: Developer forecasts are collected before randomization to measure issue difficulty, anticipated AI speedup, calibration, and beliefs about AI helpfulness.Post-study estimates are compared with measured effects to assess the relationship between beliefs and observed productivity.

3 Results

AI allowance increased implementation time across the studied issues, despite developers and experts forecasting substantial speedups. Activity labels indicate that AI shifts time away from coding and information search toward prompting, reviewing, waiting, and idling, while the factor analysis finds mixed evidence across proposed explanations.

  • 3 Results: 19% more time was required on average for AI-allowed issues than AI-disallowed issues.Developers completed 136 AI-allowed and 110 AI-disallowed issues, and nearly all observed-time quantiles were longer with AI allowed.
  • 3.1 Forecasts: 24% forecasted speedup and 20% post-study estimated speedup contrasted with the observed 19% slowdown.The raw implementation-time difference was 34%, while regression adjustment produced the 19% estimate because AI-allowed issues were slightly more difficult after randomization.
  • 3.1 Forecasts: Developers’ forecasts correlated with actual completion times at 0.64 for AI-allowed issues and 0.59 for AI-disallowed issues.Their estimates of relative issue difficulty were informative, but their expectations about AI usefulness were reversed.
  • 3.1 Forecasts: 34 economics experts predicted a 39% reduction and 54 machine-learning experts predicted a 38% reduction in implementation time.Both expert groups overestimated speedup more than the developers did.
  • 3.2 Activity Labels: With AI allowed, developers spent less time coding and reading or searching, but more time prompting, reviewing outputs, waiting for generations, and idling.These activity comparisons come from labeled recordings covering 74 issues and are also summarized as average minutes by category.
  • 3.3 Factor Analysis: The study evaluates 21 hypothesized contributors across direct productivity loss, experimental artifact, developer-performance advantages, and AI-performance limits.Evidence supports 5 factors, is mixed or unclear for 10, and contradicts 6; the authors caution that these subset analyses are suggestive rather than powered for multiple-comparison significance.

4 Discussion

The study finds that recent AI tools slowed experienced open-source developers in this setting, despite developers and experts expecting substantial speedups. The authors identify repository familiarity, repository scale, AI reliability, and developer review practices as relevant considerations while cautioning against broad generalization.

  • Setting-specific factors: Repository familiarity and size or maturity may reduce AI’s relative usefulness compared with small greenfield projects or unfamiliar codebases.Participants averaged five years and 1,500 commits on their repositories, which averaged about 10 years old and over 1,100,000 lines of code.
  • AI-specific factors: Low AI reliability generated additional work because developers accepted fewer than 44% of Cursor generations and often reviewed, edited, or reverted outputs.Seventy-five percent read every line of generated code, 56% often made major cleanup changes, and developers spent approximately 9% of time reviewing and cleaning outputs on a labeled subset.
  • AI-specific factors: AI often lacked tacit repository knowledge that experienced developers use for design and implementation decisions.Developers reported that models missed appropriate edit locations and overlooked undocumented data, compatibility, and edge-case requirements.

C.2.2 Unrepresentative task distribution (Experimental artifact)

The study’s task-selection process may have produced issues unlike developers’ average repository work, but the direction of its effect on AI’s relative usefulness is unclear.

  • Developers’ reported AI-use patterns included both experimentation and normal usage, with greater slowdown among those experimenting or using AI as much as possible.
  • Issues were intended to resemble normal repository work, and developers and manual review supported that characterization.
  • The study’s two-hour issue constraint selected tasks from the smaller end of developers’ usual work and omitted contributions such as pull-request review and design discussions.
  • Better-scoped issues could help AI perform better, but could also make experienced developers relatively harder to improve, leaving the net selection effect unclear.

C.2.3 AI increasing issue scope (Experimental artifact)

The study examines whether AI increases issue scope or introduces workflow artifacts that could explain the slowdown. Evidence is mixed: some developers expanded work with AI, but measured indicators do not clearly attribute the slowdown to scope creep or ordering.

  • Issues were defined before randomization, but flexible scope allowed developers to expand implementations differently across AI conditions.
  • Developers reporting scope creep on AI-allowed issues were slowed down less than those reporting no scope creep, providing evidence against scope expansion as the slowdown’s explanation.
  • Some developers said AI did not change their implementation scope, while others reported making more extensive tooling or code changes with AI.
  • 47% more code per hour was associated with AI-allowed issues, but the difference was not statistically significant and lines of code are a noisy productivity proxy.
  • Batch randomization allowed developers to choose issue order, creating a possible energy-related completion-time bias that the study could not fully rule out.

C.2.5 Sampling bias in developer recruitment (Experimental artifact)

Recruitment may have underrepresented developers who rely heavily on AI, potentially affecting how effectively participants used it. However, retention and quality evidence complicate that concern.

  • The participant sample may contain systematic recruitment bias because developers heavily reliant on AI could avoid a study restricting AI on half their tasks.
  • One developer explicitly said they were unlikely to join future iterations because they did not want AI restrictions on 50% of their work.
  • 69% of developers continued using Cursor after the study, suggesting participants received meaningful value from the tool beyond simple productivity miscalibration.
  • AI-allowed issues showed no statistically significant difference in mean post-review implementation time: 9 versus 15 for AI-disallowed and AI-allowed issues, respectively.
  • All developers reported submitting only high-quality pull requests, consistent with their experience and awareness of repository review standards.

C.2.8 Below-average use of AI tools (Limits AI performance)

The analyses find little evidence that limited AI-use skill or prior experience primarily explains the slowdown, though learning effects remain difficult to interpret. Tool latency and elicitation choices are additional performance constraints.

  • 44% of developers had prior Cursor experience, but prior AI-tool experience was not associated with meaningful differences in slowdown.
  • Up to 50 hours of Cursor experience showed no large speedup differences, whereas greater than 50 hours showed positive speedup, but the analysis was underpowered and difficult to interpret.
  • Excluding up to the first eight AI-allowed issues did not meaningfully change slowdown, arguing against short-term lack of basic AI-use skills as the main explanation.
  • AI-assisted work could weaken unaided development skills over time, making apparent speedup reflect weaker AI-disallowed performance rather than stronger AI-allowed performance.
  • Developers spent approximately 4% of time waiting for AI outputs on 44 recorded issues, a small but non-trivial latency cost.
  • Participants often used Cursor’s agent/composer or chat, while ordinary workflows typically sampled only a few thousand tokens and did not test higher-spend elicitation strategies.
  • The study provides suggestive evidence that six hypothesized factors do not contribute to the observed slowdown.

C.3.1 Unfamiliar development environment (Experimental artifact)

The slowdown persists when developers use comparable IDEs, and it does not vary meaningfully with their first 50 hours of Cursor use.

  • 24% slowdown remains among developers using comparable IDEs across treatment conditions.
  • Cursor unfamiliarity is unlikely to explain the slowdown because it does not meaningfully differ across developers’ first 50 hours of Cursor usage.Developers also rarely report issues or bottlenecks using Cursor.

C.3.2 Cheating or under-use of AI (Experimental artifact)

Screen recordings indicate that slowdown is not explained by failing to use AI when allowed or using it when disallowed.

  • Screen recordings provide evidence that neither AI under-use in allowed tasks nor cheating in disallowed tasks drives the slowdown.
  • AI was used on most allowed issues, with developers not using AI in only 16.4% of 44 valid recordings.
  • Only 3 of 54 disallowed recordings showed AI use, implying an approximately 6% cheating rate.At least one observed instance appeared accidental.

C.3.3 Issue dropout (Experimental artifact)

The study’s slowdown remains when analyses account for incomplete or intentionally dropped issues, reducing concern that issue dropout explains the result.

  • Issue selection could bias results if developers dropped tasks based on AI eligibility, but the robustness checks do not appear to support dropout as the main explanation.Twenty-four issues were intentionally dropped, chiefly because they were too hard, no longer relevant, or implemented by another developer.
  • Developers committed to completing all assigned issues, but 3 developers missed 7 disallowed issues and 4 missed 6 allowed issues.
  • 21% slowdown remains among developers with no dropout on any issues.The slowdown is 42% among developers with no uncompleted disallowed issues and 21% among those with no uncompleted allowed issues.

C.3.4 Non-robust outcome measure (Experimental artifact)

The slowdown is robust to alternative time measures, estimator specifications, and model choices, though the study reports uncertainty intervals and limited coverage of newer models.

  • 25% slowdown using total screen-recording time is similar to the 24% slowdown using self-reported implementation time.
  • Alternative estimators all report similar results, indicating robustness to the particular estimator specification.These include a simple ratio estimator and regression-based estimators with different covariates.
  • Developers used Claude 3.7 Sonnet thinking mode, Claude 3.7 Sonnet, and Claude 3.5 Sonnet in 25%, 34%, and 23% of allowed issues, respectively.They also used GPT-4o in 11%, Gemini 2.5 Pro in 3%, and o1 in 2%; autocomplete was used on most allowed issues.
  • Most issues were completed in February and March 2025, before Claude 4 Opus or Gemini 2.5 Pro were released.
  • The primary regression estimates log completion time from AI allowance while controlling for the pre-treatment log forecast of no-AI completion time.Random assignment identifies the AI coefficient, while forecasts proxy for issue difficulty and improve statistical power without treatment-induced bias.
  • The study reports 95% confidence intervals using HC3 standard errors, with similar clustered estimates and somewhat wider hierarchical-bootstrap intervals.

D.3 Converting to speedup

This section defines speedup from regression-based treatment effects and describes alternative estimators, uncertainty procedures, subgroup effects, and randomization checks.

  • Regression-based speedup: β represents the average treatment effect on log implementation time conditional on forecast.The regression compares expected log time with AI allowed versus disallowed at the same forecasted no-AI time.
  • Regression-based speedup: S = exp(β) −1 converts the log-scale treatment effect into a percentage speedup measure.Because the transformation is monotonic, confidence intervals for β can be transformed into confidence intervals for S.
  • Heterogeneous effects: Subgroup speedups are estimated with interaction terms, yielding β1 for X = 0 and β1 + β3 for X = 1.These effects are transformed as SX=0 = exp(β1)−1 and SX=1 = exp(β1+β3)−1.
  • Alternative estimators: The ratio-of-means estimator compares mean implementation times between AI conditions and uses pooled bootstrap quantiles for 95% confidence intervals.The estimator is used as a robustness check and is distinct from averaging per-task percentage changes.
  • Randomization checks: Randomization produced balanced forecasted difficulty and no statistically significant difference in treatment-assignment proportions.Table 4 reports no meaningful difference in forecasted difficulty, while Table 5 tests deviation from intended 50/50 allocation.

E.2 Per-developer speedup and forecast calibration

This section examines speedup across developers and forecasted effects, finding widespread slowdown but less slowdown on issues developers expected AI to help most.

  • Forecast calibration: The median forecasted implementation time is almost identical across conditions, yet AI-allowed issues take longer on average.The comparison is shown as a relationship between forecasted no-AI time and observed implementation time, using log axes.
  • Per-developer speedup: 75% of developers experience slowdown when AI is allowed.Per-developer estimates are computed using the study’s standard methodology for heterogeneous treatment effects.
  • Forecast calibration: Developers are not slowed down on issues with forecasted speedup of at least 33%, and slowdown decreases monotonically as forecasted speedup rises.Forecasted-speedup bins are chosen to be approximately similar in size, though duplicate forecasts create imbalanced tertiles.
  • Robustness: Excluding the 25 block-randomized issues leaves a 20% slowdown, so those issues are retained to increase statistical power.The block-randomization strategy was abandoned because developer issue lists were too small for it to be viable.
  • Additional analyses: The section presents additional speedup breakdowns by activities, outcome measures, data subsets, and implementation month.These analyses include high-level and fine-grained activity measures and alternative subsets of the study data.

G.3 Onboarding call and Cursor Training

The onboarding process prepared developers to use Cursor with their own codebases, collect study data, and discuss their experiences, while screen recordings were labeled with standardized activity rules.

  • Onboarding: All developers completed a 30-minute introduction and onboarding call after setting up Loom, Cursor, and study-data procedures.The call also provided an opportunity to answer questions about the data developers would collect.
  • Cursor training: Training ensured developers could create Cursor agent instances, provide file context, prompt edits, accept changes, and revert them.These actions were practiced on developers’ own codebases.
  • Study support: Periodic check-ins supported progress and captured developers’ experiences using AI during the experiment.Most developers had between 1–4 check-in calls.
  • Exit interviews: Exit interviews covered task selection, effort, code quality, scope changes, future AI use, skill development, and perceived effects on completion time.The interviews were unstructured and conducted 1 day to approximately 6 weeks after the final issue, depending on scheduling.
  • Data exclusions: Three developers’ data were excluded because two repositories paused development and one developer cheated extensively in the initial issues.All three developers were compensated fully for their participation.
Loading 2507.09089v2…