PaperlayerView exact revision on arXiv

Source-linked AI summary

AI-Assisted Programming Decreases the Productivity of Experienced Developers by Increasing the Technical Debt and Maintenance Burden

Feiyang Xu, Poonacha K. Medappa, Murat M. Tunc, Martijn Vroegindeweij, Jan C. Fransoo

arXiv:2510.10165v3econ.GNcs.CYq-fin.EC

TL;DR

Prior work has documented AI coding productivity gains, but evidence on code quality and maintenance challenges remains limited. This study analyzes OSS activity around GitHub Copilot’s introduction. It reports higher development activity alongside more rework and maintenance, with core contributors reviewing more and committing less.

  • Problem

    Prior work documents productivity gains from AI-assisted coding but provides limited insight into how those gains translate into technical debt and maintenance challenges.

    Source: §1 · p. 1, §7.1 · p. 22

  • Method

    The study analyzes GitHub OSS project and contributor activity around Copilot adoption, using PR rework and reviews to examine technical debt and maintenance.

    Source: §1 · p. 4, §6.2 · p. 20

  • Results

    Copilot adoption coincided with increased development activity and PR rework, while core contributors’ commits fell 19% and PR reviews rose 6.5%.

    Source: §1 · p. 4, §6.2 · p. 20

  • Takeaways & Limitations

    The findings indicate a trade-off in which broader peripheral-contributor activity accompanies increased review and maintenance responsibilities for core contributors.

    Source: §6.2 · p. 20, §7.1 · p. 22

    The authors raise concern that rapid AI solutions may weaken peripheral contributors’ engagement with programming principles and best practices.

    Source: §7.2 · p. 24

Abstract

from arXiv · show

GenAI solutions like GitHub Copilot have been shown to increase the productivity of software developers. Yet prior work remains unclear on the quality of code produced and the challenges of maintaining it in software projects. If quality declines as volume grows, technical debt accumulates as experienced developers face increased workloads reviewing and reworking code from less-experienced contributors. We analyze developer activity in Open Source Software (OSS) projects following the introduction of GitHub Copilot. We find that productivity indeed increases. However, the increase in productivity is primarily driven by less-experienced (peripheral) developers. We also find that code written after the adoption of AI requires more rework to satisfy repository standards, indicating a potential increase in technical debt. Importantly, the added rework burden falls on the more experienced (core) developers, who review 6.5% more code after Copilot's introduction, but show a 19% drop in their original code productivity. More broadly, this finding raises caution that productivity gains of AI may mask the growing burden of maintenance on a shrinking pool of experts, together with increased technical debt for the projects. The results highlight a fundamental tension in AI-assisted software development between short-term productivity gains and long-term system sustainability.

1. Introduction

The section asks whether Copilot’s productivity gains in OSS come with quality and maintenance costs, especially when core contributors must review and rework contributions. Using a difference-in-differences design around Copilot’s technical preview, the study reports greater development activity alongside more rework and a shift of effort toward maintenance.

1. Introduction

  • The authors frame the concern as a tension between broader participation enabled by lower coding barriers and added maintenance obligations that may divert expert attention from productive development.OSS maintenance includes reviewing contributions, suggesting corrections, and ensuring project standards; these tasks are generally assigned to contributors with expertise and community trust.Source: §1 · pp. 2–3
  • The study measures technical debt through PR rework and maintenance effort through PR reviews, capturing revisions before integration and effort evaluating and correcting contributions.It examines changes after Copilot’s introduction using a difference-in-differences design.Source: §1 · pp. 3–4
  • 2.4% more code revisions accompanied Copilot adoption, alongside project-level increases in lines of code, commits, and pull requests.The authors describe the revisions as indicating a possible decline in the quality of initially submitted code and accumulation of technical debt.Source: §1 · p. 4
  • Peripheral contributors, especially those with the lowest prior contributions, increased commits by 43.5% and submitted 17.7% more pull requests.Core contributors reduced commits by 19% while increasing PR reviews by 6.5%.Source: §1 · p. 4

2. Related Literature and Conceptual Development

Prior work documents productivity gains from AI-assisted coding, while offering limited insight into resulting maintenance and technical-debt challenges. The section frames PR rework and review as process measures suited to examining these issues in OSS, where open contribution and distributed maintenance make quality coordination distinctive.

2.1. AI Assisted Code Development in OSS Projects

  • Prior studies report increased OSS contributions after Copilot adoption, alongside shifts toward routine innovation and greater coordination time for integration.The section identifies maintenance implications as less well understood than productivity effects.Source: §2.1 · p. 5

2.2. Conceptualization of Technical Debt

  • Technical debt is framed as suboptimal code, design, or documentation decisions in the PR process that increase PR rework and PR review effort.The section describes debt as an intertemporal trade-off between short-term development speed and long-term maintainability.Source: §2.2 · p. 6

2.3. Technical Debt in OSS projects

  • OSS’s open, distributed development blurs the boundary between development and maintenance, making PR review a central coordination mechanism for quality and technical debt.The workflow allows contributors to submit changes through PRs, which core contributors review and refine before integration into the main branch.Source: §2.3 · p. 9
  • PR rework captures realized remediation effort during integration, while observing PR review and rework captures debt management at code entry before integration.The section notes that this measurement could also apply to other CI/CD software projects beyond OSS.Source: §2.3 · pp. 9–10

3. Data and Methods

The study uses Copilot’s June 2021 technical preview and its language endorsements to compare activity in Microsoft-owned OSS repositories and contributors. It assembles monthly GitHub activity data for treatment and control groups across a 24-month observation window.

3.1. Research Design

  • The study exploits the technical preview’s endorsement of Python, JavaScript, Ruby, TypeScript, and Go, comparing them with other frequently used, non-endorsed languages.The observation period spans 12 months before and 12 months after the preview, from July 2020 to July 2022.Source: §3.1 · p. 11

3.2. Data Collection

  • The project-level dataset contains 2,755 repositories: 1,660 using Copilot-endorsed primary languages and 1,095 using non-endorsed languages.The study also focuses on Microsoft-owned repositories and contributors active during the observation period because individual Copilot usage cannot be identified.Source: §3.1 · p. 11, §3.2 · p. 12
  • The individual-level dataset contains 1,699 contributors, with 1,186 in the Copilot-supported treatment group and 513 in the control group.GitHub API data collection covered PRs, commits, reviews, authors, and repository and individual metadata.Source: §3.2 · pp. 11–12

3.3. Variables

The section operationalizes technical debt through PR rework and maintenance through review activity, then distinguishes core from peripheral contributors using pretreatment commits. Descriptive statistics cover repository- and contributor-level measures.

3.3.4. Descriptive Statistics

  • Descriptive statistics cover repository-month and contributor-month observations, with tables for repository-level and individual-level analysis.The listed measures include lines of code, commits, pull requests, rework, reviews, and reviewed repositories.Source: §3.3 · p. 12, §3.3.4 · p. 14, Table 3 · p. 14

3.3.1. Dependent Variable: PR Rework

  • PR rework measures commit amendments required for a pull request to satisfy repository standards, serving as a proxy for technical debt.It captures additional rework associated with initially nonconforming code.Source: §3.3.1 · p. 12
  • PR rework counts commits added after initial submission, as contributors revise code through review until it is merged or rejected.The measure also proxies maintenance effort associated with submitted contributions.Source: §3.3.1 · p. 12

3.3.2. Dependent Variable: PR Review and PR Reviewed Repos

  • Maintenance is measured by PR reviews and distinct repositories reviewed, capturing review intensity and the breadth of contributors’ maintenance responsibilities.PR reviews count formally reviewed pull requests; repository breadth counts distinct repositories with at least one review.Source: §3.3.2 · pp. 13–14

3.3.3. Moderation Variable: Core Contributors

  • Core contributors are the pretreatment top 25% by commit activity; they perform most project development, while peripheral contributors are the remaining 75%.Figure 3 describes contribution distributions for core and peripheral contributors.Source: §3.3.3 · p. 14, Figure 3 · p. 14

3.4. Estimation Models

The study estimates Copilot’s effects with difference-in-differences comparisons across treated and control repositories or contributors over time. An event study evaluates the parallel-trends assumption and traces how rework effects unfold.

3.4.1. Main Statistical Analysis

  • The difference-in-differences models compare changes in Copilot-supported-language groups with control groups, using repository or individual and monthly fixed effects.The individual-level model includes an interaction between Copilot treatment and core-contributor status.Source: §3.4.1 · p. 15

3.4.2. Parallel Trends:

  • An event study tests whether pre-treatment PR rework trends are parallel and characterizes changes over time relative to the month before treatment.The omitted baseline is τ = −1; fixed effects account for repository differences and month-year shocks.Source: §3.4.2 · pp. 15–16

4. Main Results

Copilot adoption coincided with increased project-level development activity and PR rework. Core contributors shifted toward maintenance while reducing development activity.

4.1. Project Level Analysis - Technical Debt

  • 17.7% more code added, 4.1% more commits, and 4.3% more PRs followed Copilot introduction in treatment repositories.The analysis measured code added, commits, and PRs as development activity and PR rework as technical debt.Source: §4.1 · p. 16, Table 5 · p. 16
  • 2.4% more PR rework occurred in treatment repositories while holding PR volume constant, suggesting greater review demand per unit of development.The authors link the rework to contributions being less likely to meet repository standards and to technical-debt accumulation.Source: §4.1 · p. 16
  • Post-treatment PR-rework coefficients trend upward over time, while pre-treatment estimates are statistically insignificant.The authors interpret the upward pattern as growing rework in repositories supporting AI coding partners.Source: §4.1 · p. 17, Figure 4 · p. 17

4.2. Individual Level Analysis - Maintenance Efforts

  • Core contributors submitted 42.9% fewer commits than other contributors and reviewed more PRs across a broader set of repositories after Copilot’s launch.The authors suggest that the maintenance burden may come at the cost of experienced contributors’ productivity.Source: §4.2 · p. 17

5. Robustness Check

The paper tests whether its findings are robust to matching and omitted-variable sensitivity. The matched-sample results remain consistent, and the PR-rework estimate remains positive under the reported Oster sensitivity analysis.

5. Robustness Check

  • The authors report conducting robustness tests to assess whether findings depend on model specification or unobserved variables.The section includes coarsened exact matching and coefficient-stability sensitivity analysis.Source: §5 · p. 17, §5.2 · pp. 18–19

5.1. Matching Results

  • After matching 2,510 repositories on pretreatment development characteristics, the re-estimated results remain consistent with increased rework and maintenance activity.The matched sample included 1,486 treatment repositories and 1,024 control repositories.Source: §5.1 · p. 18, Table 7 · p. 18

5.2. Sensitivity Analysis

  • For PR rework, the Oster-adjusted coefficient remains positive and statistically meaningful for all delta values greater than 1.The authors state that unrealistically large omitted-variable selection would be required to reduce the estimated effect to zero.Source: §5.2 · p. 19, Table 9 · p. 19

6. Individual Level Heterogeneous Analyses

The subgroup analyses classify contributors by pretreatment commit activity and estimate effects separately across contributor groups. Copilot exposure is associated with increased activity among peripheral contributors and a shift by core contributors toward maintenance, alongside greater PR rework.

6. Individual Level Heterogeneous Analyses

  • Contributor groups are defined using pretreatment commit contributions, distinguishing core contributors who typically integrate and maintain code from peripheral contributors focused more on feature additions.The subgroup models are estimated separately to assess differences in Copilot effects across contributor types.Source: §6 · pp. 19–20

6.1. Sub Group Statistical Analysis

  • The subgroup DiD model uses Copilot exposure and subgroup membership to estimate individual development and maintenance outcomes over time.Subgroup membership is a moderator based on pretreatment contribution behavior; the model includes individual and monthly fixed effects.Source: §6.1 · p. 20

6.2. Sub Group Analysis Results

  • The authors associate increased PR rework with both shortcut-style AI-assisted contributions needing revision and experienced contributors shifting from original coding to review and correction.They describe the resulting combination as a decline in high-quality code creation alongside increased expert-maintenance demand.Source: §6.2 · p. 20
  • Core contributors’ commits fell 19% and PR reviews rose 6.5%, while peripheral contributors’ commits rose 43.5% and PR submissions rose 17.7%.The authors interpret this pattern as greater maintenance burden on core contributors and lower participation barriers for peripheral contributors.Source: §6.2 · p. 20, Figure 5 · p. 21
  • Core contributors also show a meaningful rise in the number of distinct repositories they review, indicating a broader oversight role.Figure 5 presents contribution-activity analyses by contributor subgroup.Source: §6.2 · p. 21

7. Discussion

The discussion reports higher development activity alongside increased rework and a redistribution of maintenance toward core contributors, framing these patterns as technical-debt and sustainability concerns.

7.1. Key Findings and Discussion

  • Copilot’s introduction coincides with higher repository- and individual-level activity, more revisions before integration, and increased review and maintenance responsibilities for core contributors.Core contributors review more PRs, contribute fewer commits, and extend maintenance responsibilities across more repositories.Source: §7.1 · p. 22
  • Before Copilot’s introduction, core contributors averaged 976 commits, 160 PRs, and 166 PR reviews annually; afterward, each is expected to review about 10 more PRs and contribute 164 fewer commits and 9 fewer PRs per year.The discussion also reports that more than one-third of contributors to the 10 most popular OSS projects made their first contribution after signing up for Copilot.Source: §7.1 · p. 23
  • The study frames AI-assisted programming as a redistribution of effort: less-experienced contributors increase output while experienced contributors face heavier review workloads and reduced development activity.It presents this redistribution as a consequence that prior work focused on productivity gains has given limited attention.Source: §7.1 · p. 22, §7.2 · p. 23

7.2. Contributions and Future Research

  • AI-enabled shortcuts may speed output while introducing code that is difficult to integrate, extend, or refactor, leaving maintainers to review, revise, or rewrite it to meet repository standards.The discussion characterizes this as a change in incoming code composition that can intensify technical-debt accumulation.Source: §7.2 · p. 23
  • The study operationalizes extensive PR rework as realized technical debt: extra modification, coordination, and revision needed to bring submitted code up to acceptable standards.This PR-level measure complements architectural metrics, defect accumulation, and long-run performance outcomes, providing a micro-level view of debt emerging and being managed through maintenance.Source: §7.2 · pp. 23–24
  • The authors raise a concern that rapid AI solutions may reduce peripheral contributors’ engagement with programming principles and best practices, leaving functional but brittle code and weakening learning through contribution.They suggest future work examine governance mechanisms and project types, and whether the documented dynamics extend beyond OSS.Source: §7.2 · p. 24
  • The authors describe OSS maintenance burdens as an early warning for similar risks in other knowledge-intensive domains where AI is promoted to boost productivity and innovation.They connect OSS sustainability to the resilience of the broader digital ecosystem because OSS components are increasingly embedded in enterprise systems and public infrastructure.Source: §7.2 · p. 24

Project Level Lead Lag Analysis

The section describes a lead-lag analysis of GitHub Copilot adoption using a 24-month window around its technical preview release.

Project Level Lead Lag Analysis

  • The analysis estimates monthly treatment effects relative to Copilot’s launch, using 12 months before and 12 months after the technical preview release.The authors describe this as a 24-month window centered on the release.Source: Project Level Lead Lag Analysis · p. 28
  • A table is identified as reporting regression results for PR rework.Source: Table 10 · p. 28

Individual Level Sub Group Analysis

The section describes subgroup analyses based on contributors’ pretreatment contribution levels and reports tables for four activity outcomes.

Individual Level Sub Group Analysis

Related papers

Loading 2510.10165v3…