PaperlayerView exact revision on arXiv

Source-linked AI summary

Debt Behind the AI Boom: A Large-Scale Empirical Study of AI-Generated Code in the Wild

Yue Liu, Ratnadira Widyasari, Yanjie Zhao, Ivana Clairine Irsan, Junkai Chen, David Lo

arXiv:2603.28592v2cs.SE

TL;DR

AI-generated code is increasingly used in production, but its real-world effects on software quality and maintenance remain unclear. The paper analyzes verified AI-authored commits with before-and-after static analysis and tracks introduced issues over time. It finds substantial issue introduction and persistence, with 22.7% of tracked issues surviving to the latest repository revision.

  • Problem

    Real-world evidence is limited about whether quality issues in AI-generated code are quickly fixed or persist as technical debt after entering production repositories.

  • Method

    The study analyzes verified AI-authored commits from multiple assistants using commit-level differential static analysis and longitudinal issue tracking.

  • Results

    22.7% of tracked AI-introduced issues survive at the latest repository revision, while code smells account for 89.3% of identified issues.

  • Takeaways & Limitations

    AI-assisted development can create long-term maintenance costs, supporting stronger quality assurance for AI-generated code.

  • Takeaways & Limitations

    The findings may not generalize beyond public GitHub repositories with at least 100 stars, Python, JavaScript, and TypeScript production files, and explicitly traceable AI-authored commits.

Abstract

from arXiv · show

AI coding assistants are now widely used in software development. Software developers increasingly integrate AI-generated code into their codebases to improve productivity. Prior studies have shown that AI-generated code may contain code quality issues under controlled settings. However, we still know little about the real-world impact of AI-generated code on software quality and maintenance after it is introduced into production repositories. In other words, it remains unclear whether such issues are quickly fixed or persist and accumulate over time as technical debt. In this paper, we conduct a large-scale empirical study on the technical debt introduced by AI coding assistants in the wild. To achieve that, we built a dataset of 302.6k verified AI-authored commits from 6,299 GitHub repositories, covering five widely used AI coding assistants. For each commit, we run static analysis before and after the change to precisely attribute which code smells, correctness issues, and security issues the AI introduced. We then track each introduced issue from the introducing commit to the latest repository revision to study its lifecycle. Our results show that we identified 484,366 distinct issues, and that code smells are by far the most common type, accounting for 89.3% of all issues. We also find that more than 15% of commits from every AI coding assistant introduce at least one issue, although the rates vary across tools. More importantly, 22.7% of tracked AI-introduced issues still survive at the latest version of the repository. These findings show that AI-generated code can introduce long-term maintenance costs into real software projects and highlight the need for stronger quality assurance in AI-assisted development.

I. INTRODUCTION

The paper studies AI-introduced technical debt in real-world repositories, addressing limited evidence about its attribution, types, and lifecycle. It builds a large dataset and combines commit-level differential analysis with issue tracking.

  • Motivation and contribution: Developers’ growing use of AI-generated code raises concerns about functional bugs, runtime errors, maintainability, and security.Prior studies also report that developers may accept AI-generated code without proper validation.
  • Method and scope: The dataset uses explicit Git metadata to identify AI-authored commits, providing reliable attribution while excluding unrecorded AI-assisted changes.This design is analogous in spirit to research on self-admitted technical debt.
  • Motivation and contribution: The study examines AI-introduced technical debt across five coding assistants and more than 6,000 GitHub repositories.The assistants are GitHub Copilot, Claude, Cursor, Gemini, and Devin.
  • Method and scope: For each AI-authored commit, differential analysis attributes code smells, correctness issues, and security issues to the individual change.The analysis compares source code immediately before and after each commit using static analysis.
  • Method and scope: The study tracks introduced issues from their originating commit to the latest repository revision to determine whether they persist or are resolved.The authors also publish the dataset and replication package.

II. BACKGROUND

AI coding assistants are rapidly moving into production development, but their generated code may introduce quality and security problems whose long-term persistence remains unclear. Real-world attribution is difficult because AI- and human-written changes are interleaved and AI use is not always recorded.

  • AI coding assistants: AI coding assistants are becoming deeply embedded in software workflows and can generate code at unprecedented speed and scale.In Anthropic’s claudes-c-compiler repository, Claude contributed 3,957 commits and nearly 500K lines within a few weeks.
  • Technical debt: Technical debt prioritizes short-term speed over long-term quality and can increase future maintenance costs.This study operationalizes debt as code smells, correctness issues, and security issues.
  • Technical debt: AI-generated code has been associated with functional bugs, runtime errors, maintainability issues, and security vulnerabilities.The paper notes that prior work found about 40% of AI-generated code in security-sensitive contexts contained critical vulnerabilities.
  • Motivating examples: A Copilot-authored commit introduced a shell=True subprocess call that increased command-injection risk when user input was involved.A later human developer removed shell=True in a security fix.
  • Motivating examples: A Copilot change in librealsense introduced an undefined constant, causing a runtime error that remained for over three weeks before a maintainer fixed it.The example illustrates that AI-introduced defects may persist after entering production code.
  • Research gap: Existing studies often examine a single tool, a small task set, or controlled settings, leaving long-term real-world implications insufficiently understood.The paper therefore focuses on whether issues are fixed, persist, or create maintenance challenges.

III. APPROACH

The approach collects AI-authored commits at scale, compares source versions before and after each commit to attribute introduced issues, and follows those issues through repository history. It combines repository filtering, static analysis, differential attribution, and lifecycle tracking.

  • Overview: The pipeline collects AI-authored commits from GitHub repositories, analyzes each change, and tracks issue and code lifecycles over time.These stages correspond to data collection, commit-level quality analysis, and issue-survival analysis.
  • Data collection: The dataset retains repositories with at least 100 stars and confirmed AI-authored commits, focusing downstream analysis on production Python, JavaScript, and TypeScript files.Repositories without source files in these languages are excluded.
  • Commit-level quality analysis: Each AI-authored commit is analyzed at its parent revision and after the commit is applied to identify issues introduced or fixed by the change.Tests, documentation, configuration, generated artifacts, and vendored dependencies are excluded from source-quality analysis.
  • Commit-level quality analysis: ESLint and Pylint detect code smells and correctness issues, while Semgrep detects security-related issues across languages.Each detected issue records its rule identifier, line number, detector, and message.
  • Differential attribution: Differential attribution matches issues across pre- and post-commit analyses, then classifies unmatched post-commit issues as introduced only when their lines fall within the commit’s changed lines.This avoids mistaking line-number shifts for newly introduced issues.
  • Lifecycle tracking: Issue survival is measured by checking whether each introduced issue remains at repository HEAD, following renamed files through history.The study also records whether affected files are modified again before HEAD.

IV. EXPERIMENTAL SETUP

The experimental setup is introduced as the section describing the study’s setup.

  • The section introduces the experimental setup for the study.

A. Dataset Summary

The study focuses on five AI coding assistants and analyzes 302.6K AI-attributed commits from 6,299 public GitHub repositories. It examines technical debt across assistants, languages, rules, and issue types.

  • Dataset overview: Figure 6 summarizes AI-authored commit growth over time and repository distribution by GitHub star count as of March 2026.
  • Dataset construction: 302.6K AI-attributed commits from 6,299 public GitHub repositories form the final analysis dataset.The dataset covers five assistants selected because each had more than 10,000 attributed commits.
  • Assistants studied: The study compares GitHub Copilot, Claude, Cursor, Gemini, and Devin.
  • Research scope: The research questions examine debt types, variation across assistants, and technical-debt lifecycle behavior.

C. Evaluation Metrics

The evaluation measures issue introduction, net impact, survival, and pipeline reliability. These metrics combine commit-level static analysis with manual validation of attribution and issue tracking.

  • Issue introduction: Issue introduction is measured by total issues, the percentage of commits introducing at least one issue, and average issues per commit.Results are broken down by issue type, programming language, rule, and AI coding assistant.
  • Debt lifecycle: Net impact compares the number of issues introduced with the number fixed by AI-authored commits.
  • Debt lifecycle: Survival rate measures the share of introduced issues still present at the latest repository revision.The study tracks issues to HEAD to determine whether they persist or are resolved.
  • Reliability: Manual inspection confirmed correct AI attribution for all 99 verifiable sampled commits, yielding 99.0% conservative attribution precision.One of 100 sampled commits could not be verified because its repository was unavailable.
  • Reliability: The issue-validity pipeline achieved 85.9% accuracy, 85.9% precision, 100.0% recall, and 92.4% F1.Survival classification achieved 84.8% accuracy, 86.7% precision, 81.2% recall, and 83.9% F1.

A. RQ1: Types and Patterns of AI-Introduced Debt

AI-generated code introduces technical debt primarily as code smells, alongside correctness and security issues. These issues span language-specific patterns and include concrete failures such as undefined variables and possible SQL injection.

  • Overview: 484,366 introduced issues were identified across 3,946 repositories and 27,677 commits.Code smells accounted for 89.3% of all introduced issues.
  • Correctness issues: 28,931 correctness issues occurred across 665 repositories and 1,650 commits, including 23,856 undefined-variable or undefined-reference cases.One example caused a runtime NameError because Devin passed an undefined cache variable.
  • Security issues: Security issues appeared in 1,643 repositories and 5,142 commits, including path traversal, unsafe format strings, non-literal regular expressions, and child-process execution.
  • Language patterns: Python issues emphasize exception handling and dynamic typing, while JavaScript/TypeScript issues emphasize scoping and variable declarations.Unused-code patterns occur in both languages.
  • Overview: AI-generated code introduces code smells (89.3%), correctness issues (6.0%), and security issues (4.7%).

B. RQ2: Comparison Across AI Coding Assistants

Technical-debt rates differ across AI coding assistants, but every studied tool exhibits the same broad pattern: code smells dominate the issues introduced.

  • Issue-bearing commits: 17.4% of GitHub Copilot commits and 29.1% of Gemini commits introduced at least one issue.More than 15% of commits from every studied assistant introduced at least one issue.
  • Issue rates: Claude had the highest average issue rate per commit at 1.95, while Devin had the lowest at 0.89.
  • Cross-tool pattern: Across all five assistants, code smells remained much more common than correctness and security issues.
  • Cross-tool pattern: The overall technical-debt pattern was consistent across the five tools despite differences in issue rates.

C. RQ3: Persistence of AI-Introduced Debt

AI-introduced issues can persist well beyond the commits that create them. Although AI commits fix slightly more code smells than they introduce, 22.7% of tracked issues remain at HEAD, including issues older than nine months.

  • Net impact: AI-authored commits fix more code smells than they introduce, producing a net reduction of 7,069 code smells.The counts are 439,817 code smells fixed versus 432,748 introduced.
  • Issue survival: Issue survival accumulates over time, with the cumulative number of surviving issues continuing to grow.The tracked issue lifecycle shows unresolved technical debt increasing across repository history.
  • Issue survival: 22.7% of tracked AI-introduced issues—105,364 of 464,900—still survive at HEAD.Surviving issues occur across all age cohorts.
  • Issue survival: 4,893 issues introduced more than nine months earlier still remain at HEAD.Cohort survival rates range from 19.4% for issues introduced 6–9 months ago to 28.2% for issues introduced 3–6 months ago.
  • Examples: Examples range from fixes within hours or one day to an undefined-variable bug fixed after 42 days and a timeout issue still unresolved.The examples illustrate substantially different repair times for AI-introduced issues.
  • Net impact: Overall, AI-authored commits fix slightly more code smells than they introduce but introduce more correctness and security issues than they fix.The paper reports this asymmetry alongside the 22.7% survival rate at HEAD.

A. Implications

The study finds that AI-assisted development introduces persistent technical debt across tools, not merely temporary low-quality code. These findings support stronger review, monitoring, and long-term code-health checks, while motivating research beyond short-term generation outcomes.

  • Implications: More than 15% of commits from each of the five studied AI assistants introduce at least one detectable issue.The issues persist regardless of repository size or popularity.
  • Implications: AI-assisted development creates persistent debt because introduced issues can remain in production systems after code is merged.The paper connects persistence with developers’ tendency to over-trust AI suggestions and accept problematic code.
  • Implications: AI co-authored commits fix a similar number of code smells as they introduce, but appear less effective at correcting correctness and security issues.The observed strength is local cleanup and repetitive maintenance, such as formatting, naming, and simple refactoring.
  • Implications: All five tools show a similar issue pattern, so switching assistants alone does not remove the technical-debt problem.The paper recommends static analysis, tests, security checks, and review beyond merge.
  • Future research: Prior work mainly studies short-term outcomes, whereas this study examines whether AI-introduced debt persists in production repositories over time.The paper argues that future research should examine persistence factors such as repository maturity, review intensity, and task type.
  • Research context: The study extends prior work by analyzing code-level debt across multiple tools and tracking its evolution after introduction.Related studies commonly focus on a single tool, narrow quality issues, or developer adoption rather than debt persistence.

VIII. THREATS TO VALIDITY

The study’s validity is constrained by its visible, attributable sample and static-analysis-based definition of technical debt. Its findings therefore concern selected public repositories, languages, and code-level issues rather than all AI-assisted work or all forms of software debt.

  • External validity: The dataset covers public GitHub repositories with at least 100 stars and production Python, JavaScript, and TypeScript files.The findings may not generalize to private repositories, smaller projects, or other programming languages.
  • External validity: Only AI-authored commits with explicit Git metadata traces are included, excluding AI-assisted contributions without visible attribution.The study therefore characterizes a visible and attributable subset rather than the full population of AI-assisted work.
  • External validity: The study does not compare AI-authored commits against a purely human-written baseline.AI-labeled commits may also contain mixed human edits, complicating a reliable human-only comparison.
  • Construct validity: Technical debt is operationalized mainly as code smells, correctness issues, and security issues detected by static-analysis tools.Architectural, design, documentation, and test-adequacy debt are outside the tools’ scope.
  • Construct validity: Static analysis can produce false positives and includes both active vulnerabilities and latent unsafe patterns without separating them quantitatively.Reported results should be interpreted as evidence about code-level technical debt, not the full spectrum of quality challenges.
  • Construct validity: Survival rates may slightly underestimate true persistence because an issue’s apparent resolution need not be a deliberate fix.This limitation affects interpretation of issue survival over repository history.
Loading 2603.28592v2…