Source-linked AI summary
AI Writes Code, Humans Pay the Debt. An Empirical Study on the Sustainability and Evolution of Agent-Generated Code
Antonino Coppola, Matteo Esposito, Rick Kazman, Valentina Lenarduzzi
TL;DR
The paper addresses the unclear long-term effects of AI coding agents on software quality and technical debt. It proposes a large-scale repository-mining study comparing counterfactual agent fixes with human implementations across localization, static-analysis measures, and software releases. The study is designed to identify differences among agents, human and agent code, and their evolution over time, while recognizing important measurement and validity boundaries.
Problem
The long-term effects of coding agents on software quality, maintainability, technical debt, and issue localization remain insufficiently understood despite short-term productivity benefits.
Method
The study reconstructs historical issue-fixing contexts, generates agent-based fixes, and compares them with human implementations using SQuaD data, static analysis, multiple agents, and release-level trajectories.
Results
The registered report is expected to provide empirical evidence on localization accuracy, technical-debt differences, agent variability, and divergence between human- and agent-driven software evolution.
Takeaways & Limitations
The study aims to support a more comprehensive understanding of agent-based software maintenance by examining strengths and limitations across entities, quality dimensions, and time.
Takeaways & Limitations
Technical-debt and maintainability measures are proxies that cannot fully capture developers’ perceived debt, while issue-to-commit linkage may miss fixes because commit practices vary.
Abstract
from arXiv · showhide
Context. The increasing adoption of Generative AI coding agents in software engineering is transforming how developers implement and maintain code. While these systems provide short-term productivity benefits, their long-term impact on software quality and technical debt remains unclear. Aim. We aim to investigate how agent-generated code affects software quality, focusing on issue localization accuracy, the introduction of technical debt, and its evolution over time. Method. We will conduct a large-scale mining software repositories study using the SQuaD dataset, employing a candidate set of 628k issue tickets. We will generate agent-based implementations for these issues, and compare them with the actual commits done by developers using static analysis metrics and tools. We will analyze differences at the commit level and across releases, and we will consider multiple LLM-based Agents selected through a systematic benchmarking strategy. Expected Results. We expect to provide empirical evidence on the trade-offs introduced by agent-based development, including differences in localization accuracy, variations in technical debt introduction, and potential divergence in long-term evolution. We expect the results to highlight variability across LLMs, to enrich our understanding of software evolution with Agents, and to inform more responsible adoption of Agents in software development.
1 Introduction
Generative AI coding agents offer short-term productivity benefits, but their effects on software quality, maintainability, and technical debt remain insufficiently understood. This study examines issue localization, debt introduction, and longer-term evolution by comparing agent-generated and human-written implementations.
- Coding agents can generate syntactically correct and functionally plausible code for bug fixing, feature implementation, and refactoring.
- Technical debt captures the trade-off between short-term gains and long-term quality, including suboptimal design decisions and maintainability issues.
- The study addresses whether coding agents introduce new technical debt or amplify existing debt, alongside whether they can reliably identify maintainability issues.
- AI-generated code may be more verbose, less consistent, or misaligned with project conventions, potentially degrading maintainability over time.
- The planned study reconstructs historical issue-fixing contexts, generates agent-based fixes, and compares them with human-written solutions using static analysis.
- Its contributions cover localization accuracy, software-quality and debt differences between agent and human code, and how those differences evolve over time.
2 Empirical Study Design
The empirical design combines repository mining, counterfactual agent-generated fixes, entity-level localization analysis, static-analysis metrics, and release-level comparisons against human implementations. It uses historical issue-fixing data and explicit statistical procedures while acknowledging limitations in linkage and measurement.
- Issue localization: The study evaluates agent localization with accuracy, precision, recall, F1-score, and MCC by comparing agent-touched entities with human reference fixes.
- Issue localization: Localization is tested at package, class, and method granularity because finer-grained entities require increasingly detailed contextual understanding.
- Technical debt and maintainability: The design asks whether agent-generated commits introduce more technical debt than human commits and compares debt and maintainability profiles across agents and implementations.
- Technical debt and maintainability: Technical debt is operationalized with static-analysis measures including maintainability, modularity, code smells, code health, cyclomatic complexity, duplication, and rule violations.
- Software evolution: Release-level analysis compares human-driven and agent-driven trajectories to test whether technical debt accumulates faster across software releases.
- Data collection: The dataset draws on SQuaD, which covers 450 mature open-source projects, 63,586 releases, more than 700 metrics, and over 628k issues.
- Data collection: Data collection extracts closed issues linked to resolving commits, restricts commits to before January 1st, 2021, and reconstructs each repository’s pre-implementation state.
- Data analysis: Statistical analysis uses Anderson-Darling tests, α = 0.05, Bonferroni-adjusted post-hoc thresholds, and Benjamini–Hochberg sensitivity analyses.
3 Threats to Validity
The study identifies limitations in its measurement proxies, validity assumptions, generalizability, and absence of conclusion-validity assessment. These constraints bound how its planned findings should be interpreted.
- Construct Validity: Static analysis metrics approximate technical debt and maintainability rather than directly measuring developers’ perceived debt.The study mitigates this construct-validity limitation by using multiple complementary metrics and focusing on consistent trends.
- Construct Validity: Fixing OpenCode while varying only underlying LLMs limits conclusions to differences among LLMs within one orchestration environment.Results may not generalize to alternative coding-assistant systems with different prompting strategies or tool integrations.
- Internal Validity: Issue-linking noise, prompt and model variability, confounding factors, and additive trajectory assumptions may affect comparisons between human- and agent-generated code.Differences may reflect issue complexity or developer expertise, while simulated trajectories may omit interactions, path dependency, and long-term architectural effects.
- Internal Validity: The proposed software-evolution trajectories are first-order approximations rather than causal reconstructions.They inject per-issue metric differences into release-level data under additive and independent-change assumptions.
- External Validity: Open-source projects and pre-2021 data may limit generalization to industrial settings and current development practices.The project sample is diversified to broaden scope, while the date restriction avoids contamination from AI-generated code.
- Conclusion Validity: Because this is a Stage 1 Registered Report, the study does not report threats to conclusion validity.
4 Related Work
Related work shows that AI-generated code differs from human-written code in defects, vulnerabilities, complexity, style, readability, and structure. These findings motivate evaluating such code beyond functional correctness and short-term productivity.
- Prior studies report differences between human-written and AI-generated code in defects, vulnerabilities, and complexity.
- Research identifies coding-style inconsistencies and reduced readability or comprehension as concerns for AI-generated code.An industrial case study also found readability to be a critical concern in real-world settings.
- Earlier work reports structural differences between AI-generated and human-written code, suggesting downstream implications for maintenance activities.
- The state of the art indicates that AI-generated code should not be evaluated solely through functional correctness or short-term outcomes.
5 Conclusion
The paper proposes a controlled, reproducible study of agent-generated code across projects, metrics, software entities, quality dimensions, and time. It aims to clarify trade-offs in issue localization, technical-debt introduction, and long-term evolution, while informing more deliberate adoption.
- The study compares human-written and agent-generated implementations across multiple projects and metrics to examine effects on software quality and technical debt.
- Its evaluation focuses on issue localization, technical-debt introduction, and how quality differences evolve across software entities, dimensions, and time.
- The proposed framework combines counterfactual trajectories, multi-metric analysis, and sensitivity analysis across different LLM configurations.
- The authors expect the findings to inform researchers and practitioners about trade-offs in adopting LLM-based agents.
- The authors used ChatGPT only for textual-clarity suggestions; the authors created the research design, analysis, interpretations, and manuscript.