Source-linked AI summary
"TODO: Fix the Mess Gemini Created": Towards Understanding GenAI-Induced Self-Admitted Technical Debt
Abdullah Al Mujahid, Mia Mohammad Imran
TL;DR
The study addresses limited evidence about how generative AI contributes to self-admitted technical debt in software development. It analyzes AI-referencing code comments and identifies recurring patterns in which developers express uncertainty about AI-generated code. The authors propose GIST as a conceptual lens, while limiting quantitative generalization because the sample is small and drawn from public Python and JavaScript repositories.
Problem
Little is known about how developers recognize and document technical debt in generative-AI-influenced code or whether AI reshapes the kinds of debt they self-admit.
Method
The study analyzes source-code comments from public Python and JavaScript GitHub repositories that explicitly reference LLM use and technical-debt markers.
Results
AI-assisted development is associated with fewer design-related issues and more requirement and testing debts; developers express uncertainty about completeness and correctness in 34/81 cases.
Takeaways & Limitations
GIST provides a conceptual lens for recurring self-admitted debt in which developers adopt AI-generated code while expressing uncertainty about its behavior or rationale.
Takeaways & Limitations
The 81 instances come from public Python and JavaScript repositories, constraining quantitative generalization to proprietary projects, other languages, and future practices.
Abstract
from arXiv · showhide
As large language models (LLMs) such as ChatGPT, Copilot, Claude, and Gemini become integrated into software development workflows, developers increasingly leave traces of AI involvement in their code comments. Among these, some comments explicitly acknowledge both the use of generative AI and the presence of technical shortcomings. Analyzing 6,540 LLM-referencing code comments from public Python and JavaScript-based GitHub repositories (November 2022-July 2025), we identified 81 that also self-admit technical debt(SATD). Developers most often describe postponed testing, incomplete adaptation, and limited understanding of AI-generated code, suggesting that AI assistance affects both when and why technical debt emerges. We term GenAI-Induced Self-admitted Technical debt (GIST) as a proposed conceptual lens to describe recurring cases where developers incorporate AI-generated code while explicitly expressing uncertainty about its behavior or correctness.
1 Introduction
Generative AI introduces uncertainty and deferred work into software development, while developers also use AI assistance to improve existing implementations. This study examines how those patterns appear in self-admitted technical-debt comments and proposes GIST as a lens for recurring AI-related debt.
- The study analyzes comments explicitly referencing both LLM use and technical-debt markers to investigate how developers identify shortcomings in AI-generated code.
- AI-assisted code can trigger uncertainty about completeness and correctness, prompting later verification or revision in 34/81 cases.
- Developers attribute errors, redundant logic, or unstable behavior to AI-assisted code in 22/81 cases.
- AI-assisted development shows fewer design-related issues but higher prevalence of requirement and testing debts.
- The study proposes GenAI-Induced Self-admitted Technical debt (GIST) for debt arising when developers adopt AI-generated code without full understanding or confidence.
2 Related Work
Prior work established SATD taxonomies, detection methods, and evidence about AI-assisted development, but left unclear how AI involvement contributes to technical-debt formation in practice. This study addresses that gap through comments explicitly mentioning both AI involvement and SATD.
- Prior SATD research developed taxonomies, manual characterizations, and NLP-based detection methods for identifying technical debt.
- Studies of generative AI report productivity gains alongside challenges involving reliability, correctness, explainability, trust, developer collaboration, and code quality.
- Existing work has also examined AI for identifying or managing technical debt.
- Little has been explored about how AI use itself contributes to technical-debt formation, motivating analysis of comments that mention both AI involvement and SATD.
3 Methodology
The study mined public GitHub comments using combined AI/LLM and SATD keyword filters, then extracted and deduplicated matching comments. Their intersection produced 96 comments, with 1.47% meeting the stated criteria.
- The collection covered public Python and JavaScript repositories from November 2022 through July 2025.
- Researchers issued 196 structured queries combining seven AI terms, six generative verbs, and four connector terms through the GitHub Code Search API.
- AST parsing extracted matched comments from 37,234 files, after which duplicate comments were removed.
- The second filter searched case-insensitively for TODO, FIXME, HACK, and XXX indicators of self-admitted technical debt.
- 96 unique comments intersected both filters, and 1.47% of collected comments met the criteria, compared with 1.86% reported in prior SATD research.
3.2 SATD Types
The study adopts a five-category SATD taxonomy and manually annotates comments against it. After removing false positives, the final dataset contains 81 annotated comments with high inter-annotator agreement.
- The adopted taxonomy contains Design, Defect, Documentation, Requirement, and Test Debt categories.
- Design Debt covers design flaws such as misplaced logic, missing abstractions, overly long methods, and temporary workarounds.
- Requirement Debt marks incomplete code or incomplete units such as classes, functions, or methods, while Test Debt marks needed test implementation or improvement.
- Two annotators independently classified 96 comments, achieving Cohen’s κ=0.896 before resolving disagreements through discussion.
- 15 false positives were removed because 10 comments mentioned AI without actual use and five lacked technical debt, yielding 81 annotated comments.
3.3 Identifying the Role of AI
The authors manually open-coded all 81 LLM-referenced SATD comments to inductively identify how developers attribute AI's contribution to technical debt.
- The two authors collaboratively developed AI-role categories through iterative review, comparison, and refinement rather than using a predefined taxonomy.The coding process was open and inductive because no prior taxonomy described AI's contribution to technical debt.
- AI roles: Developers described AI-generated code as directly introducing problems such as incorrect logic, incomplete implementation, redundant code, or temporary fixes.
- The annotation instructions for the open-coding process are available in the replication package.
4 RQ1: What types of self-admitted technical debt emerge in developer comments that acknowledge both LLM involvement and technical debt?
Among AI-referencing SATD comments, design issues were most common, while requirement, testing, defect, and documentation debts captured incomplete adaptation, deferred validation, and repairs.
- 33/81 comments identified Design Debt as the most common category, involving misplaced code, poor implementation, or temporary workarounds.Examples include Copilot-generated code needing refactoring and an AI-generated component requiring project-specific modification.
- 17/81 comments identified Requirement Debt, typically reflecting unfinished AI-generated functionality or deferred integration work.One example required adding parameters to an AI-generated check.
- 17/81 cases involved Test Debt, including deferred testing or validation of integrated AI-assisted code and incomplete AI-generated test scaffolds.
- Compared with prior SATD distributions, design-related issues were proportionally lower, whereas requirement and testing debts were higher.The reported comparison was 40.74% versus 71.84% for design, 20.98% versus 14.24% for requirement, and 20.98% versus 2.09% for testing-related concerns, but the authors describe the observations as exploratory given the dataset size.
5 RQ2: How do developers attribute the role of AI in generative-AI induced SATDs?
Developers attributed multiple roles to AI in SATD comments: most often Catalyst, sometimes Source, often Mitigator, and occasionally neutral. These roles varied across debt types.
- 34/81 cases (41.98%) classified AI as a Catalyst that surfaced uncertainty and prompted deferred testing or validation.Developers treated generated code as potentially incomplete or unreliable despite its immediate use.
- 22/81 cases (27.2%) classified AI as a Source of debt by introducing errors, redundant logic, or unstable behavior requiring later correction.
- 19/81 cases classified AI as a Mitigator that helped reduce existing debt through refactoring, test scaffolds, or design suggestions.
- Role distributions: Source roles were most often associated with Design Debt (9 instances), followed by Requirement Debt and Test Debt (6 instances each).
- Role distributions: Catalyst roles were primarily linked to Test Debt (15 instances) and Design Debt (13 instances), while Mitigator roles most often involved Requirement Debt (9) and Design Debt (8).
- Discussion: Overall, developers treated AI-generated code as provisional material in a human-in-the-loop workflow that retained human responsibility for quality assurance.
6 GenAI-Induced Self-admitted Technical debt (GIST)
The paper proposes GIST for recurring cases where developers integrate AI-generated code despite uncertainty about its correctness or internal logic. This uncertainty is deferred into future maintenance and validation work.
- GIST describes AI-generated code incorporated despite developers' uncertainty about correctness and limited understanding of its internal logic.
- Comments such as uncertainty about a Copilot-generated regex or whether GPT-generated code works exemplify the pattern.
- The observed pattern is consistent with automation bias research concerning reliance on automated suggestions despite uncertainty about correctness or rationale.The study specifically observes this uncertainty being documented as self-admitted technical debt in source-code comments.
- GIST treats unresolved uncertainty as a latent burden for future development and maintenance.
- One recurring dimension is knowledge deficit and deferred quality assurance: partial understanding of AI-generated logic can postpone validation or testing.
7 Implications and Reflections
AI assistance appears to reshape technical debt by shifting it toward requirement completion and testing, while uncertainty about AI-generated code introduces a distinct maintenance concern. The paper proposes GIST to describe these recurring patterns and suggests clearer oversight, transparency, and accountability.
- Implications and Reflections: Design Debt is less dominant, while Requirement and Test Debts become more common in AI-assisted development.The changing distribution suggests debt shifts toward later development stages, especially requirement completion and validation.
- Implications and Reflections: AI-related debt may arise unintentionally from uncertainty about the behavior or suitability of AI-generated code rather than deliberate shortcuts.Developers may adopt code without fully understanding its rationale, creating future comprehension and accountability challenges.
- Implications and Reflections: Limited understanding of AI-generated code can increase future effort to modify, validate, or extend functioning code.The paper frames this concern as a maintainability risk associated with fragmented comprehension and accountability.
- Implications and Reflections: Explainable and responsible AI practices may mitigate these risks through transparency, documented rationales, and clearer accountability.These practices are presented as ways to help developers retain insight into AI-generated code and recognize possible maintenance effort.
- Implications and Reflections: AI roles as Source, Catalyst, and Mitigator suggest the need for SDLC oversight spanning validation, testing, documentation, and provenance tracking.Differentiating these roles may help teams balance AI’s supportive functions with incomplete understanding and uncertainty.
8 Threats to Validity
The study’s validity is constrained by keyword-based case detection, possible false positives and negatives, and subjectivity in manual interpretation. Its 81 cases from public Python and JavaScript repositories also limit quantitative generalization beyond the sampled settings and period.
- Threats to Validity: Keyword-based detection may produce false positives and false negatives because alternative AI references and non-debt TODO or FIXME comments can be missed or included.Manual annotation filtered irrelevant cases, but interpretation remained partly subjective.
- Threats to Validity: The analysis covers 81 AI-related SATD instances from public Python and JavaScript repositories between November 2022 and July 2025.This focused sample supports qualitative analysis but constrains quantitative generalization.
- Threats to Validity: Findings may not generalize to proprietary projects, other programming languages, or future AI-assisted development practices.Generative AI tools and developer workflows continue to evolve, limiting the scope of the conclusions.
9 Conclusion and Future Work
The study empirically examines AI-referencing self-admitted technical debt and proposes GIST as a conceptual lens for recurring uncertainty around AI-generated code. It presents an initial foundation for broader empirical and longitudinal research across languages, ecosystems, and project lifecycles.
- Conclusion and Future Work: The study examines 81 LLM-related code comments to characterize how generative AI is associated with shifts in self-admitted technical debt.Design-related issues appear less frequently, while requirement and testing debts are more common, reflecting deferred completion and validation.
- Conclusion and Future Work: GIST describes recurring self-admitted technical debt where developers explicitly express uncertainty about AI-generated code’s behavior or rationale.The concept is proposed for further examination, refinement, and empirical validation across broader contexts.
- Conclusion and Future Work: Future work should extend analysis beyond Python and JavaScript to additional languages and ecosystems.This would assess whether the observed patterns hold in other development contexts.
- Conclusion and Future Work: Longitudinal repository studies could examine whether AI-related debts evolve, are resolved, or accumulate as systems mature.Automated analysis may further illuminate these trajectories over time.
- Conclusion and Future Work: Future research may investigate practices that improve the transparency and verifiability of AI-generated code.Such practices could support uncertainty management and long-term maintainability in AI-assisted development.