PaperlayerView exact revision on arXiv

Source-linked AI summary

Empirical Study on the Characteristics and Evolution of AI-usage in GitHub Repositories: Evidence from Code Comments

Abdullah Al Mujahid, Preetha Chatterjee, Mia Mohammad Imran

arXiv:2606.06843v1cs.SE

TL;DR

Prior research has rarely shown how developers adapt AI-generated code in real projects. This paper analyzes AI-referencing GitHub comments, associated code, and subsequent commits, finding that AI-assisted code is commonly refined through refactoring, fixes, and structural adjustments while usage shifts toward conceptual support and enhancement.

  • Problem

    Prior studies largely use controlled or short-term evaluations, leaving how developers integrate and adapt AI-generated code within real projects comparatively underexplored.

  • Method

    The study combines qualitative coding of 500 comments and code blocks with large-scale LLM annotation, probabilistic aggregation, and longitudinal analysis of associated development artifacts.

  • Results

    AI is most frequently used for code implementation, while subsequent changes commonly involve refactoring, fixes, and structural adjustments; usage increasingly includes conceptual clarification and refinement.

  • Takeaways & Limitations

    AI-assisted development appears embedded in workflows where developers integrate, refine, and correct generated output rather than treating code generation as a finished step.

  • Takeaways & Limitations

    Keyword-based AI-comment detection may produce false positives, and comments or captured code blocks may not accurately represent the actual AI-assisted task.

Abstract

from arXiv · show

Developers increasingly use AI tools such as ChatGPT, Copilot, and Claude in everyday software workflows, but prior studies often evaluate LLM outputs in isolation rather than examining how developers adapt them in real projects. We analyze 35,361 GitHub code comments that explicitly reference AI use and their associated code blocks. We first open-code 500 unique comments and code blocks to derive a taxonomy of AI-assisted development activities, then annotate the full dataset using two LLM-based classifiers and aggregate predictions with Dawid-Skene expectation-maximization. We also analyze 12,996 subsequent commit messages to study how AI-assisted code evolves after introduction, and examine temporal trends from December 2022 to March 2026. Our results show that developers primarily use LLMs for code implementation, followed by code enhancement, debugging, documentation, and testing. Subsequent commits frequently involve refactoring and cleanup, feature integration and extension, and bug fixing, indicating sustained human oversight in adapting AI-assisted code. Over time, AI-referencing comments shift from direct code generation toward knowledge and conceptual support and code enhancement. These findings suggest that AI tools are becoming embedded not only as code-generation aids, but also as collaborative support mechanisms whose outputs are refined, extended, and corrected by developers over time.

1 Introduction

This study examines self-admitted AI use in real-world GitHub development through code comments, associated code blocks, and subsequent commits. It characterizes how developers integrate AI, modify AI-assisted code, and change their usage over time.

  • Research gap: Prior research primarily evaluates AI-assisted programming through controlled experiments and benchmarks rather than artifacts from real development workflows.The study addresses this gap by examining how developers employ, interpret, and modify AI-generated outputs in situ.
  • Research gap: Code comments provide fine-grained evidence of AI involvement because they are co-located with code and capture developers’ intent, limitations, uncertainty, and acknowledgments.Examples include comments stating that code was generated by ChatGPT, suggested by Copilot, or AI-generated and requiring review.
  • Study design: 35,361 GitHub code comments explicitly referencing AI, associated code blocks, and 12,778 first-change commits form the study’s dataset across 12,944 repositories from December 2022–March 2026.The analysis uses 500 manually coded samples to guide large-scale LLM-assisted labeling, with multiple LLM annotations consolidated through a Dawid-Skene framework.
  • Study findings: Refactoring & Cleanup and Feature Integration & Extension are dominant post-integration actions, while many commits also involve Bug Fixes & Corrective Changes.These actions are identified from first-change commit messages after AI-assisted code enters projects.
  • Study findings: Over time, code implementation remains dominant but declines relatively as code enhancement gains prominence, alongside increased use of AI for documentation and conceptual reasoning.The longitudinal analysis covers comments from December 2022 to March 2026 and tracks both task types and AI contribution types.

2 Methodology · 2.1 Data Collection

The study combines large-scale mining, qualitative coding, probabilistic annotation, and semantic clustering to examine AI-referenced code comments and their subsequent evolution. It collects Python and JavaScript examples from December 2022 to March 2026, yielding 35,278 comments with associated code blocks and 12,996 retrievable first-change commits.

  • 2 Methodology: The methodology integrates data mining, human coding, multi-model LLM annotation, probabilistic aggregation, and semantic clustering.The design uses a curated consolidated dataset for RQ1 and HDBSCAN clustering of later commits for RQ2.
  • 2 Methodology: The collection covers code comments explicitly referencing AI use from December 2022 through March 2026, with associated code blocks extracted from introductory commit patches.Later commits modifying the same blocks were subsequently examined.
  • 2.1 Data Collection: The study focuses on Python and JavaScript because they are described as widely used languages in the open-source ecosystem.Python is identified as GitHub’s most-used language in 2024, while JavaScript is described as a consistent leader in developer activity.
  • 2.1.1 Extraction of AI-Referenced Comments and Associated Code Blocks: The researchers queried GitHub’s Code Search API using templates that combine AI-tool keywords, generative action verbs, and attribution connectors.The phrase sets contain 16 AI-tool keywords, 6 action verbs, and 4 connector terms.
  • 2.1.1 Extraction of AI-Referenced Comments and Associated Code Blocks: 35,278 comments—24,882 Python and 10,396 JavaScript—were linked to code blocks across 26,490 source files and 12,944 repositories.The filtering retained matches inside actual comment spans, including inline comments, block comments, and docstrings.
  • 2.1.2 Retrieving First-Change Commits: 12,996 first-change commits were retrieved by tracking each extracted block through later file revisions and updating its line interval for insertions and deletions.For 22,282 code blocks, first-change commits could not be retrieved because of insufficient commit history.

2.2 Data Annotation and Validation · 2.3 Scaling Annotation with LLMs

The study manually annotated a statistically grounded sample, refined its taxonomy and agreement checks, then scaled labeling to the full corpus with two LLMs and Dawid–Skene aggregation. After removing false positives, the final corpus contained 27,581 valid AI-referenced comments and code blocks classified by task and AI contribution.

  • 2.2 Data Annotation and Validation: 500 comments and associated code blocks were stratified by language and independently annotated by two experienced programmers across Task Type and AI Contribution Type.The sample was designed for 95% confidence with a ±5% margin of error.
  • 2.2 Data Annotation and Validation: 173 false positives were removed over three annotation rounds, with additional samples drawn until sufficient valid AI-referenced comments were obtained.False positives included incidental appearances of AI-related terms rather than actual AI use.
  • 2.2 Data Annotation and Validation: The manual coding identified six Task Types and four AI Contribution Types, led by Code Implementation in both dimensions at 362/500 and 387/500, respectively.Other categories included enhancement, debugging, testing, documentation, knowledge support, artifact generation, and generic or indeterminate actions.
  • 2.2 Data Annotation and Validation: Inter-annotator agreement reached AC1 = 0.760 for Task type and AC1 = 0.657 for AI Contribution type, indicating substantial agreement under skewed class frequencies.Gwet’s AC1 was selected because it is less sensitive to prevalence and marginal-distribution effects.
  • 2.3 Scaling Annotation with LLMs: Two open-weight LLMs independently labeled the remaining corpus using structured prompts for six Task Types and four AI Contribution Types, with Dawid–Skene EM consolidating their outputs.The models were gemma-4:31b and nemotron-3-super:120b, run at temperature 0 to reduce model-specific bias and variance.
  • 2.3 Scaling Annotation with LLMs: Dawid–Skene EM estimated latent labels and annotator reliability, initialized from 400 gold-standard human annotations and applied to both annotation dimensions.The procedure produced posterior vectors, hard labels, and confidence margins, with convergence at ∆L < 10^-6 or 100 iterations.
  • 2.3 Scaling Annotation with LLMs: Held-out evaluation against 100 manually annotated instances yielded Task Type AC1 = 0.7005 and AI Contribution Type AC1 = 0.8128.The 400 remaining manual annotations formed the DS-EM gold set, while 100 were reserved for evaluation.
  • 2.3 Scaling Annotation with LLMs: 7,697 false positives were excluded, leaving 27,581 comments and associated code blocks distributed across six Task Types and four AI Contribution Types.The largest final categories were Code Implementation (18,149) and AI Contribution Implementation (16,137).

2.4 Topic Modeling and Semantic Grouping of First Change Commit Messages

The study filtered first-change commit messages and applied BERTopic with semantic grouping to identify higher-level developer action categories. This process retained 10,139 commits and resulted in eight developer action categories.

  • Commit Filtering: 10,139 first-change commits remained after removing false positives and 163 short messages from 12,996 collected commits.The filtering excluded 2,694 commits linked with false positives and messages shorter than four characters.
  • Topic Modeling: BERTopic embedded, reduced, clustered, and represented commit messages to identify patterns in developer actions.The pipeline used sentence embeddings, UMAP dimensionality reduction, HDBSCAN clustering, and cluster-level bag-of-words representations.
  • Semantic Grouping: Semantic grouping merged closely related topics based on keywords and representative commit messages, addressing unstable fine-grained distinctions in short messages.Authors inspected 6-10 keywords and 10 randomly selected commit messages from each topic before merging activities with the same underlying meaning.
  • Developer Action Categories: Eight developer action categories resulted, including Feature Integration & Extension, Refactoring & Cleanup, and Bug Fixes and Corrective Changes.The reported category totals include Feature Integration & Extension (130 topics, 2,769 commit messages), Refactoring & Cleanup (59 topics, 1,465 messages), and Bug Fixes and Corrective Changes (66 topics, 1,304 messages).

2.5 Longitudinal Analysis for Task Type and AI Contribution Type

The longitudinal analysis constructs corrected, normalized monthly series from December 2022 to March 2026 to isolate interpretable trends in Task Type and LLM Contribution Type. It summarizes each label’s temporal stability, variability, directionality, and trend significance.

  • Data preparation: Monthly annotation counts were normalized by each month’s total from December 2022 to March 2026, excluding Generic Mention and Indeterminate Actions.The series used each comment’s introductory commit timestamp and therefore represent proportional rather than absolute frequencies.
  • Data preparation: A three-month rolling mean smoothed short-term fluctuations, while an IQR-based anomaly cap clipped values outside [Q1 −1.5×IQR, Q3 +1.5×IQR].The correction was intended to reduce distortion from one-off category surges while preserving structural variation.
  • Descriptive statistics: For each label, the analysis computed mean, standard deviation, lag-1 autocorrelation (ρ1), monthly trend slope (β), and the p-value for β’s significance.These statistics were calculated on corrected normalized data to characterize stability, variability, directionality, and trend significance.

3 Results and Discussion

The study identifies six AI-assisted development activity categories, with generic or indeterminate mentions excluded from task-specific analysis. Across the full dataset, code implementation is the dominant AI-assisted activity, while developers also use AI for enhancement, debugging, testing, documentation, and quality assurance.

  • RQ1.A: In-situ Tasks: 500 manually annotated comments and code blocks yielded six AI-assisted development activity categories, including 97/500 generic or indeterminate cases.Generic or indeterminate cases represented 19.40% and were excluded from subsequent analysis.
  • RQ1.A: In-situ Tasks: Among 403 task-specific manually annotated cases, Code Implementation accounted for 362/403 (89.82%).Code Implementation refers to AI-generated functional code incorporated into production repositories.
  • RQ1.A: In-situ Tasks: AI-assisted activities also included Code Enhancement, Bug Identification & Fixing, Testing, and Documentation.Examples include refactoring or optimizing existing code, suggesting fixes and patches, generating test cases, and producing docstrings or comments.
  • RQ1.A: In-situ Tasks: In the full dataset, 2,388 of 27,581 instances (8.66%) were generic or indeterminate, while Code Implementation comprised 18,149 of 25,193 task-specific cases (72.04%).The full-dataset taxonomy was produced using the DS-EM framework after extending analysis beyond the manually annotated sample.

RQ1.B: (Forms of Assistance). How does AI assist developers in performing these tasks in practice?

AI assists developers primarily through direct implementation, while also providing conceptual guidance and generating supplementary artifacts. After AI-assisted code is introduced, developers commonly extend, refactor, test, document, and correct it, with usage increasingly broadening toward enhancement, testing, debugging, and conceptual support over time.

  • RQ1.B: (Forms of Assistance). How does AI assist developers in performing these tasks in practice?: Implementation dominates manual annotations at 387/452 (85.62%), followed by Knowledge & Concept Support at 50/452 (11.06%) and Artifact Generation at 15/452 (3.32%).The full dataset also contained 7,932 (28.79%) comments labeled Generic Mention and Indeterminate Actions.
  • RQ1.B: (Forms of Assistance). How does AI assist developers in performing these tasks in practice?: In Implementation, AI directly produces executable code or completes functional logic that becomes part of a repository.Examples include validation written by Copilot and a sorting function generated by ChatGPT.
  • RQ1.B: (Forms of Assistance). How does AI assist developers in performing these tasks in practice?: Knowledge & Concept Support appears in 14.76% of full-dataset comments and code blocks, covering design decisions, options, and implementation strategies.Among 50 manually annotated instances, 12 involved algorithm selection or data structure design.
  • RQ1.B: (Forms of Assistance). How does AI assist developers in performing these tasks in practice?: Artifact Generation occurs in 714 (3.63%) instances and produces supplementary artifacts such as documentation, test descriptions, or development-support materials.This category goes beyond code generation to include artifacts such as AI-generated lists.
  • 3.2 RQ2: How do developers subsequently adjust, refine, or extend AI-assisted code after its introduction into projects?: Feature Integration & Extension is the most frequent subsequent action, with 2,769 commits extending functionality and introducing new features.Examples include adding chat management, updating uploads and pagination, and broadening endpoint flexibility.
  • 3.2 RQ2: How do developers subsequently adjust, refine, or extend AI-assisted code after its introduction into projects?: Refactoring & Cleanup accounts for 1,465 commits, while Bug Fixes & Corrective Changes account for 1,304, indicating structural refinement and corrective maintenance after integration.Reported changes include removing redundant or unused code, eliminating dependencies, and correcting authorization or other bugs.
  • 3.2 RQ2: How do developers subsequently adjust, refine, or extend AI-assisted code after its introduction into projects?: Configuration, Dependency & Environment Management comprises 810 commits, while Documentation comprises 709 and Testing & Evaluation comprises 432.These actions include merge-conflict resolution, pull-request integration, README updates, and additional testing for coverage or user stories.
  • 3.3 RQ3: How has developers’ AI usage behavior evolved over time?: Implementation remains dominant but stable over time, while Code Enhancement, Testing, Bug Identification & Fixing, and Knowledge & Concept Support increase and Documentation decreases.Significant trends include Code Enhancement β = +0.00113, p = 0.049; Testing β = +0.00097, p = 0.004; Documentation β = −0.00151, p = 0.008; and Knowledge & Concept Support β = +0.00185, p = 0.025.

4 Discussion and Implications

AI-assisted development has implications for organizational memory, productivity measurement, and adoption dynamics. The findings indicate that AI shifts effort toward integration and refinement while supporting increasingly deliberate knowledge-oriented use.

  • Discussion and Implications: These findings clarify AI-assisted development’s practical implications for knowledge management, productivity measurement, and long-run adoption dynamics.The discussion connects empirical results to organizational and measurement concerns.
  • Knowledge Externalization and Organizational Memory: AI increasingly supports knowledge and concept work, helping developers bridge knowledge gaps by understanding APIs, clarifying designs, and reasoning about implementation strategies.Industry observations similarly describe conversational LLMs as tools for guidance and learning rather than ready-to-integrate artifacts.
  • Knowledge Externalization and Organizational Memory: Conversational reasoning and decision rationale remain invisible to later contributors unless teams explicitly preserve them in pull requests, issues, or design notes.Project history alone may therefore be insufficient to reconstruct decisions influenced by AI assistance.
  • The Overhead Cost of Code Integration: Frequent post-integration refactoring, testing, and debugging can make lines of code generated or time to first implementation misleading measures of productivity.AI-assisted code may redistribute effort from initial scaffolding toward integration and refinement.
  • Temporal Maturation of AI-Assisted Development Practices: AI adoption increasingly combines Knowledge & Concept Support and Code Enhancement with stable Code Implementation, indicating more differentiated and deliberate application over time.The longitudinal pattern is framed as movement from exploratory use toward more differentiated practice.

5 Related Work

Related work uses code comments and commit messages to infer developer intent and maintenance behavior, while examining how generative AI and coding assistants affect software development. Prior studies highlight benefits for productivity, code generation, and comprehension alongside reliability, maintainability, oversight, and long-term risks.

  • Code Comments and Commit Messages: Code comments and commit messages reveal developer intent, maintenance behavior, rationale, design trade-offs, and cognitive processes.Prior work also finds that developers value comments that are concise and contextually accurate.
  • AI Usage in Software Development: Studies of Copilot, ChatGPT, Cursor, and Claude examine their effects on software engineering practices and identify open problems.Research also analyzes practitioner use of ChatGPT across programming languages for comprehension, translation, and problem solving.
  • AI Usage in Software Development: Prior work identifies productivity, code generation, and comprehension benefits while raising concerns about reliability, maintainability, human oversight, and long-term risks.These findings concern the role of large language models in software engineering and real software development contexts.

6 Threats to Validity

The study faces construct, internal, and external validity threats arising from keyword-based AI-comment detection, assumptions about repository histories and file-level commits, and limited coverage of repositories and AI use. The authors mitigate some risks through manual validation, multiple annotators, metadata filtering, and scope limitations.

  • Construct validity: Keyword-based detection may produce false positives, including unrelated names such as “Claude.”The authors manually validated a random subset to assess precision, while multiple annotators and iterative guideline refinement reduced annotation bias.
  • Internal validity: Linking AI-referenced comments to subsequent commits assumes repository histories preserve temporal order and authorship.The study excluded commits with anomalous timestamps or inconsistent metadata; file-level commits may also contain unrelated changes.
  • External validity: The dataset covers public Python and JavaScript repositories and only self-admitted AI use, limiting generalizability to other ecosystems, closed-source projects, and unacknowledged use.The authors also note that AI-assisted development practices evolve rapidly, further constraining external validity.

7 Conclusion and Future Work

The study finds that generative AI is used most often for code implementation, while developers subsequently refactor, fix, and structurally adjust generated code. Future work will examine longer-term evolution and generalization beyond Python and JavaScript.

  • Conclusion: 35,361 AI-referenced code comments and associated code blocks were analyzed across GitHub repositories added between December 2022 and March 2026.The study also examined 12,996 linked first-change commits across 12,944 GitHub repositories.
  • Conclusion: Generative AI is used most frequently during code implementation, but initial integration is often followed by refactoring, fixes, and structural adjustments.These subsequent changes reflect developers’ continued role in aligning generated output with project-specific constraints.
  • Future Work: Future work will extend analysis beyond first-change commits to capture stabilization, refactoring, and repeated modification patterns over the long term.The planned analysis will also test whether observed integration and adaptation behaviors generalize across additional languages, ecosystems, and development contexts.

8 Declarations

The authors report their contributions, data-availability information, and absence of competing interests.

  • Abdullah Al Mujahid led the methodology, primary analysis, formal analysis, investigation, and original manuscript draft, while all three authors contributed to conceptualization and manuscript review and editing.
  • The study’s analyzed data are publicly available from the sources described in the manuscript.
  • The authors report no competing interests related to analyzing, studying, or publishing the research.
Loading 2606.06843v1…