Source-linked AI summary
How Developers Discuss Generative AI: A Longitudinal Study of the Visual Studio Code Community
Panida Rumriankit, Akito Monden, Hiroki Inayoshi, Pattara Leelaprute, Bundit Manaskasemsak, Kundjanasith Thonglek, Arnon Rungsawang
TL;DR
The paper addresses limited evidence about how mainstream open source developers discuss generative AI in practice. It analyzes longitudinal VS Code GitHub Issues using keyword retrieval, semantic filtering, topic modeling, and trend tests, finding that discussions center on operational concerns and shift toward conversational and agent-based development.
Problem
Little is known about how developers in mainstream software projects discuss generative AI as it evolves from code completion to conversational and agent-based development.
Method
The study analyzes VS Code GitHub Issues with keyword retrieval, preprocessing, semantic relevance filtering, BERTopic, theme validation, robustness re-clustering, and temporal trend analysis.
Results
Developer discussions primarily concern AI-assistant operation, including reliability, authentication, configuration, workflow integration, and billing, while hallucination and licensing are comparatively uncommon.
Takeaways & Limitations
GitHub Issues provide a practical, workflow-oriented view of generative AI that complements survey-based accounts of developer experience.
Takeaways & Limitations
The study uses a single annotator and one Microsoft-owned repository with first-party Copilot integration, limiting validity and generalizability.
Abstract
from arXiv · showhide
Generative AI tools such as GitHub Copilot, ChatGPT, and coding agents have rapidly become part of everyday software development, yet little is known about how mainstream open source communities discuss them in practice. This paper presents a longitudinal analysis of generative-AI-related discussions in the Visual Studio Code (VS Code) GitHub repository, using 43,806 candidate issues created between January 2021 and June 2026. To improve corpus quality, we combined keyword retrieval with semantic relevance filtering, yielding a filtered corpus of 25,227 AI-related issues. We applied BERTopic to the retrieved corpus to identify discussion topics, using the filtered corpus for theme validation and a robustness re-clustering, and analyzed their evolution over time using monthly prevalence and Mann-Kendall trend tests. The results show that developer discussions are dominated by practical concerns regarding the operation of AI-assisted development environments, including agent management, configuration, reliability, authentication, and billing, whereas risks frequently emphasized in survey-based studies, such as hallucination and licensing, rarely surface in this venue. This suggests that discussions of generative AI in the VS Code issue tracker primarily focus on operational aspects of AI-assisted software development. Furthermore, discussions evolved from AI-assisted code completion toward conversational and agent-based development, reflecting the increasing integration of generative AI into software development workflows. These findings suggest that GitHub Issues provide a practical, workflow-oriented perspective on generative AI that complements survey-based studies of developer perceptions.
I. INTRODUCTION
The study examines how a mainstream open source community discusses generative AI as development practices evolve. It finds that issue-tracker discussions emphasize operational concerns and progress from code completion toward conversational and agent-based development.
- Motivation and scope: Open source repositories preserve naturally occurring discussions that reveal how developer concerns evolve during technology adoption.This motivates using GitHub Issues as an empirical source.
- Motivation and scope: Mainstream developer discussions of generative AI remain understudied across the transition from optional code completion to conversational and autonomous agent modes.The study addresses this gap through a longitudinal analysis of the VS Code repository.
- Research questions: The study asks which generative-AI topics emerge, what concerns and expectations they reveal, and how discussions changed from 2021 to 2026.These questions organize the analysis of the VS Code community.
- Contributions: The study characterizes the transition from AI-assisted code completion toward conversational assistants and agent-based software development.This transition is the paper’s longitudinal focus.
- Contributions: Discussions are dominated by operational concerns such as infrastructure, configuration, authentication, quotas, billing, and reliability, while hallucinations, licensing, and code quality appear infrequently.The findings contrast issue-tracker discourse with conceptual risks emphasized in previous research.
- Contributions: The analysis combines semantic relevance filtering, topic modeling, manual validation, and temporal trend analysis to identify long-term changes in developer discussions.This pipeline is presented as supporting reliable theme identification.
II. RELATED WORK
Prior research has documented productivity effects, risks, and discussions in AI-specialist repositories, but these studies leave mainstream developer communities comparatively unexplored.
- Prior findings: Earlier studies measured productivity gains from AI assistants while also documenting increased coordination time and risks in AI-generated code.Reported examples include faster implementation, more project-level contributions, and security weaknesses across multiple CWE categories.
B. Mining GitHub Issues
GitHub Issues support empirical analysis of software-development discussions, but prior mining studies largely focused on AI-specialist communities. Topic modeling and temporal analysis provide the methodological basis for addressing this gap.
- Issue mining: Prior GitHub Issue studies found runtime errors and unclear instructions prominent in AI-research repositories, while sparse issue labeling required text-based analysis.One study analyzed 24,953 issues and found only 7.81% of repositories labeled their issues.
- Research gap: Studies of ChatGPT projects and AI-agent frameworks identified changing topics, but their communities were AI-specialist rather than mainstream software projects.This leaves open how mainstream developers discuss generative AI.
- Topic modeling: LDA is widely used in software engineering, whereas BERTopic combines contextual embeddings with clustering and class-based TF-IDF to capture similarities missed by bag-of-words models.The paper extends topic extraction and temporal trend analysis while following cautions about corpus-specific LDA configuration.
A. Data Collection
The study collected VS Code issues through high-recall keyword searches, preprocessing, and semantic relevance filtering, then modeled the resulting discussions with BERTopic and LDA.
- Issue retrieval: Twelve case-insensitive keywords were searched in monthly windows, with recursive splitting, deduplication, and a partial final month ending June 9, 2026.The procedure addressed GitHub Search API result caps and boolean-query limits.
- Preprocessing: 43,761 preprocessed documents remained after removing fewer-than-five-token documents from the 43,806 retrieved issues.Documents combined issue titles, bodies, and comments before standard text preprocessing.
- Semantic filtering: A MiniLM-and-logistic-regression relevance classifier filtered the keyword corpus into 25,227 AI-related issues, achieving 0.79 precision and 0.84 recall in five-fold cross-validation.The filtered corpus supported theme distribution analysis and robustness re-clustering, while primary modeling used the keyword corpus.
- Topic modeling: The primary topic model used BERTopic with sentence embeddings, UMAP, HDBSCAN, and unigram–bigram class-based TF-IDF, with outliers reassigned by embedding similarity.An optimized LDA model with k = 5–30 topics provided a comparison.
C. Validation and Temporal Analysis
The study validates topic quality through quantitative, manual, and reliability checks, then tracks monthly topic shares over time. Issue volume rises sharply after agent-mode integration, motivating share-based longitudinal analysis.
- Topic quality was evaluated with coherence, topic diversity, manual inspection, and blind re-labeling of 300 issues using Cohen’s κ.
- Monthly topic shares were analyzed with Mann-Kendall tests and Sen’s slope because corpus volume grows more than tenfold over the study period.Shares, rather than raw counts, support comparisons across months with different corpus sizes.
- Issue volume rose from 3,168 in 2024 to 20,278 in 2025 and 14,786 through 9 June 2026, coinciding with Copilot Chat and agent-mode integration.The 2026 count covers only 1 January–9 June.
B. Topic Model and Validation
The study assesses BERTopic before interpreting themes, combining coherence metrics, manual inspection, and reliability checks. BERTopic outperforms tuned LDA on coherence, while breadth sampling reveals substantial fine-grained topic diversity.
- BERTopic outperformed tuned LDA on both reported coherence metrics: Cv = 0.628 vs. 0.575 and CNPMI = 0.205 vs. 0.090.
- BERTopic was adopted as the primary method because its CNPMI advantage was 2.3×, while LDA’s higher topic diversity reflected its much smaller topic count.LDA used 5 topics versus BERTopic’s 125, making word reuse mechanically less likely.
- Among the 25 largest topics, 18 (72%) were judged coherent; across all 125 topics, 50 were AI-substantive and test–retest reliability was κ = 0.73.The AI-substantive topics represented 40.1% of documents.
- The breadth sample of 625 issues exposed fine-grained concerns involving model providers, content filtering, agent sandboxing, and billing disputes.These themes indicate that the largest topics understate discourse diversity.
C. RQ1: Topics Discussed
The validated topics show that VS Code discussions focus primarily on operating and maintaining AI-assisted development environments. Operational themes dominate the filtered sample, while hallucination, licensing, and related conceptual risks are comparatively uncommon.
- AI-substantive topics center on agent editing, session management, execution failures, configuration, responsiveness, installation, quotas, billing, and authentication.These themes indicate that developers primarily configure, operate, and troubleshoot AI assistants.
- Ordinary editor topics entered through incidental AI-keyword mentions and were excluded from interpretation of AI-related discussions.
- Overall, the issue tracker emphasizes operation and maintenance of AI-assisted development environments over underlying AI technologies.
- Bug fixing dominates the validated sample at 49.1%, followed by productivity and workflow concerns at 20.7% and security at 8.9%.
- Hallucination accounts for 2.4%, licensing 0.6%, code generation 1.2%, and code review is absent from the validated issue themes.
E. RQ3: Evolution 2021–2026
AI-related discussions shifted from code completion toward conversational and agent-based development, while increasing attention centered on operating AI-assisted environments. Apparent declines in traditional editor topics were relative-share composition effects, not established absolute declines.
- Trend analysis: 84 of 122 topics increased significantly on the keyword corpus, while 17 decreased at p < 0.05.In the filtered-corpus robustness re-clustering, 78 of 86 topics increased significantly and none decreased.
- Increasing topics: Increasing topics mainly covered agent and chat session management, file-context handling, installation and verification, and inline chat.These topics emerged after 2023 and became more prominent after Copilot Chat and agent-mode features were introduced.
- Decreasing topics: Decreasing topics primarily involved traditional editor functions, but their decline represented reduced relative share as AI discussions expanded.No decreasing topic reappeared in the filtered-corpus robustness re-clustering, so the study treats these declines as composition effects.
- Discussion phases: AI-related discussions progressed from Copilot code completion in 2021–2022 to Copilot Chat in 2023–2024 and agent-based development from 2025 onward.The broader shift was toward configurable and increasingly autonomous development partners.
V. DISCUSSION
The VS Code issue tracker emphasizes the practical operation of AI assistants rather than the conceptual risks commonly highlighted in prior research. This indicates that surveys and issue trackers capture complementary, non-interchangeable aspects of developer experience.
- Operational focus: VS Code discussions focus on deploying and operating AI assistants, including configuration, authentication, billing, and reliability.These concerns differ from the hallucination, licensing, and code-quality risks often emphasized in existing literature.
- Workflow integration: Developers increasingly treat AI assistants as development infrastructure integrated into everyday workflows rather than standalone productivity tools.This reflects a changing role for generative AI within the development environment.
VI. THREATS TO VALIDITY
The study’s validity is constrained by imperfect retrieval, single-annotator judgment, limited generalizability beyond VS Code, and trend interpretations affected by corpus composition. These constraints qualify how broadly the findings and longitudinal conclusions should be applied.
- Construct validity: Keyword retrieval included substantial incidental AI-term mentions, while the semantic classifier reduced but did not eliminate irrelevant issues and could miss relevant ones.The sampled incidental rate fell from 44.7% to 13.0%; classifier recall was 0.84, and excluding “GPT” from server-side search may further reduce recall.
- Internal validity: Topic validity relies on a single annotator, leaving coder subjectivity unresolved without a second independent human annotator.The paper identifies a second independent annotator as the priority next step.
- External validity: Because VS Code is a single Microsoft-owned repository with Copilot as a first-party feature, its profile may not generalize to projects without first-party AI integrations.This limits external validity beyond repositories with a similar AI-integration context.
- Conclusion validity: Mann–Kendall can classify topic emergence as monotonic increase, so apparent decreases that vanish under robustness re-clustering are treated as composition artifacts.The study distinguishes emergent topics from genuinely trending topics when interpreting longitudinal results.
VII. CONCLUSION
The study analyzes 43,806 candidate VS Code issues, retaining 25,227 through semantic filtering, and finds that discussions center on operational concerns while shifting toward agent-based development. GitHub Issues therefore provide a practical, workflow-oriented perspective that complements survey-based studies.
- 43,806 candidate issues were analyzed, with 25,227 retained through semantic filtering for theme validation and robustness re-clustering.
- Developer discussions are dominated by operational concerns such as configuration, reliability, authentication, and workflow integration.
- Discussions shifted from AI-assisted code completion toward conversational and agent-based development over time.
- GitHub Issues capture a practical, workflow-oriented perspective on generative AI that complements survey-based studies and other developer communities.
- Future work should validate the findings across additional open source projects and use complementary methods such as sentiment analysis and alternative topic modeling.