Source-linked AI summary
Programming by Chat: A Large-Scale Behavioral Analysis of 11,579 Real-World AI-Assisted IDE Sessions
Ningzhi Tang, Chaoran Chen, Zihan Fang, Gelei Xu, Maria Dhakal, Yiyu Shi, Collin McMillan, Yu Huang, Toby Jia-Jun Li
TL;DR
Empirical evidence about how developers use codebase-aware AI assistants in real-world IDE workflows has been limited. This study analyzes 74,998 messages from 11,579 public-repository sessions and finds progressive specification, delegated cognitive work, and active management of AI collaboration. Its scope is limited to selected public-repository users and text-based expressed behavior, not downstream development outcomes.
Problem
Empirical understanding of developer-AI conversation in real-world IDE-native settings remains limited.
Method
The study analyzes 74,998 developer messages from 11,579 sessions across 1,300 repositories and 899 developers using Cursor and GitHub Copilot.
Results
Developers progressively specify tasks, delegate diagnosis and comprehension to AI, and actively manage collaboration through plans, context, and action constraints.
Takeaways & Limitations
The study provides an empirical foundation for AI assistants that support iterative, context-dependent developer-AI collaboration.
Takeaways & Limitations
The dataset reflects public-repository SpecStory users, while the analysis captures text-based expressed intents rather than downstream quality, correctness, or productivity effects.
Abstract
from arXiv · showhide
IDE-integrated AI coding assistants, which operate conversationally within developers' working codebases with access to project context and multi-file editing, are rapidly reshaping software development. However, empirical investigation of this shift remains limited: existing studies largely rely on small-scale, controlled settings or analyze general-purpose chatbots rather than codebase-aware IDE workflows. We present, to the best of our knowledge, the first large-scale study of real-world conversational programming in IDE-native settings, analyzing 74,998 developer messages from 11,579 chat sessions across 1,300 repositories and 899 developers using Cursor and GitHub Copilot. These chats were committed to public repositories as part of routine development, capturing in-the-wild behavior. Our findings reveal three shifts in how programming work is organized: conversational programming operates as progressive specification, with developers iteratively refining outputs rather than specifying complete tasks upfront; developers redistribute cognitive work to AI, delegating diagnosis, comprehension, and validation rather than engaging with code and outputs directly; and developers actively manage the collaboration, externalizing plans into persistent artifacts, and negotiating AI autonomy through context injection and behavioral constraints. These results provide foundational empirical insights into AI-assisted development and offer implications for the design of future programming environments.
1 Introduction
The paper studies how developers use IDE-native AI assistants in real-world conversational programming. It finds progressive specification, delegated cognitive work, and active management of AI collaboration across messages and sessions.
- Motivation: IDE-native assistants let developers steer AI-generated code through dialogue with project context, multi-file editing, terminal execution, and runtime inspection.This differs from browser-based chat and extends beyond autocomplete.
- Research gap: Existing evidence is limited by small, performative, or controlled studies and by datasets focused primarily on browser-based chat rather than IDE workflows.Video studies may introduce audience effects, while controlled studies use researcher-defined tasks and unfamiliar codebases.
- Study design: The study analyzes 74,998 developer messages from 11,579 sessions across 1,300 repositories and 899 developers using Cursor and GitHub Copilot.The chats were committed to public repositories during self-directed development, capturing in-the-wild activity.
- Findings: Developers progressively specify tasks by refining outputs through successive instructions rather than describing complete tasks upfront.Opening turns frame tasks, while later turns respond to AI outputs and emerging failures; modification and debugging often persist across turns.
- Findings: Developers redistribute cognitive work by delegating diagnosis, code comprehension, and validation to AI assistants.They report symptoms and machine outputs, ask about system behavior, and request static or runtime validation.
- Findings: Developers manage collaboration by externalizing plans, constraining AI actions, injecting context, and refreshing sessions while preserving task continuity.Session-level analysis also identifies six recurring archetypes, from failure-driven debugging to extended co-development.
2 Background and Related Work
AI coding assistants have progressed from single-file autocomplete to conversational and increasingly agentic IDE systems. Prior empirical work spans observational studies, benchmarks, artifact datasets, and browser-based conversations, but does not capture IDE-native interaction comprehensively.
- Evolution of AI coding assistants: Early coding assistants primarily offered single-turn, localized inline completion without external tools, code execution, or coordinated multi-file edits.Studies from that period consequently examined narrowly scoped interactions.
- Evolution of AI coding assistants: Cursor and later GitHub Copilot versions introduced chat-based interaction within IDEs and broader project contexts.The shift also produced the practitioner term “vibe coding” for accepting generated code without reading diffs or diagnosing pasted errors.
- Existing evidence: Video, livestream, interview, discourse, and practitioner studies examine prompting, delegation, debugging, and co-creation in AI-assisted coding.These sources provide qualitative and observational evidence but do not by themselves establish large-scale IDE-native behavior.
- Existing evidence: Existing datasets include standardized benchmarks, artifact-level outcomes, field usage measurements, and persistent project specifications such as Cursor rules and CLAUDE.md files.Persistent specifications capture project-level artifacts rather than dynamic multi-turn interaction.
- Research gap: DevGPT and CodeChat analyze developer conversations from browser-based ChatGPT and WildChat, whereas IDE-native datasets capture project context and integrated actions.The paper positions the absence of IDE-native conversational data as the central related-work gap.
- Methodological precedent: Prior large-scale LLM studies use classifiers for topics, tasks, usage contexts, and follow-up utterances, providing methodological precedents for this study’s session-level analysis.Their typical focus is general conversation topics or occupational tasks rather than coding-assistant interaction in IDEs.
3 Methodology
The study uses mixed methods to analyze AI coding-assistant chats at both the message and session levels. It combines behavioral-intent classification with session archetype and intent-dynamics analysis.
- Analytical design: The methodology analyzes message-level intents with a taxonomy and LLM-based classification, then examines session-level archetypes and intent dynamics.These analyses address the paper’s two complementary levels of conversational programming behavior.
3.1 Data Collection and Preparation
The dataset was assembled from SpecStory chat exports stored in public GitHub repositories, parsed and deduplicated before excluding CLI-agent sessions and unclassifiable messages. The final corpus spans diverse repositories and multilingual developer messages.
- Collection: SpecStory exports timestamped AI coding-assistant chat histories as Markdown files stored in project repositories.The collection included IDE-integrated assistants such as Cursor and GitHub Copilot, while CLI-agent histories were later excluded.
- Collection: GitHub Code Search API retrieval used date-prefix partitioning because each query was capped at 1,000 results.Data were collected on March 4, 2026, covering sessions from September 2024 through March 2026.
- Parsing: Parsed conversational turns were identified from SpecStory role markers such as “User,” “Assistant,” and “Agent,” with a manual sample of 100 sessions showing accurate turn and role assignment.The parser and format specifications were provided in the replication package.
- Cleaning: Deduplication removed 3,512 duplicate files, and the initial 1,705 CLI-agent sessions were excluded before final analysis.Duplicates primarily arose from copied or snapshot chat histories within repositories.
- Final dataset: The final analytic dataset contains 74,998 user messages across 11,579 sessions, 1,300 repositories, and 899 developers.Messages and sessions without classifiable behavioral intents were excluded after classification.
- Dataset characteristics: Repositories skew toward personal and early-stage projects: 67.3% have one contributor, 90.8% were created in 2025 or later, and the median is 15 commits over 4 active days.TypeScript, Python, JavaScript, and HTML are the most common languages.
- Dataset characteristics: English accounts for 59.7% of messages, followed by Chinese at 18.5% and Japanese at 8.3%.Because prompts mix natural language, code, and technical tokens, these proportions are contextual indicators rather than precise linguistic annotations.
3.2 Behavioral Intent Characterization
The study builds and validates a multi-label behavioral-intent taxonomy for AI coding-assistant messages, combining iterative coding, LLM classification, and qualitative thematic analysis.
- Taxonomy Development: Messages were labeled by communicative intent rather than surface form or topic, using a multi-label scheme for turns containing multiple acts.The approach draws on Speech Act Theory and accommodates examples combining actions such as configuration reference and database-connection instruction.
- Taxonomy Development: The taxonomy was developed from scratch through four rounds of iterative abductive coding and stabilized at 7 main categories with 20 subcategories.The number of newly added or substantively revised subcategories fell from 9 to 5, 1, and 0 across the four rounds.
- Dataset and Classification: The LLM classifier achieved macro-averaged F1 = 0.802 against an adjudicated reference set of 400 messages.The reported precision was 0.774 and recall was 0.851; two researchers independently annotated the validation sample before adjudication.
- Qualitative Analysis: Two researchers reviewed 80 randomly sampled messages per subcategory and reconciled interpretations into thematic summaries of recurring behavioral patterns.The qualitative analysis prioritized consensus-based thematic synthesis rather than fixed categorical labeling.
3.3 Session Archetypes and Dynamics Analysis
The analysis represents sessions as ordered sequences of multi-label intent sets, compares them with hierarchy-aware normalized edit distance, and examines clustering and intent transitions over time and across session boundaries.
- Session Representation: Each session is represented as an ordered sequence of non-empty intent-label sets drawn from 20 subcategories.This representation preserves message order while accommodating multi-label messages.
- Session Clustering: Hierarchy-aware edit distance assigns lower substitution cost within a main category than across main categories.Matching labels cost 0, within-category substitutions cost 0.5, and cross-category substitutions cost 1.0; insertions and deletions cost 1.0.
- Session Clustering: Normalizing raw weighted edit distance by the longer session length constrains the resulting distance to d(s,s′) ∈ [0, 1].The bound follows because cross-category substitutions and indels each cost 1.0, making raw distance no greater than the longer sequence length.
- Session Clustering: Clustering 4,864 sessions with at least four messages selected k = 6 using the peak silhouette score of 0.0703.Absolute silhouette values were low across all tested k values, ranging from 0.0367 to 0.0703.
- Intent Dynamics Analysis: Intent dynamics were analyzed through within-session lift-weighted transitions, session-boundary transitions, repetition lengths, and normalized session-position trends.Opening messages were also removed in an auxiliary analysis because they strongly shaped the observed trajectories.
4 Results
Developers used AI assistants for iterative authoring, diagnosis, comprehension, validation, planning, and autonomy management. Session behavior ranged from short, focused tasks to extended, mixed-mode collaboration.
- Behavioral intents: 34.53% of messages involved Code Authoring, while New Implementation accounted for 5.86% and Iterative Modification for 24.84%, indicating refinement dominated new implementation.Alignment Correction co-occurred with Iterative Modification in 43.02% of cases.
- Behavioral intents: Failure Reporting comprised 24.00% of messages, with developers commonly supplying symptoms or machine outputs while the assistant performed causal interpretation.Symptom Description represented 14.77% and Log Paste 8.84%.
- Behavioral intents: Developers queried AI about system behavior and feature logic more often than code semantics, with Project Comprehension accounting for 8.19% of messages.Code-centric questions about specific code behavior or structure were comparatively rare.
- Behavioral intents: Validation appeared in 3.99% of messages through Code Review and Runtime Inspection, often alongside iterative modification and symptom description.Code Review accounted for 2.74% and Runtime Inspection for 1.26%.
- Collaboration management: Documentation appeared in 6.85% of messages, externalizing plans, constraints, explanations, and progress as persistent memory across turns and sessions.Artifacts included Markdown planning, requirements, progress files, module summaries, deployment audits, and architecture diagrams.
- Collaboration management: Information Injection accounted for 8.46% of messages and Behavior Specification for 6.14%, showing that developers supplied context and constrained or delegated AI behavior.Constraints included requests not to modify code, stop conditions, and open-ended delegation.
- Session structure: Six recurring archetypes ranged from failure-driven debugging and focused refinement to continuation-driven delegation, toolchain operations, planning, and extended co-development.Five archetypes had median lengths of 6–8 messages, whereas Extended Iterative Co-Development had a median of 27 and the most balanced intent profile.
5 Threats to Validity
The study’s validity is constrained by its public, text-based dataset and by methodological choices that may affect observed distributions and archetypes.
- External validity: The dataset may underrepresent private and enterprise projects, early-stage practice, and non-IDE forms of AI-assisted programming.It reflects developers who use SpecStory and commit chat histories publicly, and should be read as a snapshot of evolving practice.
- Data completeness and construct validity: The analysis excludes nontextual attachments and UI actions, and captures expressed intents rather than downstream code quality, correctness, or productivity.Information Injection and behaviors such as regeneration or code acceptance are not consistently represented.
- Internal validity: Subjective coding and residual classifier errors may influence frequency estimates, although the main findings rely on aggregate distributions and recurring patterns.The classifier was validated on a stratified sample covering all 20 subcategories.
- Internal validity: Alternative choices for sampling, context truncation, session length, and edit-distance parameters may yield different distributions or archetypes.The authors therefore interpret clusters as descriptive archetypes rather than sharply bounded classes.
6 Discussion
The discussion characterizes conversational programming as iterative, cognitively distributed, and actively coordinated, with implications for persistent planning and benchmark design.
- Progressive specification: Conversational programming distributes specification across dialogue, as developers refine outputs through successive instructions and respond to emerging failures.Alignment Correction and Symptom Description can introduce constraints that were not articulated upfront.
- Progressive specification: 33.77% of messages were 50 characters or shorter, while 9.02% exceeded 1,000 characters, reflecting communication whose meaning often depends on accumulated context.Log Paste messages had a median length of 718 characters, while Information Injection had a median of 162 characters.
- Progressive specification: Documentation appeared in 6.85% of messages, showing that developers organically externalized plans and progress into persistent documents.These documents record intent and progress across turns and resemble emerging spec-driven development practices.
- Implications: Benchmark tasks that assume complete specifications may miss IDE-native abilities to tolerate ambiguity, track evolving intent, and surface missing constraints.The discussion points to underspecified-task benchmarks as an emerging response to this mismatch.
- Distributed cognition: Developers offloaded diagnosis, comprehension, and validation to AI while retaining roles as symptom reporters, intent clarifiers, and final evaluators.Debugging often began with terse failure signals, while code understanding and validation were requested from the assistant.
- Distributed cognition: When diagnosis, comprehension, and validation rely on the same assistant, it may become the sole judge of both code behavior and correctness.The discussion identifies the absence of an independent check on the assistant’s own errors as a deeper concern.
- Collaboration management: Long, dense agentic interaction histories make collaboration state difficult to inspect, prompting status checks, action summaries, context injection, and behavioral constraints.Developers used explicit requests such as recording changes in PROGRESS.md and imposed controls ranging from hard stops to open-ended handoffs.
- Future work: The study does not determine whether interactions achieved their goals and excludes CLI-agent sessions, limiting conclusions about success and more autonomous workflows.Its predominantly single-contributor, hobby-skewed sample also limits organization, domain, and time-period comparisons.
7 Conclusion
The paper presents a large-scale empirical account of IDE-native conversational programming, describing developer intents and recurring session-level patterns from real-world conversations.
- Conclusion: The study characterizes developers’ behavioral intents and recurring session-level patterns in real-world developer–AI conversations.The authors position this empirical foundation as informing future assistants for iterative, context-dependent collaboration.
Data Availability Statement
The replication materials are available, while raw chat-session data are withheld because of copyright and privacy considerations.
- Data availability: The replication package includes the codebook, classification prompts, and analysis code, but raw chat-session data are not redistributed.The raw data can be recollected through the provided scraping pipeline, and researchers may contact the first author for access.