Developer-LLM Conversations: An Empirical Study of Interactions and Generated Code Quality

• CodeChat contains 587,568 real-world developer–LLM conversations, 1.1 million turns, and 1.7 million code snippets across more than 20 programming languages.Source: §I · p. 1
• Web design (9.6%) and machine learning (8.7%) are the most common topics, while code defects are widespread and do not consistently improve across turns.Source: §I · pp. 1–2
• AI-augmented business tool automation has the highest mean interaction length at 3.40 turns, with use-case shifts, feature revisions, and unresolved erroneous responses among frequent prompt gaps.Source: §I · p. 2
• Across Python, JavaScript, C++, Java, and C#, first-turn code commonly has undefined-variable and documentation defects, and their prevalence does not consistently decrease in multi-turn conversations.In Python, undefined-variable errors rise from 23.5% to 32.8% after five turns, while Java documentation violations fall from 78.1% to 63.4%.Source: §I · p. 2
• The paper characterizes topics and interaction duration, analyzes multi-language code quality across turns, and releases the dataset and replication package.Source: §I · p. 2

II. EXPERIMENT SETUP

• The experiment uses CodeChat, derived from WildChat, to answer questions about common developer prompts and LLM-generated code quality.Source: §II · p. 2

A. Data Collection

• WildChat is a public collection of 3.2 million conversations and 5.2 million turns, collected across 76 languages from April 2023 to July 2025.Source: A. Data Collection · p. 2
• CodeChat includes conversations where at least one LLM turn contains a triple-backtick code block followed by a programming-language identifier; this filter yields 587,568 conversations, 1.1 million turns, and 1.7 million code snippets.Source: A. Data Collection · p. 2

B. Dataset Analysis

• CodeChat contains 587,568 conversations and 1.7 million code snippets, exceeding DevGPT’s 29,778 conversations and GPT-Social’s 332.It spans 28 months, from April 2023 to July 2025, and models from GPT-3.5 to o1.Source: B. Dataset Analysis · p. 2
• 81% of conversations are single-turn, while 3% (15,409) extend to at least 10 turns; JSON and Python each account for 20% of generated languages.Bash accounts for 16%, followed by JavaScript at 7%, and Java and HTML at 3% each.Source: B. Dataset Analysis · p. 3

C. Common Topics in CodeChat

• Topic modeling uses English initial prompts, with BERTopic settings selected through coherence scoring and manual checks for coherent themes.The selection process examines representative prompts and their alignment with each topic’s keywords.Source: C. Common Topics in CodeChat · p. 3
• An intent shift is marked when the initial topic’s probability drops by more than 5% as cumulative prompts are reclassified.This assigns a topic to each turn in multi-turn conversations.Source: C. Common Topics in CodeChat · p. 3
• Conversation turn counts measure engagement, and the Scott–Knott test partitions topics into groups with similar mean turn counts.The test considers all topics together rather than making only pairwise comparisons.Source: C. Common Topics in CodeChat · pp. 3–4
• The prolonged-interaction analysis manually labels 383 conversations from the topic cluster with the highest average turn count, then measures recurring sequences of three prompt-gap types.Each gap label represents the reason for a follow-up prompt; the sample uses a 95% confidence level and 5% margin of error.Source: C. Common Topics in CodeChat · p. 4

D. Quality of LLM generated Code

• Quality analysis covers Python, JavaScript, C++, Java, and C#, excluding JSON, HTML, and Bash because of assessment suitability or limited structural complexity.JSON and HTML lack executable logic to assess, while shell scripts have limited structural complexity and weak tooling.Source: D. Quality of LLM generated Code · p. 4
• Adjacent turns with sufficiently similar code are grouped into one task sequence, while low similarity marks the start of a new task.Code clone detection is used to filter task switches and unrelated generated code.Source: D. Quality of LLM generated Code · p. 4
• Language-specific tools assess defects and code smells: Pylint and PMD, ESLint, Cppcheck, and Roslyn for Python, Java, JavaScript, C++, and C#, respectively.The tools detect issues including undefined variables, null pointer dereferences, syntax errors, and style violations.Source: D. Quality of LLM generated Code · p. 4
• The longitudinal analysis tracks first-turn issue types through Turns 1–5 in task sequences containing at least five turns, using regression or Mann-Kendall trend tests.A statistically significant regression slope at p < 0.05 is interpreted as evidence of a trend.Source: D. Quality of LLM generated Code · pp. 4–5
• Syntax-error resolution analysis samples 383 cases where an erroneous response is followed by a syntax-error-free one, then labels prompts using a nine-category scheme plus two added categories.Two annotators achieved Cohen’s kappa of 0.90.Source: D. Quality of LLM generated Code · p. 5

A. RQ1: What topics do developers most commonly prompt

• Topic 8, AI-augmented business tools and strategy automation, has the highest mean engagement at 3.4 turns and forms its own Scott–Knott group.Machine learning model training and AI bot deployment averages 2.89 turns, while Excel VBA automation and data management averages 2.86.Source: A. RQ1: What topics do developers most… · pp. 5–6
• In Topic 8, three successive Different Use Cases gaps make up 16.0% of observed three-gap chains, the most frequent pattern.Other frequent patterns alternate additional functionality and use-case changes (2.3% and 1.9%); three successive erroneous responses account for 1.9%.Source: A. RQ1: What topics do developers most… · p. 6, Figure 6 · p. 6
• Web design and development accounts for 9.6% of conversations, machine learning for 8.7%, and low-level programming and memory manipulation for 8.3%.The low-level total comprises assembly-level operations (4.9%) and binary patching (3.4%).Source: A. RQ1: What topics do developers most… · pp. 5–6

Python

• 83.4% of first-turn Python snippets have invalid naming; UndefinedVar rises from 23.5% at turn 1 to 32.8% at turn 5, while ImportError falls from 48.3% to 44.6%.Both turn-to-turn changes are statistically significant (p < 0.05).Source: B. RQ2: How high is the quality of… · p. 6, Table III · p. 6, Figure 7 · p. 7

JavaScript

• 75.3% of first-turn JavaScript snippets have undefined variables; unused variables affect 33.7%, syntax errors affect 14.4%, and top issues remain statistically unchanged across turns.Unexpected angle brackets cause syntax errors in 5.8% of snippets.Source: B. RQ2: How high is the quality of… · pp. 6–7, Table III · p. 6, Figure 7 · p. 7

C++

• 41.1% of initial-turn C++ snippets have missing #include headers; MissingIncludeSystem rises from 37.9% at turn 1 to 46.5% at turn 5, without statistical significance.SyntaxError also rises from 4.2% to 8.1%, but that change is not statistically significant.Source: Table III · p. 6, B. RQ2: How high is the quality of… · p. 7, Figure 7 · p. 7

Java

• 75.9% of initial-turn Java snippets lack required comments; CommentRequired falls from 78.1% at turn 1 to 63.4% at turn 5, and LocalVariableCouldBeFinal falls from 51.6% to 41.3%.Both declines are statistically significant (p < 0.05).Source: Table III · p. 6, B. RQ2: How high is the quality of… · p. 7, Figure 7 · p. 7

C#

• 49.2% of initial-turn C# snippets have unresolved namespaces, whose prevalence remains statistically unchanged across multi-turn conversations.The reported lack of change has p > 0.05.Source: Table III · p. 6, B. RQ2: How high is the quality of… · p. 7, Figure 7 · p. 7

Other reported issues

• Omitted accessibility modifiers (10.0%) and formatting violations (9.8%) were also reported, both affecting code readability.The cited human-written C# comparison reports aggregate usability measures rather than per-issue rates.Source: B. RQ2: How high is the quality of… · pp. 6–7

Syntax-error resolutions

• Among 383 sampled transitions, the prompt type Point out mistake then request fix was most frequent, occurring in 23.0%, and frequently preceded syntactically valid code generation.The three leading prompt types together accounted for 56.4% of resolved cases.Source: B. RQ2: How high is the quality of… · p. 7

Summary of RQ2

• Across five languages, frequent defects are language-specific, and multi-turn prevalence does not consistently decline; Python import errors and Java documentation violations do decrease.The summary identifies invalid naming in Python, undefined variables in JavaScript, missing #include in C++, missing required comments in Java, and namespace-unfound errors in C#.Source: B. RQ2: How high is the quality of… · pp. 6–7

Managing evolving developer intent

• Repeated use-case shifts are the most common prompt-gap pattern in prolonged conversations; the authors recommend detecting intent shifts and confirming redirected goals.They also call for research that models intent evolution across turns.Source: §IV · p. 8

Reusing real-world interactions

• CodeChat captures real coding sessions across 20+ languages and multiple turns, including conversations reaching ten turns or more, supporting fine-tuning and realistic multi-turn, long-context evaluation.The authors suggest extending static analysis to security and vulnerability assessment and comparing other or newer models against their ChatGPT-based baseline.Source: §IV · p. 8

Building multi-turn benchmarks

• CodeChat can support conversation-level benchmarks of intent-shift recovery, code-quality regressions, error-correction rates, turns to resolution, and robustness to long contexts.These measures would complement single-turn benchmarks; the authors identify realistic and reproducible evaluation as the dataset’s primary value.Source: §IV · p. 8

Internal validity

• Static-analysis configuration poses an internal-validity threat, which the study mitigates by following each linter’s official documentation and recommendations.The evaluated tools are selected for their recognized reliability and relevance to the languages studied.Source: §V · p. 8

Scope of quality assessment

• The quality assessment covers only issues supported by static analyzers, excluding attributes such as code complexity and security vulnerabilities.Source: §V · p. 8

External validity

• CodeChat’s likely IT-community skew means its topic distribution characterizes that population rather than developers generally; its results cover ChatGPT models collected from April 2023 to July 2025.The framework could be applied to other LLMs, but specific frequencies and trends may differ as model capabilities change.Source: §V · p. 8
• Findings may not generalize to repository-based coding agents that access project files, execute code, and revise outputs using tool feedback.CodeChat instead contains prompt-response interactions with conversational coding assistants.Source: §V · p. 8

Developer-LLM interaction datasets

• CodeChat comprises 587,568 unconstrained, unedited conversations from a public chatbot service, compared with prior curated or platform-centric sources such as DevGPT’s 29,778 conversations.The authors argue that CodeChat reflects a broader range of everyday tasks than specialized prompts shared on GitHub.Source: §VI · p. 8

Code quality of LLM-generated code

• Unlike prior studies focused on selected tasks, snippets, or single-turn settings, this study analyzes real-world code across five languages and tracks quality issues over consecutive turns.Source: §VI · p. 8

• Developers often seek help with web design, machine-learning model training, and low-level memory operations, while prolonged conversations reflect shifting use cases and iterative refinement.Source: §VII · p. 8
• CodeChat contains 587,568 real-world developer-LLM conversations and 1.7 million code snippets across more than 20 programming languages.Source: §VII · p. 8
• Generated code frequently has syntax errors, undefined variables, and maintainability problems, with undefined variables becoming more prevalent across turns.Source: §VII · p. 9