Source-linked AI summary
On the Prospects of Dynamic LLM Conversations in Software Development
Annemarie Wittig, Alina Mailach, Janet Siegmund, Norbert Siegmund
TL;DR
The paper addresses limited evidence on how to support developers’ interactions with chat-based LLMs. It evaluates minimal context-aware and proactive interventions against a control condition in a four-month longitudinal Web-development study, finding no major harmful effects and more positive perceptions with proactive guidance.
Problem
Research has rarely examined how to guide developer–LLM conversations, despite evidence that developers often provide insufficient context and that interaction quality depends on prompting.
Method
The study compares context and proactive guidance with an uninfluenced control among third-semester computer science students completing a full-stack Web project for four months.
Results
Across interaction patterns, perceptions, and downstream code outcomes, the interventions showed no major negative effects, while proactive guidance received the most positive developer perceptions.
Takeaways & Limitations
Even conservative guidance mechanisms can produce observable differences in developer perceptions, with proactive guidance identified as a promising direction for future work.
Takeaways & Limitations
Because participants were novice students rather than professional developers, the findings’ generalizability is limited to novice developers.
Abstract
from arXiv · showhide
Large language models (LLMs) have become an essential tool for assisting developers, yet we still lack knowledge on ways to effectively support their interactions during development activities. That is, the quality of interactions with a chat-based LLM still strongly depends on how developers phrase prompts and which information they include. Our goal is to evaluate whether interventions into these interactions with LLMs have an effect on software developers---be it harmful or beneficial. To this end, we conducted a four-month longitudinal study with third-semester computer science students working on a full-stack Web development project using chat-based LLMs under three conditions: (1) a \emph{context}-aware group received intent-based conversation augmentation, (2) a \emph{proactive} group received follow-up suggestions and tailored advice, and (3) a \emph{control} group without intervention. Our augmentations are minimal: (i) to reduce confounding factors and (ii) to isolate treatment effects. Analyzing interaction logs and user surveys revealed no major differences in interaction patterns, indicating no detectable harmful effects in the measured outcomes when intervening in interactions. Moreover, we observed trends of increased satisfaction with the \emph{proactive} treatment. The results indicate that even with minimal interventions, dynamic guidance mechanisms for developer-LLM interactions show observable effects, such that more severe augmentations may have the potential to substantially improve developer satisfaction.
1 Introduction
LLMs are widely used in software development, but research has rarely examined how to guide developer–LLM conversations. This study evaluates minimal context-aware and proactive interventions in a four-month software project.
- LLMs assist software engineers with tasks including code generation, requirements elicitation, debugging, and documentation.
- Among 43 LLM-centered papers at recent ICSE, ASE, and FSE research tracks, only seven analyzed conversational structure and developer interactions, while two attempted to improve them.
- The context treatment augmented the LLM system message using a user-selected intention, whereas the proactive treatment recommended follow-up prompts and tailored advice.
- The study compared both treatments with a control group of third-semester computer science students during a four-month full-stack Web development project.Researchers logged interactions and conducted monthly surveys tracking user experience.
- The interventions were deliberately minimal to reduce confounding factors and isolate treatment effects.
- The work contributes longitudinal usage analyses across control, context, and proactive assistants, plus a replication package containing the application, materials, and empirical-study data.
2 Related Work
Prior work documents limitations in developer–LLM interactions and explores prompt, context, and model improvements. However, direct guidance of developers during conversations remains comparatively underexamined.
- Studies report that LLMs can misunderstand requirements, task context, problems, goals, and rationales, producing faulty conversations.
- Some approaches fine-tune LLMs or automatically refine the latest conversational prompt using role prompting.
- Direct conversation interventions that add user-highlighted code or other contextual information have been associated with more task completions and potentially improved interactions.
- Research in other fields has enhanced chat interactions to improve empathy, efficiency, and responsiveness to users’ intentions and needs.
3 Methodology
The study examines how contextual, proactive, and absent guidance affect developers’ interactions with LLMs, their perceptions, and generated code during a four-month web-development project. It operationalizes these effects through controlled treatments and measures of behavior, perception, and code usage.
- Research Questions: The study asks how guidance influences developer behavior, perceptions of LLMs, and the code they produce.These questions correspond to interaction, experience, and code-artifact outcomes.
- Research Questions: The experiment compares contextual guidance, proactive guidance, and no guidance as three levels of developer–LLM support.Contextual guidance adds project-task and user-intention context; proactive guidance offers follow-up prompts, prompt feedback, and tailored advice.
- Dependent Variables: User behavior is measured through conversation length, prompt-token use, conversation intentions, and alignment between stated intentions and conversation content.The analysis examines interaction extent, changes over time, intention switching, and mismatches between selected intentions and prompts.
- Dependent Variables: Perception is assessed monthly through participants’ ratings of LLM helpfulness, frustration impact, and productivity impact.These questionnaires evaluate how participants experience guided interactions over the study period.
- Dependent Variables: Generated-code usage is evaluated through repository contributions, modification rates, and how long committed code remains in projects.Researchers also manually review randomly selected conversations to assess the role and persistence of LLM-supported code.
- Study Conduct: The study follows 3rd-semester computer science students completing a full-stack web-development project over four months at two universities.Participants accessed a GPT-4-connected chatbot, could use it without restriction, and contributed project-related conversations whose histories were read-only.
4 Results and Discussion
Across treatments, developers showed no major differences in prompt counts, but conversations became shorter over time and proactive conversations started shorter. Intentions frequently diverged from conversation content, while proactive guidance showed the strongest alignment and higher perceived efficiency without significant overall behavioral differences.
- User Behavior: No significant difference appeared in prompts per conversation, but prompt-token counts differed significantly between control and context and between control and proactive.Kruskal-Wallis: prompts W=3.39, p=0.182; prompt tokens W=17.18, p<0.05. Dunn tests found control–context δ=−0.21 and control–proactive δ=−0.17.
- User Behavior: Conversations became shorter as the project progressed, while proactive conversations were already shorter in the first period.The decline occurred across all groups and was most severe for context treatment.
- Specified Intentions: Control participants most often selected language questions, while code generation was the most frequent intention in context and proactive groups.Language questions accounted for 30% of control conversations; code generation accounted for 54% of context conversations, with programming-related intentions also more frequent in proactive.
- Conversation Length: Intention affected conversation length potentially, whereas treatment and treatment–intention interaction did not show significant effects.The two-way ANOVA reported F=2.02, df=7, p=0.049 for intention, but post-hoc significance vanished; treatment F=2.3 and interaction F=0.8 were not significant.
- Intention Alignment: About 25% of conversations began with an intention mismatch, and 38.52% contained only prompts matching the stated intention.Misalignment commonly appeared after the first answer, with the second prompt often marking the shift.
- Intention Alignment: The proactive group had the highest intention alignment, but treatment differences in aligned and misaligned prompts were not significant.Solely aligned conversations were 37.78% for control, 31.11% for context, and 46.67% for proactive; χ2=5.31, p=0.069.
5 Threats to Validity
The study’s validity is constrained by participant attrition, novice student developers, and the university setting, although the longitudinal project design partly addresses ecological concerns.
- Internal Validity: Participant dropout could threaten internal validity by making completers more motivated or positive toward LLMs, especially for RQ2.The authors found no systematic baseline-attitude pattern between completers and dropouts, and declining participation may reflect reduced need for LLM assistance later.
- Statistical Conclusion Validity: Superusers could substantially affect statistical conclusions because most participants sent fewer than 200 prompts.The analysis removed three unusually heavy users, including one who sent over 400 prompts, during preprocessing.
- External Validity: Because participants were novice students rather than professionals, the findings may not generalize to the entire developer population.The authors interpret the results specifically for novice developers, while noting that this remains an important practical subpopulation.
- Ecological Validity: The university context limits applicability to practical settings, despite the project’s four-month duration and realistic development setting.The controlled design required a university setting, while the project’s longevity was intended to better resemble real-world development processes.
6 Conclusion
The study tested minimal context and proactive guidance in a four-month full-stack Web development project. Interventions produced no major harmful effects in measured usage outcomes, while proactive guidance was perceived most positively and minimal guidance showed promise for satisfaction and code adoption.
- 6 Conclusion: The study asks whether minimal guidance can support developers and affect software development during LLM interactions.It evaluates direct interventions in developer–LLM conversations rather than isolated programming tasks.
- 6 Conclusion: Context treatment added intent-based context, while proactive treatment supplied advice, improved prompts, and follow-up questions.Both approaches were designed to interfere minimally with the conversation.
- 6 Conclusion: Across interaction patterns, perceptions, and downstream code outcomes, conservative guidance showed no evidence of negative intervention effects.Conversation intentions varied between treatment groups, but the reason remained unclear.
- 6 Conclusion: Proactive guidance received the most positive developer perceptions, whereas context guidance was viewed critically but led to higher usage of LLM-generated code.The reported perception measures included helpfulness, frustration, and productivity.
- 6 Conclusion: Minimal interventions were associated with improved satisfaction and code adoption or modification, without disrupting workflow.Conversations also became shorter over the project, indicating increased efficiency among participants.
- 6 Conclusion: Higher guidance levels, including updated intentions or requests for more context, are identified as promising directions for future research.The authors position the sparse-intervention setup as a starting point for guiding early-career developers.
Ethical approval
The study did not require formal ethical approval under the participating universities’ institutional regulations.
- Ethical approval: Formal ethical approval was not required under the participating universities’ institutional regulations.
Informed consent
Participants provided informed consent before data collection and were told how participation and data handling would work.
- Informed consent: Participants gave informed consent before any data was collected.They were informed about the study purpose, voluntary participation, data use and protection, and withdrawal rights.