PaperlayerView exact revision on arXiv

Source-linked AI summary

The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development

Steven I. Ross, Fernando Martinez, Stephanie Houde, Michael Muller, Justin D. Weisz

arXiv:2302.07080v1cs.HC

TL;DR

The paper examines conversational interaction with code-fluent foundation models for software development. It develops a system using conversational history and the user’s current utterance, finding varied and sometimes emergent assistance beyond code generation, including explaining code and answering programming questions.

  • Problem

    The paper asks whether modern code-fluent foundation models can support conversational interaction for software development.

  • Method

    The Programmer’s Assistant uses conversational history and the user’s current utterance to generate responses in a conversational interaction.

  • Results

    The system provided varied assistance, including producing code, explaining code, and answering general programming questions.

  • Takeaways & Limitations

    The study motivates conversational styles of interaction with large language models for software development.

  • Takeaways & Limitations

    The model sometimes produced incorrect, incomplete, irrelevant, or insubstantial responses.

Abstract

from arXiv · show

Large language models (LLMs) have recently been applied in software engineering to perform tasks such as translating code between programming languages, generating code from natural language, and autocompleting code as it is being written. When used within development tools, these systems typically treat each model invocation independently from all previous invocations, and only a specific limited functionality is exposed within the user interface. This approach to user interaction misses an opportunity for users to more deeply engage with the model by having the context of their previous interactions, as well as the context of their code, inform the model's responses. We developed a prototype system -- the Programmer's Assistant -- in order to explore the utility of conversational interactions grounded in code, as well as software engineers' receptiveness to the idea of conversing with, rather than invoking, a code-fluent LLM. Through an evaluation with 42 participants with varied levels of programming experience, we found that our system was capable of conducting extended, multi-turn discussions, and that it enabled additional knowledge and capabilities beyond code generation to emerge from the LLM. Despite skeptical initial expectations for conversational programming assistance, participants were impressed by the breadth of the assistant's capabilities, the quality of its responses, and its potential for improving their productivity. Our work demonstrates the unique potential of conversational interactions with LLMs for co-creative processes like software development.

1 INTRODUCTION

The paper investigates whether code-fluent foundation models can support conversational software-development assistance, beyond isolated autocompletion. It introduces the Programmer’s Assistant to study conversational capabilities and users’ views of this interaction style.

  • Code-fluent models support software tasks including code translation, documentation, unit-test generation, and code completion.
  • Autocompletion becomes insufficient when developers need help choosing functions, libraries, algorithms, or answers to programming questions.
  • The paper asks whether code-fluent foundation models can support a conversational agent for software development.
  • The Programmer’s Assistant explores conversational interaction and whether users find conversational programming assistance desirable and useful.
  • The proposed assistant could support follow-up questions, clarification, general programming discussion, and programming-skill improvement alongside code generation.
  • The paper contributes evidence that conversational assistance provides varied support and compares conversation, direct manipulation, and search as complementary interaction models.

2 RELATED WORK

Prior work established automated and conversational support for software development, but existing tools often lacked natural-language or conversational context. This paper builds on that work by evaluating context-aware conversation with a code-fluent LLM.

  • The Programmer’s Apprentice: The Programmer’s Apprentice envisioned an intelligent software-engineering assistant, but its prototype used a stylized command language rather than natural-language interaction.
  • Code-fluent foundation models: Code-fluent foundation models generate code, translate programs, create unit tests, and generate documentation, while their probabilistic outputs can contain errors.
  • Code-fluent foundation models: Studies of Copilot report increased feelings of productivity, a 27% acceptance rate for proposed completions, and mixed effects on task performance.
  • Code-fluent foundation models: Copilot interactions differ between accelerating familiar tasks and exploring uncertain solutions, with exploration involving iterative but context-free prompting.
  • Conversational interaction: This paper instead commits to context-aware conversational interaction using conversational history and the current utterance to generate responses.
  • Conversational interaction: Existing chatbot approaches include open-domain systems, rigid task-oriented dialogue trees, and programming assistants that do not maintain conversational context.

3 THE PROGRAMMER’S ASSISTANT

The Programmer’s Assistant is a lightweight coding environment that combines a code editor with a conversational interface to a code-fluent LLM. It maintains conversational and code-selection context, while adding interface features and safeguards to manage alternate responses, long code, uncertainty, and model limitations.

  • System design: The prototype combines a code editor with a chat interface powered by OpenAI’s Codex model.It was implemented as a lightweight coding environment for exploring conversational interaction with a code-fluent LLM.
  • Interaction design: The assistant responds on demand rather than monitoring work in progress or offering unsolicited suggestions.This interaction model was intended to provide help when requested without distracting or interfering with the user’s flow.
  • System design: Newly selected code is appended to the user’s message, while prior conversational context is retained for the model.The client constructs prompts from the initial prompt, chat transcript, and user message; newly selected code is added when applicable.
  • Interface behavior: Short code snippets appear inline, whereas responses longer than 10 lines open in a pop-up that can be revisited from the transcript.The assistant does not directly modify source code; users transfer desired code through copy/paste.
  • Model prompting: The prompt establishes a polite, helpful, humble, and non-authoritative programming assistant that communicates uncertainty about possible errors.The system uses conversational examples, code delimiters, and instructions intended to discourage users from accepting erroneous results uncritically.
  • Handling model limitations: Pilot testing exposed repeated responses and fixation on earlier conversation, alongside incorrect, incomplete, irrelevant, or insubstantial outputs.The interface included “try again,” and the system could reset conversational context while preserving the transcript in the UI.

4 EMPIRICAL STUDY OF CONVERSATIONAL PROGRAMMING ASSISTANCE

The authors conducted a qualitative empirical study of conversational programming assistance with participants varying in programming experience. They deployed the prototype as a playground with tutorials and four programming challenges, collecting surveys, event logs, and conversation logs.

  • Study goals: The study examined whether users value conversational interactions grounded in code and how they compare with other programming support such as web search.The inquiry addressed enjoyment, question types, and the experience of conversational interaction with a programming assistant.
  • Study setup: The prototype was deployed within a global technology company, where participants tried sample problems in a playground environment.A tutorial introduced the assistant’s capabilities and interaction patterns before the challenges.
  • Study setup: Four challenges covered writing code, documenting code, and writing unit tests, exposing participants to a broad range of assistant capabilities.The tasks included a queue class, a CSV scatterplot, graph-search documentation, and greatest-common-divisor tests.
  • Study scope: The study intentionally did not evaluate productivity, solution quality, or completion time because its focus was the utility of conversational interaction.Participants were not required to use the assistant to complete the challenges.
  • Participants: Data collection stopped after 42 participants when periodic reviews indicated that no new insights were emerging.Participants were selected to capture varied programming experience and balanced gender representation.
  • Data collection: Data collection combined pre-study, pre-task, and post-task surveys with timestamped event logs and extracted conversation transcripts.Logs captured conversational exchanges, assistant visibility, “try again,” “start over,” and copy/paste activity.

5 RESULTS

Across 42 participants, the Programmer’s Assistant supported sustained conversational programming, and participants generally found its capabilities, responses, and code-grounded interaction valuable despite initial skepticism. Participants used it for varied assistance, including code generation, follow-up questions, explanations, learning, and work-process support, while also reporting response variability and trust concerns.

  • Capabilities and value: Participants valued code-grounded interaction for reducing context switching, supporting learning, and providing assistance beyond code generation.They described the assistant as useful for remembering language and library details, reviewing answers, and exploring emergent behaviors unavailable from autocomplete-focused tools.
  • All participants engaged with the Programmer’s Assistant during the programming challenges.
  • Post-task UX ratings exceeded pre-task expectations, increasing from 3.0 to 3.6 out of 4; 32 participants had higher post-task ratings.The paired difference was significant, t(41) = 5.94, p < .001, Cohen’s d = 0.92.
  • Conversational interaction: 42% of 910 task-oriented utterances required historical conversational context, and 83% rated follow-up questions as important.
  • Participant experience: Less experienced participants tended to ask basic programming questions, whereas more experienced participants requested help with specific libraries or programming details.

6 DISCUSSION

The discussion argues that conversational interaction grounded in both conversational history and the artifact under development adds value to programming assistance and co-creative work. The exploratory study also motivates future research on measurable productivity, longitudinal human-AI learning, personalization, and interaction designs that preserve human control.

  • The study provides empirical evidence that conversational programming assistance can be supported and valued in software development.
  • Practical value: Participants viewed the assistant as useful and wanted to use it in their own work, especially for smaller or narrowly scoped tasks requiring varied assistance.They valued its conversational, code-grounded interaction and its support for learning programming topics.
  • Interaction models: Conversation offered value beyond direct manipulation and search by embedding interaction in the UI and surfacing emergent model behaviors.The discussion does not frame these interaction models as universally competing; direct manipulation remains appropriate when a model reliably produces a known result.
  • Conversational interaction: Conversational history and the artifact under development provide additional value to co-creative processes.Participants used historical context and code selections, and many said the tool would be less valuable without conversational interaction.
  • Contribution and scope: The study begins addressing limited research on interaction design for co-creative programming systems.Its focus is co-creative programming rather than casual or experimental activities.
  • Open questions: Further work is needed to assess whether this assistance produces measurable productivity improvements.
  • Human-AI synergy: The discussion proposes reciprocal human-AI learning, in which people improve programming skills while model adaptation could use users’ edits as feedback.Longitudinal interaction may support reciprocal representations of users and assistants, enabling more personalized explanations for differing skill levels.
  • Design implications: Human-centered systems should preserve the human skills they depend on while developing complementary skills for both humans and AI.The assistant’s non-authoritative, on-demand help leaves the user in control and responsible for the work product.

7 CONCLUSION

The Programmer’s Assistant was a conversational code-fluent LLM prototype evaluated with 42 participants. Participants reported broad, high-quality assistance and a positive experience despite imperfect outputs, supporting conversational interaction for co-creative software development.

  • 7 CONCLUSION: The Programmer’s Assistant combined a code editor with a conversational agent grounded in conversational history and users’ source-code context.Users could ask follow-up questions referencing prior utterances and responses.
  • 7 CONCLUSION: 42 participants with varied programming skill evaluated the system through quantitative and qualitative feedback alongside system usage.
  • 7 CONCLUSION: Participants reported high-quality assistance for generating and explaining code, answering programming and general-knowledge questions, and supporting productivity.
  • 7 CONCLUSION: Participants contrasted the conversational interaction with tools based on direct manipulation or search.
  • 7 CONCLUSION: The study found that conversational interaction enabled emergent behaviors in a co-creative context and supported extended assistance beyond code generation.
  • 7 CONCLUSION: The assistant did not always generate perfect code or correct answers, yet participants had an overall positive experience across programming challenges.

A.1 Screening Survey

The screening survey collected participants’ roles, Python experience, recent Python use, and familiarity with GitHub Copilot to characterize the study sample.

  • A.1 Screening Survey: The survey asked participants to identify their primary job role.
  • A.1 Screening Survey: Python experience was categorized as no familiarity, less than 1 year, 1–3 years, or 3+ years.
  • A.1 Screening Survey: Participants reported when they had most recently written Python code, ranging from within the past month to more than five years ago.
  • A.1 Screening Survey: The survey measured familiarity with Copilot from no familiarity through seeing demonstrations, trying it, or using it for work.

A.2 Pre-task Survey

The pre-task survey introduced the Programmer’s Assistant and measured participants’ expectations for a conversational programming assistant before use.

  • A.2 Pre-task Survey: The Programmer’s Assistant was introduced as a code editor integrated with a chatbot that answers questions, generates code, and consults on existing code.
  • A.2 Pre-task Survey: Participants were told to complete programming tasks while the study focused on how the assistant helped rather than evaluating their programming skills.
  • A.2 Pre-task Survey: The study materials warned participants not to include proprietary code or company-confidential information because data would be processed by a third-party AI model.
  • A.2 Pre-task Survey: The survey asked participants to anticipate whether the chatbot would be easy to use, understand requests, provide high-quality responses, and improve coding.
  • A.2 Pre-task Survey: Participants also rated expectations that the assistant would help them write code more quickly and be enjoyable to use.

A.3 Post-task Survey

The post-task survey gathered participants’ feedback after they used the Programmer’s Assistant to complete programming challenges.

  • A.3 Post-task Survey: Participants completed a post-task survey after using the Programmer’s Assistant on programming challenges.
  • A.3 Post-task Survey: The post-task survey took approximately 10–15 minutes to complete.

A.3.1 Reflections.

The reflection survey asked participants to assess their experience with the Programmer’s Assistant, including usability, response quality, coding support, and enjoyment. It also examined the value of conversational continuity, code selections, alternate responses, and restarting conversations.

  • Experience ratings: Participants were asked to rate whether the Programmer’s Assistant was easy to use, understood requests, and provided high-quality responses.The response scale ranged from “Not at all” to “A great deal.”
  • Experience ratings: The survey assessed whether the assistant helped participants write better code and write code more quickly.These items used the same four-point response scale.
  • Interaction features: Participants rated the importance of follow-up questions, code-selection references, alternate responses, and clearing conversational context.These features corresponded to multi-turn continuity, editor grounding, “Try Again,” and “Start Over.”
  • Comparisons and impressions: The reflection survey also asked what stood out about the experience and how the assistant compared with web search and GitHub Copilot.The comparison questions asked participants to characterize the experience and compare it with existing tools.

A.3.2 Value.

The value section examined how participants valued adding the Programmer’s Assistant to a preferred development environment and which changes might increase its usefulness. It also documented the tutorial’s conversational exercises and interaction boundaries.

  • Value assessment: Participants rated how valuable the Programmer’s Assistant would be if integrated into their favorite development environment.The scale ranged from “Less valuable” to “More valuable.”
  • Potential improvements: Suggested changes included making the assistant behave more like a search box and adding buttons for common queries.Examples included asking what code does or requesting code documentation.
  • Potential improvements: Participants were also asked about proactive suggestions for code improvements in chat or as comments inserted directly into code.The questionnaire separated suggestions in chat from suggestions placed in code comments.
  • Potential improvements: The study invited additional suggestions for improving the experience of working with the Programmer’s Assistant.The questionnaire included an open-ended item for further recommendations.
  • Tutorial procedure: The tutorial instructed participants to record results in pre-loaded editor text and submit their responses after completing the exercises.The tutorial was presented as text in the code editor.

C CHALLENGES

The study presented programming challenges as editor text for participants to complete and submit. The prototype supported conversational interaction with code but did not run or debug programs, while its Codex prompt preserved conversation history within a token limit.

  • Study procedure: Each challenge was presented as text in the code editor, and participants submitted their completed work after making their best attempt.The prototype did not provide the ability to run or debug code.
  • Challenge tasks: The challenges covered program generation, documentation, and testing.Examples included writing a Python Queue class, generating a plot, documenting a function, and creating a unit test.
  • Challenge tasks: The program-generation tasks specified concrete Python outputs, including a Queue class and a scatter plot saved from shampoo.csv.The plotting task specified dimensions, axes, date formatting, and a title.
  • Challenge tasks: The testing challenge asked participants to create a unit test for the Euclidean algorithm function.The function repeatedly replaces (a, b) with (b, a % b) until b equals zero.
  • Assistant implementation: On subsequent exchanges, the system augmented the initial Codex prompt with the transcript of user requests and assistant responses.When the token threshold was exceeded, it deleted the earliest request-response pairs to leave room for a response.
  • Assistant implementation: The prompt configured Socrates as a helpful software-engineering assistant that answered questions and wrote code, with generated code enclosed in language-marked delimiters.It permitted clarification questions but instructed the assistant not to assign work or quiz the user.

E STUDY PARTICIPANT DEMOGRAPHICS

The study included 42 participants with varied gender identities, Python familiarity, and recency of Python use. Most participants had recent Python experience, while the sample also included participants with limited or no familiarity.

  • Python familiarity: 17 participants (~40%) had 3+ years of Python experience, 11 (26%) had 1–3 years, 11 (26%) had less than 1 year, and 3 (7%) were unfamiliar with Python.The table caption defines the familiarity categories NF, < 1, 1–3, and 3+.
  • Recency of Python use: 29 participants (69%) had written Python within the past month, while 4 (~9%) had not written it within the past 5 years.The remaining participants had written Python within the past year or past 5 years.
Loading 2302.07080v1…