PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice

Ranim Khojah, Mazen Mohamad, Philipp Leitner, Francisco Gomes de Oliveira Neto

arXiv:2404.14901v2cs.SEcs.AIcs.CLcs.HCcs.LG

TL;DR

Empirical evidence about ChatGPT’s practical usefulness for software engineers in industry is limited, amid concerns about inaccurate outputs. The authors observed 24 engineers using it for a week and analysed their chats and survey responses. Engineers most often sought guidance, and the proposed framework links interaction purpose and other factors to usefulness and trust.

  • Problem

    There is limited empirical evidence about the practical usefulness of LLM-based tools such as ChatGPT for engineers in industry, alongside concerns that LLMs can generate inaccurate or nonsensical text.

    Source: §I · p. 1

  • Method

    The authors studied 24 engineers’ ChatGPT use over one week, analysing their dialogues and exit-survey reflections qualitatively.

    Source: A. Participants and data collection · p. 3, B. Data analysis · p. 4

  • Results

    Approximately 75% of participants found ChatGPT helpful to learn, and 50% to reduce repetitive tasks; expert consultation was the most common use.

    Source: §I · p. 1

  • Takeaways & Limitations

    The proposed framework links interaction purpose and internal and external factors to software engineers’ perceived usefulness and trust.

    Source: §VI · p. 13

    The study examined only ChatGPT powered by GPT-3.5, and the framework may not capture all other AI developer tools’ user experiences.

    Source: B. Threats to validity · pp. 12–13

Abstract

from arXiv · show

Large Language Models (LLMs) are frequently discussed in academia and the general public as support tools for virtually any use case that relies on the production of text, including software engineering. Currently there is much debate, but little empirical evidence, regarding the practical usefulness of LLM-based tools such as ChatGPT for engineers in industry. We conduct an observational study of 24 professional software engineers who have been using ChatGPT over a period of one week in their jobs, and qualitatively analyse their dialogues with the chatbot as well as their overall experience (as captured by an exit survey). We find that, rather than expecting ChatGPT to generate ready-to-use software artifacts (e.g., code), practitioners more often use ChatGPT to receive guidance on how to solve their tasks or learn about a topic in more abstract terms. We also propose a theoretical framework for how (i) purpose of the interaction, (ii) internal factors (e.g., the user's personality), and (iii) external factors (e.g., company policy) together shape the experience (in terms of perceived usefulness and trust). We envision that our framework can be used by future research to further the academic discussion on LLM usage by software engineering practitioners, and to serve as a reference point for the design of future empirical LLM research in this domain.

I. INTRODUCTION

The study examines how software engineers use ChatGPT and what shapes its perceived usefulness and trust, amid concerns about its practical reliability. It finds that engineers most often seek guidance, while prompt context, user characteristics, and external factors shape their experience.

I. INTRODUCTION

  • The study addresses practical questions about ChatGPT’s usefulness for software engineering amid concerns that it can produce plausible but inaccurate or nonsensical text.The authors observed 24 professional software engineers at 10 companies using GPT-3.5 for one week, then analysed chat protocols and exit surveys.Source: §I · p. 1
  • Expert consultation was the most common use: engineers sought task guidance to implement themselves, alongside artifact manipulation and learning.Uses spanned mostly implementation, but also planning, design, and testing.Source: §I · p. 1
  • Prompt phrasing and context, participant biases, company policies, and legal aspects influenced satisfaction with results and trust.The paper proposes a framework of purpose, internal factors, and external factors shaping perceived usefulness and trust.Source: §I · pp. 1–2
  • Approximately 75% of participants found ChatGPT helpful for learning, and 50% found it helpful for reducing repetitive tasks.Some participants also reported a lack of trust and thoroughly double-checked suggestions; missing sources or references were a challenge.Source: §I · pp. 1–2
  • The paper contributes a three-purpose usage model, a framework for assessing personal experience, a re-analysis package, and implications for practice.The package includes quantitative dialogue data and exit-survey insights, including trust levels.Source: §I · p. 2

II. RELATED WORK

Prior work studied specific chatbot applications and developer populations, leaving limited evidence about chatbots in engineers’ core software tasks. The authors conducted an observational study using chat logs and an exit survey from industrial practitioners.

II. RELATED WORK

  • Prior studies often focused on specific activities, while evidence about chatbot support for core engineering tasks such as requirements, testing, and coding remained limited.This study covers software-development lifecycle stages and industrial practitioners, unlike prior Co-Pilot interaction research centered on students.Source: §II · p. 2

III. METHODOLOGY

  • The observational study followed a pilot that refined participant-facing materials, then collected ChatGPT use and exit-survey responses during normal work.Participants used GPT-3.5 over five business days; the survey addressed experience, trust, effectiveness, efficiency, and lessons learned.Source: §III · p. 2, A. Participants and data collection · pp. 2–3

A. Participants and data collection

  • Of 25 participants who started, 24 completed the survey and shared data; researchers collected 130 chats and classified 180 software-engineering dialogues.The chats were separated into dialogues based on consecutive prompts on a similar topic or track; unrelated and casual conversations were excluded.Source: A. Participants and data collection · p. 3

B. Data analysis

The authors combined quantitative checks with dialogue classification and interpretative analysis of chats and survey responses. Their framework places interaction purpose alongside internal and external factors in shaping perceived usefulness and trust.

B. Data analysis

  • Quantitative analysis summarized closed survey responses and dialogue volume over time, while qualitative analysis examined dialogue flow and user experience rather than answer accuracy.The quantitative checks were used to identify inconsistencies and outliers.Source: B. Data analysis · p. 3
  • The authors adapted Deliberation, Expert Consultation, and Didactic dialogue types, then labeled them Artifact Manipulation, Expert Consultation, and Training for software-engineering interactions.Three authors developed a classification decision tree, and the first two authors classified all dialogues together.Source: B. Data analysis · p. 3
  • Interpretative phenomenological analysis linked each participant’s chats with survey reflections to examine their experience and how interactions evolved.The analysis considered overall experience and helpful and difficult scenarios, including frustration inferred from negative feedback to ChatGPT.Source: B. Data analysis · p. 4

IV. FINDINGS

  • The framework identifies three purposes, three internal factors, and three external factors as shaping perceived usefulness and trust.Internal factors include prompt phrasing and context, personality, and expectations; external factors include legal aspects, model knowledge, and company policies.Source: §IV · p. 4

A. Purpose

Participants used ChatGPT for artifact manipulation, problem-specific expert consultation, and learning; expert consultation was most common, while interactions covered all software-development lifecycle phases. Most participants sought conceptual guidance rather than concrete solutions.

A. Purpose

  • The classification distinguishes goal-oriented artifact manipulation from information seeking, which includes expert consultation and training.The decision tree separates practical problem-solving and guidance from task-independent learning based on whether understanding develops.Source: A. Purpose · p. 5, Figure 2 · p. 5
  • Expert consultation accounted for 62% of dialogues and was used at least once by all but two participants.Training was less common; five of 24 participants used ChatGPT for task-independent learning.Source: A. Purpose · p. 5
  • Training dialogues tended to be longer and showed less variation in length than dialogues focused on specific work tasks.The paper links this pattern to follow-up questions and differences in how participants ended conversations.Source: A. Purpose · pp. 5–6
  • ChatGPT was used in all software-development lifecycle phases, with 104 dialogues (about 58%) during implementation.About 20% of dialogues were not clearly associated with a lifecycle phase.Source: A. Purpose · p. 6
  • Most participants sought conceptual guidance rather than concrete solutions, while using ChatGPT for artifact manipulation, expert consultation, and training.Participants often varied their purposes during the study week.Source: A. Purpose · pp. 4–6

1) Artifact Manipulation:

Artifact manipulation covered about one third of interactions, spanning creation and modification of artifacts as well as brainstorming and side tasks. It ranged from simple outputs and inspiration to attempts involving complex constraints.

1) Artifact Manipulation:

  • Artifact manipulation represented about one third of interactions and included artifact generation, modification, brainstorming, and side tasks.The four types vary in the creativity and originality expected of ChatGPT’s responses.Source: 1) Artifact Manipulation · p. 6
  • Artifact manipulation dialogues were usually short, and feedback during them was usually negative when generated solutions had problems.The authors suggest users either got an expected result quickly or gave up; some persisted when trying to obtain a source for an answer.Source: 1) Artifact Manipulation · p. 6
  • Participants used artifact manipulation to create new artifacts from specifications or to improve, refactor, or fix an existing artifact.Bug-fixing prompts often included an error message, while refactoring prompts included a requirement such as using particular libraries.Source: 1) Artifact Manipulation · p. 6
  • Artifact modification requires providing the existing artifact, which can be problematic when company policy prohibits sharing production code with third parties.Artifact generation instead requires specifications of what to create.Source: 1) Artifact Manipulation · p. 6
  • Artifact manipulation requests depended less on extensive additional context than generating correct, working code from scratch often requires.The authors suggest this may leave room for more integrated generative AI tools to support artifact generation.Source: 1) Artifact Manipulation · p. 6
  • Participants also used ChatGPT for brainstorming concrete user stories and for side tasks such as file organization and refining documentation wording.The broader range included fixing existing code and inspiration-focused dialogues, not only pure code generation.Source: 1) Artifact Manipulation · pp. 6–7

2) Expert Consultation:

Expert consultation was the most common purpose: participants sought problem-specific guidance or information rather than a ready-to-use artifact. They used ChatGPT as a potentially more productive alternative to colleagues or internet searches.

2) Expert Consultation:

  • Expert consultation was the most common purpose, accounting for 62% of interactions.Participants used ChatGPT for instructions, advice, or detailed information to assist with work tasks.Source: 2) Expert Consultation · p. 7
  • Users often asked “how to” questions to get guidance toward a solution rather than a ready-to-use artifact.Problem-solving prompts sought implementation instructions, while information retrieval included commands, API syntax, and basic definitions.Source: 2) Expert Consultation · p. 7
  • The purpose included problem-solving, information retrieval, and decision-making, with many users asking ChatGPT to select the most suitable recommendation for their context.Some users instead preferred receiving multiple suggestions.Source: 2) Expert Consultation · p. 7
  • Expert consultation served as a potentially more productive alternative to asking a colleague or searching the Internet.The paper says these uses did not let participants solve problems they otherwise could not solve, but could make the solution process more productive.Source: 2) Expert Consultation · p. 8

3) Training:

Training interactions are relatively infrequent but often longer, with participants using ChatGPT to build knowledge through follow-up questions. They learn by moving from broad questions to detail or by exploring examples, and report a more consistent sense of learning something useful than users seeking expert consultation.

3) Training:

  • Training interactions make up approximately 6% of conversations, are typically longer, and include follow-ups to clarify or expand learning.Participants may confirm comprehension, request explanations or examples, or broaden the topic.Source: 3) Training · p. 8
  • Participants use drill-down learning to move from general prompts or abstract information toward more detailed explanations.The approach starts broadly and then seeks additional details about ChatGPT’s responses.Source: 3) Training · p. 8
  • With learning by example, participants use concrete cases to generalize concepts and explore related topics rather than delve deeply into definitions.The software-roadmap exchange moves from an artifact to questions about its metrics, goals, vision, features, and release schedule.Source: 3) Training · p. 8
  • Participants engaging in training reported a more consistent perception of learning something useful than those using ChatGPT as an expert consultant.Training dialogues progress from broad questions to increasingly specific ones and occur across software-development lifecycle stages, with fewer implementation-related questions than other purposes.Source: 3) Training · p. 8

B. Internal Factors

Internal factors, including how users prompt ChatGPT, shape perceived outcome quality alongside interaction purpose. In particular, project context can matter more to users’ experience than whether they phrase prompts as keywords or grammatical sentences, while company policy can restrict the context they provide.

B. Internal Factors

  • Internal factors such as user personality and prompting, together with interaction purpose, influence the perceived quality of outcomes.The study treats these as aspects of the chatbot user.Source: B. Internal Factors · p. 9

1) Prompts:

  • Prompt wording style was not associated with a difference in perceived usefulness, although grammatically correct sentences yielded more accurate answers than keyword prompts.Participants compared some ChatGPT interactions to search-engine queries.Source: 1) Prompts · p. 9
  • Project-specific context had more influence on participants’ overall experience than prompt wording style, but users sometimes withheld it over privacy or company-policy concerns.Context included requirements, code, domain constraints, and process structure; some users then provided insufficient purpose for technical questions.Source: 1) Prompts · p. 9
  • The purpose of a dialogue shaped how much context users supplied; modifying an artifact often improved perceived usefulness because the artifact was provided.Artifact-generation prompts rarely included sufficient requirements or constraints, and users sometimes stopped after ChatGPT’s answer failed to fit the supplied context.Source: 1) Prompts · p. 9
  • Users sometimes struggled to supply sufficient context, and frustration with perceived failures could lead them to end interactions early.Some did not know what context to provide, while others feared sharing it would violate company policy.Source: 1) Prompts · p. 9

2) Personality and expectations:

Participants’ expectations, experience, and attitudes toward AI shaped how they judged ChatGPT, while legal concerns and company rules constrained some uses. Skeptical or inexperienced users could quickly reject answers, and the study could not assess how ChatGPT Enterprise privacy measures might affect experience.

2) Personality and expectations:

  • Optimistic participants who understood ChatGPT’s limitations adapted their use and expressed positive sentiments more often when using it for training.Other relevant internal factors included AI knowledge and tolerance for inaccurate responses.Source: 2) Personality and expectations · pp. 9–10
  • Inexperienced users often tested ChatGPT with known-answer or unrelated questions, while expectations of fault-free answers and concerns about missing sources could fuel skepticism.Some skeptical participants withheld context they considered sensitive; skeptical users were often confirmed in their view that ChatGPT was not useful and stopped quickly.Source: 2) Personality and expectations · pp. 9–10

C. External Factors

  • Legal and ethical concerns about training data and prompt storage, plus concerns about stale knowledge, affected some participants’ experience.A minority wanted knowledge sources newer than 2021.Source: C. External Factors · p. 10
  • Some companies prohibited using generative AI to generate code or requirements, and participants’ uncertainty about permitted uses could prompt caution.The study could not determine whether some participants who shared production code were covered by company policy or unaware of or unwilling to follow it.Source: C. External Factors · p. 10
  • Because the study preceded ChatGPT Enterprise’s announcement, it could not assess how that service’s privacy measures might influence user experience.The authors also could not determine whether some participants’ production-code sharing complied with their company policies.Source: C. External Factors · p. 10

D. Personal Experience

Participants found ChatGPT useful for learning and decision-making, but its usefulness varied by task and interaction purpose. Trust was substantial but not complete; missing sources and complex or context-specific tasks were concerns, yet some participants still valued answers as starting points that required checking.

D. Personal Experience

  • Participants reported relatively high usefulness for learning new concepts and making better decisions, but not for initiating team discussions or maintaining focus.Users were split on whether ChatGPT helped reduce repetitive tasks.Source: D. Personal Experience · p. 10
  • All participants using ChatGPT for training found it helpful for learning new concepts and staying focused, while expert-consultation users found it helpful for reducing repetitive tasks but not focus.Expert-consultation users who explained or interpreted artifacts found the tool more helpful than users pursuing other information-seeking purposes.Source: D. Personal Experience · p. 10
  • Artifact-generation users found ChatGPT helpful for reducing repetitive work involving simple artifacts or blueprints rather than complex artifacts with many constraints.The authors relate this pattern to prior work suggesting that software tasks requiring complex decision-making are more challenging for ChatGPT.Source: D. Personal Experience · p. 10
  • Approximately two thirds of participants reported high or moderate trust in ChatGPT, but none trusted it entirely; lack of sources was a key concern among those with low trust.The figure presents participants’ reported usefulness on the left and trust on the right.Source: D. Personal Experience · p. 11, Figure 5 · p. 11
  • Some participants valued ChatGPT as a starting point even when they expected to fact-check and correct its answers.The study’s findings also report usefulness for learning and decision-making, and for reducing repetitive tasks when those tasks are simple or context-independent.Source: D. Personal Experience · p. 11

V. DISCUSSION

The study’s implications extend beyond code generation: ChatGPT was often used for guidance and training, and participants also used it for creative work and tasks outside implementation. Its usefulness was qualified by trust, checking behavior, and possible productivity costs.

A. Implications

  • Only around a third of dialogues involved artifact manipulation; guidance and training were more common, though integrated development tools may be used differently.The authors suggest such tools may support artifact manipulation more often because they can provide better context as part of the prompt.Source: A. Implications · pp. 11–12
  • Participants used ChatGPT in lifecycle phases beyond implementation, including process improvement and generating requirements or test cases.Source: A. Implications · p. 12
  • Participants with low trust sometimes continued using ChatGPT while carefully checking suggestions; the authors call for more transparency about response confidence.Participants were more comfortable when they could judge how plausible an answer was based on how it was generated.Source: A. Implications · p. 12
  • Participants expressed satisfaction with brainstorming for inspiration and alternatives, although they understood outputs were recombinations rather than truly novel solutions.Source: A. Implications · p. 12
  • ChatGPT could reduce productivity through less team communication and focus, while prompt tweaking sometimes consumed extraordinary time.The authors note that the limited study window may have contributed, as many participants were still experimenting with AI.Source: A. Implications · p. 12

B. Threats to validity

The authors identify limits to generalizing and interpreting the findings, including the sample, data collection, chatbot, and qualitative classifications. They conclude with a framework and research directions, while restricting access to some underlying data for confidentiality reasons.

B. Threats to validity

  • The sample’s limited size and European geography may constrain representativeness, although participating companies varied in size and domain and came from three nonneighboring countries.Source: B. Threats to validity · p. 12
  • Web-archive chat logs could not be altered, mitigating the risk that participants removed parts of their conversations before sharing them.Source: B. Threats to validity · p. 12
  • Using only ChatGPT powered by GPT-3.5 limits transfer to other tools; the authors hypothesize newer models may change usefulness across use cases.They suggest using the framework to study model-version effects, including on complex artifact generation.Source: B. Threats to validity · pp. 12–13
  • Dialogue classification involved qualitative subjectivity, and the purpose categories and Likert responses have limits that the authors partly address through refinement and open-ended questions.The schema was refined over three discussion and classification rounds; the authors consider the three levels sufficient for their scope while noting categories can expand.Source: B. Threats to validity · p. 13

VI. CONCLUSION

  • The study proposes a framework linking dialogue purpose and internal and external factors to perceived usefulness and trust, and calls for research on learning, training, and prompt engineering.Source: §VI · p. 13

DATA AVAILABILITY

  • The re-analysis package includes survey materials and classified dialogue and Likert data, but excludes chat logs and open-ended responses that could reveal identities or protected company information.Source: DATA AVAILABILITY · p. 13
Loading 2404.14901v2…