Source-linked AI summary
A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges
Jenny T. Liang, Chenyang Yang, Brad A. Myers
TL;DR
Developers often do not accept AI programming assistants’ initial suggestions, leaving practical usability questions about these tools. This paper surveys 410 developers and combines quantitative and qualitative analyses to identify usage motivations, adoption barriers, and usability challenges. It finds that speed, reduced keystrokes, and syntax recall motivate use, while unmet requirements and difficulty controlling generated output constrain adoption.
Problem
Prior work had not systematically quantified how prevalent AI programming assistants’ usability factors are in practice, despite low acceptance of initial suggestions.
Method
The authors surveyed 410 developers and analyzed responses using quantitative and qualitative techniques.
Results
Developers valued reduced keystrokes, faster task completion, and syntax recall, while unmet functional or non-functional requirements and poor controllability were key reasons for nonuse.
Takeaways & Limitations
The findings support designing AI programming assistants with interactions that reduce cognitive effort and distractions during programming.
Takeaways & Limitations
Survey responses may be affected by memory bias because participants had to recall prior experiences with AI programming assistants.
Abstract
from arXiv · showhide
The software engineering community recently has witnessed widespread deployment of AI programming assistants, such as GitHub Copilot. However, in practice, developers do not accept AI programming assistants' initial suggestions at a high frequency. This leaves a number of open questions related to the usability of these tools. To understand developers' practices while using these tools and the important usability challenges they face, we administered a survey to a large population of developers and received responses from a diverse set of 410 developers. Through a mix of qualitative and quantitative analyses, we found that developers are most motivated to use AI programming assistants because they help developers reduce key-strokes, finish programming tasks quickly, and recall syntax, but resonate less with using them to help brainstorm potential solutions. We also found the most important reasons why developers do not use these tools are because these tools do not output code that addresses certain functional or non-functional requirements and because developers have trouble controlling the tool to generate the desired output. Our findings have implications for both creators and users of AI programming assistants, such as designing minimal cognitive effort interactions with these tools to reduce distractions for users while they are programming.
1 INTRODUCTION
This survey examines how developers use AI programming assistants and which usability challenges shape their adoption. Responses from 410 developers show motivations centered on speed and syntax recall, while unmet requirements and poor controllability limit use.
- Study focus: 410 developers participated in the survey examining practices and usability challenges involving AI programming assistants.Participants were recruited from GitHub repositories related to AI programming assistants.
- Usage characteristics: GitHub Copilot users reported a median of 30.5% of their code being written with help from the tool.
- Usage motivations: Developers primarily use AI programming assistants to reduce keystrokes, complete programming tasks quickly, and recall syntax.Brainstorming potential solutions was a less resonant motivation than these direct productivity and recall benefits.
- Adoption barriers: The most important reasons for not using AI programming assistants were failure to meet functional or non-functional requirements and difficulty controlling generated output.
- Usability challenges: Prominent usability issues involved understanding which inputs caused generated code, incorporating outputted code, and controlling generations.
- Desired improvements: Developers wanted tools to support feedback for correcting or personalizing models and to better understand code context, APIs, and programming languages.
2 RELATED WORK
Prior usability studies identified challenges with expressing intent, understanding generated code, and aligning tools with user expectations. This study extends that work by quantifying how prevalent such challenges are across a broader set of tools and usability dimensions.
- Prior usability findings: Prior studies reported difficulties correcting generated code and mismatches between perceived and actual capabilities of program synthesizers.
- Prior usability findings: Studies of transformer-based assistants found that users struggled to express intent and sometimes needed background knowledge of underlying modules or frameworks.
- Prior usability findings: GitHub Copilot studies described struggles understanding and debugging generated code and distinct exploration and implementation modes of use.
- Design research: Existing work also proposed design principles for inline suggestions, including glanceable suggestions, and examined developer interactions with conversational chatbots.
- Research gap: The study addresses the unclear practical importance of previously enumerated usability challenges by quantifying their prevalence across more tools and both tools and generated code.
3 METHODOLOGY
The authors conducted a Qualtrics survey of developers recruited from AI-assistant-related GitHub repositories and analyzed responses using quantitative frequencies and qualitative coding. The sample was diverse in geography, programming context, languages, and experience.
- Sampling and participants: The survey received 410 responses from 10,530 invited GitHub users, yielding a response rate of around 4%.
- Sampling and participants: Participants represented 57 countries and varied programming contexts, including professional software engineering, end-user development, open-source projects, hobbies, and school.
- Survey design: The authors designed a 15-minute Qualtrics survey covering assistant use, concerns, project-specific experiences, programming backgrounds, and demographics.
- Survey design: The survey was piloted with 11 developers to clarify wording, improve data quality, and identify usability factors that prior literature might have missed.
- Analysis: Quantitative analysis reported frequencies and perceived frequency ratings for importance, occurrence, and concern rather than exact event frequencies.
- Analysis: Qualitative responses were analyzed through multiple rounds of open coding, with shared-codebook development and unanimous resolution of coding decisions.
4 USAGE CHARACTERISTICS
Developers primarily used AI programming assistants for convenient, repetitive, and straightforward coding tasks, while adapting their inputs to improve results. They were motivated most by autocomplete, faster completion, and syntax recall, but avoided tools when outputs were difficult to control or failed requirements.
- Motivations: 86%, 76%, and 68% of participants cited autocomplete, faster task completion, and syntax recall as important usage motivations.
- Motivations: Participants often avoided AI programming assistants because generated code failed functional or non-functional requirements or was difficult to control.These were reported as important reasons by 54% and 48% of participants, respectively.
- Motivations: 38% of participants cited modifying or debugging unhelpful generated code as an important reason for not using these tools.
- Successful use cases: Participants most successfully used AI programming assistants to generate repetitive code and code with simple logic.Repetitive code was the most frequently mentioned successful use case, at 78×, while simple-logic code appeared 68×.
- Successful use cases: Participants also used the tools for short autocompletions, quality assurance, brainstorming, learning, syntax recall, and maintaining code consistency.These uses included generating test cases, exploring multiple implementations, learning languages or libraries, recalling APIs, and applying style conventions.
- User input strategies: Participants improved outputs through clear explanations, following conventions, breaking down instructions, using existing code context, prompt engineering, or no deliberate strategy.Clear explanations were the most frequently reported strategy at 99×, while some participants found no strategy necessary.
5 USABILITY OF AI PROGRAMMING ASSISTANTS
The most frequent usability challenges involved understanding how inputs shaped outputs, abandoning generated code, and controlling model generations. Participants generally evaluated and modified outputs successfully, while frequently giving up when code failed its intended action or requirements.
- Usability challenges: 30%, 28%, and 26% of participants often struggled to identify which input influenced the output, gave up on generated code, or controlled the model.
- Understanding output: Only 5.6% of participants frequently struggled to understand generated code, making it the least frequently reported usability issue.
- Understanding output: Unfamiliar APIs, excessive length, and numerous control structures were the most frequently reported reasons for difficulty understanding outputted code.These reasons were reported often by 25%, 23%, and 19% of participants, respectively.
- Evaluating output: Participants most often evaluated generated code through visual inspection, static analysis, execution, and detailed logic examination, while API documentation was consulted less frequently.The corresponding frequencies were 74%, 71%, 69%, 64%, and 38%.
- Modifying output: 63% of participants regularly succeeded in modifying generated code, usually by changing the code itself rather than the input context.Changing generated code was reported by 62%, compared with 40% for changing input context.
- Giving up: Participants frequently gave up when generated code failed the intended action or did not meet functional or non-functional requirements.These reasons were frequently encountered by 43% and 34% of participants, respectively.
6 ADDITIONAL FEEDBACK
Participants’ strongest concerns centered on intellectual property and access to their code. The survey framed these concerns alongside broader usability issues related to AI programming assistants.
- General concerns: 46% of participants were concerned that AI programming assistants could produce code infringing intellectual property, while 41% were concerned about tools accessing their code.
C. Methods of evaluating code output
Participants evaluated generated code using methods that ranged from quick inspection to execution and deeper logic analysis, with API documentation consulted less often.
- Evaluation methods: 38% of participants often consulted API documentation when evaluating generated code.
D. Methods of modifying code output
Participants reported modifying generated code in several ways rather than relying only on unchanged output, including direct edits and regeneration after changing surrounding code or comments.
- Participants reported incorporating generated code by changing it directly.
- 44% and 24% were reported for the survey item on using generated code as-is.
- Participants also reported changing the code or comments around a suggestion before regenerating.
E. Reasons for giving up on code output
Participants gave up on generated code for several reasons, especially mismatched actions or requirements, stylistic and technical problems, and difficulty understanding or controlling the tool.
- 43% and 22% were reported for generated code not performing the desired action.
- Generated code failed to meet functional or non-functional requirements, including security and performance needs.
- Participants also encountered mismatched project style, defects, unwanted APIs, unfamiliar APIs, excessive complexity, and insufficient understanding.
- Requested improvements: Participants requested user feedback, natural-language interaction, code analysis, explanations, more suggestions, better context, and configurable generation behavior.
- Requested improvements: Participants wanted assistants to understand workspace context and support adjustable suggestion behavior, including distinctions between shorter and longer generations.
- Requested improvements: Requested improvements included source references, multiple candidate snippets, attention to readability and other non-functional requirements, and automatic or personalized feedback.
7 THREATS TO VALIDITY
The study’s validity is constrained by recalled and subjective survey responses, possible question misunderstanding, recruitment-related sampling bias, and changes in AI assistants over time.
- Recall-based survey questions may introduce memory bias, although participants were asked to ground responses in a specific project.
- Participants may have misunderstood survey wording; the researchers piloted the survey 11 times to improve question clarity.
- Recruitment from AI-assistant-related GitHub projects may overrepresent enthusiasts and underrepresent people uninterested in these tools.
- Because the survey was deployed in January 2023, some findings may not apply to future tool versions.
- Subjective frequency estimates may be inaccurate, making in-situ data more appropriate for evaluating event frequency.
- The study aimed for 385 responses to achieve a 95% confidence level with a 5% margin of error.
- The authors advise future work not to reuse their GitHub recruitment strategy and recommend Prolific as a more sustainable alternative.
8 DISCUSSION & FUTURE WORK
The discussion emphasizes that developers primarily use assistants for acceleration, while future work should improve alignment, control, context, and low-distraction interaction.
- Usage modes: The results support two usage modes: acceleration when developers know what to write and exploration when they seek possible options.
- Usage modes: Acceleration-related situations were more frequent than exploration-related situations, including repetitive code at 78× versus proof-of-concepts at 20×.
- Interaction: Six percent of participants explicitly used ChatGPT as an AI programming assistant, and participants frequently requested natural-language interaction.
- Learning: Participants often favored assistants over documentation, StackOverflow, or blogs for recalling and learning API and programming-language syntax.
- Alignment: Improving alignment requires addressing code that performs the wrong action and giving developers more control over generated outputs.
- Alignment: Participants identified non-functional requirements such as readability and performance as important targets for future generation methods.
- Practitioner implications: Practitioners could improve output by giving clear explanations, adding code, and following programming conventions, while also using assistants for testing and learning.
- Future work: Future interaction techniques should support acceleration mode with minimal cognitive effort so they do not distract developers from their tasks.
9 CONCLUSION
The study surveyed 410 developers to examine AI programming-assistant usage practices and usability challenges. Findings identify key motivations for use and directions for improving control over tool output.
- 410 developers were surveyed to understand usage practices and important usability challenges involving AI programming assistants.
- Developers were most motivated by autocompletion, quickly finishing programming tasks, and syntax recall rather than brainstorming solutions.
- The findings suggest designing interaction techniques that give developers more control over the tool’s output.