Source-linked AI summary
Grounded Copilot: How Programmers Interact with Code-Generating Models
Shraddha Barke, Michael B. James, Nadia Polikarpova
TL;DR
The paper addresses limited evidence about how programmers interact with AI programming assistants and studies this question through grounded-theory analysis of Copilot use. Observing 20 participants across diverse programming tasks, it finds bimodal interaction: acceleration when programmers know what to do next and exploration when they do not. The theory motivates design recommendations tailored to these modes.
Problem
Research on programmers’ assistance needs, communication, validation, and error handling with Copilot is scarce, despite the need for evidence to inform assistant design.
Method
The authors use grounded theory to analyze 20 participants completing programming tasks across four languages, iterating between coding interactions, refining theory, and adjusting tasks.
Results
Copilot interactions are bimodal: acceleration helps programmers who know what to do next move faster, while exploration helps uncertain programmers investigate options through deliberate prompting and validation.
Takeaways & Limitations
Future assistants should preserve flow with short, high-confidence suggestions in acceleration mode and support comparison and validation during exploration.
Takeaways & Limitations
The study uses researcher-designed, time-pressured tasks focused primarily on code authorship rather than refactoring, testing, debugging, or other software-engineering activities.
Abstract
from arXiv · showhide
Powered by recent advances in code-generating models, AI assistants like Github Copilot promise to change the face of programming forever. But what is this new face of programming? We present the first grounded theory analysis of how programmers interact with Copilot, based on observing 20 participants--with a range of prior experience using the assistant--as they solve diverse programming tasks across four languages. Our main finding is that interactions with programming assistants are bimodal: in acceleration mode, the programmer knows what to do next and uses Copilot to get there faster; in exploration mode, the programmer is unsure how to proceed and uses Copilot to explore their options. Based on our theory, we provide recommendations for improving the usability of future AI programming assistants.
1 INTRODUCTION
The paper investigates how programmers interact with Copilot, addressing limited usability evidence through a grounded-theory study. It identifies two interaction modes—acceleration and exploration—and derives design recommendations for future assistants.
- Research on programmers’ assistance needs, communication, validation, and error handling remains scarce, motivating broader study of Copilot interactions.
- The authors use grounded theory to build a data-driven account through interleaved coding, data collection, and theory refinement.
- The study observes 20 participants completing programming tasks across Python, Rust, Haskell, and Java, including contributions to existing codebases.
- Copilot interactions are bimodal: acceleration helps programmers who know their next step move faster, while exploration supports uncertain programmers investigating options.
- The authors quantitatively compare prompting and validation across modes and examine factors influencing mode prevalence, while livestream data provides additional consistency checks.
- Design recommendations include short, high-confidence suggestions for acceleration and improved comparison, testing, or live-programming affordances for exploration.
2 COPILOT-ASSISTED PROGRAMMING, BY EXAMPLE
Copilot supports both unobtrusive completion when programmers know what to write and deliberate exploration when they do not. The examples show how prompting, alternative suggestions, and validation differ across these modes.
- Copilot is integrated into development environments and powered by the OpenAI Codex family of models.
- 2.1 Copilot as Intelligent Auto-Completion: Copilot provides context-sensitive end-of-line suggestions at the cursor without explicit invocation, which programmers can accept with <tab>.
- 2.1 Copilot as Intelligent Auto-Completion: In acceleration mode, short suggestions closely matching expectations can be rapidly accepted or rejected without disrupting flow.
- 2.2 Copilot as an Exploration Tool: In exploration mode, programmers explicitly prompt Copilot and open a pane displaying up to 10 unique alternatives.
- 2.2 Copilot as an Exploration Tool: Exploration involves inspecting alternatives, selecting suitable code, and validating it by running the result or otherwise checking confidence.
3 METHOD
The authors develop their theory through an iterative qualitative study of 20 participants performing four programming tasks. Tasks and analysis evolve together to investigate acceleration, exploration, and validation.
- Participants: The study includes 20 participants from academia and industry, with varied prior Copilot and programming-language experience.
- Tasks: One participant did not have time to attempt the Python section.
- Study procedure: Sessions combine programming tasks, think-aloud interaction, interviews, recordings, and transcription for analysis.
- Grounded Theory Process: Grounded theory interleaves data collection and analysis to generate hypotheses and refine an overarching theory without fixed a priori hypotheses.
- Grounded Theory Process: The researchers identified emerging modes, created later tasks through theoretical sampling, and linked codes and notes into the final bimodal theory.
- Tasks: The four tasks included existing-codebase work, Benford’s Law, Fibonacci plotting, and string manipulation, targeting different aspects of interaction.
4 THEORY
The theory identifies two modes of Copilot interaction: acceleration, where programmers use Copilot to execute planned actions, and exploration, where they use it to plan and investigate alternatives. Acceleration depends on well-defined microtasks and preserves flow, while exploration involves more deliberate prompting, comparison, and validation.
- Acceleration: Acceleration uses Copilot to execute planned code actions within the programmer’s sense of flow.Programmers treat Copilot as an intelligent autocomplete for completing a logical unit of code or comment.
- Exploration: Exploration uses Copilot to help plan code actions, including unfamiliar syntax, API lookup, algorithm discovery, and comparison of alternatives.Unlike acceleration, exploration lets Copilot drive more of the interaction and involves deliberate effort to prompt, inspect, select, and validate suggestions.
- Acceleration: Decomposing a task into well-understood microtasks is the main condition associated with acceleration.Language expertise and prior Copilot use appear less important than understanding the algorithm well enough to break it down.
- Acceleration: Acceleration favors small logical units and short end-of-line completions rather than suggestions exceeding the programmer’s immediate focus.Participants accepted focused function-call or argument completions, while larger suggestions could be reduced to the needed structure or first line.
- Acceleration: Long, multi-line suggestions can break acceleration-mode flow, distract programmers, and contribute to disabling Copilot.Participants often dismissed or actively rejected large suggestions, describing distraction and loss of focus.
- Acceleration: Acceleration-mode validation relies on rapid pattern matching against expected keywords, control structures, names, method order, and overall code shape.Suggestions are quickly rejected when patterns conflict with expectations; nearly correct suggestions may receive obvious repairs, while mismatches can occasionally reveal overlooked corner cases.
5 ADDITIONAL ANALYSIS
The additional analysis quantitatively examined interaction modes, prompting, validation, and task-related factors, then checked the theory against five livestream videos. Exploration occupied more total study time than acceleration, while prompting and validation patterns varied systematically by mode and task.
- Time spent in interaction modes: 248.6 minutes were spent in exploration mode versus 104.7 minutes in acceleration mode across participants.The exploration-to-acceleration ratio varied with language expertise, prior Copilot usage, task, and programming language.
- Factors influencing interaction modes: Professional participants with greater language expertise spent more time in acceleration mode.Prior Copilot users also spent less time exploring and more time accelerating than novices, despite similar total interaction time.
- Factors influencing interaction modes: Unfamiliar APIs increased exploration time, particularly for Chat Client and Benford’s Law, while Advent of Code’s algorithmic difficulty and File I/O also encouraged exploration.Python and Rust showed no noticeable difference in total interaction time or acceleration-to-exploration ratio; Haskell and Java had too few data points for conclusions.
- Prompting strategies across interaction modes: Code prompts dominated acceleration mode at 71.4%, while comment prompts were more common in exploration mode at 57.2% versus 13.1%.Context prompts accounted for 15.2% of acceleration-mode prompting, and the multi-suggestion pane increased in exploration mode.
- Validation strategies: Participants spent more time reading documentation in exploration mode, whereas execution was more prevalent during acceleration.Exploration often involved unfamiliar APIs and incomplete code, while examination accounted for substantial absolute validation time.
- Validation strategies: Validation varied by task: Chat Client favored examination and documentation, while Benford’s Law had the most execution time, exceeding Advent of Code.The authors conjecture that Benford’s Law’s visual output made execution especially easy and engaging to inspect.
- Livestream validation: Five livestream videos showed prompting, validation, and mode-specific behaviors consistent with the grounded theory.The livestream analysis provided additional qualitative support beyond the main 20-participant study.
6 RECOMMENDATIONS
The recommendations focus on giving programmers more control over inputs and producing outputs that better match acceleration and exploration workflows. Proposed improvements include mode-aware suggestions, stronger validation support, clearer alternative exploration, and code structures that preserve authorial control.
- Control over the context: Future tools could customize Copilot’s context through isolated scratchpads and expert prompt syntax.These mechanisms would let programmers emphasize relevant code or constrain suggestions to requested expressions, functions, data structures, or syntax examples.
- Cross-language translation: Cross-language translation could let programmers prompt Copilot in languages they know better than the task language.The authors suggest fine-tuning on equivalence classes of syntactic constructs across programming languages.
- Awareness of the interaction mode: Mode-aware assistants should preserve acceleration flow with short, high-confidence suggestions and support exploration through comparison and automated validation.The recommendations include better affordances for contrasting alternatives, automated testing, and live programming.
- Suggestions with holes: Suggestions with holes could provide control structures while leaving implementation bodies for programmers to complete.This addresses users’ tendency to retain control-flow features and their frustration with large snippets that disrupt top-down design.
- Always-on validation: Always-on validation could pair code-generating models with property-based testing, projection boxes, or other continuously available checks.Participants specifically requested automatic testing and live displays of runtime values.
7 RELATED WORK
The paper positions itself as complementary to prior usability studies of Copilot, other language-to-code tools, programming assistants, and program synthesizers. Its distinctive emphasis is grounded theory applied to open-ended, existing-codebase tasks across multiple languages rather than comparative completion metrics.
- Usability of Copilot: Compared with Vaithilingam et al., this study uses existing-codebase tasks, multiple languages, and grounded theory instead of comparative completion-rate and completion-time evaluation.The authors present the approaches as complementary.
- Usability of Copilot: The paper agrees with prior Copilot findings that generated code can provide a useful starting point, be repaired by programmers, and be harder to debug.It also notes differences in participants’ ability to understand and modify generated code.
- Usability of other LLM tools: Studies of GenLine and code-translation assistants similarly examine developer interaction, rewriting, validation, and interface features such as confidence highlighting and alternative translations.The translation study is complementary because it addresses a different language-to-language task.
- Usability of other LLM tools: A synthesis of programming-assistant studies likewise identifies prompting difficulty, validation importance, and assistant use for boilerplate or reusable code.The paper also situates Copilot among industrial statistical-model assistants such as TabNine and Kite.
- Usability of program synthesis tools: Program-synthesis usability research spans domain-specific tools for API navigation, regular expressions, web scraping, and data querying or wrangling.These studies provide a broader usability context for code-generation systems.
- Usability of program synthesis tools: A study of six synthesizers corroborates that novices often accept and then modify generated code, while this paper links that behavior specifically to exploration mode.The paper claims to support and extend the earlier observation.
- Grounded Theory for software development: Grounded theory has established use in software engineering and programming-languages research, including studies of how statically typed functional programmers write code.This paper applies the methodology to Copilot-assisted programming.
8 LIMITATIONS AND THREATS TO VALIDITY
The study’s conclusions are bounded by its designed tasks, nonrepresentative sample, limited language and task coverage, first-time-user composition, short study period, and rapidly changing code-generating technology.
- Study scope: The study used researcher-designed tasks rather than participants’ own projects and focused on code authorship, not refactoring, testing, debugging, or other software-engineering activities.More familiar codebases and less time pressure could have produced different interactions.
- Sample and coverage: The 20-participant sample skewed toward academia and is not representative of all programmers.Other task types and programming languages could also yield different interactions.
- User experience and duration: Eleven participants had never used Copilot, so they might not represent regular users.The study also could not examine how usage patterns change over a longer period.
- Technological change: Rapid progress in code-generating models could soon render the findings obsolete.This is presented as a threat to the durability of the study’s conclusions.