Source-linked AI summary
Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
Hussein Mozannar, Gagan Bansal, Adam Fourney, Eric Horvitz
TL;DR
AI code-recommendation systems require finer-grained evidence about how programmers interact with suggestions and where that interaction incurs overhead. The paper develops CUPS and applies it to participant-labeled Copilot sessions, finding substantial time spent on Copilot-related activities and verification, while noting that the study used researcher-provided time-limited tasks.
Problem
Existing studies and metrics do not adequately identify programmers’ interaction activities, their costs, or when code-recommendation systems add overhead.
Method
The paper develops CUPS, a taxonomy of programmer activities, and applies it with telemetry to a study of 21 programmers who retrospectively labeled coding sessions.
Results
34.3% of total session time may be spent double-checking and editing Copilot suggestions, while more than half of task time may involve Copilot-related activities.
Takeaways & Limitations
CUPS-based interaction profiling reveals inefficiencies and time costs and motivates new interaction metrics and interface designs.
Takeaways & Limitations
Participants solved researcher-provided, time-limited coding tasks rather than real tasks they might perform in practice.
Abstract
from arXiv · showhide
Code-recommendation systems, such as Copilot and CodeWhisperer, have the potential to improve programmer productivity by suggesting and auto-completing code. However, to fully realize their potential, we must understand how programmers interact with these systems and identify ways to improve that interaction. To seek insights about human-AI collaboration with code recommendations systems, we studied GitHub Copilot, a code-recommendation system used by millions of programmers daily. We developed CUPS, a taxonomy of common programmer activities when interacting with Copilot. Our study of 21 programmers, who completed coding tasks and retrospectively labeled their sessions with CUPS, showed that CUPS can help us understand how programmers interact with code-recommendation systems, revealing inefficiencies and time costs. Our insights reveal how programmers interact with Copilot and motivate new interface designs and metrics.
1 Introduction
Code-recommendation systems introduce interaction work that existing metrics do not fully capture. The paper develops CUPS and a labeled user study to profile these activities, quantify their costs, and identify inefficiencies.
- Motivation: Existing metrics such as acceptance rates, time to accept, typed-token reduction, and task completion time capture only part of programmer interaction.They miss work occurring after acceptance and effort spent devising and refining prompts.
- Motivation: Programmers may accept suggestions for code highlighting or accept sequences before evaluating them together, so verification and editing can occur after acceptance.These behaviors make instantaneous acceptance events insufficient for reconstructing user activity.
- Approach: CUPS is a taxonomy of common programmer activities and serves as the main tool for answering questions about CodeRec interaction.The paper aims for a taxonomy low-level enough for interaction profiling but high-level enough to capture meaningful activities.
- Study: 21 developers retrospectively labeled 3137 coding segments while interacting with 1096 suggestions, confirming that the taxonomy was sufficiently expressive.The study also enabled learning transition weights and state dwell times.
- Analysis: The resulting data support CUPS timelines and diagrams for profiling behavior, identifying inefficiencies, and analyzing time spent across programmer states.The paper also proposes an adjustment formula for time spent verifying suggestions.
- Implications: The paper proposes using current CUPS-state identification to support programmer needs through keyboard macros or automated prediction and to motivate interface improvements.These possibilities are discussed as future work and in relation to inefficiencies in the current Copilot version.
2 Background and Related Work
Prior work shows promising but dispersed evidence about AI-assisted programming, while offline benchmarks and coarse productivity measures provide limited insight into in-situ interaction. This paper addresses that gap with participant-labeled, granular activity states.
- AI-assisted programming: LLM-based code models have been developed to accelerate software engineering, reduce workloads, and improve productivity.Related work includes models such as GPT-3 and Codex and systems trained on large code corpora.
- Evaluation gap: Offline completion benchmarks evaluate models without developer input and may overestimate their utility to developers.The paper distinguishes model performance in offline settings from assistance to programmers in situ.
- Study design: The study uses raw telemetry alongside participant-labeled states, preserving programmers’ stated intents and supporting future predictive models.The authors note that eye-tracking and raw video could provide deeper behavioral insight.
- Research question: The central question is when neural code-completion systems help programmers and when they add additional overhead.This question motivates examining the nature of utility rather than relying only on coarse outcomes.
- Interaction models: Barke et al. characterized Copilot interaction as acceleration or exploration mode, whereas CUPS adds granular labels for programmer intents.CUPS is designed to enrich this two-category account with more detailed activity states.
3 Copilot System Description
Copilot generates code suggestions from the code written so far and lets programmers accept, reject, or browse them. Because interaction telemetry records instantaneous events, the paper models the activities occurring between events to capture additional effort and time costs.
- Copilot uses an LLM to recommend code completions when the programmer pauses typing.Suggestions appear as inline, monochrome popups that programmers can accept with a keyboard shortcut.
- Programmers can engineer the prompt with natural-language comments, then browse, accept, or reject the generated suggestion.Rejection can occur implicitly when typing differs from the suggestion or explicitly through the escape key.
- Copilot telemetry records timestamps, actions, and prompt and suggestion features whenever suggestions are shown, accepted, rejected, or browsed.Each event is represented as (t_i, A_i, P_i, S_i), capturing within-session time, action, prompt features, and suggestion features.
- Telemetry events alone omit activities such as double-checking suggestions and prompt engineering that occur between consecutive instantaneous events.The paper therefore treats the intervals between events as telemetry segments containing user intentions and activities.
- The taxonomy separates segments beginning with a suggestion display from those beginning with an action and ending at the next display event.These groups are called ‘User Before Action’ and ‘User Typing or Paused’, respectively.
4 A Taxonomy for Understanding Programmer-CodeRec Interaction: CUPS
CUPS is a hierarchical taxonomy for labeling programmer activities in telemetry segments during interaction with code-recommendation systems. It combines retrospective video review with telemetry segmentation to represent both ordinary coding and activities specific to LLM-based suggestions.
- Creating the Taxonomy: CUPS was created by developing a labeling tool and refining an initial taxonomy through pilot coding sessions.The tool supports retrospective annotation of recorded telemetry segments with activity labels.
- Creating the Taxonomy: The labeling tool provides segment navigation, looping video review, taxonomy buttons, an IDK option, and a custom-state text box.Keyboard bindings support faster annotation.
- Creating the Taxonomy: Each telemetry segment is labeled through a hierarchy that first distinguishes whether typing occurred, then selects the relevant typing or non-typing state.When multiple states apply under one hierarchy, participants choose the state occupying most of the segment.
- CUPS States: A single state per telemetry segment can obscure cases where programmers perform multiple activities within that segment.Sibling states such as ‘Writing New Functionality’ and ‘Editing Written Code’ may also share many traits.
- Creating the Taxonomy: Pilot refinement added ‘Debugging/Testing Code’, ‘Looking up Documentation’, and ‘Writing Documentation’ to the taxonomy.The code book was stable by the final pilot participant, and custom labels were rarely used in the study.
- CUPS States: Typing or paused activity can include writing or editing code, debugging, documentation, prompt crafting, waiting, or reviewing suggestions.Programmers may also accept a suggestion immediately and defer reviewing it until later.
- CUPS States: The finalized taxonomy contains 12 leaf states spanning programmer activities in ‘User Typing or Paused’ and ‘User Before Action’ segments.It is intended to be general enough to capture most activities while remaining specific to LLM-based code-suggestion interactions.
5 CUPS Data Collection Study
The study asked 21 developers to complete short Copilot-assisted coding tasks and retrospectively label their sessions with CUPS. Participants labeled 3137 coding segments and interacted with 1096 suggestions, providing data for systematic analysis of programmer-CodeRec activity.
- Procedure: Participants completed randomly assigned programming tasks in a remote desktop environment using Windows 10, VS Code, and the Copilot plugin.Tasks were selected from eight pre-selected tasks designed to fit within a 20-minute block.
- Procedure: Participants reviewed their screen recordings and telemetry immediately after coding, then annotated each session with CUPS labels.Self-labeling supports scalability and participant-specific intent annotation but may produce inconsistent labels across participants.
- Participants: 21 developers with varied Copilot experience and professional programming experience participated, using proficient languages including Python, C++, and JavaScript.Nineteen of the 21 participants used Python.
- Results: 34.0% of 1024 displayed suggestions were accepted during the coding tasks.The average participant acceptance rate was 36.5%, with a median of 33.8% and a range from 14.3% to 60.7%.
- Results: The analysis excludes IDK segments, which represented 6.5% of total session time across participants.Most IDK usage came from unclear video or segments that were too short, and the authors did not re-label them.
- Results: The resulting CUPS labels support systematic investigation of frequent programmer activities, time allocation, and interaction-derived interventions.The study data are used to derive insights about programmer-CodeRec interaction.
6 Understanding Programmer Behavior with CUPS: Main Results
CUPS exposes varied programmer–Copilot interaction patterns, substantial time spent in verification and related states, and state transitions that suggest interface and metric improvements.
- Interaction patterns: CUPS timelines reveal rich and variable interaction patterns across programmers using Copilot.Timelines visualize individual participants’ labeled telemetry segments and show variance in usage patterns.
- Time costs: 22.4% of session time was spent verifying suggestions, making verification the most time-consuming state.Verification was the top state for 6 participants and among the top three for 14 of 21 participants.
- State transitions: CUPS diagrams model transitions between states and show that writing new functionality, prompt crafting, and deferred thought commonly lead to verification.The reported transition probabilities were 0.59, 0.54, and 0.54, respectively.
- Design implications: The findings motivate grouping successive suggestions, suppressing suggestions during prompt crafting, and displaying multiple suggestions afterward.These interface ideas are tied to observed deferral and prompt-crafting behavior.
- State effects: Deferred suggestions were edited with probability 0.53 versus 0.18 when verified beforehand, while acceptance probability ranged from 0.12 to 0.98 across states.Acceptance was 0.98 ± 0.02 after deferring thought and 0.12 ± 0.04 while thinking about new code.
- Time costs: Post-hoc verification increased mean verification time from 3.25 to 15.21 seconds after accounting for verification following acceptance.The adjusted median was 6.48 seconds versus a pre-adjustment median of 1.99 seconds.
7 Limitations, Future Work and Conclusion
The paper identifies scope limitations, develops future directions for predicting and comparing CUPS states, and concludes that CUPS exposes time costs and motivates new metrics and interface designs.
- 7.1 Limitations: The study used provided, time-limited coding tasks rather than participants’ real-world tasks, with limited task coverage and mostly Python experiments.Only two participants used C++ and JavaScript, despite Copilot supporting many other languages.
- 7.1 Limitations: The telemetry segmentation assumed each segment contained at most one activity and did not capture longer-term interaction costs.Examples of omitted costs include security vulnerabilities and overreliance issues.
- 7.2 Future Work: Predictive models could identify programmers’ current CUPS states from session context and support real-time interventions such as reducing latency while waiting.The current labeled dataset was not sufficient to build highly accurate models.
- 7.2 Future Work: Future studies could compare CUPS behavior across user groups, over time, across Copilot versions, or against a no-Copilot condition.A counterfactual condition could assess where Copilot suggestions add net value for programmers.
- 7.3 Conclusion: The study developed CUPS with real-time telemetry, collected over 3137 labeled instances from 21 programmers, and used timelines and diagrams to analyze behavior.CUPS contains 12 mutually unique activities between consecutive Copilot actions.
- 7.3 Conclusion: CUPS-based metrics can quantify time in each state, including verification and editing after suggestion acceptance, to characterize interaction costs.The paper proposes modifying existing time and acceptance metrics to account for post-acceptance verification.
A.1 Interfaces
The appendix includes screenshots of the labeling tool and the virtual-machine interface with VS Code.
- A.1 Interfaces: Figure 9 shows the screenshot of the labeling tool represented in Figure 4.The passage identifies the figure as a labeling-tool interface.
- A.1 Interfaces: Together, the figures document both the session-labeling interface and the coding environment used in the study.This point synthesizes the two figure captions.
- A.1 Interfaces: Figure 10 shows the virtual-machine interface with VS Code.The passage identifies the figure as a VS Code development interface.
A.2 Task Instructions
The appendix presents figures for the coding tasks used in the study, spanning data manipulation, algorithms, analysis, machine learning, editing, and testing.
- A.2 Task Instructions: The tasks were shown to participants as image files to deter copying the instructions as prompts.This design choice is stated directly in the study description.
- A.2 Task Instructions: The task figures include both general programming activities and applied data-science or machine-learning activities.This grouping follows the listed task captions.
A.3 Survey Questions Results
The appendix reports survey questions and results about programming experience, code quality concerns, flow, and progress with unfamiliar languages while using CodeRec.
- A.3 Survey Questions Results: The survey appendix includes a programming-experience question and a flow-related question about using CodeRec during the coding session.Figures 19 and 21 are labeled as survey results for these materials.
- A.3 Survey Questions Results: Figure 20 accompanies the question about concern regarding code quality when using CodeRec.The question is stated as a survey item.
- A.3 Survey Questions Results: Figure 22 accompanies the question about making progress faster with CodeRec while working with an unfamiliar language.The question is stated as a survey item.
A.4 Full User Timelines
The full-session materials represent participant activity as CUPS timelines and summarize transitions between programming states, including transitions exceeding a probability threshold.
- Participant timelines cover the first 10 minutes for participants P1–P21.
- The CUPS diagram displays transitions occurring with probability higher than 0.05.
- Table 4 reports the probability of accepting a suggestion in the next two events conditional on the user's current CUPS state.