PaperlayerView exact revision on arXiv

Source-linked AI summary

How do Copilot Suggestions Impact Developers' Frustration and Productivity?

Emanuela Guglielmi, Venera Arnoudova, Gabriele Bavota, Rocco Oliveto, Simone Scalabrino

arXiv:2504.06808v1cs.SE

TL;DR

Automatic code suggestions may speed development but can also interrupt developers, and prior work had not experimentally measured frustration from Copilot suggestion interruptions. The paper proposes a quasi-experimental study to test whether effects differ for experts and novices; it reports theories and a planned study, not empirical outcomes.

  • Problem

    Suggestions may support developers who are stuck but also interrupt cognitive flow, and the authors report no controlled experiments measuring frustration caused by Copilot suggestion interruptions.

    Source: §I · p. 1

  • Method

    The authors propose a quasi-experiment using two maintenance tasks and wearable heart-rate-variability measurements to study frustration.

    Source: B. Coding Tasks · p. 3, 2) Dependent Variables · p. 4

  • Results

    No empirical outcomes are reported; the proposed theories predict frustration and productivity costs for experts and benefits for novices.

    Source: §I · p. 2, §III · p. 3

  • Takeaways & Limitations

    If supported, the theories could inform experience-based customization of suggestion frequency and other UX choices.

    Source: §X · p. 7

    The tasks may not fully represent real work, and the planned sample may miss theorized effects; the authors plan 50 participants for 80% power with a 50% difference.

    Source: §VIII · pp. 6–7

Abstract

from arXiv · show

Context. AI-based development tools, such as GitHub Copilot, are transforming the software development process by offering real-time code suggestions. These tools promise to improve the productivity by reducing cognitive load and speeding up task completion. Previous exploratory studies, however, show that developers sometimes perceive the automatic suggestions as intrusive. As a result, they feel like their productivity decreased. Theory. We propose two theories on the impact of automatic suggestions on frustration and productivity. First, we hypothesize that experienced developers are frustrated from automatic suggestions (mostly from irrelevant ones), and this also negatively impacts their productivity. Second, we conjecture that novice developers benefit from automatic suggestions, which reduce the frustration caused from being stuck on a technical problem and thus increase their productivity. Objective. We plan to conduct a quasi-experimental study to test our theories. The empirical evidence we will collect will allow us to either corroborate or reject our theories. Method. We will involve at least 32 developers, both experts and novices. We will ask each of them to complete two software development tasks, one with automatic suggestions enabled and one with them disabled, allowing for within-subject comparisons. We will measure independent and dependent variables by monitoring developers' actions through an IDE plugin and screen recording. Besides, we will collect physiological data through a wearable device. We will use statistical hypothesis tests to study the effects of the treatments (i.e., automatic suggestions enabled/disabled) on the outcomes (frustration and productivity).

I. INTRODUCTION

AI-assisted development tools offer real-time suggestions that may speed coding but can also interrupt developers and affect frustration and productivity. The paper proposes that these effects differ for experts and novices, and plans a quasi-experiment to test the theories.

I. INTRODUCTION

  • AI-assisted development tools offer real-time code suggestions intended to reduce cognitive effort and speed task completion.Source: §I · p. 1
  • Suggestions can help developers who are stuck, but frequent or irrelevant prompts may interrupt cognitive flow, increase frustration, and reduce productivity.Source: §I · p. 1
  • The authors identify a gap: no controlled experiments had measured frustration caused by interruptions from Copilot suggestions.Source: §I · p. 1

II. THEORY

  • The proposed theories predict that suggestions may frustrate experts and reduce their productivity, while helping novices and reducing their frustration.Source: §I · p. 2, §II · p. 2

A. TE: Expert Developers and Automatic Suggestions

The expert-developer theory treats automatic suggestions as possible interruptions that can raise frustration and reduce productivity. It predicts declines in both actual and perceived productivity.

A. TE: Expert Developers and Automatic Suggestions

B. TN: Novice Developers and Automatic Suggestions

The novice-developer theory treats automatic suggestions as potential support when developers are blocked, while recognizing that incorrect or hard-to-choose suggestions may frustrate them. It predicts reduced frustration and increased productivity, conditional in part on suggestion correctness.

B. TN: Novice Developers and Automatic Suggestions

III. RESEARCH QUESTION

The study’s research questions ask whether automatic suggestions affect expert and novice frustration and productivity differently. Its stated context comprises software developers and tasks to be completed.

III. RESEARCH QUESTION

  • RQ1 asks whether suggestions increase frustration for experts and reduce it for novices.Source: §III · p. 3
  • RQ2 asks whether suggestions reduce experts’ actual productivity and increase novices’ actual productivity.Source: §III · p. 3
  • RQ3 asks whether suggestions reduce experts’ perceived productivity and increase novices’ perceived productivity.Source: §III · p. 3

IV. STUDY CONTEXT

  • The study context identifies software developers as subjects and tasks to be completed as objects.Source: §IV · p. 3

A. Participants

The study plans to recruit a balanced novice–expert sample and evaluate them on two maintenance tasks involving significant changes to existing systems. The tasks are designed for completion in under an hour on average, with a 120-minute cap.

A. Participants

  • The study targets at least 32 developers, split evenly between novices and experts, and may recruit up to 50 through convenience sampling.Novices are sought among computer science students and experts among industry developers.Source: A. Participants · p. 3
  • A three-year experience gap separates candidate student novices from professional experts, and experience outliers will be removed to keep the groups distinct.An initial questionnaire also collects programming background and prior Copilot use.Source: A. Participants · p. 3

B. Coding Tasks

  • Both tasks require significant evolution of an existing system rather than writing code from scratch, and are expected to take under 60 minutes on average.Each session will stop at 120 minutes to allow extra completion time without artificially interrupting participants.Source: B. Coding Tasks · p. 3
  • Task A extends a travel-management REST API with travel search, statistics, pagination, and cancellation features.Subtasks involve new endpoints, business logic, and repository-layer database operations.Source: B. Coding Tasks · p. 3
  • Task B extends a Java BrickBreaker game with a menu, high scores, power-ups, pause/resume, and a game-over screen.The task adds gameplay features to an existing game.Source: B. Coding Tasks · pp. 3–4

V. DATA ACQUISITION

The study will examine frustration and productivity in a quasi-experiment, controlling suggestion availability while observing other suggestion characteristics. It fixes the assistant to GitHub Copilot and records developer experience and suggestion behavior as independent variables.

V. DATA ACQUISITION

  • The quasi-experimental study will examine how automatic suggestions affect frustration and productivity, controlling suggestion availability and task order while observing suggestion size and relevance.Other factors are not directly manipulated and will be analyzed as observed variables.Source: §V · p. 4, 1) Independent Variables · p. 4

1) Independent Variables:

  • GitHub Copilot is fixed as the sole assistant because different tools may vary in suggestion quality, quantity, and intrusiveness.The authors chose it for its popularity and availability in existing IDEs such as Visual Studio Code.Source: 1) Independent Variables · p. 4
  • Developer experience is classified as novice or expert, while suggestion availability records whether automatic suggestions are enabled or disabled.Both are binary variables in this study.Source: 1) Independent Variables · p. 4
  • Suggestion characteristics include average suggested code-block size, the number of alternatives viewed, and the percentage of suggestions accepted or rejected.Screen recordings capture size and alternatives; the Tako IDE plugin records suggestion interactions.Source: 1) Independent Variables · p. 4
  • The study will also count manual Copilot activations and hypothesizes more such requests when automatic suggestions are disabled.The authors suggest this difference could be related to increased frustration.Source: 1) Independent Variables · p. 4

2) Dependent Variables:

The study measures frustration using HRV around Copilot interruptions and productivity using both task performance and self-report. It compares enabled and disabled suggestion conditions within participants while treating experience as a fixed participant characteristic.

2) Dependent Variables:

  • Frustration is assessed from heart-rate variability, particularly SDNN and RMSSD, using 30-second windows before and after Copilot interruptions and the pre-interruption window as baseline.The study examines whether responses vary by task and developer experience; it plans paired comparisons of pre- and post-interruption measures.Source: 2) Dependent Variables · pp. 4–5
  • Actual productivity is completion percentage divided by task completion time, so more correctly completed subtasks and less time correspond to higher productivity.Completion percentage is the number of correctly completed subtasks divided by the total subtasks.Source: 2) Dependent Variables · p. 5
  • Perceived productivity is self-reported in a post-task survey.Participants rate how productive they felt during each task.Source: 2) Dependent Variables · p. 5

3) Factors and Treatments:

  • Each participant receives enabled and disabled suggestion conditions, while experience is balanced across participants rather than manipulated.Suggestion characteristics such as size cannot be fixed in advance and will be checked afterward, with possible adjustment through models.Source: 3) Factors and Treatments · p. 5

A. Group Design

The protocol assigns participants to counterbalanced enabled/disabled sequences and uses two tasks, with breaks and checks for carryover. It combines wearable measurements, screen-recorded behavior, and surveys to capture physiological responses and both objective and perceived productivity.

A. Group Design

  • An AB/BA crossover assigns each participant to complete both conditions, with treatment order counterbalanced and groups balanced by novice and expert experience.A 15-minute break separates tasks, and the study will test for carryover effects.Source: A. Group Design · p. 5, Table I · p. 5

A. Experimental Protocol

  • Participants attend a four-hour in-person session, complete two tasks in their assigned order, and wear an Empatica E4 wristband for physiological data collection.The protocol includes a two-minute calming video and verification of device connection and signal quality before coding begins.Source: A. Experimental Protocol · p. 5
  • After each task, participants complete a five-minute survey, and the 15-minute inter-task break is intended to reduce fatigue.The protocol combines objective correctness and time measures with participants’ perceived productivity ratings.Source: A. Experimental Protocol · pp. 5–6
  • Researchers will manually review screen recordings to document Copilot suggestions, participant responses, alternative requests, and behavioral patterns, and to double-check automatically collected data.Source: A. Experimental Protocol · p. 6

B. Materials

The study combines IDE and wearable instrumentation with invitation and post-task surveys to classify participants and assess productivity and experience.

B. Materials

  • The Tako Visual Studio Code plugin records IDE activity, while the Empatica E4 wristband measures frustration.The study also uses invitation and post-task surveys as instruments.Source: B. Materials · p. 6
  • The post-task survey complements objective measures such as task correctness and completion time with participants’ subjective reports.The survey is administered after each task.Source: B. Materials · p. 6
  • The invitation survey gathers demographic and experience information, including Java, task-technology, and prior Copilot familiarity.These responses help select expert and novice participants and assess whether Copilot familiarity relates to outcomes.Source: B. Materials · p. 6
  • Post-task surveys assess perceived productivity, task speed, solution quality, and whether Copilot saved time by reducing external searches.They also ask whether suggestions felt distracting or beneficial and capture perceived stress and frustration.Source: B. Materials · p. 6

VII. DATA ANALYSIS

The analysis compares frustration and productivity across suggestion-enabled and disabled conditions, while the study recognizes limits in task representativeness and statistical power.

VII. DATA ANALYSIS

  • Frustration comparisons account for current and preceding interruptions within a time window, then exclude observations whose baseline window contains an interruption.This filtered analysis checks whether residual frustration from earlier interruptions affects the results.Source: §VII · p. 6
  • The planned frustration comparisons use Wilcoxon signed-rank tests, Bonferroni-corrected α′ = 0.017, and Cliff delta effect sizes.If suggestion characteristics vary clearly among developers, separate expert and novice regression models will examine their relationships with frustration.Source: §VII · p. 6
  • The analogous analyses use PRODa for RQ2 and PRODp for RQ3 in place of frustration.Source: §VII · p. 6

VIII. LIMITATIONS, CHALLENGES, AND MITIGATIONS

  • The two maintenance tasks may not fully represent coding tasks encountered in real work contexts.They avoid requiring participants to write code from scratch, an approach intended to make the tasks more realistic.Source: §VIII · pp. 6–7
  • The planned sample may miss theorized effects; the authors plan 50 participants, yielding 80% power for a 50% difference under their assumptions.HRV measurements may also be noisy because of motion artifacts or inconsistent skin contact, although minimal motion is expected and device fit will be checked.Source: §VIII · p. 7

IX. RELATED WORK

Prior work reports frustration from unsuitable or intrusive suggestions and mixed productivity findings; the conclusion proposes tailoring suggestion behavior to developer experience if the hypotheses are supported.

IX. RELATED WORK

  • Prior studies link frustration to incorrect, oversized, premature, or requirements-mismatched suggestions,,.Copilot also supplied useful starting points and reduced online research, but did not significantly improve task times or success rates in one study.Source: §IX · p. 7
  • Productivity findings include a 55.8% faster JavaScript task with Copilot [12] and suggestion acceptance as a significant predictor of perceived productivity.An early rejection mechanism was also proposed to reduce computational costs and unhelpful completions.Source: §IX · p. 7
  • Prior interruption studies associate interruptions with greater stress, frustration, time pressure, effort, and altered productivity across workplace and software tasks –.One software study found on-screen interruptions increased code-comprehension time, while code-writing interruptions were associated with greater stress.Source: §IX · p. 7

X. CONCLUSION

  • If empirical evidence supports the theories, Copilot could vary suggestion frequency by developer experience and project technology.The authors present this as a potential UX improvement for more efficient and user-friendly AI-assisted coding.Source: §X · p. 7
Loading 2504.06808v1…