Source-linked AI summary
"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers
James Prather, Brent N. Reeves, Paul Denny, Brett A. Becker, Juho Leinonen, Andrew Luxton-Reilly, Garrett Powell, James Finnie-Ansley, Eddie Antonio Santos
TL;DR
The paper examines how novice programmers use GitHub Copilot in introductory programming, addressing limited evidence about their practical interactions. It finds that students expect faster coding but worry about understanding generated code and becoming reliant on the tool, while remaining optimistic about broader integration.
Problem
Existing automated approaches do not provide contextual support for programming exercises intended to facilitate and improve student learning.
Method
The study provides the first exploration of Copilot use by novices in an introductory programming (CS1) class.
Results
Most students perceived that Copilot would help them write code faster, while worrying about understanding generated code and becoming reliant on the tool.
Takeaways & Limitations
Students are wary of Copilot's implications but optimistic about integrating the tool more fully, alongside proposed ethical considerations and design guidelines.
Takeaways & Limitations
The results reflect students' actions and thoughts during their first exposure to Copilot, before novelty effects wear off.
Abstract
from arXiv · showhide
Recent developments in deep learning have resulted in code-generation models that produce source code from natural language and code-based prompts with high accuracy. This is likely to have profound effects in the classroom, where novices learning to code can now use free tools to automatically suggest solutions to programming exercises and assignments. However, little is currently known about how novices interact with these tools in practice. We present the first study that observes students at the introductory level using one such code auto-generating tool, Github Copilot, on a typical introductory programming (CS1) assignment. Through observations and interviews we explore student perceptions of the benefits and pitfalls of this technology for learning, present new observed interaction patterns, and discuss cognitive and metacognitive difficulties faced by students. We consider design implications of these findings, specifically in terms of how tools like Copilot can better support and scaffold the novice programming experience.
1 INTRODUCTION
This paper studies how novice programmers use GitHub Copilot during a typical CS1 assignment, addressing limited evidence about their interactions and perceptions. It identifies perceived speed benefits, concerns about understanding and reliance, new interaction patterns, cognitive and metacognitive difficulties, and design implications.
- Novice programming support remains difficult because existing automated approaches do not provide contextual support for arbitrary programming exercises.
- The study observes first-year university students using Copilot on a typical introductory programming assignment for the first time.The researchers triangulate observations with interviews about students’ perceptions of the tool’s learning benefits and dangers.
- Most students perceived that Copilot would help them write code faster, while worrying that they might not understand its generated code or become reliant on it.
- Students guided Copilot toward solutions by shaping its generated prompts, a pattern called shepherding, rather than writing code from scratch and integrating suggestions.
- Some students drifted between incorrect Copilot suggestions and became lost, while students also faced cognitive and metacognitive difficulties when using the tool.
- The paper presents ethical considerations and design guidelines intended to better support and scaffold novice programmers using AI code generators.
2 RELATED WORK
Related work shows that code-generation models are capable of producing functional programs and educational materials, but their classroom impact and novice use remain insufficiently understood. Prior studies also report usability benefits alongside concerns about reliability, comprehension, and over-reliance.
- Code-generation models have solved introductory programming problems at high levels, including Codex performing in the top quartile of real students.
- Research had evaluated code generators’ capabilities, but few computing-education studies examined their educational impact or how students adopt them.
- Codex scored around 80% across two introductory programming tests and ranked 17 out of 71 compared with enrolled students.
- Generated educational exercises often contained executable sample solutions, but those solutions passed Codex-generated tests only 30% of the time.Well over 80% of generated exercises included executable sample code.
3 METHODOLOGY
The study observed novice university programmers using GitHub Copilot on a typical introductory programming assignment, then interviewed them about their experience. Observations and interviews were analyzed thematically to characterize interactions and perceptions.
- Procedure: Each participant had 30 minutes to solve a Minesweeper-modeled programming problem while using Copilot and other permitted resources.The time limit matched other invigilated in-class programming activities that semester.
- Data collection: A researcher observed participants individually under a think-aloud protocol and conducted a short interview after completion or time expiration.Interview questions addressed learning, intended use, advantages, and concerns.
- Tool context: Copilot generated one or several light-gray code suggestions in VS Code that participants could accept, reject, or bypass while typing.Suggestions were based on the file contents and generally became more relevant as coding progressed.
- Analysis: Researchers used six-phase reflexive thematic analysis, iteratively developing themes through coding, discussion, review, naming, and writing.Observational and interview data were treated as one dataset, with group meetings used to discuss coding differences rather than calculate inter-coder reliability.
4 RESULTS
The analysis identified themes covering interactions, cognition, purpose, and speculation in novices’ Copilot use. Participants developed distinctive ways of accepting, adapting, interpreting, and pursuing generated code, with both supportive and disruptive effects.
- Overview: The analysis identified four main themes—Interactions, Cognitive, Purpose, and Speculation—with sub-themes tracked across observations and interviews.The “No downside” sub-theme appeared only in interviews.
- Interactions: Participants engaged in coding activities, adapted autogenerated code, and deciphered Copilot suggestions, making interactions the dataset’s most common theme.The interaction analysis also distinguished accepting, rejecting, and user-experience issues.
- Interaction patterns: Three novel interaction patterns were described: shepherding, drifting, and slow accept, alongside backtracking and adapting.Shepherding involved trying to coerce Copilot toward needed code; slow accept involved typing suggestions without pressing Tab.
- Interaction patterns: Backtracking occurred when 13 participants deleted code immediately after accepting it, suggesting suggestions were sometimes accepted without careful review.Backtracking was the fifth most-frequently occurring behavior observed.
- Cognitive and metacognitive effects: Copilot sometimes supported stuck participants as a metacognitive scaffold, but incorrect or distracting suggestions could confuse novices and draw them into unproductive debugging.Participants also reported positive feelings when suggestions helped them feel on the right path.
4.3 Theme: Purpose
Participants described Copilot as useful for guidance, discovering approaches, and completing assignments faster, but they also recognized risks to learning through over-reliance and reduced practice.
- Guiding: Participants used Copilot for guidance through problem solving, including learning unfamiliar techniques and discovering previously unknown edge cases.Three participants specifically said Copilot taught them something they did not know.
- Outsourcing: Participants worried that outsourcing code to Copilot could make it a black box or crutch that weakens problem solving and understanding.Some feared users might accept code without processing what they were implementing or become dependent on the tool.
- Speed: Participants perceived speed benefits from avoiding manual typing, receiving suggested approaches, and reducing syntax-related effort.One participant reported reaching testing in less than 20 minutes, with most time spent reading and thinking about the problem.
- Learning trade-offs: Some participants viewed typing their own code as learning practice and worried that accepting suggestions would reduce memorization of programming details.One participant specifically mentioned forgetting semicolons and parenthesis placement when code was handed to them.
- Speculation: Participants attributed intelligence and agency to Copilot, describing it as knowing, guiding, or anticipating what they wanted.These perceptions were accompanied by speculation about future control, employment, privacy, and plagiarism.
5 DISCUSSION
The discussion returns to how novices use Copilot and considers ethical issues and design implications. It presents support and scaffolding as central design concerns for LLM-based code generators.
- Discussion focus: The discussion examines implications of novice use of LLM-based code generators such as Copilot.
- Ethical considerations: The paper discusses ethical considerations arising from the use of these systems.
- Design implications: The authors offer design implications based on the study’s findings and insights.
5.1 User Interactions
Novices used Copilot both to explore solution directions and accelerate implementation, but their interactions also included shepherding and drifting. Copilot mainly supplied “Where to next?” feedback, creating both efficiency benefits and risks of confusion or over-reliance.
- Exploration and acceleration: Copilot mainly gave feedback on “Where to next?”, while exploration addressed problem direction and acceleration addressed the current implementation strategy.
- Exploration and acceleration: Novices used Copilot for initial guidance and for generating code when they already knew what they wanted.
- Novel interaction patterns: Shepherding involved spending most of the time coercing Copilot to generate code, which the authors viewed as a potential signal of over-reliance among novices.
- Novel interaction patterns: During drifting, students hesitantly accepted, experimented with, deleted, and repeatedly reconsidered Copilot’s suggestions.
- Benefits and difficulties: Novices sometimes believed Copilot would always produce useful, correct code, despite reported Codex correctness of around 70%-80%.This belief is especially troublesome because novices may struggle to identify incorrect suggestions.
- Benefits and difficulties: The primary benefit was faster progress, while suggestions also helped students avoid syntax errors and focus mental effort on program semantics.
5.2 User Perceptions
Novices perceived Copilot as useful for producing code faster and for overcoming moments of difficulty, but their first experiences also involved metacognitive difficulties, over-reliance concerns, and negative emotion.
- Perceived benefits: Novices saw Copilot as helping them create code faster and sometimes used it like a colleague when they became stuck.
- Metacognition and self-regulation: Participants displayed reflective self-regulation by rereading prompts, working problems on paper, and revisiting misunderstood instructions.
- Metacognition and self-regulation: Some participants experienced a false sense of accomplishment from having substantial code despite being far from a working solution.
- Risks and emotions: Over-reliance was the most common participant concern, and students feared it could hinder their own learning.
- Risks and emotions: The authors hypothesize that over-reliance could worsen novices’ metacognitive programming skills and behaviors.
- Risks and emotions: Copilot’s current interaction style may promote cognitive and metacognitive difficulties and negative emotion, with potential consequences for self-efficacy.
5.3 Ethical Considerations
The study highlights ethical concerns around Copilot, including privacy, employability, academic misconduct, licensing, bias, and inappropriate or insecure generated code. It argues that educators and designers must address these implications as adoption grows.
- Student concerns: Even after brief exposure, participants raised privacy and employability concerns, suggesting Copilot may initially feel threatening to some students.
- Educational response: The authors emphasize teaching students the broader social and ethical implications of code-generating tools as adoption expands.
- Academic integrity: AI-generated solutions can increase misconduct opportunities while reducing detection likelihood because they are diverse and resilient to standard plagiarism detection.
- Academic integrity: AI code generators complicate academic integrity because generated and modified code raises unresolved questions about authorship and plagiarism.
- Licensing and attribution: Copilot may reproduce licensed repository code without clearly signaling its source, exposing developers to incompatible license obligations.
- Training data and generated code: Public repositories may contain advanced styles, confusing features, security vulnerabilities, or biased patterns that mislead novice learners.
5.4 Design Implications
The design implications focus on giving novices more control, reducing cognitive load, supporting metacognition, improving explainability, and making code reuse more transparent.
- Control and personalization: Novices should be able to select, adjust, or hide the feedback they receive as their programming skills develop.
- Control and personalization: Context about the task and user background could tailor when suggestions appear, allowing novices to attempt problems independently before requesting help.
- Suggestion design: Shorter suggestions were evaluated more quickly and were less likely to be changed or deleted after acceptance.
- Suggestion design: The authors suggest preferring or filtering shorter solutions because lengthy suggestions may increase cognitive load among novices.
- Metacognitive scaffolding: Interface scaffolds could support metacognition through structural comments, suggestion cycling, and better support for uncommon rubber-ducking behavior.
- Explainability: User-centered explainability could expose the model’s inputs, confidence, and estimate of user skill to help novices understand its behavior.
- Ethical reuse: Systems could signal source matches or hide suggestions failing user-defined license criteria, helping educators teach ethical reuse and citation.
5.5 Limitations
The study has several scope and data-collection limitations: observations did not include screen or audio recordings, some conditions resembled a lab rather than normal coursework, and only Copilot was examined. Results also primarily reflect students’ first exposure to the technology and an early data-collection period.
- The study did not record screens or audio because of IRB considerations.
- Some study conditions resembled a lab-based experience rather than students’ normal in-class programming assignments.The authors attempted to mitigate this, but it may have affected participants’ work and tool interactions.
- The study examined only Copilot because it was the only easily available code-generating tool during data collection.
- Data collection occurred shortly after Copilot’s release, limiting the time available for complex, multi-channel data collection.
- The results reflect students’ actions and thoughts during their first exposure to Copilot, so novelty effects may diminish over time.
5.6 Future Work
Future work should examine how students’ interactions with Copilot evolve over longer periods and investigate the reasons behind observed behaviors. A key question is whether persistent suggestions increase cognitive load and frustration, especially relative to findings from more experienced programmers.
- Longitudinal studies should examine whether Copilot interactions evolve over a full semester as students become more skilled programmers.
- Future research should investigate why students behaved as observed, including whether continuously visible suggestions increase cognitive load.
- Researchers should examine whether increased cognitive load could explain student frustration and differences from prior studies of more experienced programmers.
- The broader goal is to determine how AI code generators can be incorporated effectively into introductory programming classrooms.
6 CONCLUSION
This paper provides an initial study of novice Copilot use in an introductory programming course through observations and interviews. It finds that novices struggle with the tool while seeing potential benefits, and identifies interaction patterns and design implications for more usable classroom support.
- The study is the first exploration of novice Copilot use in an undergraduate introductory programming class on a typical novice task.
- Observations and interviews were used to examine how novices interacted with Copilot.
- Novices struggled to understand and use Copilot, were wary of its implications, but remained optimistic about integrating it into future development work.
- The researchers observed two novel interaction patterns and explored the ethical implications of their findings.
- The findings offer design implications for integrating AI code generators into introductory programming and creating more usable tools for novice users.
- These tools may help novices get unstuck in programming tasks at scale.