Source-linked AI summary
The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
James Prather, Brent Reeves, Juho Leinonen, Stephen MacNeil, Arisoa S. Randrianasolo, Brett Becker, Bailey Kimmel, Jared Wright, Ben Briggs
TL;DR
Novice programmers often lack metacognitive awareness, while the effects of GenAI on novice learning remain poorly understood. Using a replication study with GenAI and multimodal observation, the paper finds a divide: prepared students accelerate, whereas struggling students may be hindered and develop an illusion of competence. The authors therefore argue for scaffolding novice GenAI use, while recognizing the study’s small sample.
Problem
The impact of GenAI on novice programmers’ metacognitive awareness and actual programming use remains poorly understood.
Method
The study replicates prior novice-programming research by examining GenAI use through 21 lab sessions with participant observation, interviews, and eye tracking.
Results
Students already poised to succeed could use GenAI to accelerate, while struggling students could be hindered, retaining metacognitive difficulties and an illusion of competence.
Takeaways & Limitations
The findings support scaffolding novice programmers’ use of GenAI from the beginning of their learning experience.
Takeaways & Limitations
The study’s 21 participants, compared with 31 in the original study, may limit the generalizability of its findings.
Abstract
from arXiv · showhide
Novice programmers often struggle through programming problem solving due to a lack of metacognitive awareness and strategies. Previous research has shown that novices can encounter multiple metacognitive difficulties while programming. Novices are typically unaware of how these difficulties are hindering their progress. Meanwhile, many novices are now programming with generative AI (GenAI), which can provide complete solutions to most introductory programming problems, code suggestions, hints for next steps when stuck, and explain cryptic error messages. Its impact on novice metacognition has only started to be explored. Here we replicate a previous study that examined novice programming problem solving behavior and extend it by incorporating GenAI tools. Through 21 lab sessions consisting of participant observation, interview, and eye tracking, we explore how novices are coding with GenAI tools. Although 20 of 21 students completed the assigned programming problem, our findings show an unfortunate divide in the use of GenAI tools between students who accelerated and students who struggled. Students who accelerated were able to use GenAI to create code they already intended to make and were able to ignore unhelpful or incorrect inline code suggestions. But for students who struggled, our findings indicate that previously known metacognitive difficulties persist, and that GenAI unfortunately can compound them and even introduce new metacognitive difficulties. Furthermore, struggling students often expressed cognitive dissonance about their problem solving ability, thought they performed better than they did, and finished with an illusion of competence. Based on our observations from both groups, we propose ways to scaffold the novice GenAI experience and make suggestions for future work.
1 INTRODUCTION
GenAI may accelerate novice programming, but its effects on learning and metacognitive problem solving remain uncertain. This study replicates prior work with GenAI tools to examine benefits, difficulties, and possible widening differences among novices.
- Motivation: GenAI tools can solve many programming activities and may let students progress faster or access personalized tutoring resources.Their usefulness for novice learning remains uncertain, especially because some tools were designed primarily for experts.
- Motivation: Novices already struggled with programming problem solving because they often lacked metacognitive awareness and strategies.Prior research identified difficulties involving problem understanding, moving too quickly, and reluctance to rethink seemingly complete solutions.
- Motivation: Evaluating novice metacognition provides a starting point for determining whether and how GenAI harms programming problem solving.The study focuses on whether GenAI supports critical thinking or worsens existing metacognitive difficulties.
- Study focus: This study replicates an earlier novice-programming study while adding GenAI tools to identify which prior difficulties persist and which new difficulties emerge.The experiment asks about both benefits and difficulties of using GenAI to solve programming problems.
- Findings: All previous metacognitive difficulties remained and could be compounded by GenAI, while new metacognitive difficulties also emerged.Students with better grades and higher self-efficacy were more likely to use GenAI to accelerate, whereas struggling students maintained an unwarranted illusion of competence.
2 RELATED WORK
Related work shows that GenAI can perform introductory programming tasks and support learning, but evidence about how novices actually use it and how it affects metacognition remains limited. Prior metacognition research supplies the framework for examining these behaviors.
- Generative AI in computing education: LLMs can solve many introductory programming exercises, prompting concerns that students may over-rely on generative AI.Educators have also explored LLMs for exercises, explanations, error messages, and classroom integration with guardrails.
- Generative AI use patterns: Research on professional programmers describes Copilot use as acceleration or exploration, while novices may instead drift among suggestions or rarely write their own code.These novice patterns are termed “drifting” and “shepherding.”
- Evidence gap: Survey-based studies report that students generally find GenAI helpful and deny over-reliance, but surveys do not necessarily reveal actual use.This study addresses that gap with eye-tracking and think-aloud data.
- Metacognition in programming: Metacognition involves awareness and regulation of cognitive processes, including strategies that guide problem solving and learning.It is treated as an important component of cognitive control and programming learning.
- Metacognition in programming: Novice programming problem solving can be organized into six stages, and making students aware of their current stage has been found to increase performance.The stages span reinterpretation, analogous-problem search, solution search, solution evaluation, implementation, and evaluation of the implemented solution.
- Prior findings: Prior work suggests GenAI may support rather than replace problem solving, but its use has also been associated with lower grades, lower self-efficacy, and greater fear of failure.Researchers therefore suggest that lower-performing students may need help using GenAI without circumventing their own learning.
3 METHODS
The study replicates an earlier novice-programming experiment while adding GenAI tools, eye tracking, observation, interviews, and contextualized analyses. Participants solved a C++ problem under a 35-minute protocol using Copilot, ChatGPT, and the Athene assessment tool.
- 3.1 Context: The researchers replicated Prather et al.’s study using the same automated assessment tool and programming problem where possible.Athene compiles submissions, runs test cases, and provides feedback to users.
- 3.1 Context: The selected C++ problem asked students to determine whether more positive or negative numbers appeared in an indeterminate sequence of integers.Loops had been introduced two weeks before the study, and the researchers attempted to keep the setting similar to the original experiment.
- 3.2 Participants: Twenty-one of 27 enrolled students opted into the study at a small U.S. research university, with consent and institutional review approval.The authors also report participant demographic information and note that some marginalized students did not participate.
- 3.3 Lab Study: Students used lab computers with VSCode and GitHub Copilot, while the browser allowed access to ChatGPT during the programming task.Between sessions, ChatGPT conversations and the C++ file were cleared to reduce carryover from previous context.
- 3.4 Analysis: The study added eye-tracking and qualitative analyses to identify three new metacognitive difficulties that were added as analytic tags.The eye-tracking system recorded fixations and saccades that could be replayed and overlaid on screen recordings.
- 3.3 Lab Study: The 35-minute lab protocol combined a think-aloud warm-up, participant observation, eye tracking, interviews, and self-efficacy measurement.Researchers calibrated Tobii hardware, recorded gaze data, minimized interaction during problem solving, and collected follow-up perceptions and background information.
4 RESULTS
Twenty of 21 students completed the programming task, but results revealed contrasting GenAI use among students who accelerated and those who struggled. Struggling students retained prior metacognitive difficulties while also experiencing GenAI-related difficulties and problems evaluating or integrating suggestions.
- 20 of 21 students completed a working program within the time limit.Completion times ranged from 5 to 35 minutes and averaged 17.1 minutes.
- Nine students exhibited metacognitive difficulties from previous work, while eight exhibited newly labeled difficulties.Only one student received a new label without also receiving an old label.
- The study retained five previously identified difficulties and added Interruption, Mislead, and Progression around GenAI use.The prior difficulties were Forming, Assumption, Dislodging, Location, and Achievement.
- Students with metacognitive difficulties tended to accept Copilot suggestions at higher rates, while less-prepared students had more difficulty judging suggestions.Acceptance patterns are summarized in Tables 3 and 4.
- Participants Who Struggled: GenAI sometimes interrupted struggling students, causing them to alternate between implementing their own solution and evaluating suggestions.Participants described Copilot prompts as distracting, and these interruptions produced several Interruption difficulties.
- Participants Who Struggled: Some struggling students copied or patterned code from ChatGPT without integrating its components, producing Progression difficulties or leaving incorrect solutions unresolved.Other students continued modifying their own approach despite suggestions that differed substantially from it, showing Achievement difficulty.
5 DISCUSSION
GenAI benefits novice programmers who can align suggestions with their intended solutions, but struggling students face persistent, compounded, and newly emerging metacognitive difficulties. These difficulties can obscure limited understanding and produce an illusion of competence, motivating targeted scaffolds.
- Benefits: Several participants accelerated to solutions by using GenAI for code they already intended to create.
- Benefits: Accelerating students appeared better able to recognize and ignore incorrect or unhelpful GenAI suggestions.The authors connect this ability to developing “negative expertise” in code reading.
- Harms: All previously identified metacognitive difficulties remained, while GenAI compounded some and introduced new ones.
- Harms: GenAI facilitated Location for many participants and could replace critical thinking while making students feel they were learning.
- Harms: Struggling students often rationalized each GenAI-assisted step as understanding, making it harder to assess their own learning.Many recognized correct solutions but could not reach them independently, producing an illusion of progress or competence.
- Scaffolding: The authors propose code replays, reflective pauses, explicit instruction in problem solving, and exposure to helpful and unhelpful suggestions as possible scaffolds.Future work should test whether these interventions prevent newly identified GenAI-related metacognitive difficulties.
6 CONCLUSIONS
The conclusions distinguish accelerated students from struggling students: GenAI can speed capable novices while leaving others with an illusion of competence. The authors therefore call for novice-oriented scaffolding and explicit metacognitive instruction, while noting important limits to the evidence.
- Findings: Students poised to succeed leveraged GenAI to accelerate, whereas struggling students could be hindered and left with an illusion of competence.
- Findings: Participants with higher grades had fewer GenAI metacognitive difficulties, while lower self-efficacy was associated with more difficulties.Overall course grades may not reveal the nuances of how students struggle with GenAI.
- Limitations: The authors caution that the sample was too small to determine whether prior programming experience, socioeconomic background, race, or gender shape harmful GenAI experiences.
- Implications: The authors argue that commonly used GenAI tools are not informed by pedagogy and should be scaffolded for novice programmers from the beginning.
- Implications: Novice-friendly tools can teach responsible GenAI use without replacing the critical-thinking component of programming problem solving.
- Implications: Explicitly teaching metacognitive behaviors and the problem-solving process is presented as necessary because metacognitive failures can lead to poor GenAI outcomes.