Source-linked AI summary
How Novices Use LLM-Based Code Generators to Solve CS1 Coding Tasks in a Self-Paced Learning Environment
Majeed Kazemitabaar, Xinying Hou, Austin Henley, Barbara J. Ericson, David Weintrop, Tovi Grossman
TL;DR
LLM code generators may foster dependency and create difficulties for novice programmers, making their use in learning contexts important to understand. This study analyzes novice learners’ interactions with Codex and identifies usage patterns, coding approaches, and signs of over-reliance and self-regulation, informing future curriculum and tool development.
Problem
Novice programmers may become dependent on LLMs, struggle to express coding intent or verify generated code, and raise academic-integrity concerns.
Method
The study thematically analyzed logged interactions from novice learners using an online self-paced environment with 45 CS1 Python tasks and an embedded Codex generator.
Results
The analysis identified four coding approaches and found signs of both over-reliance and self-regulation in learners’ interactions with AI code generators.
Takeaways & Limitations
Findings can inform curriculum, introductory programming tools, and scalable self-paced learning environments that incorporate AI code generators.
Takeaways & Limitations
The analysis could not capture learners’ thought processes or motivations and involved only 33 young learners in an online, informal setting.
Abstract
from arXiv · showhide
As Large Language Models (LLMs) gain in popularity, it is important to understand how novice programmers use them. We present a thematic analysis of 33 learners, aged 10-17, independently learning Python through 45 code-authoring tasks using Codex, an LLM-based code generator. We explore several questions related to how learners used these code generators and provide an analysis of the properties of the written prompts and the generated code. Specifically, we explore (A) the context in which learners use Codex, (B) what learners are asking from Codex, (C) properties of their prompts in terms of relation to task description, language, and clarity, and prompt crafting patterns, (D) the correctness, complexity, and accuracy of the AI-generated code, and (E) how learners utilize AI-generated code in terms of placement, verification, and manual modifications. Furthermore, our analysis reveals four distinct coding approaches when writing code with an AI code generator: AI Single Prompt, where learners prompted Codex once to generate the entire solution to a task; AI Step-by-Step, where learners divided the problem into parts and used Codex to generate each part; Hybrid, where learners wrote some of the code themselves and used Codex to generate others; and Manual coding, where learners wrote the code themselves. The AI Single Prompt approach resulted in the highest correctness scores on code-authoring tasks, but the lowest correctness scores on subsequent code-modification tasks during training. Our results provide initial insight into how novice learners use AI code generators and the challenges and opportunities associated with integrating them into self-paced learning environments. We conclude with various signs of over-reliance and self-regulation, as well as opportunities for curriculum and tool development.
1 INTRODUCTION
LLM code generators may broaden access to computing education, but novice use raises concerns about dependency, comprehension, verification, and academic integrity. This paper analyzes how 33 novice learners used Codex while learning Python and examines their prompts, generated code, and coding approaches.
- LLM code generators can generate functioning programs from natural-language descriptions and may support self-paced computing education.
- Novices may struggle to express coding intent, understand technical jargon, and comprehend or verify AI-generated code.
- The study analyzes 33 learners aged 10–17 who learned Python for the first time with access to an AI code generator.
- The research examines when learners use Codex, what they request, prompt properties, generated-code properties, and how they use, modify, and verify generated code.
- The analysis identifies four coding approaches and reports signs of both overreliance and self-regulation in learners’ interactions with AI code generators.
2 RELATED WORK
Prior work has examined AI-assisted coding mainly with experts or limited novice tasks, leaving a need to understand novice interaction in authentic, larger-scale learning contexts. Related research also motivates studying scaffolding and immediate assistance during code writing.
- Existing usability research on AI coding assistants largely focuses on expert programmers and small numbers of programming tasks.
- Research on novice programmers has included different program synthesizers, but one cited study examined only three tasks.
- Understanding novice usage patterns, coding approaches, prompts, generated code, and learning impacts is identified as important for integrating LLMs into computer science education.
- Prior scaffolding research includes support before coding, such as flowcharts and explicit planning, to organize solution ideas and develop programming skills.
- Immediate assistance during coding can provide explanations, error-fixing suggestions, next-step hints, worked examples, or equivalent Parsons problems.
3 METHODOLOGY
The study reanalyzes log data from novice learners using a self-paced Python environment with Codex, documentation, feedback, and structured coding tasks. Researchers coded Codex interactions thematically using an iteratively refined codebook with reported inter-rater reliability.
- 3.1 Data Set and Data Instrumentation: The dataset came from a three-week study in which 69 learners aged 10–17 used Coding Steps, while this analysis focused on 33 learners in the Codex condition.
- 3.1 Data Set and Data Instrumentation: Coding Steps provided 45 CS1 Python tasks, an embedded programming environment, remote-instructor feedback, a Python tutorial, and an AI code generator based on Codex.
- 3.2 Procedure: Training involved 45 two-part programming tasks and 40 multiple-choice questions completed at learners’ own pace with documentation and personalized feedback.
- 3.2 Procedure: Evaluation included immediate and one-week retention post-tests, each containing five code-authoring and five code-modification tasks without AI assistance or documentation.
- 3.3 Data Analysis: Researchers analyzed prompts as 1666 units of analysis across four code dimensions and developed codes inductively through iterative researcher review.
- 3.3 Data Analysis: After codebook refinement, independent coding of another five tasks reached inter-rater reliability of 0.87 using percentage agreement.
4 RESULTS
Learners used Codex in several problem-solving contexts, most often at the task’s beginning or after prior AI output. Their interactions included full-solution requests, debugging, reattempts, and occasional verification or comparison of generated code.
- 1666 Codex usages came from 1379 submitted tasks, while 356 tasks were submitted without Codex.
- 4.1 RQ1 A: When do Learners Use Codex?: 46% of Codex uses occurred at the beginning of tasks, 34% after prior Codex use, 17% after manual coding, 5% after clearing the editor, and 1% when learners already had solutions.
- 4.1.1 Using Codex at the Beginning: Among 760 beginning-of-task uses, 66% copied the full task description, 7% rephrased it, and 26% broke the task into subgoals.
- 4.1.2 Using Codex After Clearing the Editor: After clearing the editor, learners requested entire solutions in 51% of cases and task parts in 49%, with 90% of these instances following unsuccessful Codex uses.
- 4.1.3 Using Codex After Manual Coding: In 191 cases following manual-coding problems, learners used Codex to fix existing code in 44%, generate the entire solution in 18%, or generate new code ignoring the issue in 32%.
- 4.1.4 Using Codex After Previous Codex Usage: After prior AI-generated code, learners kept it unchanged in 53%, deleted it in 36%, modified it slightly in 7%, and broke it in 4% of 572 instances.
- 4.1.5 Using Codex With Existing Solutions: In six of 16 cases where learners already had solutions, they compared their manual solution with AI-generated code and made minor edits to their own solution.
4.2 RQ1 B: What are Learners Asking from Codex?
Learners mainly asked Codex to generate complete solutions or code for subgoals, with requests varying between syntax alone and syntax plus logic.
- 43% (n=723) of Codex usages requested an entire solution, while 37% (n=626) requested new subgoals.
- Learners also used Codex to fix existing code in 7% (n=110) of usages.
- When fixing code with Codex, learners correctly localized their issues in 61% (n=73) of cases.Seven cases explicitly asked Codex to fix the code.
- Of 723 entire-solution prompts, 46% requested syntax and logic, whereas 54% requested pure syntax.
- Of 626 new-subgoal prompts, 85% requested pure syntax, showing that task decomposition usually focused requests on syntax.
4.3 RQ1 C: Prompt Properties
Learners often copied or decomposed task descriptions into prompts, but prompt detail and clarity varied and sometimes led to repeated or incomplete generations.
- Prompt Crafting Patterns: Learners used two prompt-crafting patterns: sentence-by-sentence task decomposition and repeated or slightly reworded prompts.
- Relationship to Task Descriptions: 52% (n=864) of prompts copied the task description directly, while 27% (n=233) were partial copies requesting code for part of a task.
- Relationship to Task Descriptions: Some independently written prompts reworded tasks accurately, while others omitted details, misinterpreted tasks, or were unrelated to the task or Python.
- Relationship to Task Descriptions: Learners sometimes used pseudocode with syntactical elements to specify exact algorithmic logic, indicating deeper algorithmic thinking than behavior-only prompts.
- Prompt Language: 28% (n=201) of prompts showed vagueness, including unspecified variable values, storage locations, random-number ranges, or conditional conditions.
- Prompt Clarity: Task decomposition worked for sequential independent subgoals but faltered when later sentences modified earlier instructions, often causing code regeneration.
- Prompt Clarity: 7% (n=109) of multi-prompt usages were exact repetitions and 3% (n=55) were slight rephrasings, usually after an initial prompt failed.Only 13 usages added meaningful detail in reworded prompts.
4.4 RQ1 D: AI-Generated Code Properties
Most generated code had no identifiable problems, but errors included omitted requirements, regenerated code, extra code, and curriculum-inappropriate complexity linked to prompt and editor context.
- 81% (n=1357) of 1666 Codex usages produced AI-generated code without identifiable problems.19% (n=309) exhibited problematic characteristics.
- Codex regenerated existing editor code in 86 usages and omitted necessary minor code in 60 cases.Regeneration complicated attempts to fix erroneous existing code.
- Codex generated code beyond the curriculum or from advanced topics in 22 instances, including a while loop before loops had been introduced.
- Codex produced additional unspecified code in 204 cases, often inferred from editor content or the prompt.
- Of 204 extra-code cases, 34% (n=70) were wrong and deleted, 34% (n=60) were directly usable, 14% (n=29) needed manual modification, and 12% (n=25) needed minor value changes.
- Low-quality code was associated with poorly crafted prompts (n=105), missing important details (n=71), and existing low-quality editor code (n=34).
- Copied task-description prompts produced high-quality code in 81% (n=701) of cases, compared with 46% (n=104) for less-detailed rewordings.
4.5 RQ1 E: Utilizing AI-Generated Code
Learners generally placed and tested AI-generated code, but they sometimes retained incorrect duplicates, skipped verification, or broke correct code during modification.
- Learners used AI-generated code for placement, verification, modification, and as an example for fixing or writing similar code.
- Placement: The generator inserted code at the cursor, requiring manual placement adjustment when the cursor was wrong or generated code included existing code.
- Placement: 69 of 1666 usages placed AI-generated code incorrectly, including errors in indentation, declaration order, or replacement of original code.In 18 cases, learners kept both original and generated versions.
- Verification: Learners verified code by tinkering, running it, or manually adding code; 30 instances involved temporary changes to test behavior.
- Verification: 60% (n=1005) of 1666 usages were tested; among these, 71% (n=720) ran correctly and 29% (n=285) produced errors or incorrect behaviors.
- Verification: After 485 usages, learners did not run the generated code, including 63 cases (13%) submitted as final solutions without testing.
- Modification: Learners correctly modified AI-generated code in 175 cases, but broke correct code in 57 instances and could not fix it.
- Modification: Among 166 extra-code cases, learners handled the extra code correctly in 62% (n=103), but retained incorrect code in 30% (n=50).
4.6 RQ2: Effect of AI Code Generator Coding Approaches
The analysis identified four coding approaches that novices used to incorporate Codex into programming tasks, with Hybrid showing positive post-test trends and AI Single Prompt showing negative trends. The authors caution that the relationship between approach utilization and individual learning outcomes requires further study.
- Coding approaches: Four approaches were identified: AI Single Prompt, AI Step-by-Step, Hybrid, and Manual.AI Single Prompt generated an entire solution once; AI Step-by-Step generated task parts consecutively; Hybrid combined AI-generated and manually written code; Manual solutions were self-written.
- Coding approaches: 46% of tasks used AI Single Prompt, the most frequent approach, in which learners generated the entire solution with one prompt.This included 400 tasks where learners copied the task description and submitted generated code without manual coding; 17 were submitted without testing.
- Coding approaches: 6% of tasks used AI Step-by-Step, with Codex generating the main submitted code through consecutive prompts for different task parts.Learners either identified subgoals or divided the task description by sentence before prompting Codex.
- Coding approaches: 19% of tasks used Hybrid, combining Codex-generated subgoals with manually written code, including debugging-only use on 57 tasks.The remaining 29% used Manual coding, where the submitted solution was entirely self-written.
- Learning outcomes: Hybrid utilization correlated consistently positively with post-test scores, whereas AI Single Prompt utilization correlated consistently negatively with post-test evaluation scores.The authors offer active switching between manual and AI-assisted coding as a possible interpretation, but do not establish an effect.
- Learning outcomes: Further in-depth studies are required to examine how coding approaches affect individual learning outcomes.Figure 4 reports correlations between approach utilization and four evaluation post-test scores.
5 DISCUSSION
The discussion identifies both self-regulated and over-reliant ways novices used AI-generated code. It also links prompt quality and tool design to learners’ ability to engage critically with generated programs.
- Self-regulation and over-reliance: Learners showed self-regulation by adding verification code and tinkering with generated code to understand underlying concepts.Temporarily removing code and testing its contribution was another example of active engagement in learning.
- Self-regulation and over-reliance: Frequent AI Single Prompt use was identified as a common sign of over-reliance on Codex.An extreme case involved copying the task description into a prompt and submitting the generated solution without editing.
- Self-regulation and over-reliance: Some learners appeared to assume that AI-generated code was flawless and required no further verification.The discussion connects this assumption with inserting generated output directly into the intended location.
- Prompting and tool design: Poorly crafted prompts led to low-quality code that learners struggled to use.The study analyzed prompt accuracy, language, and relationship to code, extending prior mainly offline research on prompt effects.
- Prompting and tool design: The authors propose learner sandboxes that support iterative prompting, testing, verification, and tinkering.Visual cues could differentiate editable elements and encourage learners to experiment rather than remain attached to generated code.
6 LIMITATIONS
The study’s qualitative interpretation is constrained because participants’ uninterrupted interactions did not reveal their thought processes or motivations. Its findings also come from a small, informal sample and may not generalize across settings or tools.
- Scope and interpretation: The analysis did not capture participants’ thought processes and motivations during Codex use.The authors state that think-aloud studies are needed to understand prompt meanings, modification intentions, and verification decisions.
- Scope and interpretation: The study included only 33 young learners in an online, informal setting, limiting generalization to other ages and formal classrooms.Different age groups and K-12 or undergraduate courses may reveal other behaviors and usage scenarios.
- Scope and interpretation: Some behaviors may depend on the design of the Coding Steps IDE and may not generalize to other AI code generators.The authors specifically identify incorrect placement of AI-generated code as a design-specific behavior.
7 CONCLUSION
The paper provides initial evidence about how young learners prompt, use, verify, and modify AI-generated code, including both self-regulation and over-reliance. It concludes that curriculum and tool development should adapt to effective learner practices while addressing these risks.
- Implications: The authors argue that curriculum and tool development should reflect effective AI-code-generator practices while increasing awareness of over-reliance and plagiarism concerns.This recommendation is framed within the growing accessibility of tools such as ChatGPT and GitHub Copilot.
- Conclusion: The study provides initial insights into learners’ usage patterns, prompts, verification, and use of AI-generated code.It also identifies four coding approaches and examines their relationship with learning outcomes measured one week later.
- Conclusion: Learners displayed both self-regulated practices, such as verification and tinkering, and difficulties integrating AI-generated code into solutions.Prompt language and clarity sometimes affected the quality of generated code.