Source-linked AI summary
Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
Majeed Kazemitabaar, Justin Chow, Carl Ka To Ma, Barbara J. Ericson, David Weintrop, Tovi Grossman
TL;DR
AI code generators had not been studied from a learning perspective in introductory programming, leaving questions about novice understanding, modification, reliance, and learning without these tools. This study conducted a controlled experiment with 69 novice learners using Codex during Python training and found improved code-authoring performance without reduced manual modification performance or subsequent performance without Codex.
Problem
Prior research had not established how AI code generators affect novice learning in introductory programming, including whether beginners can understand, modify, and learn beyond generated code.
Method
A controlled study assigned 69 learners aged 10–17 with no prior text-based programming experience to learn Python with or without Codex across 45 code-authoring tasks and subsequent modification tasks.
Results
Codex improved code-authoring performance while leaving manual code-modification performance unchanged, and learners with higher prior programming competency performed significantly better on retention when they had used Codex during training.
Takeaways & Limitations
AI code generators can support novice programming by enabling better and faster code writing without reducing performance on manual modification or when the generator is unavailable.
Takeaways & Limitations
The study focused on novice learners in introductory programming, leaving implications for experienced programmers and formal high-school or university settings unresolved.
Abstract
from arXiv · showhide
AI code generators like OpenAI Codex have the potential to assist novice programmers by generating code from natural language descriptions, however, over-reliance might negatively impact learning and retention. To explore the implications that AI code generators have on introductory programming, we conducted a controlled experiment with 69 novices (ages 10-17). Learners worked on 45 Python code-authoring tasks, for which half of the learners had access to Codex, each followed by a code-modification task. Our results show that using Codex significantly increased code-authoring performance (1.15x increased completion rate and 1.8x higher scores) while not decreasing performance on manual code-modification tasks. Additionally, learners with access to Codex during the training phase performed slightly better on the evaluation post-tests conducted one week later, although this difference did not reach statistical significance. Of interest, learners with higher Scratch pre-test scores performed significantly better on retention post-tests, if they had prior access to Codex.
1 INTRODUCTION
The paper examines whether novice learners can use AI code generators effectively in introductory programming without harming code understanding, manual modification, learning, or retention. A controlled study compares learners with and without Codex access across training and evaluation tasks.
- AI code generators could help learners focus on computational thinking by reducing struggles with syntax and semantic errors.
- Potential risks include dependence, difficulty expressing intentions, and limited understanding of generated code for later modification.
- Prior work had not examined AI code generators from a learning perspective in introductory programming, leaving novice understanding, modification, reliance, and tool-free performance unresolved.
- The study involved 69 learners ages 10–17 with no prior text-based programming experience, split between Codex-supported and unsupported Python learning.
- Learners completed 45 Python authoring tasks followed by modification tasks during training, then immediate and one-week retention evaluations.
- Codex access improved authoring performance—1.15x increased progress, 0.59x fewer errors, 1.8x higher correctness, and 0.57x less time—without reducing manual modification performance.
2 RELATED WORK
Related work frames AI code generation as a promising but understudied aid for introductory programming. Existing research covers natural-language programming, AI assistant capabilities, educational applications, and novice learning challenges, but not the learning effects examined here.
- Natural-language programming seeks to make programming closer to human expression, but ambiguity and abstraction make code generation challenging.
- Large language models trained on extensive code corpora support code generation, completion, translation, repair, summarization, and explanation.
- Programming can overwhelm novices, increase time demands, and create frustration, motivating tools that reduce cognitive load during introductory learning.
- Block-based and other assistive environments lower syntax barriers, but learners may still need to transition to more powerful text-based languages.
- Prior studies found mixed practical evidence: Codex outperformed most students on first-year code-writing questions, while Copilot did not improve experienced programmers’ completion time or success rate.
- The learning implications of AI code generators for introductory programming remained an open question from the learner’s perspective.
3 AI-ASSISTED LEARNING ENVIRONMENT
Coding Steps is a web-based Python learning environment combining novice-oriented programming support with an optional Codex code generator. Its task sequence and prompt design were intended to scaffold progressively harder programming work.
- Coding Steps presents task descriptions, examples, a code editor, console, novice-oriented Python documentation, and an AI code generator in one environment.
- The application supports authentication, progress storage, logging, feedback, code execution, and communication with OpenAI Codex through a client-server architecture.
- Learners could enter desired behavior in natural language, after which generated code was inserted at the cursor’s line in the editor.
- Each Codex prompt included six static examples, the current editor code, and the learner’s requested behavior to condition generation on novice-level explanations and context.
- The system logged edits, console activity, documentation access, prompts, and generated code for analyzing learner behavior.
- The 45 two-part tasks gradually increased complexity across programming topics to avoid overwhelming learners and keep challenges within their Zone of Proximal Development.
- Codex correctly solved 41/45 authoring tasks without changes to the task description; three required minor prompt modifications and one omitted the random module when randint was used.
4 USER STUDY
The user study compared Codex-supported and baseline novice learners across a three-week, ten-session learning sequence. Participants completed Scratch preparation, Python training with authoring and modification tasks, and evaluation tests without assistance.
- Study design: The study used a between-subject design with Codex and Baseline conditions across ten 90-minute sessions over three consecutive weeks.
- Introduction phase: The introduction phase taught programming and computational thinking with Scratch, followed by a 25-question Scratch pre-test.
- Training phase: During seven training sessions, learners completed 45 programming tasks and 40 multiple-choice questions covering Python basics, data types, conditionals, loops, and lists.
- Training phase: Each programming task required code authoring and code modification after learners reviewed descriptions and input-output examples.
- Training phase: Researchers synchronously accepted correct submissions or returned incomplete solutions with personalized feedback, while skipped authoring tasks supplied correct starter code for modification.
- Evaluation phase: The immediate post-test occurred one day after training, and evaluation included five authoring tasks, five modification tasks, and 40 multiple-choice questions without generators, documentation, or feedback.
- Participants: Of 90 starters, 69 learners ages 10–17 completed all three study phases and were included in the analysis.
- Participants: The 69 completers included 33 Codex and 36 Baseline learners with similar Scratch pre-test means: 62.7% versus 60%.
5 RESULTS
During training, Codex access improved completion, correctness, time, and error outcomes on authoring tasks, while modification and multiple-choice performance was generally similar between groups.
- Overall Progress: Codex learners completed significantly more training tasks than Baseline learners: 90.9% versus 79.0%.The difference was statistically significant (p=.006, d=0.68).
- Authoring Tasks: Codex learners scored 80.1% versus 44.4% on authoring-task correctness.The difference was statistically significant (p<.001, d=1.67).
- Authoring Tasks: Codex learners completed authoring tasks in 210 seconds versus 361 seconds for Baseline learners.The difference was statistically significant (p<.001, d=1.56).
- Authoring Tasks: Codex learners encountered 1.28 errors per task versus 2.17 for Baseline learners, including fewer syntax errors.Syntax errors averaged 0.87 for Codex learners and 1.67 for Baseline learners.
- AI Code Generator Usage: Codex learners used the generator 1.21 times per task on average, with no use on 26.5% of submitted authoring tasks.Usage increased from 48% on basics to 84% on loops and arrays.
- AI Code Generator Usage: Codex prompts matched task descriptions by 41% on average, and 32% were exactly identical.This analysis covered 1,646 prompts.
- AI Code Generator Usage: Learners submitted AI-generated code without modification on 49% of tasks in which the generator was used.The pattern occurred more often on loops (60%) and arrays (52%).
- Modifying Tasks: Both groups performed similarly on manual modification tasks, with correctness scores of 66.2% for Codex and 58.4% for Baseline.Documentation access and overall errors were also similar during modification tasks.
5.2 Evaluation Phase
After training, Codex and Baseline learners generally performed similarly on immediate and one-week evaluation tests, although some topic-specific differences favored Codex.
- Immediate Post-Test: Immediate post-test performance was similar between groups across authoring, modifying, and multiple-choice tasks.The immediate post-test occurred one day after training.
- Immediate Post-Test: On immediate multiple-choice questions about arrays, Codex learners scored 32.2% versus 13.2% for Baseline learners.This difference was statistically significant (p=.007, d=0.68).
- Retention Post-Test: One week later, Codex learners scored 59.1% versus 49.8% on retention authoring tasks and 47.5% versus 34.8% on modifying tasks.Neither difference reached statistical significance.
- Retention Post-Test: Retention authoring-task errors were significantly higher for Codex learners: 1.58 versus 0.99 errors per task.The difference occurred on authoring tasks without starter code (p=.042, d=0.51).
- Retention Post-Test: Retention multiple-choice scores were comparable overall, while topic-level loops and arrays differences were not statistically significant.Codex learners scored 16% higher on loops and 14% higher on arrays on average.
5.3 The Effect of Prior Programming Competency
Prior programming competency moderated retention outcomes: high-performing learners with prior Codex access generally outperformed high-performing Baseline learners, unlike low-performing learners.
- Group Formation: Learners were divided into Codex-High, Baseline-High, Codex-Low, and Baseline-Low groups using pre-test scores.High groups scored 86.1% and 82.6% on average; low groups scored 42.1% and 37.3%.
- Retention Outcomes: Codex-High learners performed significantly better than Baseline-High learners on multiple retention post-test measures.Most condition differences appeared among high performers, while low-performing groups had nearly similar performance.
5.4 Qualitative Feedback
Learners generally viewed Codex as easy and supportive for learning, while some disliked receiving direct answers instead of step-by-step guidance.
- Learning and Stress: Both groups reported learning Python concepts, while Codex learners felt slightly less stressed and more eager about future computing education.Some Codex learners attributed reduced stress to saving time and reducing pressure.
- Perceptions of Codex: Codex learners generally found the generator easy to use and believed natural-language prompts required little practice.Learners described entering ordinary sentences and receiving code.
- Perceived Learning: Learners reported that generated code could make applications of Python concepts easier to learn.Some participants said they could study the generated code when they did not know how to proceed.
- Concerns: Some learners disliked that Codex provided direct answers without giving them time to think or offering step-by-step hints.This concern reflects the broader risk of dependency and limited understanding of generated code.
6 DISCUSSION
The discussion finds that novices can use Codex successfully, improving code-authoring performance without reducing subsequent manual modification or learning gains. Benefits were strongest for learners with higher prior programming competency on retention, while proposed designs aim to limit over-utilization and cognitive overload.
- RQ1: Can novices use AI code generators?: Novice learners successfully used AI code generators to solve programming tasks, typically testing generated code and sometimes generating it incrementally by subgoal.Learners wrote prompts with almost no effort on 32% of tasks but otherwise actively tested generated code before submission.
- RQ3: Manual code modification: Prior access to AI code generators did not reduce learners’ subsequent ability to manually modify code, with improvements observed on array tasks.Both groups performed similarly overall on manual modification tasks.
- RQ4: Learning and retention: AI code generators did not impede learning gains, and learners with higher Scratch pre-test scores showed significantly better retention after Codex-supported training.The study limited generator access to code-authoring tasks, while learners may also have learned Python concepts during code-modification tasks.
- RQ5: Existing programming competency: Learners with higher pre-test scores benefited more on retention, whereas lower-scoring learners still performed significantly better when authoring code with AI assistance.The authors connect this pattern with reduced frustration for highly inexperienced users and stronger retention benefits for learners with greater prior competency.
- Design implications: The authors propose dividing generated code into semantic segments and requiring active mini-tasks before learners can use it, to reduce overwhelm and over-utilization.Suggested mini-tasks include Parsons problems or multiple-choice questions based on concepts in the generated code.
7 LIMITATIONS AND FUTURE WORK
The study’s limitations concern measurement simplicity, statistical uncertainty, population and setting, and untested transfer, task complexity, and AI-assistant capabilities.
- Correctness scores used a simple rubric, while more detailed subgoal-level grading could provide more tangible scores.
- Correctness differences increased for later topics such as loops and arrays, but the roles of learning effects and task complexity remain unresolved.
- Differences that did not reach statistical significance may reflect the study’s sample size, motivating larger studies in real classrooms.
- The study focused on novice learners in introductory programming, leaving other populations, formal educational settings, and non-English natural-language programming for future work.
- Evaluation post-tests stayed within trained material, so transfer to new topics, more complicated tasks, algorithmic thinking, and algorithm design remains unanswered.
- The study examined AI code generation only, not related assistant uses such as explanations or automatic syntax and semantic-error fixing.
8 CONCLUSION
AI code generators had not previously been studied for learning in introductory programming. This study concludes that they can benefit novice programmers and computer science educators.
- AI code generators allowed novice programmers to perform better and faster, indicating potential value for computer science educators and students.
A.1 Coding Steps Programming Tasks
The appendix lists Coding Steps’ Python programming tasks, progressing from basic output and variables through data types, conditionals, loops, and arrays.
- The task set covered five topic groups: basics, data types, conditionals, loops, and arrays.
- Basics: Basics tasks introduced output, variables, string concatenation, input, arithmetic operations, and random numbers.
- Basics: Learners modified basic programs by changing variable names, extending messages, and incrementally constructing strings.
- Data Types: Data-type tasks included casting integers to strings and strings to integers.
- Loops: Loop tasks required repeated input, accumulation, counting, and processing list elements to identify largest and smallest values.
- Conditionals: Conditional tasks addressed operators, divisibility, even-or-odd checks, comparisons, and handling an equality option.