Source-linked AI summary
Computing Education in the Era of Generative AI
Paul Denny, James Prather, Brett A. Becker, James Finnie-Ansley, Arto Hellas, Juho Leinonen, Andrew Luxton-Reilly, Brent N. Reeves, Eddie Antonio Santos, Sami Sarsa
TL;DR
Generative code-generation models challenge introductory programming education because students can easily obtain solutions to assessed problems, while their effects on learning and academic integrity remain consequential. The paper synthesizes evaluations of model performance and generated learning resources to examine challenges and opportunities for educators. It reports strong performance on many introductory tasks alongside important correctness, security, integrity, and learning risks, while identifying potential for useful educational resources and feedback.
Problem
Students can easily autogenerate solutions to programming assessments, raising questions about the adequacy of introductory programming pedagogies and assessment practices.
Method
The article synthesizes recent studies evaluating code-generation models on introductory programming problems and examining their generated exercises and code explanations.
Results
Codex scored 78.5% and 78.0% on two CS1 exams, ranking 17th of 71 students, while generated exercises were around 80% novel and often matched desired topics.
Takeaways & Limitations
Generative AI creates both challenges and opportunities for introductory computing educators, including integrity and over-reliance concerns alongside potentially lower-cost learning resources and improved feedback.
Takeaways & Limitations
Current models can generate insecure or incorrect code, so students and educators need awareness of their limitations when relying on generated solutions.
Abstract
from arXiv · showhide
The computing education community has a rich history of pedagogical innovation designed to support students in introductory courses, and to support teachers in facilitating student learning. Very recent advances in artificial intelligence have resulted in code generation models that can produce source code from natural language problem descriptions -- with impressive accuracy in many cases. The wide availability of these models and their ease of use has raised concerns about potential impacts on many aspects of society, including the future of computing education. In this paper, we discuss the challenges and opportunities such models present to computing educators, with a focus on introductory programming classrooms. We summarize the results of two recent articles, the first evaluating the performance of code generation models on typical introductory-level programming problems, and the second exploring the quality and novelty of learning resources generated by these models. We consider likely impacts of such models upon pedagogical practice in the context of the most recent advances at the time of writing.
1 INTRODUCTION
Generative AI code-generation tools are rapidly becoming accessible and capable, creating both opportunities and urgent challenges for introductory computing education. The article examines their likely effects on students, educators, assessment, and learning resources.
- Emergence of generative AI: Large language models can generate human-like prose and source code from natural-language prompts, with major organizations actively developing these tools.Codex is fine-tuned on code from over 50 million public GitHub repositories and supports multiple programming languages.
- Challenges for introductory education: Students can now autogenerate solutions to many small introductory programming exercises, challenging practice-based pedagogies and assessment approaches.Some IDE plugins allow students to accept generated suggestions with minimal effort.
- Opportunities: Generative AI also offers learners on-demand coding assistance and access to programming support without formal education opportunities.The tools are already being used by both teachers and students in classrooms.
- Article scope: The article organizes its discussion around challenges—including plagiarism, over-reliance, bias, and bad habits—and opportunities involving generated exercises, explanations, feedback, and new pedagogy.Its focus is introductory programming classrooms and the near-term impact on students and educators.
2 CHALLENGES AHEAD
The paper evaluates code-generation models on introductory programming assessments and finds strong but uneven performance. Codex performed well enough to rank near the top of a CS1 class, while errors, restrictions, underspecified outputs, and edge cases remained problematic.
- Exam performance: The evaluation used verbatim questions from two Python CS1 exams, executed Codex outputs against the students’ test cases, and corrected only trivial formatting errors.The exams covered strings, tuples, lists, dictionaries, calculations, and data manipulation.
- Exam performance: Codex scored 78.5% on Exam 1 and 78.0% on Exam 2, ranking 17th among 71 students and placing within the top quartile.Its scores were calculated using the same question weights and penalties as student scores.
- Observed weaknesses: Codex often failed problems restricting language features or requiring formatted ASCII output whose requirements had to be inferred from examples.Some failures resulted from using prohibited features such as split().
- Rainfall variants: Across 350 Rainfall responses, Codex averaged close to 50% and performed poorly when no valid input values were provided.Seven prompt variants were tested 50 times each, with every response evaluated on 10 test cases.
- Rainfall variants: Codex generated diverse program structures across prompt variants while favoring expected approaches such as for-loops for lists and while-loops for standard input.Variation was assessed using source-line counts and general algorithmic approach.
2.2 Academic Integrity
Generative code complicates academic integrity because software development already relies on reuse, while AI-generated submissions can evade conventional plagiarism detection. The paper also highlights unresolved attribution, intellectual ownership, and licensing questions.
- Detection: Common plagiarism-detection tools are often ineffective against AI-generated solutions, creating significant monitoring concerns for formal assessments.Code-structure detectors work well for copied and modified submissions but not reliably for AI-generated code.
- Detection: AI-generated solutions may appear less risky than contract cheating because they can avoid traditional similarity detection, but this changes rather than resolves integrity concerns.The passage contrasts AI-generated solutions with risks associated with outsourcing assignments to humans.
- Attribution and ownership: Educators face divergent views about acceptable attribution for reused code, compounded by opaque differences between ordinary IDE completion and Copilot suggestions.A survey of academics found a diverse range of views on code reuse.
- Attribution and ownership: AI-generated code intensifies uncertainty about how much machine-generated content can remain attributable to a student’s intellectual contribution.The issue challenges conventional interpretations of plagiarism and ownership.
- Licensing: Code-generation models may produce code requiring attribution under the licenses of publicly available training data, even when users are unaware of the obligation.The paper connects this issue to professional responsibilities when reusing code.
- Educational implications: AI-generated code creates opportunities to discuss ethics and computer use in society while potentially enabling novice learners to explore advanced ideas earlier.These opportunities coexist with concerns about integrity and responsible reuse.
2.3 Learner Over-reliance
Over-reliance on code-generation tools may weaken novices’ engagement with problem solving and metacognitive development. Generated code can also be wrong, leaving dependent learners especially vulnerable when prompting does not repair the error.
- Developing metacognition: Embedded AI suggestions may encourage novices to accept solutions without reading problem statements or reasoning through the computational steps.The developers of Codex identify user over-reliance as a key practical risk.
- Developing metacognition: Relying too heavily on code-generation tools may hinder computational thinking, metacognition, co-regulation, and socially shared regulation.These higher-order skills support problem solving and help beginners manage cognitive demands.
- When the models fail: AlphaCode produced syntactically incorrect code for 11% of Python solutions and non-compiling code for 35% of C++ solutions.Other work found that up to 20% of introductory programming problems were not solved sufficiently, even with expert prompt modification.
- When the models fail: Students overly dependent on model outputs may struggle when generated code is incorrect and cannot be fixed through natural-language prompting.Codex may suggest code that appears correct but fails to perform the intended task, with possible safety implications depending on context.
2.4 Bias and Bad Habits
Generative AI can produce biased, overly complex, and insecure code that creates particular risks for novice learners. These concerns make human oversight and awareness of current model limitations important.
- Bad habits: Developer-trained models may generate code that is too complex for novices to understand or modify.Such code may also use styles or approaches that instructors intentionally have not yet taught.
- Bias: AI code generation can reproduce negative stereotypes about gender and race in comments or identifier names.Codex developers specifically identified bias and representation issues in generated code.
- Security: Novices consistently wrote insecure code with specific vulnerabilities in string encryption and SQL injection when using AI code-generation tools.They were also more likely to believe they had written secure code.
- Security: Future models may produce more secure code than the average developer, but this remains far from certain.The paper therefore calls for increased student and educator awareness of current models’ security limitations.
3 OPPORTUNITIES AHEAD
Despite significant challenges, generative code and text models may have a transformative effect on computing education. The article frames education and exploration as potential uses and examines concrete opportunities ahead.
- OPPORTUNITIES AHEAD: Model developers identify education and exploration as possible beneficial applications of code-generation tools.Chen et al. specifically describe Codex as potentially able to aid in education and exploration.
- OPPORTUNITIES AHEAD: Code and text generation models have the potential to transform teaching and learning in computing.The article presents this potential despite the challenges educators must navigate.
- OPPORTUNITIES AHEAD: The article discusses concrete opportunities for code and text generation models in computing education.These opportunities are presented as a dedicated focus of the section.
3.1 Plentiful Learning Resources
LLMs can help educators generate programming exercises and code explanations at scale, though generated exercises still require quality checks. These tools also offer potentially novel, tailored learning resources and increasingly capable explanations.
- Implications: LLMs could reduce instructor effort for generating programming exercises and code explanations, supporting plentiful and potentially individualized learning resources.The paper describes automated filtering and tailoring to students’ interests as future directions.
- Programming exercises: Codex generated exercises from one-shot examples containing contextual themes, programming concepts, problem statements, and solutions.The priming format paired a complete example with a partial prompt for a new exercise.
- Programming exercises: 240 generated exercises were evaluated by executing their code against generated tests and measuring statement coverage.Most exercises included sample solutions and tests.
- Programming exercises: Generated sample solutions frequently failed their tests, but test suites achieved full statement coverage in all but three such cases.This indicates that automated filtering remains relevant when producing exercises at scale.
- Programming exercises: Around 80% of exercises were entirely novel, with problem-description fragments absent from search-engine indexes.A similar fraction matched the desired topics and themes.
- Code explanations: Codex explanations focused on line-by-line coverage, with 90% covering all code parts and nearly 70% meeting the reported accuracy criterion.Later work found students preferred high-level summaries to detailed explanations, while ChatGPT explanations became more consistently accurate than early Codex outputs.
3.2 Better Programming Error Messages
Programming error messages are difficult for novices to decipher, motivating LLM-generated explanations and actionable fixes. Exploratory results indicate that Codex usually produced comprehensible explanations, but fixes require correctness checking.
- Motivation: Poor programming error messages remain a longstanding problem and may contribute to novices’ perception that programming is overly challenging.Readability-focused redesigns have improved student understanding and error-message outcomes.
- LLM-generated explanations: Codex generated explanations for most collected Python error messages, and most explanations were comprehensible.The study targeted messages previously judged most unreadable and paired them with code examples producing those errors.
- Caveat: Returned fixes still needed correctness checks before being presented to students.The authors characterize the work as exploratory but identify practical potential for demystifying error messages.
3.3 Exemplar Solutions
AI-generated exemplar solutions can save instructors time and expose students to varied approaches. This may shift attention from merely obtaining correct code toward comparing quality and style across correct solutions.
- Availability: AI-generated solutions offer a time-efficient way to provide exemplars for exercises, tests, and exams that instructors may not have time to prepare.Students commonly seek exemplars to check their code or obtain help when struggling.
- Variation: Generating multiple solutions from the same prompt exposes students to different approaches for solving one programming problem.The paper connects such variation with the instructional value of appropriate variation during programming instruction.
- Assessment: Automatically generated syntactically correct solutions can shift assessment toward judging differences in solution quality and style.The paper relates this emphasis to research on the benefits of peer review and code review.
3.4 New Pedagogical Approaches
Generative AI may shift introductory programming pedagogy from low-level implementation toward specification, code modification, and testing. These approaches aim to help students begin sooner while better understanding generated code.
- From implementation to algorithms: AI-assisted implementation may let beginners focus on higher-level algorithms earlier, similar to how block-based environments abstract syntax.The proposed shift reduces emphasis on low-level implementation without eliminating higher-level programming work.
- From implementation to specification: Students could write specifications themselves, for example by inferring patterns from test suites, rather than only solving carefully specified problems.Detailed specifications currently provide useful context for code-generation models, motivating specification-focused tasks.
- Prompt engineering: Prompt engineering is an emerging skill because model outputs are highly sensitive to inputs, and algorithmic hints can improve programming-task solutions.Research on Codex found prompts describing algorithmic steps effective for some programming tasks.
- Code modification: Copilot can provide starter code that helps students overcome programmer’s writer’s block, shifting work toward rewriting, refactoring, and debugging.Students build on existing code instead of beginning with a blank editor.
- Testing and understanding: Because generated code can be opaque and untested, explicit testing may help students understand both their own code and code produced by tools.The paper connects this need to test-driven development, which supports reading, comprehending, and predicting code behavior.
3.5 Designing LLM Tools
Designers should make LLM-based programming tools support learning rather than merely generate code. Recommended features address error feedback, code volume, navigation through generated segments, and users’ mental models.
- Educational integration: Educational integrations such as auto-graders and online textbooks need guardrails and feedback adapted to maximize learning.The paper identifies integration design and learning-oriented feedback as important research areas.
- Error feedback: LLM tools should help users navigate the error-feedback loop because novices may see fewer errors yet remain poorly equipped to handle those they encounter.The recommendation targets users at all skill levels, not only beginners.
- Controlling code insertion: Users should control how much code is inserted and how they step through multi-part generated segments.Large generated blocks can intimidate novices or make users spend time reading code that does not advance their goals.
- Mental models: Because code generators are black boxes, tools should account for users’ difficulty creating correct mental models of how they work.This concern applies to programmers across skill levels.
4 WHERE DO WE GO FROM HERE?
The paper argues that computing education must address generative AI’s distinct effects on novice learners, not only its implications for professional developers. It supports responsible classroom integration while retaining code literacy and developing AI-aware pedagogies.
- Novices and code literacy: Novice programmers need code literacy because generating code from natural language does not ensure they can read, test, verify, or critically analyze it.Without these skills, users risk relying on generated content without developed expertise.
- Novices and code literacy: Writing code remains valuable for learning fundamentals, even if introductory courses place greater emphasis on modifying AI-generated code.The paper presents reading and writing code as complementary skills.
- Opportunities: Copilot and ChatGPT could explain concepts, provide exemplar code, create contextualized learning resources, and support new pedagogies.The paper frames these uses as potential assistance when the tools are harnessed correctly.
- Educator adaptation: Educators must adapt assessments, teach responsible tool use, and address generative AI’s legal, ethical, and economic implications.The paper identifies these as emerging instructional needs as tool use proliferates.
- Educator adaptation: The paper argues that classrooms should incorporate these tools from the beginning and teach responsible use through modern pedagogies.It presents adoption as a route toward a new flourishing rather than a concession.