Source-linked AI summary
Automatic Generation of Programming Exercises and Code Explanations using Large Language Models
Sami Sarsa, Paul Denny, Arto Hellas, Juho Leinonen
TL;DR
Programming educators need scalable ways to create novel exercises, solutions, tests, and code explanations. This paper evaluates OpenAI Codex for generating these resources and finds substantial promise alongside quality issues requiring human oversight.
Problem
Creating enough novel programming exercises with reliable solutions and tests is difficult, while code walkthroughs can be demanding for students and not always feasible with human collaborators.
Method
The paper evaluates OpenAI Codex by generating programming exercises with sample solutions and automated tests, testing keyword-based priming, and producing step-by-step code explanations.
Results
Codex generated novel and sensible programming exercises with sample solutions and tests, influenced both programming and contextual topics through keywords, and produced explanations covering 90% of code.
Takeaways & Limitations
Codex may help instructors brainstorm and facilitate exercise creation, while tailored keywords could support exercises adapted to individual students’ contexts.
Takeaways & Limitations
The qualitative analysis used relatively few examples and prompt types, novelty searches could miss inaccessible or deleted sources, and generated exercises often required adjustments because of missing corner cases or faulty tests.
Abstract
from arXiv · showhide
This article explores the natural language generation capabilities of large language models with application to the production of two types of learning resources common in programming courses. Using OpenAI Codex as the large language model, we create programming exercises (including sample solutions and test cases) and code explanations, assessing these qualitatively and quantitatively. Our results suggest that the majority of the automatically generated content is both novel and sensible, and in some cases ready to use as is. When creating exercises we find that it is remarkably easy to influence both the programming concepts and the contextual themes they contain, simply by supplying keywords as input to the model. Our analysis suggests that there is significant value in massive generative machine learning models as a tool for instructors, although there remains a need for some oversight to ensure the quality of the generated content before it is delivered to students. We further discuss the implications of OpenAI Codex and similar tools for introductory programming education and highlight future research streams that have the potential to improve the quality of the educational experience for both teachers and students alike.
1 Introduction
Introductory programming courses depend on many short exercises, automated feedback, and explanations, but creating sufficient novel, high-quality resources is difficult. This paper investigates OpenAI Codex for generating exercises, solutions, tests, and feedback within that lifecycle.
- 1 Introduction: Many short programming exercises support active learning and mastery through repeated practice and reflection.Auto-graders can provide immediate feedback after submission.
- 1 Introduction: Developing useful exercises requires problem statements, sample solutions, and automated tests, creating a substantial workload for educators.Large exercise pools are especially tedious to develop, while crowdsourcing introduces motivation and content-quality concerns.
- 1 Introduction: OpenAI Codex automatically generated a programming exercise, solution, explanation, and test suite whose problem description appeared entirely novel.The generated example targeted object-oriented programming and dictionary manipulation in Python.
- 1 Introduction: Unlike prior work focused on Codex solving CS1 questions, this work explores opportunities for generating learning resources and feedback for instructors and students.The authors position Codex as one tool in an introductory programming teacher’s toolbox.
- 1 Introduction: The study examines whether Codex can create sensible, novel exercises and provide comprehensive, accurate explanations of introductory programming solutions.These aims correspond to research questions RQ1 and RQ2.
2 Background
Introductory programming education combines deliberate practice, feedback, and code reasoning, while emerging language models offer new ways to generate code-related resources. Code explanations remain especially challenging because they vary in abstraction and are harder to assess.
- 2 Background: Programming courses use scaffolding and deliberate practice to introduce concepts gradually without overwhelming students.Many small exercises aim to balance difficulty, utility, motivation, and self-efficacy.
- 2 Background: Automated assessment provides correctness feedback, but formative feedback must be designed carefully because it can affect self-efficacy and learning.Formative feedback can support self-regulated learning and metacognition.
- 2 Background: Explaining code is a key novice skill that develops between lower-level tracing and higher-level code writing.This progression motivates studying explanations as part of programming instruction.
- 2 Background: Code explanations are harder to assess than tracing or writing because natural-language responses vary in abstraction.The SOLO taxonomy distinguishes line-by-line multistructural explanations from higher-level relational explanations of purpose.
- 2 Background: Line-by-line explanations may assist debugging by helping students inspect incorrect code and reducing the cognitive demands of walkthroughs.Such explanations can be useful even when higher-level purpose descriptions are premature for buggy code.
- 2 Background: Codex is a GPT-based model fine-tuned on publicly available GitHub code to translate between natural language and source code.Related systems include CodeBERT and AlphaCode, while Codex has also been studied for code repair and problem generation.
- 2 Background: This paper applies Codex beyond code generation to produce exercise variations and natural-language feedback on student code.The proposed use cases align with the programming-exercise lifecycle.
3.1 Using Codex
Codex generates content from user-supplied prompts, or primings, and its output can be controlled with generation settings. Existing exercises and contextual words can guide the model toward related content.
- 3.1 Using Codex: Users provide Codex with a priming prompt, and the model generates new content based on that input.A natural-language behavior description can prompt Codex to generate corresponding source code.
- 3.1 Using Codex: A custom stop sequence ends generation at a specified string, while maximum tokens and temperature control length and randomness.Lower temperature reduces the likelihood of less probable tokens and therefore reduces randomness.
- 3.1 Using Codex: Priming Codex with an existing exercise and context-related words guides it toward creating a similar exercise with the specified context.The prompt functions as an input signal for the content’s topic and theme.
3.2 Creating Programming Exercises and Code Explanations
The study generated programming exercises by priming OpenAI Codex with exercise materials and optional programming or contextual concepts, and generated step-by-step code explanations with structured prompts.
- Creating programming exercises: The researchers selected four introductory programming exercises as priming material, all written in Python because Codex had primarily been evaluated with Python.They used the code-davinci-001 model, described as the most capable but slowest version during the experiments.
- Creating programming exercises: The most reliable exercise-generation priming included a problem description, sample solution, and automated tests.The researchers also varied programming-related concepts and contextual concepts in the prompts.
- Creating programming exercises: Exercise generation varied contextual concepts, programming-related concept sets, exercise primings, and Codex temperature values across combinations of inputs.The input design included 10 contextual-concept conditions, three programming-concept conditions, and two exercise primings before temperature variation.
- Creating code explanations: The code-explanation study compared high-level, problem-statement-like, and step-by-step primings, focusing on step-by-step explanations.The selected format aligned with the multistructural level of understanding described in the study.
- Creating code explanations: A prompt containing source code, a stop sequence, and “Step-by-step explanation of the above program: 1.” tended to produce step-by-step explanations.The researchers generated five explanations for each of four sample solutions, producing 20 explanations at temperature 0.
3.3 Evaluation
The evaluation combined qualitative review of generated exercises with programmatic checks of executability, tests, and coverage, while code explanations were assessed for completeness and line-level correctness.
- Programming exercises: The programming-exercise evaluation used mixed methods, combining qualitative and quantitative analysis.The qualitative analysis assessed sensibleness, novelty, and readiness for use, while the quantitative analysis examined executable solutions, tests, and coverage.
- Programming exercises: The qualitative exercise analysis focused on a random sample of 120 exercises and assessed whether their problems were sensible, novel, and ready for use.Sensibleness concerned whether the exercise described a practical problem students could solve; novelty concerned verbatim matches to existing exercises.
- Programming exercises: Four researchers used a rubric to assess exercises independently, discussing ambiguous Maybe ratings in pairs to reach consensus.The rubric used Yes, No, and Maybe judgments with accompanying notes.
- Programming exercises: Programmatic analysis covered all 240 exercises and tested whether solutions ran, whether solutions passed automated tests, and the statement coverage of those tests.These checks were obtained programmatically using the automated assessment rubric.
- Code explanations: The code-explanation analysis examined whether every code part was explained and calculated the proportion of correctly explained lines.Researchers jointly evaluated 20 explanations and applied a strict standard requiring precise language for each line.
4 Results
Generated exercises were often sensible, novel, and runnable, but test alignment was weaker; code explanations usually covered all code while achieving lower line-level correctness.
- Programming Exercises: 75.0% of manually evaluated exercises were sensible, 81.8% were novel, and 76.7% had a matching sample solution.Reported issues included redundant or missing information, incorrect sample values, mismatches between statements and solutions, and tests that would not pass.
- Programming Exercises: 51 of 165 exercises with both a sample solution and automated tests had solutions that passed the tests.Among those 51 cases, 48 achieved 100% statement coverage and average coverage was 98.0%.
- Programming Exercises: Generated sample solutions were runnable around 90% of the time, tests were generated and auto-extractable roughly 70% of the time, and about 30% of solutions passed the tests.When tests passed, coverage was high; several failing tests required only minor tweaks, while missing tests could sometimes be produced on a second run.
- Code Explanations: 90% of the 20 code explanations covered all code parts, but only 117 of 174 line-by-line explanations were correct (67.2%).Incorrect explanations commonly concerned comparison and branching conditions, including reversed inequality meanings and misinterpreted modulo tests.
5 Discussion
Codex generated many sensible, novel programming exercises and generally thorough code explanations, but test quality and explanation accuracy still require oversight. The discussion highlights contextualized exercise generation, instructor support, validity limits, and future work on validation and broader scope.
- 5.1 Programming Exercises: Most generated exercises were sensible and novel, with executable sample solutions, but only around 70% included tests and fewer than a third of those test suites passed successfully.The authors suggest stepwise prompting and repeated automated regeneration as possible ways to improve test quality.
- 5.1 Programming Exercises: Codex incorporated programming concepts and contextual themes into materially different problem descriptions, including relationships, hiking, books, football, and health.The generated contexts were combined with programming requirements rather than appearing as trivial variations of the priming exercises.
- 5.1 Programming Exercises: Keyword-based contextualization could help instructors target specific constructs and let students receive exercises tailored to their interests or backgrounds.The authors identify tailored exercise generation as a future research direction, particularly for diverse first-year cohorts.
- 5.2 Code Explanations: Line-by-line explanations usually addressed essential code elements thoroughly, but minor inaccuracies questioned their suitability for direct understanding or debugging.The authors consider these explanations more useful as starting points for teaching-assistant discussions than as fully reliable standalone resources.
- 5.2 Code Explanations: Prompting Codex for step-by-step explanations produced mostly multistructural descriptions, while some responses reached a more abstract relational level and one Rainfall response was nonsensical.The paper presents higher-level explanation generation as an open question for future work.
- 5.3 Future Work: The findings motivate an instructor-facing generator with validation for sample solutions and tests, while the study’s scope remains limited by small prompt samples, English and Python focus, and mainly small exercises.The authors also stress human moderation because generated materials may perpetuate biases, and they do not claim that language models can replace teachers.
6 Conclusion
The study examined whether OpenAI Codex could help instructors create programming exercises and generate code explanations. Most exercises were sensible and novel, while explanations covered much of the code but still required oversight.
- The study investigated Codex for creating programming exercises and generating useful source-code explanations.
- Most generated programming exercises were sensible, novel, and included appropriate sample solutions.
- Exercise topics and contextual themes were easily influenced, but exercises rarely required no adjustments before course use.Problem statements sometimes omitted corner cases, and tests were sometimes missing or faulty.
- Codex explanations covered 90% of the code, while 67.2% of explanation lines were correct.The authors report that most erroneous lines contained minor mistakes that instructors or teaching assistants could fix.
- The authors conclude that generative models offer opportunities for programming-course designers despite accuracy and quality issues requiring human correction.They also identify potential challenges discussed in prior work and anticipate further improvements as models evolve.
A Sample solutions to programming exercises outlined in 3.2.1
The appendix presents sample Python solutions for conditional logic, iteration, and currency conversion exercises. The examples show both standalone functions and a class-based converter with tests.
- The speeding_check solution uses conditional branches to return different messages for speed thresholds.
- Tests call speeding_check with speeds 88, 110, and 130.
- The iteration example skips numbers whose values fall outside the range 0 to 1000 after printing an invalid-input message.
- The currency-conversion solution defines a Converter class initialized with exchange rates.
- Converter.convert converts through USD by dividing by the source rate and multiplying by the target rate.
B Sample primings for programming exercise generation
The sample primings demonstrate how keywords can steer generated programming exercises toward particular concepts and themes. Examples cover conditional functions and a dictionary-based currency-conversion class.
- A cars-themed prompt asks for a speeding_check function using parameters and conditional logic.
- The speeding exercise specifies messages for speeds above 120, above 100 but below 120, and all other speeds.
- A currency-themed prompt asks for a Converter class using a dictionary of exchange rates against USD.
- The currency example converts 100 USD to 90.0 EUR using rates of USD 1 and EUR 0.9.
- The primings include sample solutions and unit tests for both exercise types.
C Complete priming example with topic theming for programming exercise generation
The complete priming example shows a cars-themed conditional exercise followed by an ice-hockey-themed variant. Both prompts specify the task, sample solution, and tests.
- The first exercise uses cars, function parameters, and conditionals to frame speeding_check.
- Its expected outputs distinguish speeds at 100, 101, and 121.
- The second generated exercise changes the context to ice hockey while retaining a threshold-based conditional structure.
- ice_hockey_check prints a penalty message when score is greater than 5 and otherwise prints a positive message.
- Tests check the ice-hockey function at scores 5 and 6.