Source-linked AI summary
Prompt Problems: A New Programming Exercise for the Generative AI Era
Paul Denny, Juho Leinonen, James Prather, Andrew Luxton-Reilly, Thezyrie Amarouche, Brett A. Becker, Brent N. Reeves
TL;DR
As LLMs automate code generation, computing education needs ways to teach code evaluation and effective prompting. This paper introduces Prompt Problems and Promptly, then reports initial classroom experiences showing mostly positive student responses alongside resistance and anxiety about AI-generated code.
Problem
LLM code generation increases the importance of code comprehension, evaluation, and effective prompt formulation, but concrete ways to teach prompt engineering remain underdeveloped.
Method
The authors design Prompt Problems, build the Promptly web tool for generating and testing prompt-produced code, and deploy it in CS1 and CS2 courses.
Results
Most students solved Prompt Problems in a few attempts, reported positive experiences, and valued exposure to programming constructs and problem-solving skills.
Takeaways & Limitations
Prompt Problems can provide a classroom activity for practicing prompts while engaging computational thinking and exposing students to new programming constructs.
Takeaways & Limitations
Students differed in their responses, with some resisting automated code generation and a few expressing anxiety about AI’s implications for programming careers.
Abstract
from arXiv · showhide
Large Language Models (LLMs) are revolutionizing the field of computing education with their powerful code-generating capabilities. Traditional pedagogical practices have focused on code writing tasks, but there is now a shift in importance towards code reading, comprehension and evaluation of LLM-generated code. Alongside this shift, an important new skill is emerging -- the ability to solve programming tasks by constructing good prompts for code-generating models. In this work we introduce a new type of programming exercise to hone this nascent skill: 'Prompt Problems'. Prompt Problems are designed to help students learn how to write effective prompts for AI code generators. A student solves a Prompt Problem by crafting a natural language prompt which, when provided as input to an LLM, outputs code that successfully solves a specified programming task. We also present a new web-based tool called Promptly which hosts a repository of Prompt Problems and supports the automated evaluation of prompt-generated code. We deploy Promptly for the first time in one CS1 and one CS2 course and describe our experiences, which include student perceptions of this new type of activity and their interactions with the tool. We find that students are enthusiastic about Prompt Problems, and appreciate how the problems engage their computational thinking skills and expose them to new programming constructs. We discuss ideas for the future development of new variations of Prompt Problems, and the need to carefully study their integration into classroom practice.
1 INTRODUCTION
As LLMs make code generation easier, computing education increasingly needs to emphasize code comprehension, evaluation, and prompt formulation. The paper introduces Prompt Problems and Promptly to teach and assess these emerging skills.
- LLM code generation is shifting introductory education beyond code writing toward reading, understanding, and evaluating generated code.
- Effective prompt writing can require students to communicate problem-solving steps, drawing on computational thinking skills.
- Prompt Problems ask students to formulate natural-language prompts that guide an LLM to generate code solving a programming task.
- Promptly delivers Prompt Problems, converts student prompts into code through an LLM API, and executes the code against test cases.
- The authors report observations from deploying Prompt Problems in one CS1 course and one CS2 course.
2 RELATED WORK
Related work has moved from evaluating LLM capabilities toward integrating LLMs into teaching and studying prompting practices. This paper contributes to that shift by developing a learning activity and supporting tool for prompt-based programming.
- Early computing-education research on LLMs focused largely on capabilities, cheating concerns, and comparisons between LLM and student performance.
- Recent work examines LLMs as educational tools, including exercise generation, test cases, explanations, and guarded teaching assistance.
- Prompting research includes frameworks for constructing prompts from reusable patterns and datasets of novice-written prompts for benchmarking and tool development.
- Instructor interviews indicate that some educators are exploring AI tools for personalized student help and assistance with time-consuming tasks.
3 PILOT STUDY
A pilot study tested whether students could communicate a visually presented programming problem to ChatGPT through prompts. Even graduate students encountered difficulties, motivating explicit prompt-writing practice.
- The pilot presented graduate students with a visual input-output problem to solve by querying ChatGPT for a program.
- The problem was shown visually to discourage copying and pasting a textual description and encourage students to formulate prompts themselves.
- Fifteen graduate students participated and submitted open-response reflections about the activity.
- Students wrote incomplete prompts, attempted conversational refinement, and sometimes described the wrong algorithm.
- These difficulties suggested that learners could benefit from practice understanding problems, writing complete prompts, and checking generated code against test coverage.
4 PRACTICING PROMPT PROBLEMS
Promptly implements Prompt Problems as visual programming tasks whose generated code is evaluated automatically, and the authors deployed it as an optional laboratory activity in introductory courses. The evaluation examined tool interactions and student perceptions.
- 4 PRACTICING PROMPT PROBLEMS: Promptly currently supports Prompt Problems where learners cannot edit generated code, requiring prompts to be complete and self-contained.
- 4.1 Tool Design: Each course repository contains visual problem representations and associated test cases for verifying LLM-generated code.
- 4.1 Tool Design: Students complete a partial prompt, send it to ChatGPT, and receive code-only output generated from their text plus system-added instructions.
- 4.1 Tool Design: Generated responses run in a sandbox against a test suite; failed submissions reveal the first failing test and can be resubmitted after prompt edits.
- 4.1 Tool Design: The interface requires students to solve each problem before progressing to the next one.
- 4.2 Classroom Evaluation: The authors deployed Promptly as an ungraded, optional laboratory task in one Python-based CS1 course and one Python-based CS2 course.
- 4.2 Classroom Evaluation: The study investigated submission counts, successful-prompt length variation, and students’ perceptions of learning programming through prompt construction.
5 EXPERIENCES
Prompt Problems revealed varied student strategies and generally positive learning experiences, while also exposing difficulties in interpreting tasks and concerns about reliance on AI. Students experimented with prompt length and wording, and many reported gains in computational thinking and familiarity with programming constructs.
- 58 CS1 students and 182 CS2 students attempted at least one Prompt Problem, with participation optional.
- CS1 students averaged 2.3, 1.8, and 7.5 attempts on Problems 1, 2, and 3, respectively, with Problem 3 appearing most difficult.Only students successful on Problems 1 and 2 progressed to Problem 3.
- Successful prompt lengths varied substantially: CS1 averages were 18.0, 47.9, and 40.7 words, while CS2 averages were 23.0, 28.3, and 34.2 words.
- Many students solved problems on their first attempt, but successful prompts ranged from concise examples to much longer submissions.The shortest successful CS2 prompts had 12 and 16 words, compared with longest prompts of 88 and 92 words.
- Generated code exposed students to unfamiliar functions and programming constructs, which many viewed positively as expanding their programming knowledge.
- Students reported that Prompt Problems strengthened analytical and problem-solving skills by requiring them to express solution steps and logic in plain language.Participants also valued focusing on logic rather than low-level syntax.
- Some students resisted the activity or worried that AI reliance could undermine independent coding, creativity, confidence, and future programming careers.The paper identifies further exploration of these concerns as an avenue for ongoing work.
6 DISCUSSION
The discussion identifies limitations and design trade-offs in Prompt Problems and Promptly, while outlining directions for broader and more rigorous classroom integration.
- LLM outputs can vary for identical prompts and may be syntactically or semantically incorrect, creating challenges beyond those associated with conventional compilers.
- Promptly evaluates whether generated programs produce expected outputs, but does not assess prompt efficiency or penalize irrelevant prompt content.A prompt is considered successful if it passes the unit tests, even when it includes unnecessary words or code constructs.
- The initial deployment likely contains considerable self-selection bias because participation was optional and students could attempt problems only after solving previous ones.
- Promptly requires a single prompt, disallows dialogue and code editing, and evaluates one LLM response at a time, reflecting trade-offs intended for introductory students.
- Because prompt creation is new, instructors may lack intuition about Prompt Problem difficulty and when to introduce these exercises.The authors call for broader student exposure and more rigorous study of how Prompt Problems should complement existing teaching practices.
7 CONCLUSION
The paper concludes that Prompt Problems are a new pedagogical approach for teaching prompt construction, supported by initial Promptly deployments in CS1 and CS2. Most students responded positively, though some needed many attempts and expressed concerns about AI-generated code and their future.
- Prompt Problems teach students to craft natural-language prompts that generate code for programming tasks.
- Promptly supported the authors’ first deployments of Prompt Problems with students in CS1 and CS2.
- Most students solved Prompt Problems in a few attempts, while some required 20 or more and successful prompts varied widely.
- Students generally valued Prompt Problems for exposing them to programming constructs and strengthening problem-solving skills, but some hesitated about automated code generation and felt anxiety about AI’s future impact.