Source-linked AI summary
Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language
Paul Denny, Viraj Kumar, Nasser Giacaman
TL;DR
Prior work found that Codex performs well on typical CS1 problems, but little was known about the problems Copilot fails and how students might use prompt engineering to resolve errors. This paper evaluates Copilot on 166 publicly accessible programming problems and finds that natural-language prompt modifications solve many initial failures, supporting prompt engineering as a potentially useful educational interaction.
Problem
Little was known about which CS1 problems code-generation models fail on or how students might interact with them when failures occur.
Method
The study evaluates Copilot on 166 CodeCheck programming problems and manually modifies unsuccessful natural-language problem descriptions.
Results
47.6% of the 166 problems were solved on the first attempt, while modified descriptions produced successful solutions for 53 of 87 initially unsolved problems (60.9%).
Takeaways & Limitations
Prompt engineering with explicit algorithmic hints was effective across almost all problem categories and may be a useful learning activity that promotes computational thinking skills.
Takeaways & Limitations
The results are constrained by non-deterministic model behavior and a dataset limited to procedural Python problems without object-oriented or functional features.
Abstract
from arXiv · showhide
GitHub Copilot is an artificial intelligence model for automatically generating source code from natural language problem descriptions. Since June 2022, Copilot has officially been available for free to all students as a plug-in to development environments like Visual Studio Code. Prior work exploring OpenAI Codex, the underlying model that powers Copilot, has shown it performs well on typical CS1 problems thus raising concerns about the impact it will have on how introductory programming courses are taught. However, little is known about the types of problems for which Copilot does not perform well, or about the natural language interactions that a student might have with Copilot when resolving errors. We explore these questions by evaluating the performance of Copilot on a publicly available dataset of 166 programming problems. We find that it successfully solves around half of these problems on its very first attempt, and that it solves 60\% of the remaining problems using only natural language changes to the problem description. We argue that this type of prompt engineering, which we believe will become a standard interaction between human and Copilot when it initially fails, is a potentially useful learning activity that promotes computational thinking skills, and is likely to change the nature of code writing skill development.
1 INTRODUCTION
Code-generation models perform well on many introductory programming problems, but their failure cases and students’ natural-language interactions after failure remain poorly understood. This study frames prompt engineering as a central question for Copilot in CS1.
- Transformer language models can generate text, images, and source code from learned patterns.Examples include GPT-3, DALL-E, and Codex.
- Copilot is a production Codex extension for development environments such as Visual Studio Code and is free to students.
- Roughly half of CS1 exam questions were solved by Codex on its first attempt in prior work.
- Little is known about which problems code-generation models fail or how students interact with them after failures.
- The study asks how Copilot performs on public CS1 problems, whether natural-language modifications rescue failures, and what failed problems share.
2 MOTIVATING EXAMPLE
A motivating example shows Copilot producing an initially incorrect average-length solution, while natural-language comments specifying the algorithmic steps guide it toward correct code. The example motivates prompt engineering as an alternative to manually fixing generated code.
- CodeCheck is a public online autograder whose exercises can support student self-practice.
- All provided test cases fail because the generated code divides the length of the search string by the number of matching strings.
- Adding comments that count matching strings, sum their lengths, and divide total length by count guides Copilot to a correct solution.
- The corrected solution assumes at least one matching string, because CodeCheck neither specifies nor tests the no-match return value.
- The example presents natural-language descriptions of computational steps as a potentially valuable pedagogical use of Copilot.
3 RELATED WORK
Related work establishes that Codex performs strongly on several programming tasks, while prompt sensitivity and increasing problem complexity make human interaction increasingly relevant. The paper extends this literature to student prompt engineering with Copilot.
- Foundation models are large self-supervised neural networks adaptable to tasks including natural-language, image, and source-code generation.
- Copilot is Codex, a GPT-3 model fine-tuned on 159GB of code from 54 million GitHub repositories.
- Prior evaluations found Codex at the 75th percentile relative to students on the same questions and able to generate multiple correct solutions.
- As problem complexity grows, more human interaction with code-generation models is likely to be needed.
- Experienced programmers rewrite natural-language problem descriptions, and strategy hints in prompts improve code-generation performance.
4 METHOD
The study evaluates Copilot on 166 public Python exercises using a test-and-reprompt protocol, then categorizes problems that remain unsolved. Failure categories include conceptual demands, missing prompt information, verbosity, and ambiguity.
- 4.1 CodeCheck Python Problems: The evaluation uses all 166 Python programming problems in the publicly available CodeCheck test bank.
- 4.1 CodeCheck Python Problems: The problems span four main categories and 23 sub-categories, including branches, strings, lists, and two-dimensional arrays.
- 4.2 Using Copilot: Researchers pasted each description into Visual Studio Code, accepted Copilot’s suggestion, tested it in CodeCheck, and recorded passed and failed cases.
- 4.2 Using Copilot: After failures, they deleted the generated code and modified only the natural-language description with clarifications or strategies until tests passed or no obvious clarification remained.
- 4.3 Categories of Copilot Failures: The study independently reviewed 34 persistent failures and assigned categories based on original prompts, engineered prompts, and manual solutions.
- 4.3 Categories of Copilot Failures: Fifteen failures were categorized as Conceptual, including every problem requiring immediate-neighbor swaps in a string or list.
- 4.3 Categories of Copilot Failures: Four failures were classified as Poor Prompts because required manual-solution code segments had no corresponding prompt content.
- 4.3 Categories of Copilot Failures: Eleven failures were attributed to Verbose Prompts, while four involved ambiguities that prompting failed to resolve.
5 RESULTS AND DISCUSSION
Copilot solved 47.6% of 166 CodeCheck problems verbatim, while natural-language prompt modifications solved 60.9% of the 87 initially unsolved problems. Performance varied substantially across categories, with prompt engineering helping most categories but failing consistently for SNL problems and some verbose or conceptual tasks.
- Initial Copilot Performance: 79 of 166 problems were solved on Copilot’s first attempt, a 47.6% success rate.The verbatim condition used the original problem description without changes.
- Effect of Prompt Engineering: 53 of the 87 initially unsolved problems were solved after modifying their natural-language descriptions, a 60.9% recovery rate.Figure 2 reports the corresponding verbatim and modified proportions by problem sub-category.
- Effect of Prompt Engineering: Performance varied widely across categories, partly because some categories were small or contained problems sharing similar solution approaches.The SNS category included three string-search problems solvable with the same split-and-iterate approach.
- Effect of Prompt Engineering: Only two of six LMRE problems were solved, and both required prompt engineering.One successful prompt explicitly suggested constructing two lists after Copilot initially swapped only the first positive and negative numbers.
- When Prompt Engineering Fails: Prompt engineering helped every category with initial failures except SNL, whose three persistent failures involved arbitrary string manipulations and special cases.The results define success by passing all available CodeCheck tests, so broader testing could have required further prompt engineering.
- When Prompt Engineering Fails: Copilot struggled with conceptual and verbose-pattern problems: terse prompts corrected some ambiguities, whereas longer instruction chains still failed on inductive pattern inference.The authors also note a tension between unpacking concepts and keeping prompts short, and suggest rewriting rather than appending to original prompts.
6 LIMITATIONS AND FUTURE WORK
The study’s scope is limited to procedural Python problems, while several future directions remain open, including observing authentic novice interactions and examining educational, ethical, and assessment impacts. Results should also be interpreted as a current baseline because Copilot is nondeterministic and rapidly improving.
- Scope: The dataset contains only procedural Python problems and excludes object-oriented and functional capabilities.
- Future work: Authentic interactions between novice programmers and Copilot remain an important avenue for future work.
- Reproducibility: Copilot’s underlying model is nondeterministic, creating challenges for replicating the study’s results.
- Reproducibility: Different code suggestions were observed when some dataset problems were revisited several weeks later.
- Future work: Future research should examine Copilot’s effects on programming teaching, learning, assessment, academic misconduct, and code-similarity detection.
7 CONCLUSION
Generative language models may change how computing courses are taught and how students learn programming. This study finds that prompt engineering can often guide Copilot toward correct solutions and may provide pedagogical value through computational reflection and natural-language articulation.
- Conclusion: Prompt engineering is presented as an important future interaction skill because language models are highly sensitive to input prompts.
- Conclusion: Roughly half of the problems were solved using the original descriptions, while more than half of the remainder were solved with explicit algorithmic hints.
- Conclusion: These interactions may support computational thinking by requiring students to reflect on failures and express abstract concepts as concrete steps in natural language.