Source-linked AI summary
Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses
Jaromir Savelka, Arav Agarwal, Marshall An, Chris Bogart, Majd Sakr
TL;DR
The paper addresses whether GPT-4 can pass higher-education Python assessments as earlier evidence becomes outdated and learner reliance on accessible LLM tools becomes a concern. It compares GPT-4 with earlier models across course assessments and examines errors, coding limitations, and auto-grader feedback. The study provides evidence that GPT-4 can be used to obtain passing scores on assessments that currently represent programming knowledge and skills, while important scope and task limitations remain.
Problem
Existing evidence about LLMs handling assessments designed for human test-takers must be updated after GPT-4’s release, amid concerns about learners relying on accessible tools to pass assessments.
Method
The study compares GPT-4 with earlier GPT models on MCQs and coding exercises from introductory and intermediate Python courses and analyzes incorrect answers and auto-grader feedback.
Results
GPT-4 provides the strongest evidence so far that an accessible technology can obtain passing scores on viable programming knowledge and skills assessments.
Takeaways & Limitations
Programming instructors need to prepare for widespread learner access to technology that can be used to collect passing scores on current programming assessments.
Takeaways & Limitations
Findings may not generalize beyond Python, the studied assessment types, or English-language assessments, and coding performance can depend on artifacts exceeding the model’s prompt length.
Abstract
from arXiv · showhide
This paper studies recent developments in large language models' (LLM) abilities to pass assessments in introductory and intermediate Python programming courses at the postsecondary level. The emergence of ChatGPT resulted in heated debates of its potential uses (e.g., exercise generation, code explanation) as well as misuses in programming classes (e.g., cheating). Recent studies show that while the technology performs surprisingly well on diverse sets of assessment instruments employed in typical programming classes the performance is usually not sufficient to pass the courses. The release of GPT-4 largely emphasized notable improvements in the capabilities related to handling assessments originally designed for human test-takers. This study is the necessary analysis in the context of this ongoing transition towards mature generative AI systems. Specifically, we report the performance of GPT-4, comparing it to the previous generations of GPT models, on three Python courses with assessments ranging from simple multiple-choice questions (no code involved) to complex programming projects with code bases distributed into multiple files (599 exercises overall). Additionally, we analyze the assessments that were not handled well by GPT-4 to understand the current limitations of the model, as well as its capabilities to leverage feedback provided by an auto-grader. We found that the GPT models evolved from completely failing the typical programming class' assessments (the original GPT-3) to confidently passing the courses with no human involvement (GPT-4). While we identified certain limitations in GPT-4's handling of MCQs and coding exercises, the rate of improvement across the recent generations of GPT models strongly suggests their potential to handle almost any type of assessment widely used in higher education programming courses. These findings could be leveraged by educators and institutions to adapt the design of programming assessments as well as to fuel the necessary discussions into how programming classes should be updated to reflect the recent technological developments. This study provides evidence that programming instructors need to prepare for a world in which there is an easy-to-use widely accessible technology that can be utilized by learners to collect passing scores, with no effort whatsoever, on what today counts as viable programming knowledge and skills assessments.
1 INTRODUCTION
The study examines whether GPT-4 can pass programming-course assessments and whether earlier findings about GPT models remain valid after GPT-4’s release. It frames assessment design and learner reliance as urgent concerns for CS/IT education.
- Motivation: The central concern is how to assess learners’ skills when ubiquitous tools can help them pass assessments, at least partially.The paper links this concern to broader unanswered questions about curricula, learner reliance, and the future of CS/IT work.
- Motivation: GPT-4’s release requires existing evidence about models handling assessments designed for human test-takers to be confirmed or updated.The authors present their study as an empirical response to the apparent increase in GPT-4’s capabilities.
- Study focus: GPT-4 is evaluated on MCQs and coding exercises from introductory and intermediate higher-education Python courses.The study compares GPT-4 with earlier GPT models while addressing prior research questions about assessment performance and feedback use.
- Research questions: The study asks whether GPT-4 can answer programming MCQs, handle code requiring multi-hop reasoning, solve coding tasks sufficiently for course completion, and use feedback to repair solutions.These questions span both assessment accuracy and iterative improvement through auto-grader feedback.
- Contributions: The paper contributes a comprehensive evaluation of GPT-4 across diverse Python-course assessments and an analysis of MCQs and coding tasks it answers incorrectly.The contribution updates and extends earlier experiments conducted with GPT-3 models.
2 RELATED WORK
Prior work documents broad but uneven LLM performance on programming and other educational assessments, while prompting strategies can materially affect outcomes. This paper extends that literature by evaluating the more powerful GPT-4 on realistic Python-course assessments.
- Educational programming assessments: Earlier GPT models did not pass the full spectrum of typical Python-course assessments, although they could obtain over 55% of available scores.Performance remained below 70% even on entry-level modules, and multi-hop reasoning was an observed weakness.
- MCQ performance: GPT-based systems have achieved strong results on selected MCQ datasets and specialized programming assessments outside the full-course context.Reported examples include 77.4–89.2% accuracy on several non-programming datasets and 100% accuracy on 60 computational linear-algebra MCQs.
- Coding assessments: Related studies evaluate coding systems across CS1, CS2, software-testing, and bioinformatics exercises, but this paper extends the literature with GPT-4.The prior work includes prompt-engineering studies and assessments ranging from 23 programming tasks to 184 exercises.
- Educational uses: LLMs are also studied as tools for supporting computing education, including generating programming exercises, tests, solutions, explanations, and course materials.One study reported that over 75% of generated introductory exercises were novel and suitable for university use.
- Professional benchmarks: Professional programming benchmarks show variable performance, including 28.8% first-attempt success on HumanEval and a 54.3% ranking in Codeforces competitions.These results provide related evidence from coding tasks outside formal education.
- Prompting: Prompt engineering can improve LLM performance, but semantically similar prompts may produce substantially different results and verbose prompts can reduce performance.The literature also examines chain-of-thought and least-to-most prompting for reasoning tasks.
3 DATA
The dataset combines MCQ assessments from two Python Essentials courses with project-based coding activities from a practical Python course. Assessments include auto-graded tasks whose feedback supports iterative solution improvement.
- Dataset: The study uses assessment exercises collected from three real-world Python programming courses.The courses include introductory and intermediate content and a practical, project-based offering.
- PE2: PE2 contains 148 MCQs, 83 of which include code snippets, and covers intermediate topics such as modules, exceptions, file processing, and object-oriented programming.The course uses MCQ quizzes and tests exclusively.
- PPP: PPP uses project-based learning with eight units, larger projects, and 69 coding activities distributed across 32 project tasks.Projects typically contribute around 80% of the course grade, alongside tests and reflections.
- Auto-grading: The example interaction shows GPT-4 correcting a num_pages type error, an unreported similar error, and then an average_rating type error after additional feedback.The sequence illustrates iterative improvement based on auto-grader feedback.
- Auto-grading: PPP’s auto-grader dynamically generates tests, checks assessment rules such as required JSON fields and data types, and provides feedback when submissions fail.Feedback explains why a rule was not met and hints at how to iterate toward a successful solution, while usually exposing neither all failed cases nor the exact tests.
4 EXPERIMENTAL DESIGN
The experiment compares GPT-4 with earlier GPT models on multiple-choice and coding assessments, using deterministic prompts and iterative auto-grader feedback. Coding evaluation accommodates prompt-length limits by submitting individual tasks and, when necessary, splitting loosely coupled activities.
- 4.1 Models: The study compares GPT-4 primarily with GPT-3.5, and additionally with GPT-3 for multiple-choice questions.GPT-3 is excluded from coding comparisons because it was not capable of producing decent coding solutions.
- 4.2 Experimental Design: All models use temperature 0.0 to make outputs deterministic and improve reproducibility.The authors tested varied temperatures initially but selected 0.0 for their setting.
- 4.2 Experimental Design: For MCQs, GPT-3 and GPT-3.5 receive templated prompts, whereas GPT-4 uses dialogue prompts designed to return answer choices without explanations.Questions are submitted individually through the OpenAI Python library, and each model returns one or more choices.
- 4.2 Experimental Design: Coding tasks use separate prompt templates for GPT-3.5 and GPT-4, incorporating task instructions and starter-code files into the submission context.GPT-3.5 receives embedded instructions, while GPT-4 receives them as a user message.
- 4.2 Experimental Design: After an incomplete coding submission, the models receive auto-grader feedback and resubmit revised solutions until full credit or an unchanged solution indicates impasse.GPT-3.5 receives feedback through an appended prompt addendum, while GPT-4 continues the dialogue.
- 4.2 Experimental Design: Prompt-length limits require fitting boilerplate, instructions, handouts, and generated solutions into context, so large tasks are submitted individually or split into smaller activities.The evaluation uses individual project tasks and splits loosely coupled elements when necessary.
5 RESULTS AND DISCUSSION
Across the assessments, GPT-4 substantially outperformed earlier GPT models and reached passing performance on multiple course assessments, while retaining limitations on some coding, formatting, reasoning, and question-quality cases.
- 446 of 530 MCQs (84.1%) were correctly answered by GPT-4, compared with 341 (64.3%) by GPT-3.5 and 199 (37.5%) by GPT-3.
- GPT-4 passed all five PE1 assessments with an overall score of 85% and all five PE2 graded assignments with an overall score of 89.6%.
- GPT-4 scored 77.8% on PPP multiple-choice tests and 71.7% on coding projects, performance described as close to or sufficient for passing.
- GPT-4 performed better on MCQs without code than on code-containing MCQs, scoring 90.7% versus 81.0% overall.
- Remaining weaknesses included inferred-code-intent errors, inconsistent reasoning, fine-grained formatting failures, oversized input artifacts, and questions with defensible or flawed answer specifications.
- After feedback, GPT-4’s coding score increased from 71.7% to 83.4%, although it earned only 42 of 95 points on project 6 after 34 feedback iterations.
6 IMPLICATIONS FOR TEACHING PRACTICE
The findings suggest that programming education should adapt because widely accessible GPT models can obtain passing scores on assessments that currently represent programming knowledge and skills. The authors recommend emphasizing learning, growth, ethics, and assessment designs that rely on artifacts or external tools.
- GPT models can obtain passing scores with no effort on assessments currently treated as viable measures of programming knowledge and skills.
- Instructors may shift emphasis from assessment performance toward learning experiences and skills development.
- The authors argue that designing MCQs around GPT-4’s current weaknesses is unlikely to remain effective as models improve.
- Programming tasks may be hardened by requiring artifacts beyond instructions or using external tools, although the passage is truncated before explaining the full rationale.
7 LIMITATIONS ANT THREATS TO VALIDITY
The study’s conclusions are constrained by the scope of its languages, assessment formats, and English-language materials, as well as by unexplored prompt engineering and limited transparency about GPT models.
- The authors acknowledge limitations across several areas before detailing threats to validity.
- Generalizability: The findings may not generalize beyond Python, the assessment types studied, or English-language programming education.The authors specifically mention other programming languages, open questions, oral exams, and assessments in languages other than English.
- Prompt Engineering: Prompt engineering was not explored, so the reported results should be interpreted as a lower bound of model performance.
- Information on GPT Models: Unknown training data create uncertainty about whether performance reflects assessment-solving capabilities or memorization.The authors state that limited technical information about GPT-4 also makes reproduction challenging.
8 CONCLUSIONS AND FUTURE WORK
The study evaluates GPT-4’s ability to pass typical programming-course assessments and updates earlier findings based on GPT-3 and GPT-3.5. It highlights learner overreliance as a challenge and calls for assessment strategies resilient to automatically generated solutions.
- The study analyzes GPT-4 on MCQ tests and coding exercises in introductory and intermediate programming courses.
- The paper updates evidence about whether findings from GPT-3 and GPT-3.5 remain relevant after GPT-4’s release.
- The findings highlight the risk of learners becoming overly reliant on GPT models, requiring strategies to maintain programming education’s relevance and integrity.
- Future work: Future work should develop assessment techniques resilient to automatically generated solutions, including real-time problem solving and collaborative activities.
True/False
The section presents two programming-related True/False multiple-choice questions: one about code standards and one about evaluating an arithmetic expression.
- One question asks whether developers writing code individually are expected to apply code standards.
- Both items use a True/False answer format with options A and B.
- A second question asks whether 2 + 2 != 2 * 2 evaluates to True or False.
Identify True/False Statement
The assessments ask learners to identify a false statement about Python data formats and modules, and to determine the relative lengths of copied and modified lists.
- One question contrasts pandas and csv functionality, Python’s XML support, and JSON’s representation of Python data structures.
- Another question asks whether nums or vals is longer after vals is created as a slice of nums and then appended to.
Finish Statement
The assessments test the meaning and evaluation direction of Python’s exponentiation operator, including right-sided binding in chained expressions.
- One item asks what the ** operator does, distinguishing exponentiation from alternative descriptions.
- Another asks whether 1 ** 2 ** 3 is evaluated from right to left, randomly, or from left to right.
Output
The assessments cover Python outputs, Boolean conditions, file-opening syntax, nested-loop counts, list construction, and list indexing. Together, they require tracing short programs and selecting or supplying the resulting values or code.
- Output: One program reads 2 and 4, converts both inputs to integers, and prints their sum.
- Output: A loop inserts each element of at index 0 in a new list before printing it.
- Output: The is_negative function item asks for a return condition that is True for negative arguments and False otherwise.
- Output: A file-handling item asks which with statement opens myfile and assigns file.readlines() to all_lines.
- Output: A nested-loop item asks how many times print('X') executes when the outer and inner ranges are range(1, 7) and range(2, 6).
- Output: A list-indexing question asks for palindromes[1] from ['pop', 'noon', 'madam'].