Source-linked AI summary
Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors
Tung Phung, Victor-Alexandru Pădurean, José Cambronero, Sumit Gulwani, Tobias Kohn, Rupak Majumdar, Adish Singla, Gustavo Soares
TL;DR
Existing studies provide limited evidence about current LLMs across programming-education scenarios. This paper systematically compares ChatGPT and GPT-4 with human tutors across six scenarios using introductory Python tasks and buggy programs. GPT-4 generally outperforms ChatGPT and approaches human tutors in several scenarios, while remaining substantially weaker in grading feedback and task synthesis.
Problem
Prior work often used outdated models or focused on only specific programming-education scenarios, leaving no systematic benchmark of state-of-the-art models across a comprehensive scenario set.
Method
The study compares ChatGPT, GPT-4, and human tutors across six scenarios using five introductory Python problems, real-world buggy programs, and expert-based assessments.
Results
GPT-4 substantially outperforms ChatGPT and approaches human-tutor performance in several scenarios, but has substantial gaps with tutors on grading feedback and task synthesis.
Takeaways & Limitations
The findings identify both promising programming-education uses for GPT-4 and scenarios requiring techniques to improve model performance.
Takeaways & Limitations
The evaluation uses only two human experts, introductory Python in English, expert assessments rather than students, and no other programming languages or domains.
Abstract
from arXiv · showhide
Generative AI and large language models hold great promise in enhancing computing education by powering next-generation educational technologies for introductory programming. Recent works have studied these models for different scenarios relevant to programming education; however, these works are limited for several reasons, as they typically consider already outdated models or only specific scenario(s). Consequently, there is a lack of a systematic study that benchmarks state-of-the-art models for a comprehensive set of programming education scenarios. In our work, we systematically evaluate two models, ChatGPT (based on GPT-3.5) and GPT-4, and compare their performance with human tutors for a variety of scenarios. We evaluate using five introductory Python programming problems and real-world buggy programs from an online platform, and assess performance using expert-based annotations. Our results show that GPT-4 drastically outperforms ChatGPT (based on GPT-3.5) and comes close to human tutors' performance for several scenarios. These results also highlight settings where GPT-4 still struggles, providing exciting future directions on developing techniques to improve the performance of these models.
1 Introduction
The paper addresses the lack of comprehensive benchmarks for current LLMs in programming education by comparing ChatGPT and GPT-4 with human tutors across six scenarios. GPT-4 substantially outperforms ChatGPT and approaches human-tutor performance in several scenarios, while remaining weaker on grading feedback and task synthesis.
- Prior studies often used outdated models or examined only specific scenarios, leaving comprehensive benchmarking of state-of-the-art models unavailable.
- The study benchmarks ChatGPT, GPT-4, and human tutors across six programming-education scenarios spanning tutor, assistant, and peer roles.
- GPT-4 substantially outperforms ChatGPT and comes close to human tutors for several scenarios, but performs poorly relative to tutors on grading feedback and task synthesis.
- The evaluation uses five introductory Python problems, real-world buggy programs, and expert-based quantitative and qualitative assessments.
2 Evaluation Setup
The evaluation covers six programming-education scenarios using five introductory Python problems and real-world buggy submissions, with ChatGPT, GPT-4, and human tutors assessed by expert evaluators. The setup also records correctness through automated test suites and acknowledges constraints involving tutor scale and dataset provenance.
- Programming education scenarios: The study evaluates ChatGPT, GPT-4, and human tutors across program repair, hint generation, grading feedback, pair programming, contextualized explanation, and task synthesis.
- Evaluation data: Five short introductory Python problems were selected to vary in programming concepts and input-output specifications.
- Evaluation data: The benchmark includes 25 real-world buggy programs from public submissions, selected to represent different bug types and ranging from 4 to 31 lines.
- Evaluation data: The study cannot provide direct links to specific source submissions because the platform does not expose them.
- Metrics and evaluation process: Two human evaluators annotate outputs for each method, with results aggregated across 25 instances or five problem-specific instances.
- Preliminary problem-solving capability: GPT-4 solves all five problems, whereas ChatGPT solves four and repeatedly fails on FIBONACCI.
3 Program Repair Scenario
The program-repair scenario asks models to fix buggy Python programs using a problem description and the current code. GPT-4 reaches high correctness and approaches tutors, but does so with more edits than human tutors.
- Prompt and output generation: The repair task supplies a detailed problem description and buggy student program, seeking a corrected program as output.
- Output quality and performance metrics: Correctness is checked against the problem specification with automated test suites, while EditTokens measures token-based edit distance from the buggy program.
- Results: GPT-4’s correctness advantage over ChatGPT is generally consistent across all five problems, with its largest tutor gap on FIBONACCI.
- Results: GPT-4 achieves 88.0 Correct versus 68.0 for ChatGPT and 100.0 for Tutor, while using 36.6 EditTokens versus Tutor’s 19.0.
4 Hint Generation Scenario
The hint-generation scenario evaluates whether an AI tutor can produce correct, informative hints and explanations for students’ buggy programs. GPT-4 substantially outperforms ChatGPT but remains below human tutors overall.
- Prompt and output generation: The scenario supplies a problem description and buggy program, then generates a hint with an explanation of its reasoning.The prompt is instantiated with the problem description and student’s buggy program before output extraction.
- Output quality and performance metrics: Performance aggregates binary attributes measuring hint correctness, informativeness, and related output qualities.Human evaluators annotate generated outputs for all 25 instances.
- Results: GPT-4 scores 66.0% overall, ahead of ChatGPT’s 18.0% but below Tutor’s 92.0%.The GPT-4–Tutor gap is generally consistent across all five problems, though GPT-4 slightly exceeds Tutor on GCD.
- Results: GPT-4’s reasoning is usually correct when it succeeds in generating a high-quality hint.The results combine the overall score with HOverall and ECorrect to support this pattern.
5 Grading Feedback Scenario
The grading-feedback scenario tests whether AI can grade buggy student programs against a rubric. GPT-4 improves on ChatGPT but remains substantially behind human tutors, especially for edge-case correctness.
- Prompt and output generation: The method receives a problem description, buggy student program, and grading rubric, then produces grading points tied to that rubric.The generated content is manually reduced to the grading points used for evaluation.
- Scope: The study does not evaluate the quality of explanations that LLMs typically generate alongside grading points.The paper provides an illustrative example of such explanations separately.
- Output quality and performance metrics: Evaluation measures exact rubric-point matches and absolute score differences, including differences for specific rubric dimensions.Lower values are better for D-Total and the dimension-specific difference metrics.
- Results: The reported GPT-4-versus-Tutor pattern is generally consistent across all five problems, with the largest gap on FIBONACCI and PALINDROME.Match is reported as a percentage, with higher values indicating better performance.
6 Pair Programming Scenario
The pair-programming scenario asks AI to complete a student’s partial Python program while preserving existing context. GPT-4 substantially improves over ChatGPT but remains below Tutor and tends to make more edits.
- Prompt and output generation: The input is a problem description and partial student program, and the output is a completed program with as few changes as possible to existing lines.The prompt explicitly requests completion rather than repair of a complete buggy program.
- Output quality and performance metrics: Overall is one only when both correctness and context preservation are one, while Correct and EditLines use automated scripts and ContextKept uses manual annotation.Human evaluators annotate output quality across 25 instances.
- Results: GPT-4 scores 64.0% overall, versus 38.0% for ChatGPT and 82.0% for Tutor.GPT-4 closes about half the gap with Tutor, while its performance is slightly better than Tutor’s on GCD.
- Results: GPT-4 tends to make more edits, reducing how much context it preserves from the supplied partial program.The weakest GPT-4–Tutor gaps occur on FIBONACCI and PALINDROME, while GPT-4 slightly exceeds Tutor on GCD.
7 Contextualized Explanation Scenario
The contextualized-explanation scenario evaluates explanations of a specific part of a correct program in the context of the whole program. GPT-4 and ChatGPT perform relatively close to human tutors, aided by short, bug-free inputs.
- Prompt and output generation: The method receives a problem description, correct program, and specific program part, then generates an explanation situated in the whole program.The selected part is used as the target of the contextualized explanation.
- Output quality and performance metrics: The evaluation uses binary quality attributes such as correctness and completeness for the generated explanations.Each attribute assigns value 1 to the better outcome.
- Results: GPT-4 scores 84.0% overall and ChatGPT 72.0%, both relatively close to Tutor’s 92.0%.Across the five problems, the largest GPT-4–Tutor gap occurs on MERGESTRS.
- Results: High performance is attributed to the use of bug-free programs that are short solutions to popular problems.The inputs for this scenario are created by fixing all bugs and selecting a specific program line based on AST depth.
8 Task Synthesis Scenario
The task synthesis scenario asks models to generate new debugging tasks from problem descriptions, buggy programs, and fixes. GPT-4 improves substantially over ChatGPT but remains far below Tutor, especially when transferring bug patterns.
- Prompt and output generation: The method receives a problem description, a student’s buggy program, and line-diff fixes, then generates a new problem and buggy program.
- Output quality and performance metrics: Performance uses binary quality attributes, including Correct and Simpler, aggregated into scenario metrics.
- Results: GPT-4 scores 22.0 Overall versus 10.0 for ChatGPT and 74.0 for Tutor across the five problems.The aggregated results are reported as percentages.
- Results: The performance gap between GPT-4 and Tutor is largest for SimilarBugs, which requires understanding and transferring bugs into a newly generated program.
- Results: Overall results are generally consistent across all five problems, with the largest GPT-4–Tutor gaps on DIVISORSDIV3 and PALINDROME.
9 Concluding Discussions
The study benchmarks ChatGPT and GPT-4 against human tutors across programming education scenarios. GPT-4 substantially outperforms ChatGPT and approaches tutors in several scenarios, but important gaps remain and the study’s scope is limited.
- Results: GPT-4 drastically outperforms ChatGPT and comes close to human tutors in several programming education scenarios.
- Results: GPT-4 still has substantial performance gaps relative to human tutors in grading feedback and task synthesis.
- Limitations: The study is limited to two human experts, introductory Python education, primarily English-language evaluation, and expert-based assessments without students.
- Future work: Future work includes larger benchmarks, alternate and open-source models, symbolic or fine-tuning methods, automated prompting, and classroom studies.
Appendix
The appendix supplies illustrative examples for the six programming education scenarios and highlights settings where GPT-4 still struggles. It also includes generated solutions and describes the presentation conventions used.
- Appendix overview: The appendix presents one illustrative example for each of six scenarios to expose settings where GPT-4 still struggles.
- Appendix overview: Examples show prompt inputs followed by outputs from ChatGPT, GPT-4, and Tutor.
- Presentation conventions: Input programs are obfuscated by changing variable names and formatting while preserving the original bugs.
- Appendix contents: Appendix sections provide GPT-4 solution programs and examples for repair, hints, grading feedback, pair programming, contextualized explanation, and task synthesis.
A.1 Solution Programs for Five Problems
The appendix illustrates GPT-4 solutions and scenario-specific behavior across five programming problems. GPT-4 solves the standalone problems, but examples show failures or inefficiencies in several educational tasks.
- Solution programs: GPT-4 correctly solves all five programming problems, with generated programs passing automated test suites.
- Program repair: In program repair, GPT-4 produces a correct Fibonacci repair but makes 33 edits, whereas Tutor makes one edit.
- Hint generation: For hint generation, both ChatGPT and GPT-4 produce incorrect hints, while Tutor meets all quality criteria.
- Grading feedback: In grading feedback, GPT-4 wrongly assigns 30/30 for general correctness despite misreasoning about the program’s issue, while Tutor assigns zero on two dimensions.
- Other scenarios: The appendix also illustrates errors or quality problems in pair programming, contextualized explanation, and task synthesis.