Source-linked AI summary
Exploring the Responses of Large Language Models to Beginner Programmers' Help Requests
Arto Hellas, Juho Leinonen, Sami Sarsa, Charles Koutcheme, Lilja Kujanpää, Juha Sorva
TL;DR
The paper examines whether LLMs can reliably analyze beginner programmers’ problematic code and respond to help requests. It studies course help requests and code, finding that responses are often sensible but remain unreliable because models can miss issues, invent problems, and struggle with output formatting.
Problem
The paper asks how well LLMs can identify and explain issues in beginner programmers’ problematic code submitted with help requests.
Method
The study analyzed course help requests, manually annotated issues in 150 student programs, and evaluated LLM responses against those issues.
Results
LLM responses were usually sensible and potentially helpful, but they could miss issues, report imaginary problems, and struggle with output formatting.
Takeaways & Limitations
LLMs may supplement programming teachers and teaching assistants at scale, but they should not replace teachers because their tutoring remains unreliable.
Takeaways & Limitations
The study’s findings are limited because newer models, including GPT-4, were unavailable for programmatic evaluation, although GPT-4 did not eliminate the highlighted challenges.
Abstract
from arXiv · showhide
Background and Context: Over the past year, large language models (LLMs) have taken the world by storm. In computing education, like in other walks of life, many opportunities and threats have emerged as a consequence. Objectives: In this article, we explore such opportunities and threats in a specific area: responding to student programmers' help requests. More specifically, we assess how good LLMs are at identifying issues in problematic code that students request help on. Method: We collected a sample of help requests and code from an online programming course. We then prompted two different LLMs (OpenAI Codex and GPT-3.5) to identify and explain the issues in the students' code and assessed the LLM-generated answers both quantitatively and qualitatively. Findings: GPT-3.5 outperforms Codex in most respects. Both LLMs frequently find at least one actual issue in each student program (GPT-3.5 in 90% of the cases). Neither LLM excels at finding all the issues (GPT-3.5 finding them 57% of the time). False positives are common (40% chance for GPT-3.5). The advice that the LLMs provide on the issues is often sensible. The LLMs perform better on issues involving program logic rather than on output formatting. Model solutions are frequently provided even when the LLM is prompted not to. LLM responses to prompts in a non-English language are only slightly worse than responses to English prompts. Implications: Our results continue to highlight the utility of LLMs in programming education. At the same time, the results highlight the unreliability of LLMs: LLMs make some of the same mistakes that students do, perhaps especially when formatting output as required by automated assessment systems. Our study informs teachers interested in using LLMs as well as future efforts to customize LLMs for the needs of programming education.
1 INTRODUCTION
LLMs offer scalable assistance for programming students, but their pedagogical reliability remains uncertain. This study therefore examines how well they analyze beginner programmers’ help requests and how responses vary across models and languages.
- LLMs may provide explanations, feedback, and learning support rapidly and at scale, but they are not pedagogical experts.
- The study uses actual help requests and student code from a programming course to assess LLM responses in a simulated assistance scenario.
- The research asks whether responses accurately identify code issues, differ between ChatGPT-3.5 and Codex, and change with non-English prompting.
- The study also examines pedagogically relevant response characteristics, including language style and whether models provide model solutions.
- The findings aim to clarify how current LLMs analyze beginner programs and inform productive educational use and future customization.
2 BACKGROUND
Prior computing-education research has examined LLM capabilities and risks, but classroom evidence about responding to explicit programming help requests remains limited. This study addresses that gap against a background of prompt-dependent, probabilistic model behavior.
- LLM outputs depend on prompts and learned parameters, making prompt design important for producing meaningful responses.
- LLMs have been studied for programming-problem solving, code explanation, error-message enhancement, and personalized exercise generation.
- Existing evidence suggests GPT-3 can outperform Codex on some code-explanation tasks, while LLM explanations may help novice learners.
- Classroom evaluations remain relatively rare, with most computing-education research relying on expert evaluations or laboratory studies.
- To the authors’ knowledge, prior work had not examined LLM responses to programming help requests explicitly signaling that students need assistance.
3 METHODOLOGY
The study evaluates Codex and GPT-3.5 on help requests drawn from an online introductory programming course. Researchers classified student-code issues and manually assessed generated responses across seven dimensions.
- 3.1 Context and Data: The dataset comes from an open online introductory course using Dart, standard introductory topics, browser-based programming, and automated assessment.
- 3.1 Context and Data: The platform enables students to request teacher help after an exercise submission fails automated tests, sending exercise details and source code.
- 3.2 Generating LLM Responses to Help Requests: The study focuses on 15 exercises with the most help requests, which represented more than 65% of the year’s requests.
- 3.2 Generating LLM Responses to Help Requests: Responses were generated from two models, two languages, 150 help requests, and deterministic temperature 0 settings, producing 600 responses.
- 3.3 Classification of Issues in Help Requests: Researchers annotated code issues by theme and sub-theme, then manually marked each response on seven issue-identification and response-content questions.
- 3.3 Classification of Issues in Help Requests: An initial comparison used 120 responses across models and languages; subsequent analysis concentrated on GPT-3.5 English responses after its stronger initial performance.
4.1 Issues in Help Requests
Student help requests most often concerned logic and input/output problems, with substantial variation across exercises and recurring sub-themes within each category.
- 275 issues were identified across 150 help requests, averaging 1.9 issues per request; every program had at least one issue.
- 72% of help requests involved logic errors, while 34% involved input/output problems.
- Logic errors chiefly involved conditionals, iteration, and arithmetic, occurring in 37, 30, and 23 requests respectively.
- Other logic problems included misused parameters, printing instead of returning, misplaced logic, and variables placed outside functions.
- Input/output problems centered on output formatting in 25 requests, unwanted printouts in 24, and missing printouts in 10.
- Issue distributions varied considerably by exercise, with some exercises dominated by one sub-theme and others showing mixed problems.
4.2 Performance of Different LLMs
GPT-3.5 identified actual code issues more often than Codex, but both models frequently produced false positives and failed to follow instructions against supplying code.
- 90% of cases saw GPT-3.5 identify at least one actual issue in both languages, compared with 70% for Codex in English and 33% in Finnish.
- GPT-3.5 identified all issues in approximately 55% of cases in both languages, whereas Codex did so in around 15%.
- False positives were common across all model–language pairings, occurring least often with GPT-3.5 prompted in Finnish, at 23% of help requests.
- GPT-3.5 responses practically always included code and often included model-solution-like code despite instructions not to provide sample code or tests.
4.3 Deeper Analysis of GPT-3.5 Responses
An extended analysis of 150 English GPT-3.5 responses found frequent partial issue detection, model-solution-like code, and variation across issue types and exercises.
- 123 of 150 responses identified at least one actual issue, 82 identified all actual issues, and 72 identified non-existing issues.
- Almost every response included code, and roughly two responses out of three included model-solution-quality code, even when actual issues were omitted.
- Issue types: GPT-3.5 addressed all issues more often for iteration problems than for conditional or arithmetic problems, at 73%, 35%, and 57%, respectively.
- Issue types: For input/output issues, all-issue coverage was 44% for formatting, 54% for unwanted outputs, and 50% for missing outputs.
- Exercises: Exercise-specific performance showed no obvious pattern: all-issue coverage ranged from one of ten responses to ten of ten, while the best case still had four false positives.
- False positives: One example shows GPT-3.5 correctly identifying an incorrectly initialized loop variable while also inventing an extra closing-brace issue.
4.4 Further Insights: Thematic Analysis of Researcher Notes
Researcher notes portray GPT-3.5 responses as confident and often technically helpful, but frequently pedagogically unsuitable because they provide solutions, add unplanned functionality, or mismatch explanations and code.
- Language and tone: Most responses used a confident tone, including responses whose advice was completely wrong; 27 of 150 also encouraged students with supportive phrases.
- Pedagogical suitability: Almost all responses included code, and most effectively provided model solutions, while fewer than ten of 150 were considered presentable without editing.
- Instructional alignment: Seventeen responses suggested functionality not covered in the course or course plan, including error handling, Dart null safety, and list-processing libraries.
- Illustrative response: The illustrative response encouraged a student while prescribing input storage, negativity checks, averaging, and a no-jumps output.
- Consistency: At least 20 of 150 responses contained discrepancies between their explanations and their accompanying code.
5 DISCUSSION
GPT-3.5 often identified genuine issues in beginner programmers’ help requests, but neither model reliably found every issue or avoided false positives. Responses could be useful, yet frequently included unsolicited model solutions and limited adaptation to students’ level.
- LLM performance: GPT-3.5 was more accurate than Codex, although both models remained unreliable at finding all issues and avoiding false positives.The authors characterize GPT-3.5 as effective at issue-hunting but far from reliable overall.
- LLM performance: GPT-3.5 identified at least one actual issue in 82% of English help requests and all issues in 55%.Mentioning an issue also implied suggesting how to fix it.
- LLM performance: 48% of GPT-3.5 responses reported issues that did not actually exist in the student code.Confidently stated false positives may send students into unproductive debugging.
- Language: In Finnish prompts, GPT-3.5 performed similarly to English prompts for identifying code issues, and its responses were generally understandable.The authors therefore describe non-English classroom use as potentially viable.
- Pedagogical quality: Despite explicit instructions, almost every GPT-3.5 response included code, and two out of three essentially provided a model solution.A parser that removes code was suggested as one possible mitigation.
- Pedagogical quality: The responses showed little evidence of adapting technical feedback to beginners’ prior knowledge or the introductory-course context.The study did not provide the models with student-specific prior-knowledge information.
5.3 The Need to Comprehend Code
The discussion connects LLM use with the need for programmers to comprehend and debug code, including code produced by LLMs. It also shows that student difficulties and study validity depend strongly on context and rapidly changing models.
- The Need to Comprehend Code: LLM proliferation reinforces program comprehension as a key skill because programmers must understand and debug code created by others, including LLMs.The authors present LLMs as both part of the situation and potentially part of its solution.
- Rapid model evolution: GPT-3.5 outperformed the older Codex on responding to student help requests, illustrating how rapidly model evolution can affect performance studies.The authors question how long results from LLM performance studies remain valid.
- Rapid model evolution: Codex results became nearly impossible to replicate after OpenAI announced that its API support would be discontinued.The authors argue that this highlights the importance of open LLMs that can run locally.
- Student help seeking: Only 831 of 120,583 course submissions, or 0.7%, had an associated help request.This pattern was not explained simply by students submitting correct solutions, since only 47% passed all tests.
- Context-dependent issues: Input/output issues appeared in 34% of help requests and were especially common early in the course.Some formatting errors apparently arose when students copied output formats from similar worked examples.
- Context-dependent issues: Syntax errors appeared in only 8.0% of help requests, sometimes alongside other issue types.The authors suggest the sample may contain few syntax errors because students submitted code for marking.
6 LIMITATIONS
The findings have limited generalizability because the study used a specific online Dart course and a selective help-request sample. Additional constraints include outdated model access, single-coder qualitative analysis, and one-shot prompting.
- Scope: The study’s generalizability is limited by its online Dart course and student cohort, whose requested issues may differ from other introductory programming courses.Dart may also have been less represented in the models’ training than more common languages.
- Sampling: Only a minority of code submissions had help requests, and students who used the built-in functionality may differ from the broader course population.Other students may have relied on course materials or internet searches instead.
- Model scope: The evaluated models were already outdated relative to newer systems, and the authors lacked programmatic access to GPT-4.The authors report anecdotally that GPT-4 outperformed GPT-3.5 to some extent but did not eliminate the highlighted challenges.
- Analysis: The qualitative analysis used a single coder, which the authors identify as a threat to reliability.
- Interaction design: The study relied on a single request to each model rather than an ongoing conversation with evolving context.Future work proposes studying more conversational feedback approaches.
7 CONCLUSIONS
The study evaluates Codex and GPT-3.5 on real introductory-programming help requests, finding useful but unreliable responses. LLMs may supplement teaching at scale, but their findings and advice require oversight and remain context-bound.
- OpenAI Codex and GPT-3.5 were evaluated by analyzing code accompanying students’ help requests from an online programming-basics course.
- The responses were usually sensible and potentially helpful, but GPT-3.5 identified issues better while both models missed issues, reported imaginary problems, and sometimes misled students.
- Output formatting was difficult for the LLMs, whereas responses in a fairly uncommon non-English language were only slightly worse than English responses.
- LLMs could serve as scalable supplements for programming teachers and teaching assistants, but they should not replace teachers because their pedagogy is unreliable and they often provide model solutions when asked not to.
- Students infrequently requested help despite failing submissions, and most observed issues involved program logic or input/output; these findings are specific to the studied context.
- LLMs are already part of programming education, motivating educators and researchers to use them productively while avoiding pitfalls and exploring tools customized for student programmers.