Source-linked AI summary
Using Large Language Models to Enhance Programming Error Messages
Juho Leinonen, Arto Hellas, Sami Sarsa, Brent Reeves, Paul Denny, James Prather, Brett A. Becker
TL;DR
Programming error messages are difficult for novices to interpret, and improving them at scale remains challenging. This study uses Codex to generate explanations and actionable fixes for Python error messages, finding that these enhancements sometimes improve on original messages but are not yet ready for production classroom use.
Problem
Programming error messages remain difficult for novices to understand, while making them readable at scale and across languages remains challenging.
Method
The study used Codex to generate explanations and actionable fixes for selected Python error messages across program complexities and temperature settings, then evaluated them for classroom utility.
Results
Codex explanations were comprehensible in 88% of cases, while correct explanations covered 48% and correct fixes covered 33% of all inputs.
Takeaways & Limitations
LLMs could help improve programming error messages, but additional work is needed to produce high-quality enhancements and support novice debugging.
Takeaways & Limitations
The study used author-created, relatively short Python 3.6 programs with singular errors rather than student code.
Abstract
from arXiv · showhide
A key part of learning to program is learning to understand programming error messages. They can be hard to interpret and identifying the cause of errors can be time-consuming. One factor in this challenge is that the messages are typically intended for an audience that already knows how to program, or even for programming environments that then use the information to highlight areas in code. Researchers have been working on making these errors more novice friendly since the 1960s, however progress has been slow. The present work contributes to this stream of research by using large language models to enhance programming error messages with explanations of the errors and suggestions on how to fix the error. Large language models can be used to create useful and novice-friendly enhancements to programming error messages that sometimes surpass the original programming error messages in interpretability and actionability. These results provide further evidence of the benefits of large language models for computing educators, highlighting their use in areas known to be challenging for students. We further discuss the benefits and downsides of large language models and highlight future streams of research for enhancing programming error messages.
1 INTRODUCTION
Programming error messages are difficult for novices to understand, creating challenges for learners and instructors. The study investigates whether Codex can explain these messages and suggest code fixes.
- Novices can find programming error messages difficult to decipher and spend substantial programming time trying to understand them.
- Instructors also spend considerable time helping novices interpret cryptic programming error messages.
- Although researchers have improved error-message readability, creating novice-friendly messages at scale and across languages remains challenging.
- The study asks how well Codex explains different error messages and how good its suggested code fixes are.
2 BACKGROUND
Programming error messages remain a longstanding barrier for learners, while prior enhancement efforts face effectiveness and scalability challenges. Large language models offer a potential way to generate more readable, contextualized support for these messages.
- Programming Error Messages: Programming error messages include diagnostics from compilers or interpreters indicating that code violates a language specification.
- Programming Error Messages: More than fifty years of reports identify programming error messages as a difficulty and barrier for students learning to program.
- Large Language Models: Large language models can generate human-like output from contextual inputs, creating potential for enhancing programming error messages.
- Enhanced Compiler Error Messages: Enhanced compiler-error-message research has addressed poor vocabulary, strange sentence structure, and jargon, but evidence of effectiveness remains limited.
- Enhanced Compiler Error Messages: Prior approaches require manual effort to generate rewordings and lack clear guidance for addressing readability problems.
- Large Language Models: Before this study, large language models had not been explored for improving programming error messages, despite earlier AI and machine-learning work on error repair.
3 METHODOLOGY
The study generated Codex explanations and fixes for Python error messages across program complexities and temperature settings, then evaluated their quality with experienced introductory-programming instructors. The evaluation examined comprehensibility, explanation and fix correctness, and improvement over original messages.
- Data and Program Construction: The study selected nine Python error messages previously reported as most unreadable and created three example programs for each.
- Data and Program Construction: The three programs varied in complexity from short examples to code using strings, functions, and external libraries.
- Prompt Engineering: Codex prompt engineering compared five prompts seeking plain-English explanations and actionable code fixes.
- Prompt Engineering: The first prompt was selected because it produced the fewest empty responses: 4 of 81, compared with 6, 7, 16, and 27 for the alternatives.
- Generating Programming Error Messages: 81 unique combinations arose from 9 error messages, 3 program categories, and 3 Codex outputs using temperatures 0 and 0.7.
- Evaluation: Two experienced introductory-programming researchers qualitatively rated generated messages using yes-or-no judgments across seven evaluation aspects.
- Evaluation: The evaluation measured comprehensibility, unnecessary content, explanations, explanation correctness, fixes, fix correctness, and improvement over the original message.
- Evaluation: Percentages of yes answers were calculated from 162 ratings covering two raters and 81 combinations.
4 RESULTS
Codex-generated explanations were usually comprehensible, but correctness and improvement over original messages varied substantially by error message. Lower temperature produced better outputs across program categories.
- Most Codex explanations were comprehensible, with “yes” responses ranging from 67% to 100%.
- Correct explanations ranged from 11% for “unexpected EOF while parsing” to 83% for “can’t assign to function call”.
- Codex supplied fixes in 44% to 89% of cases, but fixes were correct only 33% of the time overall.
- 54% of Codex explanations and proposed fixes were judged improvements over the original error messages.
- Outputs were considered improvements in over 70% of cases at temperature 0, compared with at most 50% at temperature 0.7.
- Codex performed slightly worse on “function with strings” programs, although temperature and error-message differences were larger.
5 DISCUSSION
Codex often produced more interpretable error-message explanations, but its suggested fixes were frequently incorrect and could mislead novice programmers. The authors identify recurring failure modes and propose human oversight or tiered use of LLMs.
- Are Error Message Explanations Useful: Over 70% of Codex outputs at temperature 0 were considered improvements over the original error messages.Results using temperature 0 were better overall than those using 0.7.
- Are Error Message Explanations Useful: Although 70% of outputs included a fix, included fixes were correct only 47% of the time.At temperature 0, correctness varied by program category: 42% for functions with strings, 56% for simple programs, and 61% for library-related programs.
- Explanations and Context: Incorrect suggestions may mislead novices and introduce misconceptions about indentation because the outputs use a similarly confident tone.The authors connect this risk to novices’ existing difficulty with indentation.
- Common Pitfalls and Ways Around Them: Codex commonly misdiagnosed incomplete code or capitalization errors as indentation problems.For missing quotation marks, it also confused whether the opening or closing quotation mark was absent and sometimes attributed the issue to parentheses.
- Common Pitfalls and Ways Around Them: Some outputs were contradictory and confusing even when they contained a correct explanation.One output included a correct instruction about ending a string with three quotes while simultaneously calling that explanation incorrect.
- Common Pitfalls and Ways Around Them: The authors propose a two-tiered approach: use Codex where it performs well and have instructors validate pre-generated explanations for other cases.This is presented as a possible human-in-the-loop strategy.
- Limitations: The study used Python 3.6, author-created rather than student code, and mostly short programs, limiting direct generalization to newer Python errors and student programs.The authors note that newer Python versions improve some analyzed messages and that performance may differ on student code or larger programs.
- Limitations: The evaluation relied on a single prompt requesting both an explanation and a fix without examples of good outputs.The authors suggest that separating these requests or providing examples might improve performance.
6 CONCLUSION
The study used OpenAI Codex to generate explanations and fixes for Python programming error messages, then evaluated their quality. Explanations were often comprehensible and sometimes improved on original messages, but fix correctness remained limited, so the approach is not ready for production classroom use.
- Conclusion: Codex generated explanations and actionable fixes for Python error messages, evaluated across different error messages, program categories, and prompt settings.The study used error messages identified as unreadable in prior work and applied prompt engineering before evaluation.
- Conclusion: 48% of all inputs received explanations deemed correct, while 84% received an explanatory output and 57% of those explanations were correct.Overall explanation comprehensibility was reported as 88%.
- Conclusion: 33% of all inputs received correct proposed fixes, while 70% included a proposed fix and 47% of those fixes were correct.Fix quality was substantially less reliable than the availability of proposed fixes.
- Conclusion: Temperature 0 produced better outputs across program categories, and the authors suggest a two-step system for routing cases to LLMs or traditional support mechanisms.The routing decision would consider the error message and source-code complexity.
- Conclusion: LLM-generated explanations and fixes are not ready for production use in introductory classes because faulty outputs may be treated as authoritative and lead students toward unhelpful changes.The authors nevertheless report that LLMs could be useful for improving programming error messages with additional quality-control effort.