PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond the Traceback: Using LLMs for Adaptive Explanations of Programming Errors

Alexandru-Radu Moraru, Shreyan Biswas, Ujwal Gadiraju

arXiv:2608.20896v1cs.SEcs.HC

TL;DR

Programming error messages are difficult to interpret, and it is unclear whether clearer LLM rewrites improve debugging or should vary by programmer skill. The paper evaluates skill-targeted pragmatic and contingent Python error-message rewrites in a multi-stage crowdsourced study of 103 participants. LLM rewrites improved subjective evaluations, especially for pragmatic messages, but did not reliably improve objective debugging performance, motivating interaction-aware support.

  • Problem

    Programming error messages remain difficult to interpret, and it is unclear whether clearer LLM rewrites improve debugging or align with programmer skill.

  • Method

    A multi-stage crowdsourced evaluation calibrates proficiency and tasks, then compares standard Python messages with pragmatic and contingent LLM rewrites.

  • Results

    LLM-rewritten messages, especially pragmatic messages, received more favorable subjective evaluations but did not reliably improve objective debugging outcomes.

  • Takeaways & Limitations

    Programming-support systems should evaluate behavioral debugging outcomes alongside perceived quality and consider adapting explanations to users’ interaction state and repair trajectory.

  • Takeaways & Limitations

    The proficiency assessment was study-specific and not validated at large scale against external measures of Python expertise.

Abstract

from arXiv · show

Programming error messages are critical for software development, yet they remain difficult for novice programmers to interpret. While Large Language Models (LLMs) can rewrite these errors into clearer explanations, it remains unclear whether increased readability improves objective debugging performance or how explanation styles should align with programmer skill. We present a multi-stage crowdsourced study N=103 evaluating skill-targeted, LLM-generated Python error messages. Using a custom proficiency assessment, we categorized participants by skill level and tested standard interpreter messages against two LLM-generated styles: pragmatic (action-oriented) and contingent (scaffolded explanations). We measured both objective debugging metrics (fix rate, attempts, time-to-fix) and subjective perceptions (readability, cognitive load, tone). Our results show that while LLM-rewritten messages significantly improved subjective evaluations, with pragmatic messages rated as clearer and less cognitively demanding, these perceived gains did not translate into statistically significant improvements in objective debugging performance. This highlights a critical human-AI complementarity gap: explanations that feel better to users do not necessarily make them more effective debuggers. We discuss design implications for adaptive AI feedback systems, arguing that future tools should pivot from static skill-targeted rewriting toward dynamic adjustments based on a user's real-time repair trajectory.

1 Introduction

Programming error messages are difficult to interpret and act upon, especially for less experienced programmers. This study tests whether skill-targeted LLM rewrites improve debugging performance and user perceptions compared with standard interpreter messages.

  • Error messages identify failures and locations, but users must interpret them and translate diagnostics into repair actions.
  • LLM rewrites could improve readability and actionability, but feedback may help or hinder users differently across proficiency levels.
  • The study compares pragmatic, action-oriented messages with contingent, scaffolded explanations for programmers with different assessed skill levels.
  • The evaluation combines a debugging-oriented proficiency assessment, calibrated Python tasks, and a crowdsourced experiment with 103 participants.
  • LLM-rewritten messages received more favorable subjective evaluations, especially pragmatic messages, but did not reliably improve fix rate, time-to-fix, or attempts.
  • The paper contributes a crowdsourced evaluation pipeline and evidence of a gap between perceived explanation quality and debugging effectiveness.

2 Background and Related Work

Programming diagnostics are human-facing feedback whose usefulness depends on interpretation, trust, and repair behavior. Prior work motivates readable, actionable, appropriately scoped, and expertise-sensitive messages, while LLMs create new opportunities that require human-centered evaluation.

  • Programming error messages connect automated diagnostics with human interpretation, trust, and repair actions.
  • Diagnostics can be ambiguous, technical, poorly localized, or insufficiently explanatory, increasing cognitive load and ineffective repair attempts.
  • Message difficulty varies with language design and execution model, so a single diagnostic style may not suit all programming contexts.
  • Compiler and interpreter diagnostics should identify faults, explain causes, and suggest plausible repairs using readable, actionable, appropriately scoped guidance.
  • LLMs can combine tracebacks, surrounding code, and task instructions to produce more readable, contextual, or action-oriented explanations.
  • Fluent or preferred LLM feedback may still fail to improve repair behavior, making generated diagnostics a human-centered evaluation problem.
  • Less experienced programmers may benefit from scaffolded explanations, whereas experienced programmers may prefer concise guidance that preserves workflow.
  • The study evaluates static skill-targeted rewriting rather than a tutor that updates feedback in response to user behavior.

3 Study Design and Main Experiment

The study calibrates participant skill and debugging tasks before comparing standard, pragmatic, and contingent Python error messages in a controlled crowdsourced experiment. It measures both repair behavior and subjective message evaluations.

  • Study design: The pipeline has three stages: proficiency assessment, formative task selection, and a Prolific experiment comparing standard and two LLM-generated styles.
  • Participant Skill Calibration: The proficiency assessment covers code comprehension, error-message interpretation, fault localization, and repair selection.
  • Participant Skill Calibration: Participants are assigned to lower or higher assessed proficiency using assessment scores and self-reported Python experience.
  • Task Calibration: The task pool contains self-contained Python snippets with common error families and one primary fault, supporting controlled debugging evaluation.
  • Task Calibration: Formative ratings on code difficulty, fix difficulty, mental demand, and usefulness determine four moderately difficult snippets.
  • Message Conditions: The three conditions are an unmodified standard message, a concise action-oriented pragmatic rewrite, and a more scaffolded contingent rewrite.
  • Message Conditions: Rewrites use Llama-3.1-8B-Instruct with zero-shot prompting and temperature 0, given the snippet, standard message, and line-numbered context.
  • Main Experiment: Each participant sees one snippet and one message style under constrained randomization balancing conditions and snippets.

4 Results

Across 103 participants, the study compared standard, Pragmatic, and Contingent error messages using objective debugging outcomes and subjective evaluations across assessed skill groups. LLM-generated styles were perceived more favorably, but objective performance effects were generally inconclusive and varied by snippet.

  • Study design: 103 participants attempted one of four Python debugging tasks under Standard, Pragmatic, or Contingent message conditions.
  • Fix rate: Fix rate varied substantially across snippets, with snippet A showing the lowest success rate because the standard message pointed to a non-offending line.
  • Fix rate: LLM-enhanced messages generally improved fix rates for snippet A, but effects varied widely across snippets and did not support broad conclusions.
  • Time-to-fix: Pragmatic messages produced the fastest average time-to-fix, while skill-level analyses found no significant effects and inconclusive expert results.Across all skill levels, mean time-to-fix was Standard 415.69 seconds, Pragmatic 324.56 seconds, and Contingent 423.85 seconds.
  • Fix@k: Fix@k showed limited variation because most participants either succeeded on the first attempt or failed within three attempts, preventing meaningful further analysis.
  • Subjective evaluations: Subjective evaluations favored LLM-generated styles, with Pragmatic messages associated with lower perceived cognitive load and simpler vocabulary, while both adaptive styles differed from Standard in authoritativeness.Among novices, Pragmatic versus Standard showed lower extraneous load (p=0.039); among experts, Pragmatic versus Standard showed lower intrinsic load (p=0.0049).
  • Exploratory analysis: Higher proficiency-assessment scores were moderately associated with successful debugging, with r_pb=0.368 and p=0.0001.

5 Discussion

The study found that LLM-rewritten error messages improved subjective perceptions, especially for pragmatic messages, without reliably improving debugging performance. These results expose limits of static skill-targeted rewriting and motivate evaluation and adaptation based on users’ task-specific behavior.

  • Perceived Explanation Quality: LLM-generated messages received more favorable subjective ratings than standard interpreter messages, especially for readability, cognitive load, and tone.Pragmatic messages showed the most consistent subjective benefits, while contingent messages improved several ratings less consistently and appeared more verbose.
  • Perceived Explanation Quality: Subjective improvements did not reliably translate into better objective debugging outcomes such as fix rate, time-to-fix, or Fix@k.The objective measures were variable and did not provide strong evidence that rewritten messages improved repair performance.
  • Perceived Explanation Quality: Clearer explanations may reduce reading friction while leaving fault localization, repair selection, and correct code modification unresolved.Debugging requires users to connect diagnostics to program state, reason about intended behavior, and implement the repair.
  • Adaptive Support: Contingent explanations did not consistently benefit lower-proficiency participants, challenging the assumption that more scaffolding is necessarily better for novices.Longer explanations may increase cognitive effort for users who are still struggling to understand the code.
  • Adaptive Support: Static proficiency categories may be too coarse because assessed skill does not capture local bug understanding, conceptual familiarity, or moment-to-moment repair strategy.Future systems could adapt using failed attempts, editing time and location, recurring errors, or requests for more detail.
  • Evaluation Implications: Crowdsourced calibration improved measurement of participant skill and task difficulty, but snippet difficulty and message effects still varied by error context.The authors recommend reporting effects by error type, task difficulty, and user proficiency where possible.

6 Conclusion

The study found that LLM-rewritten Python error messages improved subjective perceptions, but these gains did not reliably improve objective debugging outcomes. The findings support evaluating feedback through behavioral measures and considering dynamic adaptation beyond static skill-targeted rewriting.

  • LLM-rewritten messages, especially Pragmatic messages, were rated as more readable, less cognitively demanding, and less authoritative than standard interpreter output.
  • Objective outcomes such as fix rate, time-to-fix, and attempts did not show statistically significant improvements from the perceptual benefits.
  • Future evaluation should combine subjective perception, behavioral outcomes, task calibration, and user proficiency assessment rather than relying only on fluency, readability, or preference.
  • The proficiency assessment covered debugging-relevant competencies including code understanding, error comprehension, fault localization, and error resolution.

A.2 Pilot Procedure

The pilot developed and evaluated a debugging-oriented Python proficiency assessment before the main study. Item selection used descriptive pilot relationships and Classical Test Theory because the sample was too small for stable Item Response Theory estimates.

  • The pilot collected self-reported experience, self-assessed skill, self-estimated accuracy, and actual assessment performance for descriptive correlation analysis.
  • 60 of 78 participants completed the proficiency pilot, yielding a 76.92% completion rate.
  • Item difficulty measured the proportion correct, while point-biserial discrimination measured how well items distinguished higher-scoring from lower-scoring participants.
  • Classical Test Theory was used instead of Item Response Theory because the pilot sample was too small for stable item-response estimates.
  • Lower and higher assessed proficiency groups combined multiple-choice scores with self-reported Python experience and were not treated as complete measures of programming expertise.

B Debugging Task Selection

The debugging-task selection process generated and screened buggy Python snippets under constraints intended to support controlled repair evaluation. Candidates were manually checked for a single primary fault and selected for compatibility with semantic testing.

  • Eight candidate snippets targeted common Python error families including NameError, TypeError, and SyntaxError.
  • Candidates had to be self-contained, use only the standard library, remain at most 60 lines, and include several functions or class methods.
  • Each snippet was designed to trigger one primary error and support evaluation of the intended repair with a semantic test suite.
  • Candidates were manually checked to ensure they triggered the intended error without multiple simultaneous faults; violations were corrected or regenerated.

B.2 Task-Selection Pilot

A task-selection pilot rated candidate snippets and interpreter messages on difficulty, mental demand, and usefulness. Four final snippets were chosen to vary error type while remaining moderately difficult and suitable for controlled evaluation.

  • Participants rated code difficulty, fix difficulty, error-message mental demand, and error-message usefulness for each candidate.
  • 20 valid responses were included in the task-selection pilot after excluding one participant who failed an attention check.
  • The final four snippets provided variation in error type and fault structure while maintaining moderate perceived code and fix difficulty.
  • Selection also required interpretable standard messages, no multiple primary faults, and compatibility with hidden semantic tests.

B.4 Selected Snippets and Standard Error Messages

The study selected buggy Python snippets and paired each with an assigned error-message style for participant debugging tasks. Participants attempted repairs under controlled conditions, and both debugging performance and message perceptions were recorded.

  • Study materials and assignment: The selected task materials included full code snippets, standard interpreter errors, and line-numbered code context.Participants were instructed to identify the cause and relevant code line rather than copying the traceback line number.
  • Message styles: Pragmatic messages were constrained to concise, actionable explanations that identified the exception, relevant line, cause, and a hinted fix.The template requested one paragraph of around 20–25 words or less without corrected code.
  • Message styles: Contingent messages used 3–5 supportive sentences that combined actionable guidance with confirmation of the programmer’s likely intent.The prompt allowed discussion of commonness or alternative circumstances only when useful or relevant.
  • Study materials and assignment: Each participant saw one calibrated buggy Python snippet paired with one assigned message style.The three styles were standard, pragmatic, and contingent, with one snippet and one style per participant.
  • Outcome measures: The experiment measured fix rate, Fix@k, time-to-fix, number of attempts, readability, cognitive load, and perceived tone.Participants had up to three repair attempts, with failed submissions resetting the editor to the original code.

D.7 Participant Allocation

The final dataset contained 103 valid participants, with allocation reported across message styles, proficiency groups, and debugging snippets. The study used an eight-item assessment selected to distinguish novices from experts.

  • Participant allocation: 103 valid participants remained after quality control.Allocation was reported by message style and assessed proficiency group, and separately by snippet and message style.
  • Participant allocation: Participants were allocated across standard, pragmatic, and contingent message styles and assessed proficiency groups.The cited allocation tables document these distributions.
  • Skill assessment: The pilot self-rating instrument used six Python-experience levels informed by the Dreyfus model of skill acquisition.The table presents the wording and response options used to report experience.
  • Skill assessment: Eight multiple-choice items were selected from an initial pool of 56 to distinguish novices from experts in debugging and error-message understanding.The selected items were carried forward into the main Prolific study.

F Fantastic Four Code Snippets and Error Messages

The appendix presents four buggy Python snippets and their standard interpreter errors, covering syntax, name-resolution, decorator, and list-access problems. These snippets formed the debugging materials used in the study.

  • Task snippets: Four buggy Python snippets were prepared with their corresponding standard interpreter errors.The appendix organizes each snippet with its error message in separate sections.
  • Snippet A: Snippet A triggers a SyntaxError because a triple-quoted string remains unclosed on line 17.The listing identifies this as a standalone-file execution error.
  • Snippet B: Snippet B triggers a NameError on line 13 because the variable maximum is undefined.The relevant method attempts to return maximum(self.scores) when scores are present.
  • Snippet C: Snippet C triggers a TypeError on line 21 because @classmethod and @staticmethod appear together on the same line.The surrounding code defines a ScoreReport class and its reporting methods.
  • Snippet D: Snippet D triggers a TypeError on line 39 because list-element access uses incorrect syntax.The failing output statement accesses vs.__getitem__[i] rather than using standard list indexing.

G.1 System Architecture

The study platform combined a web frontend, Python backend, relational database, local LLM inference service, and reverse proxy in a Docker Compose deployment. Internal services communicated over a private Docker network while Nginx provided the public entry point.

  • Architecture: The platform comprised a web frontend, Python backend, relational database, local LLM inference service, and reverse proxy.Figure 18 depicts how these components interact within the containerized environment.
  • Architecture: The frontend used Next.js, React, TypeScript, and TailwindCSS for reusable forms, multi-step flows, and interactive task components.The stack was chosen largely for familiarity and straightforward component development.
  • Architecture: The FastAPI backend handled user code execution, evaluation logic, and returning LLM-rephrased error messages.Using Python simplified integration with the evaluation logic and LLM client.
  • Architecture: PostgreSQL stored participant data, submissions, feedback, and event logs while preserving links across multi-step task records.The relational schema supported transactional integrity for submissions, feedback, and timing events.
  • Deployment and networking: Docker Compose managed the services, exposing only Nginx publicly and connecting backend, database, and LLM services through an internal Docker network.This layout reduced the attack surface and allowed internal services to restart or scale independently.
Loading 2608.20896v1…