Explainable Automated Debugging via Large Language Model-driven Scientific Debugging

1 INTRODUCTION

• Existing debugging techniques are mature enough for industry adoption, yet developers also want rationales that help them judge automated results.Prior studies found that more than 85% of surveyed developers considered rationale important.Source: §1 · p. 1
• AutoSD uses LLMs and a debugger interface to emulate Scientific Debugging, generating hypotheses, testing them, and using the resulting trace to explain a patch.The LLM decides whether observations support a hypothesis and whether further investigation is needed.Source: §1 · pp. 1–2
• 83% of the real-world bugs studied showed improved developer accuracy with AutoSD explanations, while review time remained roughly constant.The study involved 20 Python developers, including six professionals.Source: §1 · p. 2
• AutoSD achieved repair results competitive with non-explainable APR techniques and indicated higher repair performance when it judged that enough debugging information had been collected.The authors also report that repair performance rises as the language models become more capable.Source: §1 · p. 2
• 70% of participants wanted explanations for APR tools, while 55% were satisfied with AutoSD’s Scientific Debugging presentation.The study evaluated patch review with 20 Python developers, including six professional developers.Source: §1 · p. 2

2.1 Explainable Automated Debugging

• Research on automated debugging has advanced, but studies of explanations produced by repair tools have been scarce despite developer interest in explanations for reviewing patches.A review found only one APR paper mentioning explanations, while a survey reported that the most commonly requested helpful APR output was an explanation including root cause.Source: §2.1 · p. 2
• Existing generate-and-validate and semantics-based APR methods search for repairs without inherently identifying a bug’s root cause, making their reasoning traces difficult to turn into satisfactory explanations.Generate-and-validate methods try code variants, while Angelix uses variable values with SMT solvers.Source: §2.1 · pp. 2–3

2.2 Scientific Debugging

• Scientific Debugging formalizes a hypothesis-and-verification process in which developers formulate hypotheses, predict outcomes, run experiments, observe results, and draw conclusions.Developers surveyed reportedly followed this broad process even without formal debugging training.Source: §2.2 · p. 3

2.3 Large Language Models

• The paper argues that LLM reasoning abilities, prompted task-following, and use of external tool results make LLMs promising for emulating Scientific Debugging.The authors point to reasoning benchmarks, few-shot or zero-shot prompting, and tool use as supporting capabilities.Source: §2.3 · p. 3

3.1 Constructing the Input Prompt

• The initial prompt combines a detailed description and examples of Scientific Debugging with bug-specific code, a failing test, an error message, and any available bug report.The prompt also provides debugger commands and a DSL for editing code and rerunning the test; ChatGPT follows it zero-shot.Source: §3.1 · p. 3

3 AUTOMATED SCIENTIFIC DEBUGGING

• AutoSD generates hypotheses and experiments with an LLM, runs experiments through a debugger or code executor, evaluates observations, and generates a fix with an explanatory reasoning trace.The hypothesize-observe-conclude cycle repeats until debugging concludes or the iteration limit is reached.Source: §3 · p. 3, Figure 1 · p. 4

3.2 Hypothesize-Observe-Conclude

• Each loop step has the LLM propose a hypothesis and executable experiment, obtain real execution observations, and judge whether the hypothesis is supported, rejected, or undecided.The LLM may emit <DONE> when it considers debugging complete; otherwise, the loop restarts until its iteration limit.Source: §3.2 · pp. 4–5

3.3 Fix Suggestion

• AutoSD can also localize faults during hypothesis generation, making that localization part of the explanation.After interaction, it removes rejected hypotheses from the prompt before generating the fix, while retaining them as possible context for the developer.Source: §3.2 · p. 4, §3.3 · p. 5
• The final fix is generated from the problem description and code-interaction information, while the intermediate reasoning steps can be shown to explain its origin.Rejected hypotheses are excluded from the fix-generation prompt because doing so empirically improved repair performance.Source: §3.3 · p. 5

4.1 Research Questions

• RQ1 tests whether AutoSD’s repair performance is competitive with prior APR techniques and direct LLM fix prompting on ARHE and Defects4J versions 1.2 and 2.0.The study frames this comparison as testing whether explainability comes with a significant performance cost.Source: §4.1 · p. 5
• RQ2 examines whether the <DONE> token marks stronger repair performance and whether it remains informative when debugger observations are replaced by LLM-generated observations; RQ3 varies the LLM.ChatGPT was selected as the default model based on preliminary performance findings.Source: §4.1 · p. 5
• RQ4 measures patch-review time and accuracy with or without explanations, while RQ5 assesses acceptance and RQ6 examines explanation examples and sampled failure causes.RQ6 analyzes a random sample of 25 cases in which AutoSD classified all generated hypotheses as incorrect.Source: §4.1 · p. 5

4.2 Environment

• The evaluation uses Defects4J 1.2 and 2.0, and constructs ARHE by mutating HumanEval solutions so exactly one test fails.ARHE contains 200 bugs; BugsInPy supplies real-world Python bugs for the human study, but its repair performance is not reported because many bugs required undocumented environment setup.Source: §4.2.1 · pp. 5–6

4.2.1 Evaluating APR Performance.

• For repair evaluation, AutoSD generates 10 patches per dataset and a patch is correct only if it passes all tests and is semantically equivalent to the developer patch.The setup uses perfect method-level fault localization and bug reports, whereas the cited comparison work uses perfect statement-level fault localization.Source: §4.2.1 · p. 6

4.2.2 Human Study Parameters.

• The human study samples 12 bugs with initially failing tests made to pass by AutoSD, then divides them into two six-bug review groups.Twenty participants reviewed six problems each; explanations were provided in a randomly selected three of six cases, and participants judged patch correctness and explanation helpfulness.Source: §4.2.2 · p. 6

5.1 RQ1: Feasibility

• AutoSD performs competitively with other repair approaches on ARHE and Defects4J, while retaining LLM-Base-like performance on ARHE and outperforming Codex under the reported Defects4J setting.The ARHE comparison also reports significantly weaker repair performance for the template-based baseline; Codex used 200 patch candidates versus AutoSD's 10.Source: Table 1 · p. 6, §5.1 · pp. 6–7, Table 2 · p. 7

5.2 RQ2: Debugger Ablation

• For both ARHE and Defects4J, <DONE> predictions correspond to higher precision; among plausible patches, 89% with <DONE> were correct fixes versus 82% without it.The Defects4J comparison sampled 100 patches with and without <DONE> because manually labeling all 1,045 plausible patches was infeasible.Source: §5.2 · p. 7
• When debugger results were hallucinated rather than obtained by execution, <DONE>-predicted solutions were 11%p less likely to be plausible, versus 12.4%p more likely with execution.Plausible patches occurred in 73% of individual AutoSD runs, compared with 63% when the debugger was ablated.Source: §5.2 · p. 7

5.3 RQ3: Varying LLM

• AutoSD repair performance increases as the underlying language model improves, and ultimately becomes comparable to querying the LLM directly.In the reported zero-shot setting, CodeGen-6B returned the original buggy code; in a few-shot setting it fixed 44 bugs.Source: §5.3 · p. 7

5.4 RQ4: Developer Benefit

• With explanations, participants reviewed patches in roughly the same time and were more accurate in seven cases, including five of six real-world bugs; explanations helped in eight of twelve questions.Accuracy fell in two cases, including one where participants accepted AutoSD's reasoning without significant scrutiny.Source: §5.4 · pp. 7–8

5.5 RQ5: Developer Acceptance

• Although 70% of participants considered explanations important, 55% found AutoSD's Scientific Debugging details satisfactory, with only one of six professional developers satisfied.Participants suggested linking explanations to specifications and improving the interface; the study's overall satisfaction results differed between students and professionals.Source: §5.5 · pp. 8–9

5.6 RQ6: Qualitative Analysis

• A common reasoning failure was proposing breakpoints the test never hit, which occurred in 13 of 25 cases where all hypotheses were rejected.Other reported failures included invalid experiment expressions in 2/25 cases and multiple print commands in 2/25; a qualitative example showed explanation-guided code review helping participants connect a patch to the test.Source: §5.6 · p. 9

6.1 Threats to Validity

• Participant accuracy improved on five of six BugsInPy problems, though the authors note possible bias in human-study responses.They describe this accuracy result as difficult to attribute to bias.Source: §6.1 · p. 9
• The authors identify implementation errors, patch-correctness assessment, and human-study response bias as internal-validity threats.They plan to make the implementation and repair results publicly available for scrutiny.Source: §6.1 · p. 9
• The authors identify possible overlap between LLM training data and evaluation data as an external-validity concern.They constructed the ARHE dataset for repair to mitigate this concern; they also consider their explanations likely absent from training data.Source: §6.1 · p. 10

6.2 Limitations

• AutoSD may take about five times longer than LLM-Base, and currently handles only single-method bugs.The evaluation also assumes method-level fault localization before AutoSD performs explainable statement-level fault localization.Source: §6.2 · p. 10
• The explanations include concrete code-execution results, which the authors say aid developer decision-making.Figure 6 is referenced for this point.Source: §6.2 · p. 10

7 CONCLUSION

• The paper identifies a lack of automated debugging techniques that provide explanations adequate for developers.It argues that existing techniques’ reasoning differs from human debugging.Source: §7 · p. 10
• AutoSD uses LLMs to emulate Scientific Debugging, bridging automated and manual debugging practices.The paper presents this as its approach to producing more human-aligned reasoning.Source: §7 · p. 10
• AutoSD achieves competitive repair performance against other repair baselines and indicates confidence in its output.The conclusion also reports that repair performance improves as language models become more capable.Source: §7 · p. 10
• The human study found that generated explanations could improve developer patch assessment, and most students said they missed explanations when unavailable.Interviews suggested explanations could fit varied developer thought patterns and might improve through tighter development-process integration, such as links to written specifications.Source: §7 · p. 10

1 DISCUSSION OF EXPLAINABLE FAULT LOCALIZATION

• Fault-localization explanations matter because developers may not know what to do with a result, and surveyed developers connect explanations to fixing and actionability.Kochhar et al. report that 85% of developers want explanations, and a cited developer wanted to know exactly why an automated tool thought code had a bug.Source: §1 · p. 13
• SBFL uses failing and passing test coverage patterns, while IRFL uses textual similarity to rank likely fault locations.The section also mentions MBFL, which mutates statements and observes test-behavior changes, and machine-learning-based fault localization, which combines features from fault-localization techniques.Source: §1 · p. 13
• The section argues that these techniques do not deduce in a humanlike way, making their reasoning traces difficult to use for understanding results.For machine-learning-based fault localization, it is also unclear whether a reasoning trace exists.Source: §1 · p. 13
• Prior fault-localization explanation work includes state-based approaches, but the section notes that comparable user studies of those explanations were absent.The described work includes explaining failure differences, applying delta debugging to program states, and using slicing to refine state replacement.Source: §1 · p. 13

2 ARHE BENCHMARK MUTATOR BREAKDOWN

• The mutators alter literals and operators: integer literals switch between 0 and 1, and operators may change from plus to minus.String literals can be emptied or have their case changed; emptying is not reversible, while reversing case changes depends on the problem.Source: §2 · p. 14
• If Remover deletes a branch block and may remove the if statement when no children remain; If Negator adds not to a condition.The descriptions compare these operations to PIT mutators.Source: §2 · p. 14
• Augmented Assignment Changer switches += and -=, while Binary Operator Remover removes an operator and leaves one operand.The benchmark description compares these mutations to PIT’s Increments Mutator and Arithmetic Operator Deletion Mutator, respectively.Source: §2 · p. 14
• 90 ARHE bugs cannot be repaired by applying the same mutation set.This includes 24 If Remover bugs, 24 Binary Operator Remover bugs, and 42 of the 63 String Literal Changer bugs.Source: §2 · p. 14
• The benchmark includes Integer Literal Changer (45), If Remover (24), String Literal Changer (63), and Operator Changer (40) bugs.It also lists Binary Operator Remover (24), Augmented Assignment Changer (3), and If Negator (1).Source: §2 · p. 14

• Figure 1 is a screenshot of the human-study webpage.Source: §3 · pp. 14–15
• The left column shows the original buggy code and error message, the middle suggests a patch, and the right displays its explanation.Source: §3 · p. 14
• The questions appear at the bottom center of the webpage.Source: §3 · p. 14

4 SCIENTIFIC DEBUGGING PROMPT

• The prompt asks for a hypothesis, prediction, experiment, observation, and conclusion, marking debugging done when a confirmed hypothesis yields a concrete fix.Source: §4 · pp. 15–16
• Hypotheses should explain buggy behavior, build on prior information, and avoid repetition; examples show revising after rejection or experiment error.Source: §4 · p. 15
• Predictions specify an expected value or symptom, such as a debugger-observed expression differing from its intended value or a test passing after an edit.Source: §4 · pp. 15–16
• Experiments may use jdb scripts to inspect a value at a real statement or edit scripts that support REPLACE, ADD, and DEL.Source: §4 · p. 16
• Observations record script output, which the prompt uses to judge whether a hypothesis is supported, rejected, or undecided after an error.Source: §4 · p. 16

5 DEFECTS4J AUTOSD EXAMPLES

• A successful Defects4J trace identifies END_OBJECT as the bug condition, tests that hypothesis, and reaches a patch identical to the developer patch that passes all tests.Source: §5 · p. 16
• A failing trace repeats the same hypothesis and experiments, failing to identify the right breakpoint.Source: §5 · pp. 16–17