Source-linked AI summary
DebugBench: Evaluating Debugging Capability of Large Language Models
Runchu Tian, Yining Ye, Yujia Qin, Xin Cong, Yankai Lin, Yinxu Pan, Yesai Wu, Haotian Hui, Weichuan Liu, Zhiyuan Liu, Maosong Sun
TL;DR
LLM debugging remains relatively unexplored, and existing evaluations are constrained by leakage risk, small scale, and limited bug diversity. DebugBench evaluates models across a broader set of bug types and finds performance differences by model and error category, as well as a positive relationship between debugging and code generation for closed-source models.
Problem
LLM debugging capability remains relatively unexplored, while prior evaluations face data-leakage risk, small datasets, and limited bug diversity.
Source: §1 · p. 1Method
DebugBench uses 4,253 LeetCode-derived instances with GPT-4-implanted bugs, covering four major categories and 18 minor types in C++, Java, and Python, with quality checks.
Source: §1 · p. 2, §2 · p. 2, §2.3 · p. 4Results
In zero-shot evaluation, closed-source models trail humans while open-source models score lower; difficulty varies by bug type, feedback is category-dependent, and closed-source debugging correlates with code generation.
Source: §1 · p. 2, §5 · pp. 8–9Takeaways & Limitations
The findings are presented as useful for understanding LLM debugging capabilities and developing more comprehensive code models.
Source: §1 · p. 2The bugs are synthetic, and the study covers only four open-source and two closed-source models.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Large Language Models (LLMs) have demonstrated exceptional coding capability. However, as another critical component of programming proficiency, the debugging capability of LLMs remains relatively unexplored. Previous evaluations of LLMs' debugging ability are significantly limited by the risk of data leakage, the scale of the dataset, and the variety of tested bugs. To overcome these deficiencies, we introduce `DebugBench', an LLM debugging benchmark consisting of 4,253 instances. It covers four major bug categories and 18 minor types in C++, Java, and Python. To construct DebugBench, we collect code snippets from the LeetCode community, implant bugs into source data with GPT-4, and assure rigorous quality checks. We evaluate two commercial and four open-source models in a zero-shot scenario. We find that (1) while closed-source models exhibit inferior debugging performance compared to humans, open-source models relatively lower pass rate scores; (2) the complexity of debugging notably fluctuates depending on the bug category; (3) incorporating runtime feedback has a clear impact on debugging performance which is not always helpful. As an extension, we also compare LLM debugging and code generation, revealing a strong correlation between them for closed-source models. These findings will benefit the development of LLMs in debugging.
1 Introduction
LLM debugging remains relatively unexplored, and prior evaluations face leakage risk, limited scale, and limited bug diversity. DebugBench supports a broader evaluation, finding that model performance varies by bug type and that debugging and code-generation performance are positively related for closed-source models.
1 Introduction
- Prior debugging evaluations risked training-data leakage, covered only 40–151 examples, and often reported aggregate pass rates instead of distinguishing bug types.Debugging is also a substantial part of programming, consuming 35–50% of development time and 50–75% of total budget.Source: §1 · pp. 1–2
- DebugBench contains 4,253 instances across four major bug categories and 18 minor types in C++, Java, and Python, built from LeetCode snippets with GPT-4 bug implantation and quality checks.The instances were released after July 2022 to reduce data-leakage risk.Source: §1 · p. 2
- Closed-source models outperform open-source models but remain below humans; multiple and logical errors are harder than syntax and reference errors, and runtime feedback helps syntax and reference errors but is unhelpful for logic errors.The evaluation uses zero-shot scenarios.Source: §1 · p. 2
- For closed-source models, syntax and reference debugging is generally easier than code generation, while logic and multiple errors can be equally hard or harder; debugging and code-generation performance are correlated.The reported correlation indicates that performance on the two tasks is positively related.Source: §1 · p. 2
2 Benchmark Construction
DebugBench is constructed from LeetCode programming material by implanting bugs with GPT-4, then filtering and manually inspecting instances. The section defines debugging against desired input-output behavior and describes data selection, bug diversity, quality control, and a comparison with real-world bugs.
2 Benchmark Construction
- LeetCode material is turned into DebugBench through GPT-4 bug implantation, automatic filtering, and final human inspection.The construction also includes human and LLM evaluation of the benchmark.Source: §2 · p. 2, Figure 2 · p. 3
2.1 Formulation of Debugging
- A code script is buggy when it fails at least one input-output pair, and an ideal debugger produces code that passes all such pairs.The formulation treats each input and its desired output as defining the programming problem.Source: §2.1 · p. 3
2.2 Source Data Collection
- LeetCode was chosen partly because its comprehensive test suites support automated evaluation, and all instances were released after June 2022 (average release date April 2023), minimizing data-leakage risk.The dataset includes 1,438 C++ instances, 1,401 Java instances, and 1,414 Python instances.Source: §2.2 · p. 3
2.3 Bug Implantation
- GPT-4 generates single-error variants, while multiple-error instances are created by rule-based merging; the taxonomy covers four major categories and 18 minor types.Synthetic bugs are chosen to allow broader error-diversity design and lower data-leakage risk than traditional debugging datasets.Source: §2.3 · pp. 3–4
2.4 Quality Control
- Manual inspection by three experienced programmers covered 180 cases and judged benchmark quality against bug validity, sensitive-information security, and scenario alignment.The results are reported in Table 3.Source: §2.4 · p. 4, Table 3 · p. 4
2.5 Analysis of Bug Realism
- Synthetic bug implantation raises a realism boundary: benchmark evaluations may not adequately reflect a model’s ability to fix bugs in actual situations.The authors compare DebugBench with QuixBugs, reporting that DebugBench covers most QuixBugs bug types and that its bugs do not significantly differ from real-world scenarios in this evaluation.Source: §2.5 · pp. 4–5
- The realism analysis compares bug-type coverage and model judgments of real-world plausibility, concluding that DebugBench bugs are not significantly different from real-world bugs.Table 4 maps QuixBugs types to DebugBench; Table 5 reports win-rate comparisons judged by GPT-4 and Claude 3 Sonnet.Source: §2.5 · p. 5, Table 4 · p. 5, Table 5 · p. 5
3 Experiments
The experiments compare closed- and open-source models with human debugging, examine error-type difficulty and runtime feedback, and relate debugging performance to code generation. Across these analyses, performance varies by model and bug type, runtime feedback has mixed effects, and debugging and coding performance are positively correlated for closed-source models.
3 Experiments
- Three programmers with over four years of experience each debugged 72 bugs, with IDE access but no deep-learning tools.Each spent approximately 20 hours debugging after training and a trial session.Source: §3 · p. 5
3.1.1 Overall Results
- 75.0% for gpt-4-0613 and 62.1% for gpt-3.5-0613 remained below human performance, while most open-source models scored lower than commercial models.DeepSeek-Coder-33B-Instruct was described as close to commercial models.Source: Table 6 · p. 6, §3.1.1 · p. 6
3 Experiments
- The evaluation assessed two closed-source and four open-source LLMs across 18 programming-error types in three distinct scenarios, using LeetCode test suites with 1–3 known and 8–100 unknown cases per instance.Users need a LeetCode account to access the test suites.Source: §3 · p. 5, §3.1 · p. 6
3.1.2 Effect of Bug Types
- Syntax and reference errors were easier to fix than logic errors, and debugging complexity rose with the number of bugs in a snippet.The paper highlights logic errors and multiple concurrent errors as important challenges for model improvement.Source: §3.1.2 · pp. 6–7
3.1.3 In-depth Analysis
- Runtime feedback helped resolve syntax and reference errors but could impair logic-error resolution.Tracebacks can locate syntax and reference faults, while their low-level detail may disrupt logical debugging.Source: Figure 4 · p. 7, §3.1.3 · p. 7
- For logical-error instances, more sampling attempts improved debugging performance, trading additional inference tokens for better pass rates.The figure notes a significant improvement from one to four samples.Source: §3.1.3 · p. 7, Figure 3 · p. 7
3.2 Interplay between Debugging and Coding
- On identical problems, syntax and reference debugging was typically easier than code generation, while logical or multiple-error debugging could be as difficult as code generation.The comparison used gpt-4-0613 and gpt-3.5-turbo-0613.Source: §3.2 · p. 7
- For closed-source models, all bug categories had positive Phi-Coefficients with code generation, ranging from 0.1 to 0.3.The authors describe debugging and coding capabilities as explicitly correlated.Source: Table 7 · p. 8, §3.2 · p. 8
4 Related Work
Prior work had advanced LLM code generation and explored LLM-based program repair, but debugging received less attention and existing evaluations faced leakage, scale, and bug-diversity constraints.
4.1 LLM-based Coding
- LLM code-generation research included specialized training corpora and methods using runtime-message feedback or multi-agent collaboration.The cited work also reports strong coding performance from general-purpose models trained on code-rich data.Source: §4.1 · p. 8
- LLM debugging had drawn less attention, partly because evaluation benchmarks were lacking.The authors introduce DebugBench to address that deficiency.Source: §4.1 · p. 8
4.2 Automated Program Repair
- Earlier LLM-based program-repair studies reported promising results on QuixBugs, including gains from conversational patch refinement.Zhang et al. (2023) identified substantial leakage risk in traditional APR datasets and evaluated ChatGPT on a 151-bug competitive-programming benchmark.Source: §4.2 · p. 8
- Prior debugging evaluations remained constrained by data leakage, small test scale, and limited bug diversity, motivating a systematic DebugBench evaluation.The passage refers to Table 1 for the evaluation constraints.Source: §4.2 · p. 8
5 Conclusion
The paper presents DebugBench, built from LeetCode source data with GPT-4 bug implantation and quality control. Its experiments report model and bug-type differences, mixed runtime-feedback effects, and a debugging–coding relationship for closed-source models.
5 Conclusion
- DebugBench uses LeetCode source data, prompted GPT-4 bug implantation, and a stringent quality-control process.The benchmark is designed to evaluate LLM debugging capabilities.Source: §5 · p. 8
- In zero-shot evaluation, closed-source models performed below humans, while open-source models attained still lower pass rates; multiple and logical errors were harder than syntax and reference errors.Runtime feedback helped with syntax and reference errors but was unhelpful for logical errors.Source: §5 · pp. 8–9
- For closed-source LLMs, syntax and reference debugging was easier than code generation, whereas logic or multiple-error debugging could be equally hard or harder, and coding and debugging abilities were correlated.The conclusion frames these findings as relevant to advancing LLMs in automatic debugging.Source: §5 · p. 9
- The authors identify repository-level debugging and scenarios involving human interaction as more practical and complex settings for future evaluation.They also suggest evaluating LLMs' ability to write reliable test cases and interact with IDEs.Source: §5 · p. 9
Limitations
The study notes limits from synthetic bugs, model coverage, and LeetCode-based evaluation, alongside appendix details on prompts and instance structure.
Limitations
- Synthetic bugs may not fully reflect real-world coding, and the tested four open-source and two closed-source models do not span all LLMs.Evaluation also relies on LeetCode test cases, which may be less accessible than local tests and require accounts subject to rate and query limits.Source: Limitations · p. 9
A Prompts
- The appendix includes a zero-shot bug-fixing prompt and a three-shot evaluation prompt with bug-fix examples.The zero-shot prompt asks models to rewrite faulty code and briefly explain the modification.Source: A Prompts · p. 12
C Instance Feature Schema
- Each instance includes a programming question, examples, an oracle solution, buggy code, and a GPT-4-written bug explanation.The examples provide one to three input-output pairs; the oracle solution passes all test cases, and the buggy code is the debugging target.Source: C Instance Feature Schema · p. 12
E Cases Failing Manual Inspection
Manual-inspection failures include bugs assigned to the wrong category and bugs considered unlikely in practical scenarios.
Cases Failing Manual Inspection
- Manual inspection checked both whether bugs caused the intended failure and whether they resembled real debugging scenarios.The examples show that category correctness and practical plausibility were separate checks.Source: E Cases Failing Manual Inspection · pp. 12, 14
- A prompt for an undefined-method bug instead produced an infinite loop, violating the assigned bug category.The inspection criteria required bugs to cause the intended malfunction and match their assigned type.Source: E Cases Failing Manual Inspection · pp. 12, 14
- Some correctly categorized bugs were still judged implausible, such as an undefined object named “undefined.”The scenario-alignment criterion excludes bugs unlikely to appear in practical settings.Source: E Cases Failing Manual Inspection · p. 14
F Runtime Messages
Runtime messages can help diagnose runtime errors, but low-level traces may provide no useful clue for some bugs.
Runtime Messages
- Runtime messages can directly identify bugs when the fault raises a runtime error.The section introduces examples of messages that may or may not help debugging.Source: F Runtime Messages · p. 14
- Runtime feedback therefore varies in diagnostic value: some errors point to the bug, while low-level messages may not.Source: F Runtime Messages · pp. 14–15
- A low-level “stackoverflow” message did not help diagnose a prime-arrangements bug caused by changing “+n” to “-n.”Source: F Runtime Messages · p. 15
G Comparison of Bug Realism
The comparison found DebugBench bugs similarly realistic to QuixBugs bugs, using paired judgments by two language models. The section also gives examples of bugs in Python and C++ tasks.
G Comparison of Bug Realism
- The study compared all 40 Python and Java QuixBugs cases with five random same-language, same-type DebugBench cases per case.GPT-4 and Claude 3 Sonnet judged which buggy code was more realistic, with pair order switched to mitigate bias.Source: G Comparison of Bug Realism · p. 15
- DebugBench bug instances were judged to have similar realism to QuixBugs instances.Source: G Comparison of Bug Realism · pp. 15–16
1. GPT-4, Python3, colon missing
- The Python example is a sentence-reordering task whose bug explanation identifies a missing colon after the function definition.Source: §1 · p. 16
2. GPT-4, C++, undefined objects
- The C++ undefined-objects example concerns reconstructing a binary array from adjacent-value XOR results.Source: §2 · p. 16
3. GPT-4, Java, operation error
These examples pair programming tasks with debugging cases: a Java ordering problem, a Python tree-level-sum problem, and a C++ string-literal issue. The supplied snippets identify an operation error, differing kth-sum selection logic, and a syntax correction.
3. GPT-4, Java, operation error
- For the Java relative-order sort, modifying the array during sorting without considering later operations caused incorrect output.The task orders arr1 according to arr2, then places remaining elements in ascending order.Source: §3 · pp. 17–18
4. GPT-4, Python3, triple bugs
- The Python tree snippets contrast ascending a[k] selection with descending a[k-1] selection for the kth-largest level sum.The task defines each level sum as the sum of node values at that depth.Source: §4 · pp. 19–20
5. GPT-3.5-Turbo, C++, unclosed string
- For the distinct-integers task, the C++ fix adds double quotes and removes the invalid “Start of sub-string” text.The task treats digit runs as integers after replacing nondigits with spaces and disregards leading zeros when comparing them.Source: §5 · pp. 20–21
6. GPT-3.5-Turbo, Java, undefined methods
These GPT-3.5-Turbo examples cover a Java temperature-wait task, a Python middle-index variable error, and two C++ buddy-string code corrections.
6. GPT-3.5-Turbo, Java, undefined methods
- The Java task asks for the days until a warmer temperature, using zero when no future warmer day exists.The supplied passage states the task but does not describe the undefined-method bug.Source: §6 · p. 21
7. GPT-3.5-Turbo, Python3, variable error
- In the Python middle-index task, using left instead of right in the sum expression can produce incorrect sums and premature termination.The explanation identifies the erroneous expression as “left_ind = dncl, mid, lo, hi + left.”Source: §7 · pp. 23–24
8. GPT-3.5-Turbo, C++, double bugs
- The C++ buddyStrings fixes change the loop bound to i<n and the frequency-array index from s[i]-'z' to s[i]-'a'.The explanation says the bound change prevents out-of-range access and the index change selects the correct frequency-array position.Source: §8 · pp. 24–26