Source-linked AI summary
From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debugging
Yuling Shi, Songsong Wang, Chengcheng Wan, Min Wang, Xiaodong Gu
TL;DR
Holistic LLM debugging treats programs as single units, despite failures ranging from syntax errors to algorithmic flaws. MGDebugger decomposes code into subfunctions and debugs them bottom-up with LLM-simulated execution. It outperforms existing methods on reported benchmarks, including HumanEval and HumanEvalFix.
Problem
Existing LLM debugging treats programs holistically despite failures spanning syntax errors to algorithmic flaws, limiting effectiveness on complex code.
Source: §1 · p. 1Method
MGDebugger decomposes code into a hierarchical subfunction tree and debugs bottom-up, using LLM-simulated execution to trace variable states on failed tests.
Source: §1 · pp. 1–2Results
MGDebugger raises accuracy from 75.6% to 94.5% on HumanEval and achieves a 97.6% repair success rate on HumanEvalFix.
Source: §1 · p. 2Takeaways & Limitations
The reported results extend across error types and code complexity, with strong performance on complex logical errors and longer code snippets.
Source: §1 · p. 2, §6 · p. 11The authors identify potential limits to generalizability across programming languages and domains, and possible performance variation across LLM architectures.
Source: §4.5 · p. 10
Abstract
from arXiv · showhide
While large language models have made significant strides in code generation, the pass rate of the generated code is bottlenecked on subtle errors, often requiring human intervention to pass tests, especially for complex problems. Existing LLM-based debugging systems treat generated programs as monolithic units, failing to address bugs at multiple levels of granularity, from low-level syntax errors to high-level algorithmic flaws. In this paper, we introduce Multi-Granularity Debugger (MGDebugger), a hierarchical code debugger by isolating, identifying, and resolving bugs at various levels of granularity. MGDebugger decomposes problematic code into a hierarchical tree structure of subfunctions, with each level representing a particular granularity of error. During debugging, it analyzes each subfunction and iteratively resolves bugs in a bottom-up manner. To effectively test each subfunction, we propose an LLM-simulated Python executor, which traces code execution and tracks important variable states to pinpoint errors accurately. Extensive experiments demonstrate that MGDebugger outperforms existing debugging systems, achieving an 18.9% improvement in accuracy over seed generations in HumanEval and a 97.6% repair success rate in HumanEvalFix. Furthermore, MGDebugger effectively fixes bugs across different categories and difficulty levels, demonstrating its robustness and effectiveness.
1 Introduction
MGDebugger addresses the gap between holistic debugging and failures spanning syntax to algorithmic flaws by decomposing code and debugging subfunctions bottom-up. Experiments report improvements on generated-code benchmarks and repairs of real-world software defects.
1 Introduction
- Holistic debugging overlooks code structure and the range of failures, from syntax errors to algorithmic flaws, limiting effectiveness on complex programs.The paper frames efficient repair of LLM-generated code as the “last mile” because generated code can still require human fixes to pass tests.Source: §1 · p. 1
- MGDebugger decomposes code into a subfunction tree and uses public tests plus LLM-simulated execution to identify and repair bugs bottom-up.The simulator tracks key variable changes while reasoning through failed tests.Source: §1 · pp. 1–2
- MGDebugger raises accuracy from 75.6% to 94.5% on HumanEval and achieves a 97.6% repair success rate on HumanEvalFix.The paper also reports +6.1% accuracy improvements for GPT-4o and Claude 3.5 Sonnet on HumanEval.Source: §1 · p. 2
- Ablations report a repair success rate drop from 76.3% to 52.6% without hierarchical debugging; DeepSeek-Coder also reaches 100% on five listed bug categories.The categories are missing logic, excess logic, operator misuse, variable misuse, and function misuse; MGDebugger also performs robustly across code lengths and complexities, especially longer code.Source: §1 · p. 2
- On Defects4J, MGDebugger repairs 129 bugs in v1.2 and 64 in v2.0, outperforming specialized program repair tools.The paper presents this as generalization to real-world software defects and human-written code.Source: §1 · p. 2
2 Methodology
MGDebugger turns whole-program debugging into recursive, bottom-up debugging of subfunctions. It decomposes code, derives subfunction tests from public tests, and uses simulated execution to diagnose failures before propagating fixes upward.
2.1 Overview
- MGDebugger’s workflow decomposes code, generates subfunction tests, simulates execution to diagnose failures, then propagates fixes through dependent functions.The hierarchy is traversed depth-first, debugging subfunctions before higher-level functions.Source: §2.1 · p. 2
2.2 Hierarchical Code Decomposition
- A subfunction is a contiguous, logically coherent minimal code unit with defined inputs and outputs, organized in a tree of caller and child functions.An LLM performs decomposition to support isolated testing and debugging at multiple granularities.Source: §2.2 · p. 2
2.3 Generating Test Cases for Subfunctions
- For each subfunction, MGDebugger derives test inputs and expected outputs by tracing its role in the main function’s public test cases.The method assumes public main-function tests are available; it may instead use LLM-generated tests.Source: §2.3 · p. 3
2.4 Debugging Subfunctions with LLM-simulated Execution
- The LLM-simulated executor traces key variables step by step on failed tests, avoiding an external debugger and accommodating unavailable environments or execution constraints.The paper cites compilation errors and security concerns as cases where code may not be executable directly.Source: §2.4 · p. 4
2.5 Bottom-up Debugging
- MGDebugger recursively debugs the hierarchy from granular subfunctions upward, updating fixes in dependent functions to address errors at different abstraction levels.The approach covers bugs from low-level syntax errors to high-level logical flaws.Source: §2.5 · pp. 4–5
3 Experiments
The experiments evaluate MGDebugger across five research questions, from debugging LLM-generated code and component contributions to bug categories, performance consistency, and real-world defects. Across these settings, MGDebugger outperforms existing methods, with ablations highlighting the importance of hierarchical debugging.
3.1 Setup
- The experiments assess five questions using HumanEval, MBPP, HumanEvalFix, and real-world Defects4J defects, with accuracy and repair success rate as evaluation metrics.The setup includes open-source and advanced models; HumanEvalFix contains six bug categories.Source: §3 · p. 5, §3.1 · p. 5
3.2 Main Results (RQ1)
- MGDebugger gains +15.3% to +18.9% accuracy on HumanEval and +11.4% to +13.4% on MBPP across models and datasets.On HumanEval, it reaches 94.5% accuracy with DeepSeek-Coder-V2-Lite and Codestral; on advanced models, it improves GPT-4o and Claude 3.5 Sonnet by +6.1% each and LLaMA 3.1 70B by +13.4%.Source: §3.2 · pp. 5–6, Table 1 · p. 6, Table 2 · p. 6
3.3 Ablation Study (RQ2)
- Removing hierarchical debugging lowers repair success rate from 76.3% to 52.6% on HumanEval and from 39.0% to 33.5% on MBPP.Replacing simulated execution with real execution lowers rates from 76.3% to 47.4% on HumanEval and from 39.0% to 36.0% on MBPP.Source: §3.3 · pp. 6–7, Table 3 · p. 6
3.4 Debugging Different Types of Bugs (RQ3)
- MGDebugger achieves 97.6% repair success with DeepSeek-Coder on HumanEvalFix, reaching 100% in every bug category except value misuse.The authors report particular strength on bottom-level bugs such as missing and excess logic.Source: §3.4 · p. 7, Table 4 · p. 7
3.5 Performance Consistency (RQ4)
- MGDebugger outperforms other methods across code lengths, with a particularly pronounced advantage on longer HumanEval and HumanEvalFix snippets.Across repeated attempts, it has the highest cumulative repair success rate; on HumanEvalFix it exceeds 95% after five attempts and gradually reaches 97.6%.Source: §3.5 · pp. 7–8, Figure 3 · p. 8, Figure 4 · p. 8
3.6 Generalization to Real-World Software Defects (RQ5)
- MGDebugger fixes 129 bugs in Defects4J v1.2 and 64 in v2.0, outperforming specialized repair approaches overall.It uniquely fixes 25 bugs in the comparison with four strong baselines, while 36 bugs are fixed by all methods.Source: Table 5 · p. 7, Figure 5 · p. 8, §3.6 · p. 9
4 Discussion
The discussion illustrates MGDebugger’s bottom-up debugging on a multi-step bug, reports component and bug-granularity analyses, and evaluates it on unseen data. It also acknowledges threats involving evaluation scope, model variation, and patch validation.
4.1 Case Study
- In the Collatz case, subfunction decomposition helped MGDebugger correct the sequence logic and retain the final 1, which other methods missed.MGDebugger separated sequence generation from odd-number filtering; the other methods introduced a bug while correcting the recurrence.Source: §4.1 · p. 9
4.2 Individual Component Analysis
- Manual checks found 100% semantic correctness across 38 decompositions, 93.2% accuracy for generated tests, and 94.2% accuracy in simulated breakpoint predictions.The tests covered a mean 71.3% of code; correcting simulation errors enabled fixes in two of nine failed cases.Source: §4.2 · p. 9
4.3 Bug Granularity Analysis
- Among 23 successfully fixed HumanEval problems, 56.5% involved low-level bugs and 43.5% high-level bugs, spanning syntactic/API/operator/name errors and logic or problem-understanding errors.The authors interpret this distribution as evidence that MGDebugger addresses faults at different granularity levels.Source: §4.3 · p. 10
4.4 Evaluation on Unseen Data
- On 131 post-cutoff LiveCodeBench problems, GPT-4o with MGDebugger reaches 36.6% accuracy and 10.8% repair success rate, beating No-Debugging by 7.6% and Reflexion by 3.0%.The authors present these results as evidence of debugging performance on completely unseen data.Source: §4.4 · p. 10, Table 6 · p. 10
4.5 Threats to Validity
- The authors identify limited generalizability across languages and domains, model-architecture variation, and manual patch validation as threats to validity.They evaluated across four benchmarks in Python and Java, several model architectures, and had three engineers independently validate Defects4J patches.Source: §4.5 · p. 10
- The authors argue that evaluation across models, datasets, bug types, and code complexities supports broad applicability and robust performance.They characterize extension to real-world debugging scenarios as likely, not established.Source: §4.5 · pp. 10–11
5 Related Work
Related work situates MGDebugger among code-generation methods, LLM-based repair, and automated program repair. The paper distinguishes its post-generation, bottom-up debugging approach from methods that generate code correctly at the outset or debug whole functions.
5.1 Large Language Models for Code
- Code-generation work improves correctness through planning, sampling and ranking, or multi-agent collaboration, whereas MGDebugger targets errors after code generation.The cited approaches aim to optimize production of correct code from the outset [66] [71], [6] [8] [46], [11] [17] [68].Source: §5.1 · p. 11
5.2 Repairing LLM-Generated Code
- LLM repair includes task-specific repair training [10] [28] [43] [73] and external-feedback methods [9] [16] [24] [47] [75]; MGDebugger uses pretrained models without task-specific retraining.Prior feedback-based methods include Reflexion [53], Self-Debugging [9], and LDB [75]; the paper says whole-function, sequential debugging can limit effectiveness on complex code [9] [15].Source: §5.2 · p. 11
5.3 Automated Program Repair
- Traditional and learning-based APR have scalability, patch-diversity, or training-data constraints; MGDebugger instead combines bottom-up hierarchy, test generation, and simulated execution.The paper frames its objective as helping LLMs correct their own code-generation mistakes.Source: §5.3 · p. 11
6 Conclusion
The conclusion presents MGDebugger as a bottom-up framework that decomposes code, generates targeted tests, and uses LLM-simulated execution to address bugs at multiple granularities. It reports stronger performance than existing methods and identifies extensions for future work.
6 Conclusion
- MGDebugger decomposes code hierarchically and uses targeted tests plus LLM-simulated execution to identify and fix syntax errors through logical flaws bottom-up.The conclusion reports superior performance across models and datasets, particularly for complex logical errors and longer code snippets.Source: §6 · p. 11
- The authors identify multi-file projects and codebases with multiple dependencies as targets for extending MGDebugger to more complex bugs and structures.They also propose combining hierarchical code generation with debugging and integrating corrected outputs into self-training.Source: §6 · p. 11
- The paper proposes integrating MGDebugger into self-training systems, using corrected base-model outputs for retraining and potentially improving performance iteratively.This is presented as a possible future direction, not a demonstrated outcome.Source: §6 · p. 11