Source-linked AI summary
Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-step
Li Zhong, Zilong Wang, Jingbo Shang
TL;DR
Existing LLM code-refinement methods largely treat programs as indivisible and depend on post-execution feedback, leaving runtime execution underused for debugging complex programs. LDB decomposes programs into basic blocks, tracks intermediate values, and verifies execution step by step against the task description. It consistently improves code-generation accuracy by up to 9.8% across HumanEval, MBPP, and TransCoder, achieving state-of-the-art debugging performance across model selections.
Problem
Existing refinement methods treat generated programs as indivisible and rely on post-execution feedback, while runtime execution information is underused for debugging complex logic flows and data operations.
Method
LDB segments programs into control-flow-based basic blocks, tracks intermediate variables after each block, and has LLMs verify block correctness against the task description.
Results
LDB consistently improves code-generation accuracy by up to 9.8% across HumanEval, MBPP, and TransCoder and achieves state-of-the-art performance in program debugging.
Takeaways & Limitations
Runtime execution information provides LDB with a distinct debugging paradigm that improves generated-program quality across different LLM backbones.
Takeaways & Limitations
LDB requires correct visible test cases to execute programs and compare their execution flows against the task description.
Abstract
from arXiv · showhide
Large language models (LLMs) are leading significant progress in code generation. Beyond one-pass code generation, recent works further integrate unit tests and program verifiers into LLMs to iteratively refine the generated programs. However, these works consider the generated programs as an indivisible entity, which falls short for LLMs in debugging the programs, especially when the programs contain complex logic flows and data operations. In contrast, when human developers debug programs, they typically set breakpoints and selectively examine runtime execution information. The execution flow and the intermediate variables play a crucial role in the debugging process, yet they are underutilized in the existing literature on code generation. In this study, we introduce Large Language Model Debugger (LDB), a novel debugging framework that enables LLMs to refine their generated programs with the runtime execution information. Specifically, LDB segments the programs into basic blocks and tracks the values of intermediate variables after each block throughout the runtime execution. This allows LLMs to concentrate on simpler code units within the overall execution flow, verify their correctness against the task description block by block, and efficiently pinpoint any potential errors. Experiments demonstrate that LDB consistently enhances the baseline performance by up to 9.8% across the HumanEval, MBPP, and TransCoder benchmarks, archiving new state-of-the-art performance in code debugging for various LLM selections.
1 Introduction
Existing code-refinement methods rely mainly on post-execution feedback and treat programs as indivisible, limiting debugging for complex control flows and data structures. LDB instead uses runtime execution information to verify basic blocks step by step and improve generated programs.
- Existing code-generation approaches use multiple sampling, self-consistency, candidate ranking, or iterative feedback, but still fall short of reliably generating correct programs.
- Human-style debugging examines execution traces and intermediate variables, whereas prior LLM debugging methods primarily use post-execution outputs, unit-test results, or error messages.
- LDB segments programs into control-flow-based basic blocks, tracks intermediate variables after each block, and asks LLMs to verify block correctness against the task description.
- LDB enables LLMs to focus on simpler code units and pinpoint potential bugs through step-by-step verification of intermediate execution states.
- LDB is evaluated on HumanEval, MBPP, and TransCoder with GPT-3.5, StarCoder, and CodeLlama, demonstrating consistent accuracy improvements across LLM backbones.
- The paper’s contributions are runtime-informed LLM debugging, basic-block decomposition for stepwise verification, and experiments across three benchmarks and multiple model backbones.
2 Large Language Model Debugger
LDB profiles failed seed programs, verifies their runtime behavior block by block, and regenerates them using the resulting debugging feedback. Its workflow represents execution through control-flow-based traces and intermediate variable states while using selective or batched inspection to manage long traces.
- Profiling: LDB begins with a seed program and failed visible test case, then collects its execution trace and runtime variable values for debugging.
- Profiling: A control-flow graph represents basic blocks as straight-line code units with one entry and one exit, while an execution trace records the traversed block sequence.
- Profiling: After each block, LDB records variables in scope and their runtime values, defining the intermediate state as entry state, current block, and resulting state.
- Debugging: For each intermediate state, an LLM assigns a correctness verdict and explanation by comparing the block’s behavior with the task description.
- Debugging: Selective debugging limits inspection when loops or recursion create lengthy traces that could exceed the LLM context window.
- Debugging: Batch debugging groups intermediate states from execution into a single query, improving token efficiency during iterative refinement.
- Regeneration: LDB feeds verdicts and explanations, together with the task description, into regeneration and repeats profiling, debugging, and regeneration until tests pass or the iteration limit is reached.
3 Experiments
Experiments evaluate LDB across three benchmarks, multiple LLM backbones, debugging baselines, decomposition levels, iteration counts, and a case study. LDB consistently improves code-generation accuracy, with runtime-informed, block-level feedback supporting more effective debugging.
- Evaluation Setup: LDB is evaluated on HumanEval, TransCoder, and MBPP using GPT-3.5, StarCoder, and CodeLlama, with Pass@1 measured on hidden tests.HumanEval and MBPP cover text-to-code generation; TransCoder covers C++-to-Python translation.
- Main Results: Up to 9.8% improvement is observed over the no-debugger baseline across datasets and LLM backbones.Compared with Self-Debugging, LDB provides higher and more stable gains through actual runtime execution information.
- Main Results: LDB’s fine-grained debugging feedback and runtime information help models focus on buggy regions and align programs with task descriptions.The framework uses accurate intermediate values and execution flows to generate debugging verdicts and explanations.
- Results on Advanced Code Generators: 95.1% accuracy is achieved on HumanEval when LDB debugs Reflexion seed programs, indicating complementarity with advanced code generators.The result is reported as a new state-of-the-art outcome and as evidence that LDB can correct bugs overlooked by stronger generators.
- Performance vs. Debugging Iterations: LDB continues improving across debugging iterations, whereas Self-Debugging nearly stops improving after two iterations and resembles repeated sampling after three rounds.Runtime execution information is described as continually moving models toward correct programs.
- Decomposition Levels: Block-level decomposition achieves the highest improvement, while line-level units may lose semantic completeness and function-level units provide insufficient runtime detail.Function-level debugging requires approximately 8.1 iterations on average and has the highest token cost.
- Case Study: In a HumanEval case study, LDB identifies the bug in Block-5 and changes lst.count(x) > 1 to lst.count(x) > 2.The correction matches the task requirement that the list contain no more than one duplicate of the same number.
4 Related Work
LDB extends feedback-based code refinement by using runtime information and decomposing programs into smaller units for step-by-step debugging.
- LDB complements existing code-generation methods without expanding sampling numbers, while their outputs can provide better seeds for debugging.
- Existing code-refinement methods use human feedback, interpreter outputs, self-generated explanations, or other users’ submissions to revise generated programs.
- LDB decomposes execution traces into blocks and queries language models for debugging verdicts and explanations.
- LDB uses batch debugging to improve debugging efficiency after segmenting programs into smaller code units.
5 Conclusion
The paper presents LDB as a framework that refines generated programs with runtime execution information. Its experiments report improved code-generation accuracy and state-of-the-art program-debugging performance.
- LDB refines generated programs with runtime execution information by segmenting programs into basic blocks and tracking intermediate values.
- The authors report significantly improved code-generation accuracy and state-of-the-art performance in program debugging.
Limitation
LDB requires correct test cases to execute programs and compare their execution flows against task descriptions.
- Correct test cases are mandatory because LDB executes programs and compares execution flows with task descriptions.
- Test-case-free debugging remains an open question for future work.
- The framework inherits limitations associated with human-developer debugging methods.
Ethic Statements
The paper studies debugging of LLM-generated code using open-source and publicly available proprietary models and publicly available datasets.
- The paper focuses on debugging code generated by large language models.
- Its architecture uses open-source models and publicly available proprietary models.
- All datasets used in the paper are available online, and the authors did not hire human annotators.
A Performance with Better Reasoners
LDB achieves higher performance with stronger debugging backbones, reaching up to 98.2% on HumanEval with GPT-4o. The section also situates execution analysis through basic blocks and control-flow traces.
- Up to 98.2% performance is achieved on HumanEval when LDB uses GPT-4o as the debugging backbone and Reflexion as the seed generator.
- LDB performs better with GPT-4 and GPT-4o debugging backbones than with GPT-3.5 on Reflexion seed programs.
- A basic block is a linear code sequence with one entry and one exit whose instructions execute sequentially once entered.
- A control-flow graph represents possible execution paths, with nodes as basic blocks and directed edges as control-flow transitions.
- Execution traces are sequences of consecutively executed basic blocks corresponding to paths from program entry to exit.
C Implementation Details
LDB uses deterministic, bounded debugging settings and follows an automated iterative-refinement paradigm shared with Self-Debugging. Its workflow executes a program, obtains LLM feedback, and regenerates the program.
- LDB generates debugging verdicts and explanations with greedy decoding at temperature T = 0 for reproducibility.
- The maximum number of debugging iterations is 10, with thresholds of 10 sampled blocks and 3,097 input tokens.
- LDB fully automates all debugging steps, with time cost comparable to other baseline methods.
- The shared iterative-refinement paradigm executes the buggy program, queries an LLM for feedback, and regenerates the program from that feedback.
E Tradeoffs in Debugging in Different Decomposition Levels
LDB compares line-, block-, and function-level decomposition, ranging from fine to coarse runtime tracing. Block-level debugging achieves the highest accuracy with the lowest token cost for GPT-3.5 and CodeLlama.
- Decomposition levels: LDB evaluates line-level, block-level, and function-level debugging as decomposition levels from fine to coarse granularity.
- Decomposition levels: Line-level debugging collects intermediate states around each line, while block-level debugging samples the first and last five blocks when traces exceed the threshold.
- Decomposition levels: Function-level debugging collects intermediate states only at the solution function’s entry and exit, with line sampling when the trace exceeds context length.
- Accuracy and cost: Using GPT-3.5, line-level debugging costs fewer tokens than function-level debugging because it requires fewer debugging turns.
- Accuracy and cost: For both GPT-3.5 and CodeLlama, block-level debugging achieves the highest accuracy and least token cost on HumanEval.
F Complexity Analysis of Batch Debugging
Batch debugging improves token efficiency by sending runtime information for selected blocks together, while LDB’s accuracy continues increasing through 20 debugging turns. On HumanEval with GPT-3.5, it reaches 84.1%.
- Batch debugging: Batch debugging sends selected blocks’ runtime information together, reducing repeated context transmission during iterative refinement.
- Granularity comparison: Block-level debugging achieves the highest accuracy and fewest debugging turns for both GPT-3.5 and CodeLlama on HumanEval.
- Batch debugging: Without batching, debugging B blocks costs approximately O(B^2 ∗ N) tokens, compared with O(B ∗ N) for batch debugging.
- Extended debugging: 84.1% is achieved by LDB after 20 debugging turns on HumanEval with GPT-3.5, and accuracy continues growing after 10 turns.
H Error Analysis of LDB
LDB’s error analysis examines bug-localization accuracy and bug categories, using block-level runtime traces to identify concrete failures in generated programs. The examples show LDB isolating semantic errors in specific blocks, while the framework remains dependent on correct test cases and is primarily demonstrated with Python.
- Evaluation dimensions: LDB evaluates debugging through bug-localization accuracy and bug categorization across HumanEval, MBPP, and TransCoder.GPT-4 is used to assess whether LDB identifies the buggy basic block, while detected bugs are categorized as syntax or semantic errors.
- Bug categorization: Around 80% of LDB’s fixes across three datasets address semantic errors.The paper attributes this pattern to the relative frequency of syntax errors and LDB’s use of detailed runtime execution information.
- Scope and limitations: LDB requires correct test cases for execution-based comparison, and its demonstrated implementation uses Python while remaining adaptable through language-specific executors and analysis tools.Multilingual experiments such as HumanEval-X are identified as future work.
- Case analysis: For a regex example, LDB identifies the block defining r"\b\w{4}\b" as incorrect because it matches only exactly four-character words.The observed output contains only “move” and “back,” despite the task requiring words at least four characters long.