PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisit Self-Debugging with Self-Generated Tests for Code Generation

Xiancai Chen, Zhengwei Tao, Kechi Zhang, Changzhi Zhou, Wanli Gu, Yuanpeng He, Mengdi Zhang, Xunliang Cai, Haiyan Zhao, Zhi Jin

arXiv:2501.12793v1cs.SEcs.AI

TL;DR

Self-debugging with self-generated tests could help address the limited availability of high-quality tests, but its efficacy across programming problems has not been fully explored. The paper evaluates post-execution and in-execution approaches on self-contained Python tasks, finding different outcomes across task difficulty and promising results from intermediate-state feedback.

  • Problem

    Self-debugging with execution feedback has promise, but high-quality oracle tests are unavailable for every code snippet and the efficacy of self-generated tests remains underexplored.

    Source: §1 · pp. 1–2

  • Method

    The authors evaluate four LLMs on self-contained Python benchmarks and compare post-execution debugging with in-execution debugging using intermediate runtime states.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    In-execution debugging consistently outperforms post-execution debugging on basic and competitive tasks, while post-execution struggles on basic tasks and shows potential on competitive ones.

    Source: §1 · p. 2, §6 · p. 11

  • Takeaways & Limitations

    More sophisticated collection and analysis of runtime information, including clearer execution traces, may further enhance self-debugging.

    Source: §5 · p. 10

    Post-execution debugging depends on high-quality tests and on the model recognizing faulty feedback from invalid tests while preserving correct original programs.

    Source: §4.3 · p. 8

Abstract

from arXiv · show

Large language models (LLMs) have shown significant advancements in code generation, but still face challenges on tasks beyond their basic capabilities. Recently, the notion of self-debugging has been proposed to boost the performance of code generation by leveraging execution feedback from tests. Despite its promise, the availability of high-quality tests in real-world scenarios is limited. In this context, self-debugging with self-generated tests is a promising solution but lacks a full exploration of its limitations and practical potential. Therefore, we investigate its efficacy on diverse programming problems. To deepen our understanding, we propose two distinct paradigms for the process: post-execution and in-execution self-debugging. Within the scope of self-contained Python programming tasks, we find that post-execution self-debugging struggles on basic problems but shows potential for improvement on competitive ones, due to the bias introduced by self-generated tests. On the other hand, in-execution self-debugging enables LLMs to mitigate the bias by solely leveraging intermediate states during execution, thereby enhancing code generation.

1 INTRODUCTION

This study examines self-debugging with self-generated tests, addressing the lack of oracle tests in many real-world code-development settings. It distinguishes post-execution feedback from intermediate runtime states and finds that their performance differs across task difficulty.

1 INTRODUCTION

  • Self-debugging with self-generated tests generates an initial program and test suite, executes the program on those tests, then uses execution feedback to revise it.The approach may reduce reliance on feedback from humans or stronger models.Source: §1 · p. 1
  • The study evaluates four advanced LLMs on self-contained Python problems from HumanEval, MBPP, and LiveCodeBench.It compares models’ ability to reflect on and debug code using post-execution and in-execution information.Source: §1 · p. 2
  • Self-generated tests may mislabel program correctness, and post-execution debugging also depends on recognizing when feedback is faulty.The authors define this bias as misalignment between self-testing labels and true program labels.Source: §1 · p. 2
  • In-execution self-debugging shows promising improvements on basic and competitive tasks, while post-execution struggles on basic problems but may improve on LiveCodeBench.The authors attribute the discrepancy to bias from self-generated tests.Source: §1 · p. 2

2 RELATED WORK

Related work covers methods that improve generated code through training, candidate selection, and execution-based refinement. This paper revisits self-debugging, focusing on the potential and limitations of using self-generated tests.

Code Generation

  • Self-debugging uses execution feedback to refine generated code, and this paper assesses its effectiveness with self-generated tests on basic and competitive benchmarks.The cited work describes self-debugging as a cost-effective way to improve inference efficiency without increasing the sample budget [Zhang et al., 2023a].Source: §2 · p. 3
  • Prior code-generation approaches include optimizing model training, selecting candidates with code-and-test agreement, and ranking multiple candidates.The passage cites CodeT for agreement-based filtering [Chen et al., 2023], coder-reviewer [Zhang et al., 2023b], and code-ranker [Inala et al., 2022].Source: §2 · p. 3

Self-Debug with LLMs

  • Execution-based refinement is common in self-debugging, with prior work using execution results, explanations, example tests, runtime information, or human intervention.The passage cites Self-Debugging [Chen et al., 2024b], Self-Edit [Zhang et al., 2023a], and LDB [Zhong et al., 2024] among these approaches.Source: §2 · p. 3

3 SELF-DEBUGGING WITH SELF-GENERATED TESTS

The framework uses execution feedback from model-generated tests to revise programs, distinguishing information collected after execution from intermediate runtime states. In-execution debugging represents execution as a trace of basic blocks and variable states that can guide code refinement.

Framework Overview

  • The model first generates an initial program and test cases from the problem specification, then an executor runs the program on the generated tests and collects execution information.Each test pairs an input with its expected output.Source: §3 · p. 3

Post-Execution Self-Debugging

  • Post-execution debugging compares each program output with the test’s expected output and uses failed tests, outputs, and related errors to revise the program.The revised program is generated from the original code and the failed test’s input, expected output, and observed output.Source: §3 · p. 4

In-Execution Self-Debugging

  • In-execution debugging uses intermediate states during execution, rather than only post-run outputs or error messages, to provide finer-grained information about program behavior.The approach divides the program into basic blocks, each a linear sequence of statements with one entry and one exit.Source: §3 · pp. 3–4
  • In-execution debugging records a trace of blocks and intermediate variable states, which the model analyzes to identify issues and refine the program.The revised code is generated from the original program, test input, and execution trace.Source: §3 · p. 4

4 EXPERIMENTS

Across self-contained Python tasks, the experiments compare post-execution and in-execution self-debugging with self-generated tests on basic and competitive benchmarks. Post-execution results vary by task and can be undermined by unreliable tests, while in-execution debugging shows gains across benchmarks but is not uniformly effective.

4.1 EXPERIMENTAL SETUP

  • The study evaluates four chat models, classifies HumanEval and MBPP as basic and LiveCodeBench as competitive programming problems, and prompts models to generate ten diverse, extensive tests per problem.The study selects 450 LiveCodeBench problems published between September 2023 and September 2024.Source: §4.1 · pp. 4–5

4.2 RQ1: POST-EXECUTION SELF-DEBUGGING STRUGGLES ON BASIC PROBLEMS

  • On HumanEval and MBPP, self-generated-test debugging declined for Llama-3-70B-Instruct and Qwen2.5-Coder-7B-Instruct, while other models consistently declined on HumanEval.MBPP could improve initially for the other models, but further feedback and iterations eventually put performance below initial generation; test-output prediction was harder than input generation, and unreliable outputs complicate repair.Source: Table 2 · p. 6, §4.2 · pp. 6–7

4.3 RQ2: BIAS FROM SELF-TESTING LEADS TO INCONSISTENCY ACROSS TASKS

  • On LiveCodeBench, GPT-4o’s label-feedback results improved across difficulty levels, but detailed feedback reduced easy-problem performance and underperformed label feedback overall.Other models’ overall performance decreased, with notable declines on easy problems.Source: Table 4 · p. 7, §4.3 · pp. 7–8
  • Self-generated tests can mislabel correct programs as false negatives and flawed programs as false positives, biasing post-execution evaluation and potentially preventing needed repairs.The reported label analysis describes false negatives as more prevalent than true negatives on HumanEval and MBPP, while lower performance on harder tasks makes negative labels more likely to align with actual correctness.Source: §4.3 · p. 8, Figure 2 · p. 8

4.4 RQ3: IN-EXECUTION REASONING HELPS SELF-DEBUGGING

  • In-execution debugging segments programs into control-flow-graph basic blocks and exposes intermediate runtime states, leaving models to assess correctness from test inputs and those states.Post-execution labels and execution-result details are withheld.Source: §4.4 · pp. 8–9
  • In-execution self-debugging improved results on LiveCodeBench for all models; GPT-4o’s MBPP pass rate rose from 76.5% to 79.1% after two iterations.Some tasks and iterations nevertheless showed slight degradation.Source: Table 6 · p. 9, Table 5 · p. 9, §4.4 · p. 9
  • In-execution debugging depends heavily on LLM code reasoning and has no formal guarantee of success; the paper reports a pass-rate drop for Llama-3-70B-Instruct on MBPP.The conclusion presents this paradigm as having potential to better align programs with requirements in real-world scenarios.Source: §4.4 · p. 10

5 DISCUSSION

The discussion identifies test quality and runtime information as promising directions for improving self-debugging, while noting that effective refinement requires strong code reasoning. Self-generated tests may also reduce reliance on labor-intensive human test creation.

5 DISCUSSION

  • Improving the quality of LLM-generated tests is important because biased evaluations make post-execution self-debugging struggle on basic problems.The authors suggest iterative test refinement using execution information and debugging outcomes.Source: §5 · p. 10
  • More sophisticated runtime-information collection and analysis, including clearer traces, code coverage, and execution paths, are proposed for future work.The paper reports superior in-execution performance to post-execution in certain tasks.Source: §5 · p. 10
  • Effective self-debugging requires models to locate program faults and reason through how code logic relates to execution feedback.Source: §5 · p. 10
  • Self-generated tests offer a route toward self-improvement that can ease reliance on labor-intensive human-generated tests and supervision.Source: §5 · p. 10

6 CONCLUSION

The paper examines self-debugging with self-generated tests through post-execution and in-execution paradigms. In self-contained Python tasks, in-execution self-debugging consistently outperforms post-execution across basic and competitive problems, while post-execution struggles on basic tasks.

6 CONCLUSION

  • Post-execution self-debugging struggles on basic tasks, and self-generated-test bias can produce inconsistent results across problem levels.Source: §6 · p. 10
  • The paper formalizes post-execution and in-execution self-debugging as two paradigms for studying code-generation repair with self-generated tests.Source: §6 · p. 10
  • In-execution self-debugging, using intermediate runtime information, consistently outperforms post-execution on both basic and competitive tasks.Source: §6 · p. 11

A CASE STUDY

A GPT-4o case study contrasts post-execution self-debugging, which changes a correct program after a faulty self-generated test, with in-execution self-debugging, which retains the original answer after inspecting its trace.

A CASE STUDY

  • A faulty self-generated test leads post-execution self-debugging to wrongly revise an initially correct program, while in-execution self-debugging keeps it.The in-execution approach analyzes runtime traces block by block against expected behavior.Source: A CASE STUDY · p. 15, Figure 3 · p. 15
  • An initially correct completion was judged against an erroneous self-generated test expecting 1 when the specification calls for 4, causing a wrongly revised program and biasing feedback away from the specification.Source: A CASE STUDY · p. 15

B EXAMPLES OF PROGRAM CONTRACTS

The examples show program contracts as assertions specifying conditions necessary for a valid input.

B EXAMPLES OF PROGRAM CONTRACTS

  • Program contracts consist of assertions that specify conditions necessary for a valid input.Source: Figure 4 · p. 16

C ANALYSIS ON THE NUMBER OF SELF-GENERATED TESTS

More self-generated tests introduce harder edge cases and lower test-suite accuracy; post-execution self-debugging declines slightly as test counts rise, while in-execution self-debugging improves over initial generation.

C ANALYSIS ON THE NUMBER OF SELF-GENERATED TESTS

D PROMPTS

The appendix presents prompts for code generation, test generation, and debugging across HumanEval, MBPP, and LiveCodeBench, including distinct LiveCodeBench input formats.

D PROMPTS

Related papers

Loading 2501.12793v1…