Large Language Model Guided Self-Debugging Code Generation

1 Introduction

• Existing multi-agent code-generation frameworks can require substantial computational resources and complex coordination, challenging real-world deployment and scalability.Source: §1 · p. 2
• Its programmer and executor agents are paired with an error handler, example call detector, and function signature converter for Python code generation and debugging.The framework is designed to reduce computational consumption while supporting code validation, case testing, and error analysis.Source: §1 · p. 2
• PyCapsule reports up to 5% accuracy improvement on HumanEval, 10% on HumanEval-ET, and 25% on BigCodeBench.The reported gains use GPT-3.5-Turbo-1106 and GPT-4-Preview-1106 on the first two benchmarks, and Qwen-2.5-coder-instruct 7B on BigCodeBench.Source: §1 · p. 2
• The authors report that normalized debugging success decreases as self-debugging attempts increase, possibly because problem complexity combines with limited and noisy error messages.Source: §1 · p. 2

2 Related Work

• Prior work spans prompt engineering, multi-agent systems, and iterative debugging with runtime feedback [5] [13] [16].Source: §2 · p. 2
• MapCoder [16] uses retrieval, planning, coding, and debugging agents, while AgentCoder [13] uses programmer, test-designer, and test-executor agents.LDB [38] instead uses runtime information, code segmentation, intermediate-state tracking, and stepwise verification; CodeCoT [12] is described as limited to syntax-error refinement.Source: §2 · pp. 2–3
• Earlier approaches can be resource-intensive and struggle to use error messages fully; added test cases may increase computational demands, reduce reliability, and require more sophisticated agent architectures.Source: §2 · p. 3
• PyCapsule addresses these challenges with a streamlined modular architecture that combines operations-research and computer-science knowledge in a two-agent pipeline.Source: §2 · p. 3

3.3 Code Generation Workflow

• The workflow extracts generated code and setup instructions, then executes the processed code and dataset tests in Docker; failures trigger refined feedback for up to five debugging attempts.A runtime timer guards execution, and each fix attempt uses feedback from the preceding execution.Source: Figure 1 · p. 3, §3.3 · p. 5

3.1 Programmer and Executor Agents

• The programmer generates and fixes code; the executor validates it, returns successful code, or provides diagnostics that initiate another debugging iteration.The programmer uses generation and fix prompts, with only the latest problem-solution pair retained; on MBPP with Qwen2.5-Coder-7B-Instruct, performance fell from 80.7% with one pair to 77.6% with two and 76.7% with three.Source: §3.1 · p. 4

3.2 Supportive Modules

• Three deterministic modules support the agents: a signature converter, an example call detector, and an error handler that refines execution feedback.They provide structured function signatures and example calls, remove potentially looping internal example calls, and filter irrelevant tracebacks or verbose recursion errors to limit debugging tokens.Source: §3.2 · pp. 4–5

4 Experiments

• PyCapsule was evaluated on HumanEval, HumanEval-ET, MBPP, MBPP-ET, and BigCodeBench(full), using success rate as its measure and one RTX 3090 for GPU experiments.The ET benchmarks include more test cases and enriched problem descriptions.Source: §4.1 · p. 5, §4 · p. 5

4.2 Results and Analysis

• PyCapsule reaches state of the art on HumanEval, HumanEval-ET, and BigCodeBench(full); on BigCodeBench(full), Qwen-2.5-Coder-Instruct 7B improves 25% over its 41.0% standalone zero-shot result and exceeds the 32B model’s 49.6% by more than 15%.The reported success-rate comparison is in Table 1.Source: Table 1 · p. 6, §4.2 · pp. 6–7
• PyCapsule performs strongly on MBPP but trails AgentCoder and Qwen-2.5-Coder-Instruct, whose comparison uses three-shot prompting; the authors retain consistent prompting across datasets for comparability.The signature converter generates actionable function signatures under that consistent prompting approach.Source: §4.2 · p. 6
• Across datasets, the relative success ratio of successful test cases declines exponentially as the number of self-debugging attempts increases.The ratio is the share of all successful test cases solved at a given attempt.Source: Figure 2 · p. 7
• Independent debugging-attempt influence decays exponentially, with effectiveness approaching zero after five attempts for Qwen-2.5-Coder-Instruct 7B and three for GPT-4-Preview-1106; the architecture therefore caps attempts at five.The fitted exponential decay had R2 close to one across every model and dataset.Source: §4.2 · p. 7

4.3 Discussion

• First-attempt accuracy varies by dataset and model configuration, and the discussion attributes strong first-attempt performance partly to dynamically augmented system prompts.The authors note that framework design and prompt engineering affect performance before self-debugging begins.Source: §4.3 · p. 7
• The authors report that PyCapsule surpasses other self-debugging methods overall while limiting the programmer agent to five attempts and at most six API calls per problem.Mean HumanEval API calls were 2.09, 1.38, and 1.55 for GPT-3.5-Turbo-1106, GPT-4-Preview-1106, and Qwen-2.5-Coder-Instruct-7B, respectively.Source: §4.3 · p. 8

• PyCapsule integrates two AI agents with high-efficiency self-debugging modules for automated code generation.Source: §5 · p. 8
• Across multiple datasets, PyCapsule surpasses existing debugging-based approaches.Source: §5 · p. 8
• PyCapsule enhances accuracy while using smaller language models and fewer API calls, offering a cost-effective alternative without compromising performance.Source: §5 · p. 8
• Iterative self-debugging feedback and dynamic prompt augmentation contribute to PyCapsule’s robustness and improvements.Source: §5 · p. 8

• Cumulative accuracy increases as debugging attempts resolve additional problems.Source: A Contribution of Each Debugging Attempt · p. 12
• The independent contribution of successive attempts gradually decreases even as their cumulative effect improves problem-solving accuracy.Source: A Contribution of Each Debugging Attempt · p. 12
• Table 2 reports accuracy and sample standard deviations for attempts 0–5 with GPT-4-Preview-1106, GPT-3.5-Turbo-1106, and Qwen 2.5 Coder Instruct.Source: Table 2 · p. 12

• 81.9% vs. 48.1% and 57.3% on HumanEval, and 77.2% vs. 37.2% and 42.7% on HumanEval-ET, are PyCapsule’s GPT-3.5 first-attempt accuracies against MapCoder and AgentCoder.Source: B First Attempt Accuracy · p. 13, Table 3 · p. 13
• With GPT-4, PyCapsule remains competitive but trails MapCoder on some MBPP benchmarks; Qwen2.5-Coder-Instruct outperforms it on MBPP using three-shot prompting.Source: B First Attempt Accuracy · p. 13
• PyCapsule’s first-attempt performance is attributed to clear prompts, contextual information, chain-of-thought reasoning, a defined programmer persona, and tuned implementation settings.Source: B First Attempt Accuracy · p. 13

C HumanEval Experiments

• Figures 4–5 trace HumanEval task attempts; Figure 4 summarizes three-experiment means and standard deviations, while Figure 5 shows three GPT-4-1106 repeats of up to five attempts.Source: C HumanEval Experiments · p. 14, Figure 4 · p. 14, Figure 5 · p. 14

D Pseudo-code of PyCapsule

• The pseudocode generates code, tests cases in a Python container, then repeats generation in error-guided Fix Mode until tests pass or the attempt limit is reached.Source: D Pseudo-code of PyCapsule · p. 15

E.1 Signature Converter

• The Signature Converter uses the first test case to infer a function name, structured signature, and example call without revealing the expected output.Source: §E.1 · p. 16

E.2 Example Call Detection

• 21 of 164 HumanEval problems and 44 of 344 MBPP problems contained embedded example calls in one GPT-3.5-Turbo-1106 experiment.The calls appeared despite explicit system prompts to avoid them.Source: §E.2 · p. 16
• Embedded calls can execute automatically during test evaluation, potentially bypassing runtime safety mechanisms for controlled code execution.Source: §E.2 · p. 16
• The module scans generated solutions and removes detected example calls before execution.For example, it removes calls such as foo(4).Source: §E.2 · p. 16

E.3 Error-Handling

• Error handling identifies error types, filters tracebacks to main.py, and truncates verbose recursion errors to keep feedback actionable and within the LLM context length.AssertionError feedback can tell the programmer that a test failed and ask it to improve the solution logic.Source: §E.3 · p. 16

F.1 Code Generation Mode

• Code Generation Mode asks for step-by-step reasoning, preserves the function signature, excludes example calls, and specifies requirements and code sections.Source: §F.1 · p. 17

F.2 Fix Mode

• Fix Mode supplies the prior error and conversation history, while retaining the step-by-step reasoning, signature, example-call, requirements, and code-format instructions.Source: §F.2 · p. 17

G Token Usage

• 1.35–3.31 debugging attempts across models and datasets highlight PyCapsule’s efficiency in reaching solutions with minimal iterations.The analysis covers GPT-3.5, GPT-4, and Qwen2.5-Coder-Instruct on standard code-generation benchmarks.Source: G Token Usage · p. 18, Table 4 · p. 18
• Token use was measured with tiktoken for GPT-3.5 and GPT-4 and Qwen2.5-Coder-Instruct’s native tokenizer, then averaged per problem.Total token usage was divided by the number of problems; inference calls, including debugging iterations, were logged.Source: G Token Usage · p. 18
• Token-counting methodologies vary across frameworks, making direct numerical comparisons challenging.Source: G Token Usage · p. 18