PaperlayerView exact revision on arXiv

Source-linked AI summary

Teaching Large Language Models to Self-Debug

Xinyun Chen, Maxwell Lin, Nathanael Schärli, Denny Zhou

arXiv:2304.05128v2cs.CLcs.AI

TL;DR

Generating correct code in one attempt is difficult, and prior methods often rely on external feedback. SELF-DEBUGGING teaches LLMs to iteratively inspect and explain their own predictions using few-shot prompting. It achieves state-of-the-art performance across code-generation domains, with improvements of up to 12% on tasks with unit tests and consistent gains on Spider without them.

  • Problem

    Generating correct code in one attempt is challenging, and prior work indicates that LLMs struggle to correct code without external feedback such as unit tests or human instructions.

  • Method

    SELF-DEBUGGING uses few-shot prompting to have an LLM execute, explain, and iteratively debug its own predicted code without additional model training.

  • Results

    SELF-DEBUGGING achieves state-of-the-art performance across text-to-SQL, code translation, and text-to-Python tasks, improving baselines by 2−3% on Spider and up to 12% on TransCoder and MBPP.

  • Takeaways & Limitations

    The results support improving LLM coding performance by teaching models to iteratively debug their own predictions rather than generate correct code from scratch.

  • Takeaways & Limitations

    The paper identifies improving the model’s ability to understand code, identify errors, and follow error messages to fix bugs as future work.

Abstract

from arXiv · show

Large language models (LLMs) have achieved impressive performance on code generation. However, for complex programming tasks, generating the correct solution in one go becomes challenging, thus some prior works have designed program repair approaches to improve code generation performance. In this work, we propose Self-Debugging, which teaches a large language model to debug its predicted program via few-shot demonstrations. In particular, we demonstrate that Self-Debugging can teach the large language model to perform rubber duck debugging; i.e., without any human feedback on the code correctness or error messages, the model is able to identify its mistakes by investigating the execution results and explaining the generated code in natural language. Self-Debugging achieves the state-of-the-art performance on several code generation benchmarks, including the Spider dataset for text-to-SQL generation, TransCoder for C++-to-Python translation, and MBPP for text-to-Python generation. On the Spider benchmark where there are no unit tests to verify the correctness of predictions, Self-Debugging with code explanation consistently improves the baseline by 2-3%, and improves the prediction accuracy on problems of the hardest level by 9%. On TransCoder and MBPP where unit tests are available, Self-Debugging improves the baseline accuracy by up to 12%. Meanwhile, by leveraging feedback messages and reusing failed predictions, Self-Debugging notably improves sample efficiency, and can match or outperform baseline models that generate more than 10x candidate programs.

1 INTRODUCTION

Correct code is difficult to generate in one attempt, motivating approaches that repair predictions using execution results and feedback. SELF-DEBUGGING instead teaches LLMs to inspect and explain their own code through few-shot prompting, improving results across code-generation tasks.

  • Single-attempt code generation is challenging for many programming tasks, motivating methods that repair incorrect predictions rather than discard them.
  • SELF-DEBUGGING teaches an LLM to debug its predicted code via few-shot prompting without additional model training or human feedback.
  • 2−3% baseline improvement was consistent on Spider with code explanation, while accuracy on the most complicated SQL queries improved by 9%.
  • Up to 12% accuracy improvements were achieved on TransCoder and MBPP using unit tests with code explanation, while code explanation alone improved code translation by 2−3%.
  • The method generates code, executes it, explains it in natural language, and uses the explanation and execution results as iterative feedback.

2 PROMPTING FOR CODE GENERATION

The paper situates SELF-DEBUGGING within few-shot prompting and execution-based selection, where multiple generated programs can be evaluated using execution behavior and unit tests.

  • Few-shot prompting supplies input-output demonstrations so the model follows the demonstrated format for generating a solution.
  • For text-to-SQL generation, few-shot prompts prepend the target question with question–SQL pairs and may include a high-level task instruction.
  • Multiple sampled programs can improve code-generation performance, and execution can help select the final prediction.
  • Some code-generation tasks provide unit tests as input-output pairs, and programs that fail them can be filtered before execution-based majority voting.

3 SELF-DEBUGGING FRAMEWORK

SELF-DEBUGGING iteratively combines generation, explanation, and feedback to refine predicted programs. Its feedback can range from correctness messages to unit-test results, execution traces, and model-generated code explanations.

  • One SELF-DEBUGGING turn has three steps: Generation, Explanation, and Feedback, using a pretrained LLM without finetuning.
  • Generation produces candidate programs, Explanation describes their semantics or execution traces, and Feedback communicates predicted correctness for further debugging.
  • The process stops when feedback states that the prediction is correct or when the maximum number of debugging turns is reached.
  • Simple feedback provides only a correctness judgment, omitting the Explanation step in a full debugging turn.
  • Unit-test feedback adds runtime errors and execution results from failed tests, supplying richer debugging information when tests are available.
  • Code Explanation feedback asks the LLM to describe generated code instead of predicting error messages, enabling rubber duck debugging without extra guidance.

4 APPLICATIONS

The paper applies SELF-DEBUGGING to text-to-SQL, C++-to-Python translation, and text-to-Python generation under settings with no tests, full tests, or only partial tests.

  • Text-to-SQL generation: Text-to-SQL generation maps a question and database information to an SQL query, and Spider provides no unit tests for verifying predicted queries.
  • Text-to-SQL generation: The text-to-SQL debugging process infers the required return type, executes the query, and explains its clauses, returned columns, and overall meaning.
  • Code translation: C++-to-Python translation uses TransCoder’s 560 problems, each with 10 unit tests, to evaluate translated Python functions.
  • Code translation: For translation, SELF-DEBUGGING is applied when predicted Python code fails all unit tests and continues until tests pass or the debugging-turn limit is reached.
  • Text-to-Python generation: MBPP evaluates text-to-Python generation with 500 problems and 3 unit tests per problem, while only the first test is shown and the remaining two remain hidden.

5 EXPERIMENTS

Experiments show that SELF-DEBUGGING consistently improves code-generation performance across prompting baselines, models, and feedback settings. It also improves sample efficiency, performs especially well on harder Spider problems, and benefits from execution information.

  • 5.1 MAIN RESULTS: SELF-DEBUGGING consistently improves performance over prior prompting-based code reranking approaches.
  • 5.1 MAIN RESULTS: Richer feedback generally improves SELF-DEBUGGING, especially when execution information is included.
  • 5.1 MAIN RESULTS: Over 12% improvement on TransCoder and 8% improvement on MBPP are achieved by GPT-4 through self-debugging.
  • 5.1 MAIN RESULTS: 6% improvement on MBPP is obtained by StarCoder with unit-test execution and execution-trace feedback despite its lower baseline performance.
  • 5.2 ABLATION STUDIES: Using SELF-DEBUGGING from 8 samples outperforms the baseline using 32 samples, while greedy-decoding SELF-DEBUGGING matches the baseline using 16 samples on Spider.
  • 5.2 ABLATION STUDIES: 9% accuracy improvement occurs on extra-hard Spider problems, where code explanation helps identify discrepancies between questions and predicted SQL queries.

6 RELATED WORK

Related work spans code-generation models, execution-based selection, prompting with intermediate outputs or feedback, code repair, and feedback-based training. SELF-DEBUGGING combines execution results and self-generated explanations for repair without training a separate model.

  • Code-generation research covers text-to-code generation, code translation, and models designed for specialized domains or broad code tasks.
  • Execution-based approaches select among multiple generated programs using majority voting or reranking based on execution outcomes.
  • Prompting methods elicit useful intermediate outputs, and SELF-DEBUGGING uses code explanation plus separate feedback-generation prompts for debugging.
  • Neural code-repair methods use code alone or additional execution traces, compiler errors, and execution errors as repair information.
  • SELF-DEBUGGING uses execution results and self-generated code explanations to aid repair without training a separate repair model.
  • Feedback-based training includes RLHF and methods that train models to refine outputs using external feedback on quality or preferences.

7 CONCLUSION

The paper presents SELF-DEBUGGING as a way for language models to debug their own generated code through rubber duck debugging without human instructions. Across code-generation domains, it improves accuracy and sample efficiency, while better debugging capabilities remain future work.

  • SELF-DEBUGGING enables a language model to identify and fix bugs in its own generated code without human instructions.
  • 2−3% baseline improvement occurs on text-to-SQL generation without specified unit tests, with a 9% gain on the hardest problems.
  • Up to 12% baseline accuracy improvement occurs on code translation and text-to-Python tasks with available unit tests.
  • The authors identify improving code understanding, error identification, error-message following, and code explanation as important future work.

A SELF-DEBUGGING WITH INITIAL SAMPLES FROM DIFFERENT LLMS

Table 5 compares SELF-DEBUGGING across Codex and GPT-3.5 on Spider, TransCoder, and MBPP, using task-specific pre-debugging baselines. The results show that model choice affects initial generation and subsequent debugging performance.

  • Table 5 reports SELF-DEBUGGING results for GPT-3.5 and Codex on Spider, TransCoder, and MBPP.
  • Spider uses Codex as its baseline, whereas TransCoder and MBPP use GPT-3.5 as their highest pre-debugging baselines.
  • GPT-3.5 Spider accuracy rises from 59.9% with zero-shot greedy decoding to 71.1% with few-shot prompting.
  • Codex SELF-DEBUGGING is on par with GPT-3.5 on TransCoder and close to it on MBPP, outperforming GPT-3.5 on MBPP Simple and UT.
  • Codex outperforms GPT-3.5 for Spider SELF-DEBUGGING, while code explanation improves GPT-3.5 performance.

B TRANSCODER COMPARISON TO BASELINE MODELS

The supplied passages identify Table 6 as a comparison between SELF-DEBUGGING and baselines without debugging, and identify Table 6’s dataset as TransCoder C++-to-Python translation.

  • Table 6 compares SELF-DEBUGGING with baseline models that do not use debugging.
  • The TransCoder dataset evaluates C++-to-Python translation.

C.1 CODE TRANSLATION

On TransCoder, SELF-DEBUGGING gains most of its improvement from the first debugging turn and can outperform baselines using substantially more initial samples. Code explanation also contributes a consistent additional gain.

  • Debugging turns: Around 12%: one debugging turn with full feedback improves TransCoder accuracy over greedy decoding.The largest improvement comes from the first debugging turn.
  • Initial samples: SELF-DEBUGGING with greedy decoding outperforms the baseline using 5 samples and approaches the baseline using 10 samples.This indicates improved sample efficiency relative to generating multiple baseline candidates.
  • Code explanation: 2 −3%: code explanation alone consistently improves performance across different numbers of samples.This comparison does not include SELF-DEBUGGING, but isolates the contribution of adding explanations.
  • Ablation setup: The ablation varies both the number of debugging turns and the number of initial samples, while distinguishing Codex+Expl. from SELF-DEBUGGING.Codex+Expl. generates explanations for input C++ code without performing SELF-DEBUGGING.

C.2 TEXT-TO-PYTHON GENERATION

On MBPP, SELF-DEBUGGING improves greedy-decoding accuracy, while combining unit-test execution with code explanation further improves debugging performance. Examples show that the corrected prediction can differ substantially from the initial code.

  • Accuracy: Over 8%: SELF-DEBUGGING increases MBPP accuracy with greedy decoding.The improvement is reported for initial Python code generation using a 3-shot prompt.
  • Explanation effects: Code-explanation gains are less significant on MBPP because unit tests make wrong predictions easier to recognize.The context-length limit also reduces the number of exemplars when code explanation is added to feedback.
  • Feedback signals: A unit-test-feedback example shows SELF-DEBUGGING fixing a code translation error where simple feedback fails.The comparison is presented in Figure 9.
  • Prediction changes: Adding code explanation can correct a translation by making the model check the end of a string.Figure 10 contrasts this corrected translation with the baseline prompt.
  • Prediction changes: SELF-DEBUGGING can produce an MBPP prediction that is very different from the initial code.Figure 11 provides an example of this change after debugging.

E TEXT-TO-SQL GENERATION

The text-to-SQL material presents question-and-query examples spanning joins, aggregation, set intersection, and nested filtering. The examples pair database schemas and questions with candidate SQL queries, sometimes followed by explanation-based correctness checks.

  • Schemas: The examples include schemas for departments, cities, farms, competitions, customers, orders, stations, status records, students, and allergies.These schemas provide the database context for the text-to-SQL demonstrations.
  • Query construction: The demonstrations translate natural-language questions into SQL queries involving joins, grouping, ordering, limiting, and set intersection.Examples include finding a hosted city, the most common department-creation year, and customers with two order statuses.
  • Query construction: One customer query uses INTERSECT to return names appearing among customers with both “On Road” and “Shipped” orders.The question explicitly asks for customer names with both statuses.
  • Nested filtering: A station query asks for the average longitude of stations that never exceeded 10 available bikes and uses a grouped subquery with MAX(bikes_available) > 10.The schema includes station and status tables linked by station identifiers.

F CODE TRANSLATION

The code-translation examples use explanation and feedback to identify semantic mismatches between C++ programs and Python translations, then correct the translations.

  • Baseline and explanation prompts: The prompts ask the model to explain source code before translating C++ programs into Python.The examples include line-by-line explanations of bitwise incrementation, factorial, and pairwise-product programs.
  • Baseline and explanation prompts: The generated explanations connect individual C++ operations to their intended Python behavior.Examples explain recursion base cases, nested loop bounds, accumulated products, and bitwise carry propagation.
  • Feedback-based correction: Incorrect translations are repaired using direct feedback that states the Python code does not match the C++ program.The feedback prompts reuse the original translation and request a corrected Python version.
  • Unit-test feedback: Unit-test feedback exposes concrete translation failures, including an incorrect factorial base case and an undercounted pairwise-product result.The examples report RecursionError for factorial at n=0 and an actual result of 16 when the expected pairwise-product result is 25.
  • Unit-test feedback: The corrected factorial translation uses n == 0 as its base case, matching the original C++ program.The correction returns 1 at zero and otherwise recurses on n - 1.

G TEXT-TO-PYTHON GENERATION

The text-to-Python examples pair natural-language programming tasks with assertions and candidate implementations, providing executable checks for generated functions.

  • Unit-test prompts: Assertions define concrete expected outputs for the generated functions.For example, the prompts test tuple similarity, primality classification, and selecting three largest numbers.
  • Task prompts: The examples specify Python tasks through docstrings describing the required function behavior.Tasks include selecting largest list elements, finding tuple intersections, identifying non-primes, and other small programming problems.
  • Candidate implementations: Candidate solutions implement the requested tasks with ordinary Python and library operations.Examples use heapq.nlargest, set intersection, and a square-root loop for primality testing.

G.2 SIMPLE FEEDBACK PROMPT (6-SHOT)

The simple-feedback prompt presents six-shot examples in which candidate Python programs receive binary correctness feedback, sometimes alongside assertions and problem descriptions.

  • G.2 SIMPLE FEEDBACK PROMPT (6-SHOT): Some examples combine assertions with binary feedback to show whether a candidate satisfies the requested output.The prompt includes assertions for bit comparison, word extraction, domino tiling, and smallest-item selection.
  • G.2 SIMPLE FEEDBACK PROMPT (6-SHOT): The domino-tiling example replaces a recursive candidate with a dynamic-programming implementation after negative feedback.The revised program initializes A and B arrays and iterates from 2 through n before returning A[n].
  • G.2 SIMPLE FEEDBACK PROMPT (6-SHOT): The six-shot prompt includes tasks for bit comparison, regex-based word extraction, squaring list elements, string rotations, domino tilings, and selecting smallest items.Each example pairs a task description with a candidate function and a feedback message.
  • G.2 SIMPLE FEEDBACK PROMPT (6-SHOT): Incorrect candidates receive feedback instructing the model to fix the code.The examples mark failed regex, domino-tiling, and other candidate programs as wrong.
  • G.2 SIMPLE FEEDBACK PROMPT (6-SHOT): Correct candidates receive feedback stating that the code above is correct.This feedback appears for bit comparison, regex extraction, squaring numbers, and selecting smallest items.
  • G.3 UNIT TEST FEEDBACK PROMPT (6-SHOT): The same binary feedback pattern is illustrated for unit-test examples, including candidates that pass or fail assertions.The examples report both successful and unsuccessful outputs for bit comparison, regex extraction, rotations, and domino tiling.
  • G.3 UNIT TEST FEEDBACK PROMPT (6-SHOT): A corrected regex example adds the missing re import after feedback reports a NameError.The revised candidate begins with import re and then defines the same regex-based function.
  • G.3 UNIT TEST FEEDBACK PROMPT (6-SHOT): The unit-test examples also show correct feedback for square-number, rotation, smallest-item, and domino-tiling candidates.The feedback explicitly reports outputs matching the corresponding assertions for these examples.

G.4 UNIT TEST + EXPL. FEEDBACK PROMPT (3-SHOT)

This 3-shot prompt combines unit-test assertions, line-by-line code explanations, and feedback to iteratively diagnose and repair generated Python programs. The examples show both successful validation and correction of implementations that fail their assertions.

  • G.4 UNIT TEST + EXPL. FEEDBACK PROMPT (3-SHOT): The prompt supplies a task, assertion tests, generated code, and a line-by-line explanation before requesting feedback on correctness.This structure is illustrated for both domino tiling and regex-based word extraction tasks.
  • G.4 UNIT TEST + EXPL. FEEDBACK PROMPT (3-SHOT): The regex example explains that find_char_long uses re.findall() with a pattern matching words of at least four characters.The explanation covers the word-boundary, word-character, and repetition components of the regular expression.
  • G.4 UNIT TEST + EXPL. FEEDBACK PROMPT (3-SHOT): A missing re import is identified from the execution error, after which the corrected function passes its assertion.The feedback explicitly reports NameError: name ’re’ is not defined and contrasts it with the successful assertion result.
  • G.4 UNIT TEST + EXPL. FEEDBACK PROMPT (3-SHOT): The initial domino-tiling implementation is marked wrong despite passing count_ways(2) == 3, prompting a correction.The feedback separates passing the displayed assertion from the overall correctness judgment.
  • G.4 UNIT TEST + EXPL. FEEDBACK PROMPT (3-SHOT): The corrected domino-tiling implementation uses dynamic-programming arrays A and B with recurrences for fully filled and L-shaped endings.The explanation states that A[n] is returned as the number of tilings.
  • G.4 UNIT TEST + EXPL. FEEDBACK PROMPT (3-SHOT): For find_Rotations, the prompt tests whether the implementation returns the required minimum rotations for the string "aaaa".The task asks for the minimum number of rotations needed to obtain the same string.
  • G.4 UNIT TEST + EXPL. FEEDBACK PROMPT (3-SHOT): The feedback exposes a mismatch because find_Rotations("aaaa") returns 0 while the assertion requires 1, directing the model to fix the code.The example therefore demonstrates feedback-driven repair after an assertion failure.
Loading 2304.05128v2…