PaperlayerView exact revision on arXiv

Source-linked AI summary

CodeCoT: Tackling Code Syntax Errors in CoT Reasoning for Code Generation

Dong Huang, Qingwen Bu, Yuhao Qing, Heming Cui

arXiv:2308.08784v2cs.SEcs.AI

TL;DR

CoT can generate logically coherent code that fails syntax requirements during execution, limiting its effectiveness in code generation. CodeCoT pairs CoT-based code development with generated tests and local self-examination, iteratively refining code from error feedback. It reports improved pass@1, including gains on HumanEval and MBPP.

  • Problem

    CoT-generated code may follow the task’s logic but contain syntax errors that prevent execution and lower pass@1 below zero-shot results.

    Source: §1 · pp. 1–2

  • Method

    CodeCoT uses CoT for initial code generation, then executes the code against generated tests locally and iteratively revises it using error feedback.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    CodeCoT raises pass@1 from 75.6% to 79.3% on HumanEval and from 69.8% to 89.5% on MBPP.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The evaluations support self-examination as a way to address runtime errors in CoT-based code generation.

    Source: §5 · p. 8

    The study focuses on closed-source models and does not evaluate open-source models; the authors also note that self-examination requires multiple LLM interactions, increasing generation overhead.

    Source: §6 · p. 8

Abstract

from arXiv · show

Chain-of-thought (CoT) has emerged as a groundbreaking tool in NLP, notably for its efficacy in complex reasoning tasks, such as mathematical proofs. However, its application in code generation faces a distinct challenge, i.e., although the code generated with CoT reasoning is logically correct, it faces the problem of syntax error (e.g., invalid syntax error report) during code execution, which causes the CoT result's pass@1 in HumanEval even lower than the zero-shot result. In this paper, we present Code Chain-of-Thought (CodeCoT) that integrates CoT with a self-examination process for code generation. CodeCoT begins with the LLMs using CoT for initial code development to ensure the generated code follows the correct logic flow. Then, CodeCoT will generate test cases to validate whether the code has syntax errors during the execution. CodeCoT then employs a self-examination phase, in which the generated code is executed against these test cases in the local environment. If the local environment raises error information (e.g., invalid syntax error), CodeCoT will iteratively refine the code based on the feedback information. Within this loop, CodeCoT can make sure their generated codes not only follow the logic flow of the code description, but the syntax error will also be addressed with the self-examination process. Our evaluation results reveal that CodeCoT improves the effectiveness of code generation. For example, CodeCoT increases pass@1 from 75.6% to 79.3% for the HumanEval dataset.

1. Introduction

CoT can produce logically structured code that nevertheless fails syntax requirements, reducing pass@1. CodeCoT adds self-examination with generated tests and local execution; reported results improve pass@1 on HumanEval and MBPP.

1. Introduction

  • CoT-generated code can follow the task’s logic yet violate syntax requirements, lowering pass@1 below zero-shot results.The stated examples include syntax errors that prevent code execution.Source: §1 · pp. 1–2
  • CodeCoT combines CoT-based code development with self-examination: it generates test cases, runs code locally, and refines code using error feedback.The process iterates when execution reports a syntax error.Source: §1 · p. 2
  • The paper presents CodeCoT as a way to address the gap between narrative logical reasoning and strict executable-code syntax.Its stated aim is to preserve logic flow while addressing syntax errors.Source: §1 · p. 2
  • 79.3% vs. 75.6% pass@1 on HumanEval and 89.5% vs. 69.8% on MBPP are reported for CodeCoT.The introduction reports these as improvements over existing methods.Source: §1 · p. 2

2. Related Work

The related work spans language-model scaling and chain-of-thought prompting, then turns to code-generation methods that use execution feedback or structural reasoning. CodeCoT is distinguished by targeting syntax requirements alongside logical reasoning.

2.1. Large language models

  • Language-model scaling work traces successive large models and studies relationships between model and dataset sizes and performance, including power-law findings.The passage cites work examining scaling and resulting performance.Source: §2.1 · p. 3

2.2. Chain of Thought Prompting

  • CoT research developed prompting variants including self-consistency, least-to-most prompting, bootstrapping, self-training, and verifier methods.Auto-CoT is described as clustering questions to support diverse zero-shot answer generation.Source: §2.2 · p. 3

2.3. Chain of thought application in LLMs

  • Applications of CoT and related prompting span reasoning, dialogue, translation, and code execution for arithmetic and date tasks.The passage contrasts Python-interpreter code generation with specialized modules and ad-hoc fixes.Source: §2.3 · p. 3

2.4. Enhancing Code Generation through Prompt Engineering

  • Prior code-generation approaches filter or rank outputs using execution behavior, or revise code using test outcomes, model predictions, or generated explanations.Examples include Self-Edit, Self-correct, CodeRL, Self-debug, Self-refine, and Reflexion.Source: §2.4 · p. 3
  • CodeCoT differs from parallel CoT strategies such as SCOT and CodeChain by targeting syntax requirements rather than submodule- or structure-level generation.The passage characterizes those strategies as focusing on code structure while CodeCoT uses self-examination for logic and syntax correctness.Source: §2.4 · p. 4

3. Methodology

CodeCoT starts from CoT-guided code generation and adds generated tests and local execution to examine and refine code. Its iterative feedback loop addresses execution errors while retaining the intended logic flow.

3.1. Overview

  • CodeCoT’s overview combines a CoT prompt, test-case generation, code generation, and self-examination through local execution.Execution errors are fed back to refine code, with a default of five iterations that users can specify.Source: §3.1 · p. 4

3.2. CoT in Code

  • The CoT prompt supplies a task description and a guidance example, and CoT reasoning guides the initial code’s logical flow.The guidance example need not directly match the task; it illustrates expected logic or structure.Source: §3.2 · p. 4

3.3. CodeCoT

  • CodeCoT generates test cases with the code-generation prompt, executes the draft in a local environment, and revises it using error feedback.The paper commonly requests five tests because dataset-provided tests average fewer than five; revised code is re-tested until it passes all tests.Source: §3.3 · p. 5

4. Evaluation

CodeCoT is evaluated across standard and enhanced code-generation benchmarks against model and prompt-engineering baselines. Results examine overall pass@1, runtime errors, refinement steps, component combinations, and generated-test effectiveness.

4. Evaluation

  • Evaluation uses pass@1 on HumanEval, MBPP, and their enhanced ET versions, comparing CodeCoT with LLM baselines and prompt-engineering strategies.The paper describes the ET versions as including more adequate test cases and being more challenging.Source: §4 · pp. 5–6

4.1. RQ1. How does CodeCoT perform?

  • CodeCoT reaches 79.3% and 89.5% pass@1 on HumanEval and MBPP, versus ChatGPT’s 57.3% and 52.2%; on ET datasets it reaches 69.5% and 63.0%.The ET results are compared with Self-Collaboration’s 56.1% and 49.5%.Source: Table 1 · p. 6, §4.1 · pp. 6–7

4.2. RQ2. Does CodeCoT fix syntax errors during the self-examination process?

  • RuntimeError shares on HumanEval are 30% for SCOT and 35% for CodeChain, versus 27% for 1-step and 2% for 5-step CodeCoT.RuntimeError includes errors such as SyntaxError; the table caption classifies Non-AssertError as SyntaxErrors.Source: Table 2 · p. 7, §4.2 · p. 7

4.3. How do self-examination steps affect CodeCoT’s effectiveness?

  • From 1 to 5 refinement steps, pass@1 rises from 71.3% to 79.3% on HumanEval, 81.7% to 89.5% on MBPP, 60.4% to 69.5% on HumanEval-ET, and 58.4% to 63.0% on MBPP-ET.The evaluation systematically varies the number of self-examination rounds.Source: Table 3 · p. 7, §4.3 · p. 7

4.4. RQ4. How does CodeCoT each component perform?

  • Combining CoT and self-examination yields 79.3% and 89.5% pass@1 on HumanEval and MBPP, above baseline strategies’ 75.6% and 69.8%.The individual CoT-prompt and self-examination components also improve on the Coder-only results.Source: Table 4 · p. 8, §4.4 · p. 8

4.5. Further discuss for the CodeCoT’s effectiveness with its components

  • Canonical-solution test-case pass@1 rises from 47.0% and 57.2% with Coder+Self-examination to 67.1% and 79.0% with CodeCoT on HumanEval and MBPP.The paper reports that CodeCoT generates more accurate self-testing cases that guide code refinement.Source: Table 5 · p. 8, §4.5 · p. 8

5. Conclusion

The conclusion presents CodeCoT as a self-examination approach for runtime errors in CoT-based code generation and reports improved pass@1 across datasets.

5. Conclusion

  • CodeCoT uses generated tests in self-examination to detect runtime errors and improves code-generation pass@1 across datasets.For ChatGPT, the reported pass@1 is 79.3% on HumanEval and 89.5% on MBPP.Source: §5 · p. 8

6. Limitations

The paper identifies limits involving evaluated models, interaction overhead, and the reliability of generated test cases.

6. Limitations

  • Experiments focus mostly on closed-source ChatGPT and GPT-4; repeated LLM interactions add time and API costs, and incorrect tests may lead refinement to introduce errors.The authors also note that open-source models were not evaluated, citing the parameter demands of CoT.Source: §6 · p. 8

A.1. Effectiveness of CodeCoT with GPT-4

The appendix reports improved HumanEval pass@1 with CodeCoT using GPT-4.

A.1. Effectiveness of CodeCoT with GPT-4

  • HumanEval pass@1 increases from 67.0% for GPT-4 to 87.20% with CodeCoT.The accompanying table is described as evaluating CodeCoT with GPT-4.Source: Table 6 · p. 14, §A.1 · p. 14

A.2. Refinement with test case

The section compares code-only refinement with refining both code and test cases during self-examination, reporting gains in test-case accuracy and pass@1.

A.2. Refinement with test case

  • The section contrasts code-snippet-only refinement with refining test cases as well during self-examination.The initial procedure leaves test cases unrefined; the experiments also evaluate refinement of both code and test cases.Source: §A.2 · p. 14, Table 8 · p. 14
  • Pass@1 rises from 79.3% and 89.5% to 82.9% and 90.7% when test cases are refined with code snippets.The comparison is for HumanEval and MBPP, respectively.Source: §A.2 · p. 14, Table 8 · p. 14
  • For 5 steps, pass@1 rises from 69.5% and 63.0% to 74.4% and 70.0% on HumanEval-ET and MBPP-ET.Source: §A.2 · p. 14, Table 8 · p. 14
  • Canonical-solution test-case accuracy rises from 67.1% and 79.0% to 84.1% and 93.4% on HumanEval and MBPP when cases are refined.Source: §A.2 · p. 14, Table 7 · p. 14

A.3. Test Cases’ Coverage Discussion

The section assesses whether CodeCoT-generated tests cover more corner cases by comparing code-line coverage against baseline strategies.

A.3. Test Cases’ Coverage Discussion

  • The comparison asks whether CodeCoT covers more corner cases than strategies that also generate test cases to guide code generation.Source: §A.3 · p. 14
  • CodeCoT obtains 74.7% and 79.3% code line coverage, while CodeT obtains 67.1% and 73.5% on HumanEval and MBPP.The paper interprets the higher coverage as indicating that CodeCoT can cover more corner cases in the canonical solution.Source: §A.3 · pp. 14–15
  • Coverage is calculated in the canonical solution using each tester’s first five test cases for each function.Source: Table 9 · p. 15

A.4. Case Illustration for CodeCoT

Figures 4–9 illustrate CodeCoT’s code-generation, response-preprocessing, execution, and self-examination templates.

A.4. Case Illustration for CodeCoT

Loading 2308.08784v2…