Source-linked AI summary
Self-planning Code Generation with Large Language Models
Xue Jiang, Yihong Dong, Lecheng Wang, Zheng Fang, Qiwei Shang, Ge Li, Zhi Jin, Wenpin Jiao
TL;DR
LLMs struggle with complicated human intents that require decomposition and planning. The paper proposes self-planning code generation, which first generates a plan from few-shot demonstrations and then produces code guided by that plan. Across multiple code-generation datasets and programming languages, self-planning outperforms direct generation and improves assessed code quality.
Problem
LLMs have difficulty generating code for complicated human intents, while plan-aided generation ordinarily requires costly intent-plan annotation.
Method
Self-planning uses few-shot prompting to generate a plan, then generates code step by step guided by the intent and plan.
Results
Self-planning outperforms direct generation on multiple datasets and improves correctness, readability, and robustness in human evaluation.
Takeaways & Limitations
Planning strategies can benefit most LLMs, although self-planning itself emerges only on sufficiently large LLMs.
Takeaways & Limitations
Prompt construction, including example selection and plan writing, affects the achievable improvement and remains open to optimization.
Abstract
from arXiv · showhide
Although large language models (LLMs) have demonstrated impressive ability in code generation, they are still struggling to address the complicated intent provided by humans. It is widely acknowledged that humans typically employ planning to decompose complex problems and schedule solution steps prior to implementation. To this end, we introduce planning into code generation to help the model understand complex intent and reduce the difficulty of problem-solving. This paper proposes a self-planning code generation approach with large language models, which consists of two phases, namely planning phase and implementation phase. Specifically, in the planning phase, LLM outlines concise and formatted planning steps from the intent. Subsequently, in the implementation phase, the model generates code step by step, guided by the preceding planning steps. We conduct extensive experiments on various code-generation benchmarks across multiple programming languages. Experimental results show that self-planning code generation achieves a relative improvement of up to 25.4% in Pass@1 compared to direct code generation, and up to 11.9% compared to Chain-of-Thought code generation. Moreover, our self-planning approach also enhances the quality of the generated code with respect to correctness, readability, and robustness, as assessed by humans.
1 INTRODUCTION
LLMs handle straightforward code-generation intents but struggle with complicated requirements that humans typically decompose and plan before implementation. The paper introduces self-planning code generation, which uses few-shot planning followed by guided implementation and improves Pass@1 over direct generation and Code CoT.
- Motivation: Complex intents are difficult for LLMs to translate directly into code, motivating explicit planning before implementation.The paper frames intent as “what to do” and code as “how to do.”
- Motivation: Few-shot prompting offers a way to generate plans without training a separate intent-to-plan model or labeling a large plan corpus.The paper identifies CoT as a relevant prompting technique but notes challenges when applying it to code generation.
- Approach: Self-planning uses a planning phase to generate plans from a few intent-to-plan demonstrations, then an implementation phase to generate code guided by those plans.This approach avoids extra training and plan-corpus annotation.
- Results: 25.4% relative improvement in Pass@1 over direct generation and 11.9% over Code CoT are reported for self-planning code generation.These are maximum relative improvements reported in the paper’s empirical evaluations.
- Evaluation: The study also evaluates self-planning across Python, Java, Go, and JavaScript and assesses correctness, readability, and robustness through human evaluation.The supplied introduction reports multilingual evaluation and human assessment of code quality.
2 SELF-PLANNING
Self-planning decomposes an intent into scheduled, concise subproblems and uses the resulting plan to guide step-by-step code generation. In a HumanEval example, this decomposition preserves a combined Fibonacci-and-primality requirement that direct generation mishandles while improving readability.
- Overview: Self-planning performs planning before code generation and divides inference into planning and implementation phases.The plan is generated first and then used to guide final code generation.
- Planning phase: In the planning phase, an LLM abstracts and decomposes the intent into a plan using only a few labeled examples in a prompt.The test intent is appended to the prompt, and the LLM generates a test-time plan.
- Planning phase: Plans schedule decomposed subproblems as concise, numbered, high-level imperative steps that are individually easy to implement.The prompt-writing principles emphasize one implementable sub-task per step and avoid excessive implementation detail.
- Implementation phase: In the implementation phase, the plan is appended to the intent, and the LLM generates final code by predicting the next token.The plan functions as navigation for code generation rather than as separately trained supervision.
- Example: For a combined Fibonacci-and-primality task, planning produced subproblems and four steps, while direct generation lost the primality requirement.The planned implementation also wrapped primality checking in a subfunction and improved readability.
- Rationale: The approach is presented as necessary for handling the complexity of code-generation tasks as those tasks become more complex.The paper connects this strategy to abstraction and decomposition of human intent.
3 EVALUATION
The evaluation compares self-planning with multiple code-generation baselines on monolingual, multilingual, and expanded-test benchmarks. It measures execution-based correctness and reference-code similarity, while also validating generated ground-truth plans.
- Research questions: The evaluation addresses five questions covering baseline performance, model variation, design choices, multilingual generation, and problem complexity.These are stated as RQ1 through RQ5.
- Benchmarks: The study uses MBPP, HumanEval, HumanEval-X, MBPP-ET, and HumanEval-ET to evaluate code generation across languages and test-set extensions.HumanEval-X supplies multilingual JavaScript, Go, and other-language samples, while the ET versions add edge cases.
- Metrics: Pass@k and AvgPassRatio measure functional correctness by executing tests, whereas CodeBLEU measures similarity to reference code.The evaluation therefore combines execution-based and match-based metrics.
- Baselines: Direct, Code CoT, and Ground-truth Planning serve as basic baselines across experiments.Direct uses only the intent, Code CoT generates intermediate reasoning, and Ground-truth Planning skips planning by supplying plans directly.
- Plan validation: Ground-truth plans were generated from existing code using few-shot prompting and manually validated on HumanEval.Most plans satisfied the requirements, with about 3% containing poorly described steps.
- Implementation details: Experiments use code-davinci-002 for basic baselines, greedy decoding for single outputs, and sampling configurations for Pass@k where k≥2.The supplied configuration passage specifies generation lengths, temperatures, and top-p settings.
4 EXPERIMENTAL RESULTS
Across code-generation benchmarks, self-planning generally outperforms direct generation and Code CoT, with benefits spanning model sizes, planning variants, programming languages, and problem difficulties.
- Comparison With Baselines: Ground-truth planning improves relative performance by over 50% on HumanEval and over 30% on MBPP-sanitized.The authors describe this as an approximate upper bound for self-planning and report a larger improvement on HumanEval.
- Performance on Different LLMs: Self-planning is weaker in small models, emerges around 13B parameters, and consistently beats Direct at 175B.Code training and RLHF also enhance self-planning ability; code-davinci-002 and text-davinci-003 show stronger planning than their respective comparisons.
- Variants of Self-planning: Among design variants, multi-turn usually fails to generate correct code, while the proposed two-phase method is reported as the optimal overall choice.Multi-turn generation faces truncation difficulties, whereas one-stage and extremely concise-plan variants have practical or readability drawbacks.
- Variants of Self-planning: Increasing few-shot examples improves self-planning until input-length limits and performance saturation constrain further increases.The authors generally recommend 8-shot or [the supplied passage ends before the second recommendation].
- Multilingual and Complexity Results: Self-planning uses fewer tokens than other few-shot approaches, and its performance remains positive across Python, Java, Go, and JavaScript.It also performs especially well on higher-difficulty problems, with the largest gains reported for Python and complex tasks.
5 HUMAN EVALUATION
Human evaluation finds that self-planning improves generated-code quality, especially readability, while correctness and robustness approach ground-truth planning. Qualitative cases suggest planning helps address more of a complex intent than direct generation or Code CoT.
- Evaluation Design: Human evaluation scores generated code on correctness, readability, and robustness using five-code comparisons across 50 HumanEval tasks.Ten developers with 2–5 years of Python experience scored codes from 0 to 4.
- Results: Self-planning outperforms Direct and Code CoT in correctness but remains below Ground-truth planning and Ground-truth.Correctness scoring also considers partial correctness and closely aligns with Pass@1.
- Results: Self-planning achieves the highest readability, while Ground-truth planning is close behind and Code CoT is hindered by excessive, potentially outdated detail.The authors connect self-planning’s readability to accuracy and a coherent outline of planned steps.
- Results: Self-planning surpasses Code CoT in robustness and performs comparably to Ground-truth planning by considering edge cases and input legality.Incorrect code usually receives a robustness score of 0, producing a broader lower end in the robustness distributions.
- Conclusion: Across human evaluation, self-planning has the best readability, with correctness and robustness on par with ground-truth planning.The conclusion summarizes the three human-assessed quality dimensions.
- Qualitative Examples: In qualitative cases, direct and Code CoT generation address limited aspects of complex intents, whereas self-planning systematically resolves the planned solution steps.Examples involve finding a minimum sub-array sum and implementing a special factorial rather than standard factorial.
- Qualitative Examples: Code CoT and direct generation can produce nearly identical semantics, suggesting that generating Code CoT from intent is comparably difficult to generating code.The comparison concerns expression form versus underlying code semantics.
6 THREATS TO VALIDITY
The study identifies prompt construction and result generalizability as its two primary threats to validity. Prompt example selection and plan writing can affect improvement, while the evaluation spans seven datasets and four programming languages.
- Threats to Validity: Prompt sensitivity makes example selection and plan writing a primary threat because they can affect the improvement achieved by self-planning.The authors note that automated example selection and prompt-generation methods could optimize performance.
- Threats to Validity: Generalizability is a second threat, addressed by evaluating self-planning on seven public benchmark datasets spanning Python, Java, Javascript, and Go.Pass@k is used as the accepted metric for validating generated-code quality.
7 RELATED WORK
Related work spans supervised and pre-trained code-generation models, few-shot reasoning and code-producing prompts, self-improving LLM methods, and LLM-based planning for complex tasks.
- Code Generation Models: Traditional code generation uses supervised learning and progressively incorporates code-specific features such as abstract syntax trees and API calls.Other work uses static analysis or PDA-based methods to address source-code generation and grammatical correctness.
- Code Generation Models: Pre-trained models including CodeT5, UniXcoder, Codex, InCoder, CodeGen, AlphaCode, and CodeGeeX advanced code generation alongside increasing model scale.The passage identifies parameter growth as a trend associated with improved code-generation performance.
- Prompting Techniques: Few-shot prompting provides task examples without fine-tuning, while CoT generates intermediate reasoning steps before an answer.Related techniques include least-to-most prompting, PAL, and PoT, which use ordered subproblems or executable code as intermediate reasoning.
- Self-Improving LLMs: Self-improving LLM methods use generated data, self-feedback, or self-evaluation to improve reasoning and instruction-following capabilities.Examples include Self-Instruct, Self-Refine, and Self-Evaluation.
- LLM-Based Planning: LLM-based planning has been applied to embodied-agent planning and to coordinating AI-model invocation for complex tasks.LLM-Planner and HuggingGPT illustrate planning across different application settings.
8 DISCUSSION AND FUTURE WORK
The discussion frames prompt crafting and data efficiency as practical considerations for self-planning code generation. It also connects planning with abstraction and decomposition for handling complex software requirements.
- Manual prompt crafting is identified as a major limitation of self-planning code generation.
- Unlike approaches requiring many training examples, self-planning teaches planning with only a few examples that people without programming knowledge can craft.
- The approach represents plans as approximate sequential executed lists resembling functional points in requirements documents.
- Planning reduces code-generation difficulty by applying abstraction and decomposition to complex human intent, although fully functional software remains challenging.
9 CONCLUSION
The conclusion presents self-planning as a simple, effective plan-aided approach for code generation. It reports improvements over direct generation and in human-rated code quality, while noting benefits across most models despite planning being emergent.
- Self-planning code generation is presented as a simple but effective approach that performs self-planning before generating code with LLMs.
- Self-planning outperforms direct LLM generation by a large margin across multiple code-generation datasets.
- Human evaluation indicates that self-planning improves the correctness, readability, and robustness of generated code.
- Although self-planning is emergent, incorporating planning strategies benefits most models.
C DETAILS OF METRICS
The metrics section defines execution-based and code-feature-based evaluation measures used for generated programs. It also distinguishes Code CoT from vanilla CoT as an adaptation for code generation.
- Pass@k uses an unbiased estimator based on n generated samples, c correct samples, and test-case passage results.
- AvgPassRatio measures the average proportion of test cases passed by generated code.
- Code CoT is implemented as an early code-generation adaptation in which the LLM generates a complete reasoning trace from intent before generating code.
- CodeBLEU extends BLEU by incorporating abstract-syntax-tree and dataflow matching alongside n-gram matching.
D EXPERIMENTAL VERIFICATION OF SUBPROBLEM DECOMPOSITION IN SELF-PLANNING
The experiments examine whether self-planning decomposes coding problems into sub-functions. Across examples, self-planning produces more sub-functions than direct generation and Code CoT, supporting its use for breaking problems into sub-problems.
- Self-planning generated 37 sub-functions on HumanEval, compared with 1 for Direct and 9 for Code CoT.
- For selected multi-objective problems, Direct, Code CoT, and Self-planning generated 11, 18, and 40 sub-functions, respectively.
- The experiments indicate that self-planning helps break problems down into sub-problems, facilitating code generation.
- The examples pair programming intents with numbered plans covering tasks such as parsing inputs, computing results, validating conditions, and returning outputs.
H HUMAN EVALUATION QUESTIONNAIRE
The human-evaluation questionnaires for a task are presented in Fig. 7 and Fig. 8. The study distributed 50 questionnaires corresponding to 50 tasks.
- 50 questionnaires were distributed for the human evaluation.These questionnaires corresponded to 50 tasks.
- Fig. 7 presents Part 1 of the questionnaire.
- Fig. 8 presents Part 2 of the questionnaire.