Planning-Driven Programming: A Large Language Model Programming Workflow

1 Introduction

• LLM code generation remains challenging because generated programs must satisfy stringent lexical, grammatical, and semantic constraints.Prior refinement approaches can lack precise correction instructions, and correcting programs that diverge substantially from problem descriptions remains an open challenge.Source: §1 · p. 1
• LPW combines solution planning and verification with initial code development and subsequent refinements in an end-to-end workflow.Apart from runtime information, its messages are generated by LLMs using few-shot prompting, without annotated corpora or additional training.Source: §1 · p. 2
• Plan verification uses visible tests to check a solution plan and clarify the logic specifications needed to solve them.It is reviewed for intermediate-result accuracy and then guides both initial code development and later refinements.Source: §1 · p. 2
• LPW improves Pass@1 over LDB by 2% to 6.1% across benchmarks with GPT-3.5 and up to 16.4% on MBPP with Llama-3.With GPT-4o, it improves over LDB by around 5% on LiveCode, 10% on APPS, and 5% on CodeContests.Source: §1 · pp. 2–3
• LPW’s contributions include deriving an intended solution for visible tests and evaluating the workflow across seven benchmarks and various LLM backbones.The authors also analyze performance, token usage, and failure cases.Source: §1 · p. 3

2 Problem Formulation

• The task is represented as P = ⟨Q, Tv, Th⟩, where Q is the problem description and Tv and Th are sets of visible and hidden input-output tests.The target program must map each test input to its corresponding output across both sets.Source: §2 · p. 3
• Hidden tests remain unavailable during solution generation and code implementation, becoming visible only if the generated program passes the visible tests.Source: §2 · p. 3
• LPW concatenates the problem description with task-specific prompts to elicit each component’s response from the LLM.Source: §2 · p. 3

Solution Generation

• LPW decomposes the problem into a solution plan, derives intermediate and final outputs for visible tests, and reviews those results before treating the verification as the intended solution.The plan and its verification then guide code development and refinements.Source: §3 · p. 3

Code Implementation

• LPW prompts for an initial program using the problem description, plan, and verification, then executes instrumented code on visible tests.If the visible tests pass, it validates the program on hidden tests for Pass@1 accuracy.Source: §3 · p. 4
• When a visible test fails, LPW compares expected intermediate outputs with the execution trace to identify inconsistencies and suggest code refinements.It combines error analysis with a code explanation as feedback, replaces the program, and revalidates it against visible tests.Source: §3 · p. 4

Iterative Updates

• The solution-generation phase revises the plan when its inferred output disagrees with a visible-test result and regenerates verification when intermediate values are incorrect.It reports failure if the maximum iteration limit is reached before verification is correct and free of erroneous intermediate outputs.Source: §3 · p. 4
• Code refinement repeats when bugs remain, while reaching the maximum iteration limit without resolving them results in failure.Source: §3 · p. 4

Benchmarks

• LPW is evaluated on HumanEval, MBPP, HumanEval-ET, and MBPP-ET; the extended variants add approximately 100 hidden edge-case tests per problem, with MBPP-ET sharing MBPP’s problems and visible tests.For HumanEval and HumanEval-ET, task-description tests, typically 2-5 per task, are visible. MBPP has 500 problems with three hidden tests each; the first is treated as visible and the other two as hidden.Source: §4 · pp. 4–5

Experimental Setup

• Comparisons include Self-Planning, MapCoder, Self-Debugging, and LDB; the seed programs for SD and LDB serve as the Baseline.LPW uses 2-shot prompting and up to 12 iterations per phase; SD and LDB also have a 12-iteration maximum, while MapCoder generates three plans with up to four refinement iterations each.Source: §4 · p. 5

Results on various LLMs

• Up to 16.4% improvement over LDB on MBPP with Llama-3 accompanies LPW’s lead over competing methods across benchmarks and evaluated LLM backbones.With GPT-3.5, its Pass@1 improvements over LDB are 6.1%, 4.8%, 3.6%, and 2% on HumanEval, HumanEval-ET, MBPP, and MBPP-ET, respectively.Source: §4 · p. 5, Table 1 · p. 5

Results on Advanced LLM

• 98.2% Pass@1 accuracy on HumanEval is among LPW’s GPT-4o results, and LPW outperforms Baseline and LDB across all evaluated benchmarks.LPW also exceeds LDB by around 10% on APPS and 5% on CodeContests.Source: §4 · p. 6, Table 2 · p. 6

Learning from Test

• On MBPP-ET-3, LPW reaches 62.0% Pass@1 accuracy and improves by 4.4% over MBPP-ET.MBPP-ET-3 has two more visible tests per problem than MBPP-ET; the authors report that LPW’s step-by-step solutions clarify development logic and inform refinements.Source: §4 · p. 6, Table 3 · p. 6

Performance Analysis

• LPW begins HumanEval debugging at 79.9% accuracy, above SP’s 77.4%, and surpasses SD and LDB after one iteration.It reaches its highest accuracy by the 10th iteration; Baseline and SP are no-debugging controls at 74.4% and 77.4%.Source: §4 · pp. 6–7, Figure 4 · p. 6

Ablation Study

• Removing either the solution generation phase or the code implementation phase decreases LPW’s performance, while excluding plan verification also reduces performance.The -V variant uses an unverified plan for initial code and relies on code explanation and runtime information for repairs; -S excludes solution generation, and -C excludes code refinements.Source: §4 · p. 7, Table 4 · p. 7

Accuracy and token cost

• On LiveCode, APPS, and CodeContests, LPW has higher accuracy-cost ratios than LDB: 0.60% vs. 0.50%, 0.43% vs. 0.39%, and 0.18% vs. 0.14% per 1000 tokens, respectively.LDB has better accuracy-cost ratios on HumanEval and MBPP, despite using fewer tokens per problem overall.Source: §5 · p. 7

Accuracy by token consumption

• On APPS and CodeContests, LPW surpasses Repeated Sampling and LDB after around 104.6 and 104.8 tokens, respectively, while repeated sampling has the same token budget as LPW.Repeated Sampling and LDB show negligible accuracy improvement after around 103.8 tokens on APPS and 103.3 tokens on CodeContests.Source: §5 · p. 8

Case study context

• LPW solves the 120th HumanEval problem with GPT-3.5 even though the other approaches fail.The example omits some components and shows incomplete prompts and responses to save space.Source: §6 · p. 8

Output-order ambiguity

• The visible test establishes reverse order after descending sort, resolving ambiguity in the problem description that confused other methods.LPW’s initial plan struggles with the issue, which is addressed in the revised plan during plan verification.Source: §6 · p. 8

Bug identification and refinement

• Comparing the initial code’s execution trace with plan verification identifies that the array was not reversed, and the refined code passes the problem.The final solution sorts descending, then reverses the first k elements into ascending order.Source: §6 · p. 8

• LPW integrates code-generation techniques in two phases, using plan verification to draft initial programs and correct bugs.Source: §7 · p. 8
• With GPT-4o, LPW reports new state-of-the-art Pass@1 of 98.2% on HumanEval, 84.8% on MBPP, 59.3% on LiveCode, 62.6% on APPS, and 34.7% on CodeContests.Source: §7 · p. 8
• The authors identify automatically generated additional visible tests as a future direction for improving LPW.Source: §7 · pp. 8–9

8 Limitations

• LPW remains constrained by imperfect LLM reasoning; its plans and verification also require substantial token consumption.Reducing token use is identified as a critical area for improvement.Source: §8 · p. 9

A.1 Program Synthesis

• LPW uses natural-language intended solutions to avoid the limited generalization of hand-crafted DSLs and the large search spaces of program synthesis.Source: §A.1 · p. 12

A.2 Prompting Techniques

• LPW decomposes problems into subproblems and verifies plans against visible tests with step-by-step analysis, following chain-of-thought prompting.Figure 8 plots Pass@1 against token consumption for LDB, Repeated Sampling, and LPW on LiveCode with GPT-4o.Source: §A.2 · p. 12, Figure 8 · p. 12

A.3 Code Refinement

• Plan verification guides initial code toward the intended solution and helps locate bugs when execution traces diverge from expected intermediate outputs.Source: §A.3 · p. 12

• The experiment used 20 iterations in both the solution generation and code implementation phases.Source: B Parameter Study · p. 12
• 81.1% initial program accuracy with 20 iterations exceeded 79.9% with 12; accuracy later peaked at 89.6% after 18 debugging turns versus a 89.0% maximum with 12 iterations.Figure 9 shows Pass@1 accuracy across iterations on HumanEval using GPT-3.5.Source: B Parameter Study · pp. 12–13, Figure 9 · p. 13
• The authors report that the larger iteration count improves both initial code generation and subsequent refinement, but significantly increases token consumption.Source: B Parameter Study · p. 13

C Additional Ablation Study

• Removing code explanation in LPW-E decreases Pass@1 accuracy on both HumanEval and MBPP with GPT-3.5.Table 5 reports Pass@1 accuracy for LPW variants using GPT-3.5.Source: C Additional Ablation Study · p. 13

D.1 Performance Across Different Difficulty Levels

• 85.7% Pass@1 on Hard HumanEval problems for LPW compares with competing approaches below 70%; LPW exceeds 85% across all difficulty levels.Figure 10 compares methods on Easy, Medium, and Hard problems.Source: §D.1 · p. 13, Figure 10 · p. 13

D.2 Failure Reasons

• Half of LPW’s HumanEval failures are classified as No Code, while 5.6% involve code that differs from the intended solution.No Code includes plans or verifications that fail before the maximum iteration threshold; the latter category covers code deviating from a manually judged-correct plan and verification.Source: §D.2 · pp. 13–14, Table 6 · p. 14

D.3 Failure on Hidden Tests

• On HumanEval, MapCoder is more likely than the other evaluated methods to pass visible tests but fail hidden tests; on MBPP, their percentages are similar.For the share of all problems, MapCoder has the highest such rate on MBPP and LPW the lowest; all methods show this issue more often on MBPP than HumanEval.Source: §D.3 · p. 14, Table 7 · p. 14, Table 8 · p. 14

E.1 Plans and Plan Verifications

• GPT-3.5 produces valid plans and plan verifications for 94.5% of HumanEval problems, with correct plans and verifications each reported at 92.7%.The accompanying table reports valid generation, manually assessed plan correctness, and manually assessed verification correctness across 164 problems.Source: §E.1 · p. 15, Table 9 · p. 15
• Plan verifications closely follow the plan: omitted conditions, including uppercase cases and negative numbers, can therefore appear in both.The authors report that correct plans lead to accurate verifications, and vice versa.Source: §E.1 · p. 15

E.2 Plan Verifications and Programs

• In LPW, inaccurate plan verifications are associated with erroneous code, as descriptions and missed conditions in the plan can propagate into the generated program.Table 10 evaluates conditional probabilities between incorrect code and incorrect verifications, and correct code and correct verifications.Source: §E.2 · p. 15
• 96.1% of LPW program solutions are correct when their plan verifications are accurate.The remaining incorrect solutions arise from unclear conditions in the verification that do not account for hidden tests.Source: §E.2 · p. 15
• Plan and verification generation accuracy is typically higher than code-generation accuracy, motivating the authors to explore alternative solution representations alongside natural language.The paper identifies lexical, grammatical, and semantic constraints as challenges in translating natural-language solutions into programs.Source: §E.2 · p. 15

F Refinement Consistency in LPW

• LPW compares a failed test’s execution trace with its plan verification to produce error analysis, then uses that analysis to guide code refinement.Refinement explanations describe the changes made to address the identified errors.Source: F Refinement Consistency in LPW · pp. 15–16
• The accompanying LiveCode figure compares Pass@1 accuracy across Easy, Medium, and Hard difficulty levels for Baseline, LDB, and LPW with GPT-4o.Tables 11 and 12 describe visible-test passes that fail hidden tests, reported respectively as a share of failures and of all 140 LiveCode problems.Source: Figure 12 · p. 16, Table 11 · p. 16, Table 12 · p. 16
• Across problems solved through debugging, only one refined code deviates from the error analysis, and that code still produces a correct solution.The authors report that this consistency supports the debugging steps’ effectiveness in producing correct programs through precise refinements.Source: F Refinement Consistency in LPW · p. 16

G.1 HumanEval

• 98.2% Pass@1 on HumanEval leaves 3 unsolved problems; LPW resolves 2 of them and reaches 99.4% by adding a visible test or clarifying the problem description.The unresolved cases are HumanEval problems 91, 132, and 145.Source: §G.1 · p. 16

G.1.1 The 91st Problem

• On HumanEval problem 91, GPT-4o confuses a sentence starting with the word “I” with one starting with the letter “I”; making a failed hidden test visible enables a correct program.The original visible tests did not expose the error, allowing the incorrect program to pass them.Source: §G.1.1 · pp. 16–17, Figure 15 · p. 20

G.1.2 The 145th Problem

• Clarifying that HumanEval problem 145 requires sorting by digit sum with explicit handling of negative numbers fixes LPW’s incorrect plan verification and yields a correct program.The original description omitted how to handle negative-number signs.Source: §G.1.2 · p. 17, Figure 17 · p. 22

G.2.1 Performance Across Different Difficulty Levels

• LPW leads on every LiveCode difficulty level: over 5% above LDB at each level, over 15% above Baseline at Easy and Medium, and 7.5% at Hard.The section also reports that LPW exceeds Baseline and LDB by approximately 15% and 5%, respectively, across difficulty levels.Source: §G.2.1 · p. 17, §G.2 · p. 17

G.2.2 Failure on Hidden Tests

• 31.6% of LiveCode LPW failures pass visible but fail hidden tests, versus 23.4% for LDB; this occurs in 12.9% of problems for LPW and 10.7% for LDB.The first percentages are shares of failed problems; the latter are shares of all problems.Source: §G.2.2 · pp. 17–18, Table 14 · p. 17

G.3 APPS and CodeContests

• 62.6% Pass@1 on APPS and 34.7% on CodeContests exceed LDB by around 10% and 5%, respectively, but underscore the challenge of complex programming problems.Because these benchmarks lack structured function signatures, GPT-4o was instructed to derive signatures and identify visible tests before experiments.Source: §G.3 · p. 18

G.3.1 Performance Across Different Difficulty Levels

• 34.8% Pass@1 on APPS Competition problems exceeds LDB’s 28.3% and Baseline’s 17.4%, yet all approaches decline substantially at this most challenging level.On Introductory and Interview problems, LPW exceeds LDB by around 9% and 13%, respectively, and Baseline by over 20%.Source: Figure 13 · p. 17, §G.3.1 · p. 18

G.3.2 Failure on Hidden Tests

• 23.1% of APPS failures pass only visible tests, while the share is around 30% on CodeContests for both LDB and LPW.Across all problems, visible-only solutions account for around 10% on APPS and 19% on CodeContests; each averages about 2 visible tests, while hidden tests average about 5 on APPS and 23 on CodeContests.Source: Table 13 · p. 17, Table 14 · p. 17, §G.3.2 · p. 18

G.4 Prompts for LPW

• The addOne([-1,2]) example falsely validates [-1,1] even though its reasoning treats the input as positive 12 and derives [1,3].The verification incorrectly claims the derived result matches the expected output.Source: §G.4 · p. 27
• Plan-verification prompts track intermediate variables, compare derived results with expected test outputs, and label each verification correct or incorrect after checking its steps.The prompt specifies [Record Analysis] and [Record] tags for identifying and updating tracked values.Source: §G.4 · p. 27
• Code-generation prompts require programs to follow the solution plan and remain consistent with its test-case verifications.The prompt also requires incorporating conditions identified in those verifications.Source: §G.4 · p. 29
• A tracing prompt adds print statements for variables tracked by the plan verification, with examples showing how to log intermediate values as execution proceeds.The prompt asks that values identified for tracking in the verification be printed.Source: §G.4 · pp. 29–30
• An explanation prompt asks for word-by-word accounts of program lines, and its encode example describes incrementing characters and skipping successors that are vowels.The example explanation says non-vowel successors are appended to the encoded message.Source: §G.4 · pp. 30–31
• LPW’s discrepancy prompt compares an error execution trace with verified reasoning that records intermediate updates, then diagnoses deviations and suggests fixes.The verification explicitly records the checking result, while the prompt asks for discrepancy and error analyses before code repair.Source: §G.4 · pp. 32–33
• In the palindrome example, the error trace drops the minus sign from -121, returns True, and is diagnosed against verification that preserves the sign and returns False.The proposed fix is to retain the negative sign when converting the integer to a string.Source: §G.4 · pp. 32–33
• The repair prompt supplies the error program, its explanation and analysis, then requests refined Python code and an explanation of the changes.The example changes str(abs(num)) to str(num) so negative integers retain their sign.Source: §G.4 · pp. 34–35