PaperlayerView exact revision on arXiv

Source-linked AI summary

Unlocking Reasoning Potential in Large Langauge Models by Scaling Code-form Planning

Jiaxin Wen, Jian Guan, Hongning Wang, Wei Wu, Minlie Huang

arXiv:2409.12452v2cs.CL

TL;DR

LLMs' limited planning ability is a bottleneck for complex multi-step reasoning, and planning signals in text corpora are often obscured. CodePlan uses automatically extracted code-form plans to guide reasoning, reporting a 25.1% relative gain over direct response generation across 13 benchmarks, with increasing gains on more complex problems.

  • Problem

    LLMs' limited planning ability is a bottleneck for complex multi-step reasoning, while pre-training corpora often obscure the high-level planning signals models should learn.

    Source: §1 · p. 1

  • Method

    CodePlan trains LLMs to generate and follow code-form plans, pseudocode blueprints automatically extracted from broad text corpora without curated task-specific datasets.

    Source: §1 · p. 2

  • Results

    A 25.1% relative performance gain over direct response generation is reported on average across 13 benchmarks, with gains increasing on more complex problems.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Code-form planning is presented as a scalable approach to structured reasoning across tasks, with analyses indicating generalization ability and data efficiency.

    Source: §1 · p. 2, §5 · p. 11

    The explicit-plan approach assumes a gold annotator supplies a high-quality plan for each prompt-response pair; implicit plans are left for future work.

    Source: §2.1 · p. 4

Abstract

from arXiv · show

Despite the remarkable success of large language models (LLMs) on traditional natural language processing tasks, their planning ability remains a critical bottleneck in tackling complex multi-step reasoning tasks. Existing approaches mainly rely on prompting or task-specific fine-tuning, often suffering from poor robustness and cross-task generalization. To address the limitation, we introduce CodePlan, a scalable framework that empowers LLMs to generate and follow \textit{code-form plans} -- pseudocode that outlines high-level, structured reasoning processes. By leveraging the structured and versatile nature of code, CodePlan effectively captures the rich semantics and control flows inherent to sophisticated reasoning tasks. Importantly, CodePlan allows automatic extraction of code-form plans from massive, wide-ranging text corpora without the need for curated, task-specific datasets. This enables it to scale up efficiently and improve LLM's reasoning capabilities across diverse scenarios. To train CodePlan, we construct a large-scale dataset of 2M examples that integrate code-form plans with standard prompt-response pairs from existing corpora. With minimal computation overhead during both training and inference, CodePlan achieves a 25.1\% relative improvement compared with directly generating responses, averaged across 13 challenging multi-step reasoning benchmarks, spanning mathematical reasoning, symbolic reasoning, instruction-following, multi-hop QA, and decision-making tasks. Further analysis reveals CodePlan's increasing performance gains on more complex reasoning tasks, as well as significant data efficiency thanks to its generalization ability.

1 INTRODUCTION

CodePlan addresses the planning bottleneck in multi-step reasoning with code-form plans that can be extracted automatically from broad text corpora. Across challenging reasoning benchmarks, it reports gains over direct response generation and advantages that increase with problem complexity.

1 INTRODUCTION

  • LLMs' limited planning ability is described as a bottleneck for complex multi-step tasks, while prompting and task-specific fine-tuning have robustness or generalization constraints.The introduction notes failure modes including repetitive steps, incoherent logic, focus drift, and early answering.Source: §1 · pp. 1–2
  • CodePlan uses code-form plans—pseudocode blueprints for structured reasoning—and automatically extracts them from broad corpora without curated task-specific datasets.The framework uses code's control flows and structure to represent reasoning processes across diverse tasks.Source: §1 · p. 2
  • A 25.1% relative performance gain over direct response generation is reported on average across 13 challenging reasoning benchmarks.The benchmarks span mathematical, symbolic, instruction-following, multi-hop question-answering, and decision-making tasks.Source: §1 · p. 2
  • The authors construct a dataset of 2M prompt-response pairs with corresponding code-form plans to train the framework.They describe the dataset as enabling scalable training data construction from wide-ranging corpora.Source: §1 · p. 2
  • CodePlan is evaluated across multiple backbone models from 7B to 13B, with advantages that grow as problem complexity increases and strong data efficiency.The introduction describes the framework's efficacy and generality across 13 challenging reasoning benchmarks.Source: §1 · p. 2

2 METHODOLOGY

The method separates generation into planning and surface realization: a model first produces a structured plan, then uses it to guide the final response. Code-form pseudocode represents the plan, and the approach trains on prompt-plan-response examples.

2 METHODOLOGY

  • Generation is decomposed into planning, which outlines control flow for a prompt, and surface realization, which expands the plan into a natural-language response.The method models the objective through the plan and the response conditioned on that plan.Source: §2 · pp. 2–3, §2.1 · p. 3

2.1 FORMALIZATION

  • The formulation separates planning from realization, but marginalizing over plans is generally intractable, motivating a variational upper bound.The authors use an explicit plan from a gold annotator and reserve implicit plans for future work.Source: §2.1 · pp. 3–4

2.2 PLANNING

  • Code-form plans use pseudocode to express reasoning structure, including conditional branches, iterative loops, modular functions, and hierarchical subtasks.Plans are synthesized from prompt-response pairs and need not be executable; the model is trained to generate the plan.Source: §2.2 · pp. 4–5
  • The training data pairs prompts with code-form plans and responses, and the plan-generation instruction is provided in Table 2.The planning stage optimizes plan generation; generated pseudocode is not required to execute.Source: §2.2 · pp. 4–5, Table 2 · p. 5

2.3 SURFACE REALIZATION

  • Surface realization generates a response conditioned on the prompt and plan, with branches and loops guiding the response's reasoning path.The model is optimized to generate the response given the prompt and code-form plan.Source: §2.3 · p. 5

3 EXPERIMENTS

The experiments evaluate CODEPLAN against prompting, implicit-planning, and vanilla-training baselines across multi-step reasoning tasks. CODEPLAN improves over direct response generation, with further analyses examining task complexity, data efficiency, plan representations, and executable-code reasoning.

3.1 TRAINING DATA CURATION

  • The training data pairs prompts and responses from WebInstruct with code-form plans synthesized by Llama-3-8B-Instruct, enabling construction from broad existing corpora.The paper reports a dataset of 2M examples and describes the curation process as readily extensible to other corpora.Source: §3.1 · p. 5

3.2 BASELINES / 3.3 EXPERIMENTAL SETUP

  • The comparisons include Plan-and-Solve prompting, Quiet-STaR implicit plans, and vanilla training on WebInstruct followed by direct response generation.Experiments use Mistral-7B and Llama-2-7B/13B; unless stated otherwise, evaluations are few-shot and do not fine-tune on evaluation benchmarks.Source: §3.2 · p. 5, §3.3 · p. 6

3.4 EVALUATION BENCHMARKS

  • Evaluations cover mathematical, symbolic, instruction-following, multi-hop QA, and decision-making benchmarks, with gold reference passages supplied for multi-hop QA.Metrics include accuracy, EM and F1, win rate, and GPT-4 scores; Dyck Language receives additional fine-tuning to mitigate long-generation degeneration.Source: §3.4 · p. 6

3.5 MAIN RESULTS

  • CODEPLAN yields a relative 25.1% performance gain over vanilla training on average across five types of reasoning tasks.The reported comparisons cover all backbone models and most benchmarks.Source: §3.5 · pp. 6, 8, Table 3 · p. 7

3.6 ANALYSIS

  • On MuSiQue, CODEPLAN’s relative EM improvement rises from 6.3% on 2-hop questions to 43.8% on 4-hop questions.MuSiQue includes questions with varying reasoning complexity; Figure 2 presents EM and F1 by hop count.Source: §3.6 · p. 8, Figure 2 · p. 8
  • With the same raw prompt-response data, CODEPLAN almost always outperforms vanilla training and maintains a more stable performance ascent.The paper interprets this as evidence of better knowledge acquisition and generalization to out-of-distribution reasoning challenges.Source: §3.6 · p. 8, Figure 3 · p. 8
  • CODEPLAN consistently outperforms natural-language plans across benchmarks, with relative gains of 4% in mathematical reasoning, 27.2% in symbolic reasoning, 6.5% in instruction-following, and 27.5% in decision-making.Natural-language planning also moderately improves over the vanilla baseline; Figure 4 averages scores across benchmarks within each task type.Source: §3.6 · p. 9, Figure 4 · p. 9
  • CODEREASON performs best on SVAMP but underperforms CODEPLAN on MATH, MuSiQue, and HotpotQA, and even trails vanilla training on MATH and MuSiQue.CODEREASON replaces pseudocode plans with executable code solutions; CODEPLAN combines intermediate planning with the model’s language understanding.Source: §3.6 · pp. 9–10, Figure 5 · p. 9

4 RELATED WORK

Related work groups LLM reasoning approaches into prompting, task-specific fine-tuning, and tool integration, while noting limits to generalization or domain scope. CODEPLAN instead uses code as a plan representation for planning across domains.

LLMs for Reasoning

  • Prior reasoning work includes prompting, task-specific fine-tuning, and tool integration, while CODEPLAN targets high-level planning across diverse domains without task-specific prompts, fine-tuning data, rewards, or tools.The passage notes that prompting and task-specific fine-tuning often generalize poorly beyond intended tasks.Source: §4 · p. 10

Deliberate Planning in LLMs

  • Earlier deliberate-planning approaches use natural-language plans, while implicit-planning methods may struggle to uncover effective signals and incur training overhead from online sampling.The authors describe multi-path planning and iterative plan refinement as orthogonal directions for future integration.Source: §4 · p. 10

Code-aided Reasoning

  • Code-aided reasoning often generates executable code for an interpreter, a strategy described as limited in data scalability and narrow in domain, whereas CODEPLAN represents reasoning structures with code.Prior uses of code also include graph generation, event structure prediction, and decision-making.Source: §4 · p. 10

5 CONCLUSION

CODEPLAN uses explicit code-form plans to structure LLM reasoning and can extract them automatically from broad text corpora. Trained on 2M examples, it improves performance across 13 reasoning benchmarks, with gains increasing on complex problems.

5 CONCLUSION

  • CODEPLAN extracts code-form plans automatically from wide-ranging text corpora, without requiring curated task-specific datasets.It trains LLMs to generate and follow pseudocode plans that structure reasoning.Source: §5 · p. 10
  • Across 13 challenging multi-step reasoning benchmarks, CODEPLAN shows increasing performance gains on complex problems and generalization ability.The training set pairs natural-language problems with their corresponding code-form plans in 2M examples.Source: §5 · p. 11
  • The paper identifies plan distributions, multi-path planning, verification, reflection, refinement, and external knowledge or APIs as directions for future work.Source: §5 · p. 11

A.1 TRAINING DATA CURATION DETAILS

Training-data curation filters out plans without valid terminations to preserve logical coherence. More complex filtering based on plan information gain was tested but omitted after yielding only marginal improvements.

A.1 TRAINING DATA CURATION DETAILS

  • Plans without valid terminations are filtered because valid endings are essential for logical coherence.Source: §A.1 · p. 16
  • Figure 6 depicts the training-data curation pipeline.Source: Figure 6 · p. 16, §A.1 · p. 16
  • Information-gain filtering compared response perplexity with and without a plan, but was omitted after pilot tests found only marginal improvements.The authors cite efficiency and avoiding potential overfitting as reasons for omitting these more complex filters.Source: §A.1 · p. 16

A.2 DATA STATISTICS

The appendix reports training-data and benchmark statistics and describes prompts for annotating natural-language plans and executable reasoning code.

A.2 DATA STATISTICS

  • Evaluation covers math, symbolic reasoning, instruction-following, multi-hop QA, and decision-making, with specified benchmark subsets and sampling choices.The setup includes 200 MuSiQue questions sampled from its 2/3/4-hop subsets and 500 randomly sampled HotpotQA questions.Source: §A.2 · p. 16
  • Table 5 reports curated training-data statistics, while Table 6 reports the number of test examples in evaluation benchmarks.Source: Table 5 · p. 16, Table 6 · p. 16

A.3 PROMPTS

B.1 ASSESSING THE ROBUSTNESS OF CODEPLAN TO PLAN QUALITY DRIFT

Annotation-model comparisons show a substantial performance gap between Gemma-2B-Instruct and Llama-3-8B-Instruct, but a relatively small gap between Llama-3-8B-Instruct and Llama-3-70B-Instruct. The authors report that CODEPLAN remains effective with imperfect annotations, including under domain shift or annotation noise.

B.1 ASSESSING THE ROBUSTNESS OF CODEPLAN TO PLAN QUALITY DRIFT

  • Llama-3-8B-Instruct was the initial plan annotator; Gemma-2B-Instruct and Llama-3-70B-Instruct were tested to assess the impact of label quality.Source: §B.1 · p. 17
  • The performance gap between Gemma-2B-Instruct and Llama-3-8B-Instruct is substantial, while the gap between Llama-3-8B-Instruct and Llama-3-70B-Instruct is relatively small across reasoning tasks.Table 9 reports scores averaged across benchmarks for three representative reasoning-task types.Source: §B.1 · p. 17, Table 9 · p. 17
  • Once annotation reaches sufficient competence, CODEPLAN is largely insensitive to further plan-quality improvements and maintains effectiveness despite domain shift or annotation noise.The authors say this reduces the need to use the most computationally expensive annotation models for data curation and training.Source: §B.1 · pp. 17–18

B.2 EFFICIENCY ANALYSIS

CODEPLAN retains computational efficiency close to vanilla training, with only a marginal increase in inference time and negligible additional inference memory.

B.2 EFFICIENCY ANALYSIS

  • Memory usage and average per-example inference time were evaluated at batch size 1 on one A100 GPU, averaging time across 1000 generations.Memory consumption was calculated with PyTorch toolkits.Source: §B.2 · p. 18
  • ∼5% higher average inference time than vanilla training accompanies only a negligible increase in inference memory.The comparison evaluates CODEPLAN against the vanilla training baseline.Source: §B.2 · p. 18
  • The authors report that pseudocode concisely outlines reasoning structures, allowing substantial performance improvements without much computation overhead.The efficiency discussion characterizes the overhead as small relative to the performance improvements.Source: §B.2 · p. 18

B.3 CASE STUDY

The case studies illustrate code-form plans across reasoning and decision tasks, showing how they structure multi-step operations and support reusable functions across examples.

B.3 CASE STUDY

  • For mathematical reasoning, CODEPLAN enumerates candidate ranges and applies constraints through logical operations, whereas the vanilla model misses conditions and answers incorrectly.The example requires finding a number satisfying multiple constraints.Source: §B.3 · p. 18
  • In symbolic reasoning, the plan iterates through words, extracts each last letter, and appends it to the result, avoiding the vanilla model’s concatenation-order error.The example uses the Last Letter Concatenation benchmark.Source: §B.3 · p. 19, Table 12 · p. 19
  • In multi-hop QA, the plan identifies entities, queries relevant facts, and combines them to answer a question that the vanilla model answers incorrectly.The example asks for the date associated with the Treaty of Waitangi after connecting a film backdrop to New Zealand.Source: §B.3 · p. 19, Table 13 · p. 19
  • For simulated decision-making, the plan specifies locating and obtaining an object, cooling it, and placing it at the target location, unlike the vanilla model’s incoherent action sequence.The example is an ALFWorld task to cool a mug and put it in a coffeemachine.Source: §B.3 · pp. 19–20, Table 14 · p. 20
  • Across the examples, code-form plans represent reasoning with explicit logic, control flow, and hierarchical task decompositions.The case study presents examples from several task types to illustrate these structured representations.Source: §B.3 · pp. 18–19
  • Reusable code blocks abstract common operations across instances, which the authors identify as a source of CODEPLAN’s higher data efficiency.The examples include mathematical operations reusable across multiple questions.Source: §B.3 · pp. 20–21, Table 15 · p. 21
  • For value comparison and letter counting, plans explicitly parse number parts or iterate through letters and increment counts, clarifying the required operations.The examples include comparing 9.8 with 9.11 and counting “r”s in “strawberry.”Source: §B.3 · p. 21

B.4 ERROR ANALYSIS

The error analysis distinguishes failures in executing subtasks, following plans, and generating plans, identifying remaining weaknesses at both planning and execution stages.

B.4 ERROR ANALYSIS

  • A correct high-level plan can still yield a wrong answer when individual subtasks are executed inaccurately, such as through an arithmetic error.The Last Letter Concatenation example also has a correct plan but an incorrect reasoning path and final result.Source: §B.4 · p. 21, Table 17 · p. 22
  • A correct plan may not be followed during execution, leading the model to omit crucial steps.The cited example is an AlpacaEval task.Source: §B.4 · p. 22, Table 18 · p. 22
  • Plans themselves may be flawed or incomplete; one math example uses “days in february % 7” instead of “(days in february -1 ) % 7”.The authors identify nuanced multi-step reasoning as an area where planning needs refinement.Source: §B.4 · p. 23, Table 19 · p. 23

Related papers

Loading 2409.12452v2…