Source-linked AI summary
Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks
Wenhu Chen, Xueguang Ma, Xinyi Wang, William W. Cohen
TL;DR
Numerical reasoning requires language models to handle both reasoning and computation, but CoT is prone to arithmetic and complex-expression errors. PoT generates programs for the reasoning process and delegates computation to an external interpreter, improving performance across math and financial benchmarks, especially with self-consistency. Its scope is constrained by execution-safety restrictions and difficulty with diverse algebraic questions.
Problem
CoT requires language models to perform both reasoning and computation, which is prone to arithmetic errors and struggles with complex expressions and lengthy iteration.
Method
PoT has language models express reasoning as programs executed by an external interpreter, with PoT optionally combined with CoT for additional textual reasoning.
Results
PoT outperforms CoT across evaluated datasets, with average gains of 8% on few-shot MWP, 15% on few-shot financial QA, and 12% on zero-shot MWP; PoT+SC reaches best-known MWP results.
Takeaways & Limitations
Separating computation from reasoning enables stronger numerical-reasoning performance and supports combining symbolic execution with language models for other symbolic reasoning tasks.
Takeaways & Limitations
PoT execution requires restricting generated code for safety, while the method achieves only 58% accuracy on AQuA’s complex algebraic questions.
Abstract
from arXiv · showhide
Recently, there has been significant progress in teaching language models to perform step-by-step reasoning to solve complex numerical reasoning tasks. Chain-of-thoughts prompting (CoT) is by far the state-of-art method for these tasks. CoT uses language models to perform both reasoning and computation in the multi-step `thought' process. To disentangle computation from reasoning, we propose `Program of Thoughts' (PoT), which uses language models (mainly Codex) to express the reasoning process as a program. The computation is relegated to an external computer, which executes the generated programs to derive the answer. We evaluate PoT on five math word problem datasets (GSM, AQuA, SVAMP, TabMWP, MultiArith) and three financial-QA datasets (FinQA, ConvFinQA, TATQA) for both few-shot and zero-shot setups. Under both few-shot and zero-shot settings, PoT can show an average performance gain over CoT by around 12\% across all the evaluated datasets. By combining PoT with self-consistency decoding, we can achieve SoTA performance on all math problem datasets and near-SoTA performance on financial datasets. All of our data and code are released in Github https://github.com/wenhuchen/Program-of-Thoughts
1 Introduction
Numerical reasoning benchmarks test language models on math word problems and financial questions, but CoT requires the model to perform both reasoning and computation. PoT separates these roles by generating executable programs, yielding stronger results across the evaluated datasets.
- Numerical reasoning benchmarks include math word problems and financial questions expressed through text, tables, and conversation.
- CoT makes language models perform both reasoning and computation, creating arithmetic, complex-expression, and iteration challenges.
- PoT expresses reasoning as Python programs and delegates computation to an external interpreter.
- 8% and 15% are the average few-shot gains over CoT on MWP and financial datasets, respectively.
- 12% is PoT’s average zero-shot gain over CoT on the evaluated MWP datasets.
- PoT+SC achieves the best-known results on all evaluated MWP datasets and near-best-known results on financial datasets excluding GPT-4.
2 Program of Thoughts
PoT uses programs as a structured representation of reasoning while an external interpreter performs computation. It supports few-shot and zero-shot prompting and can be combined with CoT when textual reasoning is additionally required.
- 2.1 Preliminaries: In-context learning prompts language models with input-output exemplars without changing model parameters.
- 2.2 Program of Thoughts: Unlike CoT, PoT assigns the language model reasoning-expression while a Python interpreter performs computation.
- 2.2 Program of Thoughts: PoT differs from direct equation generation by decomposing equations into multi-step programs with semantically meaningful variables.
- 2.1 Preliminaries: Few-shot PoT uses question-program exemplars, whereas zero-shot PoT uses an instruction without demonstrations.
- 2.1 Preliminaries: Zero-shot PoT can return an answer directly, unlike zero-shot CoT, which requires an additional answer-extraction step.
- 2.2 Program of Thoughts: For problems requiring textual reasoning, PoT execution results can be supplied to CoT to derive the final answer.
- 2.2 Program of Thoughts: AQuA requires this combined strategy because its computed numeric result may need conversion into the answer option’s format.
3 Experiments
The experiments evaluate PoT across heterogeneous mathematical and financial reasoning datasets under few-shot and zero-shot settings, alongside implementation, baseline, ablation, and error analyses. PoT consistently outperforms CoT, with gains linked to external computation, multi-step programs, semantic variable binding, and challenging problem types.
- Experimental Setup: The evaluation covers five math word-problem and three financial-QA datasets with heterogeneous text, table, and conversational inputs.The datasets are used to assess PoT’s generalizability across input formats.
- Main Results: Roughly 20% are PoT’s few-shot gains over CoT on FinQA and ConvFinQA, while gains reach 8% on TATQA and exceed 8% on several math datasets.The larger financial gains are attributed to reduced large-number miscalculations from using an external computer.
- Main Results: 12% is PoT’s average zero-shot gain over CoT across the evaluated datasets.PoT significantly outperforms zero-shot CoT across all evaluated math word-problem datasets and can generalize without dataset-specific exemplars.
- Ablation Studies: PoT’s largest improvements occur on linear or polynomial equations, iterative, symbolic, and combinatorics questions, whereas arithmetic, probability, and geometry show similar performance to CoT.Ablations also indicate that semantic variable names and decomposing equations into multiple steps improve performance; performance becomes more stable with more exemplars.
4 Related Work
Related work spans numerical-reasoning benchmarks, in-context learning, chain-of-thought prompting, extensions to PoT, and broader tool-use approaches for grounding language models.
- Numerical-reasoning research includes arithmetic and algebraic questions, increasingly challenging datasets, math word problems, and financial problems.
- In-context learning lets large language models solve tasks from a few input-output exemplars without training or fine-tuning.
- Chain-of-thought prompting elicits reasoning through natural-language rationales and has achieved strong results on reasoning benchmarks.
- Follow-up methods enhance PoT through self-evaluation, self-critique, or more detailed planning instructions.
- Tool-use methods generalize Python-program assistance to APIs such as search and string extraction for more complex reasoning and grounding problems.
5 Discussion
The discussion finds PoT effective for numerical reasoning in math and finance, while positioning CoT as broader for semantic reasoning tasks.
- PoT works efficiently on numerical reasoning tasks such as math and finance problem solving.
- PoT is considered suitable for highly symbolic reasoning, whereas CoT can solve broader reasoning tasks.
6 Conclusions
The paper disentangles reasoning from computation by having language models generate programs while an external interpreter performs computation, improving performance on math datasets and motivating symbolic-execution research.
- PoT prompts language models to generate programs expressing complex reasoning while an external interpreter handles computation separately.
- The approach boosts language-model performance on several math datasets.
- The work aims to combine language models with symbolic execution to solve challenging math problems.
- The authors suggest combining symbolic execution with language models for other symbolic reasoning tasks.
Limitations
The paper identifies execution safety, generalization under restricted modules, and difficult diversified algebraic questions as limitations of PoT.
- Generated PoT code may contain dangerous operations, so the implementation blocks additional imports and restricts execution to predefined modules.
- Restricting imports works reasonably for math question answering but may hurt PoT generalization to unknown symbolic tasks.
- PoT reaches only 58% accuracy on AQuA because its diverse algebraic questions cannot be fully covered by demonstrations.
7 Appendix
The appendix demonstrates PoT prompting through an intermediate-answer workflow, pseudocode, and concrete Python exemplars spanning arithmetic, symbolic, iterative, and commonsense-style problems.
- PoT workflow: PoT first generates an intermediate program, executes it, and can use the result to prompt a model for the final answer.The workflow is presented as PoT(question), execution of the program, and subsequent prompting with the computed intermediate result.
- PoT workflow: The appendix states that PoT can serve as an intermediate step for questions requiring both symbolic and commonsense reasoning.For additional textual reasoning, the executed program provides an intermediate result that is combined with the question to derive the final answer.
- Prompting exemplars: The appendix includes Python exemplars that compute answers through direct arithmetic, symbolic algebra, conditionals, and loops.Examples cover egg sales, fiber quantities, house profit, feed allocation, equation solving, divisibility checks, and iterative calculations.
- Prompting exemplars: A cake-scheduling exemplar converts preparation, baking, cooling, and frosting durations into a latest start time.The program totals the durations in minutes, converts to hours, and subtracts the total from the serving time.
- Prompting exemplars: Additional exemplars use named variables and executable operations to solve contribution, interest, boat-speed, percentage, and divisibility questions.The snippets use symbolic expressions, equation solving, simplification, and explicit iteration or condition checks.