Source-linked AI summary
Solving Math Word Problems by Combining Language Models With Symbolic Solvers
Joy He-Yueya, Gabriel Poesia, Rose E. Wang, Noah D. Goodman
TL;DR
Math word-problem solvers must handle both procedural calculation and declarative reasoning, while LLMs can struggle with arithmetic. The paper combines incremental LLM formalization into variables and equations with an external symbolic solver, matching original PAL on GSM8K and exceeding PAL by 20% on ALGEBRA.
Problem
Prior LLM-based approaches are less effective for word problems requiring declarative reasoning, while LLMs often struggle with arithmetic operations.
Method
The approach incrementally formalizes word problems as variables and equations with an LLM, then solves those equations using an external symbolic solver.
Results
20%: the approach outperforms PAL by an absolute 20% on ALGEBRA and achieves comparable accuracy to original PAL on GSM8K.
Takeaways & Limitations
Declarative and incremental representations are effective when an external tool solves complex math word problems.
Abstract
from arXiv · showhide
Automatically generating high-quality step-by-step solutions to math word problems has many applications in education. Recently, combining large language models (LLMs) with external tools to perform complex reasoning and calculation has emerged as a promising direction for solving math word problems, but prior approaches such as Program-Aided Language model (PAL) are biased towards simple procedural problems and less effective for problems that require declarative reasoning. We propose an approach that combines an LLM that can incrementally formalize word problems as a set of variables and equations with an external symbolic solver that can solve the equations. Our approach achieves comparable accuracy to the original PAL on the GSM8K benchmark of math word problems and outperforms PAL by an absolute 20% on ALGEBRA, a new dataset of more challenging word problems extracted from Algebra textbooks. Our work highlights the benefits of using declarative and incremental representations when interfacing with an external tool for solving complex math word problems. Our data and prompts are publicly available at https://github.com/joyheyueya/declarative-math-word-problem.
1 Introduction
Math word problems are important yet challenging, motivating tools that generate step-by-step solutions. The paper proposes combining incremental declarative formalization by an LLM with an external symbolic solver, achieving PAL-level GSM8K performance and a 20% ALGEBRA improvement.
- Motivation: Math word problems are important for students but challenging to solve.Automated step-by-step solutions could support students and educators.
- Prior approaches: LLM prompting can improve word-problem solving, but arithmetic errors remain a challenge.Prior methods use calculators or Python interpreters to offload calculations.
- Research gap: Declarative representations support mathematical declarations beyond problems that yield straightforward procedures.This addresses a limitation of procedure-oriented program representations.
- Contribution: The proposed approach incrementally formalizes problems as variables and equations, then uses an external symbolic solver.It combines language-model reasoning with symbolic equation solving.
- Contribution: 20%: the approach outperforms PAL by an absolute 20% on ALGEBRA while matching original PAL performance on GSM8K.ALGEBRA contains more challenging word problems collected from Algebra textbooks.
2 Related work
Prior work uses few-shot prompting, chain-of-thought reasoning, calculators, and PAL to improve mathematical word-problem solving, but these approaches remain oriented toward procedural calculation.
- LLM prompting: Few-shot prompting and chain-of-thought reasoning use demonstrations and explicit intermediate steps to improve LLM math solving.LLMs nevertheless often make arithmetic errors.
- External tools: External calculators generally improve GSM8K performance by less than 5%.PAL instead generates Python programs to perform more complex arithmetic.
3 Our Approach: Equipping an LLM With an External Symbolic Solver
The approach has an LLM incrementally formalize a word problem as variables and equations, then passes those equations to a symbolic solver. Its DECLARATIVE prompt uses few-shot examples and interleaved natural-language and formal statements.
- Pipeline: The pipeline first performs declarative, incremental formalization and then solves the resulting equations symbolically.These are the two central stages of the approach.
- Prompting: Few-shot prompting supplies demonstrations so the LLM can formalize a new word problem.The demonstrations are concatenated with the new input at inference time.
- DECLARATIVE prompt: The DECLARATIVE prompt represents each solution with interleaved natural-language statements and formal variable or equation declarations.A new problem is appended to the prompt, and the LLM generates its step-by-step solution.
- Prompt design: The prompt is designed around principles that specify properties desired in the generated solutions.These principles guide the formalization format.
- Symbolic solving: The generated solution supplies variables and equations but not the final answer; SymPy solves the extracted system algebraically.This prevents the LLM from having to calculate the equations directly.
4 Experimental Setup
The experiments evaluate multiple prompting methods and DECLARATIVE variants on GSM8K and ALGEBRA, using a fixed Codex configuration and symbolic-solver comparisons.
- Datasets: The evaluation uses GSM8K and ALGEBRA, a new collection of 222 Algebra textbook word problems.GSM8K contributes 1,319 grade-school math word problems.
- Methods: The study compares chain-of-thought, PAL, and DECLARATIVE prompting with both 8-shot and 3-shot prompts.The 3-shot examples were designed to illustrate step-by-step and declarative thinking.
- Ablations: The DECLARATIVE experiments include a SymPy variant, a direct-LLM-solving variant, and a one-step formalization variant.These variants test solver use and incremental versus single-step formalization.
- Resources: The paper provides its prompts publicly and makes the ALGEBRA dataset available.The resources are hosted in the authors’ public repository.
- Implementation: All methods use Codex with top-1 decoding, temperature 0, and max_tokens set to 600.These settings are held fixed across methods.
5 Results
Across GSM8K and ALGEBRA, the experiments compare prompting strategies and show that declarative, incrementally formalized solutions are especially effective on challenging algebra problems.
- Evaluation setup: Table 2 reports problem solve rates (%) as averages and standard deviations across three runs, with separate results for GSM8K and ALGEBRA.COT and PAL use both original 8-shot prompts and newly designed 3-shot prompts.
- ALGEBRA: Figure 4 contrasts incorrect procedural solutions from PAL and COT with a correct declarative solution for a challenging algebra word problem.The declarative solution describes properties of the goal rather than a procedural plan.
- GSM8K: The declarative approach matches the original PAL performance on GSM8K when using the 3-shot prompt with principles and SymPy.PAL remains stronger across comparable examples, but the specified declarative configuration reaches performance equivalent to original PAL.
- GSM8K: Adding principles to the declarative prompt improves GSM8K performance, while asking the LLM to solve equations directly reduces accuracy from 69.4% to 22.4%.The comparison highlights the benefit of using an external symbolic solver rather than relying on the LLM for equation solving.
- GSM8K: Incremental formalization improves declarative prompting performance relative to the corresponding one-step declarative variant.The result is shown by the performance gap between the incremental and non-incremental configurations.
- ALGEBRA: On ALGEBRA, the declarative approach achieves the highest accuracy and outperforms PAL by an absolute 20%.The dataset contains harder algebra problems, where procedural solutions are less intuitive for problems requiring declarative reasoning.
6 Conclusion
The paper equips an LLM with an external symbolic solver to generate step-by-step solutions through incremental declarative formalization. It finds this approach effective for complex math word problems and particularly useful when problems combine declarative and procedural reasoning.
- Contribution: The approach incrementally formalizes word problems as variables and equations, then uses an external symbolic solver to solve them and avoid arithmetic errors.This design is presented as the basis for automatically generating step-by-step solutions.
- Conclusion: Declarative formalization is effective when interfacing with an external tool to solve complex math word problems.The conclusion identifies this as a central finding of the work.
- Conclusion: Incremental formalization is beneficial, especially when using declarative representations.The paper connects this benefit to its incremental solution-generation process.
- Implications: The approach is particularly useful for math education because advanced problems can be divided into conceptual pieces involving declarative and procedural knowledge.This educational relevance is stated as a supported application scope.