Source-linked AI summary
MathPrompter: Mathematical Reasoning using Large Language Models
Shima Imani, Liang Du, Harsh Shrivastava
TL;DR
LLMs often produce incorrect answers on arithmetic reasoning tasks, while existing chain-of-thought prompting does not validate intermediate steps or provide confidence. MathPrompter generates multiple algebraic and Pythonic solutions, verifies their outputs across randomized inputs, and uses consensus to produce answers. On MultiArith, it reports 92.5% accuracy versus 78.7% for the prior zero-shot-CoT result.
Problem
LLMs have limited performance on arithmetic reasoning tasks, and existing chain-of-thought prompting lacks checks on intermediate-step validity and prediction confidence.
Method
MathPrompter uses zero-shot chain-of-thought prompts to generate multiple algebraic expressions or Python functions, then verifies their outputs through randomized evaluations and consensus.
Results
92.5% accuracy on MultiArith is reported for MathPrompter, compared with 78.7% for the state-of-the-art zero-shot-CoT baseline.
Takeaways & Limitations
MathPrompter improves LLM performance on mathematical reasoning and provides users with greater reliance in predictions through cross-checking and multiple solution approaches.
Takeaways & Limitations
Matching algebraic and Pythonic outputs do not always guarantee correctness, although increasing the number of prompts may mitigate this issue.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have limited performance when solving arithmetic reasoning tasks and often provide incorrect answers. Unlike natural language understanding, math problems typically have a single correct answer, making the task of generating accurate solutions more challenging for LLMs. To the best of our knowledge, we are not aware of any LLMs that indicate their level of confidence in their responses which fuels a trust deficit in these models impeding their adoption. To address this deficiency, we propose `MathPrompter', a technique that improves performance of LLMs on arithmetic problems along with increased reliance in the predictions. MathPrompter uses the Zero-shot chain-of-thought prompting technique to generate multiple Algebraic expressions or Python functions to solve the same math problem in different ways and thereby raise the confidence level in the output results. This is in contrast to other prompt based CoT methods, where there is no check on the validity of the intermediate steps followed. Our technique improves over state-of-the-art on the MultiArith dataset ($78.7\%\rightarrow92.5\%$) evaluated using 175B parameter GPT-based LLM.
1 INTRODUCTION
LLMs perform poorly on multi-step mathematical reasoning, and existing zero-shot chain-of-thought prompting does not verify intermediate steps or provide prediction confidence. MathPrompter addresses these gaps by generating and checking multiple solution expressions.
- Prompting techniques that work well for single-step tasks remain inadequate for system-II tasks requiring multi-step reasoning.
- Zero-shot-CoT improved MultiArith accuracy from 17.7% to 78.7%, but its intermediate reasoning steps were not validated.
- MathPrompter generates multiple algebraic expressions or Python functions to solve an algebraic template in different ways.
- The method evaluates these solutions on randomized variable assignments and accepts a final answer when outputs reach consensus across approximately five repetitions.
2 METHOD
MathPrompter transfers student-like verification into LLM answer generation by combining multiple solution approaches, intermediate-step cross-checking, computation, and repeated consensus testing. The method illustrates this process on a MultiArith restaurant-cost problem.
- MathPrompter transfers students' verification process to LLM answer generation through cross-checking and multiple approaches.
- The MultiArith example asks for the cost of 15 restaurant customers when 8 are children and children eat free.
- The method replaces numerical entries with variables to create an algebraic template before prompting the LLM for alternative solutions.
- Two prompts request a mathematical equation and a Python function, producing algebraic and Pythonic solution forms.
- Generated expressions are evaluated with randomized variable mappings, and matching outputs provide a higher level of confidence before computing the final answer.
- The procedure repeats prompting and verification approximately five times and reports the most frequent answer value.
3 EXPERIMENT
MathPrompter is evaluated on MultiArith against standard Zero-shot and Zero-shot-CoT baselines, using accuracy and example comparisons to assess its performance and reasoning outputs.
- 3.1 DATASET: MultiArith tests models on problems requiring multiple arithmetic operations and logical reasoning.
- 3.2 BASELINE: The standard Zero-shot baseline and Kojima et al.’s Zero-shot-CoT model provide comparison points for MathPrompter.Kojima et al.’s method prompts the model to think step-by-step before producing a final Arabic-numerical answer.
- 3.3.1 ACCURACY COMPARISONS: 92.5% accuracy is reported for MathPrompter, compared with 78.7% for Zero-shot-CoT and 17.7% for standard Zero-shot.The comparison uses 175B-parameter models for MathPrompter, Zero-shot-CoT, and standard Zero-shot.
- 3.3.2 EXAMPLE COMPARISONS: MathPrompter can avoid some one-step errors through repeated outputs and consensus, while its Algebraic and Pythonic approaches typically require fewer tokens.The method also uses Python’s eval() function when the reasoning is correct but the final computation is incorrect.
- 3.3.2 EXAMPLE COMPARISONS: In some cases, Algebraic and Pythonic outputs agree yet remain erroneous, as illustrated by the last question in Table 2.
4 CONCLUSIONS & DISCUSSIONS
The paper presents MathPrompter as an approach for improving mathematical reasoning performance and increasing reliance on LLM predictions through cross-checking and multiple solution methods.
- MathPrompter improves LLM performance on mathematical reasoning problems.
- The approach transfers aspects of students’ arithmetic problem-solving process to LLM answer generation.
- MathPrompter cross-checks intermediate steps and solves the same problem using multiple approaches.
- The method is reported as comparable to state-of-the-art Few-shot-CoT models and larger 540B-parameter Zero-shot-CoT models.
5 LIMITATION
MathPrompter’s repeated, multi-approach verification increases accuracy but does not always guarantee correct output.
- Repeated MathPrompter runs in different ways do not always guarantee correctness.Both Algebraic and Pythonic expressions can be incorrect even when their prompt outputs match.
- Increasing the number of prompts may mitigate this failure, while more principled techniques remain under investigation.