Source-linked AI summary
How well do Large Language Models perform in Arithmetic tasks?
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang
TL;DR
The paper addresses the limited direct evaluation of arithmetic ability in large language models, which is needed because arithmetic underlies step-by-step solutions to math word problems. It introduces MATH 401, evaluates diverse models and arithmetic expressions, and finds that GPT-4 and ChatGPT lead substantially, while several factors affect performance.
Problem
Direct evidence about large language models’ arithmetic ability is limited, despite arithmetic being required at each step of chain-of-thought solutions to math word problems.
Method
The paper constructs MATH 401 and evaluates large language models on diverse arithmetic expressions using accuracy, relative error, and non-number ratio.
Results
GPT-4 and ChatGPT outperform other evaluated models by a large margin across arithmetic abilities, while tokenization, pre-training corpus, prompts, and parameter counts are important factors.
Takeaways & Limitations
MATH 401 provides a focused basis for analyzing and improving the arithmetic ability of large language models.
Takeaways & Limitations
The evaluation cannot assess earlier ChatGPT versions because OpenAI’s reported improvement began with the January 30 version.
Abstract
from arXiv · showhide
Large language models have emerged abilities including chain-of-thought to answer math word problems step by step. Solving math word problems not only requires abilities to disassemble problems via chain-of-thought but also needs to calculate arithmetic expressions correctly for each step. To the best of our knowledge, there is no work to focus on evaluating the arithmetic ability of large language models. In this work, we propose an arithmetic dataset MATH 401 to test the latest large language models including GPT-4, ChatGPT, InstrctGPT, Galactica, and LLaMA with various arithmetic expressions and provide a detailed analysis of the ability of large language models. MATH 401 and evaluation codes are released at \url{https://github.com/GanjinZero/math401-llm}.
1 Introduction
The paper argues that arithmetic computation is a necessary upper bound for solving math word problems with chain-of-thought, motivating direct evaluation of LLM arithmetic ability. It introduces MATH 401 to test diverse arithmetic expressions and analyze factors affecting performance.
- Motivation: Chain-of-thought decomposes math word problems into steps, but each step still requires correct arithmetic computation.The paper therefore treats arithmetic ability as an upper bound on solving math word problems.
- Contribution: MATH 401 is an arithmetic dataset designed to evaluate large language models.
- Scope: The project is identified as work in progress.
- Dataset scope: The dataset covers arithmetic expressions involving integers, decimals, irrational numbers, brackets, trigonometric functions, and logarithms.It includes operations from addition through exponentiation, trigonometry, and logarithms.
- Main finding: GPT-4 and ChatGPT outperform other evaluated models by a large margin across arithmetic abilities.The paper reports this pattern across the evaluated arithmetic categories.
- Motivation: Arithmetic evaluation is also presented as a gauge for general intelligence because arithmetic supports more intricate mathematical tasks.The paper connects arithmetic ability with symbolic mathematics and automatic theorem proving.
2 Related Works
Prior work evaluated mathematical reasoning and limited arithmetic capabilities, while this paper broadens evaluation to instruction-tuned large language models and comprehensive arithmetic abilities. The supplied related-work passages also identify version-dependent evaluation conditions for ChatGPT and GPT-4.
- Math ability evaluation: Earlier studies evaluated language models on math word problems and mathematical ability benchmarks.
- Evaluation context: The reported ChatGPT and GPT-4 evaluations may differ from other results because they used specific model versions, interfaces, and an evaluation date.ChatGPT used gpt-3.5-turbo-0301, while GPT-4 was evaluated through the chat UI on March 16th.
- Arithmetic evaluation: Prior arithmetic studies focused on simple addition, subtraction, or multiplication and identified tokenization and token frequency as important factors.
- Arithmetic evaluation: MATH 401 differs by evaluating instruction-finetuned large language models on comprehensive arithmetic abilities with different expression types.
3 Evaluation Settings
The evaluation uses 401 arithmetic expressions spanning elementary operations, numerical types, long expressions, trigonometry, and logarithms, with accuracy, relative error, and non-number ratio as output metrics. It compares many LLM families under varied prompts, input formats, and decoding settings.
- Dataset construction: The benchmark contains 401 arithmetic expressions organized into groups, including an Euler equation group and 25 problems for each of groups 1–16.
- Dataset construction: The benchmark also includes multiplication, division, exponentiation, irrational-number operations, long expressions, trigonometry, and logarithms.These groups extend coverage beyond basic integer addition and subtraction.
- Dataset construction: The authors classify groups 1, 2, 3, 5, 6, 7, 8, and 11 as easy and the remaining groups as hard, computing reference results with Python and rounding to four decimal places.
- Metrics: For evaluation, the authors preserve the last decoded number after ignoring numbers in parentheses and convert fractions to decimals except for the division group.
- Metrics: Accuracy counts predictions within 1e-3 of the target, while relative error is capped at 10 and non-number outputs count as failures.
- Models and prompts: The experiments compare GPT-4, GPT-3.5, InstructGPT, Galactica, LLaMA, OPT, Bloom, T0++, GLM, Flan-T5, and smaller variants.
- Models and prompts: The study varies prompts, input formats, system instructions, and decoding settings, including greedy decoding for non-OpenAI models.Tested prompts include no prompt, “Calculate:”, dollar-sign prompts, and an equation environment.
4 Results and Analysis
GPT-4 and ChatGPT outperform other models on MATH 401, while arithmetic performance varies sharply by operation, number type, expression length, and prompting. GPT-4 is strongest on long expressions and several difficult operations, but both models retain specific failure modes.
- GPT-4 surpasses ChatGPT with accuracy of 10 points and reduces relative error by half; InstructGPT ranks third by accuracy.
- Grouped Results: GPT-4 ranks first and ChatGPT second across all arithmetic groups, while most other LLMs handle mainly addition, subtraction, and some multiplication.
- Grouped Results: Division, exponentiation, trigonometry, and logarithms are hard for most LLMs, whereas GPT-4 and ChatGPT handle big numbers above 1e12 and complex long queries.
- ChatGPT passes all Group 7 and 8 multiplication queries but gets every Group 9 query wrong, including a close estimate for 71786 × 21638.
- ChatGPT’s accuracy improves when prompted for four decimal places, because answers correct to two decimal places can be marked incorrect by the evaluation metric.
- ChatGPT handles integer-base exponentiation and special trigonometric inputs, but decimal-base exponentiation and erroneous explanations can produce wrong answers.
- GPT-4 solves long equations step by step and obtains much higher accuracy, while ChatGPT may generate wrong answers when it decodes long-expression results directly.
4.2 Tokenization
Tokenization is associated with large differences in arithmetic performance: models using digit-level tokenization outperform comparable models, while T5-series models lack several arithmetic-specific tokens.
- T0++-11B and Flan-T5-xxl-11B perform worse than similarly sized Opt-13B and LLaMA-13B models on arithmetic tasks because arithmetic expressions contain special tokens absent from T5-series models.
- Galactica and LLaMA split numbers into individual digit tokens, and their accuracy exceeds OPT, Bloom, and GLM models at comparable larger scales.
- Galactica-30B reaches 45.14 accuracy and LLaMA-30B 30.17, compared with OPT-30B 15.96, Bloom-176B 20.2, and GLM-130B 25.94.
4.3 Training
Training choices are associated with arithmetic performance: instruction tuning improves measured metrics, and RLHF further improves performance over supervised fine-tuning in the reported comparison.
- Code-davinci-002 performs well on reasoning-related tasks but worse in arithmetic than other LLMs, separating mathematical reasoning ability from arithmetic ability.
- Galactica’s extensive pretraining on LaTeX sources is proposed as a possible reason for its relatively strong arithmetic performance.
- Instruction tuning boosts accuracy, relative error, and non-number ratio across Opt, Bloom, and code-davinci comparisons.
- Text-davinci-003 with RLHF outperforms text-davinci-002 with SFT on arithmetic tasks, indicating a reported benefit from RLHF.
4.4 Prompts
Prompt design substantially affects arithmetic evaluation, with optimal prompts varying across models. System-level instructions also improve ChatGPT’s accuracy and reduce relative error.
- Input Prompts: Prompt sensitivity varies across LLMs, and omitting a prompt is the worst option for most models.“Calculate” performs best for InstructGPT and ChatGPT, while LATEX-related prompts perform best for other LLMs.
- System Prompts: System-level messages improve ChatGPT’s accuracy and reduce relative error significantly.The largest differences occur for irrational-number and logarithm-function groups.
- System Prompts: Requiring four decimal places gives ChatGPT a zero non-number ratio.This instruction also improves accuracy for multiplication and division, where two-decimal answers can otherwise be marked incorrect.
4.5 Interpolation and Extrapolation
The paper uses easy and hard arithmetic groups to approximate interpolation and extrapolation, then examines how model scale and prompting relate to performance. ChatGPT performs best on hard queries, while larger parameter counts generally help but do not guarantee superior arithmetic ability.
- 4.5 Interpolation and Extrapolation: Easy queries approximate interpolation, whereas hard queries with large numbers, decimals, or long expressions test extrapolation.Easy cases may occur in pretraining or instruction tuning; hard cases are unlikely to be covered there.
- 4.5 Interpolation and Extrapolation: ChatGPT performs best on hard queries, while all other tested models show limited hard-query performance.The result is interpreted as evidence of limited extrapolation in the other models.
- Scaling Laws: Larger parameter counts generally correspond to better arithmetic ability.The comparison excludes models without smaller versions using the same setting.
- Scaling Laws: Models above 30B parameters do not improve significantly over 30B models, with Galactica’s 120B model matching its 30B model.The authors hypothesize that 30B may be sufficient for arithmetic ability.
- Chain-of-Thought: Zero-shot chain-of-thought does not improve any arithmetic group, including long-expression queries.The “Calculate:” prompt already produces chain-of-thought for long expressions and direct answers for easy questions.
4.8 In-context Learning
The study tests in-context learning by providing eight similar arithmetic examples before each query. It does not improve Galactica’s accuracy, though it reduces relative error, and small Flan models fail to produce numbers under this setting.
- 4.8 In-context Learning: The in-context-learning setup provides eight similar arithmetic expressions before each query.The examples are different from the query and are tested on Galactica and Flan-T5.
- 4.8 In-context Learning: For Galactica, in-context learning does not improve accuracy but reduces relative error significantly.This evaluates a comparatively well-behaved model under the same task setting.
- 4.8 In-context Learning: Flan-T5 models smaller than 3B cannot generate any number under the in-context-learning setting.The failure is reported specifically for the small-sized Flan models.
5 Conclusion
The paper introduces MATH 401 to evaluate arithmetic ability in LLMs and identifies tokenization, pretraining data, prompts, and parameter counts as important factors. It also emphasizes that arithmetic is only one part of broader mathematical evaluation.
- 5 Conclusion: MATH 401 evaluates LLM arithmetic ability across arithmetic expressions and model families.The paper analyzes factors associated with arithmetic performance, including tokenization, pretraining corpus, prompts, and parameter counts.
- 5 Conclusion: The reason for ChatGPT’s strong arithmetic performance remains uncertain, including the roles of parameter counts and instruction datasets.The paper identifies these factors as unresolved explanations rather than established causes.
- 5 Conclusion: The study is limited to arithmetic, leaving symbolic mathematics, equations, calculus, algebra, geometry, probability, and graph theory for future evaluation.These topics are presented as additional interesting areas rather than evaluated components of this paper.
A Examples from MATH 401
This section lists examples drawn from the MATH 401 dataset.
- The section presents examples from each group in MATH 401.