MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark

1 Introduction

• Existing benchmarks offer limited perspectives and difficulty grading, making comprehensive evaluation across theory, application, levels, disciplines, and languages difficult.The introduction notes that prior resources also overlook fundamental theory understanding needed for application problems.Source: §1 · p. 1
• MathBench spans five stages from basic arithmetic to college mathematics and evaluates both theoretical knowledge and practical applications bilingually.Its taxonomy follows educational stages and is designed to assess mathematical understanding in breadth and depth.Source: §1 · p. 2
• The paper details MathBench’s hierarchical knowledge system, data collection, and question-selection criteria, and reports experiments across models to identify current bottlenecks.The benchmark covers fundamental concepts through real-world applications.Source: §1 · p. 2

2.1 The Hierarchical Knowledge System

• The framework has five stages—Arithmetic, Primary, Middle, High, and College—and three levels, with Subject Area and Topic providing finer-grained knowledge categories.Arithmetic covers the four basic operations; the taxonomy extends from foundational arithmetic to abstract college-level concepts.Source: §2.1 · pp. 2–3
• Stage, subject-area, and topic tags support fine-grained analysis of model strengths and weaknesses, while Arithmetic emphasizes basic operations as a foundation for later learning.Subject areas include disciplines such as algebra, geometry, calculus, statistics, and probability.Source: §2.1 · p. 3

2.2 Data Collection and Statistics

• Collection covers theoretical questions testing formulas, theories, and corollaries, alongside application questions testing their use in practice.Theoretical questions are built topic by topic from textbooks and online sources; application questions are selected to match educational level and taxonomy coverage, including exams and competitions.Source: §2.2 · p. 3
• Some questions with complex answers are reformulated as multiple choice, typically with four options, with unique correct answers and highly confusable distractors.A semi-automated filtering process addresses intrinsic question errors and alignment with educational stages.Source: §2.2 · p. 3
• MathBench contains 3,709 bilingual questions: 2,209 theoretical questions in MathBench-T and 1,500 practical-application questions in MathBench-A.Each question underwent semi-automated vetting.Source: §2.2 · p. 4

3.1 Configuration

• Evaluations use CircularEval for Chat models and Perplexity for Base models, with standardized greedy decoding and output limits across more than 20 evaluated models.Open-ended questions use few-shot CoT; multiple-choice Chat questions use zero-shot CoT, while Base-model PPL evaluation uses few-shot prompting.Source: §3.1 · p. 4

3.2.1 MathBench-A

• Qwen1.5-110B-Chat is the leading open-source application model, while DeepSeek-Math-7B-RL is the top open-source math model and surpasses DeepSeek-67B-Chat by 24.8%.Qwen1.5-110B-Chat scores 58.4 on application problems, comparable to GPT-4-0125-Preview at 58.8.Source: §3.2.1 · pp. 4–5

3.2 Main Results

• GPT-4o leads overall on MathBench-A and scores 87.0 on MathBench-T, the highest theoretical average among tested closed-source and open-source chat models.The overall comparison covers both MathBench-A and MathBench-T; the theory score is balanced with an application score of 70.9.Source: §3.2.1 · p. 4, Table 2 · p. 5, §3.2.2 · pp. 5–6

3.2.2 MathBench-T

• GPT-4o leads theoretical performance across nearly all stages, while Qwen1.5-110B-Chat scores highest in Primary theory and DeepSeek-Math-7B-RL excels among math models.GPT-4o scores 84.1 in College theory, 16.9 points above the best open-source math model; high application performers also tend to perform well in theory.Source: §3.2.2 · pp. 5–6

3.3 Evaluation of Base Models

• Base-model performance generally aligns with Chat-model trends, but ChatGLM3-6B-Base ranks second among 7B Base models while ChatGLM3-6B trails Qwen-7B-Chat.The reported gaps are 95.2% on MathBench-A and 104.7% on MathBench-T; the authors suggest differing fine-tuning strategies may contribute.Source: §3.3 · p. 6

3.4 Detailed Analysis

• From Middle stage onward, application performance declines, and the theory–application gap widens at advanced stages; GPT-4 leads application scores across stages.Against Qwen-72B-Chat, GPT-4’s theoretical and application score differences rise from (1.4, 8.7) in Middle to (13.5, 23.0) in College.Source: §3.4 · p. 7
• Models score better on elementary topics such as Unit Conversion and Four Operations than on Double Integrals, Mathematical Logic, and Set Theory; CoT plus knowledge infusion performs best in exploratory tests.On 200 Qwen-72B-Chat questions, the reported order is Straight < CoT < Straight-Knowledge < CoT-Knowledge.Source: §3.4 · pp. 7–9, Figure 6 · p. 9
• GPT-4 has a 67.1 bilingual average, with scores of 65.2 in Chinese and 69.0 in English; most evaluated models have larger language gaps.CircularEval–Accuracy discrepancies narrow as model performance improves.Source: §3.4 · pp. 6, 8, Figure 4 · p. 8, Figure 5 · p. 8

4.2 Effect of Model Size on Math Capabilities

• Qwen, MammoTH, and Yi models generally improved as parameter size increased, but DeepSeekMath-7B outperformed DeepSeek-72B and MammoTH-70B despite having 10x fewer parameters.The figure compares model parameter size with average MathBench score; its red dotted line marks GPT-4’s score.Source: §4.2 · p. 10, Figure 8 · p. 10

4.1 How Models Perform with Code Agent on MathBench

• InternLM2-7B-Chat’s Arithmetic Application score rose about 64%, from 53.0 to 87.3, with the Code Agent, while College Application performance did not notably improve.The Code Agent also did not significantly enhance theoretical performance across stages.Source: §4.1 · p. 10, Figure 9 · p. 11

4.3 Error Analysis

• 49.5% of mistakes involved misconceptions, 33.4% flawed but logically consistent reasoning, 9.6% reasoning off the intended query, and 4.0% responses exceeding the token limit.Reasoning-related errors increased with task difficulty; the figure covers theoretical and application questions.Source: §4.3 · p. 11, Figure 10 · p. 11
• Models with enhanced reasoning capabilities can critically assess presented options and offer answers beyond predetermined choices.Source: §4.3 · p. 11

4.4 Reasoning Path

• Small chat models can falter on symbol interpretation and relations, large chat models can divert despite broader knowledge, and specialized math models show systematic reasoning on complex problems.GPT-4 is described as handling complex paths, mathematical symbols, and ambiguities effectively.Source: §4.4 · pp. 11–12

Preliminary Mathematical Datasets

• Early datasets targeted elementary algebra or basic arithmetic, with assessments limited in mathematical form and content.Examples include Alg514, SingleEq, DRAW-1K, AddSub, SingleOp, and MultiArith.Source: §5 · p. 12

Benchmarks tailored to specific educational tiers

• Benchmarks designed for educational tiers largely focus on elementary mathematics and seldom assess college-level knowledge.Examples include Math23k, ASDiv, and GSM8K.Source: §5 · p. 12

Enriching the diversity of mathematical problem types within benchmarks

• Other datasets broaden mathematical domains or problem types, but some still lack diversity in question formulation.MathQA, Mathematics Dataset, and MATH [Hendrycks et al., 2021b] cover varied domains or complexity; NumGLUE and Lila incorporate other reasoning skills or background knowledge.Source: §5 · p. 12

• MathBench categorizes questions by stage and knowledge level across mathematics topics to support comprehensive evaluation of LLM mathematical proficiency.Source: §6 · p. 12

• About 19% of questions come from open-source datasets and may be contaminated, limiting assurance that models have not previously encountered them.The authors plan to automate question construction across stages in future iterations.Source: §7 · pp. 12–13
• MathBench does not provide detailed reasoning paths for each question, making it challenging to unlock the questions’ full potential.The authors aim to investigate semi-automated natural-language and code-based reasoning approaches.Source: §7 · p. 13

8 Ethical Considerations

• The benchmark work used publicly accessible reference materials and closed-source models, with generated data selected and processed by humans to protect privacy and confidentiality.The authors state that no personally identifiable information was used and that all data were anonymized before analysis.Source: §8 · p. 13
• ChatGPT and Grammarly were used to refine the manuscript’s language.Source: §8 · p. 13

A.1 Dataset Statistics

• Table 5 covers theoretical and application question distributions across stages, while Table 6 reports fine-grained knowledge levels.Source: §A.1 · p. 15

A.2 Data collection details

• GPT-4 supports two Primary-stage datasets: Chinese GSM-X-CN questions and English GSM-X-Plus questions.Source: §A.2 · p. 15
• GSM-X-CN questions are produced by translating GSM8k test questions into Chinese and replacing entity names while preserving meaning.Source: §A.2 · p. 15
• GSM-X-Plus questions modify numeric parameters using multiples of k, with k ∈(2, 10), and derive new answers by executing modified solution code.Source: §A.2 · p. 15
• Collected exam questions are processed and annotated by domain experts, with undergraduate students handling primary and secondary levels and graduate specialists handling university-level exams.Source: §A.2 · p. 15
• The benchmark also incorporates CE-VAL, MMLU, Arithmetic-HG, Math401, and SciBench, whose used datasets are stated to have MIT licenses.Source: §A.2 · p. 15

A.3 Quality Screening

• GPT-4 evaluates each question four times, and questions it answers incorrectly 0, 1, or 2 times are selected for manual review.The screening addresses intrinsic errors and questions judged too difficult or trivial for their intended education stage, which may destabilize CircularEval responses.Source: §A.3 · pp. 15–16

B.1 Overall Results

• Overall experimental results for chat models appear in Table 7, and results for base models appear in Table 8.Source: §B.1 · p. 16
• For stages other than Arithmetic, the reported score is the average of theoretical and application question scores.Source: §B.1 · p. 16
• Arithmetic is excluded from that averaging approach because it predominantly indicates pure computational prowess.Source: §B.1 · p. 16

B.2 RESULTS WITH ACCURACY

• Detailed accuracy results are presented in Table 9.Source: §B.2 · p. 16
• The supplied passages identify the results tables but do not state their accuracy values or comparisons.Source: §B.2 · p. 16, §B.3 · p. 16

B.3 BILINGUAL

• Bilingual results are presented in Table 10.Source: §B.3 · p. 16

C.2.3 Misaligned with the question; C.3.3 Math model

• The appendix includes tables comparing chat models, base models, accuracy, and bilingual MathBench performance.Source: Table 7 · p. 17, Table 8 · p. 18, Table 9 · p. 19

C.2 Error Types Demonstration

• The error examples cover limited theoretical capability, conceptual misunderstanding, flawed reasoning, misalignment with intent, and responses exceeding the output limit.Source: Figure 11 · p. 21, Figure 12 · p. 22, Figure 13 · p. 23

C.2.5 Responses constrained to Options; C.2.6 Non-adherence to the prompt

• Separate examples illustrate responses constrained to options and non-adherence to the prompt.Source: Figure 16 · p. 26, Figure 17 · p. 27

C.3.1 Small-scale chat model

• Figure 18 presents an example reasoning path for a small-scale chat model.Source: Figure 18 · p. 28

C.3.2 Large-scale chat model

• Figure 19 presents an example reasoning path for a large-scale chat model.Source: Figure 19 · p. 29

C.3.3 Math model

• Figure 20 presents an example reasoning path for a math model.Source: Figure 20 · p. 30

C.1.1 English Open-ended test; C.1.2 Chinese Open-ended test

• Figures 21 and 22 show example English and Chinese open-ended test prompts.Source: §C.1.1 · p. 16, §C.1.2 · p. 16, Figure 21 · p. 31

C.1.3 English single choice with reasoning; C.1.4 Chinese single choice with reasoning

• Figures 23 and 24 show example English and Chinese single-choice prompts with reasoning.Source: §C.1.3 · p. 16, §C.1.4 · p. 16, Figure 23 · p. 32