CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?

1 Introduction

• The authors argue that math evaluation is difficult to interpret because existing datasets are often English-language or excessively difficult, and scores lack intuitive meaning.They motivate grade-level reporting as a more accessible way to convey model ability.Source: §1 · p. 1
• CMATH comprises 1.7k elementary-school math word problems sourced from actual Chinese workbooks and exams.Each problem has grade information, enabling fine-grained results such as performance on a fourth-grade exam.Source: §1 · pp. 1–2
• The paper evaluates popular LLMs available through commercial APIs or released model weights.The evaluation includes both commercial and open-source options as described in the paper context.Source: §1 · p. 2
• GPT-4 is the only evaluated model to achieve accuracy ≥60% across all six elementary-school grades and maintain robustness to distracting information.Other models falter at different grades or are misled by distractors.Source: §1 · p. 2

2.1 Motivation

• CMATH is motivated by the question of which elementary-school math grade level popular LLMs' abilities correspond to, with results intended to be intuitive to the public.The authors consider elementary problems a more appropriate probe of general-purpose reasoning and arithmetic than high-school or college problems, and more interpretable.Source: §2.1 · p. 2

2.2 Data Collection

• The problems are collected from Chinese elementary-school exercise books and exams freely available online.The materials are converted to text, graphic-based problems are discarded, and the remaining questions undergo preprocessing and human validation.Source: §2.2 · p. 2

2.3 Data annotation

• The dataset provides annotations for grade, answer, number of reasoning steps, and number of digits.Table 1 reports average problem length, reasoning steps, and digits involved in the solution.Source: §2.3 · p. 2, Table 1 · p. 2

2.3.1 Grade

• Each problem is assigned its elementary-school grade so that grade-specific subsets can support targeted, fine-grained evaluation.The supplied passages do not specify the grade-assignment procedure.Source: §2.3.1 · p. 2

2.3.2 Answer

• Ground-truth answers are standalone numerals in integer, decimal, fraction, or percentage form, and the test-only dataset does not provide reasoning processes.The answer categories are specified as the permitted forms of the annotated answer.Source: §2.3.2 · p. 2

2.3.3 Number of Reasoning Steps

• The number of reasoning steps is manually annotated as a proxy for reasoning complexity, though a small fraction of problems allow different step counts depending on the solution.More steps generally indicate a more intricate thought process and potentially more opportunities for errors or loss of problem structure.Source: §2.3.3 · p. 3

2.3.4 Number of Digits

• The digit-count measure D is the maximum digit length among the problem's numbers and its answer, used as a rough proxy for arithmetic complexity.Digits 0–9 are counted; decimal points, percentage signs, and slashes are excluded, and a program computes D automatically.Source: §2.3.4 · p. 3

3.1 Models

• The evaluated Chinese-capable models vary in organization, size, and access mode, with their lineup summarized in Table 2.The models are intended as general-purpose task solvers.Source: §3.1 · p. 4, Table 2 · p. 4

3.2.1 Zero-shot Evaluation

• Problems are presented in their original form for zero-shot evaluation, without few-shot or chain-of-thought prompting.The authors consider this setup a pragmatic assessment of fine-tuned models intended for direct deployment.Source: §3.2.1 · p. 4

3.2.2 Automated Evaluation

• A regular-expression script extracts likely answer numerals and counts a response as correct if any extracted numeral matches the annotated answer.On a manual review of 200 random problems and responses, the method achieved 96% precision and 98% recall.Source: §3.2.2 · p. 5

4.1 Main results

• Accuracy declines as grade rises, and half the models struggle even at grade 1; elementary problems still challenge open-source LLMs.Source: §4.1 · p. 5
• GPT-4 alone exceeds 60% accuracy across all six grades; ChatGPT succeeds in grades 1–4, ChatGLM2-6B in grades 1–2, and the remaining models fail.Source: Table 3 · p. 5, §4.1 · p. 5

4.2 Arithmetic Complexity and Reasoning Complexity

• Accuracy falls as digit count or reasoning steps increase, with reasoning complexity generally having the larger impact.Digit count and reasoning steps serve as approximate measures of arithmetic and reasoning complexity.Source: §4.2 · p. 5

4.3 Robustness

• GPT-4 alone maintains robustness as distractors increase; ChatGPT loses 30% accuracy on problems with just two distractors.The authors report that ChatGPT’s injected-distractor responses can contain erroneous reasoning, while GPT-4 sticks to relevant information.Source: Figure 2 · p. 7, §4.3 · p. 7
• The robustness test uses 60 problems, 10 per grade, each paired with versions containing 1–5 distractors of exactly one number each.Source: §4.3 · p. 7

• Many math datasets are English-language, while Chinese math subsets in broader evaluations range from middle school to college level.The authors describe these datasets as unsuitable for Chinese LLM evaluation or challenging for measuring progress given current LLM capabilities.Source: §5 · p. 7
• Math23K and APE210K (Zhao et al., 2020) cover elementary problems but lack fine-grained grade and reasoning-step annotations.Source: §5 · p. 8

• All LLMs except GPT-4 falter at a certain grade, and GPT-4 alone maintains robustness amid distracting information.Source: §6 · p. 8
• CMATH enables fine-grained evaluation of LLMs on elementary-school math word problems.Source: §6 · p. 8
• The authors anticipate that the study will expose limitations in LLM capabilities and catalyze their ongoing development and improvement.Source: §6 · p. 8