Source-linked AI summary
A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers
Shen-Yun Miao, Chao-Chun Liang, Keh-Yih Su
TL;DR
Existing MWP corpora provide limited diversity in language patterns and problem types, weakening their coverage of solver capability. The paper constructs ASDiv, adds annotations and a lexicon diversity metric, and reports that diverse evaluation better reflects solver performance, which remains far below human performance.
Problem
Existing MWP corpora are limited in language-pattern or problem-type diversity, and some lack reliable difficulty information.
Method
The paper constructs ASDiv with diverse language patterns, broad elementary-school problem-type coverage, consistent equations and answers, problem-type and grade-level annotations, and a lexicon usage diversity metric.
Results
ASDiv contains no identical MWPs, while identical-MWP rates reach 85% in MathQA, 28% in IL, 22% in AI2, and 20% in ALGES; solver accuracy on ASDiv is around 36%.
Takeaways & Limitations
The authors conclude that low-diversity corpora exaggerate state-of-the-art performance, whereas ASDiv is more suitable for assessing real MWP-solver performance.
Takeaways & Limitations
ASDiv is designed to cover most problem types found in primary school and approximate real human tests, rather than cover all possible MWP domains.
Abstract
from arXiv · showhide
We present ASDiv (Academia Sinica Diverse MWP Dataset), a diverse (in terms of both language patterns and problem types) English math word problem (MWP) corpus for evaluating the capability of various MWP solvers. Existing MWP corpora for studying AI progress remain limited either in language usage patterns or in problem types. We thus present a new English MWP corpus with 2,305 MWPs that cover more text patterns and most problem types taught in elementary school. Each MWP is annotated with its problem type and grade level (for indicating the level of difficulty). Furthermore, we propose a metric to measure the lexicon usage diversity of a given MWP corpus, and demonstrate that ASDiv is more diverse than existing corpora. Experiments show that our proposed corpus reflects the true capability of MWP solvers more faithfully.
1 Introduction
Existing MWP corpora lack either broad problem-type coverage, diverse language patterns, or reliable difficulty information, which can inflate solver evaluations. ASDiv addresses these gaps with a diverse, wide-coverage corpus and a lexicon diversity metric.
- Motivation: Existing MWP corpora are limited in problem-type or lexicon-pattern diversity, or lack difficulty annotations.Mixed-type corpora may also contain inconsistent annotated answers or formulas.
- Motivation: Low-diversity corpora can produce over-optimistic solver results because similar training problems enable template-based answering.Removing highly similar MWPs has been reported to lower performance, making diversity important for judging true capability.
- ASDiv: ASDiv is a diverse, wide-coverage MWP corpus with consistent equations and answers, plus problem-type and grade-level annotations.The annotations support system evaluation and indicate problem difficulty.
- ASDiv: The paper proposes a lexicon usage diversity metric and uses it to evaluate MWP corpora.The metric is intended to assess whether solvers understand problem text rather than match similar patterns.
- ASDiv: ASDiv is designed to assess solver capability more faithfully than low-diversity corpora and to approximate real human tests.The authors report that low-diversity corpora exaggerate state-of-the-art performance relative to human performance.
2 Problem Type
ASDiv organizes elementary-school MWPs into problem types defined by their mathematical operation patterns. Its categories span basic arithmetic, aggregative operations, and problems requiring additional domain knowledge.
- Problem-type framework: A problem type identifies a crucial mathematical operation pattern used to solve an MWP.MWPs sharing a type can exhibit similar language, logic representations, or inferences.
- Problem-type framework: ASDiv annotates each MWP with one of 24 common problem types taught in elementary school.The types are grouped into three main categories according to the operations involved.
- Basic arithmetic operations: Basic arithmetic types include addition, subtraction, difference, multiplication, three divisions, sum, surplus, number-operation, three TVQ types, and multi-step.TVQ represents a variable updated sequentially across events, while multi-step problems require multiple arithmetic operations.
- Aggregative operations: Aggregative operations include comparison, set-operation, ratio, number-pattern, Algebra-1, and Algebra-2.Number-pattern problems infer integer sequences; Algebra-1 and Algebra-2 contain one and two unknown variables, respectively.
- Additional domain knowledge required: Additional-domain-knowledge types include greatest common divisor, least common multiple, geometry, and unit conversion.Geometry requires facts such as area = length * width, while UnitTrans requires metric conversion.
3 ASDiv Math Word Problem Corpus
ASDiv is designed as a diverse, wide-coverage MWP corpus with consistent annotations for evaluating and developing solvers. Its diversity metric exposes repeated language patterns in existing corpora and supports more faithful assessment of solver capability.
- Corpus design: ASDiv follows three design goals: diverse lexicon usage, broad coverage of primary-school problem types, and sufficient annotations for evaluation and system development.The corpus guidelines emphasize reducing mechanical pattern matching, approximating real human tests, and supporting both assessment and development.
- Corpus diversity metrics: The proposed lexicon diversity metric ranges from 0 to 1, with higher values indicating greater corpus diversity.The metric uses tokenization, part-of-speech tagging, lemmatization, stop-word removal, and normalization of names and quantities.
- Corpus diversity metrics: LD measures lexicon diversity between one MWP and other MWPs sharing its problem type, while mean LD gives corpus lexicon diversity (CLD).BLEU is measured bidirectionally with n-grams up to n=4 to identify repeated lexicon and phrase usage.
- Existing-corpus analysis: 85% of MathQA MWPs have LD=0, and a subset contains 105 problems sharing the same sentence pattern.The skewed distribution reflects repeated sentence patterns among members of some MathQA subsets.
- Annotation quality: 27% of MathQA arithmetic problems have annotated formulas that do not match their labeled answers.Among 30 inspected inconsistencies, 67% involved incorrect formulas, 23% problematic descriptions, and 10% valueless answers.
- Corpus construction: ASDiv contains 2,305 MWPs collected from 28 websites, with problems pruned or revised for high similarity and manually annotated for type, equation, answer, and grade level.A program automatically verified consistency between equations and answers; grade level indicates difficulty and supports evaluation.
4 Experiments
The experiments compare established MWP solvers on consistent and diverse arithmetic subsets, then evaluate them across ASDiv. Results indicate that corpus diversity tracks solver difficulty and that grade level reflects problem difficulty.
- Experimental setup: The evaluation uses three SOTA solvers: LCA++, UnitDep, and GTS.The first two are statistical models, while GTS uses a neural network with a goal-driven tree-structured approach.
- Datasets: 4,117 MathQA problems were filtered to arithmetic-only, non-external-constant, non-reused-operand problems, leaving 3,000 consistent MathQA-C problems.ASDiv-A contains 1,218 arithmetic problems satisfying the same constraints.
- Dataset diversity: MathQA-C and ASDiv-A have CLDs of 0.08 and 0.50, respectively, while 82% of MathQA-C problems have LD = 0.These measurements contrast the low diversity of the consistent MathQA subset with the higher diversity of ASDiv-A.
- Results: Around 36% overall performance on ASDiv shows that current SOTA systems remain below human performance, while lower diversity is associated with higher performance.Grades 5 and 6 are significantly harder than grades 1 to 4, making grade level useful for indicating MWP difficulty.
5 Conclusion and Future Work
The paper presents ASDiv as a diverse, wide-coverage MWP corpus with consistent annotations and a lexicon-usage diversity metric. Experiments support using corpus diversity and grade level to assess solver performance and problem difficulty.
- Conclusion: ASDiv is highly diverse in lexicon usage and covers most problem types taught in elementary school.The corpus contains annotations for problem type, equation, and grade level.
- Conclusion: The proposed metric measures lexicon-usage diversity and supports comparing ASDiv with widely used MWP corpora.The paper reports that ASDiv is more suitable for assessing real MWP-solver performance.
- Conclusion: Low-diversity corpora exaggerate SOTA performance, while grade level is useful for indicating MWP difficulty.The conclusion states that current systems remain far behind human-level performance on the proposed corpus.
Appendix A: Examples of a few Selected Problem Types
The appendix illustrates selected problem types through examples organized across three broad categories: arithmetic, aggregative, and problems requiring additional domain knowledge.
- Appendix A: Table 5 provides examples of selected problem types in three categories.The categories are basic arithmetic operations, aggregative operations, and additional domain knowledge required.
- Appendix A: The selected examples span arithmetic operations, aggregation, and cases requiring knowledge beyond basic arithmetic.These categories reflect different operation or knowledge demands in MWP solving.
- Appendix A: The table is intended to show representative examples rather than the full set of problem types.Its caption identifies the contents as examples of selected problem types.
Appendix B: Problematic MWPs in MathQA
The appendix documents problematic MathQA annotations and repetitive wording patterns. It includes inconsistent-answer examples, an absent proper operation, and many problems sharing one sentence pattern.
- Appendix B: Table 6 presents examples of inconsistent MWPs in MathQA.The table is specifically described as showing inconsistent answers.
- Appendix B: 105 MathQA problems in one subset share the same sentence pattern after normalization.The repeated pattern concerns a train crossing a pole and asks for the train's length.
- Appendix B: MathQA contains cases for which its operation set has no proper operation.The appendix repeats this issue in two listed examples.
Appendix C: Additional Experiments for Corpus Diversity Metrics
The appendix evaluates corpus diversity within datasets and between training and test sets using lexicon- and syntactic-pattern metrics. ASDiv shows greater diversity than several existing corpora, while MathQA’s split similarity may exaggerate solver performance.
- Syntactic pattern diversity: Syntactic-pattern diversity compares POS-tag sequences among MWPs sharing the same problem type using BLEU-based similarity.For each problem, the metric aggregates similarity against the other problems in its set.
- Syntactic pattern diversity: ASDiv has only 4% identical syntactic patterns, compared with 87% for MathQA, 54% for IL, 46% for AI2, and 33% for ALGES.The percentages denote MWPs with SD_i = 0 in each dataset.
- Train-test diversity: Train-test diversity metrics measure how lexicon and syntactic patterns in test problems differ from those in training problems.A low diversity index indicates that a test problem may be solvable through a training problem with similar patterns.
- Test-set diversity: Within-test-set CLD is also necessary because duplicating a high-diversity training problem can produce a misleadingly high train-test CLD without a meaningful test set.The appendix therefore compares within-test-set CLDs; MathQA is 0.27 versus means of 0.57 for ASDiv-A and ASDiv.
- Train-test diversity: 48% of ASDiv-A MWPs have lexicon diversity indices at least 0.5, versus 35% for AI2 and 7% for MathQA-C.The corresponding proportions are 34% for IL, 29% for ALGES, 23% for KAZB, and 23% for AllArith and DRAW.
- Test-set diversity: MathQA has the highest train-test CLD at 0.85, while ASDiv-A, IL, AllArith, and AI2 have CLDs of 0.52, 0.44, 0.42, and 0.42, respectively.MathQA’s official split CLD is also far above its whole-corpus CLD of 0.05.