Source-linked AI summary
MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms
Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, Hannaneh Hajishirzi
TL;DR
Math word-problem datasets have been limited by small scale or imprecise annotations, making precise operational supervision difficult. The paper introduces an operation-based representation, the 37k-problem MathQA dataset, and a categorized sequence-to-program model; these models outperform prior AQuA results but remain below human performance.
Problem
Existing math word-problem datasets are often small or lack precise operational annotations across diverse problem types, partly because accurate annotation requires background math knowledge.
Method
The paper constructs MathQA with operation programs and trains encoder-decoder sequence-to-program models that execute predicted programs and match their results to multiple-choice options, using domain categorization.
Results
The category-aware sequence-to-program model outperforms previous state-of-the-art on AQuA despite smaller training data, achieves competitive MathQA results, and remains below human performance.
Takeaways & Limitations
Operation-based formalisms provide extra supervision and human-interpretable solution paths while reducing the influence of statistical bias in datasets such as AQuA.
Takeaways & Limitations
The dataset still includes challenges involving problems beyond current models’ scope, including sequence problems, ambiguous or unsolvable problems, categorization noise, and insufficient textual context.
Abstract
from arXiv · showhide
We introduce a large-scale dataset of math word problems and an interpretable neural math problem solver that learns to map problems to operation programs. Due to annotation challenges, current datasets in this domain have been either relatively small in scale or did not offer precise operational annotations over diverse problem types. We introduce a new representation language to model precise operation programs corresponding to each math problem that aim to improve both the performance and the interpretability of the learned models. Using this representation language, our new dataset, MathQA, significantly enhances the AQuA dataset with fully-specified operational programs. We additionally introduce a neural sequence-to-program model enhanced with automatic problem categorization. Our experiments show improvements over competitive baselines in our MathQA as well as the AQuA dataset. The results are still significantly lower than human performance indicating that the dataset poses new challenges for future research. Our dataset is available at: https://math-qa.github.io/math-QA/
1 Introduction
Math word problems require extracting salient information and translating narratives into precise, executable meanings while using implied quantities, formulas, and domain knowledge. The paper addresses dataset and annotation limitations with an operation-based language, MathQA, and a categorized sequence-to-program model.
- Problem: Math word-problem solving requires extracting salient information from narratives and transforming them into executable meaning representations.The task demands high precision and, for story problems, significant world knowledge.
- Problem: Implied constants such as pi and domain-specific formulas such as the area of a square complicate operation-program construction.
- Contributions: MathQA is a 37k-problem dataset covering multiple math domains with operation programs modeled from AQuA problems.
- Contributions: The paper introduces an operation-based representation language and a neural model that maps problems to operation programs with domain categorization.
- Results: The model outperforms previous state-of-the-art on AQuA despite smaller training data, achieves competitive MathQA results, and remains below human performance.
2 Background and Related Work
Prior math word-problem datasets range from large and diverse but noisy resources to smaller datasets focused on narrower problem types. Existing neural and symbolic approaches motivate formal representations that preserve problem-solving structure while addressing annotation and interpretability challenges.
- Large-Scale Datasets: AQuA provides over 100K multiple-choice GRE- and GMAT-level problems across a wide range of domains.Its scale and diversity support training deep-learning models.
- Large-Scale Datasets: AQuA contains incorrect solutions, brute-force-only problems, and rationales missing many required solution steps.
- Large-Scale Datasets: MathQA aims to preserve AQuA’s challenge while removing noise that hinders learning signals for logical reasoning.
- Additional Datasets: Smaller datasets primarily target algebra word problems and often align equations or equation systems with text.
- Prior Solvers: Earlier systems used pattern matching, statistical modeling, semantic parsing, entity extraction, and equation-tree representations.
- Neural and Interpretable Solvers: Recent neural approaches exploit large-scale datasets, while interpretability remains challenging for neuralized math problem solvers.
3 Representing Math Word Problems
The paper represents each math word problem as a sequence of dependent, domain-aware operations whose arguments may come from the text or earlier steps. This formalism supports correctness, domain awareness, human interpretability, annotation, and neural execution of solution paths.
- Motivation: Math word problems require reasoning about implied actions and relations, including operations such as addition and division that may not be stated explicitly.
- Operation Programs: An operation program consists of sequential dependent operations and arguments that can be executed to solve a word problem.
- Formalism: The representation language expresses a program as n operations, where each operation o_i takes a list of arguments a_i.
- Formalism: The Figure 2 example executes three additions followed by one division: add1(85, 89), add2(174, 80), add3(254, 95), and divide4(349, 4).
- Design Objectives: The language contains 58 operations and targets correctness, domain awareness, and human interpretability.
- Benefits: Sequential formalisms provide annotators with clear solution steps, give neural models a continuous execution path, and enable reconstruction of inferred logical hops.
4 Dataset
MathQA contains 37,200 multiple-choice math word problems paired with aligned operation programs, organized through category-based formalisms. The dataset is constructed with dynamic crowdsourcing, constrained argument selection, quality control, and program validation.
- MathQA contains 37,200 math word problems, multiple-choice options, and aligned operation programs.
- Math problems are categorized into domains, which organize the operation formalisms and prune possible operations during alignment.
- The annotation platform gives workers category-specific operations and valid arguments, then dynamically calculates intermediate results after each submission.
- Valid arguments include problem numbers, category constants, and previous calculations, restricting annotators from introducing noisy or dangling numbers.
- Quality control evaluates workers with test questions and removes prior annotations from trust when accuracy falls below a threshold.
- 94.64% validation accuracy was achieved across categories, with programs accepted when at least two of three validators ranked them valid.
5 Models
The paper develops sequence-to-program neural models that translate math word problems into executable operation programs, including a category-aware extension. The model decodes operations and arguments, uses attention and informed generation, and executes candidate programs to select answers.
- Sequence-to-Program: The encoder-decoder model maps each word problem to feasible operation programs and matches executed program results against multiple-choice options.The predicted program is executed, and the resulting value is compared with answer choices to produce the final output.
- Sequence-to-Program: The base model frames operation-program alignment as neural machine translation between problem text and operation-program sequences.The source is the word problem, while the target vocabulary contains operations and valid arguments from the representation language.
- Sequence-to-Program: Attention-based encoding summarizes the input through weighted encoder states, while the decoder recurrently predicts operators or their arguments.At each timestep, predictions depend on the input and previously generated tokens, producing a complete program sequence.
- Categorized Sequence-to-Program Model: The category-aware extension makes decoder hidden-state computation depend on a deterministically extracted domain category.The category label acts as a hard switch selecting the parameter set used for hidden-state computation.
- Categorized Sequence-to-Program Model: Domain categories are assigned by comparing problem n-gram frequencies against a lexicon of domain-associated n-grams.The selected label is the category with the highest associated n-gram frequency among categories appearing in the problem.
- Inference: At inference, the system executes a beam of decoded programs and chooses a solution using thresholded option matching and minimum distance.This procedure converts generated operation sequences into a selected multiple-choice answer.
6 Experimental Setup
MathQA contains 37k problems and fully annotates a portion of AQuA problems with formal operation programs. The experimental setup addresses AQuA data quality, crowdsourced annotation, model training, and accuracy evaluation on MathQA and AQuA test sets.
- Datasets: MathQA contains 37k problems split randomly into 80% training, 12% development, and 8% test sets.The dataset fully annotates a portion of solvable AQuA problems with formal operation programs.
- Datasets: AQuA includes near-duplicate problems, inconsistent rationales, and examples that current neural frameworks cannot solve.Sequence problems are identified as one class requiring reasoning beyond the models described.
- Annotation: The annotation process uses a documented strategy, with additional annotator information tested but not accepted as a sufficient solution method.The supplied passages indicate that extra information could lead annotators to construct programs without reading the problem.
- Annotation: Expert evaluation found 92% valid annotations and 87% agreement between expert validation and crowdsourcing validation.These figures come from evaluation of the annotation procedure on a collection of 500 problems.
- Annotation: Annotation expansion identifies closely similar AQuA problems and transfers generalized operation-program annotations using a four-word Levenshtein-distance threshold.Numeric values are replaced with generic numbers before expanding annotations to additional problems.
- Model and Training: The models use LSTM encoder-decoder architectures, Adam optimization, 0.001 learning rate, hidden size d = 100, two layers, and beam search.Beam sizes are 200 for AQuA and 100 for MathQA.
- Model and Training: The program vocabulary contains representation-language operations and source-indexed arguments rather than their actual numeric values.Constants, problem numbers, and intermediate calculated numbers are maintained in ordered lists.
- Evaluation: Table 3 evaluates accuracy on MathQA and AQuA test sets, comparing Seq2prog and Seq2prog + cat with baselines.Accuracy is defined by performance on an actual math test.
7 Experimental Results
The experiments show that category-aware sequence-to-program models outperform their base models and baselines, while analysis identifies categorization noise, representation gaps, and difficult reasoning cases. The models remain limited by problem types beyond the current formalism and by insufficient or erroneous textual context.
- Results: The model improves despite proportionally smaller training data and a simpler architecture than the state-of-the-art AQuA model.The authors attribute this result to the formal representation language and the quality of MathQA annotations.
- Error Analysis: The most common errors occur on problems requiring complicated or long chains of mathematical reasoning.One analyzed example requires reasoning beyond a single sentence.
- Error Analysis: Some errors arise because the representation language lacks required operations, such as factorization.The authors propose extending the language to additional mathematical domains, including logic and number factors.
- Error Analysis: Other errors result from noisy categorization or problems with insufficient textual context and erroneous problem statements.A problem containing “m, cm, liter” was assigned to physics instead of geometry.
- Impact of Categorization: The category-aware model outperforms the base model on both AQuA and MathQA, although the gain is relatively small because categorization is used only as a hard decoding constraint.Categorization decisions may also be noisy because domain-specific n-grams can have multiple mathematical interpretations.
- Discussion: The formalism can represent systems of equations, but sequence problems and other multiply interpretable or currently unsolvable problems remain beyond the current models.Problems with multiple interpretations can yield multiple correct solutions.
8 Conclusion
The paper presents an operation-based representation and annotation system for large-scale math word-problem datasets, then evaluates sequence-to-program models trained with that representation. Both base and category-aware models outperform AQuA baselines, while the gap to human performance and uncovered problem types remain open challenges.
- Conclusion: The paper introduces a representation language and annotation system addressing dataset noise and the lack of formal operation-based representations.The system is demonstrated by transforming solvable AQuA problems into operation formalisms.
- Conclusion: Both base and category-aware sequence-to-program models outperform baselines and previous AQuA results when trained on data aligned with the representation language.The representation supplies an additional supervision layer and supports human-interpretable generated operation programs.
- Conclusion: The gap between model and human performance shows that MathQA retains the challenging nature of AQuA problems.Future work will extend the representation language and models to sequence and high-order polynomial problems.