Source-linked AI summary
Reverse Operation based Data Augmentation for Solving Math Word Problems
Qianying Liu, Wenyu Guan, Sujian Li, Fei Cheng, Daisuke Kawahara, Sadao Kurohashi
TL;DR
Math word problem solvers face a shortage of high-quality training data, while paraphrase augmentation may damage essential information and offers no new mathematical reasoning. RODA generates new problems by reversing mathematical logic, and its evaluation reports state-of-the-art Math23K performance and stronger results than back translation on Math23K, with comparable performance on AllArith.
Problem
MWP solvers have reached a performance bottleneck because available training data is insufficient, while paraphrase augmentation risks losing key information and adds no new mathematical reasoning.
Source: §I · p. 1Method
RODA mimics human double-checking by reversing an original problem’s mathematical logic to generate new high-quality problems and knowledge points.
Source: §I · pp. 1–2Results
RODA achieves state-of-the-art performance on Math23K, significantly outperforms back translation on Math23K, and achieves comparable performance on AllArith.
Source: §V · p. 9Takeaways & Limitations
Abstract
from arXiv · showhide
Automatically solving math word problems is a critical task in the field of natural language processing. Recent models have reached their performance bottleneck and require more high-quality data for training. We propose a novel data augmentation method that reverses the mathematical logic of math word problems to produce new high-quality math problems and introduce new knowledge points that can benefit learning the mathematical reasoning logic. We apply the augmented data on two SOTA math word problem solving models and compare our results with a strong data augmentation baseline. Experimental results show the effectiveness of our approach. We release our code and data at https://github.com/yiyunya/RODA.
I. INTRODUCTION
Math word problem solvers need more high-quality training data, but paraphrase-based augmentation risks losing essential information and adds no new mathematical reasoning. RODA reverses problems’ mathematical logic to create new problems, and the paper reports state-of-the-art results on Math23K.
I. INTRODUCTION
- Existing MWP solvers have reached a performance bottleneck amid limited training data, making data augmentation a key consideration.Source: §I · p. 1
- Paraphrase methods can omit critical details that make a problem unsolvable and add lexical variation without new mathematical reasoning.Source: §I · p. 1
- RODA mimics human double-checking by reversing a problem’s mathematical logic to generate a new problem with a different unknown.Source: §I · pp. 1–2
- The authors describe the reverse operation as simple and reliable, able to preserve key information and introduce knowledge points for learning reasoning logic.Source: §I · p. 2
- The paper reports that RODA-augmented data achieves state-of-the-art performance on Math23K.Source: §I · p. 2
II. RELATED WORK
Prior MWP work spans rule-based, parsing, and neural approaches, while datasets differ in scale, language, and answer format. The paper focuses on equation-output datasets, including Math23K and AllArith.
1) Models:
- MWP systems progressed from predefined rules to semantic parsing and end-to-end prediction, including seq2seq equation generation and tree-structured models.Source: 1) Models · p. 2
2) Datasets:
- Math23K contains 23,162 Chinese elementary-school problems with equation solutions, while Dolphin18K has 18,460 English problems, including multi-unknown cases, typos, and wrong solutions.Source: 2) Datasets · p. 2
- The study excludes Dolphin18K because RODA targets high-quality source data, and it does not apply to datasets whose answers are semantic parses or descriptive text rather than equations.Source: 2) Datasets · pp. 2–3
- Experiments use the large, complex Chinese Math23K and the small, simple English AllArith to assess the method across datasets and languages.Source: 2) Datasets · p. 3
B. Data Augmentation
Prior paraphrase-based augmentation can alter essential MWP information without supplying new reasoning knowledge; RODA instead reverses problems using mathematical logic. Its methodology prioritizes data quality and filters numbers that cannot be safely reversed.
B. Data Augmentation
- Paraphrase methods risk changing key information in MWPs and provide lexical variation rather than new reasoning knowledge.Source: B. Data Augmentation · p. 3
- RODA uses reversed operations on complex mathematical computation, unlike prior task-specific augmentation based on one or two simple logic steps.Source: B. Data Augmentation · p. 3
III. METHODOLOGY
- RODA augments a high-quality MWP dataset with accurate new problems, and the number of generated questions correlates linearly with the number of known variables.Source: §III · p. 3
A. Reversion based Data Augmentation
- The reversion pipeline has three stages—number identification, problem transformation, and equation generation—and prioritizes quality over quantity using empirical rules.Source: A. Reversion based Data Augmentation · p. 3
Number Identification:
- An LSTM classifier identifies significant numbers; filtering and exact matching then align eligible numbers in the problem text and equation.Source: Number Identification · p. 3
- The method excludes duplicated problem or equation numbers, numbers in power operations, and constant terms because they cannot be reliably reversed within the stated approach.Source: Number Identification · p. 3
Problem Transformation:
RODA transforms each original question into a declarative statement containing its answer, then asks about a reversible candidate number. It derives a corresponding equation by recursively reversing the original equation and normalizes equations to reduce stylistic mismatch.
Problem Transformation:
- The original question becomes a declarative sentence with its answer, while a candidate number becomes the focus of a new question.The method replaces the original question's interrogative pronoun with its answer, then converts a sentence containing the candidate number into a question.Source: Problem Transformation · p. 4
- The transformed problem replaces the original question and answer sentences with their converted versions, appended to the text.For English, the interrogative-pronoun candidates follow manually encoded rules from Heilman et al..Source: Problem Transformation · p. 4
Equation Generation:
- Equation conversion builds a quasi-binary tree, moves the variable-free child across the equality using reversion rules, and recursively repeats until only the new variable remains on the right.The variable-containing child is recursively decomposed; numbers and operators end up on the left side.Source: Equation Generation · pp. 4–5
- Generated equations are normalized by shortening convertible expressions and ordering numbers to match their occurrence in the problem, reducing prediction-target domain shift.The method also uses SymPy simplification to broaden normalization beyond limited rule-based cases.Source: Equation Generation · p. 5
B. MWP Solving Model
The paper evaluates RODA with two sequence-to-sequence solvers and a Transformer classification model. The models predict equation outputs or templates, and augmented examples are mixed with the original training data.
B. MWP Solving Model
- RODA is evaluated with the prefix seq2seq baseline, the GTS tree-structured solver, and a Transformer-based equation-template classifier.The authors describe the augmentation method as combinable with any preferred neural model.Source: B. MWP Solving Model · p. 5
- The prefix model encodes input tokens with a BiLSTM and uses an attention-equipped LSTM decoder to generate equation tokens.The decoder's attention-aware representation is passed to an MLP to produce each output token.Source: B. MWP Solving Model · p. 5
- GTS extends the prefix baseline with bottom-up subtree representations that inform equation decoding.A recursive neural network computes subtree representations using a gate mechanism.Source: B. MWP Solving Model · p. 5
- The seq2seq models predict equation templates rather than raw numbers to address the large numeric vocabulary and its out-of-vocabulary problem.Numbers in each problem are represented by their order of appearance, with extra constants such as π and 1 added to the decoder vocabulary.Source: B. MWP Solving Model · p. 5
- The classification model encodes question text with a Transformer and classifies its [CLS] representation into an equation template; training mixes shuffled augmented and original examples.The authors note that RODA is not limited to the three evaluated models.Source: B. MWP Solving Model · p. 5
A. Experiment Setup
Experiments use Math23K and three model configurations, with RODA applied to the training set. The augmented set also increases coverage of training-set equation templates on the development set.
A. Experiment Setup
- Math23K contains 23,162 Chinese elementary-level problems; the train-test split uses 21,162 training, 1,000 validation, and 1,000 test examples.GTS is evaluated both with this split and with five-fold cross-validation; the other models use the test set.Source: A. Experiment Setup · p. 6
- The Transformer classifier uses a 12-layer, 768-hidden-size model with 12 attention heads and 5,693 labels, while the seq2seq models use 128-dimensional embeddings.The prefix model uses 512-dimensional encoder and 1,024-dimensional decoder states; GTS uses 512-dimensional states for both.Source: A. Experiment Setup · p. 6
- RODA generated 47,318 new problem-equation pairs from Math23K's 21,162 training problems, more than doubling the original training-set size.The process filtered 1,490 problems consisting only of numbers and operators and excluded 7,399 numbers judged difficult to reverse.Source: A. Experiment Setup · p. 6
- RODA reduces the development-set rate of uncovered templates by 21.5%.The comparison is between templates from the original and augmented training sets.Source: A. Experiment Setup · p. 6, Table IV · p. 6
B. MWP Solving Results
RODA improves Math23K solving results when added to generation and classification models, and the analysis compares augmentation amounts and back-translation. The authors also report a limitation when training uses augmented data alone.
B. MWP Solving Results
- RODA improves MWP-solving accuracy by 8.8 points over the Transformer baseline classification model, and boosts Prefix and GTS by about 2.7 and 2.3 points; under 5-fold cross-validation, it improves GTS by 1.7 points.Table V identifies the metric as Math Word Problem Solving Accuracy on Math23K and specifies that other models are tested on the test set.Source: B. MWP Solving Results · p. 7, Table V · p. 7
C. Analysis of Data Augmentation
- Training only on augmented data yields 50% accuracy, below the 68% from only the original Math23K training data.The authors suggest possible causes including filtered question types, lexical similarity that may introduce noise, and a mismatch between generated examples and the original distribution.Source: C. Analysis of Data Augmentation · p. 7
- As augmented-data volume increases, performance rises stably; the model performs similarly with a 1.5 proportion and the full augmented set.The authors use the full augmented set for reported results because performance did not decline as the amount increased.Source: C. Analysis of Data Augmentation · p. 7, Table VI · p. 7
- At the same augmentation proportion, back-translation has a significant performance gap with RODA, which the authors attribute to paraphrase noise and the lack of new mathematical knowledge points.The comparison is described as controlling for the size of the augmented data.Source: C. Analysis of Data Augmentation · p. 7, Table VII · p. 7
D. Human Evaluation
Human evaluation compares original, back-translated, and RODA data for coherence and equation–text correctness. The paper also illustrates both a successful multi-question augmentation and a discourse-related failure.
D. Human Evaluation
- RODA data has only a small quality gap from original data and scores higher than back-translation in both coherence and correctness.The evaluation samples 100 problems from each dataset, with two annotators; coherence is rated 1–5 and correctness is binary.Source: D. Human Evaluation · p. 8, Table X · p. 8
- One complex source problem yields four new question–answer pairs, each corresponding to a new mathematical knowledge point.The example involves shared travel time among A, B, and a dog, enabling questions about five variables.Source: D. Human Evaluation · p. 8
- Sentence reordering can change apple coreference, making a reversed question unnatural or incorrect because the question-generation module does not model discourse dependencies.The authors identify an additional discourse-analysis module as a possible way to reduce this error.Source: D. Human Evaluation · p. 9
- In the comparison example, back-translation changes place terms and omits a key directional detail, while RODA forms a natural reversed question with a new knowledge point.Source: D. Human Evaluation · p. 9
E. A Study on English Dataset
The paper evaluates RODA with GTS on AllArith, an English math word-problem dataset, and reports improvement when augmented data is added. Its conclusion contrasts stronger results on Math23K with comparable performance to back-translation on AllArith.
E. A Study on English Dataset
- On AllArith, adding RODA data improves performance, while back-translation also improves performance but slightly less.The experiment uses GTS and 5-cross validation; the dataset contains 831 problems, and RODA generates 715 new examples.Source: E. A Study on English Dataset · p. 9, Table XI · p. 9, Table XII · p. 9
- The authors suggest RODA's smaller advantage on AllArith may reflect fewer variables per problem and lower mathematical reasoning complexity than Math23K.They report averages of 2.35 variables for AllArith and 2.77 for Math23K.Source: E. A Study on English Dataset · p. 9
V. CONCLUSION
- The authors report that RODA significantly outperforms back-translation on large, complex Math23K, but achieves comparable performance on small, simple AllArith.Source: §V · p. 9
APPENDIX A REMOVING NEGATIVE TERMS
The appendix describes a normalization procedure for equations beginning with negative terms and a constraint on interrogative-pronoun detection in discourse.
APPENDIX A REMOVING NEGATIVE TERMS
- The normalization procedure moves the first addition term to the front when an equation begins with a negative term, changing its written form but not its mathematical value.This addresses the stated issue that a leading negative can prevent formation of a binary abstract syntax tree.Source: Appendix A · p. 9, Appendix B · p. 10
APPENDIX B INTERROGATIVE PRONOUNS
- Interrogative pronouns are detected only in the sentence's last discourse unit because they can have other meanings in declarative discourse units.The appendix lists the pronouns used for question conversion.Source: Appendix B · p. 10