Source-linked AI summary
Generate & Rank: A Multi-task Framework for Math Word Problems
Jianhao Shen, Yichun Yin, Lin Li, Lifeng Shang, Xin Jiang, Ming Zhang, Qun Liu
TL;DR
Math word problem generation can produce minor expression mistakes that change answers, while generation objectives do not explicitly distinguish correct from incorrect expressions. Generate & Rank adds a jointly trained ranker, using tree-based disturbance and online updates to construct candidates, and consistently outperforms baselines across the evaluated datasets, including a 7% Math23K improvement.
Problem
MWP generation objectives do not explicitly distinguish correct expressions from expressions containing minor mistakes, although such mistakes can change the answer and performance degrades on longer expressions.
Method
Generate & Rank jointly trains a BART-based generator and ranker, using tree-based disturbance and online-updated candidate expressions to train ranking.
Results
Generate & Rank consistently outperforms baselines across the evaluated datasets, with a reported 7% improvement on Math23K.
Takeaways & Limitations
The framework enables the model to learn from generated mistakes and select correct expressions from candidates.
Abstract
from arXiv · showhide
Math word problem (MWP) is a challenging and critical task in natural language processing. Many recent studies formalize MWP as a generation task and have adopted sequence-to-sequence models to transform problem descriptions to mathematical expressions. However, mathematical expressions are prone to minor mistakes while the generation objective does not explicitly handle such mistakes. To address this limitation, we devise a new ranking task for MWP and propose Generate & Rank, a multi-task framework based on a generative pre-trained language model. By joint training with generation and ranking, the model learns from its own mistakes and is able to distinguish between correct and incorrect expressions. Meanwhile, we perform tree-based disturbance specially designed for MWP and an online update to boost the ranker. We demonstrate the effectiveness of our proposed method on the benchmark and the results show that our method consistently outperforms baselines in all datasets. Particularly, in the classical Math23k, our method is 7% (78.4% $\rightarrow$ 85.4%) higher than the state-of-the-art.
1 Introduction
Math word problem solving is commonly treated as expression generation, but minor expression errors can change answers and generation models struggle with longer expressions. Generate & Rank addresses this gap by jointly training generation and ranking, using disturbed and newly generated candidates to improve selection.
- Motivation: MWP studies commonly map problem texts to mathematical expressions with sequence-to-sequence generation models.The problem text is the source sequence and the mathematical expression is the target sequence.
- Motivation: Minor mathematical-expression mistakes can change the whole semantic, while generation likelihood does not explicitly teach models to distinguish correct from slightly incorrect expressions.Generation performance also degrades rapidly as expressions become longer.
- Method: Generate & Rank adds a ranker that distinguishes correct from incorrect candidate expressions alongside a generator.The generator produces candidates, while the ranker selects among them; both are jointly trained with generation and ranking losses.
- Method: Tree-based disturbance and online candidate updates provide informative ranking examples, including expressions produced by the generator and newly generated candidates.The expression bank is updated during training so the model can learn from its own mistakes and new examples.
- Results: 7% improvement is reported on Math23K, and experiments show consistent outperformance over state-of-the-art models and baselines.The study also reports that joint training improves the generator and ranker, while candidate construction and online updating are important to ranker performance.
2 Preliminaries
MWPs describe quantities and relationships before asking for an unknown quantity, whose solution is represented as a number-mapped mathematical expression. The paper uses BART's Transformer encoder-decoder foundation and its denoising pre-training, while introducing the Generate & Rank framework for MWP solving.
- 2.1 Math Word Problem: A math word problem describes a partial quantitative state, updates or relationships, and a question about an unknown quantity.Its solution is a mathematical expression containing operators and numbers.
- 2.1 Math Word Problem: Problem and solution numbers are mapped to ordered special tokens such as NUM#i, so solving focuses on expression structure rather than specific values.Table 1 illustrates the number-mapped problem representation.
- 2.2 BART: BART is a pre-trained Transformer encoder-decoder model that can be fine-tuned for sequence classification and generation.The encoder uses bidirectional self-attention, while the decoder attends to encoder outputs and previous decoder tokens.
- 2.2 BART: BART pre-training reconstructs original documents from corrupted inputs using span masking and sentence shuffling.Its objective minimizes cross-entropy between decoder generation probabilities and the original document.
- Generate & Rank: Generate & Rank shares BART between a generator and ranker, jointly training expression generation and candidate selection.An expression bank supplies ranking examples, connecting the BART foundation to the paper's multi-task framework.
3 Methodology
Generate & Rank combines BART-based generation with ranking to select correct mathematical expressions from candidates. It jointly trains both tasks using model-generated and tree-disturbed expressions, with online expression-bank updates.
- Multi-task Training: Generate & Rank uses a shared BART encoder-decoder with a generator for solution expressions and a ranker for selecting correct candidates.The two modules are jointly trained with generation and ranking objectives.
- Multi-task Training: The generation task maps problem-token sequences to solution-expression sequences using BART fine-tuning and autoregressive decoding.Problem text is encoded as P, while the target expression is S; start and end tokens mark decoding boundaries.
- Multi-task Training: The ranking task classifies problem-expression pairs and selects the candidate with the highest score as the final solution.An MLP uses the final decoder-token representation, which attends to the problem text and candidate expression.
- Expression Bank: Online updates regenerate the expression bank during iterative joint training, while the training algorithm fine-tunes BART, constructs samples, and optimizes the joint loss.The procedure repeatedly refreshes candidates and trains the generator and ranker so the modules can enhance each other.
- Expression Bank: The expression bank combines beam-search candidates with tree-based disturbances, labeling expressions by whether their numerical results match the ground truth.Tree-based disturbance transforms ground-truth expressions into ASTs and modifies nodes or substructures; four disturbance types are designed for this process.
- Inference: At inference, beam search generates top-K expressions and the ranker selects the candidate with the highest score.This two-stage process first generates candidate solutions and then ranks all generated expressions.
4 Experiment
Experiments on Math23K and MAWPS evaluate solution accuracy, ablations, expression-bank strategies, expression complexity, and problem domains. Generate & Rank consistently improves over baselines, with joint training, online updating, and complementary candidate construction contributing to performance.
- Experimental Setup: Experiments use Math23K and MAWPS, evaluating solution accuracy where an expression is correct if it yields the ground-truth number.Math23K results are reported on both the public test set and 5-fold cross-validation; MAWPS uses 5-fold cross-validation.
- Results: Generate & Rank outperforms mBART baselines by more than 4% in every evaluation setting and previous best models by 7% on Math23K† and 7.4% on Math23K‡.Table 2 reports solution accuracy on MAWPS and Math23K, with † denoting test-set results and ‡ denoting 5-fold cross-validation.
- Results: 99.9% of expressions generated by mBART are valid, addressing concerns that sequence decoding may produce invalid expressions.The validity check covers all expressions generated by mBART.
- Effect of Joint Training: Joint training improves performance by 2.2% over two-stage training and improves the generator by 2.6% over mBART trained alone.The ablation compares separate generator-ranker training with joint training and evaluates the generator without the ranker.
- Expression Bank Strategy: Tree-based disturbance combined with model-based generation outperforms either random sampling or model-based generation alone, while online updating improves performance.Tree-based disturbance supplies informative examples not covered by the generator, whereas omitting online updating causes a performance drop.
- Expression Bank Size: With model-based generation, ranker performance is best at expression-bank size 20, whereas tree-based disturbance performs best at size 10.The results suggest that expression banks should avoid being either too small or too large; larger banks can introduce low-quality expressions.
- Model Analysis: Generate & Rank shows larger improvements over fine-tuned mBART on longer expressions, while accuracy generally decreases as the number of operations increases.The analysis excludes six-operation expressions because they have too few examples and high variance.
- Model Analysis: Fine-tuned mBART is limited on geometry problems, while Generate & Rank mainly improves those domains requiring external mathematical knowledge.Domains are defined by keyword lists, and domain proportions do not sum to 100% because some problems belong to no specified domain.
5 Related Work
Research on math word problems progressed from manually designed rules and templates to statistical parsing, two-stage scoring, and deep learning methods. Pre-trained language models form a broader sequence-to-sequence modeling family within this evolution.
- Rule-based methods: Rule-based methods rely heavily on manual design and solve only a limited scope of problems.
- Parsing-based methods: Parsing-based methods use statistical techniques to identify entities, quantities, and operators from problem text.
- Two-stage methods: Two-stage methods first obtain expression templates, then map numbers to template slots and use scoring functions to guide candidate selection.
- Deep learning methods: Deep learning methods became prevailing approaches, using recurrent models, equation normalization, attention, tree structures, graph information, and multiple encoders or decoders.
- Pre-trained language models: Pre-trained language models use Transformer-based architectures and self-supervised training, including encoder-only, decoder-only, and encoder-decoder designs.
6 Conclusion and Future Work
The paper proposes Generate & Rank, a multi-task framework combining a generator and ranker through joint training for math word problems. Tree-based disturbance and online updating further improve performance, while the framework is positioned for future use beyond this task.
- Conclusion: Generate & Rank combines a generator and ranker that enhance each other through joint training.
- Conclusion: Tree-based disturbance and online updating further improve the framework’s performance.
- Conclusion: The model consistently outperforms baselines across all evaluated datasets.
- Future Work: Future work will explore the generation-and-ranking framework for summarization and translation.