Source-linked AI summary
Translating a Math Word Problem to an Expression Tree
Lei Wang, Yan Wang, Deng Cai, Dongxiang Zhang, Xiaojiang Liu
TL;DR
Math word problems can admit multiple valid equations, making the problem-to-equation mapping non-deterministic and complicating maximum likelihood estimation. The paper normalizes equivalent equations into unique expression trees, evaluates three SEQ2SEQ models, and ensembles their outputs. On Math23K, the combined approach improves accuracy from 60.7% to 68.4%.
Problem
Multiple valid equations can solve the same math word problem, creating a non-deterministic transduction that harms maximum likelihood estimation.
Method
The paper normalizes duplicated equation templates into unique expression trees, compares three SEQ2SEQ models, and selects ensemble outputs by generation probability.
Results
68.4% accuracy on Math23K is achieved with equation normalization and model ensembling, compared with 60.7%.
Takeaways & Limitations
Equation normalization and model ensembling significantly improve the reported performance of SEQ2SEQ methods for math word problem solving.
Takeaways & Limitations
The evaluated problems are linear algebra questions with only one unknown variable.
Abstract
from arXiv · showhide
Sequence-to-sequence (SEQ2SEQ) models have been successfully applied to automatic math word problem solving. Despite its simplicity, a drawback still remains: a math word problem can be correctly solved by more than one equations. This non-deterministic transduction harms the performance of maximum likelihood estimation. In this paper, by considering the uniqueness of expression tree, we propose an equation normalization method to normalize the duplicated equations. Moreover, we analyze the performance of three popular SEQ2SEQ models on the math word problem solving. We find that each model has its own specialty in solving problems, consequently an ensemble model is then proposed to combine their advantages. Experiments on dataset Math23K show that the ensemble model with equation normalization significantly outperforms the previous state-of-the-art methods.
1 Introduction
The paper addresses automatic math word problem solving with SEQ2SEQ models, focusing on duplicated equations that create a non-deterministic output space. It compares three SEQ2SEQ models and proposes an ensemble with equation normalization, improving Math23K accuracy from 60.7% to 68.4%.
- Motivation: Equation duplication lets one math word problem map to multiple valid equations, creating a non-deterministic output space that harms data-driven methods.Examples include different operand orders and equivalent bracket placements.
- Contribution: The paper proposes equation normalization based on unique expression trees to address duplicated equation templates.
- Contribution: The study compares three state-of-the-art SEQ2SEQ models for automatic math word problem solving.
- Contribution: Each model solves different math word problems correctly, motivating an ensemble model that combines their strengths.
- Contribution: 68.4% accuracy is achieved on Math23K, up from 60.7%, using equation normalization and model ensembling.
2 SEQ2SEQ Framework
The SEQ2SEQ framework converts significant numbers into ordered number tokens, then generates an equation template from the problem text and solves it numerically.
- Number Mapping: The number mapping stage identifies significant numbers and maps them to ordered tokens {n1, ..., nm}.The paper uses the significant number identification module from Wang et al. (2017).
- Template Generation: SEQ2SEQ models take the problem text as the source sequence and an equation template as the target sequence.
- Training and Decoding: During training, the model maximizes the conditional probability of the target template given the problem, decomposed into tokenwise probabilities.
- Training and Decoding: During decoding, beam search finds a likely equation template, whose number tokens are replaced with actual numbers before a math solver calculates the solution.
3 Equation Normalization
Equation normalization removes order and bracket duplications by mapping equivalent templates to canonical forms, including a unique expression-tree representation. The normalized tree is serialized for SEQ2SEQ prediction.
- Duplication Problem: Equation templates create a non-deterministic transduction because equivalent solutions can differ in operand order or bracket placement.
- Order Normalization: Unequal-length duplicated templates are normalized to the shorter template.For example, redundant repeated tokens are removed when a shorter equivalent template exists.
- Order Normalization: Number tokens are ordered as closely as possible to their order in the number mapping.The canonical example normalizes three order variants to n1 + n2 + n3.
- Expression-Tree Normalization: Bracket duplication is resolved by converting equation templates into expression trees with binary operator nodes and number-token leaves.
- SEQ2SEQ Representation: After normalization, models use the postorder traversal of a unique expression tree as the target sequence.
4 Models
The paper evaluates BiLSTM, ConvS2S, and Transformer architectures and combines their outputs with an ensemble based on generation probability. Each architecture contributes a distinct sequence-modeling design.
- Models: The evaluated models are BiLSTM, Convolutional SEQ2SEQ, and Transformer, combined with a simple ensemble method.
- BiLSTM: BiLSTM encodes each token using forward and backward LSTMs and decodes with global attention.
- ConvS2S: ConvS2S replaces recurrent networks with convolutional encoder and decoder structures using gated linear units.
- Transformer: Transformer uses stacked layers containing multi-head self-attention and positionwise feed-forward networks without convolutional or recurrent architecture.
- Ensemble: The ensemble selects the model output with the highest generation probability.
5 Experiment
Experiments on Math23K evaluate equation normalization, several SEQ2SEQ architectures, and their ensemble, showing gains in accuracy and model-specific strengths.
- Math23K contains 23,162 labeled math word problems, all framed as linear algebra questions with one unknown variable.
- The experiments compare DNS and DNS-Hybrid baselines with Bi-LSTM, ConvS2S, Transformer, and an ensemble model.The test set is Math23K’s testing set, with 1,000 training problems randomly reserved for validation.
- Equation normalization increases different models’ accuracy by 2.7% to 7.1%.
- The ensemble model adds an additional 1.7% accuracy increase by combining SEQ2SEQ models.
- For Bi-LSTM, accuracies for SE, OE, and EB are 63.1%, 63.7%, and 65.3%, respectively.These ablations separate the contributions of three equation-normalization techniques.
- The case study reports distinct model capabilities: Transformer handles complex inference, ConvS2S captures number-token context, and Bi-LSTM and ConvS2S rarely generate mathematically incorrect templates.The Transformer’s occasional incorrect templates are attributed to insufficient training data for multi-head self-attention structures.
6 Conclusion
The paper introduces equation normalization to map duplicate equation templates to unique expression trees, evaluates multiple SEQ2SEQ models, and combines them in an ensemble.
- The proposed equation normalization method maps duplicated equation templates to an expression tree.
- The paper tests different SEQ2SEQ models for math word problem solving and proposes an ensemble model for higher performance.
- Experimental results show that equation normalization and model ensembling significantly improve performance over state-of-the-art methods.