PaperlayerView exact revision on arXiv

Source-linked AI summary

Are NLP Models really able to Solve Simple Math Word Problems?

Arkil Patel, Satwik Bhattamishra, Navin Goyal

arXiv:2103.07191v2cs.CL

TL;DR

Elementary one-unknown arithmetic MWPs are widely treated as solved because existing systems perform well on benchmarks, but the paper examines whether that performance reflects genuine solving. It tests models without question text or word order and creates SVAMP, where state-of-the-art accuracy falls substantially, showing that these simple problems remain an open challenge.

  • Problem

    Existing solvers achieve high benchmark accuracy on elementary one-unknown arithmetic MWPs, raising questions about whether they genuinely solve them.

  • Method

    The paper evaluates question-removed and bag-of-words models and creates SVAMP by applying variations to existing elementary problems.

  • Results

    64.4% on ASDiv-A and 77.7% on MAWPS were achieved without the question, while state-of-the-art models solved fewer than half of SVAMP’s problems.

  • Takeaways & Limitations

    Benchmark performance can be achieved through shallow heuristics, making existing benchmarks unreliable for robustly measuring MWP-solving ability.

  • Takeaways & Limitations

    The study focuses on English one-unknown arithmetic MWPs taught up to grade four.

Abstract

from arXiv · show

The problem of designing NLP solvers for math word problems (MWP) has seen sustained research activity and steady gains in the test accuracy. Since existing solvers achieve high performance on the benchmark datasets for elementary level MWPs containing one-unknown arithmetic word problems, such problems are often considered "solved" with the bulk of research attention moving to more complex MWPs. In this paper, we restrict our attention to English MWPs taught in grades four and lower. We provide strong evidence that the existing MWP solvers rely on shallow heuristics to achieve high performance on the benchmark datasets. To this end, we show that MWP solvers that do not have access to the question asked in the MWP can still solve a large fraction of MWPs. Similarly, models that treat MWPs as bag-of-words can also achieve surprisingly high accuracy. Further, we introduce a challenge dataset, SVAMP, created by applying carefully chosen variations over examples sampled from existing datasets. The best accuracy achieved by state-of-the-art models is substantially lower on SVAMP, thus showing that much remains to be done even for the simplest of the MWPs.

1 Introduction

The paper questions whether strong benchmark performance means elementary math word problems are genuinely solved. It presents evidence of shallow heuristics and introduces SVAMP to test more robustly.

  • 1 Introduction: MWPs are semantic-parsing tasks requiring systems to extract relevant information from natural language and perform mathematical reasoning.The goal is to generate a mathematical expression that can be evaluated to obtain the answer.
  • 1 Introduction: Existing research often treats one-unknown arithmetic MWPs as well handled, shifting attention toward more complex problem types.These include multiple-unknown linear problems, geometry, and probability.
  • 1 Introduction: The paper argues that state-of-the-art methods may not robustly solve even the simplest MWPs.This challenges the assumption that high benchmark performance reflects genuine problem-solving ability.
  • 1 Introduction: Models can solve a majority of benchmark MWPs without question text or word-order information, indicating reliance on shallow heuristics.The question-removed and bag-of-words findings show that superficial patterns can support high accuracy.
  • 1 Introduction: SVAMP is a challenge set of elementary one-unknown arithmetic MWPs created by applying variations to existing problems for more robust evaluation.The set targets weaknesses in benchmark evaluation and tests capabilities expected of MWP solvers.

2 Related Work

Research on Math Word Problems spans statistical, semantic-parsing, and deep-learning methods, alongside continuing efforts to identify dataset deficiencies. Challenge sets have been developed in other NLP tasks, and this work targets robust evaluation specifically for Math Word Problems.

  • MWP research has used statistical machine learning, semantic parsing, and deep learning methods.
  • MAWPS was curated from previous datasets, while MAWPS and Math23k became benchmarks for recent MWP research.
  • Prior studies identified dataset artifacts and shallow heuristics in NLI, relation extraction, and ROC stories cloze tasks.
  • Challenge sets and manually perturbed contrast sets have been proposed for tasks including NLI and machine translation.
  • The authors introduce a challenge set targeted specifically at robust evaluation of Math Word Problems.

3 Background

The paper formalizes an MWP as a body and question mapped to a mathematical expression, evaluated by execution accuracy. Its analysis focuses on one-unknown arithmetic problems in English benchmark datasets and establishes baseline model performance for comparison.

  • An MWP is represented as tokens divided into a body B and question Q, with the goal of producing an expression using numbers from the problem and +, −, /, and ∗.
  • Execution Accuracy compares the answer obtained by evaluating the predicted expression EP with the annotated answer.
  • Table 2 reports five-fold cross-validation accuracies for baseline models, distinguishing RoBERTa embeddings from models trained from scratch.
  • The analysis is restricted to one-unknown arithmetic word problems.
  • The benchmark data comprise 2373 MAWPS problems and 1218 ASDiv-A arithmetic problems, mostly up to grade level 4.
  • Graph2Tree with RoBERTa embeddings achieves state-of-the-art performance on both datasets.
  • Table 3 reports five-fold cross-validation accuracies on datasets with the question removed.
  • The implementations exceed previously reported highest scores of 78% on ASDiv-A and 83.7% on MAWPS.

4 Deficiencies in existing datasets

Experiments show that benchmark accuracy can be achieved without question text or word order, revealing reliance on shallow correlations and making robustness estimates unreliable.

  • 4.1 Evaluation on Question-removed MWPs: 64.4% on ASDiv-A and 77.7% on MAWPS were achieved without models seeing the test problems’ questions.The results suggest direct correlations between narrative bodies and output equations.
  • 4.1 Evaluation on Question-removed MWPs: Models’ success was concentrated on Easy problems correctly answered without questions, while many Hard problems were also answered correctly.This pattern indicates that benchmark performance is overestimated by body-level heuristics.
  • 4.2 Performance of a constrained model: 51.2 accuracy on ASDiv-A and 77.9 on MAWPS was obtained by a constrained model without word-order information.The model used non-contextual RoBERTa embeddings and could associate specific words with equations.
  • 4.3 Analyzing the attention weights: The constrained model often attended to a single word regardless of context and retained the same prediction after a subtle change made that equation incorrect.These examples expose spurious word-to-equation correlations.
  • 4.3 Analyzing the attention weights: The analysis does not claim that every model uses specific-word occurrence in this brittle way.It establishes that such brittle models can nevertheless score well, making the datasets unreliable for robust evaluation.

5 SVAMP

SVAMP is a challenge set of elementary one-unknown arithmetic MWPs designed to test question sensitivity, reasoning ability, and structural invariance. It applies controlled variations to ASDiv-A seeds, and state-of-the-art models perform substantially worse than on existing benchmarks.

  • Dataset construction: SVAMP contains grade-level one-unknown arithmetic problems created by applying variations to seed examples from ASDiv-A.The dataset targets robust evaluation while remaining similar in scope and human difficulty to existing elementary-level datasets.
  • Variation categories: The variations test question sensitivity, reasoning ability, and structural invariance.They alter questions, relevant or changed information, unknown quantities, irrelevant information, object order, and phrase order.
  • Creation protocol: 100 seed examples were distributed among workers, who manually created and annotated variation examples with equations and variation types.Workers could combine multiple variation types in a single example.
  • Dataset properties: SVAMP is less lexically and typologically diverse than ASDiv-A, yet it is more challenging for models.Its construction prioritizes controlled robustness tests over dataset diversity.
  • Results: 43.8% accuracy was achieved by the best-performing Graph2Tree model on SVAMP.This result is reported despite SVAMP having the same scope and type as ASDiv-A and MAWPS.
  • Results: SVAMP is less vulnerable to simple-pattern solutions, requiring models to use contextual information and respond to changes in reasoning.Some structural-invariance categories make the challenge set harder, while Invert Operation examples can resemble training examples.

6 Final Remarks

The paper concludes that existing NLP models do not robustly solve elementary math word problems. Benchmark artifacts allow shallow heuristics to succeed, while SVAMP exposes substantial remaining weaknesses and motivates more robust evaluation and methods.

  • Conclusion: The paper gives a negative answer to whether existing NLP models can solve elementary math word problems.The conclusion concerns English elementary-level benchmark tasks and the proposed challenge set.
  • Benchmark reliability: Most existing benchmark problems can be solved using simple heuristics without word-order information or question text.This makes the benchmark datasets unreliable for gauging MWP solver performance.
  • Implications: SVAMP is intended to enable more robust evaluation of elementary-level MWP methods.The authors expect genuine advances beyond heuristic reliance to perform well on SVAMP even when trained on other datasets.
  • Open problem: Developing robust methods for elementary MWPs remains a significant open problem.The paper notes that research attention has shifted toward more difficult MWPs such as nonlinear equations and multiple-unknown problems.

A Experiments with Transformer

The paper additionally evaluates Transformer models on MAWPS, ASDiv-A, question-removed datasets, and SVAMP.

  • Transformer experiments: Transformer experiments report 5-fold cross-validation accuracies on MAWPS and ASDiv-A, plus results on question-removed datasets and SVAMP.The corresponding results are provided in Tables 16–18.

B Implementation Details

The experiments used specified hardware, hyperparameters, and dataset splits for MAWPS and ASDiv-A.

  • Hardware and configuration: Experiments used 8 NVIDIA Tesla P100 GPUs with 16 GB memory each.Model-specific hyperparameters are listed separately for the evaluated systems and Transformer model.
  • Dataset splits: MAWPS was divided into five equal-test-size folds, while ASDiv-A used the authors’ split sizes of 238, 238, 238, 238, and 266.These splits support the reported cross-validation evaluations.

C Creation Protocol

The protocol creates SVAMP by templating seed problems, generating controlled variations, annotating equations and variation types, and validating the resulting examples.

  • Template construction: The preprocessing script masks person names and numbers, while workers correct its output and add tags to create base examples.Workers also provide alternate tag values and modifier words for later lexicon-based scaling.
  • Variation generation: Workers first test whether each variation applies, then generate examples while maintaining consistent indexing for newly introduced tags.Multiple variations are listed from latest to earliest when combined.
  • Variation generation: Workers apply specified variations to base examples and record the variation types, including combinations in application order.The protocol covers question sensitivity, inversion, relevant or irrelevant information, information changes, and reordered objects or events.
  • Quality control: A trial using five shared seed examples found that different workers produced mostly the same examples aside from minor linguistic differences.This supported the effectiveness of the creation protocol.

D Analyzing Attention Weights

The constrained model’s learned word-to-equation correlations are illustrated through examples of attention paid to specific words.

  • Attention analysis: Table 26 provides examples of the specific word-to-equation correlation learned by the constrained model.The table focuses on which words receive attention.

E Examples of Simple Problems

The paper presents simple SVAMP problems that the best-performing Graph2Tree model fails to solve.

  • Failure examples: Table 27 lists simple SVAMP examples that the best-performing Graph2Tree model could not solve.These examples document failures on the challenge set.

F Ethical Considerations

The paper frames robust MWP solving as relevant to educational applications and presents SVAMP as an author-created challenge set with documented experimental details and no identified risks.

  • Motivation: Robust MWP solvers could support educational applications that provide elementary students with detailed solution explanations.The paper gives learning and practice as example uses.
  • Dataset: SVAMP is an in-house challenge set of one-unknown English MWPs created by applying simple variations to examples from ASDiv-A.The authors provide a detailed creation protocol and report no known risks associated with the dataset.
Loading 2103.07191v2…