PaperlayerView exact revision on arXiv

Source-linked AI summary

The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers

Dongxiang Zhang, Lei Wang, Luming Zhang, Bing Tian Dai, Heng Tao Shen

arXiv:1808.07290v2cs.CL

TL;DR

Automatic MWP solving remains difficult because human-readable language must be mapped to machine-understandable logic, and strong results on small datasets do not generalize reliably to large, diverse ones. The paper surveys solver technologies, semantic features, task types, and evaluations, concluding that existing methods still have substantial room for improvement.

  • Problem

    MWP solvers must bridge the semantic gap between human-readable words and machine-understandable logic, while reported success on small datasets has not established strong performance on large, diverse datasets.

  • Method

    The paper provides a comprehensive survey of MWP solvers, organizing semantic-parsing techniques, extracted features, task types, datasets, evaluations, and related geometric and miscellaneous problems.

  • Results

    Large and diversified evaluations showed that many previously reported solver accuracies dropped sharply, indicating that existing methods are not yet generally robust.

  • Takeaways & Limitations

    The survey offers a structured reference for comparing MWP-solving techniques across arithmetic, equation-set, geometric, and other math tasks.

  • Takeaways & Limitations

    Dolphin18K contains 18,460 problems and 5,871 equation templates, but its online-forum annotations and answers may contain errors and may be lower quality than professionally sourced or paid-crowd datasets.

Abstract

from arXiv · show

Solving mathematical word problems (MWPs) automatically is challenging, primarily due to the semantic gap between human-readable words and machine-understandable logics. Despite the long history dated back to the1960s, MWPs have regained intensive attention in the past few years with the advancement of Artificial Intelligence (AI). Solving MWPs successfully is considered as a milestone towards general AI. Many systems have claimed promising results in self-crafted and small-scale datasets. However, when applied on large and diverse datasets, none of the proposed methods in the literature achieves high precision, revealing that current MWP solvers still have much room for improvement. This motivated us to present a comprehensive survey to deliver a clear and complete picture of automatic math problem solvers. In this survey, we emphasize on algebraic word problems, summarize their extracted features and proposed techniques to bridge the semantic gap and compare their performance in the publicly accessible datasets. We also cover automatic solvers for other types of math problems such as geometric problems that require the understanding of diagrams. Finally, we identify several emerging research directions for the readers with interests in MWPs.

1 INTRODUCTION

MWP solving has evolved from hand-crafted pattern matching toward semantic parsing and deep learning, but large, diverse datasets expose substantial limits in generality and robustness. This survey organizes the field’s methods, features, task types, evaluations, and future directions.

  • Technology Evolution: Early MWP solvers manually crafted rules and schemas, relying heavily on human intervention and handling only predefined scenarios.This pioneering stage spans roughly 1960 to 2010 and includes systems such as STUDENT, DEDUCOM, WORDPRO, and ROBUST.
  • Technology Evolution: Semantic-parsing methods map problem statements into structured logic representations and use feature engineering and statistical learning to support quantitative reasoning.The survey reviews these methods and examines their techniques across subtasks with organized experimental evaluations.
  • Technology Evolution: Large and diversified evaluations revealed that MWP solvers’ performance can fall sharply, motivating more accurate and robust approaches.Earlier systems often reported promising results on small or self-collected datasets, but later evidence showed substantial room for improvement.
  • Technology Evolution: Deep learning introduced an emerging direction that exploits large-scale training data, alongside continued refinement of semantic representations.The survey highlights models including Deep Neural Solver, Seq2SeqET, StackDecoder, MathDQN, CASS, and T-RNN.
  • Survey Scope and Contributions: The survey covers arithmetic, equation-set, geometric, and miscellaneous math problems while separately organizing related solver methods, features, preprocessing, evaluations, and future directions.Its coverage is intended to help researchers identify relevant approaches and distinguish arithmetic problems from more general equation-set problems.

2 ARITHMETIC WORD PROBLEM SOLVER

Arithmetic word problem solvers span rule-based, statistical, tree-based, and deep-learning approaches, with trade-offs between annotation, search, feature design, and scalability. Results show strong performance on some small datasets but sharp degradation and unclear generalization on larger, more diverse problems.

  • 2.2 Statistic-based Methods: Statistical methods require additional annotations and predefined templates, limiting their usability on large-scale and diversified datasets.The templates are brittle and rigid, and extending them to operators such as multiplication and division requires substantial effort.
  • 2.3 Tree-Based Methods: Tree-based methods avoid equation-template, tag, and logic-form annotations by constructing arithmetic expression trees bottom-up.ALGES instead enumerates syntactically valid trees with integer linear programming, at a computation cost dozens of times higher than another tree-based method.
  • 2.5.2 Performance Analysis: 88.64% accuracy was achieved by statistic-based methods on AI2 with advanced logic annotations, exceeding tree-based methods.Their advantage is attributed to fine-grained reasoning enabled by manually defined templates and role annotations, but the methods are brittle and not scalable.
  • 2.5.2 Performance Analysis: 75.5% accuracy on the challenging multi-step CC dataset improved over ALGES's 65%, with MathDQN modeling tree construction as a Markov Decision Process.The survey attributes this gain to deep Q-network reward feedback and reports leading or comparable performance across AI2, IL, and CC.
  • 2.5.2 Performance Analysis: UNITDEP performs best on SingleEQ and AllArith because its unit dependency graph constrains unit compatibility during expression-tree construction.The graph helps filter false candidates for multiplication and division problems but requires additional annotation overhead.
  • 2.5.2 Performance Analysis: Existing methods' accuracies fall below 25% on Dolphin18K equation-set problems, exposing unclear generalization from small datasets to larger, diverse collections.Dolphin18K contains 18,460 problems and 5,871 templates with one or multiple equations.

3 EQUATION SET SOLVER

Equation set problems require mapping text to multiple unknowns and a system of equations, creating a larger search space than arithmetic problems. The survey reviews semantic, similarity-based, template-based, and deep-learning approaches, finding that performance remains limited on large, diverse datasets.

  • Equation set problems involve multiple unknown variables and require several equations, making their candidate-equation search space substantially larger than arithmetic problems.
  • Large and diverse evaluation shows that methods previously reporting over 70% accuracy on small self-collected datasets can perform very poorly, indicating limited generality and robustness.
  • Similarity-Based Methods: SIM retrieves a similar training problem using TF-IDF and Jaccard similarity, then transfers its equation template and aligns quantities by ordered minimum-edit-distance matching.
  • Template Based Methods: Template-based methods address limited annotations through MixedSP, which jointly learns from fully annotated examples and noisy online problems with implicit supervision.
  • Template Based Methods: FG-Expression decomposes whole equation templates into fine-grained fragments to reduce problems caused by sparse instances and large lexical-syntactic feature spaces.
  • Performance Analysis: Performance comparisons are difficult because many table cells are empty due to implementation limits, dataset-release timing, or methods being unsuitable for particular datasets.
  • Performance Analysis: On Dolphin18K, CASS reaches 29% accuracy, while the survey reports that DNS did not achieve higher accuracy there and attributes challenges partly to thousands of templates.

4 FEATURE EXTRACTION

Feature extraction is central to MWP solvers because effective feature construction can improve accuracy, while deep learning can learn latent representations and reduce reliance on manual engineering.

  • Feature engineering is a vital component of MWP solvers, and effective construction can significantly boost accuracy.
  • Deep learning can automatically learn latent feature representations when sufficient training data is available, replacing labor-intensive manual feature engineering.
  • DSN is the only reviewed deep-learning approach that extracts features without manual engineering, using word embeddings and a GRU encoder.

4.1 Preprocessing

MWP preprocessing prepares text for feature extraction through syntactic parsing and coreference resolution, supporting structural analysis and consistent entity interpretation.

  • Preprocessing is introduced as a common prerequisite for subsequent feature extraction in MWP solvers.
  • Syntactic parsing organizes lexical units and semantic dependencies into tree structures used for feature selection.
  • Parser-derived representations support coreference resolution and automatic feature generation in several MWP solvers.
  • Coreference resolution identifies noun phrases referring to the same entity, helping solvers apply arithmetic operations or update values consistently.
  • MWP systems use coreference resolvers or declarative coreference rules to identify pronoun referents and support semantic labeling or operator determination.

4.2 Common Features

Common MWP features capture quantities, contexts, relationships, questions, verbs, and document-level signals to identify operands and operators or assign equation-template slots.

  • The survey emphasizes common features as more general and effective than unique features proposed only once.
  • Quantity-related features help identify relevant operands and operators, including rate indicators associated with multiplication or division.
  • Quantity contexts use lemmas, part-of-speech tags, and dependency types to distinguish numeric quantities with different roles.
  • Quantity relationships provide operator clues: shared units suggest addition or subtraction, whereas rate-unit pairings suggest multiplication or division.
  • In equation-template methods, dependency paths and sentence co-occurrence help assign quantities to slots and equations.
  • Question features identify the unknown and relevance through question-linked units and matching tokens, while dependent verbs provide operator cues such as subtraction for “lose.”
  • Global features include quantity counts, word unigrams, and bigrams that can influence quantity identification and ordering.

5 GEOMETRIC WORD PROBLEM (GWP) SOLVER

Geometric word problem solvers must jointly interpret diagrams and text because diagrams can contain essential information absent from the textual description.

  • Geometry-solving research spans diagram understanding, spatial descriptions, shape detection, and geometry theorem proving.
  • Geometric word problems require simultaneous understanding of visual diagrams and textual mentions because diagrams may contain information missing from the text.
  • G-ALINGER jointly addresses geometry and text understanding by detecting diagram primitives and aligning visual and textual information.
  • GEOS tackles complete geometric word problems by parsing text and diagrams into logical expressions, then optimizing their numerical alignment.
  • GeoShader represents diagram facts and deductibility relations as a hypergraph, formulating shaded-area calculation as a path-finding problem.

6 MISCELLANEOUS MATH TASKS

The survey covers miscellaneous math tasks beyond core algebraic problems, including non-diagram questions, multilingual solvers, and automated problem generation for algebra and geometry.

  • The survey reviews assorted math tasks, including non-diagram multiple-choice questions and knowledge-base-supported question answering.Aristo combines five specialized solvers, while another system uses Wikipedia as a knowledge base.
  • Math word problem solving in languages other than English remains at an early stage, with small, relatively unchallenging datasets and limited techniques.The surveyed Chinese and Arabic efforts use language-specific parsing or entity-recognition adaptations, but the area still has substantial room for improvement.
  • The survey also reviews generators that produce large, diverse, configurable corpora for algebraic and geometry problems.Covered topics include algebra problems with basic arithmetic operators and geometry problems.
  • Expression-tree generation constructs narratives recursively from sub-stories and value slots, whereas another approach rewrites existing problems into themed settings.These methods provide alternative mechanisms for generating or transforming math word problems.
  • GeoTutor generates geometry proof problems from a figure and axioms, outputting figure assumptions alongside explicit facts to infer as goals.The input-output formulation supports automated generation of high-school geometry proof exercises.

7 CONCLUSIONS AND FUTURE DIRECTIONS

The survey concludes that current MWP solvers still have substantial room for improvement and highlights deep learning, visual-text alignment, interpretability, and multilingual solving as future directions.

  • Conclusions: The survey compares techniques across math tasks, categorizes them, analyzes experiments, and reviews feature engineering and syntactic features.Its coverage spans arithmetic, equation-set, geometry, and miscellaneous math problems.
  • Conclusions: Current MWP solvers still have great room for improvement, motivating continued research as larger datasets become publicly available.The survey specifically mentions Dolphin18K and Math23K as large-scale datasets likely to support future work.
  • Future Directions: Deep learning is presented as a way to reduce non-trivial feature engineering and human intervention through end-to-end MWP models.DNS initiated this direction, followed by models such as T-RNN that generate quantity representations and perform inference.
  • Future Directions: Visual-text alignment is important for geometry problems, but existing evaluations rely on self-collected, small-scale datasets and require larger, more diverse testing.The survey warns that methods validated only in narrow settings may not work well on broader datasets.
  • Future Directions: Interpretability is important for online tutoring, while deep learning-based solvers may create new challenges for explaining their decisions.The survey contrasts interpretable domain knowledge and reasoning with difficult-to-interpret near-optimal actions in AlphaGo and AlphaZero.
  • Future Directions: Most MWP research focuses on English, leaving multilingual solving underexplored despite substantial non-English educational problem resources.The survey notes especially large Chinese collections of K12 questions and solutions.
Loading 1808.07290v2…