Source-linked AI summary
A Survey of Deep Learning for Mathematical Reasoning
Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, Kai-Wei Chang
TL;DR
Deep learning for mathematical reasoning lacks a comprehensive account of its task taxonomy, capabilities, benchmarks, and limitations despite the importance of mathematical reasoning and rapid research growth. This survey reviews over 180 papers, covering tasks, datasets, architectures, pretrained models, benchmarks, and methods. It finds limited benchmark coverage of multimodal and low-resource settings, insufficient current numeracy representations, and inconsistent reasoning performance, and identifies future directions including generalization, robustness, trustworthy reasoning, feedback, and multimodal reasoning.
Problem
The field lacks a clear taxonomy of mathematical reasoning tasks and required deep learning capabilities, while prior reviews covered only specific aspects.
Method
The paper surveys over 180 papers on mathematical reasoning tasks, datasets, deep learning approaches, benchmarks, and methods from the past decade.
Results
The survey finds less benchmark focus on multimodal and low-resource settings, insufficient numeracy representations, and inconsistent deep learning methods for mathematical reasoning.
Takeaways & Limitations
The paper identifies future research directions in generalization and robustness, trustworthy reasoning, learning from feedback, and multimodal mathematical reasoning.
Takeaways & Limitations
The survey covers only the intersection of mathematical reasoning and deep learning over the past decade and evaluates a curated set of papers that may not fully represent the field.
Abstract
from arXiv · showhide
Mathematical reasoning is a fundamental aspect of human intelligence and is applicable in various fields, including science, engineering, finance, and everyday life. The development of artificial intelligence (AI) systems capable of solving math problems and proving theorems has garnered significant interest in the fields of machine learning and natural language processing. For example, mathematics serves as a testbed for aspects of reasoning that are challenging for powerful deep learning models, driving new algorithmic and modeling advances. On the other hand, recent advances in large-scale neural language models have opened up new benchmarks and opportunities to use deep learning for mathematical reasoning. In this survey paper, we review the key tasks, datasets, and methods at the intersection of mathematical reasoning and deep learning over the past decade. We also evaluate existing benchmarks and methods, and discuss future research directions in this domain.
1 Introduction
Mathematical reasoning is important across human intelligence and practical domains, yet deep learning research lacks a clear taxonomy of tasks and required capabilities. This survey organizes the field, reviews over 180 papers, evaluates benchmarks and methods, and identifies research gaps and future directions.
- Mathematical reasoning supports numerical understanding and decision-making across science, engineering, finance, and everyday life.
- Deep learning models have advanced mathematical reasoning, but the field still lacks a clear taxonomy of task types and required capabilities.
- Prior literature mainly addressed isolated areas such as math word problems, number representation, or informal problem solving.
- The survey covers over 180 papers, spanning math word problems, theorem proving, geometry, math question answering, and other quantitative problems.
- The taxonomy groups mathematical reasoning datasets into textual, multimodal, formal, informal, and combined formal-plus-informal settings.
- Benchmark analysis finds less focus on multimodal and low-resource settings, while evidence-based studies find insufficient numeracy representations and inconsistent reasoning methods.
2 Mathematical Reasoning Tasks
The survey covers several mathematical reasoning tasks, including word problems, theorem proving, geometry problem solving, and mathematical question answering. These tasks combine language understanding, semantic parsing, multimodal interpretation, logical argumentation, and numerical reasoning.
- Math Word Problem Solving: Math word problems require language comprehension, semantic parsing, and multiple mathematical reasoning skills across one or more arithmetic operations.
- Theorem Proving: Theorem proving seeks to establish a theorem through a sequence of logical arguments, testing strategy selection, background knowledge, and symbolic manipulation.
- Geometry Problem Solving: Geometry problems combine textual descriptions with diagrams to represent geometric entities, attributes, and relationships while solving for an unknown numeric variable.
- Math Question Answering: Math question answering includes benchmarks requiring discrete reasoning over passages, such as identifying which kicker made the most field goals.
3 Neural Networks for Mathematical Reasoning
Deep learning approaches to mathematical reasoning include sequence, graph, attention, convolutional, multimodal, recurrent, transformer, and reinforcement-learning architectures. These methods differ in how they represent mathematical structure, long-range dependencies, visual information, and search.
- Seq2Seq Networks: Seq2Seq models formulate mathematical reasoning as mapping an input problem sequence to an output sequence such as an equation or proof.
- Graph-based Networks: Graph-based networks explicitly capture structural information in mathematical expressions that Seq2Seq methods do not represent directly.
- Attention-based Networks: Attention-based methods identify important relationships and learn long-distance dependencies in word problems, geometry solving, and theorem proving.
- Other Neural Networks: Convolutional and multimodal networks model long-term symbolic relationships or combine visual and textual representations for mathematical reasoning.
- Other Neural Networks: Transformers outperform GRU in generating mathematical equations in DDT, while MathDQN introduced reinforcement learning to math word problem solving for search.
4 Pre-trained Language Models for Mathematical Reasoning
Pre-trained language models have been applied to mathematical reasoning tasks, but their natural-language training and limited mathematical data create challenges. Research explores model scale, mathematical pre-training corpora and tasks, and task-specific fine-tuning across downstream applications.
- Pre-trained language models have shown promising performance on math word problems, theorem proving, and other mathematical tasks.
- Large language models face challenges because they are not specifically trained on mathematical data and have less mathematical or scientific pre-training data than text data.
- Model scale: Model scale has increased substantially, from BERT and RoBERTa backbones of 110M and 123M parameters to Minerva at up to 540B parameters.
- Pre-training corpus: Mathematical language models use curated corpora such as AMPS, which contains step-by-step solutions in natural language and LATEX from Khan Academy and Mathematica.
- Pre-training tasks: Researchers pre-train models on mathematical or scientific corpora using masked language modeling and causal language modeling, alongside customized mathematical reasoning tasks.
- Task-specific fine-tuning: Task-specific fine-tuning adapts pre-trained models to downstream problems including math word problems, MathQA, geometry, linear algebra, and theorem proving.
5 In-context Learning for Mathematical Reasoning
In-context learning enables large language models to perform mathematical tasks from examples or reasoning prompts without updating model parameters. Research therefore focuses on improving example selection and reasoning-chain quality through decomposition, diverse demonstrations, and multiple reasoning paths.
- In-context learning conditions a large language model with task examples at inference time without updating its parameters.
- Standard few-shot prompting provides input-output examples before a test example, but has not proved sufficient for high performance in mathematical reasoning.
- Reasoning chains: Chain-of-thought prompting uses intermediate natural-language rationales so models generate reasoning chains before predicting an answer.
- Recent work improves few-shot chain-of-thought reasoning by selecting better in-context examples and creating better reasoning chains.
- Example selection: Few-shot learning can be unstable across example selections, making effective in-context reasoning examples an open problem.
- Reasoning-chain strategies: Process-based approaches decompose complex problems into sub-problems and solve them sequentially, while outcome-based approaches aggregate multiple reasoning paths.
6 Discussion and Findings
The survey finds important gaps in mathematical-reasoning benchmarks and methods, including underexplored multimodal and low-resource settings, insufficient number representations, and inconsistent performance.
- Benchmark gaps: Multimodal mathematical reasoning remains underexplored, while most benchmarks target text-only inputs.Visual information can support reasoning over quantitative relations in images, diagrams, figures, charts, and tables.
- Benchmark gaps: Low-resource mathematical reasoning is largely under-explored despite emerging financial, scientific, and non-English datasets.Existing work includes benchmarks for financial and scientific domains and datasets in Chinese and other non-English settings.
- Dataset design: Intermediate rationales are represented as logic forms, programs, reasoning graphs, and natural-language solutions to support complex reasoning.Python programs are used in some datasets because they offer enhanced accessibility and readability.
- Numeracy representation: Token-based number representations can collapse rare numbers into UNK or split them into suboptimal subword tokens.Insufficient numerical representations contribute to out-of-distribution problems, and GPT-3 performs poorly when reasoning over large numbers.
- Consistency and robustness: 75.0% on MMLU-STEM and 93.0% on MultiArith demonstrate strong results, but complex-problem capability remains uncertain.These figures correspond to zero-shot-CoT Minerva 540B on MMLU-STEM and few-shot-CoT GPT-3 175B on MultiArith, respectively.
- Consistency and robustness: On SVAMP, Graph2Tree reaches 43.8% accuracy and zero-shot-CoT GPT-3 reaches 63.7%, while performance varies across slight description changes.Human performance remains unchanged in the cited comparisons, indicating inconsistent reasoning by state-of-the-art large language models.
7 Future Work
Future work targets failures in generalization, robustness, trustworthiness, feedback-based improvement, and multimodal reasoning. The survey emphasizes better evaluation, accessible resources, and methods that handle more complex settings.
- 7.1 Generalization and Robustness: Future research should address generalization and robustness failures on larger numbers, nearby problems, and longer problems than seen during training.Suggested approaches include new inference-time and fine-tuning strategies, alongside analysis of memorization.
- 7.1 Generalization and Robustness: Understanding memorization in complex solutions and proofs requires further analysis with accessible datasets and models.Existing findings differ on whether complex outputs depend on memorized solutions, and some training sets and models are unavailable for inspection.
- 7.2 Trustworthy Reasoning: Mathematical language models need methods that reduce ungrounded answers, hallucinated statements, flawed reasoning, and wrong answers.Rationales produced through prompting do not eliminate these trustworthiness problems.
- 7.3 Learning from Feedback: Learning from human or theorem-proof-engine feedback is proposed as a route toward continual improvement of output quality and safety.RLHF ranks generated outputs and uses a learned reward function to fine-tune the language model.
- 7.4 Multi-modal Mathematical Reasoning: Multimodal datasets are often small, template-generated, or topic-specific, and VQA-based systems can create semantic gaps for mathematical figures and plots.Future work should develop unified multimodal models and stronger evaluation benchmarks for complex, realistic problems.
8 Conclusion
The paper surveys deep learning for mathematical reasoning by reviewing tasks, datasets, and approaches, identifying gaps, and outlining future research directions. It aims to provide a useful resource for researchers and practitioners.
- 8 Conclusion: The survey reviews mathematical-reasoning tasks, datasets, and deep learning approaches while identifying gaps in existing datasets and methods.It also outlines future research directions and provides a continually updated reading list in a GitHub repository.
- 8 Conclusion: The paper is intended as a comprehensive resource for readers interested in developing deep learning systems for mathematical reasoning.
Limitations
The survey focuses on mathematical reasoning and deep learning over the past decade, evaluates a curated paper set, and may not reflect the entire field or its latest developments.
- The survey’s decade-long focus may not encompass the entire field or its history.
- Its evaluation relies on a curated set of papers that may not fully represent the field’s state of the art.
- Because the field is fast-paced, the survey may not reflect the latest developments and advancements.
Broader Impact
The survey provides an overview of mathematical reasoning tasks, datasets, and methods developed over the past decade. It evaluates existing benchmarks and methods while identifying gaps and future research directions.
- The survey organizes key tasks, datasets, and methods in deep learning for mathematical reasoning.
- It evaluates existing benchmarks and methods to clarify the current state of the art.
- It discusses future research directions intended to identify gaps and guide further development.
A Mathematical Reasoning Datasets
This section examines datasets available for studying mathematical reasoning with deep learning methods and summarizes commonly used datasets in Table 7.
- The section examines datasets currently available for mathematical reasoning with deep learning methods.
- The datasets are presented as resources for studying mathematical reasoning using deep learning.
- Table 7 summarizes the commonly used mathematical reasoning datasets.
A.1 Math Word Problem Solving
The survey covers datasets for mathematical reasoning across word problems, theorem proving, geometry, and question answering. These resources vary in modality, proof form, scale, and evaluation goals, including robustness and generalization.
- Math Word Problem Solving: Math word problem datasets model brief narratives with quantities as equations whose solutions produce final answers.
- Math Word Problem Solving: GSM8K includes problems requiring up to 8 solution steps, while SVAMP tests robustness to simple problem variations.
- Theorem Proving: Theorem-proving resources include interactive environments, proof-library datasets, informal proofs, and benchmarks linking informal and formal statements.
- Geometry Problem Solving: Geometry problem solving combines textual and diagrammatic inputs and requires multimodal parsing, symbolic abstraction, theorem knowledge, and quantitative reasoning.
- Math Question Answering: Mathematical question-answering benchmarks span numerical reasoning, tabular contexts, diagrams, and generalization beyond dataset-specific shortcuts.