A Survey of Deep Learning for Mathematical Reasoning

1 Introduction

• The survey addresses the lack of a clear taxonomy of mathematical reasoning tasks and the capabilities deep learning models need to solve them.Earlier work focused on specific topics, including math word problems (Bhattacharya, 2017; Zhang et al., 2019), number representation (Thawani et al., 2021), and informal problems (Meadows and Freitas, 2022).Source: §1 · p. 1
• The survey covers more than 180 papers, spanning mathematical reasoning tasks and neural networks, pretrained language models, and in-context learning for large language models.Tasks include math word problems, theorem proving, geometry problem solving, math question answering, and other quantitative problems.Source: §1 · pp. 1–2
• The survey finds less benchmark focus on multimodal and low-resource settings, insufficient current numeracy representations, and inconsistent deep learning performance on mathematical reasoning.It proposes future directions in generalization and robustness, trustworthy reasoning, learning from feedback, and multimodal mathematical reasoning.Source: §1 · p. 2

Math Word Problem Solving

• Math word problems require language comprehension, semantic parsing, and one or more arithmetic operations to reach a solution.Problems may involve the four basic arithmetic operations and single or multiple steps.Source: §2 · p. 2

Theorem Proving

• Theorem proving seeks to establish a mathematical claim through logical arguments, requiring multi-step strategies, background knowledge, and symbolic manipulation.The task is to demonstrate the truth of a theorem through a sequence of logical arguments.Source: §2 · p. 2

Geometry Problem Solving

• Geometry problems combine a textual description and diagram to represent geometric entities, attributes, and relationships, with the goal of finding an unknown numeric value.Source: §2 · p. 2

Math Question Answering

• Math question answering covers benchmarks that use mathematical reasoning, including DROP (Dua et al., 2019), which requires discrete reasoning over paragraphs.An example asks which kicker made the most field goals based on paragraph content.Source: §2 · p. 3

3.1 Seq2Seq-based Networks for Math

• Seq2Seq encoder–decoders map problem inputs to outputs, supporting word problems, theorem proving, geometry solving, and math question answering.The surveyed work reports performance advantages over earlier statistical-learning approaches.Source: §3.1 · p. 3

3.2 Graph-based Networks for Math

• Graph-based models address Seq2Seq’s lack of explicit structure by generating tree- or graph-structured expressions and embedding graph information in input encodings.Seq2Tree models use equation ASTs, while Seq2Graph can capture complex relationships among variables (Liu et al., 2019a; Cao et al., 2021).Source: §3.2 · p. 3

3.3 Attention-based Networks for Math

• Attention methods identify important mathematical relationships, using self-attention for long-distance dependencies and variants for word problems, geometry, and theorem proving.Examples include MATH-EN (Wang et al., 2018a), Group-ATT, and graph attention for knowledge-aware information.Source: §3.3 · p. 3

3.4 Other Neural Networks for Math

• CNNs encode text to capture long-term symbol relationships, while multimodal geometry and diagram reasoning combine visual and textual representations through fusion models.CNNs were used for premise selection in the first deep-neural theorem-proving application (Alemi et al., 2016); multimodal systems use visual encoders such as ResNet or Faster-RCNN and textual encoders such as GRU or LTSM, with BAN, FiLM, or DAFA fusion.Source: §3.4 · p. 4
• Other approaches use GNNs for geometry parsing, WaveNet for theorem proving, Transformers for equation generation, and reinforcement learning for math word problems.Transformers outperform GRU in DDT, and MathDQN is the first work to explore reinforcement learning for math word-problem solving (Meng and Rumshisky, 2019; Wang et al., 2018b).Source: §3.4 · p. 4

4 Pre-trained Language Models for Mathematical Reasoning

• Despite promise in word problems, theorem proving, and other tasks, pre-trained models face limited math-specific data, costly full-model training, and varied downstream input formats.These challenges motivate fine-tuning models for downstream tasks or adapting their architectures.Source: §4 · p. 4

4.1 Self-Supervised Learning for Math

• Model scale has grown from MWP-BERT’s 110M- and 123M-parameter backbones to Minerva at up to 540B parameters; a study reports accuracy-leading models are at least 50B parameters.The cited study also finds model scale within a family reliably predicts accuracy (Liang et al., 2022a).Source: §4.1 · pp. 4–5
• Mathematical pre-training uses curated corpora such as AMPS and Minerva’s 38.5B-token dataset, as well as synthetic data generated from templates or interactions with engines.Examples include TaPEX’s executable SQL queries and outputs, LISA’s extracted formal proofs, and GenBERT’s template-based data.Source: §4.1 · p. 5
• Researchers apply masked-language and causal-language modeling to mathematical or scientific corpora, and also design numeracy-augmented or synthetic tasks to teach reasoning primitives and skills.Liang et al. (2022b) use eight numeracy-augmented tasks; LIME teaches deduction, induction, and abduction before more complex reasoning.Source: §4.1 · p. 5

4.2 Task-specific Fine-tuning for Math

• Task-specific fine-tuning adapts pre-trained models across word problems, MathQA, geometry, linear algebra, and theorem proving; some systems also ensemble model encoders with other modules.This approach is common when data is insufficient to train large models from scratch.Source: §4.2 · p. 6

5.1 In-context Example Selection

• Standard few-shot example selection is unstable, so studies explore retrieving semantically similar examples, choosing complex reasoning chains, and reinforcement-learning-based selection (Rubin et al., 2022; Fu et al., 2023; Lu et al., 2022b).Which reasoning examples make the most effective prompts remains an open question; early approaches often selected examples randomly or heuristically.Source: §5.1 · p. 6

5.2 High-quality Reasoning Chains

• Process-based approaches improve chains by decomposing complex problems into sub-problems solved sequentially or with diverse decomposition structures (Zhou et al., 2023; Khot et al., 2022).Least-to-most prompting first creates a list of sub-problems, then solves them in sequence, using answers to earlier sub-problems to facilitate later ones.Source: §5.2 · p. 7
• Program-of-thoughts expresses reasoning as code for an external computer to execute, while Chameleon integrates tools for compositional reasoning (Chen et al., 2022b; Gao et al., 2022; Lu et al., 2023).These are alternative ways to address complex reasoning beyond multi-step prompting.Source: §5.2 · p. 7
• Outcome-based approaches use multiple reasoning paths to mitigate errors in any single path; self-consistency selects the most common sampled answer, while selfteaching adds diverse prompts (Wang et al., 2023; Li et al., 2022a).Self-consistency samples multiple paths from one prompt and marginalizes over them by choosing the most common answer.Source: §5.2 · p. 7

6.1 Analysis of Benchmarks

• Most benchmarks target text alone, but emerging datasets support quantitative reasoning over images, diagrams, figures, charts, tables, conversations, and reports.Source: §6.1 · p. 7
• Low-resource mathematical reasoning remains underexplored despite emerging financial and scientific benchmarks and non-English datasets for Chinese and Arabic.Source: §6.1 · p. 7
• Datasets annotate intermediate reasoning as logic forms, programs, graphs, or natural-language solutions; Python programs are used for accessibility and readability.Source: §6.1 · pp. 7–8

Is the current representation of numeracy suf-

• Zero-shot-CoT Minerva 540B scores 75.0% on MMLU-STEM, while few-shot-CoT GPT-3 175B achieves 93.0% accuracy on MultiArith.The discussion raises whether such methods can tackle more complex problems.Source: Is the current representation of… · p. 8
• On SVAMP, Graph2Tree achieves only 43.8% accuracy and zero-shot-CoT GPT-3 (175B) reaches 63.7%; slight description changes also expose inconsistent GPT-3 performance.The discussion reports that human performance remains unchanged across those description variations.Source: Is the current representation of… · p. 8, Table 6 · p. 8

7.1 Generalization and Robustness

• Models still struggle to generalize across larger numbers, nearby problems, and longer problems, motivating new inference-time or fine-tuning strategies and deeper analysis of memorization.Term frequency affects simple arithmetic accuracy, while evidence about memorization in complex outputs remains inconclusive; progress requires accessible datasets and models for analysis.Source: §7.1 · pp. 8–9

7.2 Trustworthy Reasoning

• Because language models can hallucinate, reason incorrectly, and give wrong answers, future systems should support reasoning with evidence, express uncertainty, and detect mistakes.Rationales in prompts do not by themselves prevent flawed reasoning or wrong answers.Source: §7.2 · p. 9

7.3 Learning from Feedback

• Models could improve through feedback, using human rankings or theorem-proof engine outcomes and generated-script execution as reward signals for training.RLHF aligns models with instructions by learning a reward function from human rankings and fine-tuning with policy gradient.Source: §7.3 · p. 9

7.4 Multi-modal Mathematical Reasoning

• Multimodal mathematical reasoning needs more complex, realistic tasks, unified models for integrating modalities, and stronger benchmarks.Existing datasets are often small, template-generated, or topic-specific, while VQA frameworks trained on natural images can leave semantic gaps when analyzing mathematical figures and plots.Source: §7.4 · p. 9

• The survey reviews tasks, datasets, and deep learning approaches for mathematical reasoning.Source: §8 · p. 9
• It identifies gaps in existing mathematical reasoning datasets and methods.Source: §8 · p. 9
• It outlines future research directions and highlights potential for further exploration in the field.Source: §8 · p. 9
• The authors provide a reading list in a GitHub repository that will be continually updated.Source: §8 · p. 9

Limitations

• The survey covers mathematical reasoning and deep learning over the past decade, so it may not encompass the field’s full history.Its evaluation also draws on a curated set of papers that may not fully represent the state of the art.Source: Limitations · p. 10
• Because the field develops rapidly, the survey may omit advances published close to or after it was conducted.Source: Limitations · p. 10
• Despite these limitations, the survey provides an overview of the field’s current state and key trends for researchers and practitioners.Source: Limitations · p. 10

• The survey’s overview of key tasks, datasets, and methods helps researchers and practitioners understand the state of the art and make informed research decisions.Source: Broader Impact · p. 10
• By evaluating benchmarks and methods and identifying gaps, the survey aims to guide future research toward more advanced and effective mathematical reasoning systems.Source: Broader Impact · p. 10
• The survey aims to contribute to advances in mathematical reasoning and deep learning, with potential impact on machine learning and natural language processing.Source: Broader Impact · p. 10

• The section examines datasets available for studying mathematical reasoning with deep learning.Source: A Mathematical Reasoning Datasets · p. 21
• Table 7 summarizes commonly used mathematical reasoning datasets.Source: A Mathematical Reasoning Datasets · p. 21

A.1 Math Word Problem Solving

• Math word problems require language comprehension, semantic parsing, and mathematical reasoning to translate short narratives into equations and solve for an answer.Datasets range from grade-school problems to more diverse or multi-step tasks, including Ape210K, GSM8K, SVAMP, IconQA, and TabMWP (Zhao et al., 2020; Cobbe et al., 2021; Patel et al., 2021; Lu et al., 2021b, 2022b).Source: §A.1 · p. 21

A.2 Theorem Proving

• Formal theorem proving generates verified proof steps in interactive provers, with resources including CoqGym, PISA, LeanStep, Lean-Gym, and the cross-prover miniF2F benchmark.Other tasks include synthetic generalization tests and filling in missing intermediate propositions (Wu et al., 2021c; Li et al.).Source: §A.2 · pp. 21–22
• Informal theorem proving uses natural language and mathematical symbols, with resources for premise selection, retrieval, proof generation, and human evaluation.Work also connects informal and formal proving by translating informal statements or proofs into formal ones (Welleck et al., 2021, 2022a; Wu et al., 2022b; Jiang et al., 2022a).Source: §A.2 · p. 22

A.3 Geometry Problem Solving

• Geometry problem solving combines natural-language descriptions with diagrams to find unknown values, requiring multimodal parsing, symbolic abstraction, theorem knowledge, and quantitative reasoning.Geometry3K and later datasets provide annotated problems or executable programs for neural solvers (Lu et al.; Chen et al., 2021a; Cao and Xiao, 2022; Chen et al., 2022a).Source: §A.3 · p. 22

A.4 Math Question Answering

• Math question-answering datasets cover diverse quantities, temporal commonsense, approximate Fermi estimation, tabular reasoning, and unified multi-task evaluation.Examples include QuaRel, McTaco, Fermi, FinQA, TAT-QA, MultiHiertt, NumGLUE, and Lila, which spans 23 mathematical reasoning tasks (Tafjord et al., 2019; Zhou et al., 2019; Kalyan et al., 2021; Chen et al., 2021c; Zhu et al., 2021; Zhao et al., 2022; Mishra et al., 2022a,b).Source: §A.4 · pp. 22–23
• Benchmarks such as Mathematics and MATH target brittleness by testing algebraic generalization and problem solving on challenging competition mathematics.Prior systems may rely on spurious signals and plug-and-chug calculations that work on specific datasets (Hendrycks et al., 2021b; Mishra et al., 2022b).Source: §A.4 · p. 22

A.5 Other Quantitative Problems

• Everyday quantitative benchmarks examine reasoning over diagrams, numerical commonsense, natural-language inference, financial and scientific reports, and program behavior.These tasks include diagram-grounded quantitative relationships, numerical commonsense induction, conversational finance QA, and finding inputs that make programs return True (Kahou et al., 2018; Lin et al., 2020; Chen et al., 2022c; Lu et al., 2022a; Schuster et al., 2021).Source: §A.5 · p. 23