Source-linked AI summary
Analysing Mathematical Reasoning Abilities of Neural Models
David Saxton, Edward Grefenstette, Felix Hill, Pushmeet Kohli
TL;DR
Neural models need evaluation on mathematical reasoning that depends on rules, symbolic manipulation, and composition rather than pattern matching alone. The paper builds a free-form benchmark across generated mathematics problems and evaluates sequence-to-sequence architectures. Models perform well on some arithmetic modules but struggle with mixed and multi-step reasoning, while the dataset currently omits linguistic variation.
Problem
Neural models show limited robustness, flexibility, and generalization beyond experienced environments, motivating evaluation of mathematical reasoning involving rules and algebraic composition.
Method
The paper creates a free-form mathematics dataset with controlled generation, compositional questions, and interpolation and extrapolation tests, then evaluates recurrent and Transformer sequence-to-sequence architectures.
Results
Transformer performance reaches 90% or more on several-number addition or subtraction and multiplication or division, but drops to around 50% on mixed arithmetic with parentheses.
Takeaways & Limitations
The dataset provides an analyzable benchmark for developing models with stronger algebraic and symbolic reasoning abilities.
Takeaways & Limitations
The dataset does not address linguistic variation or complexity, including distinct phrasings of the same underlying mathematical problem.
Abstract
from arXiv · showhide
Mathematical reasoning---a core ability within human intelligence---presents some unique challenges as a domain: we do not come to understand and solve mathematical problems primarily on the back of experience and evidence, but on the basis of inferring, learning, and exploiting laws, axioms, and symbol manipulation rules. In this paper, we present a new challenge for the evaluation (and eventually the design) of neural architectures and similar system, developing a task suite of mathematics problems involving sequential questions and answers in a free-form textual input/output format. The structured nature of the mathematics domain, covering arithmetic, algebra, probability and calculus, enables the construction of training and test splits designed to clearly illuminate the capabilities and failure-modes of different architectures, as well as evaluate their ability to compose and relate knowledge and learned processes. Having described the data generation process and its potential future expansions, we conduct a comprehensive analysis of models from two broad classes of the most powerful sequence-to-sequence architectures and find notable differences in their ability to resolve mathematical problems and generalize their knowledge.
1 INTRODUCTION
The paper introduces mathematics as a demanding test of neural models’ compositional and algebraic generalization, using a broad dataset to expose their capabilities and failure modes. It motivates the domain as a structured setting for evaluating knowledge transfer and systematic reasoning.
- Neural models remain brittle beyond experienced environments, especially on adversarially constructed inputs.
- Mathematical function composition requires parsing, planning, sub-algorithms, working memory, and rule-based transformations.
- The dataset spans diverse mathematics problems so success across problem types and generalization tests requires some abilities supporting algebraic generalization.
- Mathematics supports architecture analysis through self-consistent notation, extensible problem types, and reusable rules that enable knowledge transfer.
- The benchmark provides interpolation and extrapolation tests across many generated question types to measure mathematical reasoning and generalization.
- Experiments find that state-of-the-art neural architectures solve some question types but not all, with only moderate generalization and identifiable failure modes.
2 THE DATASET
The dataset uses procedurally generated, free-form mathematical questions and answers spanning multiple modules, with controlled difficulty and tests for compositionality and extrapolation. Its design prioritizes flexible textual representation, correctness, scale, and analyzable generalization.
- Procedural generation provides scalable examples, precise difficulty control, question-type analysis, correctness guarantees, and efficient generalization testing.
- 2.1 DESIGN CHOICES: Free-form character sequences express diverse question types, require semantic parsing, and simplify dataset and model development compared with trees or graphs.
- 2.1 DESIGN CHOICES: Free-form inputs and outputs match the format required by various real-world mathematics exam questions and can support benchmark-based unit testing.
- 2.1 DESIGN CHOICES: Composed questions chain modules with matching input and output types, using intermediate subproblem values as later inputs.
- 2.2 BRIEF OVERVIEW OF MODULES: The modules cover algebra, arithmetic, calculus, comparisons, measurement, and number-related tasks, extending a school mathematics curriculum with algebraic reasoning problems.
- Extrapolation tests vary axes such as number size, quantity, composition depth, recursion, and sampler size beyond training conditions.
3 MODELS EXAMINED
The paper examines recurrent, attentional LSTM, relational-memory, and Transformer sequence-to-sequence models for character-level mathematical question answering, including added computation steps before decoding.
- Model scope: General-purpose models are evaluated without built-in mathematical operations or symbolic solvers, preserving their domain-general modeling assumption.The study excludes approaches with direct access to arithmetic operations or mathematical templates.
- Recurrent architectures: Simple LSTM reads the question character by character and generates the answer character by character using the previous correct character.The model uses a 2048-unit hidden state selected through hyperparameter search.
- Recurrent architectures: Attentional LSTM encodes the question into key-value sequences that the decoder attends to while producing the answer autoregressively.Attention addresses input-ordering issues and the single-vector information bottleneck of the Simple LSTM.
- Computation steps: Both architectures receive additional zero-input computation steps before answer generation, while Adaptive Computation Time performed worse.These steps allow models to integrate question information before decoding.
- Recurrent architectures: RMC replaces the LSTM with interacting attention-based memory slots, but the best setting used one slot rather than the full relational memory.The reported configuration used 2048 total units, 16 attention heads, and 1 block.
- Transformer: The Transformer uses encoder-decoder self- and input-attention and represents questions and answers as character sequences.Its configuration uses d_model = 512, 8 attention heads, d_k = d_v = 64, and d_ff = 2048.
4 ANALYSIS
The analysis finds that performance varies sharply by task structure and architecture: models handle shallow or perceptual operations better than tasks requiring intermediate values, robust composition, or broader extrapolation. Transformers generally match or outperform recurrent models, but both show brittle and non-human-like failures.
- Architecture comparison: The Transformer performed the same as or significantly better than recurrent models across nearly all modules.The paper highlights more calculations per parameter, shallower depth, and sequential internal memory as possible advantages.
- Task difficulty: The easiest tasks were place-value identification and rounding, while primality detection and factorization were among the hardest.The models obtained nearly perfect scores on the easiest tasks; a Transformer produced a plausible but incorrect factorization of 235232673.
- Task difficulty: 90% or more performance on adding or subtracting several numbers and multiplying or dividing several numbers fell to around 50% on mixed arithmetic with parentheses.The authors attribute this gap to the former tasks being relatively linear, shallow, or parallel, while mixed expressions require intermediate-value calculation.
- Architecture comparison: The Transformer did significantly better than recurrent models on polynomial expansion, collecting terms, addition, composition, differentiation, and named-coefficient extraction.The authors speculate that parallel sequential processing helps keep multiple interacting coefficients in memory.
- Generalization: Good extrapolation covered larger numbers for rounding, comparison, addition, and subtraction, but models failed when adding more numbers than seen during training.The authors connect this failure with a suspicion that models add numbers in parallel rather than calculating subsums.
- External benchmark: 14/40 exam questions were answered correctly, proportionally equivalent to an E grade student.The model solved some simultaneous equations and sequence questions, but received no credit for plausible yet incorrect attempts.
5 CONCLUSIONS AND FUTURE WORK
The dataset provides a modular, extensible benchmark for mathematical reasoning, while identifying linguistic complexity and visual reasoning as important directions for future expansion.
- Current state-of-the-art neural models achieve moderate performance, with several multi-step modules largely unsolved and extrapolation performance low.The benchmark is intended to support development of models with stronger algebraic and symbolic reasoning abilities.
- The dataset is modular and easily extendable because its modules share a common input/output format and mathematical language.Its main restriction is that answers must be uniquely determined, although this still permits broad coverage up to university-level mathematics.
- Linguistic variation and complexity remain unaddressed, especially where translating algebraic word problems into formal mathematics contributes substantially to difficulty.The authors suggest extending the dataset with differently phrased versions of the same underlying mathematical problem.
- Mechanical collection of additional question templates is suggested as another way to expand the dataset.
- Visual problems such as geometry are proposed as a distinct extension requiring intermediate visual representations and visual working memory.The proposed reasoning scope is richer than merely analyzing visual domains in visual question-answering datasets.
A RECURRENT ENCODER AND DECODER WITH ATTENTION
The architecture uses an encoder-decoder sequence-to-sequence model with attention, representing input characters as key-value pairs and generating answer characters through decoder queries.
- The encoder maps a character-level question into a sequence of key-value pairs, with keys and values each having dimension 256.Questions are represented as sequences of 1-hot character vectors before encoding.
- Two encoder cores are tested: a unidirectional LSTM and a bidirectional LSTM that concatenates reverse-direction hidden states before a linear mapping.
- The decoder LSTM has hidden size 2048 and produces attention queries plus logits over 96 possible answer characters.Queries are dot-producted with encoder keys to form softmax weightings over encoded question values.
B.1 ALGEBRA
The algebra suite covers equation solving, polynomial manipulation, and sequence reasoning, with extrapolation tests extending polynomial size beyond training examples.
- Algebra modules include solving one-variable and simultaneous two-variable linear equations and finding or factorizing polynomial roots.
- Sequence modules require predicting the next term or deriving an nth-term expression from initial sequence values.
- The algebra tests include extrapolation beyond the training distribution.
- polynomial_roots_big applies polynomial-root finding or factorization to larger polynomials than those seen during training.
B.2 ARITHMETIC
The arithmetic suite spans operations on integers, decimals, fractions, bases, roots, surds, and longer expressions, with extrapolation tests increasing numerical size or expression length.
- Many arithmetic modules participate in module composition.
- Additional modules calculate nearest integer roots and simplify square-root expressions.
- Arithmetic extrapolation tests use larger integers or more terms than those seen during training.They include longer mixed and multiplication-division expressions.
B.3 CALCULUS
The calculus-related modules test differentiation, alongside comparison, conversion, and time operations that can be combined with outputs from other modules.
- Calculus: The differentiate module accepts inputs from and passes outputs to other modules, enabling it to participate fully in module composition.
- Calculus: Differentiation covers first- and higher-order derivatives of multivariate polynomials, including derivatives obtained through module composition.An example asks for the derivative of f(g(x)).
- Calculus: Comparison modules include finding closest values, selecting kth-largest or kth-smallest values, pairwise comparison, and sorting.
- Calculus: Extrapolation tests extend closest, kth_biggest, and sort tasks to lists larger than those seen during training.
- Calculus: Conversion and time modules handle unit conversion and clock-time calculations, with conversion extrapolation using larger values than during training.
B.6 NUMBERS
The numbers section spans arithmetic, number properties, functions, and probability, with compositional inputs and extrapolation tests that increase numerical or sampling difficulty.
- Numbers: Number modules include base conversion, division remainders, greatest common divisors, factor recognition, primality, least common multiples, prime factorization, place value, and rounding.
- Numbers: Place-value and rounding extrapolation tests use larger numbers than those seen during training.
- Functions: Function modules are fully compositional: they accept functions from other questions and define functions for use in other modules.
- Probability: Sampling-without-replacement probability modules operate on repeated-letter bags specified either by counts or by unsorted letter lists.
- Probability: Probability tasks calculate either the probability of letter-count sets or the probability of specified letter sequences.
- Probability: Probability extrapolation tests use more sampled letters than those seen during training.
C PER-MODULE PERFORMANCE
Per-module results are presented separately for interpolation and extrapolation, using the bidirectional LSTM encoder selected as the strongest recurrent-attention encoder.
- C PER-MODULE PERFORMANCE: Interpolation test performance is reported for the different modules in Figure 4.
- C PER-MODULE PERFORMANCE: Extrapolation test performance is reported for the different modules in Figure 5.
- C PER-MODULE PERFORMANCE: The per-module recurrent-attention results use the bidirectional LSTM encoder because it has the greatest performance among the encoders compared.