Source-linked AI summary
NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks
Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, Peter Clark, Chitta Baral, Ashwin Kalyan
TL;DR
Current AI systems remain brittle at simple arithmetic reasoning when problems are presented in slightly different ways. NumGLUE introduces an eight-task benchmark and shows that models perform substantially below humans, while joint training improves performance across tasks.
Problem
Current AI systems often fail to perform underlying arithmetic reasoning when numerically similar problems are phrased differently.
Method
NumGLUE is a multi-task benchmark comprising eight tasks that require arithmetic reasoning, with datasets retained in their naturally imbalanced distributions.
Results
All baseline models perform significantly below the human baseline, while joint training yields an average improvement of 3.4% on each task over task-specific training.
Takeaways & Limitations
NumGLUE provides evidence that robust arithmetic reasoning in language remains unsolved and that multi-task training can promote knowledge sharing across tasks.
Takeaways & Limitations
The benchmark retains substantial task imbalance, ranging from approximately 400 examples for one task to nearly 50K for another, to mimic real-world distributions.
Abstract
from arXiv · showhide
Given the ubiquitous nature of numbers in text, reasoning with numbers to perform simple calculations is an important skill of AI systems. While many datasets and models have been developed to this end, state-of-the-art AI systems are brittle; failing to perform the underlying mathematical reasoning when they appear in a slightly different scenario. Drawing inspiration from GLUE that was proposed in the context of natural language understanding, we propose NumGLUE, a multi-task benchmark that evaluates the performance of AI systems on eight different tasks, that at their core require simple arithmetic understanding. We show that this benchmark is far from being solved with neural models including state-of-the-art large-scale language models performing significantly worse than humans (lower by 46.4%). Further, NumGLUE promotes sharing knowledge across tasks, especially those with limited training data as evidenced by the superior performance (average gain of 3.4% on each task) when a model is jointly trained on all the tasks as opposed to task-specific modeling. Finally, we hope that NumGLUE will encourage systems that perform robust and general arithmetic reasoning within language, a first step towards being able to perform more complex mathematical reasoning.
1 Introduction
NumGLUE targets brittle arithmetic reasoning in language by benchmarking eight tasks designed to require simple arithmetic across varied presentations and contexts. State-of-the-art models perform substantially below humans, while joint training improves performance across tasks.
- 1 Introduction: Existing systems produce inconsistent answers when equivalent mathematical problems are presented with minor stylistic variations.The paper links this brittleness to reliance on spurious dataset-specific signals rather than underlying arithmetic reasoning.
- 1 Introduction: NumGLUE introduces eight arithmetic-reasoning tasks, including four new tasks, to test numeric reasoning across varied language settings.The benchmark includes roughly 100K problems and may combine arithmetic with commonsense reasoning or reading comprehension.
- 1 Introduction: 46.4% lower human-level performance shows that state-of-the-art large-scale models remain far from solving NumGLUE.The supplied conclusion describes current systems as incapable of general simple arithmetic reasoning.
- 1 Introduction: 3.4% average improvement per task occurs when the proposed model is jointly trained on NumGLUE rather than trained separately on each task.The authors attribute this result to beneficial transfer from the shared arithmetic-reasoning theme.
2 Related Work
Prior work developed datasets and models for quantitative reasoning and broader multi-task language understanding. NumGLUE extends these directions by evaluating arithmetic reasoning across multiple tasks and presentation styles.
- 2 Related Work: Earlier numerical-reasoning datasets progressively increased question difficulty, while multi-task benchmarks evaluated shared knowledge across tasks.The related work includes arithmetic, quantitative question answering, BAbI, GLUE, and SuperGLUE.
- 2 Related Work: NumGLUE is presented as the first benchmark to consider multiple tasks in the numerical-reasoning space.Its motivation is to test simple arithmetic reasoning independently of context or presentation style.
- 2 Related Work: Prior multi-task models show that shared underlying reasoning can support more robust and accurate performance across datasets.Examples include models targeting commonsense reasoning, problem decomposition, or linguistic understanding.
3 NUMGLUE
NUMGLUE is a multi-task benchmark of eight arithmetic-reasoning tasks spanning self-contained problems, external knowledge, reading comprehension, quantitative NLI, and word problems.
- NUMGLUE Benchmark: NUMGLUE contains eight tasks totaling about 100K questions, all requiring arithmetic reasoning at their core.The suite includes both newly created and existing tasks.
- Task Composition: Tasks range from self-contained arithmetic and fill-in-the-blanks problems to questions requiring commonsense, domain knowledge, reading comprehension, or quantitative comparison.Reading-comprehension tasks distinguish explicit numerical answers from implicit numerical reasoning.
- NUMGLUE Benchmark: The dataset preserves substantial task imbalance, ranging from about 400 examples for Task 1 to nearly 50K questions for Task 5.The authors retain this imbalance to better mimic real-world distributions.
- Data Partition and Evaluation: Data are split into training, development, and test sets, with passage-level assignment for reading-comprehension questions to discourage leakage.The reported evaluation uses F1 for each task and the unweighted average across tasks.
- Novel Datasets: The benchmark combines four novel tasks with four existing tasks and includes diverse linguistic and mathematical properties.The novel tasks include commonsense arithmetic, domain-specific arithmetic, quantitative comparison, and fill-in-the-blanks problems.
4 Experiments
The experiments compare heuristic, zero-shot, few-shot, fine-tuned, human, neuro-symbolic, and end-to-end baselines, including a memory-augmented Ex-NumNet model with retrieval.
- Baselines: The study evaluates heuristic, zero-shot, few-shot, fine-tuning, and human baselines using neuro-symbolic and end-to-end architectures.The neuro-symbolic family extends Ex-NumNet, while the end-to-end family uses GPT3.
- Architectures: Multi-task Ex-NumNet converts all tasks to reading-comprehension format, enabling a shared pipeline and injection of retrieved context.Question-only and context-only variants serve as bias-checking baselines.
- Baselines: Few-shot experiments compare task-specific examples with task-agnostic multi-task examples, while fine-tuning includes GPT3-13B in the multi-task setting.The multi-task few-shot setting conditions the model on instances from all tasks.
- Evaluation: The benchmark reports F1 across tasks and uses the unweighted task average as its aggregate evaluation measure.Human performance is estimated from annotated test samples for each task.
- Memory-Augmented Model: The memory-augmented model retrieves needed knowledge from the MATH KB and supplies it to Ex-NumNet.The modification targets Ex-NumNet's lack of external-knowledge handling.
5 Results and Discussion
NUMGLUE remains difficult for all tested baselines, while retrieval and joint training improve selected settings but expose trade-offs across task types and dataset sizes.
- Overall Results: All baseline models score significantly below the human baseline, showing that NUMGLUE remains challenging even for strong systems.The comparison covers the benchmark test set across tasks.
- Task Difficulty: Task 1 is hardest, whereas Task 7 performs best; numeric-answer tasks are comparatively difficult because they require accurate calculation.Task 6 slightly outperforms Task 5, and Task 3 benefits from its multiple-choice format.
- Information Retrieval: Conditional information retrieval is the strongest baseline, improving knowledge-dependent tasks while avoiding Task 3's drop from unnecessary information.Retrieval helps Tasks 1, 2, and 4; conditional retrieval fixes the adverse effect observed on Task 3.
- Data Imbalance: Oversampling helps some small-data tasks but significantly harms performance on large-data tasks and never surpasses conditional retrieval in the multi-task setting.The result highlights a trade-off from correcting task imbalance.
- Error Analysis: Most analyzed errors involve incorrect calculation, including wrong operations, number parsing, or numerical commonsense, while GPT3 produces more redundant text.The analysis also identifies invalid outputs, copying numbers, and redundant text as error categories.
- Cross-Task Transfer: 3.4% average improvement separates multitask from single-task Ex-NumNet training, indicating beneficial transfer across arithmetic-reasoning tasks.The reported gain is averaged across tasks.
6 Conclusion
NUMGLUE is an eight-task benchmark for arithmetic understanding, including tasks that combine arithmetic with external knowledge. State-of-the-art models perform substantially below humans, while joint training improves performance across tasks.
- NUMGLUE is a multi-task benchmark of eight arithmetic-understanding tasks, including four new tasks and tasks requiring commonsense or domain-specific knowledge.Some tasks are self-contained, while others combine arithmetic reasoning with additional knowledge.
- State-of-the-art large-scale models perform considerably worse than humans on NUMGLUE, showing that general simple-arithmetic reasoning remains unsolved.The benchmark remains challenging in both zero- or few-shot settings and after fine-tuning.
- Joint multi-task training yields an average 3.4% improvement on each task over task-specific training, indicating beneficial transfer from their shared arithmetic-reasoning theme.The paper presents memory-augmented baselines, including Ex-NumNet, to study modeling choices and cross-task knowledge sharing.
Ethical Considerations
The authors report that the source datasets are licensed for research use, modification, and redistribution. Distribution will credit the original data and task creators.
- All source-dataset licenses were verified to permit research use, modification, and redistribution.
- NUMGLUE will be distributed with full credit assigned to the original data and task creators.The planned distribution follows the SuperGLUE model.
A.1 NUMGLUE vs Other Datasets:
NUMGLUE selects tasks from clusters of numerical-reasoning datasets while excluding the multi-model reasoning cluster to keep the benchmark text-only.
- Tasks are selected from numerical-reasoning dataset clusters, excluding the multi-model reasoning cluster to maintain a text-only benchmark.
A.2 Construction of NUMGLUE :
The authors describe a structured process for creating NUMGLUE questions across the first four tasks and apply the same procedure to other examples within each task.
- Figures 5 and 6 illustrate the data-creation process for Tasks 1–4 using one example per task.
- The same question-creation procedure is followed for additional examples within each task.
A.3 GPT3-Instruct’s Response
GPT3-Instruct remained unreliable on simple arithmetic questions across multiple linguistic formats, despite expert tuning. The surrounding analyses describe NUMGLUE’s diverse, format-invariant tasks and the Ex-NumNet model’s conversion and retrieval mechanisms.
- GPT3-Instruct’s Response: GPT3-Instruct still could not solve basic arithmetic questions reliably after expert tuning across multiple parameter settings.The evaluation used varied question forms, including comparison, NLI, and fill-in-the-blanks formats.
- Data Quality Analysis: NUMGLUE’s novel datasets generally show higher vocabulary diversity and lower sentence similarity than existing datasets.Task 2 is an exception because its chemistry and physics principles produce templated questions with lower vocabulary and higher similarity.
- Ex-NumNet: Ex-NumNet converts each task into reading-comprehension format and uses task-specific heuristics to identify passages, questions, and answer options.For NLI, the premise becomes the passage and the hypothesis becomes the question, with answer labels appended.
- Memory-Augmented Model: The memory-augmented model retrieves relevant knowledge from MATH KB, filters redundant statements using semantic similarity, and appends the retained knowledge to the passage.The retrieval procedure uses Elasticsearch and parameters including Z = 50, v = 10, th = 0.75, and b = 0.1.