Source-linked AI summary
MathNet: a Global Multimodal Benchmark for Mathematical Reasoning and Retrieval
Shaden Alshammari, Kevin Wen, Abrar Zainal, Mark Hamilton, Navid Safaei, Sultan Albarakati, William T. Freeman, Antonio Torralba
TL;DR
Existing mathematical reasoning benchmarks have limited size, language coverage, task diversity, and analysis of mathematical retrieval. MathNet introduces a large multilingual, multimodal Olympiad corpus and benchmarks solving, structure-aware retrieval, and retrieval-augmented solving; evaluations show a gap between strong generative solving and weak retrieval, with retrieval-augmented performance depending on retrieval quality.
Problem
Existing benchmarks provide limited size, language coverage, task diversity, expert solutions, and analysis of mathematical problem retrieval.
Method
MathNet combines a 30K-problem Olympiad corpus with a fine-grained similarity taxonomy, expert-validated pairs, and benchmarks for solving, math-aware retrieval, and retrieval-augmented problem solving.
Results
Evaluations show strong generative problem solving but weak retrieval of mathematically equivalent or structurally related problems, while retrieval-augmented reasoning improves only with structure-aligned neighbors.
Takeaways & Limitations
MathNet provides a ground-truth foundation for studying mathematical generalization, analogical reasoning, retrieval quality, and retrieval-augmented reasoning across languages and modalities.
Takeaways & Limitations
Current retrieval systems still struggle to capture mathematical structure, and visual augmentation provides limited gains for symbolic tasks.
Abstract
from arXiv · showhide
Mathematical problem solving remains a challenging test of reasoning for large language and multimodal models, yet existing benchmarks are limited in size, language coverage, and task diversity. We introduce MathNet, a high-quality, large-scale, multimodal, and multilingual dataset of Olympiad-level math problems together with a benchmark for evaluating mathematical reasoning in generative models and mathematical retrieval in embedding-based systems. MathNet spans 47 countries, 17 languages, and two decades of competitions, comprising 30,676 expert-authored problems with solutions across diverse domains. In addition to the core dataset, we construct a retrieval benchmark consisting of mathematically equivalent and structurally similar problem pairs curated by human experts. MathNet supports three tasks: (i) Problem Solving, (ii) Math-Aware Retrieval, and (iii) Retrieval-Augmented Problem Solving. Experimental results show that even state-of-the-art reasoning models (78.4% for Gemini-3.1-Pro and 69.3% for GPT-5) remain challenged, while embedding models struggle to retrieve equivalent problems. We further show that retrieval-augmented generation performance is highly sensitive to retrieval quality; for example, DeepSeek-V3.2-Speciale achieves gains of up to 12%, obtaining the highest scores on the benchmark. MathNet provides the largest high-quality Olympiad dataset together with the first benchmark for evaluating mathematical problem retrieval, and we publicly release both the dataset and benchmark at https://mathnet.mit.edu.
1 INTRODUCTION
MathNet addresses gaps in open, diverse Olympiad-level benchmarks by combining a large multilingual corpus with tasks for solving, math-aware retrieval, and retrieval-augmented problem solving. It highlights a mismatch between strong generative solving and weaker retrieval of mathematical equivalence.
- Existing Olympiad-level datasets are limited in openness, quality, diversity, and competition coverage, constraining research progress.
- MathNet studies Problem Solving and Math-Aware Retrieval, where retrieval must recognize symbolic structure, invariances, and transformations rather than semantic similarity alone.
- Superficial lexical overlap can cause retrieval systems to rank mathematically non-equivalent expressions closer than truly equivalent formulations.
- MathNet-Solve contains 30K Olympiad-level problems with aligned LaTeX and natural-language statements, expert solutions, and metadata across 47 countries, 17 languages, and 65+ domains.
- MathNet-Retrieve provides 40K synthetic problems from 10K anchors with one equivalent positive and three hard negatives, while MathNet-RAG contains 70 expert-curated structurally similar IMO-level problems.
- Embedding-model performance in Math-Aware Retrieval lags behind LLM and LMM performance in Problem Solving, while retrieval-augmented generation helps only with structure-aligned, mathematically relevant neighbors.
2 RELATED WORK
Prior mathematical reasoning benchmarks span text-only, multimodal, and aggregated datasets, but remain limited in scale, language diversity, difficulty, expert solutions, or retrieval analysis. MathNet is positioned to address these gaps with Olympiad-level, multilingual, multimodal data and expert-validated similarity pairs.
- Text-only benchmarks range from grade-school arithmetic to competitive mathematics, but many remain limited in scale, language diversity, or structured similarity.
- Multimodal benchmarks incorporate charts and diagrams, yet they remain comparatively easy and do not capture the full difficulty of Olympiad-level problem solving.
- Formula-aware indexing captures formulas but misses broader conceptual and structural similarities, while modern semantic retrieval often misses symbolic equivalence and cross-modal cues.
- Prior benchmarks commonly lack expert solutions, visual multilingual content for high-difficulty problems, and substantial analysis of mathematical problem retrieval.
- MathNet expands benchmark coverage to 17 languages and emphasizes proof- and process-based evaluation with national contest problems.
3 DATASET AND BENCHMARK DESIGN
MATHNET combines three benchmark datasets and tasks for mathematical solving, retrieval, and retrieval-augmented solving, built from officially published Olympiad materials and organized around graded mathematical similarity. Its curation pipeline extracts, aligns, normalizes, and verifies problem–solution pairs from heterogeneous multilingual documents.
- Datasets and Tasks: MATHNET evaluates Problem Solving, Math-Aware Retrieval, and Retrieval-Augmented Problem Solving across three associated datasets.The datasets are MathNet-Solve, MathNet-Retrieve, and MathNet-RAG.
- Datasets and Tasks: MathNet-Solve contains 30,676 expert-written Olympiad problems with solutions spanning 47 countries, 17 languages, and 143 competitions.The corpus is divided into 23,776 training samples and 6,400 test samples.
- Data Collection: MATHNET draws exclusively on officially published national materials collected from 47 countries, comprising 1,595 PDF volumes and more than 25,000 pages.The sources include problems and solutions authored and disseminated by national teams.
- Extraction and Verification: Its extraction pipeline segments documents, extracts and normalizes problem–solution content, and verifies retained pairs through rule-based, GPT-4.1, and human checks.A pair is retained only when all three verification mechanisms agree unanimously.
- Mathematical Similarity: MATHNET categorizes similarity as Invariance, Resonance, or Affinity, ranging from strict equivalence through shared solution ideas to broad thematic relatedness.This taxonomy supports analysis across different levels of structural and semantic overlap.
4 EXPERIMENTS
Experiments evaluate problem solving, math-aware retrieval, and retrieval-augmented problem solving across diverse models and grading settings. Frontier reasoning models solve many Olympiad problems, but retrieval remains difficult and downstream gains depend on retrieval quality.
- Problem Solving: 76.3% overall accuracy makes gemini-3.1-pro the strongest MathNet-Solve model, ahead of gemini-2.5-pro at 71.9%.
- Problem Solving: 56.3% Geometry accuracy and 64.1% Discrete Mathematics accuracy show the hardest domains for gpt-5, despite 79.4% Algebra accuracy.
- Problem Solving: 72.7 points separate gemini-3.1-pro-preview from Ministral-3B overall, while mid-tier models score between 38.8% and 43.9%.
- Math-Aware Retrieval: ∼5% Recall@1 is achieved by the strongest retrieval models, although Recall@10 exceeds 80% in several domains.
- Math-Aware Retrieval: 68.88% Recall@5 and 83.79% Recall@10 make Gemini-embedding-001 the strongest aggregate retrieval model.
- Retrieval-Augmented Problem Solving: 97.3% is achieved by DeepSeek-V3.2-Speciale with Expert-RAG under human grading, while GPT-5 rises from 76.8% Zero Shot to 86.6% Expert-RAG.
- Retrieval-Augmented Problem Solving: Expert-RAG is generally strongest, but Embed-RAG can underperform Zero Shot when embedding retrieval returns near-miss problems.
5 DISCUSSION AND LIMITATION
MathNet exposes a gap between strong generative problem solving and weak retrieval of mathematically equivalent problems. The results also indicate that structured representations matter and that multimodal gains for symbolic tasks remain limited.
- Frontier generative models perform strongly on Problem Solving, while embedding systems struggle with mathematical-equivalence retrieval.
- Limited visual-augmentation gains suggest that multimodal integration for symbolic tasks remains underdeveloped.
- Structured, non-textual representations are crucial for retrieval, as indicated by the strong performance of the formula-aware baseline.
6 CONCLUSION
MathNet provides a human-validated foundation for studying mathematical generalization, analogical reasoning, and retrieval-augmented problem solving. Its evaluations show that generative models can solve complex problems while retrieval systems still struggle to find equivalent or structurally related problems.
- 30,676 problems and fine-grained equivalence labels support rigorous analysis of mathematical generalization and analogical reasoning.
- Human experts validate similarity labels and student evaluators assess extracted problem–solution alignment and completeness.
- Frontier generative models solve complex problems, but retrieval systems still struggle to retrieve mathematically equivalent or structurally related problems.
APPENDIX
The appendix provides additional materials supporting reproducibility and extending the paper’s main-text analysis.
- Additional results, dataset examples, tables, figures, prompts, and implementation details are included for reproducibility and further analysis.
A OVERVIEW OF COMPETITIONS COVERED BY MATHNET
MathNet-Solve covers a broad range of mathematical olympiads, along with related training and team-selection materials.
- MathNet-Solve draws problems from a wide range of mathematical olympiads.
- The collection also includes related training and team-selection materials.
- The dataset’s sources are cataloged with their covered years and competition names.
- International and multi-country competitions appear separately from national sources grouped by country.
Source Years Competitions National competitions and team selection tests by country
MathNet includes national competitions, international contests, team-selection tests, and training materials across many countries and years.
- Argentina: Argentina’s listed sources span 2003–2024 and include national, regional, international, and school-level mathematics competitions.
- Australia: Australia’s listed sources span 2010–2024 and include APMO, EGMO, IMO, national olympiads, and other contests.
- China: China’s sources span 2007–2025 and include national olympiads, team-selection tests, girls’ competitions, and regional contests.
- Croatia: Croatia’s sources span 2011–2019 and include national olympiads, junior and multi-round competitions, and IMO-related tests.
- Table 6 organizes the included competitions and countries by source category and national grouping.
B TAXONOMY OF TOPICS COMMONLY USED IN MATH OLYMPIAD
MathNet uses a curated, fine-grained taxonomy to label mathematical domains, subjects, topics, subtopics, and key concepts for consistent analysis.
- The taxonomy labels domains, subjects, topics, and subtopics to support consistent cross-competition comparisons.
- Geometry: Geometry categories include triangles, quadrilaterals, circles, concurrency or collinearity, and transformations.
- Algebra: Algebra categories include integers, fractions, equations, polynomials, and sequences or series.
- Number Theory: Number theory categories include divisibility, primes, GCD, LCM, factorization, modular arithmetic, and Diophantine equations.
- The taxonomy also records specialized concepts such as parametrization, inequalities, infinite descent, and structural methods.
C DATASET EXAMPLES AND ADDITIONAL STATISTICS
The examples section presents benchmark problems and solution excerpts spanning combinatorial games, modular subset sums, information recovery, and an asymptotic number-theory argument.
- MathNet-Solve examples: MathNet-Solve examples include a connected-strip cutting game whose winner depends on the parity of n.
- MathNet-Solve examples: The cutting-game solution uses symmetry and end-column control to establish that Olesya wins for odd n and Andrew for even n.
- MathNet-Solve examples: Another example asks for the smallest N ensuring a nonempty subset sum divisible by n when an N-element set contains a complete residue system modulo m.
- MathNet-Solve examples: Its solution constructs lower-bound examples and uses subset-sum divisibility facts, splitting the proof according to the relation between m, n, and their gcd.
- MathNet-Solve examples: A further example studies how many pairwise-set queries are needed to recover 2013 distinct numbers.
- Additional statistics: The section also includes an asymptotic argument concluding that infinitely many primes divide some sequence terms.
D PERFORMANCE SENSITIVITY TO IMAGE PRESENCE, LANGUAGE
MathNet reports performance separately by figure presence and language, revealing modality- and language-specific variation in problem-solving accuracy.
- Results are reported separately for samples with and without figures and across different languages.
- Top multimodal reasoning models gain substantially on text+image problems.
- Rankings are broadly consistent across languages, with Gemini-3.1-Pro and Gemini-3-Flash leading overall.
- Performance is weakest on Chinese and strongest on Italian and Portuguese.
E LLM GRADERS VS HUMAN EXPERT GRADERS ON MATHNET-RAG
MathNet-RAG evaluates whether LLM graders agree with human experts across solvers and retrieval settings. LLM graders broadly track the solver ranking established by human grading.
- Retrieval-Augmented Problem Solving is reported under zero-shot, embed-RAG, and expert-RAG settings.
- LLM graders broadly track the same solver ranking as human experts on MathNet-RAG.
- The evaluation covers 35 problems and compares four LLM graders, their mean score, and human expert scores.