MathNet: a Global Multimodal Benchmark for Mathematical Reasoning and Retrieval

Math-Aware Retrieval

• Math-aware retrieval must recognize symbolic structure, invariances, and transformations; existing models can mistake lexical overlap for mathematical equivalence.This matters in workflows such as finding equivalent problems across different notations, formats, and languages.Source: §1 · p. 2

Main Corpus

• MathNet-Solve contains 30K Olympiad problems with aligned LaTeX and natural-language statements, expert solutions, and metadata spanning 47 countries, 17 languages, and 65+ mathematical domains.The collection is intended to support analysis of mathematical generalization and analogical reasoning.Source: §1 · p. 2

Datasets for Retrieval

• MathNet-Retrieve pairs 10K anchor problems with 40K synthetic problems, each having one equivalent positive and three hard negatives; MathNet-RAG contains 70 expert-curated, structurally similar IMO-level problems.These datasets support math-aware retrieval and retrieval-augmented problem solving, respectively.Source: §1 · p. 2

Benchmark Evaluation

• The benchmark evaluates 27 state-of-the-art models on problem-solving accuracy, retrieval Recall@k, and retrieval-augmented solving accuracy, using automatic and human expert grading.The three benchmarks use MathNet-Solve, MathNet-Retrieve, and MathNet-RAG, respectively.Source: §1 · p. 2

Analysis: Solving vs. Retrieving

• Embedding-model retrieval lags behind LLM and LMM problem solving, and retrieval-augmented generation improves reasoning only when retrievers surface structurally aligned, mathematically relevant neighbors.MathNet studies Problem Solving, Math-Aware Retrieval, and Retrieval-Augmented Problem Solving.Source: §1 · p. 2

Text-Only Mathematical Benchmarks

• Text-only benchmarks span arithmetic to Olympiad mathematics, but many remain limited in scale, language diversity, or structured similarity annotations.Examples include GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021b), and Omni-MATH’s 4,428 Olympiad-level problems (Gao et al., 2024b), alongside bilingual and competition-level datasets (He et al., 2024; Wang et al., 2024).Source: §2 · p. 3

Multimodal Mathematical Benchmarks and Large-Scale Aggregates

• Multimodal benchmarks such as MATH-Vision (Wang et al., 2024) and MathVista (Lu et al., 2024) include visual contexts but remain comparatively easy for Olympiad-level problem solving.Large-scale aggregates such as NuminaMath (Li et al., 2024b) and (Albalak et al., 2025) typically lack curated multimodal content, multilingual coverage, and fine-grained annotations.Source: §2 · p. 3

Math-Aware Retrieval

• Formula-aware indexing (Zanibbi et al., 2025; Das et al., 2025) captures formulas but misses broader conceptual and structural similarities, while modern information retrieval often misses symbolic equivalence and cross-modal cues.The cited formula-indexing systems predate LLMs and typically operate at the formula level.Source: §2 · p. 3

3.1 DATASETS AND TASKS

• MathNet-Solve contains 30,676 problems with solutions from 47 countries, 17 languages, and 143 competitions; the three datasets support problem solving, retrieval, and retrieval-augmented solving.MathNet-Retrieve is described here as using 19K anchor problems, while MathNet-RAG evaluates retrieved and oracle contexts.Source: §3.1 · p. 4

3.2 DATA COLLECTION, EXTRACTION AND ANNOTATION

• The corpus comes from official national contest booklets, and its LLM-based alignment pipeline retains problem–solution pairs only after rule-based, GPT-4.1, and human verification agree.The source collection covers 47 countries from 1985–2025 and includes 1,595 PDF volumes comprising more than 25,000 pages.Source: §3.2 · p. 5

3.3.1 UNDERSTANDING DIFFERENT TYPES OF MATHEMATICAL SIMILARITY

• The taxonomy distinguishes Invariance (strict equivalence), Resonance (shared ideas or strategies), and Affinity (thematic relatedness without shared structure).It enables analysis of solver and retriever performance across different degrees of structural and semantic overlap.Source: §3 · p. 4, §3.3.1 · pp. 6–7

3.3.2 BUILDING MATHNET-RETRIEVE

• MathNet-Retrieve tests whether embeddings identify mathematical equivalence beyond lexical overlap, using generated positives and hard negatives verified by agreement between Gemini-3-flash and GPT-5.The construction passage specifies 15K anchors, with one equivalent positive and three hard negatives for each difficulty level.Source: §3.3.2 · p. 7

3.3.3 BUILDING MATHNET-RAG

• MathNet-RAG pairs 35 anchors with 35 expert-curated real Olympiad problems, all in Structural Resonance through generalization, common lemmas, or structural reduction.The 70 problems are drawn from real competitions, and the pairings reflect expert similarity beyond simple algebraic transformation.Source: §3.3.3 · p. 7

4.1 EXPERIMENTAL SETUP

• The evaluation covers 27 models across MathNet-Solve accuracy, MathNet-Retrieve Recall@k, and MathNet-RAG accuracy.Solve evaluations include language and multimodal models, retrieval uses cosine similarity over embeddings, and RAG compares seven models across Zero Shot, Embed-RAG, and Expert-RAG.Source: §4.1 · p. 8

Problem Solving on MathNet-Solve.

• MathNet-Solve uses GPT-5 to score generated solutions from 0–7, counting scores ≥6 as correct and reporting results by subject domain.The protocol is designed to distinguish consistent reasoning from reaching the correct answer by coincidence.Source: Problem Solving on MathNet-Solve · p. 8

Math-Aware Retrieval on MathNet-Retrieve.

• MathNet-Retrieve reports Recall@1 and Recall@5, measuring whether a top-k result matches an equivalent version of the query problem.The analysis also examines cosine-similarity distributions for equivalent pairs, unrelated pairs, and hard-negative near misses.Source: Math-Aware Retrieval on MathNet-Retrieve · p. 8

Retrieval-Augmented Problem Solving on MathNet-RAG.

• The RAG comparison isolates retrieval quality by measuring performance with no example, an embedding-retrieved example, or an expert-paired example, each with its official solution.The Zero Shot-to-Embed-RAG gap measures the value of embedding-retrieved context; the Embed-RAG-to-Expert-RAG gap reflects remaining limits from retrieval errors.Source: Retrieval-Augmented Problem Solving on… · p. 8

4.3.1 PROBLEM SOLVING ON MAT HNE T-SO L V E

• Gemini-3.1-Pro achieves the highest MathNet-Solve overall accuracy at 76.3%, while GPT-5 scores 56.3% on Geometry.Gemini-2.5-Pro reaches 71.9% overall; Geometry and Discrete Mathematics are the most challenging domains, and Gemini-3.1-Pro-preview leads Ministral-3B by 72.7 points overall.Source: Table 3 · p. 9, §4.3.1 · p. 9

4.3.2 MATH-AWARE RETRIEVAL ON MA T HNE T-RE T R I E V E

• Gemini-embedding-001 achieves aggregate Recall@5 of 68.88% and Recall@10 of 83.79%, but top models reach only ∼5% Recall@1.The passages attribute weak top-1 retrieval to superficial textual overlap rather than mathematical structure, motivating embeddings trained to represent mathematical relationships.Source: §4.3.2 · pp. 9–10, Table 4 · p. 10

4.3.3 RETRIEVAL-AUGMENTED PROBLEM SOLVING ON MA T HNE T-RAG

• Under human grading, DeepSeek-V3.2-Speciale reaches 97.3% with Expert-RAG, while GPT-5 rises from 76.8% in Zero Shot to 86.6% in Expert-RAG.Expert-RAG is strongest overall, but gains vary by model and grading protocol; Embed-RAG can fall below Zero Shot when near-miss problems add noise.Source: Table 5 · p. 10, §4.3.3 · p. 10

5 DISCUSSION AND LIMITATION

• Current embedding-based systems struggle to capture mathematical structure despite frontier models’ strong performance on answer-generation benchmarks.The results reveal a clear gap between problem solving and mathematical-equivalence retrieval.Source: §5 · p. 11
• Limited gains from visual augmentation suggest that multimodal integration for symbolic tasks remains underdeveloped.Source: §5 · p. 11
• The formula-aware baseline’s strong performance indicates that structured, non-textual representations are crucial for mathematical retrieval.The paper suggests progress in mathematical reasoning may require moving beyond next-token prediction toward architectures that explicitly integrate symbolic reasoning.Source: §5 · p. 11

6 CONCLUSION

• MATHNET provides 30,676 problems with a fine-grained equivalence taxonomy, supported by expert review of similarity labels and student evaluation of extracted problem–solution pairs.These human checks establish ground truth intended to capture deep mathematical structure rather than superficial overlap.Source: §6 · p. 11
• Frontier generative models can solve complex problems, but retrieval systems still struggle to find mathematically equivalent or structurally related problems in large corpora.The retrieval gap highlights a limitation in current learned representations of mathematical knowledge.Source: §6 · p. 11
• MATHNET is intended to support research on improved retrieval-augmented reasoning, symbolic AI, and more capable and reliable problem-solving models.Source: §6 · p. 11

APPENDIX

• It includes additional results, dataset examples, tables, figures, prompts, and implementation details.These materials support reproducibility and provide further analysis beyond the main text.Source: APPENDIX · p. 14

• The dataset spans a wide range of mathematical olympiads and related training and team-selection materials.Source: A OVERVIEW OF COMPETITIONS COVERED BY… · p. 14
• Table 6 catalogs the dataset’s sources and the years covered.Source: A OVERVIEW OF COMPETITIONS COVERED BY… · p. 14
• International and multi-country competitions include the Asia Pacific Mathematics Olympiad, Balkan Mathematical Olympiad, and Baltic Way.Source: A OVERVIEW OF COMPETITIONS COVERED BY… · p. 14
• National sources are grouped by country and include olympiads, international-team selection tests, other national contests, and training materials.Source: A OVERVIEW OF COMPETITIONS COVERED BY… · p. 14

Taxonomy

• The curated taxonomy provides consistent labels for analyzing domains, subjects, topics, and subtopics across competitions.It is intended to ground the paper’s analyses and enable cross-competition comparisons.Source: B TAXONOMY OF TOPICS COMMONLY USED IN… · p. 17

Geometry

• Geometry ranges from plane and solid geometry to differential and non-Euclidean topics, with coordinate methods, transformations, configurations, and inequalities.Listed concepts include triangle centers and classical theorems, 3D volume and surface area, curvature, spherical triangles, and hyperbolic geometry.Source: B TAXONOMY OF TOPICS COMMONLY USED IN… · p. 17

Algebra

• Algebra covers foundational number properties and equations alongside polynomials, sequences, functional equations, and classical and functional inequalities.The taxonomy includes techniques and concepts such as factorization, Vieta’s formula, recurrences, and Cauchy-Schwarz and Jensen’s inequalities.Source: B TAXONOMY OF TOPICS COMMONLY USED IN… · pp. 17–18

Discrete Mathematics

• Discrete mathematics includes graph theory, combinatorics, probability, invariants, induction, games, logic, and algorithms.Topics range from matchings and connectivity to counting, expected values, coloring, greedy strategies, and dynamic programming.Source: B TAXONOMY OF TOPICS COMMONLY USED IN… · p. 18

Number Theory

• Number theory spans divisibility, modular arithmetic, residues, Diophantine equations, arithmetic functions, and algebraic number theory.The taxonomy also lists methods such as infinite descent, Pell-equation techniques, quadratic reciprocity, and parametrization of solutions.Source: B TAXONOMY OF TOPICS COMMONLY USED IN… · pp. 18–19

MathNet-Solve examples

• Olesya wins the cutting game when n is odd, while Andrew wins when n is even.For even n, Andrew’s strategy prevents either end from becoming a removable 2 × 2 square, controlling the parity of the remaining moves.Source: C DATASET EXAMPLES AND ADDITIONAL… · p. 20

MathNet-RAG examples

• An Expert-RAG example pairs a polynomial prime-divisibility problem with a sequence problem asking for infinitely many prime divisors.The retrieved sequence problem also states that each number appears at most 5,000 times.Source: C DATASET EXAMPLES AND ADDITIONAL… · p. 23
• Another Expert-RAG example retrieves a 12-coefficient real-root game for a target asking who wins a positive-coefficient polynomial game.The retrieved problem asks players to fill in coefficients so that the resulting equation has a real root.Source: C DATASET EXAMPLES AND ADDITIONAL… · p. 24

Dataset examples

• Figure 5 shows scanned Olympiad booklet pages from Ukraine, Argentina, Russia, Saudi Arabia, India, Japan, and Australia.Source: Figure 5 · p. 25

D PERFORMANCE SENSITIVITY TO IMAGE PRESENCE, LANGUAGE

• The analysis compares results by figure presence and sample language on MathNet-Solve-Test.Source: D PERFORMANCE SENSITIVITY TO IMAGE… · p. 26
• Top multimodal reasoning models gain substantially on text-plus-image problems compared with text-only problems.Table 8 reports modality-specific accuracy on the full evaluation set and text-only and text-plus-image subsets.Source: Table 8 · p. 26
• Model rankings are broadly consistent across languages, with Gemini-3.1-Pro and Gemini-3-Flash leading overall.Performance is weakest on Chinese and strongest on Italian and Portuguese.Source: Table 9 · p. 26

E LLM GRADERS VS HUMAN EXPERT GRADERS ON MATHNET-RAG

• The evaluation compares LLM grader judgments with human expert grading for Retrieval-Augmented Problem Solving on MathNet-RAG.It assesses automatic grading reliability, cross-model consistency, and alignment with human scoring.Source: E LLM GRADERS VS HUMAN EXPERT GRADERS… · p. 27
• For each model, performance is reported under zero-shot, embed-RAG, and expert-RAG settings.The comparison includes four LLM graders, their mean score, and human expert scores.Source: E LLM GRADERS VS HUMAN EXPERT GRADERS… · p. 27, Table 10 · p. 27
• LLM graders broadly track the solver ranking assigned by human experts on 35 MathNet-RAG problems.Table 10 reports accuracy for four LLM graders, their mean, and human expert scores across solvers and retrieval settings.Source: Table 10 · p. 27

F LLM USAGE AND PROMPTS

• The solution-generation prompt requests full reasoning and a boxed final response, with only the final expression boxed for answer problems and the entire proof boxed for proof problems.Source: F LLM USAGE AND PROMPTS · p. 28
• The grading prompt calls for rigorous IMO-standard assessment, scoring solutions on a 0–7 rubric that distinguishes fully correct, almost-correct, partial-progress, and incorrect work.It directs graders to inspect the official solution and student work line by line before assigning a score.Source: F LLM USAGE AND PROMPTS · pp. 28–29
• The grading prompt separates final-answer matching from proof quality, marking answer correctness inapplicable for proof-only problems or when no boxed answer is supplied.A correct boxed answer can count as correct even without a valid argument, while incorrect boxed answers can be marked incorrect despite mostly correct reasoning.Source: F LLM USAGE AND PROMPTS · p. 29
• The extraction prompt requires taxonomy-exact topic paths, key solution tools, and a symbol-free natural-language problem description.Source: F LLM USAGE AND PROMPTS · p. 30
• The extraction schema also classifies problem type and extracts a final answer when required, with a confidence score and JSON-only output.Source: F LLM USAGE AND PROMPTS · pp. 30–31

[SYSTEM PROMPT]

• The retrieval-variant prompt asks for mathematically equivalent problems using varied transformations and near-miss problems whose small changes require a different solving strategy.The instructions conflict on counts: they first request five of each, but the output rules specify exactly one equivalent and three near-miss variants.Source: [SYSTEM PROMPT] · pp. 32–33
• The variant prompt requires each item to include a problem, justification, and nonempty transformation tags in a strict JSON object.Source: [SYSTEM PROMPT] · pp. 32–33
• The authors used LLMs primarily to polish writing and assist with the project website and annotator interface.Source: [SYSTEM PROMPT] · p. 33