RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics

1 Introduction

• Current benchmarks focus on coursework, competitions, formal proofs, or expert-crafted extreme challenges, which do not capture much of the structure and topic diversity of research mathematics.Research mathematics rarely relies on formal proofs and is not exclusively designed to maximize difficulty.Source: §1 · p. 1
• REALMATH combines research-paper and forum content, correctness checking through verifiable answers, and a refreshable dataset intended to mitigate contamination.Its pipeline extracts statements from sources such as arXiv and Stack Exchange; evaluation checks answers rather than assessing proofs.Source: §1 · p. 2
• The benchmark converts mathematical statements into question-answer pairs while preserving context and relevant definitions and notation.The introduction illustrates this with a theorem about counting permutations, converted into a question with a verifiable answer.Source: §1 · p. 2
• Frontier LLMs perform more strongly on REALMATH research mathematics than on deliberately challenging competition or FrontierMath datasets, though they still struggle with the most advanced problems.The paper frames statement verification as a useful signal about mathematical-assistant potential, not an assessment of proof generation or verification.Source: §1 · p. 2

Math benchmarks sourced from exams and competitions

• Coursework and competition benchmarks use abundant problems with solutions, while newer sets draw on advanced contests such as the IMO, AIME, and Putnam.Early datasets include GSM8K [5], GHOSTS [7], and MATH [11]; advanced contest examples include the IMO [2] [15] and Putnam.Source: §2 · p. 3

Math benchmarks sourced from research experts

• FrontierMath and part of HLE [16] use extremely challenging expert-crafted problems, but cover a narrow slice of practice and require labor-intensive curation.FrontierMath keeps its test set private to guard against contamination, limiting reproducibility; REALMATH spans a broader range and can be refreshed automatically.Source: §2 · p. 3

Benchmarks for theorem proving

• Formal-proof benchmarks such as LeanDojo and MiniF2F [25] test proof generation in interactive theorem provers, unlike much research mathematics that is not fully formalized.Source: §2 · p. 3

Benchmarks for LLM capabilities “in the wild”

• REALMATH joins a broader shift toward evaluating real-world tasks, exemplified by software-engineering and cyberoffensive benchmarks rather than curated competitions.Examples include SWE-bench, Lancer, BaxBench, and exploit-focused benchmarks.Source: §2 · p. 3

3.1 Design Criteria

• REALMATH targets practical research and learning tasks, uses single exact numerical or symbolic answers for automated verification, and excludes ambiguous, multiple-answer, or non-constructive proof-focused statements.The benchmark also collects from public sources so fresh evaluation sets can be gathered after model releases to mitigate contamination.Source: §3.1 · p. 3

3.2 An Automatic and Refreshable Data Collection Pipeline

• The arXiv pipeline retrieves papers, parses LaTeX, selects constructive fixed-answer theorems, converts them into question-answer pairs, and filters trivial questions.From 4,000 papers, 3,922 had usable sources; 407 theorems were selected, yielding 401 question-answer pairs and finally 280 samples.Source: §3.2 · p. 4
• The pipeline can be refreshed with new mathematical problems, but its data quality depends on source materials and LLM judgments that may introduce inaccuracies.Stack Exchange posts can be poorly formulated, mathematically inaccurate, or unanswered, so obtaining high-quality samples requires filtering more submissions.Source: §3.2 · p. 5
• More than 94% of the pipeline’s arXiv samples were high-quality, and manual review found approximately 6% of 633 judge-selected theorems did not meet quality criteria.The pipeline produced these samples without human annotation before the manual quality review.Source: §3.2 · pp. 5–6
• The construction assumes arXiv theorems are correct, although papers may be unreviewed and errors or ambiguous claims may occur.The authors note that incorrect statements could add noise, while ambiguity may reflect mathematics “in the wild.”Source: §3.2 · p. 6

4.1 Experimental Setup

• For CS.arXiv and Math.arXiv, models receive the generated question and preceding paper context; for Math.StackExchange, they receive only the generated question.Table 2 is captioned as a summary of the final evaluation datasets.Source: §4.1 · p. 6, Table 2 · p. 6

4.2 Main Experimental Results

• o3 leads on CS.arXiv and Math.arXiv, while o4-mini slightly leads on Math.StackExchange; DeepSeek-R1 and Gemini-2.5-pro score 25.6%–32.5% on arXiv and 60.9%–62.2% on StackExchange.The reported accuracy ranges are for the two mid-tier models collectively.Source: §4.2 · p. 6
• 97.5% accuracy on easy questions falls to 81.4% on medium and 27.9% on hard questions for o3, with similar declines for Gemini-2.5-Pro and DeepSeek-R1.Figure 4 presents accuracy by difficulty alongside the distribution of difficulty levels.Source: §4.2 · p. 7, Figure 4 · p. 7
• o3 exceeds 60% accuracy in Representation Theory, Number Theory, and Analysis of PDEs, but reaches only 23.8% in Machine Learning; Gemini 2.5-pro’s strongest domains include Machine Learning and Optimization and Control.The two models’ profiles differ: Gemini’s strongest domains are among o3’s weakest, while o3 performs well on highly theoretical tasks.Source: §4.2 · pp. 7–8, Figure 5 · p. 8

4.3 Additional Analysis

• Most analyzed errors are flawed reasoning, followed by conceptual misunderstanding and missed insights, pointing to difficulty maintaining multistep logical coherence.The error analysis compares o3, Gemini 2.5-pro, and DeepSeek R1 on Math.arXiv.Source: §4.3 · p. 8, Figure 6 · p. 8
• 21.6% accuracy without context versus 42.3% with full context is reported for o4-mini on CS.arXiv, although models still perform rather well without context.The authors note that some theorem notation or concepts can be understood or inferred in isolation.Source: §4.3 · pp. 8–9, Figure 7 · p. 9
• Models performed better on 2025 than 2022 Math.arXiv samples; the authors suggest familiarity with topics near training cutoffs as a likely explanation.The comparison uses papers before and after the respective cutoffs of GPT-4o-mini, Llama-3.1-405B, and Claude-3.5-Sonnet.Source: Figure 5 · p. 8, §4.3 · p. 9
• Fine-tuning GPT-4o-mini on 500 Math.arXiv samples did not improve accuracy on the remaining 133, which the authors interpret as evidence that simple fine-tuning may not supply needed specialized skills.Source: §4.3 · p. 9

• Leading models achieve 43–49% accuracy on Math.arXiv papers and up to 70% on Math.StackExchange questions, while still struggling with the most challenging problems.Source: §5 · p. 9
• REALMATH is a refreshable data-collection pipeline intended to reflect current mathematical practice and protect against contamination.The authors suggest it may support evaluating and improving LLMs’ mathematical capabilities and broader use as collaborative tools.Source: §5 · p. 9

Limitations

• The authors report discussing limitations in source-data quality, frontier-model performance, and LLM-judge reliability.Source: NeurIPS Paper Checklist · p. 13

Reproducibility and experimental details

• The authors report providing the datasets, retrieval code, and LLM prompts, with pipeline, setup, and results described in the paper.The checklist also says the experimental details needed to understand the results are specified.Source: NeurIPS Paper Checklist · pp. 14–15

Statistical significance

• The checklist answers no on reporting error bars or other statistical-significance information; results are described as illustrating dataset difficulty composition and model trends.Source: NeurIPS Paper Checklist · p. 15

Societal impact

• The checklist says the paper discusses positive and negative societal impacts and the benchmark's relevance to evaluating LLM contributions to research communities.Source: NeurIPS Paper Checklist · p. 16

Human subjects

• The study is reported to use no human subjects, relying on arXiv and Mathematics Stack Exchange samples without human intervention in experimental work.Source: NeurIPS Paper Checklist · p. 18

A.1 Prompts for Judge Models

• The o3-mini prompts filter theorems for unique answers and generate question-answer pairs from retained theorems.Source: §A.1 · p. 20
• The filtering criteria favor unambiguous fixed answers and exclude ambiguous results, inequalities, and asymptotic statements such as Big-O claims.Source: §A.1 · pp. 20–21
• The generation prompt turns verified theorems into precise graduate-level questions with uniquely determined answers and without adding assumptions or changing theorem scope.Source: §A.1 · p. 21

A.2 Prompts for Evaluation Model

• The GPT-4o evaluation prompt checks whether generated answers are mathematically correct and equivalent to ground truth, treating correctness strictly rather than as a matter of formatting or style.Source: §A.2 · p. 23

A.3 Manual Review

• Human review removed 5.2% of Math.arXiv samples and 7.5% of CS.arXiv samples; 35 and 9 rejects, respectively, lacked unique answers.These were theorems or QA pairs that earlier pipeline stages had not filtered out.Source: §A.3 · p. 23

A.4 Category Distribution Across Construction Stages

• Combinatorics (math.CO) grows increasingly dominant through construction, because those papers often contain many theorems suited to Q&A.Figure 9 tracks category percentages across construction stages for Math.arXiv papers collected from May to September 2022.Source: §A.4 · pp. 23–24, Figure 9 · p. 24

A.5 Pipeline for Mathematics Stack Exchange

• For Math.StackExchange, authors retrieve tagged questions and top answers, convert posts to statement-proof form, manually verify fixed correct answers, then generate and filter QA pairs.Source: §A.5 · p. 24

A.6 Optional LaTeX Conversion

• Optional LLM-based LaTeX conversion expands macros into self-contained code, and authors report no harm to QA generation or quality filtering.Source: §A.6 · p. 25

• The evaluation was repeated five times, with mean and standard deviation reported.Source: §B.1 · p. 25
• Fine-tuning GPT-4o-mini on the dataset produced no statistically significant accuracy improvement across five evaluation runs.The authors report the mean and standard deviation across the five runs.Source: Figure 10 · p. 25, §B.1 · p. 25
• Training loss fell rapidly and stayed low, while test loss fluctuated and rose over time, a pattern the authors say may explain failed generalization.Source: §B.1 · p. 25, Figure 11 · p. 25

• 69.4% of CS.arXiv samples were classified Hard under a four-model scheme, meaning none of the older models answered them correctly.Medium means one or two models answered correctly; Easy means three or four did.Source: §B.2 · p. 26, Figure 12 · p. 26
• On CS.arXiv, DeepSeek-R1 accuracy falls from 100% on Easy samples to 11.7% on Hard samples; o3 is an exception to the broader downward trend.Source: §B.2 · p. 26
• On CS.arXiv, o3 scores 75% on Medium versus 71.4% on Easy; the authors relate this to the larger Medium share, 18% versus 12.6% Easy.Source: §B.2 · p. 26
• Math.StackExchange difficulty is more spread, with Medium questions most common, unlike the predominantly Hard Math.arXiv and CS.arXiv distributions.Source: Figure 13 · p. 27, §B.2 · p. 27

B.3 Performance Across Categories

• On CS.arXiv, o3 outperforms Gemini-2.5-Pro in every category except cs.LO, where accuracies match; o3 favors applied areas, while Gemini favors theoretical topics.Source: Figure 14 · p. 28, §B.3 · p. 28
• On Math.StackExchange, o3 exceeds Gemini-2.5-Pro in every category; both are strongest on Limits, with more than 80% accuracy, and weaker on advanced Complex Analysis.Source: Figure 14 · p. 28, §B.3 · p. 28

B.4 Category Distribution of Questions

• Domain and tag tables provide category breakdowns, and categories are nonexclusive, so multi-tag papers or questions count under each applicable category.Source: §B.4 · p. 29, Table 4 · p. 29, Table 5 · p. 30
• Combinatorics (math.CO) is among the most frequent domains in both arXiv collections and likely supplies many fixed-answer problems suited to automated verification.Source: §B.4 · p. 29

B.5 How Context Improves LLM Performance on Mathematical Problems

• Without context, LLMs may fail because paper-specific notation is undefined or because intermediate results and proof examples needed for a theorem are absent.Manual verification identified both failure modes; adding context improves the models' approach to correctly proving questions in the latter cases.Source: §B.5 · p. 32
• Figure 15 illustrates CS.arXiv questions that models solve correctly with relevant context but answer improperly without it.The examples include a question whose gcover(BF(r)) notation is ambiguous without a prior definition.Source: Figure 15 · p. 32, §B.5 · p. 32