Source-linked AI summary
MathArena: Evaluating LLMs on Uncontaminated Math Competitions
Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, Martin Vechev
TL;DR
Existing math benchmarks can be contaminated and often omit proof-writing, motivating a more current and transparent evaluation framework. MATHARENA uses newly released competitions with automated or human grading, and its evaluations show strong performance on answer-based tasks but substantial room for improvement on proofs.
Problem
Widely available math benchmarks are vulnerable to contamination and generally do not evaluate proof-writing capabilities.
Method
MATHARENA evaluates models on newly released competitions through a pipeline that parses problems, collects solutions, and verifies final answers or proofs.
Results
Over 50 models were evaluated on 162 problems from seven competitions; leading models performed strongly, while IMO 2025 scores remained below 40%.
Takeaways & Limitations
A dynamic, public benchmark can support ongoing evaluation while retaining proof-based tasks and reducing contamination risk.
Takeaways & Limitations
Small competitions produce relatively wide confidence intervals, and answer-based benchmarks may soon become saturated as models solve nearly all but the hardest questions.
Abstract
from arXiv · showhide
The rapid advancement of reasoning capabilities in large language models (LLMs) has led to notable improvements on mathematical benchmarks. However, many of the most commonly used evaluation datasets (e.g., AIME 2024) are widely available online, making it difficult to disentangle genuine reasoning from potential memorization. Furthermore, these benchmarks do not evaluate proof-writing capabilities, which are crucial for many mathematical tasks. To address this, we introduce MathArena, a new benchmark based on the following key insight: recurring math competitions provide a stream of high-quality, challenging problems that can be used for real-time evaluation of LLMs. By evaluating models as soon as new problems are released, we effectively eliminate the risk of contamination. Using this framework, we find strong signs of contamination in AIME 2024. Nonetheless, evaluations on harder competitions, such as CMIMC 2025, demonstrate impressive reasoning capabilities in top-performing models. MathArena is also the first benchmark for proof-writing capabilities. On IMO 2025, top models achieve slightly less than 40%, demonstrating both notable progress and significant room for improvement. So far, we have evaluated over $50$ models across seven competitions, totaling $162$ problems. As an evolving benchmark, MathArena will continue to track the progress of LLMs on newly released competitions, ensuring rigorous and up-to-date evaluation of mathematical reasoning.
1 Introduction
Existing math benchmarks face contamination, limited transparency, and insufficient evaluation of proof-writing. MATHARENA addresses these gaps with newly released competitions and reports strong overall performance alongside weaker proof-based results.
- Publicly available competition datasets can be contaminated through training inclusion or benchmark-guided model selection, undermining accurate progress measurement.
- Private benchmarks reduce leakage but limit reproducibility, transparency, access equality, and scalability.HLE reportedly required a $500,000 prize pool to incentivize contributions.
- Final-answer benchmarks can miss proof-writing and multi-step reasoning abilities important to mathematical applications.
- MATHARENA evaluates newly released, pre-vetted competitions to reduce contamination while remaining public, reproducible, transparent, and continuously updated.It also includes proof-based competitions such as IMO 2025.
- Over 50 models were evaluated across seven competitions and 162 problems; leading models exceeded top-1% human performance, while IMO 2025 scores remained below 40%.
2 Related Work
Prior mathematics benchmarks trade off public accessibility, difficulty, contamination resistance, proof coverage, and evaluation scalability. MATHARENA combines dynamic competition-based evaluation with broader reasoning coverage.
- Public answer-based benchmarks are widely used but many are nearing saturation and draw on past competitions vulnerable to contamination.
- Private benchmarks offer greater difficulty but hinder reproducibility, transparency, fair comparison, and progress tracking.
- Formal proof benchmarks enable automatic verification but can underuse natural-language capabilities and remain limited by model support for formal systems.
- Dynamic benchmarks address contamination and evolving capabilities, but existing examples provide less difficult mathematics or do not evaluate rigorous proving.
- Perturbing existing problems reduces overlap without fully eliminating contamination because underlying reasoning patterns remain shared.
- MATHARENA instead introduces entirely new problems requiring new high-level reasoning strategies and supports scalable evaluation across problem formats.
3 MATHARENA
MATHARENA selects challenging competitions, evaluates models under standardized protocols, and uses distinct grading branches for final answers and proofs. Results are published with rankings and uncertainty estimates for transparent comparison.
- MATHARENA pipeline: The pipeline selects reputable competitions, extracts problems and solutions, evaluates selected models, grades outputs, and performs statistical post-processing.
- Competition selection: MATHARENA currently includes seven competitions totaling 162 problems, spanning final-answer and proof-based formats.
- Competition selection: Proof-based competitions receive expert human grading because scalable automated proof evaluation remains an open problem.The core proof competitions are USAMO, IMO, and Putnam.
- Competition selection: Project Euler evaluation focuses on recent problems and permits Python and C++ execution in a sandbox, with a limit of 20 executions per problem.
- Solution generation: Models are evaluated once per competition with provider-recommended hyperparameters, using boxed final answers or complete proofs according to problem type.
- Grading: Answer-based problems use custom rule-based parsing into structured sympy expressions, while proof-based problems use anonymized expert grading with partial-credit schemes.
- Leaderboard and post-processing: Reported results include public leaderboard rankings and confidence intervals for model rank and raw accuracy.
4 Evaluation
MATHARENA evaluates over 50 models on 162 problems across seven 2025 competitions, combining numerical and proof-based tasks with analyses of performance, contamination, cost, and variance. Latest models perform strongly on numerical competitions, while AIME 2024 shows contamination signals and IMO proof performance remains below 40%.
- Evaluation setup: Over 50 models were evaluated across seven 2025 competitions spanning 162 problems in algebra, combinatorics, geometry, and number theory.USAMO and IMO are proof-based; the other listed competitions require numerical final answers.
- Numerical answer competitions: GPT-5, GROK 4, and GROK 4 FAST achieved accuracies of 91.3%, 90.6%, and 90.4%, respectively, on the main final-answer evaluation.GROK 4 FAST was significantly cheaper, while GPT-OSS-120B led among open-source models.
- Performance over time: Models released before September 2024 achieved less than 10% accuracy on HMMT 2025, followed by substantial gains from chain-of-thought reasoning models and later iterations.Models released before the competition date are guaranteed to be uncontaminated in the release-date analysis.
- Data contamination: Most models scored 10%−20% above the human-aligned comparison line on AIME 2024, while the discrepancy was much smaller for HMMT.QWQ-PREVIEW-32B exceeded expected human-aligned performance by nearly 60%, indicating especially strong contamination evidence.
- Cross-competition correlation: AIME, HMMT, and CMIMC rankings correlated above 80%, indicating that results from one similar competition generalize well to the others.The high overall correlation supports using a single competition as representative of similar competitions.
- Natural-language proofs: GPT-5 achieved 38% on IMO 2025, or 16 points, below the 19 points required for a bronze medal.Because the number of proof-based problems is small, rank confidence intervals are wider and similar scores require caution.
5 Discussion
MathArena remains constrained by the limited number of sufficiently challenging annual competitions, producing wide confidence intervals, while its dynamic design supports continued relevance as models improve. The authors also note residual contamination concerns but judge the risk minimal because evaluations occur shortly after competitions conclude.
- Limitations: A limited number of sufficiently challenging annual competitions keeps MathArena small and produces relatively wide confidence intervals.The authors expect uncertainty to decrease as more competitions are added.
- Limitations: Answer-based competitions may soon become saturated because current state-of-the-art models already solve nearly all but the most difficult questions.The authors anticipate saturation possibly as early as 2026 and expect more challenging competitions to be needed.
- Broader impact: MathArena’s dynamic structure allows it to evolve alongside model capabilities and remain relevant as the field progresses.This distinguishes it from static benchmarks in the authors’ discussion.
- Limitations: Residual contamination is theoretically possible because closed-source models could be updated during the gap between competition release and evaluation.Evaluations occur only a few hours to at most a few days after competitions conclude, and the authors therefore believe contamination risks are minimal.
- Broader impact: Several major model providers have cited MathArena results in release notes, and the benchmark has attracted significant community interest.The cited providers include PHI-4-REASONING, GEMINI-2.5-PRO, and GROK-3.
6 Conclusion
The paper introduces MathArena as a benchmark using uncontaminated problems from human math competitions and supports it with a scalable evaluation pipeline. Evaluating over 50 LLMs on 162 problems from seven 2025 competitions, the authors report substantial progress while confirming contamination in prior benchmarks.
- 6 Conclusion: MathArena evaluates LLM mathematical performance using uncontaminated problems from human math competitions.The benchmark is designed around diverse, challenging competition problems.
- 6 Conclusion: The scalable pipeline parses problems and answers, samples model solutions, extracts final answers, and verifies correctness.This framework supports the benchmark’s evaluation process across competitions.
- 6 Conclusion: Over 50 LLMs were evaluated on 162 problems from seven math competitions held in 2025.The evaluation framework was used to assess mathematical capabilities across the included competitions.
- 6 Conclusion: The results show substantial progress in LLM mathematical capabilities while confirming the impact of data contamination in prior benchmarks.The conclusion reports both progress and contamination-related findings.
A Code and Data Availability and Reprodicibility
The benchmark’s code and data are made publicly available, with datasets typically published on HuggingFace under a license permitting non-commercial use and modification with attribution.
- A Code and Data Availability and Reprodicibility: MathArena’s code is publicly available on GitHub, and its datasets are typically published on HuggingFace.The repositories are https://github.com/eth-sri/matharena and https://huggingface.co/MathArena.
- A Code and Data Availability and Reprodicibility: The data use a CC-BY-NC-SA 4.0 license allowing non-commercial use and modification with attribution.The license was chosen after consultation with competition organizers.
B Additional Results
Additional results document the benchmark’s full model results, domain-specific performance, confidence intervals, release-date accuracy frontiers, token usage, and searches for possible contamination sources. Models struggle more with combinatorial and geometric problems than with algebraic and number-theoretic tasks, while Figure 6 relates accuracy to release date and cost.
- Full main results: Table 6 reports complete benchmark results across many proprietary and open-source LLMs, excluding poorly performing or superseded models.The exclusions apply to models superseded by better versions on CMIMC or BRUMO.
- Domain-specific results: Problems are manually classified into Algebra, Combinatorics, Geometry, and Number Theory for domain-specific analysis.Calculus is grouped under Algebra, while non-standard or word-based problems are categorized under Combinatorics.
- Domain-specific results: Nearly all models struggle more with combinatorial and geometric problems, whereas Algebra and Number Theory show stronger performance.The paper links the contrast to creativity, spatial intuition, synthetic reasoning, standard techniques, and symbolic manipulation.
- Confidence intervals per competition: Tables 9–12 provide confidence intervals for results from each competition using the method described in Section 3.These tables report uncertainty separately for the competition results.
- Token usage per model: Token usage includes prompt and response tokens averaged over final-answer competition problems, with GPT-5’s higher input count attributed to response-token caching in some cases.The measurements are reported in Table 13.
- Data contamination of past competitions: The authors searched online for problems similar to AIME 2025 and HMMT 2025 problems to identify possible contamination sources.The appendix lists examples from Quora, Math StackExchange, Wyzant, and Puzzling Stack Exchange.
- Timeline for all competitions: Figure 6 plots model accuracy against release date and cost, with red Pareto curves and black dotted competition-release markers.Each dot represents a released model; the red curves trace release-date-versus-accuracy Pareto-optimal points.
C Permutation Test for Rank Confidence Interval
The rank confidence interval procedure compares models pairwise using paired permutation tests on correctness outcomes from the same problems. Across competitions, each competition contributes equal weight to the aggregate statistic.
- Rank confidence intervals: 50% pair-flipping permutations generate the null distribution used to assess whether the observed performance difference is significant.The null hypothesis treats the paired outcomes as exchangeable, and the observed statistic is compared with its permutation distribution.
- Rank confidence intervals: Paired samples record whether two models answer the same problem correctly, enabling a paired comparison of their performance.The test statistic is the total performance difference between the two models for a single competition.
- Rank confidence intervals: When combining competitions, each problem receives weight 1/Nc, so every competition contributes the same total weight regardless of its size.The resulting aggregate statistic is then used to compute each model’s rank confidence interval.
- Prompts: The evaluation prompts explicitly require thorough, self-contained proofs rather than proof outlines or unsupported conclusions.The proof prompt instructs models to explain every step and acknowledge uncertainty when an intermediate result cannot be established.
E.1 AIME 2025 - Problem 15
The solution counts residue classes modulo 37 by decomposing each integer into base-27 components and separating cases according to divisibility by powers of 3. This yields N = 885735 and remainder 735 modulo 1000.
- Problem: The problem asks for the remainder of the number of ordered triples with coordinates at most 36 whose cube sum is divisible by 37.The triples consist of positive integers a, b, c satisfying a^3 + b^3 + c^3 ≡ 0 modulo 37.
- Counting method: Writing each n as u + 27v gives n^3 ≡ u^3 + 81u^2v modulo 2187, reducing the count to congruence conditions on u and v.The decomposition is unique for 1 ≤ n ≤ 36, with (u,v)=(0,0) representing n=36.
- Counting method: The three divisibility cases d = 1, 9, and 27 contribute N1 = 486, N9 = 54, and N27 = 9 base triples, respectively.These cases correspond to increasing divisibility constraints on the relevant residue components.
- Result: 885735 total triples produce remainder 735 modulo 1000.The total is assembled as N = 729 · 486 + 729 · 9 · 54 + 27^3 · 9.
E.2 HMMT Feb 2025 - Problem 20
The solution models the merging snakes through gap sizes and a Markov chain on partitions of 15. It reports an expected absorption time of 448/3 seconds, with quadratic scaling suggested by small-n computations.
- Problem: Fifteen snakes initially occupy every third arc on a circle of 45 arcs, leaving 15 gaps of two empty arcs each.The process ends when all snakes have merged into one.
- Markov-chain model: Scaling each gap by two makes the initial state a partition of total size 15 with all parts equal to 1.At each step, each current cluster independently moves one arc left or right.
- Markov-chain model: Merging occurs when an updated normalized gap reaches zero, so the state evolves as a Markov chain on partitions of 15.The number of parts equals the current number of snake clusters.
- Result: 448/3 seconds is the reported expected time until one snake remains.The text also states that small-n computations fit a quadratic form, consistent with symmetric random-walk gap dynamics.
E.3 IMO 2025 - Problem 1
The proposed solution reduces the sunny-line covering problem to covering a smaller triangular grid with sunny lines. It concludes that k ∈ {0, 1, 3}, although the grading record reports substantial proof gaps and a score of 2/7.
- Problem: The problem asks which nonnegative k allow n distinct lines to cover the triangular lattice Sn with exactly k sunny lines.A sunny line is not parallel to the x-axis, y-axis, or x + y = 0.
- Reduction: The solution classifies the remaining lines after using n − k non-sunny lines and reduces the task to covering a smaller triangular grid Sk with k sunny lines.It claims the non-sunny lines can be chosen as horizontals without changing the relevant reduced problem.
- Impossibility: Boundary counting gives 2k ≥ 3k − 3, so the proposed argument rules out every k ≥ 4.Each sunny line is claimed to meet at most two boundary lattice points of Sk.
- Characterization: The solution checks k = 0, 1, 2, 3 and concludes that only k ∈ {0, 1, 3} are possible.It claims constructions for k = 0, 1, and 3, while rejecting k = 2.
- Evaluation: The human grading record assigns 2/7 points and identifies missing or incorrect arguments in the reduction, boundary coverage, and finalization steps.The reported answer is correct, but the grading notes state that several required proof components were absent or unsupported.
Conclusion
The conclusion combines structural properties of f with a tight construction to establish the target constant, with the submitted solution receiving full credit.
- f(n) ≤ 4n for every bonza function f, giving the upper bound c ≤ 4.
- A construction satisfying the inequality tightly gives f(4)/4 = 4 and the matching lower bound c ≥ 4.
- If f is not the identity, then f(a) = 1 for every odd a, while f(a) is a power of 2 for every even a.
- The exponent of 2 in f(a) exceeds that in a by at most 2, expressed as ν2(f(a)) ≤ ν2(a) + 2.
- The submitted solution earned 7 / 7 points under human evaluation.