Source-linked AI summary
FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI
Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de Oliveira Santos, Olli Järviniemi, Matthew Barnett, Robert Sandler, Matej Vrzala, Jaime Sevilla, Qiuyu Ren, Elizabeth Pratt, Lionel Levine, Grant Barkley, Natalie Stewart, Bogdan Grechuk, Tetiana Grechuk, Shreepranav Varma Enugandla, Mark Wildon
TL;DR
FrontierMath addresses the need for advanced, contamination-resistant mathematics benchmarks by introducing original problems spanning most branches of modern mathematics with automated verification. The benchmark was created and vetted by expert mathematicians, and leading AI models solve under 2% of its problems. Its scope is constrained to automatically verifiable answers and hour-scale problems rather than proof-writing or longer-term mathematical research.
Problem
Existing benchmarks are nearing saturation and provide limited evaluation of advanced mathematical reasoning, while contamination can inflate apparent model performance.
Method
FrontierMath combines over 60 mathematician-created, unpublished problems spanning modern mathematics with automated verification and expert review.
Results
Leading AI models cannot solve over 2% of FrontierMath problems, while expert mathematicians assessed sampled problems as exceptionally difficult.
Takeaways & Limitations
FrontierMath provides a rigorous benchmark for measuring advanced mathematical reasoning while reducing risks from benchmark saturation and training-data contamination.
Takeaways & Limitations
Automatic verification excludes proof-writing and open-ended exploration, and hour-scale problems do not test research requiring weeks, months, or years.
Abstract
from arXiv · showhide
We introduce FrontierMath, a benchmark of hundreds of original, exceptionally challenging mathematics problems crafted and vetted by expert mathematicians. The questions cover most major branches of modern mathematics -- from computationally intensive problems in number theory and real analysis to abstract questions in algebraic geometry and category theory. Solving a typical problem requires multiple hours of effort from a researcher in the relevant branch of mathematics, and for the upper end questions, multiple days. FrontierMath uses new, unpublished problems and automated verification to reliably evaluate models while minimizing risk of data contamination. Current state-of-the-art AI models solve under 2% of problems, revealing a vast gap between AI capabilities and the prowess of the mathematical community. As AI systems advance toward expert-level mathematical abilities, FrontierMath offers a rigorous testbed that quantifies their progress.
1 Introduction
FrontierMath addresses the saturation and contamination limits of existing mathematics benchmarks with original, expert-created problems spanning modern mathematics. Its problems require substantial expert effort, while leading AI models solve under 2% of them.
- Existing benchmarks largely assess high-school and early undergraduate mathematics, where state-of-the-art models achieve near-perfect performance.
- FrontierMath was created with over 60 mathematicians and contains exclusively new, previously unpublished problems spanning most branches of modern mathematics.
- FrontierMath problems demand deep theoretical understanding, creative insight, and specialized expertise, often requiring multiple hours from expert mathematicians.
- Under 2% of FrontierMath problems are solved by current state-of-the-art AI models, even with multiple attempts.
- Four prominent mathematicians, including three Fields Medalists, uniformly characterized sampled problems as exceptionally challenging and time-intensive.
2 Data collection
FrontierMath problems were contributed and reviewed by a broad community of mathematicians under requirements for originality, verifiability, guessproofness, and computational tractability. Automated checking and multi-stage review support scalable evaluation, while the authors report quality and difficulty-rating limitations.
- Problem creation: Over 60 mathematicians from more than a dozen countries contributed problems spanning diverse mathematical fields and career stages.
- Problem creation: Contributors designed problems to be original, automatically verifiable, resistant to guessing, and computationally tractable.
- Review and verification: Each problem included a detailed solution, verification script, metadata, and peer review by a relevant domain expert.
- Review and verification: Automated verification checks submitted Python objects against exact or symbolic answer criteria, enabling efficient and scalable evaluation without human grading.
- Quality assurance: Two of 35 additionally reviewed questions contained author-provided incorrect answers missed in first review, corresponding to an estimated posterior error rate of approximately 6.9%.
- Quality assurance: Reviewers identified missing hypotheses, easier guessing strategies, inconsistent difficulty ratings, and other issues requiring continued quality control.
3 Dataset composition
FrontierMath spans broad contemporary mathematics, with core fields frequently combined across problems and more than 200 techniques represented. Its composition emphasizes number theory, combinatorics, and group theory while retaining methodological diversity.
- The benchmark covers most top-level MSC2020 subject codes across foundational and specialized research domains.
- Number theory and combinatorics appear together most frequently, with 13% of problems requiring both subjects.
- Number theory, combinatorics, and group theory appear in 44%, 39%, and 22% of problems, respectively, and each combines with more than a dozen other domains.
- More than 200 techniques occur in the solutions, while generating functions, recurrence relations, and special functions each appear in less than 5% of problems.
- The most frequently co-occurring techniques appear together in at most 3 problems, indicating that solutions typically use unique combinations of approaches.
4 Evaluation
FrontierMath evaluates language models through direct answering or iterative code-assisted experimentation. Across six leading models, performance remains below 2%, with substantial variability in repeated trials and experimentation behavior.
- The framework lets models analyze problems, execute Python implementations, receive outputs and errors, and refine solution strategies iteratively.
- Six leading language models achieved less than a 2% success rate on the full FrontierMath benchmark.
- Repeated trials on four problems solved at least once showed high variability, with only one model-question pair solved on all five runs.
- A correct answer does not necessarily indicate correct reasoning, because simple simulations sometimes enabled accurate guesses without deeper mathematical understanding.
- o1-preview averaged 1.29 responses per problem, whereas Grok 2 Beta averaged 3.81 responses.
- Gemini 1.5 Pro 002 reached the 10,000-token limit in 16.8% of questions, while Claude 3.5 Sonnet, GPT-4o, and Grok 2 Beta exceeded it in more than 45% of attempts.
5 Related work
Existing mathematics benchmarks range from elementary reasoning to graduate-level and formal theorem-proving tasks, but FrontierMath targets research-level evaluation with novel problems and broader access.
- GSM8K and MATH primarily assess elementary to undergraduate mathematics, and state-of-the-art models now achieve near-perfect performance on many such benchmarks.
- ARB, GHOSTS, and related benchmarks extend coverage toward university, graduate, and natural-language mathematical reasoning.
- Putnam-AXIOM reduces o1-preview accuracy from 50% to 34% on generated rather than unmodified problems, but relies on variants of existing problems.
- AIMO creates novel olympiad-level problems but requires open-weight runtime models, excluding frontier closed-source systems from official evaluation.
- FrontierMath uses unpublished research-level problems, automated verification, and evaluation applicable to closed-source frontier models.
6 Interviews with mathematicians
Interviews with four prominent mathematicians characterize FrontierMath’s research problems as exceptionally challenging and requiring substantial expertise and time. They anticipate human-AI collaboration before fully autonomous solutions, while emphasizing limited specialized training data as a major barrier.
- Four mathematicians, including three Fields Medalists, provided expert perspectives on FrontierMath’s difficulty, significance, and prospects.
- The interviewees described the research problems as exceptionally challenging, with the hardest requiring deep domain expertise and significant time investment.
- Tao expected FrontierMath to resist AI systems for several years because the problems require substantial domain expertise and relevant training data is insufficient.
- Chen and Tao anticipated human experts working with AI could tackle FrontierMath problems within around three years, before fully autonomous systems could do so.
- The mathematicians identified extremely limited specialized training data and branch-specific expertise as major barriers to solving FrontierMath problems.
- Potential uses include assisting with routine research tasks such as checking conjectures and saving researchers time.
7 Discussion
FrontierMath evaluates advanced mathematical reasoning with difficult, broad-domain problems and novel data, but its practical design leaves important research capabilities and current model comparisons incompletely measured.
- FrontierMath spans most branches of modern mathematics and emphasizes problems requiring multiple hours of concentrated effort by expert mathematicians.
- Leading AI models solve under 2% of FrontierMath problems, while four expert mathematicians uniformly assessed a ten-problem sample as exceptionally difficult.
- FrontierMath addresses benchmark saturation and contamination by using exceptionally challenging research-level problems that are novel and unreleased.
- Automatically verifiable numerical answers exclude proof-writing and open-ended exploration, which are significant parts of modern mathematical research.
- Limiting problems to hours rather than weeks, months, or years makes the benchmark practical but misses crucial long-term research skills.
- Current models solve only a small fraction of problems, temporarily limiting FrontierMath’s usefulness for evaluating relative model performance.
8 Future work
Future work will expand and improve FrontierMath’s problem collection and evaluation methodology while maintaining its public, expert-vetted role. The benchmark’s examples span difficulty levels and mathematical domains represented by MSC2020 codes.
- 8 Future work: Future releases will add rigorously vetted problems using the established collection and review process, with improved quality assurance for answer accuracy and difficulty ratings.
- 8 Future work: Planned evaluations will test longer token limits, more experiments per problem, and multiple runs to report statistics and confidence intervals.
- 8 Future work: Epoch AI plans to maintain FrontierMath publicly and continue working with mathematicians on problem creation, quality assurance, difficulty validation, and transparent evaluations.
- A Sample problems and solutions: The sample-problem section presents five problems selected across difficulty quintiles, ordered from research-level number theory to undergraduate algebraic geometry.
- A Sample problems and solutions: Figure 7 identifies MSC2020 codes 11, 14, and 20 as number theory, algebraic geometry, and group theory and generalizations.
A.1.2 Solution
The solution computes a density of primes using field-theoretic formulas, quasi-multiplicativity, and finite corrections, while relying on GRH to control error terms. With T = 10^5 and x = 10^8, the resulting answer has error at most 2.5 · 10^-8.
- Background: Artin’s primitive-root question asks when a has order p − 1 modulo infinitely many primes, and the relevant result is conditional on a generalized Riemann hypothesis.The density problem considered here extends this motivation to comparisons and prescribed relations between multiplicative orders.
- Method: The method replaces ord_p(a) with ord_p,x(2), eliminating error terms and reducing inclusion-exclusion to finitely many asymptotic formulas.This replacement is made specifically to avoid requiring sharp error bounds.
- Method: The density formulas are quasi-multiplicative, with explicit O(1)-time functions and sums that can be evaluated in O(a) time.The formulas depend on local valuations, with special corrections at q = 2 and q = 3.
- Method: The naive independence heuristic is almost correct, but requires a small finite number of revisions because of correlations at q = 2 and q = 3.At these primes, the relevant field degrees lose a factor of 4.
A.2 Sample problem 2 — high-medium difficulty
The problem is solved by translating irreducible components of a fibre product into monodromy orbits and constraining the degree-19 permutation group. This forces a scaled Chebyshev polynomial, whose value at 19 is computed explicitly.
- Problem: The problem asks for an odd, monic, real degree-19 polynomial with linear coefficient −19 and at least 3 but not all linear irreducible components.The requested value is p(19).
- Result: 1876572071974094803391179 is the computed value of p(19).The polynomial has 10 irreducible components, including the diagonal.
- Method: Irreducible components of the fibre product correspond to orbits of the monodromy group on ordered pairs.The associated covering has degree 19^2.
- Method: Riemann–Hurwitz and fixed-point bounds constrain the branch structure, while Burnside’s theorem and Sylow’s theorem force the order-19 subgroup to be normal.This rules out the doubly transitive case and narrows the possible monodromy.
- Result: The only possibility is affine equivalence to the degree-19 Chebyshev polynomial T19, whose action has 10 irreducible components including the diagonal.The remaining components have bi-degree (2,2).
- Result: p(x) = 2T19(x/2) satisfies the imposed parity, monicity, reality, and linear-coefficient conditions.These conditions determine the scaling of the affine-equivalent Chebyshev polynomial.
A.3 Sample problem 3 — medium difficulty
This problem determines when a recurrence sequence extends continuously from Z to Z_p by analyzing its characteristic roots p-adically. The candidate prime is narrowed through divisibility conditions to p = 9811.
- Problem: The task asks for the smallest prime p ≡ 4 mod 7 permitting a continuous extension of n ↦ a_n from Z to Z_p.The sequence is defined by a fourth-order recurrence with initial values a_i = i for 0 ≤ i ≤ 3.
- Method: The solution expresses a_n through the characteristic roots α_i and studies their p-adic valuations in an extension of Q_p.The roots lie in an extension of degree at most 4! over Q_p.
- Result: p = 9811 is the only possible prime after factoring the relevant divisibility conditions and imposing p ≡ 4 mod 7.The factorization of 198130309625 − 4 yields candidate factors 37, 673, 811, and 9811.
- Method: When every root satisfies v_L(α_i − 1) > 0, p-adic exponential and logarithm formulas produce the required continuous extension.The resulting extension is unique because Z is dense in Z_p.
- Limitation: For other primes, separate residue-class extensions are incompatible when the roots are not congruent to 1, preventing a single continuous extension in that setting.The construction extends each residue class modulo p − 1 separately, but these functions are not compatible.
A.5 Sample problem 5 — low difficulty
The paper illustrates benchmark problems spanning finite-field point counting, algebraic structure, recurrence sequences, and large-scale symbolic computation. Solutions combine mathematical reasoning with executable Python experiments and automated answer submission.
- Problem: The finite-field problem counts nonzero points on a curve over F_{5^18} up to scaling.The solution uses counts over F5, F25, and F125 to identify a pattern and then applies the Weil conjectures.
- Solution: The observed counts 6, 26, and 126 over F5, F25, and F125 lead to the formula |C(F_{5^18})| = 5^18 + 6 · 5^9 + 1.The curve is verified to be smooth at p = 5 and to have genus 3.
- Prompts: The prompts encourage mathematical reasoning, code-based experiments, and strict formatting for final executable answers.Models are instructed to verify mathematical steps explicitly with Python.
- Prompts: Continuation prompts provide execution errors, outputs, or timeout information and urge new approaches when an answer is not yet correct.A final prompt requests the best attempt when the token limit is reached.
- Evaluation: Final submissions must contain a specific comment, pickle the answer to final_answer.p, and run correctly within the prescribed time limit.The script must be self-contained and execute in less than 20 seconds for the relevant prompt requirement.
B.2 Analysis of problems solved at least once
The analysis isolates four problems solved at least once by a model and evaluates model reliability across repeated trials.
- Four problems were selected because at least one model solved each of them.
- Each model was tested five times on each of the four problems.
- Table 2 reports model success rates out of five runs for the four test problems.
B.3 Percentage of unsolved problems across benchmarks
FrontierMath remains substantially harder for current AI models than other evaluated mathematics benchmarks, with even repeated evaluation producing very low accuracy.
- Figure 2 compares the fraction of unsolved problems for FrontierMath with seven prominent mathematics benchmarks.
- The comparison uses one minus each benchmark’s top reported pass@1 accuracy.
- No model achieved higher than 2% mean accuracy across eight FrontierMath evaluations.
- The best model achieved around 6% pass@8 accuracy, compared with around 2% for Grok 2 Beta.