Source-linked AI summary
Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models
Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang, Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, Baobao Chang
TL;DR
Existing benchmarks have become less challenging for rapidly improving LLMs, creating a need for a mathematics-only Olympiad-level evaluation set. The paper introduces Omni-MATH, a 4,428-problem benchmark with hierarchical domain and difficulty annotations plus model-based evaluation, and finds that advanced models still perform modestly on its problems. It also reports that common Best-of-N scaling does not consistently improve performance.
Problem
Existing benchmarks increasingly lose their ability to differentiate LLM mathematical capabilities as models improve, while prior challenging benchmarks often do not isolate mathematics.
Method
The paper constructs Omni-MATH from 4,428 competition-level problems, classifies them across 33 sub-domains and more than 10 difficulty levels, and evaluates solutions with GPT-4o and Omni-Judge.
Results
OpenAI o1-mini achieves 60.54% accuracy and o1-preview 52.55% on Omni-MATH, while the highest open-source model achieves 36.2%.
Takeaways & Limitations
Omni-MATH remains challenging for current models and reveals weaker performance in discrete mathematics alongside ineffective Best-of-N scaling.
Takeaways & Limitations
Best-of-N scaling did not yield consistent improvements, potentially because the reward model has limited Olympiad-level supervision and the policy model struggles to search for correct solutions.
Abstract
from arXiv · showhide
Recent advancements in large language models (LLMs) have led to significant breakthroughs in mathematical reasoning capabilities. However, existing benchmarks like GSM8K or MATH are now being solved with high accuracy (e.g., OpenAI o1 achieves 94.8\% on MATH dataset), indicating their inadequacy for truly challenging these models. To bridge this gap, we propose a comprehensive and challenging benchmark specifically designed to assess LLMs' mathematical reasoning at the Olympiad level. Unlike existing Olympiad-related benchmarks, our dataset focuses exclusively on mathematics and comprises a vast collection of 4428 competition-level problems with rigorous human annotation. These problems are meticulously categorized into over 33 sub-domains and span more than 10 distinct difficulty levels, enabling a holistic assessment of model performance in Olympiad-mathematical reasoning. Furthermore, we conducted an in-depth analysis based on this benchmark. Our experimental results show that even the most advanced models, OpenAI o1-mini and OpenAI o1-preview, struggle with highly challenging Olympiad-level problems, with 60.54\% and 52.55\% accuracy, highlighting significant challenges in Olympiad-level mathematical reasoning.
1 INTRODUCTION
Existing mathematical benchmarks are becoming less discriminative as LLMs improve, motivating Omni-MATH, a text-only Olympiad-level benchmark with broad coverage and challenging evaluation. The benchmark exposes substantial remaining weaknesses in advanced models’ mathematical reasoning.
- Existing benchmarks such as GSM8K and MATH are increasingly losing their ability to differentiate model capabilities as LLMs evolve.
- Omni-MATH contains 4,428 competition-level problems focused exclusively on text-only mathematical reasoning.
- Its hierarchical organization spans 33 sub-domains and more than 10 difficulty levels, supporting analysis across mathematical disciplines and complexity levels.
- Omni-MATH supplements rule-based evaluation with GPT-4o assessment and the open-source Omni-Judge verifier.Omni-Judge achieves over 91% consistency with GPT-4o and 86% consistency with human judgments.
- 60.54% and 52.55% accuracy are achieved by OpenAI o1-mini and o1-preview, respectively, while the highest open-source score is 36.2%.
- LLMs perform marginally better on algebra than discrete mathematics, and Best-of-N test-time scaling is ineffective for Olympiad-level problems.
2 OMNI-MATH BENCHMARK
Omni-MATH is constructed from diverse competition sources through filtering, verification, annotation, and structured classification. Its benchmark design addresses variation in problem difficulty, mathematical domain, answer format, and evaluation reliability.
- Omni-MATH construction combines data generation, manual annotation, domain classification, difficulty classification, and GPT-4o-based and Omni-Judge evaluation.
- 2.1 DATA COLLECTION AND ANNOTATION: Problems are collected from official competition sources, AoPS Wiki, and AoPS forums, with different verification procedures depending on source reliability.
- 2.1 DATA COLLECTION AND ANNOTATION: Graduate and doctoral annotators manually verify dataset solutions and answers after automated construction.
- 2.1 DATA COLLECTION AND ANNOTATION: 92.7% accuracy was obtained through cross-validation of 1,100 forum-sourced problems, rising to 97.3% after manual sampling and exclusion of inconsistent cases.
- 2.2 DIFFICULTY CLASSIFICATION: Problem difficulty is assessed at the instance level using AoPS ratings because competition-level difficulty and popularity alone are insufficient.
- 2.2 DIFFICULTY CLASSIFICATION: The observed difficulty distribution across competitions closely aligns with prior tiered competition classifications, supporting the reliability of the domain classification.
- 2.3 DOMAIN CLASSIFICATION: A hierarchical domain tree based on mathematics guidebooks enables accuracy analysis at different levels of granularity.
- 2.4 EVALUATION: GPT-4o evaluates whether model-generated solutions align with reference answers because diverse answer formats make fixed answer-level rules difficult to apply.
3 OLYMPIAD-LEVEL MATH EVALUATION ON EXISTING LLMS
The paper evaluates 15 mathematically capable models on Omni-MATH, finding that Olympiad-level performance remains limited and varies across domains.
- 15 models are evaluated on Omni-MATH using GPT-4o to assess whether generated outputs match correct answers.
- 60.54% accuracy is achieved by OpenAI o1-mini, while OpenAI o1-preview reaches 52.55% and the strongest vanilla model reaches 36.2%.
- Models perform better in algebra, calculus, and number theory but show significant weaknesses in discrete mathematics.
- Best-of-N scaling does not consistently improve performance, potentially because reward and policy models struggle with Olympiad-level reasoning.
4 ANALYSIS
The analysis examines contamination, evaluation reliability, data selection, and difficulty consistency to characterize model behavior on Omni-MATH.
- Most models exhibit some data leakage, but contaminated-and-correct samples are extremely rare, indicating little impact on the conclusions.
- GPT-4o shows strong consistency with human annotations and is therefore treated as a reliable proxy for human judgment.
- Algebra interferes with applied mathematics when used as in-context demonstration data.
- Greater domain similarity between demonstrations and target problems increases performance consistency, with same-first-level-domain data improving most over random selection.
- Increasing demonstration difficulty generally improves performance, except for a slight decline at T4.
- The difficulty-consistency measure models four interval accuracies and finds that all models’ accuracy declines as difficulty increases.
5 RELATED WORK
Earlier benchmarks include Olympiad-related mathematics, but Omni-MATH focuses exclusively on text-only Olympiad reasoning and adds finer-grained difficulty analysis.
- GSM8K emphasizes practical application problems, while MATH covers university-level and introductory competition-level mathematics.
- Prior Olympiad-oriented resources include geometry, high-school competition, and comprehensive multi-field benchmarks.
- Omni-MATH addresses the scarcity of text-only Olympiad data by categorizing competition problems across multiple difficulty levels.
6 CONCLUSION
The paper presents Omni-MATH as a challenging Olympiad-level benchmark and analyzes model errors, contamination, annotation, and dataset construction.
- OpenAI o1-mini reaches 60.5% accuracy and the strongest open-source model reaches 36.2%, showing that the highest difficulty level remains challenging.
- Process-level assessment uses GPT-4o to classify incorrect reasoning steps as unrelated, accumulation, calculation, logic, or other errors.
- Logical errors are the most common error type among strong inference models, while OpenAI o1-mini has fewer errors across all categories.
- Annotation included filtering multi-part or proof-based questions, verifying answers, and correcting incorrect original answers.
- The dataset combines contest-page, AoPS Forum, and AoPS Wiki questions, with 3,295 contest-page questions and 298 AoPS Wiki questions reported.
- The final dataset contains 4,428 questions after removing 53 questions from an initial 4,481.
C.3 DETAILS OF META-EVALUATION SET
The meta-evaluation set samples 100 non-overlapping problems across difficulty levels to assess whether GPT-4o and Omni-Judge align with human judgments. Human annotation involved four student annotators, while the benchmark also documents data collection, classification, and evaluation procedures.
- Meta-evaluation set: 100 non-overlapping problems were sampled across difficulty levels from inference results produced by three models.The sampled outputs came from DeepSeek-Coder-V2, GPT-4o, and Qwen2.5-MATH-72b-Instruct.
- Meta-evaluation set: The evaluation target was the human alignment of GPT-4o and Omni-Judge judgments.All examples were designed for these evaluators to provide reliable human evaluation.
- Human annotation: GPT-4o achieved 97% inter-agreement under conditions of high agreement among human annotators.Two PhD students and two Master’s students each labeled 50 problems, primarily using correct answers to judge model responses.
- Dataset organization: The benchmark’s domain counts can exceed 4,428 because a question may belong to multiple domains.The overall domain tree and its problem counts are shown in Figure 12.
- Dataset organization: Omni-MATH organizes problems through hierarchical domain and difficulty classifications, with domain assignments generated using GPT-4o prompts.The dataset includes a domain tree and difficulty-level assignment procedure, alongside figures documenting the prompts.
- Evaluation setup: The evaluation setup includes baseline models, model-specific prompts, and deterministic decoding settings.Most models use release-page instructions; temperature is 0, top p is 1, and maximum completion length is 2,048 tokens, or 4,096 for o1 models.
G METRICS WITH DISTINCT DIFFICULTY LEVEL
Performance generally declines as Omni-MATH difficulty increases, while the easiest category provides little differentiation among models. GSM8K and MATH correspond mainly to the benchmark’s lowest difficulty levels.
- Difficulty methodology: Difficulty levels are grouped into integer buckets because some decimal levels contain too few instances.For example, 4.75 is grouped into #D5 and 2.25 into #D3.
- Difficulty trends: Model performance consistently declines as difficulty increases, except in #D10, where only 16 instances cause substantial fluctuations.Difficulty levels with few examples can produce unstable estimates.
- Difficulty trends: Nearly all models achieve exceptionally high accuracy on #D1, making that category ineffective for differentiating performance.Performance differences emerge more clearly at higher difficulty levels.
- Benchmark comparison: GSM8K mainly corresponds to #D1 difficulty, whereas MATH aligns with #D2 difficulty.This comparison places both established datasets below much of Omni-MATH’s difficulty range.
H.2 CASE STUDY OF EVALUATION ISSUE
The case study shows why answer-level evaluation is difficult for Olympiad problems: model outputs may require reasoning to recognize equivalent answers and may use complex formats that resist fixed rules. Omni-MATH therefore uses model-based evaluation to cover diverse answer types.
- Model-based evaluation: GPT-4o correctly assesses both cases where rule-based evaluation is impractical.The two cases involve reasoning-based equivalence and complex formatting.
- Reasoning-based equivalence: A model answer of m ≥ 2 is equivalent to the reference answer m ≠ 1 under the problem’s positive-integer domain.Recognizing the equivalence requires reasoning about the stated domain and answer constraints.
- Formatting challenges: Complex formatting can misalign answer order with generated LaTeX tuples, making rule-based evaluation difficult.This is presented as a second evaluation scenario in the case study.
- Answer-format distribution: Number, LaTeX, and Tuple answers comprise 74% of a random sample, leaving a substantial share with more complex formats.The sampled answer categories also include Multi-LaTeX, Function, Multi-Function, and free-form Text.
- Evaluation design: Omni-MATH applies GPT-4 evaluation because fixed rules cannot reliably cover the diverse answer formats in Olympiad-level outputs.The evaluator checks whether the generated solution content aligns with the standard answer.
I FURTHER DISCUSSION ON TEST-TIME SCALING RESULTS
Test-time sampling helps policy models find correct solutions, but reward-model selection becomes harder as the number of sampled reasoning traces grows. This creates a limitation for scaling through reward-model-guided selection.
- Reward-model limitation: RM@256 does not outperform RM@8 on Qwen2.5-MATH-72b-Instruct, suggesting insufficient reward-model supervision for Olympiad-level reasoning tasks.The authors attribute the issue to difficulty selecting the correct reasoning path from numerous candidates.
- Pass@K behavior: After 32 samples, 33.8% of problems still lack a correct solution.Increasing inference samples enables the policy model to solve more problems, but does not eliminate failures.
- Reward-model selection: At 32 samples, the correct-answer proportion among passed cases falls from 100% to 58.5%.The decline accompanies a rising proportion of interference items as sampling increases.
- Reward-model selection: Increasing the sampling count makes it increasingly difficult for the reward model to select the correct chain of thought.The policy model may produce correct traces that the reward model fails to identify among many candidates.
J ANALYSIS OF OMNI-JUDGE
Omni-Judge is evaluated as a lower-cost evaluator for complex mathematical answers, emphasizing both output-format success and agreement with GPT-4o judgments. Its strongest version achieves similar model outcomes and identical capability rankings to GPT-4o.
- Evaluation criteria: Omni-Judge evaluates generated mathematical answers using parsing Success rates and judgment Consistency rates against golden judgments.The evaluation tests both whether judgments follow the required format and whether they are correct.
- Evaluation results: Success rates are approximately 100% across settings, whereas Consistency rates vary substantially by base model.LLaMA-3.1-8b-Instruct performs best overall, while LLaMA-2-7b-Chat performs worst.
- Evaluation results: Omni-Judge trained with LLaMA-3.1-8b-Instruct is selected as the default version because it has the highest Consistency with GPT-4o.The authors attribute this to stronger capture of optimization directions during fine-tuning or greater domain knowledge.
- Ranking comparison: Omni-Judge produces outcomes similar to GPT-4o for each tested model and an identical ranking of model capabilities by predicted accuracy.This comparison supports its usability as a practical judgment model offering reliable feedback.
K THE CONSTRUCTION DETAILS OF OMNI-MATH-RULE
Omni-MATH-Rule filters Omni-MATH outputs into cases suitable for efficient rule-based evaluation, combining rule matching with GPT-4o comparisons and manual annotation. The resulting testable subset contains 2,821 problems, while 1,607 are assigned to the untestable set.
- Subset construction: Omni-MATH-Rule is a filtered subset designed to provide efficient and reliable rule-based evaluation.The filtering retains cases that rules can match effectively while reducing uncertain cases.
- Rule-based pipeline: The rule-based pipeline rewrites matching code using SymPy, filters the full dataset, and extracts answers from O1-mini outputs with the last boxed method.Answers are first constrained to boxed format before rule evaluation.
- Cross-evaluator filtering: Rule-based and GPT-4o-based judgments are compared to identify positive, uncertain, and inconsistent evaluation cases.Positive cases are accepted by both systems, while inconsistent cases are assigned to the untestable set.
- Cross-evaluator filtering: Fewer than 66 of 4,428 cases, or less than 2%, were judged incorrect by GPT-4o despite matching the rules correctly.Thirty of these 66 cases arose from parsing failures that matched both predicted and ground-truth outputs as empty strings.
- Manual annotation: 2,925 problems entered manual annotation, yielding a testable set of 2,821 problems and an untestable set of 1,607 problems.Two PhD students annotated each problem, achieving 98% cross-validation accuracy, or 2,869 of 2,925.
- Validation: Model rankings and accuracies on the testable subset were consistent with GPT-4o evaluation results.This additional validation supports the reliability of the testable set and its rule-based evaluation.